Changeset 226085 in webkit


Ignore:
Timestamp:
Dec 18, 2017 4:19:17 PM (6 years ago)
Author:
Wenson Hsieh
Message:

[Attachment Support] Insert images as inline attachments when pasting and dropping
https://bugs.webkit.org/show_bug.cgi?id=180853
<rdar://problem/35756268>

Reviewed by Tim Horton.

Source/WebCore:

Adds support for transforming dropped and pasted web content fragments prior to insertion, such that inline
elements (so far, only images) are replaced with attachment elements that have an inline representation. See
below comments for more detail.

Tests: WKAttachmentTests.InsertPastedImageAsAttachment

WKAttachmentTests.InsertPastedAttributedStringContainingImage
WKAttachmentTestsMac.InsertPastedFileURLsAsAttachments
WKAttachmentTestsIOS.InsertDroppedImageAsAttachment
WKAttachmentTestsIOS.InsertDroppedAttributedStringContainingAttachment

  • editing/WebContentReader.h:
  • editing/cocoa/EditorCocoa.mm:

(WebCore::Editor::replaceSelectionWithAttributedString):

Add a new helper to replace elements in a DOM fragment with inline attachment elements instead, using the given
Blobs. So far, we only replace image elements with these attachments, by mapping the source of each image to a
Blob, and constructing a replacement attachment backed by a File constructed from the image's corresponding
Blob. However, this mechanism can be generalized in the future to handle transformations from arbitrary elements
to attachment elements capable of representing the same elements using inline display mode.

This function is a noop if the attachment elements are disabled via runtime-enabled features.

  • editing/cocoa/WebContentReaderCocoa.mm:

(WebCore::replaceRichContentWithAttachmentsIfNecessary):
(WebCore::createFragmentAndAddResources):
(WebCore::sanitizeMarkupWithArchive):

Add out-params to both of these helper functions that capture the map of blob URLs to Blobs being used to
replace subresource URLs in the pasted/dropped DOM fragment.

(WebCore::WebContentReader::readWebArchive):
(WebCore::WebContentMarkupReader::readWebArchive):
(WebCore::createFragmentFromAttributedString):
(WebCore::WebContentReader::readRTFD):
(WebCore::WebContentMarkupReader::readRTFD):
(WebCore::WebContentReader::readRTF):
(WebCore::WebContentMarkupReader::readRTF):
(WebCore::WebContentReader::readImage):

In these places where we swap out subresource URLs for blob URLs, collect a map of blob URL => Blob, and use it
to replace images in the DOM with attachments (if needed, and possible).

  • editing/mac/WebContentReaderMac.mm:

(WebCore::WebContentReader::readFilenames):

Augment existing logic to generate attachment elements when pasting or dropping file URLs, so that the generated
attachment elements also have titles, subtitles, and content type information.

  • html/HTMLAttachmentElement.cpp:

(WebCore::HTMLAttachmentElement::setFile):

Add an optional second param, UpdateDisplayAttributes. If UpdateDisplayAttributes::Yes is passed in, then we set
the elements's attributes that are displayed in the attachment representation (title, subtitle, and type) using
the given File.

(WebCore::HTMLAttachmentElement::updateFileWithData):
(WebCore::HTMLAttachmentElement::populateShadowRootIfNecessary):

Make a small tweak to correctly handle the case where an attachment with a content type that is a UTI is being
displayed inline. Content type can either be a UTI or a MIME type, but the code to construct the shadow tree of
an attachment element only handles MIME types.

  • html/HTMLAttachmentElement.h:

Tools:

Adds new WKAttachment API tests and test support. See comments below for more detail.

  • TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm:

Tweak the editing test page markup, such that document.body.innerHTML doesn't additionally contain the contents
of the script tag.

(webViewForTestingAttachments):
(testImageFileURL):
(testImageData):
(testPDFFileURL):
(testPDFData):
(platformCopyRichTextWithAttachment):
(platformCopyPNG):
(platformImageWithData):

Add some platform helper functions, which some of the platform-agnostic tests below use to resolve differences
between iOS and macOS when writing to the pasteboard and creating an image from data.

(TestWebKitAPI::TEST):

Add new API tests to exercise drag and drop/copy and paste of rich content (namely, images and files) on iOS and
macOS. iOS attachment tests use DataInteractionSimulator to simulate UIKit drag and drop coordination. On macOS,
handling of dropped content is much closer to handling of pasted content (they use the same codepaths to read
from the platform pasteboard), so exercising paste codepaths on Mac is sufficient.

  • TestWebKitAPI/ios/DataInteractionSimulator.h:
  • TestWebKitAPI/ios/DataInteractionSimulator.mm:

(-[DataInteractionSimulator _resetSimulatedState]):
(-[DataInteractionSimulator insertedAttachments]):
(-[DataInteractionSimulator removedAttachments]):

Teach the iOS drag and drop simulator to keep track of attachment elements that are inserted during a drop. We
also keep track of removed attachments here too, though no default drop handling scenario should trigger
attachment removal, so we simply use this to check that no _WKAttachments were removed during a drop.

(-[DataInteractionSimulator _webView:didInsertAttachment:]):
(-[DataInteractionSimulator _webView:didRemoveAttachment:]):

Location:
trunk
Files:
11 edited

Legend:

Unmodified
Added
Removed
  • trunk/Source/WebCore/ChangeLog

    r226084 r226085  
     12017-12-18  Wenson Hsieh  <wenson_hsieh@apple.com>
     2
     3        [Attachment Support] Insert images as inline attachments when pasting and dropping
     4        https://bugs.webkit.org/show_bug.cgi?id=180853
     5        <rdar://problem/35756268>
     6
     7        Reviewed by Tim Horton.
     8
     9        Adds support for transforming dropped and pasted web content fragments prior to insertion, such that inline
     10        elements (so far, only images) are replaced with attachment elements that have an inline representation. See
     11        below comments for more detail.
     12
     13        Tests: WKAttachmentTests.InsertPastedImageAsAttachment
     14               WKAttachmentTests.InsertPastedAttributedStringContainingImage
     15               WKAttachmentTestsMac.InsertPastedFileURLsAsAttachments
     16               WKAttachmentTestsIOS.InsertDroppedImageAsAttachment
     17               WKAttachmentTestsIOS.InsertDroppedAttributedStringContainingAttachment
     18
     19        * editing/WebContentReader.h:
     20        * editing/cocoa/EditorCocoa.mm:
     21        (WebCore::Editor::replaceSelectionWithAttributedString):
     22
     23        Add a new helper to replace elements in a DOM fragment with inline attachment elements instead, using the given
     24        Blobs. So far, we only replace image elements with these attachments, by mapping the source of each image to a
     25        Blob, and constructing a replacement attachment backed by a File constructed from the image's corresponding
     26        Blob. However, this mechanism can be generalized in the future to handle transformations from arbitrary elements
     27        to attachment elements capable of representing the same elements using inline display mode.
     28
     29        This function is a noop if the attachment elements are disabled via runtime-enabled features.
     30
     31        * editing/cocoa/WebContentReaderCocoa.mm:
     32        (WebCore::replaceRichContentWithAttachmentsIfNecessary):
     33        (WebCore::createFragmentAndAddResources):
     34        (WebCore::sanitizeMarkupWithArchive):
     35
     36        Add out-params to both of these helper functions that capture the map of blob URLs to Blobs being used to
     37        replace subresource URLs in the pasted/dropped DOM fragment.
     38
     39        (WebCore::WebContentReader::readWebArchive):
     40        (WebCore::WebContentMarkupReader::readWebArchive):
     41        (WebCore::createFragmentFromAttributedString):
     42        (WebCore::WebContentReader::readRTFD):
     43        (WebCore::WebContentMarkupReader::readRTFD):
     44        (WebCore::WebContentReader::readRTF):
     45        (WebCore::WebContentMarkupReader::readRTF):
     46        (WebCore::WebContentReader::readImage):
     47
     48        In these places where we swap out subresource URLs for blob URLs, collect a map of blob URL => Blob, and use it
     49        to replace images in the DOM with attachments (if needed, and possible).
     50
     51        * editing/mac/WebContentReaderMac.mm:
     52        (WebCore::WebContentReader::readFilenames):
     53
     54        Augment existing logic to generate attachment elements when pasting or dropping file URLs, so that the generated
     55        attachment elements also have titles, subtitles, and content type information.
     56
     57        * html/HTMLAttachmentElement.cpp:
     58        (WebCore::HTMLAttachmentElement::setFile):
     59
     60        Add an optional second param, UpdateDisplayAttributes. If UpdateDisplayAttributes::Yes is passed in, then we set
     61        the elements's attributes that are displayed in the attachment representation (title, subtitle, and type) using
     62        the given File.
     63
     64        (WebCore::HTMLAttachmentElement::updateFileWithData):
     65        (WebCore::HTMLAttachmentElement::populateShadowRootIfNecessary):
     66
     67        Make a small tweak to correctly handle the case where an attachment with a content type that is a UTI is being
     68        displayed inline. Content type can either be a UTI or a MIME type, but the code to construct the shadow tree of
     69        an attachment element only handles MIME types.
     70
     71        * html/HTMLAttachmentElement.h:
     72
    1732017-12-18  Youenn Fablet  <youenn@apple.com>
    274
  • trunk/Source/WebCore/editing/WebContentReader.h

    r223678 r226085  
    3333
    3434class ArchiveResource;
     35class Blob;
    3536
    3637class FrameWebContentReader : public PasteboardWebContentReader {
     
    106107};
    107108
    108 RefPtr<DocumentFragment> createFragmentAndAddResources(Frame&, NSAttributedString*);
     109void replaceRichContentWithAttachmentsIfNecessary(DocumentFragment&, HashMap<AtomicString, RefPtr<Blob>>&& urlToBlobMap);
     110RefPtr<DocumentFragment> createFragmentAndAddResources(Frame&, NSAttributedString*, HashMap<AtomicString, RefPtr<Blob>>& urlToBlobMap);
    109111#endif
    110112
  • trunk/Source/WebCore/editing/cocoa/EditorCocoa.mm

    r225264 r226085  
    223223
    224224    if (m_frame.selection().selection().isContentRichlyEditable()) {
    225         RefPtr<DocumentFragment> fragment = createFragmentAndAddResources(m_frame, attributedString);
    226         if (fragment && shouldInsertFragment(*fragment, selectedRange().get(), EditorInsertAction::Pasted))
    227             pasteAsFragment(fragment.releaseNonNull(), false, false, mailBlockquoteHandling);
     225        HashMap<AtomicString, RefPtr<Blob>> urlToBlobMap;
     226        if (auto fragment = createFragmentAndAddResources(m_frame, attributedString, urlToBlobMap)) {
     227            replaceRichContentWithAttachmentsIfNecessary(*fragment, WTFMove(urlToBlobMap));
     228            if (shouldInsertFragment(*fragment, selectedRange().get(), EditorInsertAction::Pasted))
     229                pasteAsFragment(fragment.releaseNonNull(), false, false, mailBlockquoteHandling);
     230        }
    228231    } else {
    229232        String text = attributedString.string;
  • trunk/Source/WebCore/editing/cocoa/WebContentReaderCocoa.mm

    r225675 r226085  
    3535#import "DocumentFragment.h"
    3636#import "DocumentLoader.h"
     37#import "File.h"
    3738#import "Frame.h"
    3839#import "FrameLoader.h"
    3940#import "FrameLoaderClient.h"
     41#import "HTMLAttachmentElement.h"
    4042#import "HTMLBodyElement.h"
    4143#import "HTMLIFrameElement.h"
     
    4547#import "Page.h"
    4648#import "PublicURLManager.h"
     49#import "RuntimeEnabledFeatures.h"
    4750#import "Settings.h"
    4851#import "SocketProvider.h"
     
    5356#import <pal/spi/cocoa/NSAttributedStringSPI.h>
    5457#import <wtf/SoftLinking.h>
     58#import <wtf/UUID.h>
    5559
    5660#if (PLATFORM(IOS) && __IPHONE_OS_VERSION_MIN_REQUIRED >= 110000) || (PLATFORM(MAC) && __MAC_OS_X_VERSION_MIN_REQUIRED >= 101300)
     
    6973
    7074namespace WebCore {
     75
     76#if ENABLE(ATTACHMENT_ELEMENT)
     77
     78void replaceRichContentWithAttachmentsIfNecessary(DocumentFragment& fragment, HashMap<AtomicString, RefPtr<Blob>>&& urlToBlobMap)
     79{
     80    if (!RuntimeEnabledFeatures::sharedFeatures().attachmentElementEnabled() || urlToBlobMap.isEmpty())
     81        return;
     82
     83    Vector<Ref<Element>> elementsToRemove;
     84    Vector<std::pair<Ref<File>, Ref<Element>>> filesForElementsToReplace;
     85    for (auto& image : descendantsOfType<HTMLImageElement>(fragment)) {
     86        auto url = image.attributeWithoutSynchronization(HTMLNames::srcAttr);
     87
     88        if (url.isEmpty()) {
     89            elementsToRemove.append(image);
     90            continue;
     91        }
     92
     93        auto blob = urlToBlobMap.get(url);
     94        if (!blob) {
     95            elementsToRemove.append(image);
     96            continue;
     97        }
     98
     99        auto title = image.attributeWithoutSynchronization(HTMLNames::titleAttr);
     100        if (title.isEmpty())
     101            title = AtomicString("media");
     102
     103        filesForElementsToReplace.append({ File::create(*blob, title), image });
     104    }
     105
     106    for (auto& fileAndElement : filesForElementsToReplace) {
     107        auto& file = fileAndElement.first;
     108        auto& elementToReplace = fileAndElement.second;
     109        auto parent = makeRefPtr(elementToReplace->parentNode());
     110        if (!parent)
     111            continue;
     112
     113        auto attachment = HTMLAttachmentElement::create(HTMLNames::attachmentTag, fragment.document());
     114        attachment->setUniqueIdentifier(createCanonicalUUIDString());
     115        attachment->setFile(WTFMove(file), HTMLAttachmentElement::UpdateDisplayAttributes::Yes);
     116        attachment->updateDisplayMode(AttachmentDisplayMode::InPlace);
     117        parent->replaceChild(attachment, elementToReplace);
     118    }
     119
     120    for (auto& elementToRemove : elementsToRemove)
     121        elementToRemove->remove();
     122}
     123
     124#else
     125
     126void replaceRichContentWithAttachmentsIfNecessary(DocumentFragment&, HashMap<AtomicString, RefPtr<Blob>>&&)
     127{
     128}
     129
     130#endif
    71131
    72132#if (PLATFORM(IOS) && __IPHONE_OS_VERSION_MIN_REQUIRED >= 110000) || (PLATFORM(MAC) && __MAC_OS_X_VERSION_MIN_REQUIRED >= 101300)
     
    160220};
    161221
    162 RefPtr<DocumentFragment> createFragmentAndAddResources(Frame& frame, NSAttributedString *string)
     222RefPtr<DocumentFragment> createFragmentAndAddResources(Frame& frame, NSAttributedString *string, HashMap<AtomicString, RefPtr<Blob>>& urlToBlobMap)
    163223{
    164224    if (!frame.page() || !frame.document())
     
    179239        String blobURL = DOMURL::createObjectURL(document, blob);
    180240        blobURLMap.set(subresource->url().string(), blobURL);
     241        urlToBlobMap.set(blobURL, WTFMove(blob));
    181242    }
    182243    replaceSubresourceURLs(*fragmentAndResources.fragment, WTFMove(blobURLMap));
     
    208269}
    209270
    210 static String sanitizeMarkupWithArchive(Document& destinationDocument, MarkupAndArchive& markupAndArchive, const std::function<bool(const String)>& canShowMIMETypeAsHTML)
     271static String sanitizeMarkupWithArchive(Document& destinationDocument, MarkupAndArchive& markupAndArchive, const std::function<bool(const String)>& canShowMIMETypeAsHTML, HashMap<AtomicString, RefPtr<Blob>>& urlToBlobMap)
    211272{
    212273    auto page = createPageForSanitizingWebContent();
     
    220281        String blobURL = DOMURL::createObjectURL(destinationDocument, blob);
    221282        blobURLMap.set(subresource->url().string(), blobURL);
     283        urlToBlobMap.set(blobURL, WTFMove(blob));
    222284    }
    223285
     
    235297        MarkupAndArchive subframeContent = { String::fromUTF8(subframeMainResource->data().data(), subframeMainResource->data().size()),
    236298            subframeMainResource.releaseNonNull(), subframeArchive.copyRef() };
    237         auto subframeMarkup = sanitizeMarkupWithArchive(destinationDocument, subframeContent, canShowMIMETypeAsHTML);
     299        auto subframeMarkup = sanitizeMarkupWithArchive(destinationDocument, subframeContent, canShowMIMETypeAsHTML, urlToBlobMap);
    238300
    239301        CString utf8 = subframeMarkup.utf8();
     
    245307        String subframeBlobURL = DOMURL::createObjectURL(destinationDocument, blob);
    246308        blobURLMap.set(subframeURL.string(), subframeBlobURL);
     309        urlToBlobMap.set(subframeBlobURL, WTFMove(blob));
    247310    }
    248311
     
    275338    }
    276339
     340    HashMap<AtomicString, RefPtr<Blob>> urlToBlobMap;
    277341    String sanitizedMarkup = sanitizeMarkupWithArchive(*frame.document(), *result, [&] (const String& type) {
    278342        return frame.loader().client().canShowMIMETypeAsHTML(type);
    279     });
     343    }, urlToBlobMap);
    280344    fragment = createFragmentFromMarkup(*frame.document(), sanitizedMarkup, blankURL(), DisallowScriptingAndPluginContent);
    281345
     346    if (!fragment)
     347        return false;
     348
     349    replaceRichContentWithAttachmentsIfNecessary(*fragment, WTFMove(urlToBlobMap));
    282350    return true;
    283351}
     
    299367    }
    300368
     369    HashMap<AtomicString, RefPtr<Blob>> urlToBlobMap;
    301370    markup = sanitizeMarkupWithArchive(*frame.document(), *result, [&] (const String& type) {
    302371        return frame.loader().client().canShowMIMETypeAsHTML(type);
    303     });
     372    }, urlToBlobMap);
    304373
    305374    return true;
     
    349418}
    350419
     420static RefPtr<DocumentFragment> createFragmentFromAttributedString(Frame& frame, NSAttributedString *string)
     421{
     422    HashMap<AtomicString, RefPtr<Blob>> urlToBlobMap;
     423    auto fragment = createFragmentAndAddResources(frame, string, urlToBlobMap);
     424    if (!fragment)
     425        return nullptr;
     426
     427    replaceRichContentWithAttachmentsIfNecessary(*fragment, WTFMove(urlToBlobMap));
     428    return fragment;
     429}
     430
    351431bool WebContentReader::readRTFD(SharedBuffer& buffer)
    352432{
     
    354434        return false;
    355435
    356     auto fragment = createFragmentAndAddResources(frame, adoptNS([[NSAttributedString alloc] initWithRTFD:buffer.createNSData().get() documentAttributes:nullptr]).get());
     436    auto string = adoptNS([[NSAttributedString alloc] initWithRTFD:buffer.createNSData().get() documentAttributes:nullptr]);
     437    auto fragment = createFragmentFromAttributedString(frame, string.get());
    357438    if (!fragment)
    358439        return false;
     
    366447    if (!frame.document())
    367448        return false;
    368     auto fragment = createFragmentAndAddResources(frame, adoptNS([[NSAttributedString alloc] initWithRTFD:buffer.createNSData().get() documentAttributes:nullptr]).get());
     449    auto string = adoptNS([[NSAttributedString alloc] initWithRTFD:buffer.createNSData().get() documentAttributes:nullptr]);
     450    auto fragment = createFragmentFromAttributedString(frame, string.get());
     451    if (!fragment)
     452        return false;
     453
    369454    markup = createMarkup(*fragment);
    370455    return true;
     
    376461        return false;
    377462
    378     auto fragment = createFragmentAndAddResources(frame, adoptNS([[NSAttributedString alloc] initWithRTF:buffer.createNSData().get() documentAttributes:nullptr]).get());
     463    auto string = adoptNS([[NSAttributedString alloc] initWithRTF:buffer.createNSData().get() documentAttributes:nullptr]);
     464    auto fragment = createFragmentFromAttributedString(frame, string.get());
    379465    if (!fragment)
    380466        return false;
     
    388474    if (!frame.document())
    389475        return false;
    390     auto fragment = createFragmentAndAddResources(frame, adoptNS([[NSAttributedString alloc] initWithRTF:buffer.createNSData().get() documentAttributes:nullptr]).get());
     476    auto string = adoptNS([[NSAttributedString alloc] initWithRTF:buffer.createNSData().get() documentAttributes:nullptr]);
     477    auto fragment = createFragmentFromAttributedString(frame, string.get());
    391478    if (!fragment)
    392479        return false;
     
    413500    String blobURL = DOMURL::createObjectURL(document, blob);
    414501    addFragment(createFragmentForImageAndURL(document, blobURL));
    415     return true;
    416 }
    417 
    418 }
     502
     503    if (!fragment)
     504        return false;
     505
     506    replaceRichContentWithAttachmentsIfNecessary(*fragment, {{ blobURL, WTFMove(blob) }});
     507    return true;
     508}
     509
     510}
  • trunk/Source/WebCore/editing/mac/WebContentReaderMac.mm

    r224755 r226085  
    4848#import "WebCoreNSURLExtras.h"
    4949#import "markup.h"
     50#import <wtf/UUID.h>
    5051
    5152namespace WebCore {
     
    6667        if (RuntimeEnabledFeatures::sharedFeatures().attachmentElementEnabled()) {
    6768            auto attachment = HTMLAttachmentElement::create(HTMLNames::attachmentTag, document);
    68             attachment->setFile(File::create([[NSURL fileURLWithPath:text] path]));
     69            attachment->setUniqueIdentifier(createCanonicalUUIDString());
     70            attachment->setFile(File::create([NSURL fileURLWithPath:text].path), HTMLAttachmentElement::UpdateDisplayAttributes::Yes);
    6971            fragment->appendChild(attachment);
    7072            continue;
  • trunk/Source/WebCore/html/HTMLAttachmentElement.cpp

    r225422 r226085  
    4646#include <pal/FileSizeFormatter.h>
    4747
     48#if PLATFORM(COCOA)
     49#include "UTIUtilities.h"
     50#endif
     51
    4852namespace WebCore {
    4953
     
    113117}
    114118
    115 void HTMLAttachmentElement::setFile(RefPtr<File>&& file)
     119void HTMLAttachmentElement::setFile(RefPtr<File>&& file, UpdateDisplayAttributes updateAttributes)
    116120{
    117121    m_file = WTFMove(file);
    118122
    119123    setAttributeWithoutSynchronization(HTMLNames::webkitattachmentbloburlAttr, m_file ? m_file->url() : emptyString());
     124
     125    if (updateAttributes == UpdateDisplayAttributes::Yes) {
     126        if (m_file) {
     127            setAttributeWithoutSynchronization(HTMLNames::titleAttr, m_file->name());
     128            setAttributeWithoutSynchronization(HTMLNames::subtitleAttr, fileSizeDescription(m_file->size()));
     129            setAttributeWithoutSynchronization(HTMLNames::typeAttr, m_file->type());
     130        } else {
     131            removeAttribute(HTMLNames::titleAttr);
     132            removeAttribute(HTMLNames::subtitleAttr);
     133            removeAttribute(HTMLNames::typeAttr);
     134        }
     135    }
    120136
    121137    if (auto* renderAttachment = attachmentRenderer())
     
    224240    auto contentType = newContentType ? *newContentType : File::contentTypeForFile(filename);
    225241    auto file = File::create(Blob::create(WTFMove(data), contentType), filename);
    226 
    227     setAttributeWithoutSynchronization(titleAttr, filename);
    228     setAttributeWithoutSynchronization(subtitleAttr, fileSizeDescription(file->size()));
    229     setAttributeWithoutSynchronization(typeAttr, contentType);
    230     setFile(WTFMove(file));
     242    setFile(WTFMove(file), UpdateDisplayAttributes::Yes);
    231243}
    232244
     
    267279void HTMLAttachmentElement::populateShadowRootIfNecessary()
    268280{
     281    if (!m_file)
     282        return;
     283
    269284    auto mimeType = attachmentType();
    270     if (!m_file || mimeType.isEmpty())
     285
     286#if PLATFORM(COCOA)
     287    if (isDeclaredUTI(mimeType))
     288        mimeType = MIMETypeFromUTI(mimeType);
     289#endif
     290
     291    if (mimeType.isEmpty())
    271292        return;
    272293
  • trunk/Source/WebCore/html/HTMLAttachmentElement.h

    r225422 r226085  
    4646    WEBCORE_EXPORT URL blobURL() const;
    4747    WEBCORE_EXPORT File* file() const;
    48     void setFile(RefPtr<File>&&);
     48
     49    enum class UpdateDisplayAttributes { No, Yes };
     50    void setFile(RefPtr<File>&&, UpdateDisplayAttributes = UpdateDisplayAttributes::No);
    4951
    5052    WEBCORE_EXPORT String uniqueIdentifier() const;
  • trunk/Tools/ChangeLog

    r226062 r226085  
     12017-12-18  Wenson Hsieh  <wenson_hsieh@apple.com>
     2
     3        [Attachment Support] Insert images as inline attachments when pasting and dropping
     4        https://bugs.webkit.org/show_bug.cgi?id=180853
     5        <rdar://problem/35756268>
     6
     7        Reviewed by Tim Horton.
     8
     9        Adds new WKAttachment API tests and test support. See comments below for more detail.
     10
     11        * TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm:
     12
     13        Tweak the editing test page markup, such that document.body.innerHTML doesn't additionally contain the contents
     14        of the script tag.
     15
     16        (webViewForTestingAttachments):
     17        (testImageFileURL):
     18        (testImageData):
     19        (testPDFFileURL):
     20        (testPDFData):
     21        (platformCopyRichTextWithAttachment):
     22        (platformCopyPNG):
     23        (platformImageWithData):
     24
     25        Add some platform helper functions, which some of the platform-agnostic tests below use to resolve differences
     26        between iOS and macOS when writing to the pasteboard and creating an image from data.
     27
     28        (TestWebKitAPI::TEST):
     29
     30        Add new API tests to exercise drag and drop/copy and paste of rich content (namely, images and files) on iOS and
     31        macOS. iOS attachment tests use DataInteractionSimulator to simulate UIKit drag and drop coordination. On macOS,
     32        handling of dropped content is much closer to handling of pasted content (they use the same codepaths to read
     33        from the platform pasteboard), so exercising paste codepaths on Mac is sufficient.
     34
     35        * TestWebKitAPI/ios/DataInteractionSimulator.h:
     36        * TestWebKitAPI/ios/DataInteractionSimulator.mm:
     37        (-[DataInteractionSimulator _resetSimulatedState]):
     38        (-[DataInteractionSimulator insertedAttachments]):
     39        (-[DataInteractionSimulator removedAttachments]):
     40
     41        Teach the iOS drag and drop simulator to keep track of attachment elements that are inserted during a drop. We
     42        also keep track of removed attachments here too, though no default drop handling scenario should trigger
     43        attachment removal, so we simply use this to check that no _WKAttachments were removed during a drop.
     44
     45        (-[DataInteractionSimulator _webView:didInsertAttachment:]):
     46        (-[DataInteractionSimulator _webView:didRemoveAttachment:]):
     47
    1482017-12-18  Fujii Hironori  <Hironori.Fujii@sony.com>
    249
  • trunk/Tools/TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm

    r225422 r226085  
    2626#import "config.h"
    2727
     28#import "DataInteractionSimulator.h"
    2829#import "PlatformUtilities.h"
    2930#import "TestWKWebView.h"
     31#import <WebKit/WKPreferencesRefPrivate.h>
    3032#import <WebKit/WKWebViewPrivate.h>
    3133#import <WebKit/WebKit.h>
    3234#import <WebKit/WebKitPrivate.h>
    3335#import <wtf/RetainPtr.h>
     36
     37#if PLATFORM(IOS)
     38#import <MobileCoreServices/MobileCoreServices.h>
     39#endif
    3440
    3541#if WK_API_ENABLED
     
    123129    auto configuration = adoptNS([[WKWebViewConfiguration alloc] init]);
    124130    [configuration _setAttachmentElementEnabled:YES];
     131    WKPreferencesSetCustomPasteboardDataEnabled((WKPreferencesRef)[configuration preferences], YES);
    125132
    126133    auto webView = adoptNS([[TestWKWebView alloc] initWithFrame:CGRectMake(0, 0, 320, 500) configuration:configuration.get()]);
    127     [webView synchronouslyLoadHTMLString:@"<body contenteditable></body><script>document.body.focus();</script>"];
     134    [webView synchronouslyLoadHTMLString:@"<script>focus = () => document.body.focus()</script><body onload=focus() contenteditable></body>"];
    128135
    129136    return webView;
     
    135142}
    136143
     144static NSURL *testImageFileURL()
     145{
     146    return [[NSBundle mainBundle] URLForResource:@"icon" withExtension:@"png" subdirectory:@"TestWebKitAPI.resources"];
     147}
     148
    137149static NSData *testImageData()
    138150{
    139     NSURL *url = [[NSBundle mainBundle] URLForResource:@"icon" withExtension:@"png" subdirectory:@"TestWebKitAPI.resources"];
    140     return [NSData dataWithContentsOfURL:url];
     151    return [NSData dataWithContentsOfURL:testImageFileURL()];
    141152}
    142153
     
    147158}
    148159
     160static NSURL *testPDFFileURL()
     161{
     162    return [[NSBundle mainBundle] URLForResource:@"test" withExtension:@"pdf" subdirectory:@"TestWebKitAPI.resources"];
     163}
     164
    149165static NSData *testPDFData()
    150166{
    151     NSURL *url = [[NSBundle mainBundle] URLForResource:@"test" withExtension:@"pdf" subdirectory:@"TestWebKitAPI.resources"];
    152     return [NSData dataWithContentsOfURL:url];
     167    return [NSData dataWithContentsOfURL:testPDFFileURL()];
    153168}
    154169
     
    292307@end
    293308
     309#pragma mark - Platform testing helper functions
     310
     311void platformCopyRichTextWithImage()
     312{
     313    auto richText = adoptNS([[NSMutableAttributedString alloc] init]);
     314    auto image = adoptNS([[NSTextAttachment alloc] initWithData:testImageData() ofType:(NSString *)kUTTypePNG]);
     315
     316    [richText appendAttributedString:[[[NSAttributedString alloc] initWithString:@"Lorem ipsum "] autorelease]];
     317    [richText appendAttributedString:[NSAttributedString attributedStringWithAttachment:image.get()]];
     318    [richText appendAttributedString:[[[NSAttributedString alloc] initWithString:@" dolor sit amet."] autorelease]];
     319
     320#if PLATFORM(MAC)
     321    NSPasteboard *pasteboard = [NSPasteboard generalPasteboard];
     322    [pasteboard clearContents];
     323    [pasteboard writeObjects:@[ richText.get() ]];
     324#elif PLATFORM(IOS)
     325    auto item = adoptNS([[NSItemProvider alloc] init]);
     326    [item registerObject:richText.get() visibility:NSItemProviderRepresentationVisibilityAll];
     327    [UIPasteboard generalPasteboard].itemProviders = @[ item.get() ];
     328#endif
     329}
     330
     331typedef void(^ItemProviderDataLoadHandler)(NSData *, NSError *);
     332
     333void platformCopyPNG()
     334{
     335#if PLATFORM(MAC)
     336    NSPasteboard *pasteboard = [NSPasteboard generalPasteboard];
     337    [pasteboard declareTypes:@[NSPasteboardTypePNG] owner:nil];
     338    [pasteboard setData:testImageData() forType:NSPasteboardTypePNG];
     339#elif PLATFORM(IOS)
     340    UIPasteboard *pasteboard = [UIPasteboard generalPasteboard];
     341    auto item = adoptNS([[UIItemProvider alloc] init]);
     342    [item setPreferredPresentationStyle:UIPreferredPresentationStyleAttachment];
     343    [item registerDataRepresentationForTypeIdentifier:(NSString *)kUTTypePNG visibility:NSItemProviderRepresentationVisibilityAll loadHandler:[] (ItemProviderDataLoadHandler completionHandler) -> NSProgress * {
     344        completionHandler(testImageData(), nil);
     345        return nil;
     346    }];
     347    pasteboard.itemProviders = @[ item.get() ];
     348#endif
     349}
     350
     351#if PLATFORM(MAC)
     352typedef NSImage PlatformImage;
     353#else
     354typedef UIImage PlatformImage;
     355#endif
     356
     357PlatformImage *platformImageWithData(NSData *data)
     358{
     359#if PLATFORM(MAC)
     360    return [[[NSImage alloc] initWithData:data] autorelease];
     361#else
     362    return [UIImage imageWithData:data];
     363#endif
     364}
     365
    294366namespace TestWebKitAPI {
     367
     368#pragma mark - Platform-agnostic tests
    295369
    296370TEST(WKAttachmentTests, AttachmentElementInsertion)
     
    676750}
    677751
     752TEST(WKAttachmentTests, InsertPastedImageAsAttachment)
     753{
     754    platformCopyPNG();
     755
     756    RetainPtr<_WKAttachment> attachment;
     757    auto webView = webViewForTestingAttachments();
     758    {
     759        ObserveAttachmentUpdatesForScope observer(webView.get());
     760        [webView _synchronouslyExecuteEditCommand:@"Paste" argument:nil];
     761        EXPECT_EQ(1U, observer.observer().inserted.count);
     762        attachment = observer.observer().inserted[0];
     763    }
     764
     765    auto size = platformImageWithData([attachment synchronouslyRequestData:nil]).size;
     766    EXPECT_EQ(215., size.width);
     767    EXPECT_EQ(174., size.height);
     768
     769    {
     770        ObserveAttachmentUpdatesForScope observer(webView.get());
     771        [webView _synchronouslyExecuteEditCommand:@"SelectAll" argument:nil];
     772        [webView _synchronouslyExecuteEditCommand:@"DeleteBackward" argument:nil];
     773        observer.expectAttachmentUpdates(@[attachment.get()], @[]);
     774    }
     775}
     776
     777TEST(WKAttachmentTests, InsertPastedAttributedStringContainingImage)
     778{
     779    platformCopyRichTextWithImage();
     780
     781    RetainPtr<_WKAttachment> attachment;
     782    auto webView = webViewForTestingAttachments();
     783    {
     784        ObserveAttachmentUpdatesForScope observer(webView.get());
     785        [webView _synchronouslyExecuteEditCommand:@"Paste" argument:nil];
     786        EXPECT_EQ(0U, observer.observer().removed.count);
     787        EXPECT_EQ(1U, observer.observer().inserted.count);
     788        attachment = observer.observer().inserted[0];
     789    }
     790
     791    [attachment expectRequestedDataToBe:testImageData()];
     792    EXPECT_WK_STREQ("Lorem ipsum  dolor sit amet.", [webView stringByEvaluatingJavaScript:@"document.body.textContent"]);
     793    EXPECT_WK_STREQ("image/png", [webView valueOfAttribute:@"type" forQuerySelector:@"attachment"]);
     794
     795    {
     796        ObserveAttachmentUpdatesForScope observer(webView.get());
     797        [webView _synchronouslyExecuteEditCommand:@"SelectAll" argument:nil];
     798        [webView _synchronouslyExecuteEditCommand:@"DeleteBackward" argument:nil];
     799        observer.expectAttachmentUpdates(@[attachment.get()], @[]);
     800    }
     801}
     802
     803#pragma mark - Platform-specific tests
     804
     805#if PLATFORM(MAC)
     806
     807TEST(WKAttachmentTestsMac, InsertPastedFileURLsAsAttachments)
     808{
     809    NSPasteboard *pasteboard = [NSPasteboard generalPasteboard];
     810    [pasteboard clearContents];
     811    [pasteboard declareTypes:@[NSFilenamesPboardType] owner:nil];
     812    [pasteboard setPropertyList:@[testPDFFileURL().path, testImageFileURL().path] forType:NSFilenamesPboardType];
     813
     814    RetainPtr<NSArray<_WKAttachment *>> insertedAttachments;
     815    auto webView = webViewForTestingAttachments();
     816    {
     817        ObserveAttachmentUpdatesForScope observer(webView.get());
     818        [webView _synchronouslyExecuteEditCommand:@"Paste" argument:nil];
     819        insertedAttachments = [observer.observer() inserted];
     820        EXPECT_EQ(2U, [insertedAttachments count]);
     821    }
     822
     823    NSArray<NSData *> *expectedAttachmentData = @[ testPDFData(), testImageData() ];
     824    EXPECT_TRUE([expectedAttachmentData containsObject:[[insertedAttachments firstObject] synchronouslyRequestData:nil]]);
     825    EXPECT_TRUE([expectedAttachmentData containsObject:[[insertedAttachments lastObject] synchronouslyRequestData:nil]]);
     826    EXPECT_WK_STREQ("application/pdf", [webView stringByEvaluatingJavaScript:@"document.querySelectorAll('attachment')[0].getAttribute('type')"]);
     827    EXPECT_WK_STREQ("test.pdf", [webView stringByEvaluatingJavaScript:@"document.querySelectorAll('attachment')[0].getAttribute('title')"]);
     828    EXPECT_WK_STREQ("image/png", [webView stringByEvaluatingJavaScript:@"document.querySelectorAll('attachment')[1].getAttribute('type')"]);
     829    EXPECT_WK_STREQ("icon.png", [webView stringByEvaluatingJavaScript:@"document.querySelectorAll('attachment')[1].getAttribute('title')"]);
     830
     831    {
     832        ObserveAttachmentUpdatesForScope observer(webView.get());
     833        [webView _synchronouslyExecuteEditCommand:@"SelectAll" argument:nil];
     834        [webView _synchronouslyExecuteEditCommand:@"DeleteBackward" argument:nil];
     835        NSArray<_WKAttachment *> *removedAttachments = [observer.observer() removed];
     836        EXPECT_EQ(2U, removedAttachments.count);
     837        EXPECT_TRUE([removedAttachments containsObject:[insertedAttachments firstObject]]);
     838        EXPECT_TRUE([removedAttachments.lastObject isEqual:[insertedAttachments lastObject]]);
     839    }
     840}
     841
     842#endif // PLATFORM(MAC)
     843
     844#if PLATFORM(IOS)
     845
     846TEST(WKAttachmentTestsIOS, InsertDroppedImageAsAttachment)
     847{
     848    auto webView = webViewForTestingAttachments();
     849    auto draggingSimulator = adoptNS([[DataInteractionSimulator alloc] initWithWebView:webView.get()]);
     850    auto item = adoptNS([[NSItemProvider alloc] init]);
     851    [item setPreferredPresentationStyle:UIPreferredPresentationStyleAttachment];
     852    [item registerDataRepresentationForTypeIdentifier:(NSString *)kUTTypePNG visibility:NSItemProviderRepresentationVisibilityAll loadHandler:[] (ItemProviderDataLoadHandler completionHandler) -> NSProgress * {
     853        completionHandler(testImageData(), nil);
     854        return nil;
     855    }];
     856    [draggingSimulator setExternalItemProviders:@[ item.get() ]];
     857    [draggingSimulator runFrom:CGPointZero to:CGPointMake(50, 50)];
     858
     859    EXPECT_EQ(1U, [draggingSimulator insertedAttachments].count);
     860    EXPECT_EQ(0U, [draggingSimulator removedAttachments].count);
     861    auto attachment = retainPtr([draggingSimulator insertedAttachments].firstObject);
     862    [attachment expectRequestedDataToBe:testImageData()];
     863    EXPECT_WK_STREQ("public.png", [webView valueOfAttribute:@"type" forQuerySelector:@"attachment"]);
     864
     865    {
     866        ObserveAttachmentUpdatesForScope observer(webView.get());
     867        [webView _synchronouslyExecuteEditCommand:@"SelectAll" argument:nil];
     868        [webView _synchronouslyExecuteEditCommand:@"DeleteBackward" argument:nil];
     869        observer.expectAttachmentUpdates(@[attachment.get()], @[]);
     870    }
     871}
     872
     873TEST(WKAttachmentTestsIOS, InsertDroppedAttributedStringContainingAttachment)
     874{
     875    auto webView = webViewForTestingAttachments();
     876    auto draggingSimulator = adoptNS([[DataInteractionSimulator alloc] initWithWebView:webView.get()]);
     877    auto image = adoptNS([[NSTextAttachment alloc] initWithData:testImageData() ofType:(NSString *)kUTTypePNG]);
     878    auto item = adoptNS([[NSItemProvider alloc] init]);
     879    [item registerObject:[NSAttributedString attributedStringWithAttachment:image.get()] visibility:NSItemProviderRepresentationVisibilityAll];
     880
     881    [draggingSimulator setExternalItemProviders:@[ item.get() ]];
     882    [draggingSimulator runFrom:CGPointZero to:CGPointMake(50, 50)];
     883
     884    EXPECT_EQ(1U, [draggingSimulator insertedAttachments].count);
     885    EXPECT_EQ(0U, [draggingSimulator removedAttachments].count);
     886    auto attachment = retainPtr([draggingSimulator insertedAttachments].firstObject);
     887
     888    auto size = platformImageWithData([attachment synchronouslyRequestData:nil]).size;
     889    EXPECT_EQ(215., size.width);
     890    EXPECT_EQ(174., size.height);
     891    EXPECT_WK_STREQ("image/png", [webView valueOfAttribute:@"type" forQuerySelector:@"attachment"]);
     892
     893    {
     894        ObserveAttachmentUpdatesForScope observer(webView.get());
     895        [webView _synchronouslyExecuteEditCommand:@"SelectAll" argument:nil];
     896        [webView _synchronouslyExecuteEditCommand:@"DeleteBackward" argument:nil];
     897        observer.expectAttachmentUpdates(@[attachment.get()], @[]);
     898    }
     899}
     900
     901#endif // PLATFORM(IOS)
     902
    678903} // namespace TestWebKitAPI
    679904
  • trunk/Tools/TestWebKitAPI/ios/DataInteractionSimulator.h

    r221907 r226085  
    139139    RetainPtr<NSMutableArray<UITargetedDragPreview *>> _liftPreviews;
    140140
     141    RetainPtr<NSMutableArray<_WKAttachment *>> _insertedAttachments;
     142    RetainPtr<NSMutableArray<_WKAttachment *>> _removedAttachments;
     143
    141144    bool _isDoneWaitingForInputSession;
    142145    BOOL _shouldPerformOperation;
     
    169172@property (nonatomic, readonly) NSArray<UITargetedDragPreview *> *liftPreviews;
    170173
     174@property (nonatomic, readonly) NSArray<_WKAttachment *> *insertedAttachments;
     175@property (nonatomic, readonly) NSArray<_WKAttachment *> *removedAttachments;
     176
    171177@end
    172178
  • trunk/Tools/TestWebKitAPI/ios/DataInteractionSimulator.mm

    r224517 r226085  
    318318    _isDoneWithCurrentRun = false;
    319319    _observedEventNames = adoptNS([[NSMutableArray alloc] init]);
     320    _insertedAttachments = adoptNS([[NSMutableArray alloc] init]);
     321    _removedAttachments = adoptNS([[NSMutableArray alloc] init]);
    320322    _finalSelectionRects = @[ ];
    321323    _dragSession = nil;
     
    577579}
    578580
     581- (NSArray<_WKAttachment *> *)insertedAttachments
     582{
     583    return _insertedAttachments.get();
     584}
     585
     586- (NSArray<_WKAttachment *> *)removedAttachments
     587{
     588    return _removedAttachments.get();
     589}
     590
    579591#pragma mark - WKUIDelegatePrivate
    580592
     
    618630}
    619631
     632- (void)_webView:(WKWebView *)webView didInsertAttachment:(_WKAttachment *)attachment
     633{
     634    [_insertedAttachments addObject:attachment];
     635}
     636
     637- (void)_webView:(WKWebView *)webView didRemoveAttachment:(_WKAttachment *)attachment
     638{
     639    [_removedAttachments addObject:attachment];
     640}
     641
    620642#pragma mark - _WKInputDelegate
    621643
Note: See TracChangeset for help on using the changeset viewer.