Changeset 290211 in webkit
- Timestamp:
- Feb 19, 2022, 11:24:02 AM (3 years ago)
- Location:
- trunk/Source/WebKit
- Files:
-
- 13 edited
Legend:
- Unmodified
- Added
- Removed
-
trunk/Source/WebKit/ChangeLog
r290203 r290211 1 2022-02-19 Wenson Hsieh <wenson_hsieh@apple.com> 2 3 [macOS] Hovering over "Copy Cropped Image" context menu item should reveal the cropped image 4 https://bugs.webkit.org/show_bug.cgi?id=236845 5 rdar://89152746 6 7 Reviewed by Dean Jackson. 8 9 Add support for revealing the "cropped" region of an image element (while dimming out the rest of the image) 10 when the user highlights the new "Copy Cropped Image" context menu item on macOS. See below for more details. 11 12 * Platform/cocoa/TextRecognitionUtilities.h: 13 14 Adjust `requestImageAnalysisMarkup` to take a completion handler with both the resulting CGImageRef, as well as 15 a CGRect that represents the frame of the cropped image with respect to the source image, normalized to the unit 16 square. Additionally, tweak all call sites of `requestImageAnalysisMarkup` to pass in completion handlers that 17 accept both arguments. 18 19 * Shared/ContextMenuContextData.cpp: 20 (WebKit::ContextMenuContextData::ContextMenuContextData): 21 (WebKit::ContextMenuContextData::encode const): 22 (WebKit::ContextMenuContextData::decode): 23 * Shared/ContextMenuContextData.h: 24 (WebKit::ContextMenuContextData::hitTestedElementContext const): 25 26 Add an optional ElementContext representing the inner hit-tested element for the context menu. We use this below 27 in `WebPageProxy::willHighlightContextMenuItem` to know where we need to install the cropped image overlay. 28 29 * UIProcess/WebPageProxy.cpp: 30 (WebKit::WebPageProxy::didDismissContextMenu): 31 32 Reset the cached cropped image result as well as the cropped image overlay state when the context menu is 33 dismissed. 34 35 (WebKit::WebPageProxy::resetStateAfterProcessExited): 36 37 Also clear the above state if the process crashes. 38 39 * UIProcess/WebPageProxy.h: 40 * UIProcess/ios/WKContentViewInteraction.mm: 41 (-[WKContentView doAfterComputingImageAnalysisResultsForMarkup:]): 42 (-[WKContentView actionSheetAssistant:copyCroppedImage:sourceMIMEType:]): 43 * UIProcess/mac/WebContextMenuProxyMac.mm: 44 (-[WKMenuDelegate menu:willHighlightItem:]): 45 46 Implement this NSMenu delegate method and call out to WebPageProxy whenever the highlighted context menu item 47 changes. See below for more details. 48 49 (WebKit::WebContextMenuProxyMac::applyMarkupToControlledImage): 50 * UIProcess/mac/WebPageProxyMac.mm: 51 (WebKit::WebPageProxy::willHighlightContextMenuItem): 52 53 This method coordinates showing or hiding the cropped image overlay when the user highlights (but does not yet 54 select) the "Copy Cropped Image" context menu item. When the user first highlights this item, we trigger an 55 analysis request in VisionKit; when we later obtain the results, we send the image to the web process as TIFF 56 data, and inject it into the image using `ImageOverlay::CroppedImage::install`. Un-highlighting and re- 57 highlighting the menu item then respectively hides and shows the cropped image overlay by setting visibility. 58 59 (WebKit::WebPageProxy::handleContextMenuCopyCroppedImage): 60 61 Adjust this to use the cached `m_croppedImageResult` if it has already been computed. 62 63 * WebProcess/WebPage/WebContextMenu.cpp: 64 (WebKit::WebContextMenu::show): 65 * WebProcess/WebPage/WebPage.cpp: 66 (WebKit::WebPage::didDismissContextMenu): 67 68 Destroy the `m_croppedImageOverlay` after the context menu is dismissed (which uninstalls the cropped image 69 overlay in the process). 70 71 (WebKit::WebPage::installCroppedImageOverlay): 72 (WebKit::WebPage::setCroppedImageOverlayVisibility): 73 74 Add methods to install and toggle the visibility of the cropped image overlay. 75 76 * WebProcess/WebPage/WebPage.h: 77 * WebProcess/WebPage/WebPage.messages.in: 78 1 79 2022-02-19 Kimmo Kinnunen <kkinnunen@apple.com> 2 80 -
trunk/Source/WebKit/Platform/cocoa/TextRecognitionUtilities.h
r289623 r290211 60 60 #if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS) 61 61 void requestImageAnalysisWithIdentifier(CocoaImageAnalyzer *, const String& identifier, CGImageRef, CompletionHandler<void(WebCore::TextRecognitionResult&&)>&&); 62 void requestImageAnalysisMarkup(CGImageRef, CompletionHandler<void(CGImageRef )>&&);62 void requestImageAnalysisMarkup(CGImageRef, CompletionHandler<void(CGImageRef, CGRect)>&&); 63 63 #endif 64 64 -
trunk/Source/WebKit/Shared/ContextMenuContextData.cpp
r289882 r290211 44 44 } 45 45 46 ContextMenuContextData::ContextMenuContextData(const WebCore::IntPoint& menuLocation, const Vector<WebKit::WebContextMenuItemData>& menuItems, const ContextMenuContext& context)46 ContextMenuContextData::ContextMenuContextData(const IntPoint& menuLocation, std::optional<ElementContext>&& hitTestedElementContext, const Vector<WebKit::WebContextMenuItemData>& menuItems, const ContextMenuContext& context) 47 47 #if ENABLE(SERVICE_CONTROLS) 48 48 : m_type(context.controlledImage() ? Type::ServicesMenu : context.type()) … … 51 51 #endif 52 52 , m_menuLocation(menuLocation) 53 , m_hitTestedElementContext(WTFMove(hitTestedElementContext)) 53 54 , m_menuItems(menuItems) 54 55 , m_webHitTestResultData({ context.hitTestResult(), true }) … … 95 96 encoder << m_type; 96 97 encoder << m_menuLocation; 98 encoder << m_hitTestedElementContext; 97 99 encoder << m_menuItems; 98 100 encoder << m_webHitTestResultData; … … 120 122 121 123 if (!decoder.decode(result.m_menuLocation)) 124 return false; 125 126 if (!decoder.decode(result.m_hitTestedElementContext)) 122 127 return false; 123 128 -
trunk/Source/WebKit/Shared/ContextMenuContextData.h
r289882 r290211 47 47 48 48 ContextMenuContextData(); 49 ContextMenuContextData(const WebCore::IntPoint& menuLocation, const Vector<WebKit::WebContextMenuItemData>& menuItems, const WebCore::ContextMenuContext&);49 ContextMenuContextData(const WebCore::IntPoint& menuLocation, std::optional<WebCore::ElementContext>&& hitTestedElementContext, const Vector<WebKit::WebContextMenuItemData>& menuItems, const WebCore::ContextMenuContext&); 50 50 51 51 Type type() const { return m_type; } … … 53 53 const Vector<WebKit::WebContextMenuItemData>& menuItems() const { return m_menuItems; } 54 54 55 std::optional<WebCore::ElementContext> hitTestedElementContext() const { return m_hitTestedElementContext; } 55 56 const std::optional<WebHitTestResultData>& webHitTestResultData() const { return m_webHitTestResultData; } 56 57 const String& selectedText() const { return m_selectedText; } … … 87 88 88 89 WebCore::IntPoint m_menuLocation; 90 std::optional<WebCore::ElementContext> m_hitTestedElementContext; 89 91 Vector<WebKit::WebContextMenuItemData> m_menuItems; 90 92 -
trunk/Source/WebKit/UIProcess/WebPageProxy.cpp
r290186 r290211 7006 7006 7007 7007 pageClient().didDismissContextMenu(); 7008 7009 #if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS) 7010 m_croppedImageResult = { }; 7011 m_croppedImageOverlayState = CroppedImageOverlayState::Inactive; 7012 #endif 7008 7013 } 7009 7014 … … 8139 8144 m_fullscreenVideoExtractionTimer.stop(); 8140 8145 m_currentFullscreenVideoSessionIdentifier = std::nullopt; 8146 #endif 8147 8148 #if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS) 8149 m_croppedImageResult = { }; 8150 m_croppedImageOverlayState = CroppedImageOverlayState::Inactive; 8141 8151 #endif 8142 8152 -
trunk/Source/WebKit/UIProcess/WebPageProxy.h
r290186 r290211 1372 1372 void contextMenuItemSelected(const WebContextMenuItemData&); 1373 1373 void handleContextMenuKeyEvent(); 1374 void willHighlightContextMenuItem(WebCore::ContextMenuAction); 1374 1375 #endif 1375 1376 … … 3212 3213 RunLoop::Timer<WebPageProxy> m_fullscreenVideoExtractionTimer; 3213 3214 #endif 3215 3216 #if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS) 3217 enum class CroppedImageOverlayState : uint8_t { 3218 Inactive, 3219 Analyzing, 3220 Hidden, 3221 Showing, 3222 }; 3223 WebCore::PlatformImagePtr m_croppedImageResult; 3224 CroppedImageOverlayState m_croppedImageOverlayState { CroppedImageOverlayState::Inactive }; 3225 #endif // ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS) 3214 3226 }; 3215 3227 -
trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.mm
r290009 r290211 4766 4766 } 4767 4767 4768 WebKit::requestImageAnalysisMarkup(cgImage.get(), [sourceMIMEType, context, completion = WTFMove(completion), weakSelf](CGImageRef result ) mutable {4768 WebKit::requestImageAnalysisMarkup(cgImage.get(), [sourceMIMEType, context, completion = WTFMove(completion), weakSelf](CGImageRef result, CGRect) mutable { 4769 4769 auto strongSelf = weakSelf.get(); 4770 4770 if (!strongSelf) { … … 10904 10904 { 10905 10905 auto changeCount = UIPasteboard.generalPasteboard.changeCount; 10906 WebKit::requestImageAnalysisMarkup(image.CGImage, [changeCount, weakSelf = WeakObjCPtr<WKContentView>(self), originalImage = RetainPtr { image }, sourceMIMEType = RetainPtr { sourceMIMEType }](CGImageRef result ) mutable {10906 WebKit::requestImageAnalysisMarkup(image.CGImage, [changeCount, weakSelf = WeakObjCPtr<WKContentView>(self), originalImage = RetainPtr { image }, sourceMIMEType = RetainPtr { sourceMIMEType }](CGImageRef result, CGRect) mutable { 10907 10907 auto strongSelf = weakSelf.get(); 10908 10908 if (!strongSelf) -
trunk/Source/WebKit/UIProcess/mac/WebContextMenuProxyMac.mm
r290103 r290211 187 187 } 188 188 189 - (void)menu:(NSMenu *)menu willHighlightItem:(NSMenuItem *)item 190 { 191 _menuProxy->page()->willHighlightContextMenuItem(static_cast<WebCore::ContextMenuAction>(item.tag)); 192 } 193 189 194 @end 190 195 … … 344 349 return; 345 350 346 requestImageAnalysisMarkup(image.get(), [weakPage = WeakPtr { page() }, preferredMIMEType = m_context.controlledImageMIMEType(), elementContext = WTFMove(*elementContext)](CGImageRef result ) {351 requestImageAnalysisMarkup(image.get(), [weakPage = WeakPtr { page() }, preferredMIMEType = m_context.controlledImageMIMEType(), elementContext = WTFMove(*elementContext)](CGImageRef result, CGRect) { 347 352 RefPtr protectedPage = weakPage.get(); 348 353 if (!protectedPage || !result) -
trunk/Source/WebKit/UIProcess/mac/WebPageProxyMac.mm
r289851 r290211 776 776 #endif // ENABLE(IMAGE_ANALYSIS) 777 777 778 void WebPageProxy::willHighlightContextMenuItem(ContextMenuAction action) 779 { 778 780 #if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS) 781 if (action != ContextMenuItemTagCopyCroppedImage) { 782 if (m_croppedImageOverlayState == CroppedImageOverlayState::Showing) { 783 m_croppedImageOverlayState = CroppedImageOverlayState::Hidden; 784 send(Messages::WebPage::SetCroppedImageOverlayVisibility(false)); 785 } 786 return; 787 } 788 789 if (m_croppedImageOverlayState == CroppedImageOverlayState::Hidden) { 790 m_croppedImageOverlayState = CroppedImageOverlayState::Showing; 791 send(Messages::WebPage::SetCroppedImageOverlayVisibility(true)); 792 return; 793 } 794 795 if (m_croppedImageOverlayState != CroppedImageOverlayState::Inactive) 796 return; 797 798 auto elementContext = m_activeContextMenuContextData.hitTestedElementContext(); 799 if (!elementContext) 800 return; 801 802 auto& hitTestData = m_activeContextMenuContextData.webHitTestResultData().value(); 803 auto imageBitmap = hitTestData.imageBitmap; 804 if (!imageBitmap) 805 return; 806 807 auto image = imageBitmap->makeCGImageCopy(); 808 if (!image) 809 return; 810 811 m_croppedImageOverlayState = CroppedImageOverlayState::Analyzing; 812 813 requestImageAnalysisMarkup(image.get(), [weakPage = WeakPtr { *this }, elementContext = WTFMove(*elementContext)](CGImageRef resultImage, CGRect normalizedCropRect) { 814 if (!resultImage || CGRectIsEmpty(normalizedCropRect)) 815 return; 816 817 RefPtr protectedPage = weakPage.get(); 818 if (!protectedPage) 819 return; 820 821 protectedPage->m_croppedImageResult = resultImage; 822 823 if (protectedPage->m_croppedImageOverlayState != CroppedImageOverlayState::Analyzing) 824 return; 825 826 auto tiffData = transcode(resultImage, (__bridge CFStringRef)UTTypeTIFF.identifier); 827 if (!tiffData) 828 return; 829 830 auto sharedMemory = SharedMemory::allocate([tiffData length]); 831 if (!sharedMemory) 832 return; 833 834 [tiffData getBytes:sharedMemory->data() length:[tiffData length]]; 835 836 SharedMemory::Handle handle; 837 sharedMemory->createHandle(handle, SharedMemory::Protection::ReadOnly); 838 protectedPage->send(Messages::WebPage::InstallCroppedImageOverlay(elementContext, { WTFMove(handle), sharedMemory->size() }, "image/tiff"_s, normalizedCropRect)); 839 protectedPage->m_croppedImageOverlayState = CroppedImageOverlayState::Showing; 840 }); 841 #else 842 UNUSED_PARAM(action); 843 #endif 844 } 845 846 #if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS) 779 847 780 848 void WebPageProxy::handleContextMenuCopyCroppedImage(ShareableBitmap& imageBitmap, const String& preferredMIMEType) 781 849 { 782 auto originalImage = imageBitmap.makeCGImage();783 if (!originalImage)784 return;785 786 850 auto changeCount = NSPasteboard.generalPasteboard.changeCount; 787 requestImageAnalysisMarkup(originalImage.get(), [changeCount, originalImage, preferredMIMEType](CGImageRef resultImage) {851 auto performCopy = [changeCount, preferredMIMEType](CGImageRef resultImage) { 788 852 auto pasteboard = NSPasteboard.generalPasteboard; 789 if (changeCount != pasteboard.changeCount )790 return; 791 792 auto [data, type] = WebKit::transcodeWithPreferredMIMEType(resultImage ?: originalImage.get(), preferredMIMEType.createCFString().get(), (__bridge CFStringRef)UTTypeTIFF.identifier);853 if (changeCount != pasteboard.changeCount || !resultImage) 854 return; 855 856 auto [data, type] = transcodeWithPreferredMIMEType(resultImage, preferredMIMEType.createCFString().get(), (__bridge CFStringRef)UTTypeTIFF.identifier); 793 857 if (!data) 794 858 return; … … 796 860 [pasteboard declareTypes:@[(__bridge NSString *)type.get()] owner:nil]; 797 861 [pasteboard setData:data.get() forType:(__bridge NSString *)type.get()]; 862 }; 863 864 if (m_croppedImageResult) { 865 performCopy(m_croppedImageResult.get()); 866 return; 867 } 868 869 auto originalImage = imageBitmap.makeCGImageCopy(); 870 if (!originalImage) 871 return; 872 873 requestImageAnalysisMarkup(originalImage.get(), [performCopy = WTFMove(performCopy)](auto image, auto) { 874 performCopy(image); 798 875 }); 799 876 } -
trunk/Source/WebKit/WebProcess/WebPage/WebContextMenu.cpp
r284142 r290211 64 64 menuItemsWithUserData(menuItems, userData); 65 65 66 std::optional<ElementContext> hitTestedElementContext; 67 if (RefPtr hitTestedElement = controller.hitTestResult().innerNonSharedElement()) 68 hitTestedElementContext = m_page->contextForElement(*hitTestedElement); 69 66 70 auto menuLocation = view->contentsToRootView(controller.hitTestResult().roundedPointInInnerNodeFrame()); 67 71 68 ContextMenuContextData contextMenuContextData(menuLocation, menuItems, controller.context());72 ContextMenuContextData contextMenuContextData(menuLocation, WTFMove(hitTestedElementContext), menuItems, controller.context()); 69 73 70 74 // Mark the WebPage has having a shown context menu then notify the UIProcess. -
trunk/Source/WebKit/WebProcess/WebPage/WebPage.cpp
r290183 r290211 2906 2906 { 2907 2907 corePage()->contextMenuController().didDismissContextMenu(); 2908 2909 #if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS) 2910 m_croppedImageOverlay = nullptr; 2911 #endif 2908 2912 } 2909 2913 … … 7792 7796 } 7793 7797 7798 #if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS) 7799 7800 void WebPage::installCroppedImageOverlay(const ElementContext& context, const SharedMemory::IPCHandle& imageData, const String& mimeType, FloatRect normalizedCropRect) 7801 { 7802 auto sharedMemory = SharedMemory::map(imageData.handle, SharedMemory::Protection::ReadOnly); 7803 if (!sharedMemory) 7804 return; 7805 7806 RefPtr element = dynamicDowncast<HTMLElement>(elementForContext(context).get()); 7807 if (!element) 7808 return; 7809 7810 m_croppedImageOverlay = ImageOverlay::CroppedImage::install(*element, sharedMemory->createSharedBuffer(imageData.dataSize), mimeType, normalizedCropRect); 7811 } 7812 7813 void WebPage::setCroppedImageOverlayVisibility(bool visible) 7814 { 7815 if (m_croppedImageOverlay) 7816 m_croppedImageOverlay->setVisibility(visible); 7817 } 7818 7819 #endif // ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS) 7820 7794 7821 #if ENABLE(MEDIA_CONTROLS_CONTEXT_MENUS) && USE(UICONTEXTMENU) 7795 7822 void WebPage::showMediaControlsContextMenu(FloatRect&& targetFrame, Vector<MediaControlsContextMenuItem>&& items, CompletionHandler<void(MediaControlsContextMenuItem::ID)>&& completionHandler) -
trunk/Source/WebKit/WebProcess/WebPage/WebPage.h
r290186 r290211 269 269 struct TranslationContextMenuInfo; 270 270 #endif 271 272 namespace ImageOverlay { 273 class CroppedImage; 274 } 271 275 } 272 276 … … 1453 1457 1454 1458 void requestImageBitmap(const WebCore::ElementContext&, CompletionHandler<void(const ShareableBitmap::Handle&, const String& sourceMIMEType)>&&); 1459 #if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS) 1460 void installCroppedImageOverlay(const WebCore::ElementContext&, const SharedMemory::IPCHandle& imageData, const String& mimeType, WebCore::FloatRect normalizedCropRect); 1461 void setCroppedImageOverlayVisibility(bool); 1462 #endif 1455 1463 1456 1464 #if HAVE(TRANSLATION_UI_SERVICES) && ENABLE(CONTEXT_MENUS) … … 2442 2450 WebCore::HighlightVisibility m_appHighlightsVisible { WebCore::HighlightVisibility::Hidden }; 2443 2451 #endif 2452 2453 #if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS) 2454 std::unique_ptr<WebCore::ImageOverlay::CroppedImage> m_croppedImageOverlay; 2455 #endif 2444 2456 }; 2445 2457 -
trunk/Source/WebKit/WebProcess/WebPage/WebPage.messages.in
r290186 r290211 135 135 RequestImageBitmap(struct WebCore::ElementContext elementContext) -> (WebKit::ShareableBitmap::Handle image, String sourceMIMEType) Async 136 136 137 #if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS) 138 InstallCroppedImageOverlay(struct WebCore::ElementContext elementContext, WebKit::SharedMemory::IPCHandle imageData, String mimeType, WebCore::FloatRect normalizedCropRect) 139 SetCroppedImageOverlayVisibility(bool visible) 140 #endif 141 137 142 SetControlledByAutomation(bool controlled) 138 143
Note:
See TracChangeset
for help on using the changeset viewer.