⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 293730 in webkit


Ignore:
Timestamp:
May 3, 2022, 10:37:47 AM (4 years ago)
Author:
commit-queue@webkit.org
Message:

[iOS] The "Copy Cropped Image" context menu action should be gated on cropped image results
​https://bugs.webkit.org/show_bug.cgi?id=240013
rdar://88941787

Reviewed by Tim Horton.

Only show this item in the context menu when long pressing in the case where requestImageAnalysisMarkup
computes a non-null result for the given image bitmap. This gating logic runs alongside existing gating logic
for both the visual search item ("Look Up") and "Show Text" actions. See below for more details.

  • UIProcess/ios/WKContentViewInteraction.h:
  • UIProcess/ios/WKContentViewInteraction.mm:

(-[WKContentView _setUpImageAnalysis]):
(-[WKContentView _tearDownImageAnalysis]):

Introduce a _croppedImageResult ivar to cache the CGImage result after running image analysis over the image
corresponding to the element for which we're showing the context menu. This is reset in the same lifecycle as
the extant _hasVisualSearchResults and _hasSelectableTextInImage flags which are used for the same purpose.

(-[WKContentView imageAnalysisGestureDidBegin:]):
(-[WKContentView _completeImageAnalysisRequestForContextMenu:requestIdentifier:hasTextResults:]):

This is the codepath that currently prevents us from showing the context menu until we know whether or not there
are relevant visual search results, such that we can conditionally show the "Look Up" context menu item. Adjust
this so that it calls -_invokeAllActionsToPerformAfterPendingImageAnalysis: only after we've also determined
whether or not there is a non-null cropped image result, so that we can also conditionally show the "Copy
Cropped Image" item.

To achieve this, we move the call to -_invokeAllActionsToPerformAfterPendingImageAnalysis: into a
WTF::CallbackAggregator, and ref/deref the aggregator when invoking both of the async image analysis
operations. When the callback aggregator is destroyed (i.e., after both async image analysis operations are
complete), we proceed with showing the context menu.

(-[WKContentView imageAnalysisGestureDidTimeOut:]):

Implement similar logic as above, but for the scenario where we show the context menu after the user continues
to long press after selecting text inside of an image.

(-[WKContentView actionSheetAssistantShouldIncludeCopyCroppedImageAction:]):

Only show the item if _croppedImageResult is non-null.

(-[WKContentView actionSheetAssistant:copyCroppedImage:sourceMIMEType:]):

Instead of running image analysis and writing the resulting image to the clipboard, simply transcode the cached
image in _croppedImageResult.

Canonical link: ​https://commits.webkit.org/250218@main

Location:
trunk/Source/WebKit
Files:
3 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebKit/ChangeLog

    r293729 r293730  
     12022-05-03  Wenson Hsieh  <wenson_hsieh@apple.com>
     2
     3        [iOS] The "Copy Cropped Image" context menu action should be gated on cropped image results
     4        https://bugs.webkit.org/show_bug.cgi?id=240013
     5        rdar://88941787
     6
     7        Reviewed by Tim Horton.
     8
     9        Only show this item in the context menu when long pressing in the case where `requestImageAnalysisMarkup`
     10        computes a non-null result for the given image bitmap. This gating logic runs alongside existing gating logic
     11        for both the visual search item ("Look Up") and "Show Text" actions. See below for more details.
     12
     13        * UIProcess/ios/WKContentViewInteraction.h:
     14        * UIProcess/ios/WKContentViewInteraction.mm:
     15        (-[WKContentView _setUpImageAnalysis]):
     16        (-[WKContentView _tearDownImageAnalysis]):
     17
     18        Introduce a `_croppedImageResult` ivar to cache the CGImage result after running image analysis over the image
     19        corresponding to the element for which we're showing the context menu. This is reset in the same lifecycle as
     20        the extant `_hasVisualSearchResults` and `_hasSelectableTextInImage` flags which are used for the same purpose.
     21
     22        (-[WKContentView imageAnalysisGestureDidBegin:]):
     23        (-[WKContentView _completeImageAnalysisRequestForContextMenu:requestIdentifier:hasTextResults:]):
     24
     25        This is the codepath that currently prevents us from showing the context menu until we know whether or not there
     26        are relevant visual search results, such that we can conditionally show the "Look Up" context menu item. Adjust
     27        this so that it calls `-_invokeAllActionsToPerformAfterPendingImageAnalysis:` only after we've also determined
     28        whether or not there is a non-null cropped image result, so that we can also conditionally show the "Copy
     29        Cropped Image" item.
     30
     31        To achieve this, we move the call to `-_invokeAllActionsToPerformAfterPendingImageAnalysis:` into a
     32        `WTF::CallbackAggregator`, and ref/deref the aggregator when invoking both of the async image analysis
     33        operations. When the callback aggregator is destroyed (i.e., after both async image analysis operations are
     34        complete), we proceed with showing the context menu.
     35
     36        (-[WKContentView imageAnalysisGestureDidTimeOut:]):
     37
     38        Implement similar logic as above, but for the scenario where we show the context menu after the user continues
     39        to long press after selecting text inside of an image.
     40
     41        (-[WKContentView actionSheetAssistantShouldIncludeCopyCroppedImageAction:]):
     42
     43        Only show the item if `_croppedImageResult` is non-null.
     44
     45        (-[WKContentView actionSheetAssistant:copyCroppedImage:sourceMIMEType:]):
     46
     47        Instead of running image analysis and writing the resulting image to the clipboard, simply transcode the cached
     48        image in `_croppedImageResult`.
     49
    1502022-05-03  Philippe Normand  <pnormand@igalia.com> and Yury Semikhatsky <yurys@chromium.org>
    251

  • trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.h

    r293231 r293730  
    546546    WebCore::FloatRect _imageAnalysisInteractionBounds;
    547547    std::optional<WebKit::ImageAnalysisMarkupData> _imageAnalysisMarkupData;
     548    RetainPtr<CGImageRef> _croppedImageResult;
    548549#endif
    549550}

  • trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.mm

    r293663 r293730  
    149149#import <wtf/BlockObjCExceptions.h>
    150150#import <wtf/BlockPtr.h>
     151#import <wtf/CallbackAggregator.h>
    151152#import <wtf/Scope.h>
    152153#import <wtf/SetForScope.h>
    … …  
    1071510716#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
    1071610717    _imageAnalysisMarkupData = std::nullopt;
     10718    _croppedImageResult = nil;
    1071710719#endif
    1071810720}
    … …  
    1074410746    [self uninstallImageAnalysisInteraction];
    1074510747    _imageAnalysisMarkupData = std::nullopt;
     10748    _croppedImageResult = nil;
    1074610749#endif
    1074710750}
    … …  
    1083810841    _hasVisualSearchResults = NO;
    1083910842#endif // USE(QUICK_LOOK)
     10843
     10844#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
     10845    _croppedImageResult = nil;
     10846#endif
    1084010847
    1084110848#if USE(UICONTEXTMENU) && ENABLE(IMAGE_ANALYSIS_FOR_MACHINE_READABLE_CODES)
    … …  
    1088810895
    1088910896        auto requestForTextSelection = [strongSelf createImageAnalyzerRequest:VKAnalysisTypeText image:cgImage.get()];
    10890         auto requestForContextMenu = [strongSelf createImageAnalyzerRequest:VKAnalysisTypeVisualSearch | VKAnalysisTypeMachineReadableCode | VKAnalysisTypeAppClip image:cgImage.get()];
    10891 
    1089210897        if (information.elementContainsImageOverlay) {
    10893             [strongSelf _completeImageAnalysisRequestForContextMenu:requestForContextMenu.get() requestIdentifier:requestIdentifier hasTextResults:YES];
     10898            [strongSelf _completeImageAnalysisRequestForContextMenu:cgImage.get() requestIdentifier:requestIdentifier hasTextResults:YES];
    1089410899            return;
    1089510900        }
    1089610901
    1089710902        auto textAnalysisStartTime = MonotonicTime::now();
    10898         [[strongSelf imageAnalyzer] processRequest:requestForTextSelection.get() progressHandler:nil completionHandler:[requestIdentifier = WTFMove(requestIdentifier), weakSelf, elementContext, requestLocation, requestForContextMenu, gestureDeferralToken, textAnalysisStartTime] (CocoaImageAnalysis *result, NSError *error) mutable {
     10903        [[strongSelf imageAnalyzer] processRequest:requestForTextSelection.get() progressHandler:nil completionHandler:[requestIdentifier = WTFMove(requestIdentifier), weakSelf, elementContext, requestLocation, cgImage, gestureDeferralToken, textAnalysisStartTime] (CocoaImageAnalysis *result, NSError *error) mutable {
    1089910904            auto strongSelf = weakSelf.get();
    1090010905            if (![strongSelf validateImageAnalysisRequestIdentifier:requestIdentifier])
    … …  
    1090410909            RELEASE_LOG(Images, "Image analysis completed in %.0f ms (request %" PRIu64 "; found text? %d)", (MonotonicTime::now() - textAnalysisStartTime).milliseconds(), requestIdentifier.toUInt64(), hasTextResults);
    1090510910
    10906             strongSelf->_page->updateWithTextRecognitionResult(WebKit::makeTextRecognitionResult(result), elementContext, requestLocation, [requestIdentifier = WTFMove(requestIdentifier), weakSelf, hasTextResults, requestForContextMenu, gestureDeferralToken] (WebKit::TextRecognitionUpdateResult updateResult) mutable {
     10911            strongSelf->_page->updateWithTextRecognitionResult(WebKit::makeTextRecognitionResult(result), elementContext, requestLocation, [requestIdentifier = WTFMove(requestIdentifier), weakSelf, hasTextResults, cgImage, gestureDeferralToken] (WebKit::TextRecognitionUpdateResult updateResult) mutable {
    1090710912                auto strongSelf = weakSelf.get();
    1090810913                if (![strongSelf validateImageAnalysisRequestIdentifier:requestIdentifier])
    … …  
    1092210927                }
    1092310928
    10924                 [strongSelf _completeImageAnalysisRequestForContextMenu:requestForContextMenu.get() requestIdentifier:requestIdentifier hasTextResults:hasTextResults];
     10929                [strongSelf _completeImageAnalysisRequestForContextMenu:cgImage.get() requestIdentifier:requestIdentifier hasTextResults:hasTextResults];
    1092510930            });
    1092610931        }];
    … …  
    1092810933}
    1092910934
    10930 - (void)_completeImageAnalysisRequestForContextMenu:(CocoaImageAnalyzerRequest *)requestForContextMenu requestIdentifier:(WebKit::ImageAnalysisRequestIdentifier)requestIdentifier hasTextResults:(BOOL)hasTextResults
    10931 {
     10935- (void)_completeImageAnalysisRequestForContextMenu:(CGImageRef)image requestIdentifier:(WebKit::ImageAnalysisRequestIdentifier)requestIdentifier hasTextResults:(BOOL)hasTextResults
     10936{
     10937#if USE(QUICK_LOOK)
     10938    _hasSelectableTextInImage = hasTextResults;
     10939#endif
     10940
     10941    auto weakSelf = WeakObjCPtr<WKContentView>(self);
     10942    auto aggregator = CallbackAggregator::create([weakSelf, requestIdentifier]() mutable {
     10943        if (auto strongSelf = weakSelf.get(); [strongSelf validateImageAnalysisRequestIdentifier:requestIdentifier])
     10944            [strongSelf _invokeAllActionsToPerformAfterPendingImageAnalysis:WebKit::ProceedWithTextSelectionInImage::No];
     10945    });
     10946
     10947    auto request = [self createImageAnalyzerRequest:VKAnalysisTypeVisualSearch | VKAnalysisTypeMachineReadableCode | VKAnalysisTypeAppClip image:image];
    1093210948    auto visualSearchAnalysisStartTime = MonotonicTime::now();
    10933     [self.imageAnalyzer processRequest:requestForContextMenu progressHandler:nil completionHandler:[requestIdentifier = WTFMove(requestIdentifier), weakSelf = WeakObjCPtr<WKContentView>(self), hasTextResults, visualSearchAnalysisStartTime] (CocoaImageAnalysis *result, NSError *error) mutable {
     10949    [self.imageAnalyzer processRequest:request.get() progressHandler:nil completionHandler:[requestIdentifier = WTFMove(requestIdentifier), weakSelf, visualSearchAnalysisStartTime, aggregator = aggregator.copyRef()] (CocoaImageAnalysis *result, NSError *error) mutable {
    1093410950        auto strongSelf = weakSelf.get();
    1093510951        if (![strongSelf validateImageAnalysisRequestIdentifier:requestIdentifier])
    … …  
    1094210958        UNUSED_PARAM(visualSearchAnalysisStartTime);
    1094310959#endif
    10944         if (!result || error) {
    10945             [strongSelf _invokeAllActionsToPerformAfterPendingImageAnalysis:WebKit::ProceedWithTextSelectionInImage::No];
     10960        if (!result || error)
    1094610961            return;
    10947         }
    1094810962
    1094910963#if USE(QUICK_LOOK)
    10950         strongSelf->_hasSelectableTextInImage = hasTextResults;
    1095110964        strongSelf->_hasVisualSearchResults = hasVisualSearchResults;
    10952 #else
    10953         UNUSED_PARAM(hasTextResults);
    1095410965#endif
    1095510966#if USE(UICONTEXTMENU) && ENABLE(IMAGE_ANALYSIS_FOR_MACHINE_READABLE_CODES)
    1095610967        [strongSelf _updateContextMenuForMachineReadableCodeForImageAnalysis:result];
    10957 #endif // USE(UICONTEXTMENU) && ENABLE(IMAGE_ANALYSIS_FOR_MACHINE_READABLE_CODES)
    10958         [strongSelf _invokeAllActionsToPerformAfterPendingImageAnalysis:WebKit::ProceedWithTextSelectionInImage::No];
     10968#endif
    1095910969    }];
     10970
     10971#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
     10972    if (_page->preferences().imageAnalysisMarkupEnabled()) {
     10973        WebKit::requestImageAnalysisMarkup(image, [weakSelf = WeakObjCPtr<WKContentView>(self), aggregator = aggregator.copyRef()](CGImageRef result, CGRect) mutable {
     10974            if (auto strongSelf = weakSelf.get())
     10975                strongSelf->_croppedImageResult = result;
     10976        });
     10977    }
     10978#endif
    1096010979}
    1096110980
    … …  
    1099811017        // making redundant image analysis requests for the same image data.
    1099911018
     11019        auto aggregator = CallbackAggregator::create([weakSelf, location]() mutable {
     11020            auto strongSelf = weakSelf.get();
     11021            if (!strongSelf)
     11022                return;
     11023
     11024            strongSelf->_contextMenuWasTriggeredByImageAnalysisTimeout = YES;
     11025            [strongSelf presentContextMenu:strongSelf->_contextMenuInteraction.get() atLocation:location];
     11026        });
     11027
    1100011028        auto visualSearchAnalysisStartTime = MonotonicTime::now();
    1100111029        auto requestForContextMenu = [strongSelf createImageAnalyzerRequest:VKAnalysisTypeVisualSearch | VKAnalysisTypeMachineReadableCode | VKAnalysisTypeAppClip image:cgImage.get()];
    11002         [[strongSelf imageAnalyzer] processRequest:requestForContextMenu.get() progressHandler:nil completionHandler:[weakSelf, location, visualSearchAnalysisStartTime] (CocoaImageAnalysis *result, NSError *error) {
     11030        [[strongSelf imageAnalyzer] processRequest:requestForContextMenu.get() progressHandler:nil completionHandler:[weakSelf, visualSearchAnalysisStartTime, aggregator = aggregator.copyRef()] (CocoaImageAnalysis *result, NSError *error) {
    1100311031            auto strongSelf = weakSelf.get();
    1100411032            if (!strongSelf)
    … …  
    1101411042#endif
    1101511043
    11016 #if USE(UICONTEXTMENU)
    11017 #if ENABLE(IMAGE_ANALYSIS_FOR_MACHINE_READABLE_CODES)
     11044#if USE(UICONTEXTMENU) && ENABLE(IMAGE_ANALYSIS_FOR_MACHINE_READABLE_CODES)
    1101811045            [strongSelf _updateContextMenuForMachineReadableCodeForImageAnalysis:result];
    11019 #endif // ENABLE(IMAGE_ANALYSIS_FOR_MACHINE_READABLE_CODES)
    11020             strongSelf->_contextMenuWasTriggeredByImageAnalysisTimeout = YES;
    11021             [strongSelf presentContextMenu:strongSelf->_contextMenuInteraction.get() atLocation:location];
    11022 #else
    11023             UNUSED_PARAM(location);
    11024 #endif // USE(UICONTEXTMENU)
     11046#endif
    1102511047        }];
     11048
     11049#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
     11050        if (strongSelf->_page->preferences().imageAnalysisMarkupEnabled()) {
     11051            WebKit::requestImageAnalysisMarkup(cgImage.get(), [weakSelf, aggregator = aggregator.copyRef()](CGImageRef result, CGRect) mutable {
     11052                if (auto strongSelf = weakSelf.get())
     11053                    strongSelf->_croppedImageResult = result;
     11054            });
     11055        }
     11056#endif
    1102611057    } forRequest:request];
    1102711058}
    … …  
    1103811069- (BOOL)actionSheetAssistantShouldIncludeCopyCroppedImageAction:(WKActionSheetAssistant *)assistant
    1103911070{
    11040     return _page->preferences().imageAnalysisMarkupEnabled();
     11071    return !!_croppedImageResult;
    1104111072}
    1104211073
    1104311074- (void)actionSheetAssistant:(WKActionSheetAssistant *)assistant copyCroppedImage:(UIImage *)image sourceMIMEType:(NSString *)sourceMIMEType
    1104411075{
    11045     auto changeCount = UIPasteboard.generalPasteboard.changeCount;
    11046     WebKit::requestImageAnalysisMarkup(image.CGImage, [changeCount, weakSelf = WeakObjCPtr<WKContentView>(self), originalImage = RetainPtr { image }, sourceMIMEType = RetainPtr { sourceMIMEType }](CGImageRef result, CGRect) mutable {
    11047         if (!result)
    11048             return;
    11049 
    11050         auto strongSelf = weakSelf.get();
    11051         if (!strongSelf)
    11052             return;
    11053 
    11054         auto [data, type] = WebKit::imageDataForCroppedImageResult(result, (__bridge CFStringRef)sourceMIMEType.get());
    11055         if (!data)
    11056             return;
    11057 
    11058         [UIPasteboard _performAsDataOwner:[strongSelf _dataOwnerForCopy] block:[data = WTFMove(data), type = WTFMove(type), changeCount] {
    11059             auto pasteboard = UIPasteboard.generalPasteboard;
    11060             if (changeCount == pasteboard.changeCount)
    11061                 [pasteboard setData:data.get() forPasteboardType:(__bridge NSString *)type.get()];
    11062         }];
    11063     });
     11076    if (!_croppedImageResult)
     11077        return;
     11078
     11079    auto [data, type] = WebKit::imageDataForCroppedImageResult(_croppedImageResult.get(), (__bridge CFStringRef)sourceMIMEType);
     11080    if (!data)
     11081        return;
     11082
     11083    [UIPasteboard _performAsDataOwner:self._dataOwnerForCopy block:[data = WTFMove(data), type = WTFMove(type)] {
     11084        [UIPasteboard.generalPasteboard setData:data.get() forPasteboardType:(__bridge NSString *)type.get()];
     11085    }];
    1106411086}
    1106511087
Note: See TracChangeset for help on using the changeset viewer.