⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 292184 in webkit


Ignore:
Timestamp:
Mar 31, 2022, 2:58:55 PM (5 years ago)
Author:
Wenson Hsieh
Message:

[iOS] Add support for temporarily installing VKCImageAnalysisInteraction on WKContentView
​https://bugs.webkit.org/show_bug.cgi?id=238582

Reviewed by Tim Horton.

Source/WebCore/PAL:

Soft link against the image analysis interaction class.

  • pal/cocoa/VisionKitCoreSoftLink.h:
  • pal/cocoa/VisionKitCoreSoftLink.mm:

Source/WebKit:

Add helper methods for temporarily creating and installing an image analysis interaction on the root view. See
below for more details.

  • UIProcess/ios/WKContentViewInteraction.h:
  • UIProcess/ios/WKContentViewInteraction.mm:

(-[WKContentView hasSelectablePositionAtPoint:]):
(-[WKContentView pointIsNearMarkedText:]):
(-[WKContentView textInteractionGesture:shouldBeginAtPoint:]):

If Live Text exists at the given location, suppress default WebKit text selection behaviors in favor of Live
Text instead.

(-[WKContentView _singleTapRecognized:]):

Since we only allow the image analysis interaction to begin when interacting directly with regions of the
content view that contain Live Text, we need to (manually) clear the native text selection here when handling a
tap over other parts of the web view that are not covered by Live Text.

(-[WKContentView _didCommitLoadForMainFrame]):

Uninstall the interaction when loading a new page.

(-[WKContentView gestureRecognizer:shouldIgnoreWebTouchWithEvent:]):
(-[WKContentView deferringGestureRecognizer:willBeginTouchesWithEvent:]):

Don't allow touch events to even begin, if the gesture starts over a region of Live Text.

(-[WKContentView _tearDownImageAnalysis]):

Uninstall the interaction when the web process terminates or swaps.

(-[WKContentView installImageAnalysisInteraction:]):
(-[WKContentView uninstallImageAnalysisInteraction]):

These are the two main helper methods for adding or removing the image analysis interaction.

(-[WKContentView contentsRectForImageAnalysisInteraction:]):
(-[WKContentView imageAnalysisInteraction:shouldBeginAtPoint:forAnalysisType:]):

Only allow Live Text interaction gestures to begin if there's interactable content at the given location. The
only reason why this isn't default behavior already is that the image analysis interaction handles taps that
happen outside of Live Text by clearing any active selection; to ensure that this continues to work, we handle
the tap ourselves, and clear the native selection directly (see -_singleTapRecognized: above).

Location:
trunk/Source
Files:
6 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebCore/PAL/ChangeLog

    r292035 r292184  
     12022-03-31  Wenson Hsieh  <wenson_hsieh@apple.com>
     2
     3        [iOS] Add support for temporarily installing VKCImageAnalysisInteraction on WKContentView
     4        https://bugs.webkit.org/show_bug.cgi?id=238582
     5
     6        Reviewed by Tim Horton.
     7
     8        Soft link against the image analysis interaction class.
     9
     10        * pal/cocoa/VisionKitCoreSoftLink.h:
     11        * pal/cocoa/VisionKitCoreSoftLink.mm:
     12
    1132022-03-28  Simon Fraser  <simon.fraser@apple.com>
    214

  • trunk/Source/WebCore/PAL/pal/cocoa/VisionKitCoreSoftLink.h

    r289763 r292184  
    3737SOFT_LINK_CLASS_FOR_HEADER(PAL, VKCImageAnalyzerRequest)
    3838SOFT_LINK_CLASS_FOR_HEADER(PAL, VKCImageAnalysis)
     39#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
     40SOFT_LINK_CLASS_FOR_HEADER(PAL, VKCImageAnalysisInteraction)
     41#endif
    3942
    4043#if USE(APPLE_INTERNAL_SDK)

  • trunk/Source/WebCore/PAL/pal/cocoa/VisionKitCoreSoftLink.mm

    r289763 r292184  
    3737SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT_AND_IS_OPTIONAL(PAL, VisionKitCore, VKCImageAnalyzerRequest, PAL_EXPORT, true)
    3838SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT_AND_IS_OPTIONAL(PAL, VisionKitCore, VKCImageAnalysis, PAL_EXPORT, true)
     39#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
     40SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT_AND_IS_OPTIONAL(PAL, VisionKitCore, VKCImageAnalysisInteraction, PAL_EXPORT, true)
     41#endif
    3942
    4043#if USE(APPLE_INTERNAL_SDK)

  • trunk/Source/WebKit/ChangeLog

    r292183 r292184  
     12022-03-31  Wenson Hsieh  <wenson_hsieh@apple.com>
     2
     3        [iOS] Add support for temporarily installing VKCImageAnalysisInteraction on WKContentView
     4        https://bugs.webkit.org/show_bug.cgi?id=238582
     5
     6        Reviewed by Tim Horton.
     7
     8        Add helper methods for temporarily creating and installing an image analysis interaction on the root view. See
     9        below for more details.
     10
     11        * UIProcess/ios/WKContentViewInteraction.h:
     12        * UIProcess/ios/WKContentViewInteraction.mm:
     13        (-[WKContentView hasSelectablePositionAtPoint:]):
     14        (-[WKContentView pointIsNearMarkedText:]):
     15        (-[WKContentView textInteractionGesture:shouldBeginAtPoint:]):
     16
     17        If Live Text exists at the given location, suppress default WebKit text selection behaviors in favor of Live
     18        Text instead.
     19
     20        (-[WKContentView _singleTapRecognized:]):
     21
     22        Since we only allow the image analysis interaction to begin when interacting directly with regions of the
     23        content view that contain Live Text, we need to (manually) clear the native text selection here when handling a
     24        tap over other parts of the web view that are not covered by Live Text.
     25
     26        (-[WKContentView _didCommitLoadForMainFrame]):
     27
     28        Uninstall the interaction when loading a new page.
     29
     30        (-[WKContentView gestureRecognizer:shouldIgnoreWebTouchWithEvent:]):
     31        (-[WKContentView deferringGestureRecognizer:willBeginTouchesWithEvent:]):
     32
     33        Don't allow touch events to even begin, if the gesture starts over a region of Live Text.
     34
     35        (-[WKContentView _tearDownImageAnalysis]):
     36
     37        Uninstall the interaction when the web process terminates or swaps.
     38
     39        (-[WKContentView installImageAnalysisInteraction:]):
     40        (-[WKContentView uninstallImageAnalysisInteraction]):
     41
     42        These are the two main helper methods for adding or removing the image analysis interaction.
     43
     44        (-[WKContentView contentsRectForImageAnalysisInteraction:]):
     45        (-[WKContentView imageAnalysisInteraction:shouldBeginAtPoint:forAnalysisType:]):
     46
     47        Only allow Live Text interaction gestures to begin if there's interactable content at the given location. The
     48        only reason why this isn't default behavior already is that the image analysis interaction handles taps that
     49        happen outside of Live Text by clearing any active selection; to ensure that this continues to work, we handle
     50        the tap ourselves, and clear the native selection directly (see `-_singleTapRecognized:` above).
     51
    1522022-03-31  Elliott Williams  <emw@apple.com>
    253

  • trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.h

    r291770 r292184  
    113113@class AVPlayerViewController;
    114114@class QLPreviewController;
     115@class VKCImageAnalysisInteraction;
    115116@class WebEvent;
    116117@class WebTextIndicatorLayer;
    … …  
    541542    uint32_t _fullscreenVideoExtractionRequestIdentifier;
    542543#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
     544    RetainPtr<VKCImageAnalysisInteraction> _imageAnalysisInteraction;
     545    WebCore::FloatRect _imageAnalysisInteractionBounds;
    543546    std::optional<WebKit::ImageAnalysisMarkupData> _imageAnalysisMarkupData;
    544547#endif

  • trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.mm

    r291863 r292184  
    232232#endif // ENABLE(IMAGE_ANALYSIS)
    233233
     234#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
     235
     236@interface WKContentView (ImageAnalysisInteraction) <VKCImageAnalysisInteractionDelegate>
     237@end
     238
     239#endif
     240
    234241#if USE(APPLE_INTERNAL_SDK)
    235242#import <WebKitAdditions/WKContentViewInteractionAdditions.mm>
    … …  
    29812988- (BOOL)hasSelectablePositionAtPoint:(CGPoint)point
    29822989{
     2990#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
     2991    if ([_imageAnalysisInteraction interactableItemExistsAtPoint:point])
     2992        return NO;
     2993#endif
     2994
    29832995    ALLOW_DEPRECATED_DECLARATIONS_BEGIN
    29842996    if (!self.webView.configuration._textInteractionGesturesEnabled)
    … …  
    30093021- (BOOL)pointIsNearMarkedText:(CGPoint)point
    30103022{
     3023#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
     3024    if ([_imageAnalysisInteraction interactableItemExistsAtPoint:point])
     3025        return NO;
     3026#endif
     3027
    30113028    ALLOW_DEPRECATED_DECLARATIONS_BEGIN
    30123029    if (!self.webView.configuration._textInteractionGesturesEnabled)
    … …  
    30283045- (BOOL)textInteractionGesture:(UIWKGestureType)gesture shouldBeginAtPoint:(CGPoint)point
    30293046{
     3047#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
     3048    if ([_imageAnalysisInteraction interactableItemExistsAtPoint:point])
     3049        return NO;
     3050#endif
     3051
    30303052    ALLOW_DEPRECATED_DECLARATIONS_BEGIN
    30313053    if (!self.webView.configuration._textInteractionGesturesEnabled)
    … …  
    33673389    if (!_isExpectingFastSingleTapCommit)
    33683390        [self _finishInteraction];
     3391
     3392#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
     3393    if (![_imageAnalysisInteraction interactableItemExistsAtPoint:_lastInteractionLocation])
     3394        [_imageAnalysisInteraction resetSelection];
     3395#endif
    33693396}
    33703397
    … …  
    50565083#if HAVE(UIKIT_WITH_MOUSE_SUPPORT)
    50575084    [_hoverPlatter dismissPlatterWithAnimation:NO];
     5085#endif
     5086
     5087#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
     5088    [self uninstallImageAnalysisInteraction];
    50585089#endif
    50595090}
    … …  
    78787909    _touchEventsCanPreventNativeGestures = YES;
    78797910
     7911#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
     7912    if ([_imageAnalysisInteraction interactableItemExistsAtPoint:[gestureRecognizer locationInView:self]])
     7913        return YES;
     7914#endif
     7915
    78807916    return [self gestureRecognizer:gestureRecognizer isInterruptingMomentumScrollingWithEvent:event];
    78817917}
    … …  
    82878323{
    82888324    self.gestureRecognizerConsistencyEnforcer.beginTracking(deferringGestureRecognizer);
     8325
     8326#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
     8327    if ([_imageAnalysisInteraction interactableItemExistsAtPoint:[deferringGestureRecognizer locationInView:self]])
     8328        return WebKit::ShouldDeferGestures::No;
     8329#endif
    82898330
    82908331    return [self gestureRecognizer:deferringGestureRecognizer isInterruptingMomentumScrollingWithEvent:event] ? WebKit::ShouldDeferGestures::No : WebKit::ShouldDeferGestures::Yes;
    … …  
    1062910670    [self _invokeAllActionsToPerformAfterPendingImageAnalysis:WebKit::ProceedWithTextSelectionInImage::No];
    1063010671#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
     10672    [self uninstallImageAnalysisInteraction];
    1063110673    _imageAnalysisMarkupData = std::nullopt;
    1063210674#endif
    … …  
    1094510987        }];
    1094610988    });
     10989}
     10990
     10991- (void)installImageAnalysisInteraction:(VKCImageAnalysis *)analysis
     10992{
     10993    if (!_imageAnalysisInteraction) {
     10994        _imageAnalysisInteraction = adoptNS([PAL::allocVKCImageAnalysisInteractionInstance() init]);
     10995        [_imageAnalysisInteraction setActiveInteractionTypes:VKImageAnalysisInteractionTypeTextSelection | VKImageAnalysisInteractionTypeDataDetectors];
     10996        [_imageAnalysisInteraction setDelegate:self];
     10997        [_imageAnalysisInteraction setWantsAutomaticContentsRectCalculation:NO];
     10998        [self addInteraction:_imageAnalysisInteraction.get()];
     10999    }
     11000    [_imageAnalysisInteraction setAnalysis:analysis];
     11001    [_imageAnalysisDeferringGestureRecognizer setEnabled:NO];
     11002    [_imageAnalysisGestureRecognizer setEnabled:NO];
     11003}
     11004
     11005- (void)uninstallImageAnalysisInteraction
     11006{
     11007    if (!_imageAnalysisInteraction)
     11008        return;
     11009
     11010    [self removeInteraction:_imageAnalysisInteraction.get()];
     11011    _imageAnalysisInteraction = nil;
     11012    [_imageAnalysisDeferringGestureRecognizer setEnabled:WebKit::isLiveTextAvailableAndEnabled()];
     11013    [_imageAnalysisGestureRecognizer setEnabled:WebKit::isLiveTextAvailableAndEnabled()];
     11014}
     11015
     11016#pragma mark - VKCImageAnalysisInteractionDelegate
     11017
     11018- (CGRect)contentsRectForImageAnalysisInteraction:(VKCImageAnalysisInteraction *)interaction
     11019{
     11020    auto unitInteractionRect = _imageAnalysisInteractionBounds;
     11021    WebCore::FloatRect unobscuredRect = self.bounds;
     11022    unitInteractionRect.moveBy(-unobscuredRect.location());
     11023    unitInteractionRect.scale(1 / unobscuredRect.size());
     11024    return unitInteractionRect;
     11025}
     11026
     11027- (BOOL)imageAnalysisInteraction:(VKCImageAnalysisInteraction *)interaction shouldBeginAtPoint:(CGPoint)point forAnalysisType:(VKImageAnalysisInteractionTypes)analysisType
     11028{
     11029    return [_imageAnalysisInteraction interactableItemExistsAtPoint:point];
    1094711030}
    1094811031
Note: See TracChangeset for help on using the changeset viewer.