⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 286106 in webkit


Ignore:
Timestamp:
Nov 21, 2021, 3:41:28 PM (5 years ago)
Author:
Wenson Hsieh
Message:

Add an API test for batch image analysis (-[WKWebView _startImageAnalysis:])
​https://bugs.webkit.org/show_bug.cgi?id=233331

Reviewed by Sam Weinig.

Add a new API teset to exercise the -_startImageAnalysis: SPI introduced in r285949, along with helper
functions to simulate VKImageAnalysis results from VisionKitCore. See below for more details.

  • TestWebKitAPI/SourcesCocoa.txt:
  • TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
  • TestWebKitAPI/Tests/WebKitCocoa/ImageAnalysisTests.mm:

(-[TestWKWebView waitForImageAnalysisRequests:]):
(TestWebKitAPI::createWebViewWithTextRecognitionEnhancements):
(TestWebKitAPI::swizzledProcessRequestWithResults):
(TestWebKitAPI::swizzledProcessRequestWithError):
(TestWebKitAPI::TEST):
(TestWebKitAPI::swizzledProcessRequest): Deleted.

Add an API test that uses -_startImageAnalysis: to perform batch image analysis over 5 image elements on the
page, and verify that the text results are injected back into the page.

  • TestWebKitAPI/Tests/WebKitCocoa/multiple-images.html: Added.
  • TestWebKitAPI/cocoa/ImageAnalysisTestingUtilities.h: Added.
  • TestWebKitAPI/cocoa/ImageAnalysisTestingUtilities.mm: Added.

(-[TestVKQuad initWithTopLeft:topRight:bottomLeft:bottomRight:]):
(-[TestVKWKTextInfo initWithString:quad:]):
(-[TestVKWKTextInfo string]):
(-[TestVKWKTextInfo quad]):
(-[TestVKWKLineInfo initWithString:quad:children:]):
(-[TestVKWKLineInfo children]):
(-[TestVKImageAnalysis initWithLines:]):
(-[TestVKImageAnalysis allLines]):
(TestWebKitAPI::createQuad):
(TestWebKitAPI::createTextInfo):
(TestWebKitAPI::createLineInfo):
(TestWebKitAPI::createImageAnalysis):
(TestWebKitAPI::createImageAnalysisWithSimpleFixedResults):

Add helper functions to create and return mock objects that represent VKImageAnalysis, VKWKLineInfo,
VKWKTextInfo and VKQuad.

Location:
trunk/Tools
Files:
3 added
4 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r286102 r286106  
     12021-11-21  Wenson Hsieh  <wenson_hsieh@apple.com>
     2
     3        Add an API test for batch image analysis (-[WKWebView _startImageAnalysis:])
     4        https://bugs.webkit.org/show_bug.cgi?id=233331
     5
     6        Reviewed by Sam Weinig.
     7
     8        Add a new API teset to exercise the `-_startImageAnalysis:` SPI introduced in r285949, along with helper
     9        functions to simulate `VKImageAnalysis` results from VisionKitCore. See below for more details.
     10
     11        * TestWebKitAPI/SourcesCocoa.txt:
     12        * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
     13        * TestWebKitAPI/Tests/WebKitCocoa/ImageAnalysisTests.mm:
     14        (-[TestWKWebView waitForImageAnalysisRequests:]):
     15        (TestWebKitAPI::createWebViewWithTextRecognitionEnhancements):
     16        (TestWebKitAPI::swizzledProcessRequestWithResults):
     17        (TestWebKitAPI::swizzledProcessRequestWithError):
     18        (TestWebKitAPI::TEST):
     19        (TestWebKitAPI::swizzledProcessRequest): Deleted.
     20
     21        Add an API test that uses `-_startImageAnalysis:` to perform batch image analysis over 5 image elements on the
     22        page, and verify that the text results are injected back into the page.
     23
     24        * TestWebKitAPI/Tests/WebKitCocoa/multiple-images.html: Added.
     25        * TestWebKitAPI/cocoa/ImageAnalysisTestingUtilities.h: Added.
     26        * TestWebKitAPI/cocoa/ImageAnalysisTestingUtilities.mm: Added.
     27        (-[TestVKQuad initWithTopLeft:topRight:bottomLeft:bottomRight:]):
     28        (-[TestVKWKTextInfo initWithString:quad:]):
     29        (-[TestVKWKTextInfo string]):
     30        (-[TestVKWKTextInfo quad]):
     31        (-[TestVKWKLineInfo initWithString:quad:children:]):
     32        (-[TestVKWKLineInfo children]):
     33        (-[TestVKImageAnalysis initWithLines:]):
     34        (-[TestVKImageAnalysis allLines]):
     35        (TestWebKitAPI::createQuad):
     36        (TestWebKitAPI::createTextInfo):
     37        (TestWebKitAPI::createLineInfo):
     38        (TestWebKitAPI::createImageAnalysis):
     39        (TestWebKitAPI::createImageAnalysisWithSimpleFixedResults):
     40
     41        Add helper functions to create and return mock objects that represent VKImageAnalysis, VKWKLineInfo,
     42        VKWKTextInfo and VKQuad.
     43
    1442021-11-21  Andres Gonzalez  <andresg_22@apple.com>
    245

  • trunk/Tools/TestWebKitAPI/SourcesCocoa.txt

    r285547 r286106  
    2727
    2828cocoa/DaemonTestUtilities.mm
     29cocoa/ImageAnalysisTestingUtilities.mm
    2930cocoa/NSItemProviderAdditions.mm
    3031cocoa/PlatformUtilitiesCocoa.mm

  • trunk/Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj

    r285770 r286106  
    10171017                F42D634422A1729F00D2FB3A /* AutocorrectionTestsIOS.mm in Sources */ = {isa = PBXBuildFile; fileRef = F42D634322A1729F00D2FB3A /* AutocorrectionTestsIOS.mm */; };
    10181018                F42DA5161D8CEFE400336F40 /* large-input-field-focus-onload.html in Copy Resources */ = {isa = PBXBuildFile; fileRef = F42DA5151D8CEFDB00336F40 /* large-input-field-focus-onload.html */; };
     1019                F42F081227449892007E0D90 /* multiple-images.html in Copy Resources */ = {isa = PBXBuildFile; fileRef = F42F0811274497C8007E0D90 /* multiple-images.html */; };
    10191020                F434CA1A22E65BCA005DDB26 /* ScrollToRevealSelection.mm in Sources */ = {isa = PBXBuildFile; fileRef = F434CA1922E65BCA005DDB26 /* ScrollToRevealSelection.mm */; };
    10201021                F4352F9F26D403DE00E605E4 /* editable-responsive-body.html in Copy Resources */ = {isa = PBXBuildFile; fileRef = F4352F9E26D4037000E605E4 /* editable-responsive-body.html */; };
    … …  
    14651466                                9BCD411A206DBCA3001D71BE /* mso-list-on-h4.html in Copy Resources */,
    14661467                                9BF356CD202D458500F71160 /* mso-list.html in Copy Resources */,
     1468                                F42F081227449892007E0D90 /* multiple-images.html in Copy Resources */,
    14671469                                5797FE331EB15AB100B2F4A0 /* navigation-client-default-crypto.html in Copy Resources */,
    14681470                                2E4838472169DF30002F4531 /* nested-lists.html in Copy Resources */,
    … …  
    29382940                F42D634322A1729F00D2FB3A /* AutocorrectionTestsIOS.mm */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.objcpp; path = AutocorrectionTestsIOS.mm; sourceTree = "<group>"; };
    29392941                F42DA5151D8CEFDB00336F40 /* large-input-field-focus-onload.html */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.html; name = "large-input-field-focus-onload.html"; path = "Tests/WebKitCocoa/large-input-field-focus-onload.html"; sourceTree = SOURCE_ROOT; };
     2942                F42F0811274497C8007E0D90 /* multiple-images.html */ = {isa = PBXFileReference; lastKnownFileType = text.html; path = "multiple-images.html"; sourceTree = "<group>"; };
     2943                F42F081627449FFD007E0D90 /* ImageAnalysisTestingUtilities.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; name = ImageAnalysisTestingUtilities.mm; path = cocoa/ImageAnalysisTestingUtilities.mm; sourceTree = "<group>"; };
     2944                F42F081727449FFD007E0D90 /* ImageAnalysisTestingUtilities.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = ImageAnalysisTestingUtilities.h; path = cocoa/ImageAnalysisTestingUtilities.h; sourceTree = "<group>"; };
    29402945                F434CA1922E65BCA005DDB26 /* ScrollToRevealSelection.mm */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.objcpp; path = ScrollToRevealSelection.mm; sourceTree = "<group>"; };
    29412946                F4352F9E26D4037000E605E4 /* editable-responsive-body.html */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.html; path = "editable-responsive-body.html"; sourceTree = "<group>"; };
    … …  
    32113216                                5C7C24FB237C972300599C91 /* HTTPServer.h */,
    32123217                                5C7C24FA237C972300599C91 /* HTTPServer.mm */,
     3218                                F42F081727449FFD007E0D90 /* ImageAnalysisTestingUtilities.h */,
     3219                                F42F081627449FFD007E0D90 /* ImageAnalysisTestingUtilities.mm */,
    32133220                                F44A530D21B8976900DBB99C /* InstanceMethodSwizzler.h */,
    32143221                                F44A531021B8976900DBB99C /* InstanceMethodSwizzler.mm */,
    … …  
    41034110                                9BCD4119206D5ED7001D71BE /* mso-list-on-h4.html */,
    41044111                                9BF356CC202D44F200F71160 /* mso-list.html */,
     4112                                F42F0811274497C8007E0D90 /* multiple-images.html */,
    41054113                                2E4838462169DD42002F4531 /* nested-lists.html */,
    41064114                                466C3842210637CE006A88DE /* notify-resourceLoadObserver.html */,

  • trunk/Tools/TestWebKitAPI/Tests/WebKitCocoa/ImageAnalysisTests.mm

    r281090 r286106  
    2828#if ENABLE(IMAGE_ANALYSIS)
    2929
     30#import "ImageAnalysisTestingUtilities.h"
    3031#import "InstanceMethodSwizzler.h"
    3132#import "PlatformUtilities.h"
    … …  
    3738#import <pal/spi/cocoa/VisionKitCoreSPI.h>
    3839
     40static unsigned gDidProcessRequestCount = 0;
     41
     42@interface TestWKWebView (ImageAnalysisTests)
     43- (void)waitForImageAnalysisRequests:(unsigned)numberOfRequests;
     44@end
     45
     46@implementation TestWKWebView (ImageAnalysisTests)
     47
     48- (void)waitForImageAnalysisRequests:(unsigned)numberOfRequests
     49{
     50    TestWebKitAPI::Util::waitForConditionWithLogging([&] {
     51        return gDidProcessRequestCount == numberOfRequests;
     52    }, 3, @"Timed out waiting for %u image analysis to complete.", numberOfRequests);
     53
     54    [self waitForNextPresentationUpdate];
     55    EXPECT_EQ(gDidProcessRequestCount, numberOfRequests);
     56}
     57
     58@end
     59
    3960namespace TestWebKitAPI {
     61
     62static RetainPtr<TestWKWebView> createWebViewWithTextRecognitionEnhancements()
     63{
     64    RetainPtr configuration = [WKWebViewConfiguration _test_configurationWithTestPlugInClassName:@"WebProcessPlugInWithInternals" configureJSCForTesting:YES];
     65    for (_WKInternalDebugFeature *feature in WKPreferences._internalDebugFeatures) {
     66        if ([feature.key isEqualToString:@"TextRecognitionEnhancementsEnabled"]) {
     67            [[configuration preferences] _setEnabled:YES forInternalDebugFeature:feature];
     68            break;
     69        }
     70    }
     71    return adoptNS([[TestWKWebView alloc] initWithFrame:NSMakeRect(0, 0, 300, 300) configuration:configuration.get()]);
     72}
     73
     74static void swizzledProcessRequestWithResults(id, SEL, VKImageAnalyzerRequest *, void (^)(double progress), void (^completion)(VKImageAnalysis *, NSError *))
     75{
     76    gDidProcessRequestCount++;
     77    auto analysis = createImageAnalysisWithSimpleFixedResults();
     78    completion(analysis.get(), nil);
     79}
    4080
    4181#if PLATFORM(IOS_FAMILY)
    … …  
    4686}
    4787
    48 static bool gDidProcessRequest = false;
    49 static void swizzledProcessRequest(id, SEL, VKImageAnalyzerRequest *, void (^)(double progress), void (^completion)(VKImageAnalysis *analysis, NSError *error))
     88static void swizzledProcessRequestWithError(id, SEL, VKImageAnalyzerRequest *, void (^)(double progress), void (^completion)(VKImageAnalysis *analysis, NSError *error))
    5089{
    51     gDidProcessRequest = true;
     90    gDidProcessRequestCount++;
    5291    completion(nil, [NSError errorWithDomain:NSCocoaErrorDomain code:1 userInfo:nil]);
    5392}
    … …  
    5695{
    5796    InstanceMethodSwizzler gestureLocationSwizzler { UIGestureRecognizer.class, @selector(locationInView:), reinterpret_cast<IMP>(swizzledLocationInView) };
    58     InstanceMethodSwizzler imageAnalysisRequestSwizzler { PAL::getVKImageAnalyzerClass(), @selector(processRequest:progressHandler:completionHandler:), reinterpret_cast<IMP>(swizzledProcessRequest) };
     97    InstanceMethodSwizzler imageAnalysisRequestSwizzler { PAL::getVKImageAnalyzerClass(), @selector(processRequest:progressHandler:completionHandler:), reinterpret_cast<IMP>(swizzledProcessRequestWithError) };
    5998
    6099    auto webView = adoptNS([[TestWKWebView alloc] initWithFrame:CGRectMake(0, 0, 400, 400)]);
    … …  
    64103    [webView _imageAnalysisGestureRecognizer].state = UIGestureRecognizerStateBegan;
    65104    [webView waitForNextPresentationUpdate];
    66     EXPECT_FALSE(gDidProcessRequest);
     105    EXPECT_EQ(gDidProcessRequestCount, 0U);
    67106}
    68107
    … …  
    70109{
    71110    InstanceMethodSwizzler gestureLocationSwizzler { UIGestureRecognizer.class, @selector(locationInView:), reinterpret_cast<IMP>(swizzledLocationInView) };
    72     InstanceMethodSwizzler imageAnalysisRequestSwizzler { PAL::getVKImageAnalyzerClass(), @selector(processRequest:progressHandler:completionHandler:), reinterpret_cast<IMP>(swizzledProcessRequest) };
     111    InstanceMethodSwizzler imageAnalysisRequestSwizzler { PAL::getVKImageAnalyzerClass(), @selector(processRequest:progressHandler:completionHandler:), reinterpret_cast<IMP>(swizzledProcessRequestWithError) };
    73112
    74113    auto webView = adoptNS([[TestWKWebView alloc] initWithFrame:CGRectMake(0, 0, 400, 400)]);
    … …  
    77116    [webView _imageAnalysisGestureRecognizer].state = UIGestureRecognizerStateBegan;
    78117    [webView waitForNextPresentationUpdate];
    79     EXPECT_TRUE(gDidProcessRequest);
     118    EXPECT_EQ(gDidProcessRequestCount, 1U);
    80119}
    81120
    82121#endif // PLATFORM(IOS_FAMILY)
    83122
     123TEST(ImageAnalysisTests, StartImageAnalysisWithoutIdentifier)
     124{
     125    InstanceMethodSwizzler imageAnalysisRequestSwizzler { PAL::getVKImageAnalyzerClass(), @selector(processRequest:progressHandler:completionHandler:), reinterpret_cast<IMP>(swizzledProcessRequestWithResults) };
     126
     127    auto webView = createWebViewWithTextRecognitionEnhancements();
     128    [webView synchronouslyLoadTestPageNamed:@"multiple-images"];
     129    [webView _startImageAnalysis:nil];
     130    [webView waitForImageAnalysisRequests:5];
     131
     132    NSArray<NSString *> *overlaysAsText = [webView objectByEvaluatingJavaScript:@"imageOverlaysAsText()"];
     133    EXPECT_EQ(overlaysAsText.count, 5U);
     134    for (NSString *overlayText in overlaysAsText)
     135        EXPECT_WK_STREQ(overlayText, @"Foo bar");
     136}
     137
    84138} // namespace TestWebKitAPI
    85139
Note: See TracChangeset for help on using the changeset viewer.