⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 285949 in webkit


Ignore:
Timestamp:
Nov 17, 2021, 12:22:11 PM (5 years ago)
Author:
Wenson Hsieh
Message:

Add a helper class to coordinate batch analysis of images
​https://bugs.webkit.org/show_bug.cgi?id=233075

Reviewed by Dean Jackson.

Source/WebCore:

Introduce ImageAnalysisQueue, which will collect suitable image elements in the main document and coordinate
image analysis tasks for up to some (fixed, but easily configurable) maximum number of simultaneous pending
tasks. See below for more details.

  • Headers.cmake:
  • Sources.txt:
  • WebCore.xcodeproj/project.pbxproj:
  • page/ImageAnalysisQueue.cpp: Added.

(WebCore::ImageAnalysisQueue::ImageAnalysisQueue):
(WebCore::ImageAnalysisQueue::enqueueAllImages):

Iterate through all image elements on the page with cached images, and add them as image analysis tasks to the
task queue as long as they are not tiny. Automatically start processing queued tasks afterwards (see below).

(WebCore::ImageAnalysisQueue::resumeProcessing):

Run the main processing loop of the task queue; while there are tasks remaining, push up to 5 images to the
client layer for image analysis at the same time, until there are no more queued tasks.

(WebCore::ImageAnalysisQueue::clear):

  • page/ImageAnalysisQueue.h: Added.

Introduce the new class.

  • page/Page.cpp:

(WebCore::Page::imageOverlayController):

Drive-by fix: m_imageOverlayController was being eagerly initialized here, despite the existence of both
imageOverlayControllerIfExists() and imageOverlayController(). Fix this by lazily creating the
ImageOverlayController instead.

(WebCore::Page::imageAnalysisQueue):

  • page/Page.h:

(WebCore::Page::imageAnalysisQueueIfExists):
(WebCore::Page::imageOverlayController): Deleted.

Source/WebKit:

Add a couple of WKWebView SPI methods to kick off (and cancel) batch image analysis for mainframe content.
See WebCore ChangeLog for more details.

  • UIProcess/API/Cocoa/WKWebView.mm:

(-[WKWebView _startImageAnalysis:]):

  • UIProcess/API/Cocoa/WKWebViewPrivate.h:
  • UIProcess/WebPageProxy.cpp:

(WebKit::WebPageProxy::startImageAnalysis):

  • UIProcess/WebPageProxy.h:
  • WebProcess/WebPage/WebPage.cpp:

(WebKit::WebPage::startImageAnalysis):

  • WebProcess/WebPage/WebPage.h:
  • WebProcess/WebPage/WebPage.messages.in:
Location:
trunk/Source
Files:
2 added
14 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebCore/ChangeLog

    r285948 r285949  
     12021-11-17  Wenson Hsieh  <wenson_hsieh@apple.com>
     2
     3        Add a helper class to coordinate batch analysis of images
     4        https://bugs.webkit.org/show_bug.cgi?id=233075
     5
     6        Reviewed by Dean Jackson.
     7
     8        Introduce ImageAnalysisQueue, which will collect suitable image elements in the main document and coordinate
     9        image analysis tasks for up to some (fixed, but easily configurable) maximum number of simultaneous pending
     10        tasks. See below for more details.
     11
     12        * Headers.cmake:
     13        * Sources.txt:
     14        * WebCore.xcodeproj/project.pbxproj:
     15        * page/ImageAnalysisQueue.cpp: Added.
     16        (WebCore::ImageAnalysisQueue::ImageAnalysisQueue):
     17        (WebCore::ImageAnalysisQueue::enqueueAllImages):
     18
     19        Iterate through all image elements on the page with cached images, and add them as image analysis tasks to the
     20        task queue as long as they are not tiny. Automatically start processing queued tasks afterwards (see below).
     21
     22        (WebCore::ImageAnalysisQueue::resumeProcessing):
     23
     24        Run the main processing loop of the task queue; while there are tasks remaining, push up to 5 images to the
     25        client layer for image analysis at the same time, until there are no more queued tasks.
     26
     27        (WebCore::ImageAnalysisQueue::clear):
     28        * page/ImageAnalysisQueue.h: Added.
     29
     30        Introduce the new class.
     31
     32        * page/Page.cpp:
     33        (WebCore::Page::imageOverlayController):
     34
     35        Drive-by fix: `m_imageOverlayController` was being eagerly initialized here, despite the existence of both
     36        `imageOverlayControllerIfExists()` and `imageOverlayController()`. Fix this by lazily creating the
     37        ImageOverlayController instead.
     38
     39        (WebCore::Page::imageAnalysisQueue):
     40        * page/Page.h:
     41        (WebCore::Page::imageAnalysisQueueIfExists):
     42        (WebCore::Page::imageOverlayController): Deleted.
     43
    1442021-11-17  Alan Bujtas  <zalan@apple.com>
    245

  • trunk/Source/WebCore/Headers.cmake

    r285822 r285949  
    955955    page/GlobalFrameIdentifier.h
    956956    page/GlobalWindowIdentifier.h
     957    page/ImageAnalysisQueue.h
    957958    page/LayoutMilestone.h
    958959    page/MediaCanStartListener.h

  • trunk/Source/WebCore/Sources.txt

    r285936 r285949  
    17261726page/FrameViewLayoutContext.cpp
    17271727page/History.cpp
     1728page/ImageAnalysisQueue.cpp
    17281729page/ImageOverlayController.cpp
    17291730page/IntersectionObserver.cpp

  • trunk/Source/WebCore/WebCore.xcodeproj/project.pbxproj

    r285936 r285949  
    54505450                F46C447E234654540039A79D /* ClipboardItemBindingsDataSource.h in Headers */ = {isa = PBXBuildFile; fileRef = F46C447C234654540039A79D /* ClipboardItemBindingsDataSource.h */; };
    54515451                F46D5386273D7E460009FA80 /* ImageOverlay.h in Headers */ = {isa = PBXBuildFile; fileRef = F46D5385273D7E3F0009FA80 /* ImageOverlay.h */; settings = {ATTRIBUTES = (Private, ); }; };
     5452                F46D53A1273EEFA00009FA80 /* ImageAnalysisQueue.h in Headers */ = {isa = PBXBuildFile; fileRef = F46D539D273EECF70009FA80 /* ImageAnalysisQueue.h */; settings = {ATTRIBUTES = (Private, ); }; };
    54525453                F473845825DDE9FB006DE8DD /* DataOwnerType.h in Headers */ = {isa = PBXBuildFile; fileRef = F473845725DDE9FB006DE8DD /* DataOwnerType.h */; settings = {ATTRIBUTES = (Private, ); }; };
    54535454                F47A09D120A93A9700240FAE /* DisabledAdaptations.h in Headers */ = {isa = PBXBuildFile; fileRef = F47A09CF20A939F600240FAE /* DisabledAdaptations.h */; settings = {ATTRIBUTES = (Private, ); }; };
    … …  
    1726717268                F46D5384273D7E3E0009FA80 /* ImageOverlay.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = ImageOverlay.cpp; sourceTree = "<group>"; };
    1726817269                F46D5385273D7E3F0009FA80 /* ImageOverlay.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = ImageOverlay.h; sourceTree = "<group>"; };
     17270                F46D539D273EECF70009FA80 /* ImageAnalysisQueue.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = ImageAnalysisQueue.h; sourceTree = "<group>"; };
     17271                F46D539E273EECF70009FA80 /* ImageAnalysisQueue.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = ImageAnalysisQueue.cpp; sourceTree = "<group>"; };
    1726917272                F473845725DDE9FB006DE8DD /* DataOwnerType.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = DataOwnerType.h; sourceTree = "<group>"; };
    1727017273                F47A09CF20A939F600240FAE /* DisabledAdaptations.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = DisabledAdaptations.h; sourceTree = "<group>"; };
    … …  
    2318423187                                BC94D1510C275C8B006BC617 /* History.h */,
    2318523188                                BC94D1520C275C8B006BC617 /* History.idl */,
     23189                                F46D539E273EECF70009FA80 /* ImageAnalysisQueue.cpp */,
     23190                                F46D539D273EECF70009FA80 /* ImageAnalysisQueue.h */,
    2318623191                                F482434A260C32F10022497C /* ImageOverlayController.cpp */,
    2318723192                                F4824348260C32F10022497C /* ImageOverlayController.h */,
    … …  
    3385633861                                830D0B6924F57D67007ED122 /* IIRProcessor.h in Headers */,
    3385733862                                B27535700B053814002CE64F /* Image.h in Headers */,
     33863                                F46D53A1273EEFA00009FA80 /* ImageAnalysisQueue.h in Headers */,
    3385833864                                55A336F91D821E3C0022C4C7 /* ImageBackingStore.h in Headers */,
    3385933865                                31815A311F9A6C8F00FCBF89 /* ImageBitmap.h in Headers */,

  • trunk/Source/WebCore/page/Page.cpp

    r285655 r285949  
    7777#include "HistoryItem.h"
    7878#include "IDBConnectionToServer.h"
     79#include "ImageAnalysisQueue.h"
    7980#include "ImageOverlay.h"
    8081#include "ImageOverlayController.h"
    … …  
    317318    , m_servicesOverlayController(makeUnique<ServicesOverlayController>(*this))
    318319#endif
    319     , m_imageOverlayController(makeUnique<ImageOverlayController>(*this))
    320320    , m_recentWheelEventDeltaFilter(WheelEventDeltaFilter::create())
    321321    , m_pageOverlayController(makeUnique<PageOverlayController>(*this))
    … …  
    37053705}
    37063706
     3707ImageOverlayController& Page::imageOverlayController()
     3708{
     3709    if (!m_imageOverlayController)
     3710        m_imageOverlayController = makeUnique<ImageOverlayController>(*this);
     3711    return *m_imageOverlayController;
     3712}
     3713
    37073714#if ENABLE(IMAGE_ANALYSIS)
     3715
     3716ImageAnalysisQueue& Page::imageAnalysisQueue()
     3717{
     3718    if (!m_imageAnalysisQueue)
     3719        m_imageAnalysisQueue = makeUnique<ImageAnalysisQueue>(*this);
     3720    return *m_imageAnalysisQueue;
     3721}
    37083722
    37093723void Page::updateElementsWithTextRecognitionResults()

  • trunk/Source/WebCore/page/Page.h

    r285637 r285949  
    116116class HTMLMediaElement;
    117117class HistoryItem;
     118class ImageAnalysisQueue;
    118119class ImageOverlayController;
    119120class InspectorClient;
    … …  
    532533    ServicesOverlayController& servicesOverlayController() { return *m_servicesOverlayController; }
    533534#endif
    534     ImageOverlayController& imageOverlayController() { return *m_imageOverlayController; }
     535    ImageOverlayController& imageOverlayController();
    535536    ImageOverlayController* imageOverlayControllerIfExists() { return m_imageOverlayController.get(); }
     537
     538#if ENABLE(IMAGE_ANALYSIS)
     539    WEBCORE_EXPORT ImageAnalysisQueue& imageAnalysisQueue();
     540    ImageAnalysisQueue* imageAnalysisQueueIfExists() { return m_imageAnalysisQueue.get(); }
     541#endif
    536542
    537543#if ENABLE(WHEEL_EVENT_LATCHING)
    … …  
    11931199    std::unique_ptr<ImageOverlayController> m_imageOverlayController;
    11941200
     1201#if ENABLE(IMAGE_ANALYSIS)
     1202    std::unique_ptr<ImageAnalysisQueue> m_imageAnalysisQueue;
     1203#endif
     1204
    11951205    std::unique_ptr<WheelEventDeltaFilter> m_recentWheelEventDeltaFilter;
    11961206    std::unique_ptr<PageOverlayController> m_pageOverlayController;

  • trunk/Source/WebKit/ChangeLog

    r285936 r285949  
     12021-11-17  Wenson Hsieh  <wenson_hsieh@apple.com>
     2
     3        Add a helper class to coordinate batch analysis of images
     4        https://bugs.webkit.org/show_bug.cgi?id=233075
     5
     6        Reviewed by Dean Jackson.
     7
     8        Add a couple of WKWebView SPI methods to kick off (and cancel) batch image analysis for mainframe content.
     9        See WebCore ChangeLog for more details.
     10
     11        * UIProcess/API/Cocoa/WKWebView.mm:
     12        (-[WKWebView _startImageAnalysis:]):
     13        * UIProcess/API/Cocoa/WKWebViewPrivate.h:
     14        * UIProcess/WebPageProxy.cpp:
     15        (WebKit::WebPageProxy::startImageAnalysis):
     16        * UIProcess/WebPageProxy.h:
     17        * WebProcess/WebPage/WebPage.cpp:
     18        (WebKit::WebPage::startImageAnalysis):
     19        * WebProcess/WebPage/WebPage.h:
     20        * WebProcess/WebPage/WebPage.messages.in:
     21
    1222021-11-17  Dean Jackson  <dino@apple.com>
    223

  • trunk/Source/WebKit/UIProcess/API/Cocoa/WKWebView.mm

    r285742 r285949  
    22052205}
    22062206
     2207- (void)_startImageAnalysis:(NSString *)identifier
     2208{
     2209#if ENABLE(IMAGE_ANALYSIS)
     2210    THROW_IF_SUSPENDED;
     2211
     2212    if (!_page || !_page->preferences().textRecognitionEnhancementsEnabled())
     2213        return;
     2214
     2215    _page->startImageAnalysis(identifier);
     2216#endif
     2217}
     2218
    22072219- (void)_takeFindStringFromSelection:(id)sender
    22082220{

  • trunk/Source/WebKit/UIProcess/API/Cocoa/WKWebViewPrivate.h

    r285742 r285949  
    415415- (void)_resumePage:(void (^)(BOOL))completionHandler WK_API_AVAILABLE(macos(12.0), ios(15.0));
    416416
     417- (void)_startImageAnalysis:(NSString *)identifier WK_API_AVAILABLE(macos(WK_MAC_TBA), ios(WK_IOS_TBA));
     418
    417419@end
    418420

  • trunk/Source/WebKit/UIProcess/WebPageProxy.cpp

    r285877 r285949  
    86378637}
    86388638
     8639void WebPageProxy::startImageAnalysis(const String& identifier)
     8640{
     8641    if (hasRunningProcess())
     8642        send(Messages::WebPage::StartImageAnalysis(identifier));
     8643}
     8644
    86398645#endif // ENABLE(IMAGE_ANALYSIS)
    86408646

  • trunk/Source/WebKit/UIProcess/WebPageProxy.h

    r285637 r285949  
    17021702    void updateWithTextRecognitionResult(WebCore::TextRecognitionResult&&, const WebCore::ElementContext&, const WebCore::FloatPoint& location, CompletionHandler<void(TextRecognitionUpdateResult)>&&);
    17031703    void computeHasImageAnalysisResults(const URL& imageURL, ShareableBitmap& imageBitmap, ImageAnalysisType, CompletionHandler<void(bool)>&&);
     1704    void startImageAnalysis(const String& identifier);
    17041705#endif
    17051706

  • trunk/Source/WebKit/WebProcess/WebPage/WebPage.cpp

    r285846 r285949  
    203203#include <WebCore/HistoryItem.h>
    204204#include <WebCore/HitTestResult.h>
     205#include <WebCore/ImageAnalysisQueue.h>
    205206#include <WebCore/ImageOverlay.h>
    206207#include <WebCore/InspectorController.h>
    … …  
    76327633}
    76337634
     7635void WebPage::startImageAnalysis(const String& identifier)
     7636{
     7637    if (RefPtr document = m_mainFrame->coreFrame()->document()) {
     7638        // We only consider main document content for now, to match the behavior of the corresponding feature
     7639        // that will trigger this codepath.
     7640        corePage()->imageAnalysisQueue().enqueueAllImages(*document, identifier);
     7641    }
     7642}
     7643
    76347644#endif // ENABLE(IMAGE_ANALYSIS)
    76357645

  • trunk/Source/WebKit/WebProcess/WebPage/WebPage.h

    r285846 r285949  
    14331433    void requestTextRecognition(WebCore::Element&, CompletionHandler<void(RefPtr<WebCore::Element>&&)>&&);
    14341434    void updateWithTextRecognitionResult(const WebCore::TextRecognitionResult&, const WebCore::ElementContext&, const WebCore::FloatPoint& location, CompletionHandler<void(TextRecognitionUpdateResult)>&&);
     1435    void startImageAnalysis(const String& identifier);
    14351436#endif
    14361437

  • trunk/Source/WebKit/WebProcess/WebPage/WebPage.messages.in

    r285640 r285949  
    645645#if ENABLE(IMAGE_ANALYSIS)
    646646    UpdateWithTextRecognitionResult(struct WebCore::TextRecognitionResult result, struct WebCore::ElementContext element, WebCore::FloatPoint location) -> (enum:uint8_t WebKit::TextRecognitionUpdateResult result) Async
     647    StartImageAnalysis(String identifier)
    647648#endif
    648649
Note: See TracChangeset for help on using the changeset viewer.