Changeset 278747 in webkit


Ignore:
Timestamp:
Jun 10, 2021 6:22:52 PM (13 months ago)
Author:
Wenson Hsieh
Message:

[Live Text] Add a mechanism to regenerate text in an image element when it changes dimensions
​https://bugs.webkit.org/show_bug.cgi?id=226858
rdar://77522786

Reviewed by Devin Rousso.

Source/WebCore:

Push m_elementsWithTextRecognitionResults from WebPage down to Page, and additionally keep track of a
TextRecognitionResult per element, as well as the size of the element at the time when we last injected text
recognition results. Use this information to tell elements to update their OCR text containers if their
dimenions have changed since the last update, after finishing a rendering update.

Test: fast/images/text-recognition/image-overlay-size-change.html

  • html/HTMLElement.cpp:

(WebCore::HTMLElement::updateWithTextRecognitionResult):

Refactor this to take a const TextRecognitionResult& instead of a TextRecognitionResult&&, since this method
doesn't actually take ownership of the TextRecognitionResult. This allows us to avoid explicitly copying a
TextRecognitionResult when calling this method from Page::updateElementsWithTextRecognitionResults. Also, add
a CacheTextRecognitionResults enum flag as an argument, to avoid entering Page::cacheTextRecognitionResult
every time we update a resized element in Page::updateElementsWithTextRecognitionResults; since we know that
the dimensions are the only thing that changed, we simply update the cached dimensions.

Call into Page::cacheTextRecognitionResult to store per-element TextRecognitionResults when we
finish injecting the OCR text and data detectors into the image element.

  • html/HTMLElement.h:
  • page/Page.cpp:

(WebCore::Page::didCommitLoad):

Clear out both m_textRecognitionResultsByElement and m_elementsWithTextRecognitionResults.

(WebCore::Page::doAfterUpdateRendering):

At the end of each rendering update, go through m_elementsWithTextRecognitionResults and check if any of the
elements' dimensions have changed; if so, tell those elements to re-update their OCR text quads and data
detectors in the UA shadow tree.

(WebCore::Page::resetTextRecognitionResults):
(WebCore::Page::updateElementsWithTextRecognitionResults):
(WebCore::Page::hasCachedTextRecognitionResult const):
(WebCore::Page::cacheTextRecognitionResult):

Add the element to m_elementsWithTextRecognitionResults and m_textRecognitionResultsByElement, appending a
new entry to m_textRecognitionResultsByElement only if necessary.

  • page/Page.h:

Source/WebKit:

Push m_elementsWithTextRecognitionResults from WebKit::WebPage down to WebCore::Page. See WebCore
ChangeLog for more details.

  • WebProcess/WebPage/WebPage.cpp:

(WebKit::WebPage::didCommitLoad):
(WebKit::WebPage::requestTextRecognition):
(WebKit::WebPage::updateWithTextRecognitionResult):

  • WebProcess/WebPage/WebPage.h:

LayoutTests:

  • fast/images/text-recognition/image-overlay-size-change-expected.txt: Added.
  • fast/images/text-recognition/image-overlay-size-change.html: Added.
Location:
trunk
Files:
2 added
9 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r278743 r278747  
     12021-06-10  Wenson Hsieh  <wenson_hsieh@apple.com>
     2
     3        [Live Text] Add a mechanism to regenerate text in an image element when it changes dimensions
     4        https://bugs.webkit.org/show_bug.cgi?id=226858
     5        rdar://77522786
     6
     7        Reviewed by Devin Rousso.
     8
     9        * fast/images/text-recognition/image-overlay-size-change-expected.txt: Added.
     10        * fast/images/text-recognition/image-overlay-size-change.html: Added.
     11
    1122021-06-10  Arcady Goldmints-Orlov  <agoldmints@igalia.com>
    213

  • trunk/Source/WebCore/ChangeLog

    r278746 r278747  
     12021-06-10  Wenson Hsieh  <wenson_hsieh@apple.com>
     2
     3        [Live Text] Add a mechanism to regenerate text in an image element when it changes dimensions
     4        https://bugs.webkit.org/show_bug.cgi?id=226858
     5        rdar://77522786
     6
     7        Reviewed by Devin Rousso.
     8
     9        Push `m_elementsWithTextRecognitionResults` from WebPage down to Page, and additionally keep track of a
     10        TextRecognitionResult per element, as well as the size of the element at the time when we last injected text
     11        recognition results. Use this information to tell elements to update their OCR text containers if their
     12        dimenions have changed since the last update, after finishing a rendering update.
     13
     14        Test: fast/images/text-recognition/image-overlay-size-change.html
     15
     16        * html/HTMLElement.cpp:
     17        (WebCore::HTMLElement::updateWithTextRecognitionResult):
     18
     19        Refactor this to take a `const TextRecognitionResult&` instead of a `TextRecognitionResult&&`, since this method
     20        doesn't actually take ownership of the TextRecognitionResult. This allows us to avoid explicitly copying a
     21        TextRecognitionResult when calling this method from `Page::updateElementsWithTextRecognitionResults`. Also, add
     22        a CacheTextRecognitionResults enum flag as an argument, to avoid entering `Page::cacheTextRecognitionResult`
     23        every time we update a resized element in `Page::updateElementsWithTextRecognitionResults`; since we know that
     24        the dimensions are the only thing that changed, we simply update the cached dimensions.
     25
     26        Call into `Page::cacheTextRecognitionResult` to store per-element TextRecognitionResults when we
     27        finish injecting the OCR text and data detectors into the image element.
     28
     29        * html/HTMLElement.h:
     30        * page/Page.cpp:
     31        (WebCore::Page::didCommitLoad):
     32
     33        Clear out both `m_textRecognitionResultsByElement` and `m_elementsWithTextRecognitionResults`.
     34
     35        (WebCore::Page::doAfterUpdateRendering):
     36
     37        At the end of each rendering update, go through `m_elementsWithTextRecognitionResults` and check if any of the
     38        elements' dimensions have changed; if so, tell those elements to re-update their OCR text quads and data
     39        detectors in the UA shadow tree.
     40
     41        (WebCore::Page::resetTextRecognitionResults):
     42        (WebCore::Page::updateElementsWithTextRecognitionResults):
     43        (WebCore::Page::hasCachedTextRecognitionResult const):
     44        (WebCore::Page::cacheTextRecognitionResult):
     45
     46        Add the element to `m_elementsWithTextRecognitionResults` and `m_textRecognitionResultsByElement`, appending a
     47        new entry to `m_textRecognitionResultsByElement` only if necessary.
     48
     49        * page/Page.h:
     50
    1512021-06-10  Chris Dumez  <cdumez@apple.com>
    252

  • trunk/Source/WebCore/html/HTMLElement.cpp

    r278689 r278747  
    13321332#if ENABLE(IMAGE_ANALYSIS)
    13331333
    1334 void HTMLElement::updateWithTextRecognitionResult(TextRecognitionResult&& result)
     1334void HTMLElement::updateWithTextRecognitionResult(const TextRecognitionResult& result, CacheTextRecognitionResults cacheTextRecognitionResults)
    13351335{
    13361336    RefPtr<HTMLDivElement> previousContainer;
    … …  
    15101510    if (auto frame = makeRefPtr(document().frame()))
    15111511        frame->eventHandler().scheduleCursorUpdate();
     1512
     1513    if (cacheTextRecognitionResults == CacheTextRecognitionResults::Yes) {
     1514        if (auto* page = document().page())
     1515            page->cacheTextRecognitionResult(*this, containerSize, result);
     1516    }
    15121517}
    15131518

  • trunk/Source/WebCore/html/HTMLElement.h

    r278575 r278747  
    140140
    141141#if ENABLE(IMAGE_ANALYSIS)
    142     WEBCORE_EXPORT void updateWithTextRecognitionResult(TextRecognitionResult&&);
     142    enum class CacheTextRecognitionResults : bool { No, Yes };
     143    WEBCORE_EXPORT void updateWithTextRecognitionResult(const TextRecognitionResult&, CacheTextRecognitionResults = CacheTextRecognitionResults::Yes);
    143144#endif
    144145

  • trunk/Source/WebCore/page/Page.cpp

    r278498 r278747  
    137137#include "SubframeLoader.h"
    138138#include "TextIterator.h"
     139#include "TextRecognitionResult.h"
    139140#include "TextResourceDecoder.h"
    140141#include "UserContentProvider.h"
    … …  
    12751276    resetSeenPlugins();
    12761277    resetSeenMediaEngines();
     1278
     1279#if ENABLE(IMAGE_ANALYSIS)
     1280    resetTextRecognitionResults();
     1281#endif
    12771282}
    12781283
    … …  
    16601665        document.updateTextTrackRepresentationImageIfNeeded();
    16611666    });
     1667#endif
     1668
     1669#if ENABLE(IMAGE_ANALYSIS)
     1670    updateElementsWithTextRecognitionResults();
    16621671#endif
    16631672
    … …  
    36043613}
    36053614
     3615#if ENABLE(IMAGE_ANALYSIS)
     3616
     3617void Page::updateElementsWithTextRecognitionResults()
     3618{
     3619    if (m_textRecognitionResultsByElement.isEmpty())
     3620        return;
     3621
     3622    m_textRecognitionResultsByElement.removeAllMatching([](auto& elementAndResult) {
     3623        return !elementAndResult.first;
     3624    });
     3625
     3626    for (auto& [element, resultAndSize] : m_textRecognitionResultsByElement) {
     3627        if (!element || !element->isConnected())
     3628            continue;
     3629
     3630        auto protectedElement = makeRef(*element);
     3631        auto& [result, offsetSize] = resultAndSize;
     3632        IntSize newOffsetSize { protectedElement->offsetWidth(), protectedElement->offsetHeight() };
     3633        if (offsetSize == newOffsetSize)
     3634            continue;
     3635
     3636        offsetSize = newOffsetSize;
     3637        protectedElement->updateWithTextRecognitionResult(result, HTMLElement::CacheTextRecognitionResults::No);
     3638    }
     3639}
     3640
     3641bool Page::hasCachedTextRecognitionResult(const HTMLElement& element) const
     3642{
     3643    return m_elementsWithTextRecognitionResults.contains(element);
     3644}
     3645
     3646void Page::cacheTextRecognitionResult(const HTMLElement& element, const IntSize& offsetSize, const TextRecognitionResult& result)
     3647{
     3648    m_elementsWithTextRecognitionResults.add(element);
     3649
     3650    auto index = m_textRecognitionResultsByElement.findMatching([&](auto& elementAndResult) {
     3651        return elementAndResult.first == &element;
     3652    });
     3653
     3654    if (index == notFound)
     3655        m_textRecognitionResultsByElement.append({ makeWeakPtr(element), { result, offsetSize } });
     3656    else
     3657        m_textRecognitionResultsByElement[index].second = { result, offsetSize };
     3658
     3659    m_textRecognitionResultsByElement.removeAllMatching([](auto& elementAndResult) {
     3660        return !elementAndResult.first;
     3661    });
     3662}
     3663
     3664void Page::resetTextRecognitionResults()
     3665{
     3666    m_textRecognitionResultsByElement.clear();
     3667    m_elementsWithTextRecognitionResults.clear();
     3668}
     3669
     3670#endif // ENABLE(IMAGE_ANALYSIS)
     3671
    36063672} // namespace WebCore

  • trunk/Source/WebCore/page/Page.h

    r278482 r278747  
    108108class FormData;
    109109class Frame;
     110class HTMLElement;
    110111class HTMLMediaElement;
    111112class HistoryItem;
    … …  
    113114class InspectorClient;
    114115class InspectorController;
     116class IntSize;
    115117class LibWebRTCProvider;
    116118class LowPowerModeNotifier;
    … …  
    159161
    160162struct SimpleRange;
     163struct TextRecognitionResult;
    161164
    162165using PlatformDisplayID = uint32_t;
    … …  
    861864    void setLoadSchedulingMode(LoadSchedulingMode);
    862865
     866#if ENABLE(IMAGE_ANALYSIS)
     867    WEBCORE_EXPORT bool hasCachedTextRecognitionResult(const HTMLElement&) const;
     868    void cacheTextRecognitionResult(const HTMLElement&, const IntSize& offsetSize, const TextRecognitionResult&);
     869#endif
     870
    863871private:
    864872    struct Navigation {
    … …  
    905913
    906914    WheelEventTestMonitor& ensureWheelEventTestMonitor();
     915
     916#if ENABLE(IMAGE_ANALYSIS)
     917    void resetTextRecognitionResults();
     918    void updateElementsWithTextRecognitionResults();
     919#endif
    907920
    908921    const std::unique_ptr<Chrome> m_chrome;
    … …  
    11761189    const bool m_httpsUpgradeEnabled { true };
    11771190    mutable MediaSessionGroupIdentifier m_mediaSessionGroupIdentifier;
     1191
     1192#if ENABLE(IMAGE_ANALYSIS)
     1193    // FIXME: These should be refactored to use a weak hash map of HTMLElement to std::pair<TextRecognitionResult, IntSize>.
     1194    Vector<std::pair<WeakPtr<HTMLElement>, std::pair<TextRecognitionResult, IntSize>>> m_textRecognitionResultsByElement;
     1195    WeakHashSet<HTMLElement> m_elementsWithTextRecognitionResults;
     1196#endif
    11781197};
    11791198

  • trunk/Source/WebKit/ChangeLog

    r278745 r278747  
     12021-06-10  Wenson Hsieh  <wenson_hsieh@apple.com>
     2
     3        [Live Text] Add a mechanism to regenerate text in an image element when it changes dimensions
     4        https://bugs.webkit.org/show_bug.cgi?id=226858
     5        rdar://77522786
     6
     7        Reviewed by Devin Rousso.
     8
     9        Push `m_elementsWithTextRecognitionResults` from `WebKit::WebPage` down to `WebCore::Page`. See WebCore
     10        ChangeLog for more details.
     11
     12        * WebProcess/WebPage/WebPage.cpp:
     13        (WebKit::WebPage::didCommitLoad):
     14        (WebKit::WebPage::requestTextRecognition):
     15        (WebKit::WebPage::updateWithTextRecognitionResult):
     16        * WebProcess/WebPage/WebPage.h:
     17
    1182021-06-10  Per Arne Vollan  <pvollan@apple.com>
    219

  • trunk/Source/WebKit/WebProcess/WebPage/WebPage.cpp

    r278737 r278747  
    62916291    }
    62926292    m_elementsPendingTextRecognition.clear();
    6293     m_elementsWithTextRecognitionResults.clear();
    62946293#endif
    62956294
    … …  
    73977396    }
    73987397
    7399     if (m_elementsWithTextRecognitionResults.contains(element)) {
     7398    auto htmlElement = makeRef(downcast<HTMLElement>(element));
     7399    if (corePage()->hasCachedTextRecognitionResult(htmlElement.get())) {
    74007400        if (completion) {
    7401             ASSERT(is<HTMLElement>(element));
    74027401            RefPtr<Element> imageOverlayHost;
    7403             if (is<HTMLElement>(element) && downcast<HTMLElement>(element).hasImageOverlay())
     7402            if (htmlElement->hasImageOverlay())
    74047403                imageOverlayHost = makeRefPtr(element);
    74057404            completion(WTFMove(imageOverlayHost));
    … …  
    74677466
    74687467        auto& htmlElement = downcast<HTMLElement>(*protectedElement);
    7469         htmlElement.updateWithTextRecognitionResult(WTFMove(result));
    7470         protectedPage->m_elementsWithTextRecognitionResults.add(htmlElement);
     7468        htmlElement.updateWithTextRecognitionResult(result);
    74717469
    74727470        auto matchIndex = protectedPage->m_elementsPendingTextRecognition.findMatching([&] (auto& elementAndCompletionHandlers) {
    … …  
    74857483}
    74867484
    7487 void WebPage::updateWithTextRecognitionResult(TextRecognitionResult&& result, const ElementContext& context, const FloatPoint& location, CompletionHandler<void(TextRecognitionUpdateResult)>&& completionHandler)
     7485void WebPage::updateWithTextRecognitionResult(const TextRecognitionResult& result, const ElementContext& context, const FloatPoint& location, CompletionHandler<void(TextRecognitionUpdateResult)>&& completionHandler)
    74887486{
    74897487    auto elementToUpdate = elementForContext(context);
    … …  
    74937491    }
    74947492
    7495     downcast<HTMLElement>(*elementToUpdate).updateWithTextRecognitionResult(WTFMove(result));
     7493    downcast<HTMLElement>(*elementToUpdate).updateWithTextRecognitionResult(result);
    74967494    auto hitTestResult = corePage()->mainFrame().eventHandler().hitTestResultAtPoint(roundedIntPoint(location), {
    74977495        HitTestRequest::Type::ReadOnly,

  • trunk/Source/WebKit/WebProcess/WebPage/WebPage.h

    r278712 r278747  
    245245struct RunJavaScriptParameters;
    246246struct TextCheckingResult;
     247struct TextRecognitionResult;
    247248struct ViewportArguments;
    248249
    … …  
    14051406#if ENABLE(IMAGE_ANALYSIS)
    14061407    void requestTextRecognition(WebCore::Element&, CompletionHandler<void(RefPtr<WebCore::Element>&&)>&&);
    1407     void updateWithTextRecognitionResult(WebCore::TextRecognitionResult&&, const WebCore::ElementContext&, const WebCore::FloatPoint& location, CompletionHandler<void(TextRecognitionUpdateResult)>&&);
     1408    void updateWithTextRecognitionResult(const WebCore::TextRecognitionResult&, const WebCore::ElementContext&, const WebCore::FloatPoint& location, CompletionHandler<void(TextRecognitionUpdateResult)>&&);
    14081409#endif
    14091410
    … …  
    23492350#if ENABLE(IMAGE_ANALYSIS)
    23502351    Vector<std::pair<WeakPtr<WebCore::HTMLElement>, Vector<CompletionHandler<void(RefPtr<WebCore::Element>&&)>>>> m_elementsPendingTextRecognition;
    2351     WeakHashSet<WebCore::HTMLElement> m_elementsWithTextRecognitionResults;
    23522352#endif
    23532353
Note: See TracChangeset for help on using the changeset viewer.