Changeset 278775 in webkit


Ignore:
Timestamp:
Jun 11, 2021 12:55:13 PM (13 months ago)
Author:
Wenson Hsieh
Message:

[Live Text] Text selection inside image elements should not be cleared upon resize
​https://bugs.webkit.org/show_bug.cgi?id=226911

Reviewed by Tim Horton.

Source/WebCore:

Refactor HTMLElement::updateWithTextRecognitionResult, such that it doesn't tear down and recreate the host
element's shadow DOM structure in the case where the extant DOM elements are compatible with the given text
recognition result. This prevents us from removing or inserting DOM elements in the case where an image element
is resized (and thus adjusts its shadow DOM content using the updated size), which in turn prevents us from
clearing out the text selection.

Test: fast/images/text-recognition/mac/image-overlay-maintain-selection-during-size-change.html

  • editing/cocoa/DataDetection.h:
  • editing/cocoa/DataDetection.mm:

Make this helper method return an HTMLDivElement instead of just an HTMLElement.

(WebCore::DataDetection::createElementForImageOverlay):

  • html/HTMLElement.cpp:

(WebCore::HTMLElement::updateWithTextRecognitionResult):

Split this method into two logical parts: the first builds up a TextRecognitionElements struct that contains
references to all connected elements in the image element's shadow DOM that require style updates due to the
new size; the second uses this TextRecognitionElements information to compute the new CSS transforms to apply to
each of the data detector, line containers, and text containers underneath each line container element.

Importantly, in step (1), we avoid regenerating shadow DOM content in the case where the DOM elements already
exist in their expected places within the shadow DOM.

LayoutTests:

  • fast/images/text-recognition/mac/image-overlay-maintain-selection-during-size-change-expected.txt: Added.
  • fast/images/text-recognition/mac/image-overlay-maintain-selection-during-size-change.html: Added.
Location:
trunk
Files:
2 added
5 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r278765 r278775  
     12021-06-11  Wenson Hsieh  <wenson_hsieh@apple.com>
     2
     3        [Live Text] Text selection inside image elements should not be cleared upon resize
     4        https://bugs.webkit.org/show_bug.cgi?id=226911
     5
     6        Reviewed by Tim Horton.
     7
     8        * fast/images/text-recognition/mac/image-overlay-maintain-selection-during-size-change-expected.txt: Added.
     9        * fast/images/text-recognition/mac/image-overlay-maintain-selection-during-size-change.html: Added.
     10
    1112021-06-11  Cathie Chen  <cathiechen@igalia.com>
    212

  • trunk/Source/WebCore/ChangeLog

    r278767 r278775  
     12021-06-11  Wenson Hsieh  <wenson_hsieh@apple.com>
     2
     3        [Live Text] Text selection inside image elements should not be cleared upon resize
     4        https://bugs.webkit.org/show_bug.cgi?id=226911
     5
     6        Reviewed by Tim Horton.
     7
     8        Refactor `HTMLElement::updateWithTextRecognitionResult`, such that it doesn't tear down and recreate the host
     9        element's shadow DOM structure in the case where the extant DOM elements are compatible with the given text
     10        recognition result. This prevents us from removing or inserting DOM elements in the case where an image element
     11        is resized (and thus adjusts its shadow DOM content using the updated size), which in turn prevents us from
     12        clearing out the text selection.
     13
     14        Test: fast/images/text-recognition/mac/image-overlay-maintain-selection-during-size-change.html
     15
     16        * editing/cocoa/DataDetection.h:
     17        * editing/cocoa/DataDetection.mm:
     18
     19        Make this helper method return an HTMLDivElement instead of just an HTMLElement.
     20
     21        (WebCore::DataDetection::createElementForImageOverlay):
     22        * html/HTMLElement.cpp:
     23        (WebCore::HTMLElement::updateWithTextRecognitionResult):
     24
     25        Split this method into two logical parts: the first builds up a TextRecognitionElements struct that contains
     26        references to all connected elements in the image element's shadow DOM that require style updates due to the
     27        new size; the second uses this TextRecognitionElements information to compute the new CSS transforms to apply to
     28        each of the data detector, line containers, and text containers underneath each line container element.
     29
     30        Importantly, in step (1), we avoid regenerating shadow DOM content in the case where the DOM elements already
     31        exist in their expected places within the shadow DOM.
     32
    1332021-06-11  Megan Gardner  <megan_gardner@apple.com>
    234

  • trunk/Source/WebCore/editing/cocoa/DataDetection.h

    r278575 r278775  
    4141
    4242class Document;
    43 class HTMLElement;
     43class HTMLDivElement;
    4444class HitTestResult;
    4545class QualifiedName;
    … …  
    6868
    6969#if ENABLE(IMAGE_ANALYSIS)
    70     static Ref<HTMLElement> createElementForImageOverlay(Document&, const TextRecognitionDataDetector&);
     70    static Ref<HTMLDivElement> createElementForImageOverlay(Document&, const TextRecognitionDataDetector&);
    7171#endif
    7272

  • trunk/Source/WebCore/editing/cocoa/DataDetection.mm

    r278669 r278775  
    690690#if ENABLE(IMAGE_ANALYSIS)
    691691
    692 Ref<HTMLElement> DataDetection::createElementForImageOverlay(Document& document, const TextRecognitionDataDetector& info)
     692Ref<HTMLDivElement> DataDetection::createElementForImageOverlay(Document& document, const TextRecognitionDataDetector& info)
    693693{
    694694    auto container = HTMLDivElement::create(document);

  • trunk/Source/WebCore/html/HTMLElement.cpp

    r278765 r278775  
    13421342void HTMLElement::updateWithTextRecognitionResult(const TextRecognitionResult& result, CacheTextRecognitionResults cacheTextRecognitionResults)
    13431343{
    1344     RefPtr<HTMLDivElement> previousContainer;
    1345     if (auto shadowRoot = userAgentShadowRoot(); shadowRoot && hasImageOverlay()) {
    1346         for (auto& child : childrenOfType<HTMLDivElement>(*shadowRoot)) {
     1344    static MainThreadNeverDestroyed<const AtomString> imageOverlayLineClass("image-overlay-line", AtomString::ConstructFromLiteral);
     1345    static MainThreadNeverDestroyed<const AtomString> imageOverlayTextClass("image-overlay-text", AtomString::ConstructFromLiteral);
     1346
     1347    struct TextRecognitionLineElements {
     1348        Ref<HTMLDivElement> line;
     1349        Vector<Ref<HTMLDivElement>> children;
     1350    };
     1351
     1352    struct TextRecognitionElements {
     1353        RefPtr<HTMLDivElement> root;
     1354        Vector<TextRecognitionLineElements> lines;
     1355        Vector<Ref<HTMLDivElement>> dataDetectors;
     1356    };
     1357
     1358    bool hadExistingTextRecognitionElements = false;
     1359    TextRecognitionElements textRecognitionElements;
     1360
     1361    if (hasImageOverlay()) {
     1362        for (auto& child : childrenOfType<HTMLDivElement>(*userAgentShadowRoot())) {
    13471363            if (child.getIdAttribute() == imageOverlayElementIdentifier()) {
    1348                 previousContainer = &child;
     1364                textRecognitionElements.root = &child;
     1365                hadExistingTextRecognitionElements = true;
    13491366                break;
    13501367            }
    13511368        }
    1352         if (previousContainer)
    1353             previousContainer->remove();
    1354         else
    1355             ASSERT_NOT_REACHED();
     1369    }
     1370
     1371    if (textRecognitionElements.root) {
     1372        for (auto& lineOrDataDetector : childrenOfType<HTMLDivElement>(*textRecognitionElements.root)) {
     1373            if (!lineOrDataDetector.hasClass())
     1374                continue;
     1375
     1376            if (lineOrDataDetector.classList().contains(imageOverlayLineClass)) {
     1377                TextRecognitionLineElements lineElements { lineOrDataDetector };
     1378                for (auto& text : childrenOfType<HTMLDivElement>(lineOrDataDetector))
     1379                    lineElements.children.append(text);
     1380                textRecognitionElements.lines.append(WTFMove(lineElements));
     1381            } else if (lineOrDataDetector.classList().contains(imageOverlayDataDetectorClassName()))
     1382                textRecognitionElements.dataDetectors.append(lineOrDataDetector);
     1383        }
     1384
     1385        bool canUseExistingTextRecognitionElements = ([&] {
     1386            if (result.dataDetectors.size() != textRecognitionElements.dataDetectors.size())
     1387                return false;
     1388
     1389            if (result.lines.size() != textRecognitionElements.lines.size())
     1390                return false;
     1391
     1392            for (size_t lineIndex = 0; lineIndex < result.lines.size(); ++lineIndex) {
     1393                auto& childResults = result.lines[lineIndex].children;
     1394                auto& childTextElements = textRecognitionElements.lines[lineIndex].children;
     1395                if (childResults.size() != childTextElements.size())
     1396                    return false;
     1397
     1398                for (size_t childIndex = 0; childIndex < childResults.size(); ++childIndex) {
     1399                    if (childResults[childIndex].text != childTextElements[childIndex]->textContent().stripWhiteSpace())
     1400                        return false;
     1401                }
     1402            }
     1403
     1404            return true;
     1405        })();
     1406
     1407        if (!canUseExistingTextRecognitionElements) {
     1408            textRecognitionElements.root->remove();
     1409            textRecognitionElements = { };
     1410        }
    13561411    }
    13571412
    13581413    if (result.isEmpty())
    13591414        return;
     1415
     1416    auto shadowRoot = makeRef(ensureUserAgentShadowRoot());
     1417    bool hasUserSelectNone = renderer() && renderer()->style().userSelect() == UserSelect::None;
     1418    if (!textRecognitionElements.root) {
     1419        auto rootContainer = HTMLDivElement::create(document());
     1420        rootContainer->setIdAttribute(imageOverlayElementIdentifier());
     1421        if (document().isImageDocument())
     1422            rootContainer->setInlineStyleProperty(CSSPropertyWebkitUserSelect, CSSValueText);
     1423        shadowRoot->appendChild(rootContainer);
     1424        textRecognitionElements.root = rootContainer.copyRef();
     1425        textRecognitionElements.lines.reserveInitialCapacity(result.lines.size());
     1426        for (auto& line : result.lines) {
     1427            auto lineContainer = HTMLDivElement::create(document());
     1428            lineContainer->classList().add(imageOverlayLineClass);
     1429            rootContainer->appendChild(lineContainer);
     1430            TextRecognitionLineElements lineElements { lineContainer };
     1431            lineElements.children.reserveInitialCapacity(line.children.size());
     1432            for (size_t childIndex = 0; childIndex < line.children.size(); ++childIndex) {
     1433                auto& child = line.children[childIndex];
     1434                auto textContainer = HTMLDivElement::create(document());
     1435                textContainer->classList().add(imageOverlayTextClass);
     1436                lineContainer->appendChild(textContainer);
     1437                textContainer->appendChild(Text::create(document(), makeString('\n', child.text)));
     1438                lineElements.children.uncheckedAppend(WTFMove(textContainer));
     1439            }
     1440
     1441            lineContainer->appendChild(HTMLBRElement::create(document()));
     1442            textRecognitionElements.lines.uncheckedAppend(WTFMove(lineElements));
     1443        }
     1444
     1445#if ENABLE(DATA_DETECTION)
     1446        textRecognitionElements.dataDetectors.reserveInitialCapacity(result.dataDetectors.size());
     1447        for (auto& dataDetector : result.dataDetectors) {
     1448            auto dataDetectorContainer = DataDetection::createElementForImageOverlay(document(), dataDetector);
     1449            dataDetectorContainer->classList().add(imageOverlayDataDetectorClassName());
     1450            rootContainer->appendChild(dataDetectorContainer);
     1451            textRecognitionElements.dataDetectors.uncheckedAppend(WTFMove(dataDetectorContainer));
     1452        }
     1453#endif // ENABLE(DATA_DETECTION)
     1454
     1455        if (document().quirks().needsToForceUserSelectWhenInstallingImageOverlay())
     1456            setInlineStyleProperty(CSSPropertyWebkitUserSelect, CSSValueText);
     1457    }
    13601458
    13611459    if (auto* renderer = this->renderer()) {
    … …  
    13661464    }
    13671465
    1368     auto shadowRoot = makeRef(ensureUserAgentShadowRoot());
    1369     if (!previousContainer) {
     1466    if (!hadExistingTextRecognitionElements) {
    13701467        static MainThreadNeverDestroyed<const String> shadowStyle(StringImpl::createWithoutCopying(imageOverlayUserAgentStyleSheet, sizeof(imageOverlayUserAgentStyleSheet)));
    13711468        auto style = HTMLStyleElement::create(HTMLNames::styleTag, document(), false);
    … …  
    13741471    }
    13751472
    1376     auto container = HTMLDivElement::create(document());
    1377     container->setIdAttribute(imageOverlayElementIdentifier());
    1378     if (document().isImageDocument())
    1379         container->setInlineStyleProperty(CSSPropertyWebkitUserSelect, CSSValueText);
    1380     shadowRoot->appendChild(container);
    1381 
    1382     if (document().quirks().needsToForceUserSelectWhenInstallingImageOverlay())
    1383         setInlineStyleProperty(CSSPropertyWebkitUserSelect, CSSValueText);
    1384 
    1385     static MainThreadNeverDestroyed<const AtomString> imageOverlayLineClass("image-overlay-line", AtomString::ConstructFromLiteral);
    1386     static MainThreadNeverDestroyed<const AtomString> imageOverlayTextClass("image-overlay-text", AtomString::ConstructFromLiteral);
    1387 
    1388     bool hasUserSelectNone = renderer() && renderer()->style().userSelect() == UserSelect::None;
    13891473    IntSize containerSize { offsetWidth(), offsetHeight() };
    1390     for (auto& line : result.lines) {
     1474    for (size_t lineIndex = 0; lineIndex < result.lines.size(); ++lineIndex) {
     1475        auto& lineElements = textRecognitionElements.lines[lineIndex];
     1476        auto& lineContainer = lineElements.line;
     1477        auto& line = result.lines[lineIndex];
    13911478        auto lineQuad = line.normalizedQuad;
    13921479        if (lineQuad.isEmpty())
    … …  
    13941481
    13951482        lineQuad.scale(containerSize.width(), containerSize.height());
    1396 
    1397         auto lineContainer = HTMLDivElement::create(document());
    1398         lineContainer->classList().add(imageOverlayLineClass);
    1399         container->appendChild(lineContainer);
    14001483
    14011484        auto lineBounds = rotatedBoundingRectWithMinimumAngleOfRotation(lineQuad, 0.01);
    … …  
    14271510
    14281511        for (size_t childIndex = 0; childIndex < line.children.size(); ++childIndex) {
    1429             auto& child = line.children[childIndex];
    1430             if (child.normalizedQuad.isEmpty())
    1431                 continue;
    1432 
     1512            auto& textContainer = lineElements.children[childIndex];
    14331513            bool lineHasOneChild = line.children.size() == 1;
    14341514            float horizontalMarginToMinimizeSelectionGaps = lineHasOneChild ? 0 : 0.125;
    … …  
    14511531
    14521532            FloatSize targetSize { horizontalExtent - horizontalOffset, lineBounds.size.height() };
    1453             if (targetSize.isEmpty())
     1533            if (targetSize.isEmpty()) {
     1534                textContainer->setInlineStyleProperty(CSSPropertyTransform, "scale(0, 0)");
    14541535                continue;
    1455 
    1456             auto textContainer = HTMLDivElement::create(document());
    1457             textContainer->classList().add(imageOverlayTextClass);
    1458             lineContainer->appendChild(textContainer);
    1459             textContainer->appendChild(Text::create(document(), makeString('\n', child.text)));
     1536            }
    14601537
    14611538            document().updateLayoutIfDimensionsOutOfDate(textContainer);
    … …  
    14701547
    14711548            if (sizeBeforeTransform.isEmpty()) {
    1472                 textContainer->remove();
     1549                textContainer->setInlineStyleProperty(CSSPropertyTransform, "scale(0, 0)");
    14731550                continue;
    14741551            }
    … …  
    14921569
    14931570#if ENABLE(DATA_DETECTION)
    1494     for (auto& dataDetector : result.dataDetectors) {
     1571    for (size_t index = 0; index < result.dataDetectors.size(); ++index) {
     1572        auto dataDetectorContainer = textRecognitionElements.dataDetectors[index];
     1573        auto& dataDetector = result.dataDetectors[index];
    14951574        if (dataDetector.normalizedQuads.isEmpty())
    14961575            continue;
    1497 
    1498         auto dataDetectorContainer = DataDetection::createElementForImageOverlay(document(), dataDetector);
    1499         dataDetectorContainer->classList().add(imageOverlayDataDetectorClassName());
    1500         container->appendChild(dataDetectorContainer);
    15011576
    15021577        // FIXME: We should come up with a way to coalesce the bounding quads into one or more rotated rects with the same angle of rotation.
Note: See TracChangeset for help on using the changeset viewer.