Changeset 279751 in webkit

Timestamp:

Jul 8, 2021 1:59:32 PM (13 months ago)

Author:

Wenson Hsieh

Message:

[Live Text] Selection is misaligned on some images on twitter.com
​https://bugs.webkit.org/show_bug.cgi?id=227775
rdar://77142364

Reviewed by Tim Horton.

Source/WebCore:

On Twitter, image thumbnails consist of fully transparent image elements (opacity: 0;) that are overlaid over
div elements with a background image. The images for some of these transparent image elements sometimes don't
match the background image behind it (namely, the image is stretched to exactly fit the bounds of the
transparent image element in a way that does not preserve aspect ratio, whereas the background image is cropped
to cover the div). As such, Live Text in these image elements ends up misaligned against the background image
(which is actually visible to the user).

To address this, special case fully transparent images such that we take a snapshot of the page using the
absolute bounds of the transparent image and run OCR over this snapshot, instead of using the image data as-is.

Test: fast/images/text-recognition/image-overlay-in-transparent-image.html

• html/HTMLElement.cpp:

(WebCore::HTMLElement::containerRectForTextRecognition):
(WebCore::HTMLElement::updateWithTextRecognitionResult):

• html/HTMLElement.h:
• page/Page.cpp:

(WebCore::Page::updateElementsWithTextRecognitionResults):

Source/WebKit:

See WebCore ChangeLog for more details.

• WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:

(WebKit::createShareableBitmap):

• WebProcess/WebCoreSupport/ShareableBitmapUtilities.h:

(WebKit::createShareableBitmap):

Add an option to handle the scenario where the image renderer is fully transparent by falling back to a frame-
level snapshot, using the bounds of the image renderer.

• WebProcess/WebPage/WebPage.cpp:

(WebKit::WebPage::requestTextRecognition):

• WebProcess/WebPage/ios/WebPageIOS.mm:

(WebKit::imagePositionInformation):
(WebKit::elementPositionInformation):

LayoutTests:

• fast/images/text-recognition/image-overlay-in-transparent-image-expected.txt: Added.
• fast/images/text-recognition/image-overlay-in-transparent-image.html: Added.

Location:

trunk

Files:

• LayoutTests/ChangeLog (modified) (1 diff)
• LayoutTests/fast/images/text-recognition/image-overlay-in-transparent-image-expected.txt (added)
• LayoutTests/fast/images/text-recognition/image-overlay-in-transparent-image.html (added)
• Source/WebCore/ChangeLog (modified) (1 diff)
• Source/WebCore/html/HTMLElement.cpp (modified) (2 diffs)
• Source/WebCore/html/HTMLElement.h (modified) (1 diff)
• Source/WebCore/page/Page.cpp (modified) (1 diff)
• Source/WebKit/ChangeLog (modified) (1 diff)
• Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp (modified) (2 diffs)
• Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.h (modified) (1 diff)
• Source/WebKit/WebProcess/WebPage/WebPage.cpp (modified) (1 diff)
• Source/WebKit/WebProcess/WebPage/ios/WebPageIOS.mm (modified) (2 diffs)

trunk/LayoutTests/ChangeLog

@@ +1 @@
+2021-07-08  Wenson Hsieh  <wenson_hsieh@apple.com>
+
+        [Live Text] Selection is misaligned on some images on twitter.com
+        https://bugs.webkit.org/show_bug.cgi?id=227775
+        rdar://77142364
+
+        Reviewed by Tim Horton.
+
+        * fast/images/text-recognition/image-overlay-in-transparent-image-expected.txt: Added.
+        * fast/images/text-recognition/image-overlay-in-transparent-image.html: Added.
+
 2021-07-08  Kate Cheney  <katherine_cheney@apple.com>
 

trunk/Source/WebCore/ChangeLog

@@ +1 @@
+2021-07-08  Wenson Hsieh  <wenson_hsieh@apple.com>
+
+        [Live Text] Selection is misaligned on some images on twitter.com
+        https://bugs.webkit.org/show_bug.cgi?id=227775
+        rdar://77142364
+
+        Reviewed by Tim Horton.
+
+        On Twitter, image thumbnails consist of fully transparent image elements (`opacity: 0;`) that are overlaid over
+        `div` elements with a background image. The images for some of these transparent image elements sometimes don't
+        match the background image behind it (namely, the image is stretched to exactly fit the bounds of the
+        transparent image element in a way that does not preserve aspect ratio, whereas the background image is cropped
+        to cover the `div`). As such, Live Text in these image elements ends up misaligned against the background image
+        (which is actually visible to the user).
+
+        To address this, special case fully transparent images such that we take a snapshot of the page using the
+        absolute bounds of the transparent image and run OCR over this snapshot, instead of using the image data as-is.
+
+        Test: fast/images/text-recognition/image-overlay-in-transparent-image.html
+
+        * html/HTMLElement.cpp:
+        (WebCore::HTMLElement::containerRectForTextRecognition):
+        (WebCore::HTMLElement::updateWithTextRecognitionResult):
+        * html/HTMLElement.h:
+        * page/Page.cpp:
+        (WebCore::Page::updateElementsWithTextRecognitionResults):
+
 2021-07-08  Kate Cheney  <katherine_cheney@apple.com>
 

trunk/Source/WebCore/html/HTMLElement.cpp

@@ -1341 +1341 @@
 #if ENABLE(IMAGE_ANALYSIS)
 
+IntRect HTMLElement::containerRectForTextRecognition()
+{
+    auto* renderer = this->renderer();
+    if (!is<RenderImage>(renderer))
+        return { };
+
+    if (!renderer->opacity())
+        return { 0, 0, offsetWidth(), offsetHeight() };
+
+    return enclosingIntRect(downcast<RenderImage>(*renderer).replacedContentRect());
+}
+
 void HTMLElement::updateWithTextRecognitionResult(const TextRecognitionResult& result, CacheTextRecognitionResults cacheTextRecognitionResults)
 {
@@ -1473 +1485 @@
     downcast<RenderImage>(*renderer).setHasImageOverlay();
 
-    auto containerRect = enclosingIntRect(downcast<RenderImage>(*renderer).replacedContentRect());
+    auto containerRect = containerRectForTextRecognition();
     auto convertToContainerCoordinates = [&](const FloatQuad& normalizedQuad) {
         auto quad = normalizedQuad;

trunk/Source/WebCore/html/HTMLElement.h

@@ -140 +140 @@
 
 #if ENABLE(IMAGE_ANALYSIS)
+    IntRect containerRectForTextRecognition();
     enum class CacheTextRecognitionResults : bool { No, Yes };
     WEBCORE_EXPORT void updateWithTextRecognitionResult(const TextRecognitionResult&, CacheTextRecognitionResults = CacheTextRecognitionResults::Yes);

trunk/Source/WebCore/page/Page.cpp

@@ -3631 +3631 @@
             continue;
 
-        auto newContainerRect = enclosingIntRect(downcast<RenderImage>(*renderer).replacedContentRect());
+        auto newContainerRect = protectedElement->containerRectForTextRecognition();
         if (containerRect == newContainerRect)
             continue;

trunk/Source/WebKit/ChangeLog

@@ +1 @@
+2021-07-08  Wenson Hsieh  <wenson_hsieh@apple.com>
+
+        [Live Text] Selection is misaligned on some images on twitter.com
+        https://bugs.webkit.org/show_bug.cgi?id=227775
+        rdar://77142364
+
+        Reviewed by Tim Horton.
+
+        See WebCore ChangeLog for more details.
+
+        * WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:
+        (WebKit::createShareableBitmap):
+        * WebProcess/WebCoreSupport/ShareableBitmapUtilities.h:
+        (WebKit::createShareableBitmap):
+
+        Add an option to handle the scenario where the image renderer is fully transparent by falling back to a frame-
+        level snapshot, using the bounds of the image renderer.
+
+        * WebProcess/WebPage/WebPage.cpp:
+        (WebKit::WebPage::requestTextRecognition):
+        * WebProcess/WebPage/ios/WebPageIOS.mm:
+        (WebKit::imagePositionInformation):
+        (WebKit::elementPositionInformation):
+
 2021-07-08  Kate Cheney  <katherine_cheney@apple.com>
 

trunk/Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp

@@ -29 +29 @@
 #include "ShareableBitmap.h"
 #include <WebCore/CachedImage.h>
+#include <WebCore/Frame.h>
+#include <WebCore/FrameSnapshotting.h>
 #include <WebCore/GeometryUtilities.h>
 #include <WebCore/GraphicsContext.h>
+#include <WebCore/ImageBuffer.h>
 #include <WebCore/IntSize.h>
 #include <WebCore/PlatformScreen.h>
@@ -38 +41 @@
 using namespace WebCore;
 
-RefPtr<ShareableBitmap> createShareableBitmap(RenderImage& renderImage, std::optional<FloatSize> screenSizeInPixels, AllowAnimatedImages allowAnimatedImages)
+RefPtr<ShareableBitmap> createShareableBitmap(RenderImage& renderImage, CreateShareableBitmapFromImageOptions&& options)
 {
+    Ref frame = renderImage.frame();
+    auto colorSpaceForBitmap = screenColorSpace(frame->mainFrame().view());
+    if (!renderImage.opacity() && options.useSnapshotForTransparentImages == UseSnapshotForTransparentImages::Yes) {
+        auto snapshotRect = renderImage.absoluteBoundingBoxRect();
+        if (snapshotRect.isEmpty())
+            return { };
+
+        OptionSet<SnapshotFlags> snapshotFlags { SnapshotFlags::ExcludeSelectionHighlighting, SnapshotFlags::PaintEverythingExcludingSelection };
+        auto imageBuffer = snapshotFrameRect(frame.get(), snapshotRect, { snapshotFlags, PixelFormat::BGRA8, DestinationColorSpace::SRGB() });
+        if (!imageBuffer)
+            return { };
+
+        auto snapshotImage = ImageBuffer::sinkIntoImage(WTFMove(imageBuffer), PreserveResolution::Yes);
+        if (!snapshotImage)
+            return { };
+
+        auto bitmap = ShareableBitmap::createShareable(snapshotRect.size(), { WTFMove(colorSpaceForBitmap) });
+        if (!bitmap)
+            return { };
+
+        auto context = bitmap->createGraphicsContext();
+        if (!context)
+            return { };
+
+        context->drawImage(*snapshotImage, { FloatPoint::zero(), snapshotRect.size() });
+        return bitmap;
+    }
+
     auto* cachedImage = renderImage.cachedImage();
     if (!cachedImage || cachedImage->errorOccurred())
-        return nullptr;
+        return { };
 
     auto* image = cachedImage->imageForRenderer(&renderImage);
     if (!image || image->width() <= 1 || image->height() <= 1)
-        return nullptr;
+        return { };
 
-    if (allowAnimatedImages == AllowAnimatedImages::No && image->isAnimated())
-        return nullptr;
+    if (options.allowAnimatedImages == AllowAnimatedImages::No && image->isAnimated())
+        return { };
 
     auto bitmapSize = cachedImage->imageSizeForRenderer(&renderImage);
-    if (screenSizeInPixels) {
-        auto scaledSize = largestRectWithAspectRatioInsideRect(bitmapSize.width() / bitmapSize.height(), { FloatPoint(), *screenSizeInPixels }).size();
+    if (options.screenSizeInPixels) {
+        auto scaledSize = largestRectWithAspectRatioInsideRect(bitmapSize.width() / bitmapSize.height(), { FloatPoint(), *options.screenSizeInPixels }).size();
         bitmapSize = scaledSize.width() < bitmapSize.width() ? scaledSize : bitmapSize;
     }
 
     // FIXME: Only select ExtendedColor on images known to need wide gamut.
-    auto sharedBitmap = ShareableBitmap::createShareable(IntSize(bitmapSize), { screenColorSpace(renderImage.frame().mainFrame().view()) });
+    auto sharedBitmap = ShareableBitmap::createShareable(IntSize(bitmapSize), { WTFMove(colorSpaceForBitmap) });
     if (!sharedBitmap)
-        return nullptr;
+        return { };
 
     auto graphicsContext = sharedBitmap->createGraphicsContext();
     if (!graphicsContext)
-        return nullptr;
+        return { };
 
     graphicsContext->drawImage(*image, FloatRect(0, 0, bitmapSize.width(), bitmapSize.height()), { renderImage.imageOrientation() });

trunk/Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.h

@@ -37 +37 @@
 class ShareableBitmap;
 
+enum class UseSnapshotForTransparentImages : bool { No, Yes };
 enum class AllowAnimatedImages : bool { No, Yes };
-RefPtr<ShareableBitmap> createShareableBitmap(WebCore::RenderImage&, std::optional<WebCore::FloatSize> screenSizeInPixels = std::nullopt, AllowAnimatedImages = AllowAnimatedImages::Yes);
+
+struct CreateShareableBitmapFromImageOptions {
+    std::optional<WebCore::FloatSize> screenSizeInPixels;
+    AllowAnimatedImages allowAnimatedImages { AllowAnimatedImages::Yes };
+    UseSnapshotForTransparentImages useSnapshotForTransparentImages { UseSnapshotForTransparentImages::No };
+};
+
+RefPtr<ShareableBitmap> createShareableBitmap(WebCore::RenderImage&, CreateShareableBitmapFromImageOptions&& = { });
 
 };

trunk/Source/WebKit/WebProcess/WebPage/WebPage.cpp

@@ -7407 +7407 @@
 
     auto& renderImage = downcast<RenderImage>(*renderer);
-    auto bitmap = createShareableBitmap(renderImage, std::nullopt, AllowAnimatedImages::No);
+    auto bitmap = createShareableBitmap(renderImage, { std::nullopt, AllowAnimatedImages::No, UseSnapshotForTransparentImages::Yes });
     if (!bitmap) {
         if (completion)

trunk/Source/WebKit/WebProcess/WebPage/ios/WebPageIOS.mm

@@ -2824 +2824 @@
 
     if (request.includeSnapshot || request.includeImageData)
-        info.image = createShareableBitmap(renderImage, screenSize() * page.corePage()->deviceScaleFactor());
+        info.image = createShareableBitmap(renderImage, { screenSize() * page.corePage()->deviceScaleFactor(), AllowAnimatedImages::Yes, UseSnapshotForTransparentImages::Yes });
 
     info.imageElementContext = page.contextForElement(element);
@@ -2886 +2886 @@
                     auto& [renderImage, image] = *rendererAndImage;
                     info.imageURL = element.document().completeURL(renderImage.cachedImage()->url().string());
-                    info.image = createShareableBitmap(renderImage, screenSize() * page.corePage()->deviceScaleFactor());
+                    info.image = createShareableBitmap(renderImage, { screenSize() * page.corePage()->deviceScaleFactor(), AllowAnimatedImages::Yes, UseSnapshotForTransparentImages::Yes });
                 }
             }