⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 278276 in webkit


Ignore:
Timestamp:
May 31, 2021, 6:33:16 AM (5 years ago)
Author:
magomez@igalia.com
Message:

[GStreamer] Incorrect rendering of VP9 with transparency
​https://bugs.webkit.org/show_bug.cgi?id=225961

Reviewed by Žan Doberšek.

Source/WebCore:

Fix the rendering of videos with transparency. This is done both AC and non AC modes and
with and without GSTREAMER_GL enabled.

On AC mode and using GSTREAMER_GL, a new option is added to TextureMapperGL and TextureMapperShaderProgram
to support premultiplying the components of a pixel. This is required because GStreamer uses straight alpha,
and we need it premultiplied in order to perform the correct blending blending of the video frames. Then we
use that new option to render video frames. This new option is also used when copying the video frame into
an external texture if premultiplyAlpha is requested.

On non AC mode, or when GStreamer-gl is disabled, add support to ImageGStreamerCairo to premultiply the alpha
of the video frame before creating the cairo surface to paint, and pass the appropriate composite operator when
drawing with alpha.

Test: media/video-with-alpha.html

  • platform/graphics/gstreamer/GStreamerVideoFrameHolder.cpp:

(WebCore::GstVideoFrameHolder::GstVideoFrameHolder):

  • platform/graphics/gstreamer/ImageGStreamerCairo.cpp:

(WebCore::ImageGStreamer::ImageGStreamer):

  • platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:

(WebCore::MediaPlayerPrivateGStreamer::pushTextureToCompositor):
(WebCore::MediaPlayerPrivateGStreamer::paint): update the caps of the conversion to make them match the format
expected by cairo, so we can avoid a component swap.
(WebCore::MediaPlayerPrivateGStreamer::copyVideoTextureToPlatformTexture):

  • platform/graphics/gstreamer/VideoTextureCopierGStreamer.cpp:

(WebCore::VideoTextureCopierGStreamer::copyVideoTextureToPlatformTexture):

  • platform/graphics/gstreamer/VideoTextureCopierGStreamer.h:
  • platform/graphics/texmap/TextureMapperGL.cpp:

(WebCore::TextureMapperGL::drawTexture):

  • platform/graphics/texmap/TextureMapperGL.h:
  • platform/graphics/texmap/TextureMapperShaderProgram.cpp:

(WebCore::TextureMapperShaderProgram::create):

  • platform/graphics/texmap/TextureMapperShaderProgram.h:

LayoutTests:

Add a test for the playback of transparent videos. This is initially skipped because it's not supported
on Apple ports and won't be working on GTK and WPE until GStreamer is bumped to 1.20.

  • TestExpectations:
  • media/content/video-with-alpha.webm: Added.
  • media/video-with-alpha-expected.html: Added.
  • media/video-with-alpha.html: Added.
Location:
trunk
Files:
3 added
12 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r278275 r278276  
     12021-05-31  Miguel Gomez  <magomez@igalia.com>
     2
     3        [GStreamer] Incorrect rendering of VP9 with transparency
     4        https://bugs.webkit.org/show_bug.cgi?id=225961
     5
     6        Reviewed by Žan Doberšek.
     7
     8        Add a test for the playback of transparent videos. This is initially skipped because it's not supported
     9        on Apple ports and won't be working on GTK and WPE until GStreamer is bumped to 1.20.
     10
     11        * TestExpectations:
     12        * media/content/video-with-alpha.webm: Added.
     13        * media/video-with-alpha-expected.html: Added.
     14        * media/video-with-alpha.html: Added.
     15
    1162021-05-31  Rob Buis  <rbuis@igalia.com>
    217

  • trunk/LayoutTests/TestExpectations

    r278268 r278276  
    10341034# Tests a setting behind the ENABLE_FULLSCREEN_API flag, which is WK2-only
    10351035media/video-supports-fullscreen.html [ Skip ]
     1036
     1037# Support for VP9 encoded videos with transparency is only available on ports that use GStreamer.
     1038media/video-with-alpha.html [ Skip ]
    10361039
    10371040#//////////////////////////////////////////////////////////////////////////////////////////

  • trunk/Source/WebCore/ChangeLog

    r278275 r278276  
     12021-05-31  Miguel Gomez  <magomez@igalia.com>
     2
     3        [GStreamer] Incorrect rendering of VP9 with transparency
     4        https://bugs.webkit.org/show_bug.cgi?id=225961
     5
     6        Reviewed by Žan Doberšek.
     7
     8        Fix the rendering of videos with transparency. This is done both AC and non AC modes and
     9        with and without GSTREAMER_GL enabled.
     10
     11        On AC mode and using GSTREAMER_GL, a new option is added to TextureMapperGL and TextureMapperShaderProgram
     12        to support premultiplying the components of a pixel. This is required because GStreamer uses straight alpha,
     13        and we need it premultiplied in order to perform the correct blending blending of the video frames. Then we
     14        use that new option to render video frames. This new option is also used when copying the video frame into
     15        an external texture if premultiplyAlpha is requested.
     16
     17        On non AC mode, or when GStreamer-gl is disabled, add support to ImageGStreamerCairo to premultiply the alpha
     18        of the video frame before creating the cairo surface to paint, and pass the appropriate composite operator when
     19        drawing with alpha.
     20
     21        Test: media/video-with-alpha.html
     22
     23        * platform/graphics/gstreamer/GStreamerVideoFrameHolder.cpp:
     24        (WebCore::GstVideoFrameHolder::GstVideoFrameHolder):
     25        * platform/graphics/gstreamer/ImageGStreamerCairo.cpp:
     26        (WebCore::ImageGStreamer::ImageGStreamer):
     27        * platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:
     28        (WebCore::MediaPlayerPrivateGStreamer::pushTextureToCompositor):
     29        (WebCore::MediaPlayerPrivateGStreamer::paint): update the caps of the conversion to make them match the format
     30        expected by cairo, so we can avoid a component swap.
     31        (WebCore::MediaPlayerPrivateGStreamer::copyVideoTextureToPlatformTexture):
     32        * platform/graphics/gstreamer/VideoTextureCopierGStreamer.cpp:
     33        (WebCore::VideoTextureCopierGStreamer::copyVideoTextureToPlatformTexture):
     34        * platform/graphics/gstreamer/VideoTextureCopierGStreamer.h:
     35        * platform/graphics/texmap/TextureMapperGL.cpp:
     36        (WebCore::TextureMapperGL::drawTexture):
     37        * platform/graphics/texmap/TextureMapperGL.h:
     38        * platform/graphics/texmap/TextureMapperShaderProgram.cpp:
     39        (WebCore::TextureMapperShaderProgram::create):
     40        * platform/graphics/texmap/TextureMapperShaderProgram.h:
     41
    1422021-05-31  Rob Buis  <rbuis@igalia.com>
    243

  • trunk/Source/WebCore/platform/graphics/gstreamer/GStreamerVideoFrameHolder.cpp

    r278253 r278276  
    4747
    4848#if USE(GSTREAMER_GL)
    49     m_flags = flags | (m_hasAlphaChannel ? TextureMapperGL::ShouldBlend : 0);
     49    m_flags = flags | (m_hasAlphaChannel ? TextureMapperGL::ShouldBlend | TextureMapperGL::ShouldPremultiply : 0);
    5050
    5151    GstMemory* memory = gst_buffer_peek_memory(m_buffer.get(), 0);

  • trunk/Source/WebCore/platform/graphics/gstreamer/ImageGStreamerCairo.cpp

    r261922 r278276  
    4040        return;
    4141
    42     // Right now the TextureMapper only supports chromas with one plane
    43     ASSERT(GST_VIDEO_INFO_N_PLANES(&videoInfo) == 1);
    44 
    45     m_hasAlpha = GST_VIDEO_INFO_HAS_ALPHA(&videoInfo);
     42    // The frame has to RGB so we can paint it.
     43    ASSERT(GST_VIDEO_INFO_IS_RGB(&videoInfo));
    4644
    4745    GstBuffer* buffer = gst_sample_get_buffer(sample);
    … …  
    5351        return;
    5452
     53    // The video buffer may have these formats in these cases:
     54    // { BGRx, BGRA }: on little endian:
     55    //   - When GStreamer-gl is disabled (being AC enabled or not) as VideoSinkGStreamer is used.
     56    //   - When GStreamer-gl is enabled, but the caps used in the sink are not RGB and it's converted by the player to paint it.
     57    // { xRGB, ARGB }: on big endian:
     58    //   - When GStreamer-gl is disabled (being AC enabled or not) as VideoSinkGStreamer is used.
     59    //   - When GStreamer-gl is enabled, but the caps used in the sink are not RGB and it's converted by the player to paint it.
     60    // { RGBx, RGBA }
     61    //   - When GStreamer-gl is enabled and the caps used in the sink are RGBx/RGBA.
     62    //
     63    // Internally cairo uses BGRA for CAIRO_FORMAT_ARGB32 on little endian and ARGB on big endian, so both { BGRx, BGRA }
     64    // and { xRGB, ARGB } can be passed directly to cairo. But for { RGBx, RGBA } we need to swap the R and B components.
     65    // Also, GStreamer uses straight alpha while cairo requires it to be premultiplied, so if the format has alpha
     66    // we need to premultiply the color components. So in these cases we need to create a modified copy of the original
     67    // buffer.
     68    m_hasAlpha = GST_VIDEO_INFO_HAS_ALPHA(&videoInfo);
     69    bool componentSwapRequired = GST_VIDEO_FRAME_FORMAT(&m_videoFrame) == GST_VIDEO_FORMAT_RGBA || GST_VIDEO_FRAME_FORMAT(&m_videoFrame) == GST_VIDEO_FORMAT_RGBx;
    5570    unsigned char* bufferData = reinterpret_cast<unsigned char*>(GST_VIDEO_FRAME_PLANE_DATA(&m_videoFrame, 0));
    5671    int stride = GST_VIDEO_FRAME_PLANE_STRIDE(&m_videoFrame, 0);
    5772    int width = GST_VIDEO_FRAME_WIDTH(&m_videoFrame);
    5873    int height = GST_VIDEO_FRAME_HEIGHT(&m_videoFrame);
     74    RefPtr<cairo_surface_t> surface;
    5975
    60     RefPtr<cairo_surface_t> surface;
    61     cairo_format_t cairoFormat;
    62     cairoFormat = (GST_VIDEO_FRAME_FORMAT(&m_videoFrame) == GST_VIDEO_FORMAT_RGBA) ? CAIRO_FORMAT_ARGB32 : CAIRO_FORMAT_RGB24;
    63 
    64     // GStreamer doesn't use premultiplied alpha, but cairo does. So if the video format has an alpha component
    65     // we need to premultiply it before passing the data to cairo. This needs to be both using gstreamer-gl and not
    66     // using it.
    67     //
    68     // This method could be called several times for the same buffer, for example if we are rendering the video frames
    69     // in several non accelerated canvases. Due to this, we cannot modify the buffer, so we need to create a copy.
    70     if (cairoFormat == CAIRO_FORMAT_ARGB32) {
    71         unsigned char* surfaceData = static_cast<unsigned char*>(fastMalloc(height * stride));
    72         unsigned char* surfacePixel = surfaceData;
     76    if (m_hasAlpha || componentSwapRequired) {
     77        uint8_t* surfaceData = static_cast<uint8_t*>(fastMalloc(height * stride));
     78        uint8_t* surfacePixel = surfaceData;
    7379
    7480        for (int x = 0; x < width; x++) {
    7581            for (int y = 0; y < height; y++) {
    76                 unsigned short alpha = bufferData[3];
     82                // These store the source pixel components.
     83                uint16_t red;
     84                uint16_t green;
     85                uint16_t blue;
     86                uint16_t alpha;
     87                // These store the component offset inside the pixel for the destination surface.
     88                uint8_t redIndex;
     89                uint8_t greenIndex;
     90                uint8_t blueIndex;
     91                uint8_t alphaIndex;
     92                if (componentSwapRequired) {
     93                    // Source is RGBA or RGBx.
     94                    red = bufferData[0];
     95                    green = bufferData[1];
     96                    blue = bufferData[2];
     97                    alpha = bufferData[3];
    7798#if G_BYTE_ORDER == G_LITTLE_ENDIAN
    78                 // Video frames use RGBA in little endian.
    79                 surfacePixel[0] = (bufferData[2] * alpha + 128) / 255;
    80                 surfacePixel[1] = (bufferData[1] * alpha + 128) / 255;
    81                 surfacePixel[2] = (bufferData[0] * alpha + 128) / 255;
    82                 surfacePixel[3] = alpha;
     99                    // Destination is BGRA.
     100                    redIndex = 2;
     101                    greenIndex = 1;
     102                    blueIndex = 0;
     103                    alphaIndex = 3;
    83104#else
    84                 // Video frames use RGBA in big endian.
    85                 surfacePixel[0] = alpha;
    86                 surfacePixel[1] = (bufferData[0] * alpha + 128) / 255;
    87                 surfacePixel[2] = (bufferData[1] * alpha + 128) / 255;
    88                 surfacePixel[3] = (bufferData[2] * alpha + 128) / 255;
     105                    // Destination is ARGB.
     106                    redIndex = 1;
     107                    greenIndex = 2;
     108                    blueIndex = 3;
     109                    alphaIndex = 0;
    89110#endif
     111                } else {
     112#if G_BYTE_ORDER == G_LITTLE_ENDIAN
     113                    // BGRA or BGRx both source and destination.
     114                    red = bufferData[2];
     115                    green = bufferData[1];
     116                    blue = bufferData[0];
     117                    alpha = bufferData[3];
     118                    redIndex = 2;
     119                    greenIndex = 1;
     120                    blueIndex = 0;
     121                    alphaIndex = 3;
     122#else
     123                    // ARGB ot xRGB both source and destination.
     124                    red = bufferData[1];
     125                    green = bufferData[2];
     126                    blue = bufferData[3];
     127                    alpha = bufferData[0];
     128                    redIndex = 1;
     129                    greenIndex = 2;
     130                    blueIndex = 3;
     131                    alphaIndex = 0;
     132#endif
     133                }
     134
     135                if (m_hasAlpha) {
     136                    surfacePixel[redIndex] = red * alpha / 255;
     137                    surfacePixel[greenIndex] = green * alpha / 255;
     138                    surfacePixel[blueIndex] = blue * alpha / 255;
     139                    surfacePixel[alphaIndex] = alpha;
     140                } else {
     141                    surfacePixel[redIndex] = red;
     142                    surfacePixel[greenIndex] = green;
     143                    surfacePixel[blueIndex] = blue;
     144                    surfacePixel[alphaIndex] = alpha;
     145                }
     146
    90147                bufferData += 4;
    91148                surfacePixel += 4;
    92149            }
    93150        }
    94         surface = adoptRef(cairo_image_surface_create_for_data(surfaceData, cairoFormat, width, height, stride));
     151        surface = adoptRef(cairo_image_surface_create_for_data(surfaceData, CAIRO_FORMAT_ARGB32, width, height, stride));
    95152        static cairo_user_data_key_t s_surfaceDataKey;
    96153        cairo_surface_set_user_data(surface.get(), &s_surfaceDataKey, surfaceData, [](void* data) { fastFree(data); });
    97154    } else
    98         surface = adoptRef(cairo_image_surface_create_for_data(bufferData, cairoFormat, width, height, stride));
     155        surface = adoptRef(cairo_image_surface_create_for_data(bufferData, CAIRO_FORMAT_ARGB32, width, height, stride));
    99156
    100157    ASSERT(cairo_surface_status(surface.get()) == CAIRO_STATUS_SUCCESS);

  • trunk/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp

    r278253 r278276  
    29472947            }
    29482948            frameHolder->updateTexture(layerBuffer->textureGL());
    2949             layerBuffer->setExtraFlags(m_textureMapperFlags | (frameHolder->hasAlphaChannel() ? TextureMapperGL::ShouldBlend : 0));
     2949            layerBuffer->setExtraFlags(m_textureMapperFlags | (frameHolder->hasAlphaChannel() ? TextureMapperGL::ShouldBlend | TextureMapperGL::ShouldPremultiply : 0));
    29502950        }
    29512951        proxy.pushNextBuffer(WTFMove(layerBuffer));
    … …  
    32993299            m_colorConvertInputCaps = caps;
    33003300            m_colorConvertOutputCaps = adoptGRef(gst_caps_copy(caps));
     3301
     3302            // These caps must match the internal format of a cairo surface with CAIRO_FORMAT_ARGB32,
     3303            // so we don't need to perform color conversions when painting the video frame.
    33013304#if G_BYTE_ORDER == G_LITTLE_ENDIAN
    3302             const char* formatString = GST_VIDEO_INFO_HAS_ALPHA(&videoInfo) ? "RGBA" : "BGRx";
     3305            const char* formatString = GST_VIDEO_INFO_HAS_ALPHA(&videoInfo) ? "BGRA" : "BGRx";
    33033306#else
    3304             const char* formatString = GST_VIDEO_INFO_HAS_ALPHA(&videoInfo) ? "RGBA" : "RGBx";
     3307            const char* formatString = GST_VIDEO_INFO_HAS_ALPHA(&videoInfo) ? "ARGB" : "xRGB";
    33053308#endif
    33063309            gst_caps_set_simple(m_colorConvertOutputCaps.get(), "format", G_TYPE_STRING, formatString,
    … …  
    33243327        return;
    33253328
    3326     context.drawImage(gstImage->image(), rect, gstImage->rect(), { CompositeOperator::Copy, m_shouldHandleOrientationTags ? m_videoSourceOrientation : ImageOrientation() });
     3329    context.drawImage(gstImage->image(), rect, gstImage->rect(), { gstImage->hasAlpha() ? CompositeOperator::SourceOver : CompositeOperator::Copy, m_shouldHandleOrientationTags ? m_videoSourceOrientation : ImageOrientation() });
    33273330}
    33283331
    … …  
    33333336
    33343337    if (m_isUsingFallbackVideoSink)
    3335         return false;
    3336 
    3337     if (premultiplyAlpha)
    33383338        return false;
    33393339
    … …  
    33583358    frameHolder->waitForCPUSync();
    33593359
    3360     return m_videoTextureCopier->copyVideoTextureToPlatformTexture(*layerBuffer.get(), size, outputTexture, outputTarget, level, internalFormat, format, type, flipY, m_videoSourceOrientation);
     3360    return m_videoTextureCopier->copyVideoTextureToPlatformTexture(*layerBuffer.get(), size, outputTexture, outputTarget, level, internalFormat, format, type, flipY, m_videoSourceOrientation, premultiplyAlpha);
    33613361}
    33623362

  • trunk/Source/WebCore/platform/graphics/gstreamer/VideoTextureCopierGStreamer.cpp

    r258197 r278276  
    137137}
    138138
    139 bool VideoTextureCopierGStreamer::copyVideoTextureToPlatformTexture(TextureMapperPlatformLayerBuffer& inputTexture, IntSize& frameSize, GLuint outputTexture, GLenum outputTarget, GLint level, GLenum internalFormat, GLenum format, GLenum type, bool flipY, ImageOrientation sourceOrientation)
     139bool VideoTextureCopierGStreamer::copyVideoTextureToPlatformTexture(TextureMapperPlatformLayerBuffer& inputTexture, IntSize& frameSize, GLuint outputTexture, GLenum outputTarget, GLint level, GLenum internalFormat, GLenum format, GLenum type, bool flipY, ImageOrientation sourceOrientation, bool premultiplyAlpha)
    140140{
    141141    if (!m_framebuffer || !m_vbo || frameSize.isEmpty())
    … …  
    162162    TextureMapperShaderProgram::Options options;
    163163    WTF::switchOn(inputTexture.textureVariant(),
    164         [&](const Buffer::RGBTexture&) { options = TextureMapperShaderProgram::TextureRGB; },
     164        [&](const Buffer::RGBTexture&) { options = TextureMapperShaderProgram::TextureRGB | (premultiplyAlpha ? TextureMapperShaderProgram::Premultiply : 0); },
    165165        [&](const Buffer::YUVTexture& texture) {
    166166            switch (texture.numberOfPlanes) {

  • trunk/Source/WebCore/platform/graphics/gstreamer/VideoTextureCopierGStreamer.h

    r249428 r278276  
    4646    ~VideoTextureCopierGStreamer();
    4747
    48     bool copyVideoTextureToPlatformTexture(TextureMapperPlatformLayerBuffer& inputTexture, IntSize& frameSize, GLuint outputTexture, GLenum outputTarget, GLint level, GLenum internalFormat, GLenum format, GLenum type, bool flipY, ImageOrientation sourceOrientation);
     48    bool copyVideoTextureToPlatformTexture(TextureMapperPlatformLayerBuffer& inputTexture, IntSize& frameSize, GLuint outputTexture, GLenum outputTarget, GLint level, GLenum internalFormat, GLenum format, GLenum type, bool flipY, ImageOrientation sourceOrientation, bool premultiplyAlpha);
    4949    void updateColorConversionMatrix(ColorConversion);
    5050    void updateTextureSpaceMatrix();

  • trunk/Source/WebCore/platform/graphics/texmap/TextureMapperGL.cpp

    r268923 r278276  
    505505    }
    506506
     507    if (flags & ShouldPremultiply)
     508        options |= TextureMapperShaderProgram::Premultiply;
     509
    507510    Ref<TextureMapperShaderProgram> program = data().getShaderProgram(options);
    508511

  • trunk/Source/WebCore/platform/graphics/texmap/TextureMapperGL.h

    r268923 r278276  
    5858        ShouldConvertTextureARGBToRGBA = 0x100,
    5959        ShouldNotBlend = 0x200,
    60         ShouldUseExternalOESTextureRect = 0x400
     60        ShouldUseExternalOESTextureRect = 0x400,
     61        ShouldPremultiply = 0x800
    6162    };
    6263

  • trunk/Source/WebCore/platform/graphics/texmap/TextureMapperShaderProgram.cpp

    r272361 r278276  
    265265        void applyTextureRGB(inout vec4 color, vec2 texCoord) { color = u_textureColorSpaceMatrix * SamplerFunction(s_sampler, texCoord); }
    266266
     267        void applyPremultiply(inout vec4 color) { color = vec4(color.rgb * color.a, color.a); }
     268
    267269        vec3 yuvToRgb(float y, float u, float v)
    268270        {
    … …  
    484486            applyManualRepeatIfNeeded(texCoord);
    485487            applyTextureRGBIfNeeded(color, texCoord);
     488            applyPremultiplyIfNeeded(color);
    486489            applyTextureYUVIfNeeded(color, texCoord);
    487490            applyTextureNV12IfNeeded(color, texCoord);
    … …  
    538541    SET_APPLIER_FROM_OPTIONS(TextureExternalOES);
    539542    SET_APPLIER_FROM_OPTIONS(RoundedRectClip);
     543    SET_APPLIER_FROM_OPTIONS(Premultiply);
    540544
    541545    StringBuilder vertexShaderBuilder;

  • trunk/Source/WebCore/platform/graphics/texmap/TextureMapperShaderProgram.h

    r268923 r278276  
    104104        TextureExternalOES = 1L << 22,
    105105        RoundedRectClip  = 1L << 23,
     106        Premultiply      = 1L << 24,
    106107    };
    107108
Note: See TracChangeset for help on using the changeset viewer.