Changeset 264596 in webkit


Ignore:
Timestamp:
Jul 20, 2020 5:37:46 AM (4 years ago)
Author:
commit-queue@webkit.org
Message:

[GStreamer] Add support for canvas captureStream, tests failing since they were added in r213598
https://bugs.webkit.org/show_bug.cgi?id=169811

Patch by Philippe Normand <pnormand@igalia.com> on 2020-07-20
Reviewed by Darin Adler.

Source/WebCore:

Hook captureStream support in GStreamer ports. The only missing bits were:

  1. create a MediaSample from a BGRA buffer
  2. convert a MediaSample to a RGBA buffer (used only for layout tests)
  3. set video width and height tags not only on video capture tracks (tracks created from captureStream are not GStreamer capture device tracks)
  • html/HTMLCanvasElement.cpp:

(WebCore::HTMLCanvasElement::toMediaSample):

  • platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h: Refactor createImageSample to

use unsigned for dimensions.

  • platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm:
  • platform/graphics/gstreamer/MediaSampleGStreamer.cpp:

(WebCore::MediaSampleGStreamer::createImageSample):
(WebCore::MediaSampleGStreamer::getRGBAImageData const):

  • platform/graphics/gstreamer/MediaSampleGStreamer.h:
  • platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp:

(mediaStreamTrackPrivateGetTags):

LayoutTests:

Unflag passing tests and rebaseline one platform-specific WPT test expectation which differs
from the default by only one line.

  • platform/glib/imported/w3c/web-platform-tests/mediacapture-streams/MediaStream-MediaElement-srcObject.https-expected.txt: Added.
  • platform/gtk/TestExpectations:
  • platform/gtk/imported/w3c/web-platform-tests/mediacapture-streams/MediaStream-MediaElement-srcObject.https-expected.txt: Removed.
  • platform/wpe/TestExpectations:
Location:
trunk
Files:
2 added
1 deleted
10 edited

Legend:

Unmodified
Added
Removed
  • trunk/LayoutTests/ChangeLog

    r264595 r264596  
     12020-07-20  Philippe Normand  <pnormand@igalia.com>
     2
     3        [GStreamer] Add support for canvas captureStream, tests failing since they were added in r213598
     4        https://bugs.webkit.org/show_bug.cgi?id=169811
     5
     6        Reviewed by Darin Adler.
     7
     8        Unflag passing tests and rebaseline one platform-specific WPT test expectation which differs
     9        from the default by only one line.
     10
     11        * platform/glib/imported/w3c/web-platform-tests/mediacapture-streams/MediaStream-MediaElement-srcObject.https-expected.txt: Added.
     12        * platform/gtk/TestExpectations:
     13        * platform/gtk/imported/w3c/web-platform-tests/mediacapture-streams/MediaStream-MediaElement-srcObject.https-expected.txt: Removed.
     14        * platform/wpe/TestExpectations:
     15
    1162020-07-20  Philippe Normand  <pnormand@igalia.com>
    217
  • trunk/LayoutTests/platform/gtk/TestExpectations

    r264594 r264596  
    571571fast/mediastream/MediaDevices-addEventListener.html [ DumpJSConsoleLogInStdErr ]
    572572
    573 # Canvas captureStream support is not implemented
    574 webkit.org/b/169811 fast/mediacapturefromelement [ Skip ]
    575 webkit.org/b/169811 fast/mediastream/captureStream/ [ Failure ]
    576 webkit.org/b/169811 fast/mediastream/captureStream/canvas2d.html [ Crash Timeout ]
    577 webkit.org/b/169811 fast/mediastream/captureStream/canvas3d.html [ Crash Timeout ]
    578 
    579573# No support for GPUProcess yet
    580574fast/mediastream/captureInGPUProcess.html [ Skip ]
     
    33043298webkit.org/b/186601 fullscreen/full-screen-layer-dump.html [ Failure ]
    33053299
    3306 webkit.org/b/186845 fast/mediastream/captureStream/canvas2d-heavy-drawing.html [ Failure Timeout ]
    3307 
    33083300webkit.org/b/186144 transitions/move-after-transition.html [ Failure Timeout ]
    33093301
  • trunk/LayoutTests/platform/wpe/TestExpectations

    r264544 r264596  
    131131# Crash is bug #176801
    132132webkit.org/b/160996 fast/mediastream/MediaStream-video-element-video-tracks-disabled.html [ Timeout ImageOnlyFailure ]
    133 
    134 # Canvas captureStream support is not implemented
    135 webkit.org/b/169811 fast/mediacapturefromelement [ Skip ]
    136 webkit.org/b/169811 fast/mediastream/captureStream/ [ Failure ]
    137 webkit.org/b/169811 fast/mediastream/captureStream/canvas2d.html [ Timeout ]
    138 webkit.org/b/169811 fast/mediastream/captureStream/canvas3d.html [ Timeout ]
    139133
    140134# MediaRecorder is not currently implemented
  • trunk/Source/WebCore/ChangeLog

    r264595 r264596  
     12020-07-20  Philippe Normand  <pnormand@igalia.com>
     2
     3        [GStreamer] Add support for canvas captureStream, tests failing since they were added in r213598
     4        https://bugs.webkit.org/show_bug.cgi?id=169811
     5
     6        Reviewed by Darin Adler.
     7
     8        Hook captureStream support in GStreamer ports. The only missing bits were:
     9
     10        1. create a MediaSample from a BGRA buffer
     11        2. convert a MediaSample to a RGBA buffer (used only for layout tests)
     12        3. set video width and height tags not only on video capture tracks (tracks created from
     13           captureStream are not GStreamer capture device tracks)
     14
     15        * html/HTMLCanvasElement.cpp:
     16        (WebCore::HTMLCanvasElement::toMediaSample):
     17        * platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h: Refactor createImageSample to
     18        use unsigned for dimensions.
     19        * platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm:
     20        * platform/graphics/gstreamer/MediaSampleGStreamer.cpp:
     21        (WebCore::MediaSampleGStreamer::createImageSample):
     22        (WebCore::MediaSampleGStreamer::getRGBAImageData const):
     23        * platform/graphics/gstreamer/MediaSampleGStreamer.h:
     24        * platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp:
     25        (mediaStreamTrackPrivateGetTags):
     26
    1272020-07-20  Philippe Normand  <pnormand@igalia.com>
    228
  • trunk/Source/WebCore/html/HTMLCanvasElement.cpp

    r264334 r264596  
    8888#endif
    8989
     90#if USE(CG)
     91#include "ImageBufferUtilitiesCG.h"
     92#endif
     93
     94#if USE(GSTREAMER)
     95#include "MediaSampleGStreamer.h"
     96#endif
     97
    9098#if PLATFORM(COCOA)
    9199#include "MediaSampleAVFObjC.h"
    92100#include <pal/cf/CoreMediaSoftLink.h>
    93 #endif
    94 
    95 #if USE(CG)
    96 #include "ImageBufferUtilitiesCG.h"
    97101#endif
    98102
     
    790794    makeRenderingResultsAvailable();
    791795    return MediaSampleAVFObjC::createImageSample(imageBuffer->toBGRAData(), width(), height());
     796#elif USE(GSTREAMER)
     797    makeRenderingResultsAvailable();
     798    return MediaSampleGStreamer::createImageSample(imageBuffer->toBGRAData(), width(), height());
    792799#else
    793800    return nullptr;
  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h

    r262189 r264596  
    3838    static Ref<MediaSampleAVFObjC> create(CMSampleBufferRef sample, AtomString trackID) { return adoptRef(*new MediaSampleAVFObjC(sample, trackID)); }
    3939    static Ref<MediaSampleAVFObjC> create(CMSampleBufferRef sample, VideoRotation rotation = VideoRotation::None, bool mirrored = false) { return adoptRef(*new MediaSampleAVFObjC(sample, rotation, mirrored)); }
    40     static RefPtr<MediaSampleAVFObjC> createImageSample(Vector<uint8_t>&&, unsigned long width, unsigned long height);
     40    static RefPtr<MediaSampleAVFObjC> createImageSample(Vector<uint8_t>&&, unsigned width, unsigned height);
    4141
    4242    WEBCORE_EXPORT static void setAsDisplayImmediately(MediaSample&);
  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm

    r262189 r264596  
    4747}
    4848
    49 RefPtr<MediaSampleAVFObjC> MediaSampleAVFObjC::createImageSample(Vector<uint8_t>&& array, unsigned long width, unsigned long height)
     49RefPtr<MediaSampleAVFObjC> MediaSampleAVFObjC::createImageSample(Vector<uint8_t>&& array, unsigned width, unsigned height)
    5050{
    5151    CVPixelBufferRef pixelBuffer = nullptr;
  • trunk/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.cpp

    r258844 r264596  
    2424#include "GStreamerCommon.h"
    2525
     26#include <JavaScriptCore/JSCInlines.h>
     27#include <JavaScriptCore/TypedArrayInlines.h>
    2628#include <algorithm>
    2729
     
    9698}
    9799
     100Ref<MediaSampleGStreamer> MediaSampleGStreamer::createImageSample(Vector<uint8_t>&& bgraData, unsigned width, unsigned height)
     101{
     102    size_t size = bgraData.sizeInBytes();
     103    auto buffer = adoptGRef(gst_buffer_new_wrapped_full(GST_MEMORY_FLAG_READONLY, bgraData.data(), size, 0, size, bgraData.releaseBuffer().leakPtr(), [](gpointer data) {
     104        WTF::VectorMalloc::free(data);
     105    }));
     106    gst_buffer_add_video_meta(buffer.get(), GST_VIDEO_FRAME_FLAG_NONE, GST_VIDEO_FORMAT_BGRA, width, height);
     107    auto caps = adoptGRef(gst_caps_new_simple("video/x-raw", "format", G_TYPE_STRING, "BGRA", "width", G_TYPE_INT, width, "height", G_TYPE_INT, height, "framerate", GST_TYPE_FRACTION, 1, 1, nullptr));
     108    auto sample = adoptGRef(gst_sample_new(buffer.get(), caps.get(), nullptr, nullptr));
     109    return create(WTFMove(sample), FloatSize(width, height), { });
     110}
     111
     112RefPtr<JSC::Uint8ClampedArray> MediaSampleGStreamer::getRGBAImageData() const
     113{
     114    auto* caps = gst_sample_get_caps(m_sample.get());
     115    GstVideoInfo inputInfo;
     116    if (!gst_video_info_from_caps(&inputInfo, caps))
     117        return nullptr;
     118
     119    // We could check the input format is RGBA before attempting a conversion, but it is very
     120    // unlikely to pay off. The input format is likely to be BGRA (when the samples are created as a
     121    // result of mediastream captureStream) or some YUV format if the sample is from a video capture
     122    // device. This method is called only by internals during layout tests, it is thus not critical
     123    // to optimize this code path.
     124
     125    auto outputCaps = adoptGRef(gst_caps_copy(caps));
     126    gst_caps_set_simple(outputCaps.get(), "format", G_TYPE_STRING, "RGBA", nullptr);
     127
     128    GstVideoInfo outputInfo;
     129    if (!gst_video_info_from_caps(&outputInfo, outputCaps.get()))
     130        return nullptr;
     131
     132    int width = GST_VIDEO_INFO_WIDTH(&inputInfo);
     133    int height = GST_VIDEO_INFO_HEIGHT(&inputInfo);
     134    unsigned byteLength = GST_VIDEO_INFO_SIZE(&inputInfo);
     135    auto bufferStorage = JSC::ArrayBuffer::create(width * height, 4);
     136    auto outputBuffer = adoptGRef(gst_buffer_new_wrapped_full(GST_MEMORY_FLAG_NO_SHARE, bufferStorage->data(), byteLength, 0, byteLength, nullptr, [](gpointer) { }));
     137    gst_buffer_add_video_meta(outputBuffer.get(), GST_VIDEO_FRAME_FLAG_NONE, GST_VIDEO_FORMAT_RGBA, width, height);
     138    GstMappedFrame outputFrame(outputBuffer.get(), outputInfo, GST_MAP_WRITE);
     139
     140    GUniquePtr<GstVideoConverter> converter(gst_video_converter_new(&inputInfo, &outputInfo, nullptr));
     141    GstMappedFrame inputFrame(gst_sample_get_buffer(m_sample.get()), inputInfo, GST_MAP_READ);
     142    gst_video_converter_frame(converter.get(), inputFrame.get(), outputFrame.get());
     143    return JSC::Uint8ClampedArray::tryCreate(WTFMove(bufferStorage), 0, byteLength);
     144}
     145
    98146void MediaSampleGStreamer::extendToTheBeginning()
    99147{
  • trunk/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.h

    r261922 r264596  
    3838
    3939    static Ref<MediaSampleGStreamer> createFakeSample(GstCaps*, MediaTime pts, MediaTime dts, MediaTime duration, const FloatSize& presentationSize, const AtomString& trackId);
     40    static Ref<MediaSampleGStreamer> createImageSample(Vector<uint8_t>&&, unsigned width, unsigned height);
    4041
    4142    void extendToTheBeginning();
     
    5556    PlatformSample platformSample() override;
    5657    void dump(PrintStream&) const override;
     58    RefPtr<JSC::Uint8ClampedArray> getRGBAImageData() const final;
    5759
    5860protected:
  • trunk/Source/WebCore/platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp

    r264062 r264596  
    3030#include "GStreamerAudioData.h"
    3131#include "GStreamerCommon.h"
    32 #include "GStreamerVideoCaptureSource.h"
    3332#include "MediaSampleGStreamer.h"
    3433#include "MediaStreamPrivate.h"
     
    6463        gst_tag_list_add(tagList.get(), GST_TAG_MERGE_APPEND, WEBKIT_MEDIA_TRACK_TAG_KIND, static_cast<int>(VideoTrackPrivate::Kind::Main), nullptr);
    6564
    66         if (track->isCaptureTrack()) {
    67             GStreamerVideoCaptureSource& source = static_cast<GStreamerVideoCaptureSource&>(track->source());
    68             gst_tag_list_add(tagList.get(), GST_TAG_MERGE_APPEND, WEBKIT_MEDIA_TRACK_TAG_WIDTH, source.size().width(),
    69                 WEBKIT_MEDIA_TRACK_TAG_HEIGHT, source.size().height(), nullptr);
    70         }
     65        auto& settings = track->settings();
     66        gst_tag_list_add(tagList.get(), GST_TAG_MERGE_APPEND, WEBKIT_MEDIA_TRACK_TAG_WIDTH, settings.width(),
     67            WEBKIT_MEDIA_TRACK_TAG_HEIGHT, settings.height(), nullptr);
    7168    }
    7269
Note: See TracChangeset for help on using the changeset viewer.