⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 264607 in webkit


Ignore:
Timestamp:
Jul 20, 2020, 11:30:48 AM (6 years ago)
Author:
commit-queue@webkit.org
Message:

[GStreamer] Add support for canvas captureStream, tests failing since they were added in r213598
​https://bugs.webkit.org/show_bug.cgi?id=169811

Patch by Philippe Normand <​pnormand@igalia.com> on 2020-07-20
Reviewed by Darin Adler.

Source/WebCore:

Hook captureStream support in GStreamer ports. The only missing bits were:

  1. create a MediaSample from a BGRA buffer
  2. convert a MediaSample to a RGBA buffer (used only for layout tests)
  3. set video width and height tags not only on video capture tracks (tracks created from captureStream are not GStreamer capture device tracks)
  • html/HTMLCanvasElement.cpp:

(WebCore::HTMLCanvasElement::toMediaSample):

  • platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h: Refactor createImageSample to

use unsigned for dimensions.

  • platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm:
  • platform/graphics/gstreamer/MediaSampleGStreamer.cpp:

(WebCore::MediaSampleGStreamer::createImageSample):
(WebCore::MediaSampleGStreamer::getRGBAImageData const):

  • platform/graphics/gstreamer/MediaSampleGStreamer.h:
  • platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp:

(mediaStreamTrackPrivateGetTags):

LayoutTests:

Unflag passing tests and rebaseline one platform-specific WPT test expectation which differs
from the default by only one line.

  • platform/glib/imported/w3c/web-platform-tests/mediacapture-streams/MediaStream-MediaElement-srcObject.https-expected.txt: Added.
  • platform/gtk/TestExpectations:
  • platform/gtk/imported/w3c/web-platform-tests/mediacapture-streams/MediaStream-MediaElement-srcObject.https-expected.txt: Removed.
  • platform/wpe/TestExpectations:
Location:
trunk
Files:
2 added
1 deleted
10 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r264606 r264607  
     12020-07-20  Philippe Normand  <pnormand@igalia.com>
     2
     3        [GStreamer] Add support for canvas captureStream, tests failing since they were added in r213598
     4        https://bugs.webkit.org/show_bug.cgi?id=169811
     5
     6        Reviewed by Darin Adler.
     7
     8        Unflag passing tests and rebaseline one platform-specific WPT test expectation which differs
     9        from the default by only one line.
     10
     11        * platform/glib/imported/w3c/web-platform-tests/mediacapture-streams/MediaStream-MediaElement-srcObject.https-expected.txt: Added.
     12        * platform/gtk/TestExpectations:
     13        * platform/gtk/imported/w3c/web-platform-tests/mediacapture-streams/MediaStream-MediaElement-srcObject.https-expected.txt: Removed.
     14        * platform/wpe/TestExpectations:
     15
    1162020-07-20  Youenn Fablet  <youenn@apple.com>
    217

  • trunk/LayoutTests/platform/gtk/TestExpectations

    r264601 r264607  
    571571fast/mediastream/MediaDevices-addEventListener.html [ DumpJSConsoleLogInStdErr ]
    572572
    573 # Canvas captureStream support is not implemented
    574 webkit.org/b/169811 fast/mediacapturefromelement [ Skip ]
    575 webkit.org/b/169811 fast/mediastream/captureStream/ [ Failure ]
    576 webkit.org/b/169811 fast/mediastream/captureStream/canvas2d.html [ Crash Timeout ]
    577 webkit.org/b/169811 fast/mediastream/captureStream/canvas3d.html [ Crash Timeout ]
    578 
    579573# No support for GPUProcess yet
    580574fast/mediastream/captureInGPUProcess.html [ Skip ]
    … …  
    33043298webkit.org/b/186601 fullscreen/full-screen-layer-dump.html [ Failure ]
    33053299
    3306 webkit.org/b/186845 fast/mediastream/captureStream/canvas2d-heavy-drawing.html [ Failure Timeout ]
    3307 
    33083300webkit.org/b/186144 transitions/move-after-transition.html [ Failure Timeout ]
    33093301

  • trunk/LayoutTests/platform/wpe/TestExpectations

    r264601 r264607  
    131131# Crash is bug #176801
    132132webkit.org/b/160996 fast/mediastream/MediaStream-video-element-video-tracks-disabled.html [ Timeout ImageOnlyFailure ]
    133 
    134 # Canvas captureStream support is not implemented
    135 webkit.org/b/169811 fast/mediacapturefromelement [ Skip ]
    136 webkit.org/b/169811 fast/mediastream/captureStream/ [ Failure ]
    137 webkit.org/b/169811 fast/mediastream/captureStream/canvas2d.html [ Timeout ]
    138 webkit.org/b/169811 fast/mediastream/captureStream/canvas3d.html [ Timeout ]
    139133
    140134# MediaRecorder is not currently implemented

  • trunk/Source/WebCore/ChangeLog

    r264604 r264607  
     12020-07-20  Philippe Normand  <pnormand@igalia.com>
     2
     3        [GStreamer] Add support for canvas captureStream, tests failing since they were added in r213598
     4        https://bugs.webkit.org/show_bug.cgi?id=169811
     5
     6        Reviewed by Darin Adler.
     7
     8        Hook captureStream support in GStreamer ports. The only missing bits were:
     9
     10        1. create a MediaSample from a BGRA buffer
     11        2. convert a MediaSample to a RGBA buffer (used only for layout tests)
     12        3. set video width and height tags not only on video capture tracks (tracks created from
     13           captureStream are not GStreamer capture device tracks)
     14
     15        * html/HTMLCanvasElement.cpp:
     16        (WebCore::HTMLCanvasElement::toMediaSample):
     17        * platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h: Refactor createImageSample to
     18        use unsigned for dimensions.
     19        * platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm:
     20        * platform/graphics/gstreamer/MediaSampleGStreamer.cpp:
     21        (WebCore::MediaSampleGStreamer::createImageSample):
     22        (WebCore::MediaSampleGStreamer::getRGBAImageData const):
     23        * platform/graphics/gstreamer/MediaSampleGStreamer.h:
     24        * platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp:
     25        (mediaStreamTrackPrivateGetTags):
     26
    1272020-07-20  Youenn Fablet  <youenn@apple.com>
    228

  • trunk/Source/WebCore/html/HTMLCanvasElement.cpp

    r264601 r264607  
    8888#endif
    8989
     90#if USE(CG)
     91#include "ImageBufferUtilitiesCG.h"
     92#endif
     93
     94#if USE(GSTREAMER)
     95#include "MediaSampleGStreamer.h"
     96#endif
     97
    9098#if PLATFORM(COCOA)
    9199#include "MediaSampleAVFObjC.h"
    92100#include <pal/cf/CoreMediaSoftLink.h>
    93 #endif
    94 
    95 #if USE(CG)
    96 #include "ImageBufferUtilitiesCG.h"
    97101#endif
    98102
    … …  
    790794    makeRenderingResultsAvailable();
    791795    return MediaSampleAVFObjC::createImageSample(imageBuffer->toBGRAData(), width(), height());
     796#elif USE(GSTREAMER)
     797    makeRenderingResultsAvailable();
     798    return MediaSampleGStreamer::createImageSample(imageBuffer->toBGRAData(), width(), height());
    792799#else
    793800    return nullptr;

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h

    r264601 r264607  
    3838    static Ref<MediaSampleAVFObjC> create(CMSampleBufferRef sample, AtomString trackID) { return adoptRef(*new MediaSampleAVFObjC(sample, trackID)); }
    3939    static Ref<MediaSampleAVFObjC> create(CMSampleBufferRef sample, VideoRotation rotation = VideoRotation::None, bool mirrored = false) { return adoptRef(*new MediaSampleAVFObjC(sample, rotation, mirrored)); }
    40     static RefPtr<MediaSampleAVFObjC> createImageSample(Vector<uint8_t>&&, unsigned long width, unsigned long height);
     40    static RefPtr<MediaSampleAVFObjC> createImageSample(Vector<uint8_t>&&, unsigned width, unsigned height);
    4141
    4242    WEBCORE_EXPORT static void setAsDisplayImmediately(MediaSample&);

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm

    r264601 r264607  
    4747}
    4848
    49 RefPtr<MediaSampleAVFObjC> MediaSampleAVFObjC::createImageSample(Vector<uint8_t>&& array, unsigned long width, unsigned long height)
     49RefPtr<MediaSampleAVFObjC> MediaSampleAVFObjC::createImageSample(Vector<uint8_t>&& array, unsigned width, unsigned height)
    5050{
    5151    CVPixelBufferRef pixelBuffer = nullptr;

  • trunk/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.cpp

    r264601 r264607  
    2424#include "GStreamerCommon.h"
    2525
     26#include <JavaScriptCore/JSCInlines.h>
     27#include <JavaScriptCore/TypedArrayInlines.h>
    2628#include <algorithm>
    2729
    … …  
    9698}
    9799
     100Ref<MediaSampleGStreamer> MediaSampleGStreamer::createImageSample(Vector<uint8_t>&& bgraData, unsigned width, unsigned height)
     101{
     102    size_t size = bgraData.sizeInBytes();
     103    auto* data = bgraData.releaseBuffer().leakPtr();
     104    auto buffer = adoptGRef(gst_buffer_new_wrapped_full(GST_MEMORY_FLAG_READONLY, data, size, 0, size, data, [](gpointer data) {
     105        WTF::VectorMalloc::free(data);
     106    }));
     107    gst_buffer_add_video_meta(buffer.get(), GST_VIDEO_FRAME_FLAG_NONE, GST_VIDEO_FORMAT_BGRA, width, height);
     108    auto caps = adoptGRef(gst_caps_new_simple("video/x-raw", "format", G_TYPE_STRING, "BGRA", "width", G_TYPE_INT, width, "height", G_TYPE_INT, height, "framerate", GST_TYPE_FRACTION, 1, 1, nullptr));
     109    auto sample = adoptGRef(gst_sample_new(buffer.get(), caps.get(), nullptr, nullptr));
     110    return create(WTFMove(sample), FloatSize(width, height), { });
     111}
     112
     113RefPtr<JSC::Uint8ClampedArray> MediaSampleGStreamer::getRGBAImageData() const
     114{
     115    auto* caps = gst_sample_get_caps(m_sample.get());
     116    GstVideoInfo inputInfo;
     117    if (!gst_video_info_from_caps(&inputInfo, caps))
     118        return nullptr;
     119
     120    // We could check the input format is RGBA before attempting a conversion, but it is very
     121    // unlikely to pay off. The input format is likely to be BGRA (when the samples are created as a
     122    // result of mediastream captureStream) or some YUV format if the sample is from a video capture
     123    // device. This method is called only by internals during layout tests, it is thus not critical
     124    // to optimize this code path.
     125
     126    auto outputCaps = adoptGRef(gst_caps_copy(caps));
     127    gst_caps_set_simple(outputCaps.get(), "format", G_TYPE_STRING, "RGBA", nullptr);
     128
     129    GstVideoInfo outputInfo;
     130    if (!gst_video_info_from_caps(&outputInfo, outputCaps.get()))
     131        return nullptr;
     132
     133    int width = GST_VIDEO_INFO_WIDTH(&inputInfo);
     134    int height = GST_VIDEO_INFO_HEIGHT(&inputInfo);
     135    unsigned byteLength = GST_VIDEO_INFO_SIZE(&inputInfo);
     136    auto bufferStorage = JSC::ArrayBuffer::create(width * height, 4);
     137    auto outputBuffer = adoptGRef(gst_buffer_new_wrapped_full(GST_MEMORY_FLAG_NO_SHARE, bufferStorage->data(), byteLength, 0, byteLength, nullptr, [](gpointer) { }));
     138    gst_buffer_add_video_meta(outputBuffer.get(), GST_VIDEO_FRAME_FLAG_NONE, GST_VIDEO_FORMAT_RGBA, width, height);
     139    GstMappedFrame outputFrame(outputBuffer.get(), outputInfo, GST_MAP_WRITE);
     140
     141    GUniquePtr<GstVideoConverter> converter(gst_video_converter_new(&inputInfo, &outputInfo, nullptr));
     142    GstMappedFrame inputFrame(gst_sample_get_buffer(m_sample.get()), inputInfo, GST_MAP_READ);
     143    gst_video_converter_frame(converter.get(), inputFrame.get(), outputFrame.get());
     144    return JSC::Uint8ClampedArray::tryCreate(WTFMove(bufferStorage), 0, byteLength);
     145}
     146
    98147void MediaSampleGStreamer::extendToTheBeginning()
    99148{

  • trunk/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.h

    r264601 r264607  
    3838
    3939    static Ref<MediaSampleGStreamer> createFakeSample(GstCaps*, MediaTime pts, MediaTime dts, MediaTime duration, const FloatSize& presentationSize, const AtomString& trackId);
     40    static Ref<MediaSampleGStreamer> createImageSample(Vector<uint8_t>&&, unsigned width, unsigned height);
    4041
    4142    void extendToTheBeginning();
    … …  
    5556    PlatformSample platformSample() override;
    5657    void dump(PrintStream&) const override;
     58    RefPtr<JSC::Uint8ClampedArray> getRGBAImageData() const final;
    5759
    5860protected:

  • trunk/Source/WebCore/platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp

    r264601 r264607  
    3030#include "GStreamerAudioData.h"
    3131#include "GStreamerCommon.h"
    32 #include "GStreamerVideoCaptureSource.h"
    3332#include "MediaSampleGStreamer.h"
    3433#include "MediaStreamPrivate.h"
    … …  
    6463        gst_tag_list_add(tagList.get(), GST_TAG_MERGE_APPEND, WEBKIT_MEDIA_TRACK_TAG_KIND, static_cast<int>(VideoTrackPrivate::Kind::Main), nullptr);
    6564
    66         if (track->isCaptureTrack()) {
    67             GStreamerVideoCaptureSource& source = static_cast<GStreamerVideoCaptureSource&>(track->source());
    68             gst_tag_list_add(tagList.get(), GST_TAG_MERGE_APPEND, WEBKIT_MEDIA_TRACK_TAG_WIDTH, source.size().width(),
    69                 WEBKIT_MEDIA_TRACK_TAG_HEIGHT, source.size().height(), nullptr);
    70         }
     65        auto& settings = track->settings();
     66        gst_tag_list_add(tagList.get(), GST_TAG_MERGE_APPEND, WEBKIT_MEDIA_TRACK_TAG_WIDTH, settings.width(),
     67            WEBKIT_MEDIA_TRACK_TAG_HEIGHT, settings.height(), nullptr);
    7168    }
    7269
Note: See TracChangeset for help on using the changeset viewer.