Changeset 238557 in webkit


Ignore:
Timestamp:
Nov 27, 2018 9:34:28 AM (5 years ago)
Author:
commit-queue@webkit.org
Message:

[GStreamer][WebRTC] Use LibWebRTC provided vp8 decoders and encoders
​https://bugs.webkit.org/show_bug.cgi?id=191861

Source/ThirdParty/libwebrtc:

Patch by Thibault Saunier <​tsaunier@igalia.com> on 2018-11-27
Reviewed by Philippe Normand.

  • CMakeLists.txt: Build LibVPX vp8 encoder and decoders.

Source/WebCore:

The GStreamer implementations are less feature full and less tested, now that Apple
also use the LibWebRTC provided implementations it makes a lot of sense for us to
do the same.

Basically everything related to temporal scalability is not implemented in GStreamer.

We should make sure to use GStreamer elements on low powered platforms and for
accelerated encoders and decoders.

Patch by Thibault Saunier <​tsaunier@igalia.com> on 2018-11-27
Reviewed by Philippe Normand.

This is mostly refactoring, no new test required.

  • platform/graphics/gstreamer/GStreamerCommon.h: Added GstMappedFrame similar to GstMappedBuffer but for video frames.

(WebCore::GstMappedFrame::GstMappedFrame):
(WebCore::GstMappedFrame::get):
(WebCore::GstMappedFrame::ComponentData):
(WebCore::GstMappedFrame::ComponentStride):
(WebCore::GstMappedFrame::info):
(WebCore::GstMappedFrame::width):
(WebCore::GstMappedFrame::height):
(WebCore::GstMappedFrame::format):
(WebCore::GstMappedFrame::~GstMappedFrame):
(WebCore::GstMappedFrame::operator bool const):

  • platform/graphics/gstreamer/GUniquePtrGStreamer.h:
  • platform/mediastream/gstreamer/GStreamerVideoFrameLibWebRTC.cpp:

(WebCore::GStreamerVideoFrameLibWebRTC::ToI420): Implemented support for converting frame formats with the GstVideoConverter API

  • platform/mediastream/libwebrtc/GStreamerVideoDecoderFactory.cpp:

(WebCore::GStreamerVideoDecoder::GstDecoderFactory):
(WebCore::GStreamerVideoDecoder::HasGstDecoder):
(WebCore::VP8Decoder::Create): Creates a webrtc::LibvpxVp8Decoder() if GStreamer decoder would be the LibVPX based one.
(WebCore::GStreamerVideoDecoderFactory::CreateVideoDecoder):

  • platform/mediastream/libwebrtc/GStreamerVideoEncoder.cpp:

(gst_webrtc_video_encoder_class_init):

  • platform/mediastream/libwebrtc/GStreamerVideoEncoderFactory.cpp: Stop using vp8enc and use LibWebRTC based implementation

(WebCore::GStreamerH264Encoder::GStreamerH264Encoder): Renamed H264Encoder to GStreamerH264Encoder to be more coherent with what is done in LibVPX
(WebCore::GStreamerVP8Encoder::GStreamerVP8Encoder): Renamed VP8Encoder to GStreamerVP8Encoder to be more coherent with what is done in LibVPX
(WebCore::GStreamerVideoEncoderFactory::CreateVideoEncoder):
(WebCore::GStreamerVideoEncoderFactory::GetSupportedFormats const):

Location:
trunk/Source
Files:
9 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/ThirdParty/libwebrtc/CMakeLists.txt

    r237174 r238557  
    10511051    Source/webrtc/modules/video_coding/codecs/vp8/default_temporal_layers.cc
    10521052    Source/webrtc/modules/video_coding/codecs/vp8/screenshare_layers.cc
     1053    Source/webrtc/modules/video_coding/codecs/vp8/vp8_temporal_layers.cc
     1054    Source/webrtc/modules/video_coding/codecs/vp8/libvpx_vp8_decoder.cc
     1055    Source/webrtc/modules/video_coding/codecs/vp8/libvpx_interface.cc
     1056    Source/webrtc/modules/video_coding/codecs/vp8/temporal_layers_checker.cc
     1057    Source/webrtc/modules/video_coding/codecs/vp8/libvpx_vp8_encoder.cc
    10531058    Source/webrtc/modules/video_coding/codecs/vp9/svc_config.cc
    10541059    Source/webrtc/modules/video_coding/codecs/vp9/svc_rate_allocator.cc

  • trunk/Source/ThirdParty/libwebrtc/ChangeLog

    r238182 r238557  
     12018-11-27  Thibault Saunier  <tsaunier@igalia.com>
     2
     3        [GStreamer][WebRTC] Use LibWebRTC provided vp8 decoders and encoders
     4        https://bugs.webkit.org/show_bug.cgi?id=191861
     5
     6        Reviewed by Philippe Normand.
     7
     8        * CMakeLists.txt: Build LibVPX vp8 encoder and decoders.
     9
    1102018-11-14  Youenn Fablet  <youenn@apple.com>
    211

  • trunk/Source/WebCore/ChangeLog

    r238551 r238557  
     12018-11-27  Thibault Saunier  <tsaunier@igalia.com>
     2
     3        [GStreamer][WebRTC] Use LibWebRTC provided vp8 decoders and encoders
     4        https://bugs.webkit.org/show_bug.cgi?id=191861
     5
     6        The GStreamer implementations are less feature full and less tested, now that Apple
     7        also use the LibWebRTC provided implementations it makes a lot of sense for us to
     8        do the same.
     9
     10        Basically everything related to temporal scalability is not implemented in GStreamer.
     11
     12        We should make sure to use GStreamer elements on low powered platforms and for
     13        accelerated encoders and decoders.
     14
     15        Reviewed by Philippe Normand.
     16
     17        This is mostly refactoring, no new test required.
     18
     19        * platform/graphics/gstreamer/GStreamerCommon.h: Added GstMappedFrame similar to GstMappedBuffer but for video frames.
     20        (WebCore::GstMappedFrame::GstMappedFrame):
     21        (WebCore::GstMappedFrame::get):
     22        (WebCore::GstMappedFrame::ComponentData):
     23        (WebCore::GstMappedFrame::ComponentStride):
     24        (WebCore::GstMappedFrame::info):
     25        (WebCore::GstMappedFrame::width):
     26        (WebCore::GstMappedFrame::height):
     27        (WebCore::GstMappedFrame::format):
     28        (WebCore::GstMappedFrame::~GstMappedFrame):
     29        (WebCore::GstMappedFrame::operator bool const):
     30        * platform/graphics/gstreamer/GUniquePtrGStreamer.h:
     31        * platform/mediastream/gstreamer/GStreamerVideoFrameLibWebRTC.cpp:
     32        (WebCore::GStreamerVideoFrameLibWebRTC::ToI420): Implemented support for converting frame formats with the GstVideoConverter API
     33        * platform/mediastream/libwebrtc/GStreamerVideoDecoderFactory.cpp:
     34        (WebCore::GStreamerVideoDecoder::GstDecoderFactory):
     35        (WebCore::GStreamerVideoDecoder::HasGstDecoder):
     36        (WebCore::VP8Decoder::Create): Creates a `webrtc::LibvpxVp8Decoder()` if GStreamer decoder would be the LibVPX based one.
     37        (WebCore::GStreamerVideoDecoderFactory::CreateVideoDecoder):
     38        * platform/mediastream/libwebrtc/GStreamerVideoEncoder.cpp:
     39        (gst_webrtc_video_encoder_class_init):
     40        * platform/mediastream/libwebrtc/GStreamerVideoEncoderFactory.cpp: Stop using vp8enc and use LibWebRTC based implementation
     41        (WebCore::GStreamerH264Encoder::GStreamerH264Encoder): Renamed H264Encoder to GStreamerH264Encoder to be more coherent with what is done in LibVPX
     42        (WebCore::GStreamerVP8Encoder::GStreamerVP8Encoder): Renamed VP8Encoder to GStreamerVP8Encoder to be more coherent with what is done in LibVPX
     43        (WebCore::GStreamerVideoEncoderFactory::CreateVideoEncoder):
     44        (WebCore::GStreamerVideoEncoderFactory::GetSupportedFormats const):
     45
    1462018-11-27  Javier Fernandez  <jfernandez@igalia.com>
    247

  • trunk/Source/WebCore/platform/graphics/gstreamer/GStreamerCommon.h

    r237921 r238557  
    136136}
    137137
     138class GstMappedFrame {
     139    WTF_MAKE_NONCOPYABLE(GstMappedFrame);
     140public:
     141
     142    GstMappedFrame(GstBuffer* buffer, GstVideoInfo info, GstMapFlags flags)
     143    {
     144        m_isValid = gst_video_frame_map(&m_frame, &info, buffer, flags);
     145    }
     146
     147    GstMappedFrame(GRefPtr<GstSample> sample, GstMapFlags flags)
     148    {
     149        GstVideoInfo info;
     150
     151        if (!gst_video_info_from_caps(&info, gst_sample_get_caps(sample.get()))) {
     152            m_isValid = false;
     153            return;
     154        }
     155
     156        m_isValid = gst_video_frame_map(&m_frame, &info, gst_sample_get_buffer(sample.get()), flags);
     157    }
     158
     159    GstVideoFrame* get()
     160    {
     161        if (!m_isValid) {
     162            GST_INFO("Invalid frame, returning NULL");
     163
     164            return nullptr;
     165        }
     166
     167        return &m_frame;
     168    }
     169
     170    uint8_t* ComponentData(int comp)
     171    {
     172        return GST_VIDEO_FRAME_COMP_DATA(&m_frame, comp);
     173    }
     174
     175    int ComponentStride(int stride)
     176    {
     177        return GST_VIDEO_FRAME_COMP_STRIDE(&m_frame, stride);
     178    }
     179
     180    GstVideoInfo* info()
     181    {
     182        if (!m_isValid) {
     183            GST_INFO("Invalid frame, returning NULL");
     184
     185            return nullptr;
     186        }
     187
     188        return &m_frame.info;
     189    }
     190
     191    int width()
     192    {
     193        return m_isValid ? GST_VIDEO_FRAME_WIDTH(&m_frame) : -1;
     194    }
     195
     196    int height()
     197    {
     198        return m_isValid ? GST_VIDEO_FRAME_HEIGHT(&m_frame) : -1;
     199    }
     200
     201    int format()
     202    {
     203        return m_isValid ? GST_VIDEO_FRAME_FORMAT(&m_frame) : GST_VIDEO_FORMAT_UNKNOWN;
     204    }
     205
     206    ~GstMappedFrame()
     207    {
     208        if (m_isValid)
     209            gst_video_frame_unmap(&m_frame);
     210        m_isValid = false;
     211    }
     212
     213    explicit operator bool() const { return m_isValid; }
     214
     215private:
     216    GstVideoFrame m_frame;
     217    bool m_isValid { false };
     218};
     219
     220
    138221bool gstRegistryHasElementForMediaType(GList* elementFactories, const char* capsString);
    139 void connectSimpleBusMessageCallback(GstElement *pipeline);
    140 void disconnectSimpleBusMessageCallback(GstElement *pipeline);
     222void connectSimpleBusMessageCallback(GstElement* pipeline);
     223void disconnectSimpleBusMessageCallback(GstElement* pipeline);
    141224
    142225}

  • trunk/Source/WebCore/platform/graphics/gstreamer/GUniquePtrGStreamer.h

    r238083 r238557  
    2727#include <gst/gststructure.h>
    2828#include <gst/pbutils/install-plugins.h>
     29#include <gst/video/video.h>
    2930#include <wtf/glib/GUniquePtr.h>
    3031
    … …  
    3738WTF_DEFINE_GPTR_DELETER(GstFlowCombiner, gst_flow_combiner_free)
    3839WTF_DEFINE_GPTR_DELETER(GstByteReader, gst_byte_reader_free)
     40WTF_DEFINE_GPTR_DELETER(GstVideoConverter, gst_video_converter_free)
    3941
    4042}

  • trunk/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoFrameLibWebRTC.cpp

    r237819 r238557  
    2121#if USE(GSTREAMER) && USE(LIBWEBRTC)
    2222#include "GStreamerVideoFrameLibWebRTC.h"
     23
     24#include <thread>
    2325
    2426namespace WebCore {
    … …  
    9193rtc::scoped_refptr<webrtc::I420BufferInterface> GStreamerVideoFrameLibWebRTC::ToI420()
    9294{
    93     GstVideoInfo info;
    94     GstVideoFrame frame;
     95    GstMappedFrame inFrame(m_sample, GST_MAP_READ);
    9596
    96     if (!gst_video_info_from_caps(&info, gst_sample_get_caps(m_sample.get())))
    97         ASSERT_NOT_REACHED();
     97    if (!inFrame) {
     98        GST_WARNING("Could not map frame");
    9899
    99     if (GST_VIDEO_INFO_FORMAT(&info) != GST_VIDEO_FORMAT_I420)
    100100        return nullptr;
     101    }
    101102
    102     gst_video_frame_map(&frame, &info, gst_sample_get_buffer(m_sample.get()), GST_MAP_READ);
    103 
    104     auto newBuffer = m_bufferPool.CreateBuffer(GST_VIDEO_FRAME_WIDTH(&frame),
    105         GST_VIDEO_FRAME_HEIGHT(&frame));
    106 
     103    auto newBuffer = m_bufferPool.CreateBuffer(inFrame.width(), inFrame.height());
    107104    ASSERT(newBuffer);
    108105    if (!newBuffer) {
    109         gst_video_frame_unmap(&frame);
    110106        GST_WARNING("RealtimeOutgoingVideoSourceGStreamer::videoSampleAvailable unable to allocate buffer for conversion to YUV");
    111107        return nullptr;
    112108    }
    113109
     110    if (inFrame.format() != GST_VIDEO_FORMAT_I420) {
     111        GstVideoInfo outInfo;
     112
     113        gst_video_info_set_format(&outInfo, GST_VIDEO_FORMAT_I420, inFrame.width(),
     114            inFrame.height());
     115        auto info = inFrame.info();
     116        outInfo.fps_n = info->fps_n;
     117        outInfo.fps_d = info->fps_d;
     118
     119        GRefPtr<GstBuffer> buffer = adoptGRef(gst_buffer_new_wrapped_full(GST_MEMORY_FLAG_NO_SHARE, newBuffer->MutableDataY(),
     120            outInfo.size, 0, outInfo.size, nullptr, nullptr));
     121
     122        GstMappedFrame outFrame(buffer.get(), outInfo, GST_MAP_WRITE);
     123
     124        GUniquePtr<GstVideoConverter> videoConverter(gst_video_converter_new(inFrame.info(),
     125            &outInfo, gst_structure_new("GstVideoConvertConfig",
     126            GST_VIDEO_CONVERTER_OPT_THREADS, G_TYPE_UINT, std::thread::hardware_concurrency() || 1 , nullptr)));
     127
     128        ASSERT(videoConverter);
     129
     130        gst_video_converter_frame(videoConverter.get(), inFrame.get(), outFrame.get());
     131
     132        return newBuffer;
     133    }
     134
    114135    newBuffer->Copy(
    115         GST_VIDEO_FRAME_WIDTH(&frame),
    116         GST_VIDEO_FRAME_HEIGHT(&frame),
    117         GST_VIDEO_FRAME_COMP_DATA(&frame, 0),
    118         GST_VIDEO_FRAME_COMP_STRIDE(&frame, 0),
    119         GST_VIDEO_FRAME_COMP_DATA(&frame, 1),
    120         GST_VIDEO_FRAME_COMP_STRIDE(&frame, 1),
    121         GST_VIDEO_FRAME_COMP_DATA(&frame, 2),
    122         GST_VIDEO_FRAME_COMP_STRIDE(&frame, 2));
    123     gst_video_frame_unmap(&frame);
     136        inFrame.width(),
     137        inFrame.height(),
     138        inFrame.ComponentData(0),
     139        inFrame.ComponentStride(0),
     140        inFrame.ComponentData(1),
     141        inFrame.ComponentStride(1),
     142        inFrame.ComponentData(2),
     143        inFrame.ComponentStride(2));
    124144
    125145    return newBuffer;

  • trunk/Source/WebCore/platform/mediastream/libwebrtc/GStreamerVideoDecoderFactory.cpp

    r237819 r238557  
    3030#include "webrtc/modules/video_coding/codecs/h264/include/h264.h"
    3131#include "webrtc/modules/video_coding/codecs/vp8/include/vp8.h"
     32#include "webrtc/modules/video_coding/codecs/vp8/libvpx_vp8_decoder.h"
    3233#include "webrtc/modules/video_coding/include/video_codec_interface.h"
    3334#include <gst/app/gstappsink.h>
    … …  
    215216    }
    216217
    217     bool HasGstDecoder()
    218     {
    219 
     218    static GRefPtr<GstElementFactory> GstDecoderFactory(const char *capsStr)
     219    {
    220220        auto all_decoders = gst_element_factory_list_get_elements(GST_ELEMENT_FACTORY_TYPE_DECODER,
    221221            GST_RANK_MARGINAL);
    222         auto caps = adoptGRef(gst_caps_from_string(Caps()));
     222        auto caps = adoptGRef(gst_caps_from_string(capsStr));
    223223        auto decoders = gst_element_factory_list_filter(all_decoders,
    224224            caps.get(), GST_PAD_SINK, FALSE);
    225225
    226226        gst_plugin_feature_list_free(all_decoders);
     227        GRefPtr<GstElementFactory> res;
     228        if (decoders)
     229            res = GST_ELEMENT_FACTORY(decoders->data);
    227230        gst_plugin_feature_list_free(decoders);
    228231
    229         return decoders != nullptr;
     232        return res;
     233    }
     234
     235    bool HasGstDecoder()
     236    {
     237        return GstDecoderFactory(Caps());
    230238    }
    231239
    … …  
    344352    const gchar* Name() final { return cricket::kVp8CodecName; }
    345353    webrtc::VideoCodecType CodecType() final { return webrtc::kVideoCodecVP8; }
     354    static std::unique_ptr<webrtc::VideoDecoder> Create()
     355    {
     356        auto factory = GstDecoderFactory("video/x-vp8");
     357
     358        if (factory && !g_strcmp0(GST_OBJECT_NAME(GST_OBJECT(factory.get())), "vp8dec")) {
     359            GST_INFO("Our best GStreamer VP8 decoder is vp8dec, better use the one from LibWebRTC");
     360
     361            return std::unique_ptr<webrtc::VideoDecoder>(new webrtc::LibvpxVp8Decoder());
     362        }
     363
     364        return std::unique_ptr<webrtc::VideoDecoder>(new VP8Decoder());
     365    }
    346366};
    347367
    348368std::unique_ptr<webrtc::VideoDecoder> GStreamerVideoDecoderFactory::CreateVideoDecoder(const webrtc::SdpVideoFormat& format)
    349369{
    350     GStreamerVideoDecoder* dec;
     370    webrtc::VideoDecoder* dec;
    351371
    352372    if (format.name == cricket::kH264CodecName)
    353373        dec = new H264Decoder();
    354374    else if (format.name == cricket::kVp8CodecName)
    355         dec = new VP8Decoder();
     375        return VP8Decoder::Create();
    356376    else {
    357377        GST_ERROR("Could not create decoder for %s", format.name.c_str());

  • trunk/Source/WebCore/platform/mediastream/libwebrtc/GStreamerVideoEncoder.cpp

    r237865 r238557  
    279279
    280280static void
    281 setup_vp8enc (GObject * encoder)
    282 {
    283   gst_preset_load_preset (GST_PRESET (encoder), "Profile Realtime");
    284 }
    285 
    286 static void
    287281setup_openh264enc (GObject *)
    288282{
    … …  
    294288{
    295289  g_object_set (encoder, prop_name, bitrate, NULL);
    296 }
    297 
    298 static void
    299 set_bitrate_bit_per_sec (GObject * encoder, const gchar * prop_name,
    300     gint bitrate)
    301 {
    302   g_object_set (encoder, prop_name, bitrate * KBIT_TO_BIT, NULL);
    303290}
    304291
    … …  
    345332      "video/x-h264,alignment=au,stream-format=byte-stream,profile=baseline",
    346333      setup_openh264enc, "bitrate", set_bitrate_kbit_per_sec, "gop-size");
    347   register_known_encoder (ENCODER_VP8, "vp8enc", NULL, "video/x-vp8", NULL,
    348       setup_vp8enc, "target-bitrate", set_bitrate_bit_per_sec,
    349       "keyframe-max-dist");
    350334}
    351335

  • trunk/Source/WebCore/platform/mediastream/libwebrtc/GStreamerVideoEncoderFactory.cpp

    r238027 r238557  
    3131#include "webrtc/modules/video_coding/codecs/h264/include/h264.h"
    3232#include "webrtc/modules/video_coding/codecs/vp8/include/vp8.h"
     33#include "webrtc/modules/video_coding/codecs/vp8/libvpx_vp8_encoder.h"
    3334#include "webrtc/modules/video_coding/include/video_codec_interface.h"
    3435#include "webrtc/modules/video_coding/utility/simulcast_utility.h"
    … …  
    411412};
    412413
    413 class H264Encoder : public GStreamerVideoEncoder {
     414class GStreamerH264Encoder : public GStreamerVideoEncoder {
    414415public:
    415     H264Encoder() { }
    416 
    417     H264Encoder(const webrtc::SdpVideoFormat& format)
     416    GStreamerH264Encoder() { }
     417
     418    GStreamerH264Encoder(const webrtc::SdpVideoFormat& format)
    418419        : m_parser(gst_h264_nal_parser_new())
    419420        , packetizationMode(webrtc::H264PacketizationMode::NonInterleaved)
    … …  
    509510};
    510511
    511 class VP8Encoder : public GStreamerVideoEncoder {
     512class GStreamerVP8Encoder : public GStreamerVideoEncoder {
    512513public:
    513     VP8Encoder() { }
    514     VP8Encoder(const webrtc::SdpVideoFormat&) { }
     514    GStreamerVP8Encoder() { }
     515    GStreamerVP8Encoder(const webrtc::SdpVideoFormat&) { }
    515516    const gchar* Caps() final { return "video/x-vp8"; }
    516517    const gchar* Name() final { return cricket::kVp8CodecName; }
    … …  
    536537std::unique_ptr<webrtc::VideoEncoder> GStreamerVideoEncoderFactory::CreateVideoEncoder(const webrtc::SdpVideoFormat& format)
    537538{
    538     if (format.name == cricket::kVp8CodecName)
    539         return std::make_unique<VP8Encoder>(format);
     539    if (format.name == cricket::kVp8CodecName) {
     540        GRefPtr<GstElement> webrtcencoder = adoptGRef(GST_ELEMENT(g_object_ref_sink(gst_element_factory_make("webrtcvideoencoder", NULL))));
     541        GRefPtr<GstElement> encoder = nullptr;
     542
     543        g_object_set(webrtcencoder.get(), "format", adoptGRef(gst_caps_from_string("video/x-vp8")).get(), NULL);
     544        g_object_get(webrtcencoder.get(), "encoder", &encoder.outPtr(), NULL);
     545
     546        if (encoder)
     547            return std::make_unique<GStreamerVP8Encoder>(format);
     548
     549        GST_INFO("Using VP8 Encoder from LibWebRTC.");
     550        return std::make_unique<webrtc::LibvpxVp8Encoder>();
     551    }
    540552
    541553    if (format.name == cricket::kH264CodecName)
    542         return std::make_unique<H264Encoder>(format);
     554        return std::make_unique<GStreamerH264Encoder>(format);
    543555
    544556    return nullptr;
    … …  
    559571    std::vector<webrtc::SdpVideoFormat> supportedCodecs;
    560572
    561     VP8Encoder().AddCodecIfSupported(&supportedCodecs);
    562     H264Encoder().AddCodecIfSupported(&supportedCodecs);
     573    supportedCodecs.push_back(webrtc::SdpVideoFormat(cricket::kVp8CodecName));
     574    GStreamerH264Encoder().AddCodecIfSupported(&supportedCodecs);
    563575
    564576    return supportedCodecs;
Note: See TracChangeset for help on using the changeset viewer.