Changeset 231089 in webkit


Ignore:
Timestamp:
Apr 27, 2018 12:33:53 AM (6 years ago)
Author:
commit-queue@webkit.org
Message:

[EME][GStreamer] Move the decryptor from AppendPipeline to PlaybackPipeline.
​https://bugs.webkit.org/show_bug.cgi?id=181855

Patch by Yacine Bandou <​yacine.bandou_ext@softathome.com> on 2018-04-27
Reviewed by Xabier Rodriguez-Calvar.

The goal of this move is to handle the limitation of SVP (Secure Video Path) memory size.

When the decryptor is in the AppendPipeline and we use SVP, we buffer in MediaSource queue
the decrypted GstBuffers that are in SVP memory.
This behavior cause an out-of-memory error, because we are limited in SVP memory size.

By moving the decryptor in PlaybackPipeline, we avoid to buffer the decrypted GstBuffers
which use the SVP memory and we buffer the encrypted GstBuffers that are in system memory.

This new architecture also allows to start the buffering before obtaining the DRM license
and it makes easier to manage dynamic change of the license or Key.

The decryptor is auto plugged by GStreamer playbin in PlaybackPipeline.

SVP: Secure Video Path also named trusted or protected video path, it is a memory which is
protected by a hardware access control engine, it is not accessible to other unauthorised
software or hardware components.

Tests:

media/encrypted-media/clearKey/clearKey-cenc-audio-playback-mse.html
media/encrypted-media/clearKey/clearKey-cenc-video-playback-mse.html

  • platform/graphics/gstreamer/eme/WebKitCommonEncryptionDecryptorGStreamer.cpp:

(webkitMediaCommonEncryptionDecryptSinkEventHandler):

  • platform/graphics/gstreamer/mse/AppendPipeline.cpp:

(WebCore::dumpAppendState):
(WebCore::AppendPipeline::AppendPipeline):
(WebCore::AppendPipeline::handleNeedContextSyncMessage):
(WebCore::AppendPipeline::handleAppsrcNeedDataReceived):
(WebCore::AppendPipeline::setAppendState):
(WebCore::AppendPipeline::parseDemuxerSrcPadCaps):
(WebCore::AppendPipeline::appsinkNewSample):
(WebCore::AppendPipeline::connectDemuxerSrcPadToAppsinkFromAnyThread):
(WebCore::AppendPipeline::disconnectDemuxerSrcPadFromAppsinkFromAnyThread):
(WebCore::appendPipelineElementMessageCallback): Deleted.
(WebCore::AppendPipeline::handleElementMessage): Deleted.
(WebCore::AppendPipeline::dispatchPendingDecryptionStructure): Deleted.
(WebCore::AppendPipeline::dispatchDecryptionStructure): Deleted.

  • platform/graphics/gstreamer/mse/AppendPipeline.h:
  • platform/graphics/gstreamer/mse/MediaPlayerPrivateGStreamerMSE.cpp:

(WebCore::MediaPlayerPrivateGStreamerMSE::attemptToDecryptWithInstance):

  • platform/graphics/gstreamer/mse/PlaybackPipeline.cpp:
Location:
trunk/Source/WebCore
Files:
6 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebCore/ChangeLog

    r231088 r231089  
     12018-04-27  Yacine Bandou  <yacine.bandou_ext@softathome.com>
     2
     3        [EME][GStreamer] Move the decryptor from AppendPipeline to PlaybackPipeline.
     4        https://bugs.webkit.org/show_bug.cgi?id=181855
     5
     6        Reviewed by Xabier Rodriguez-Calvar.
     7
     8        The goal of this move is to handle the limitation of SVP (Secure Video Path) memory size.
     9
     10        When the decryptor is in the AppendPipeline and we use SVP, we buffer in MediaSource queue
     11        the decrypted GstBuffers that are in SVP memory.
     12        This behavior cause an out-of-memory error, because we are limited in SVP memory size.
     13
     14        By moving the decryptor in PlaybackPipeline, we avoid to buffer the decrypted GstBuffers
     15        which use the SVP memory and we buffer the encrypted GstBuffers that are in system memory.
     16
     17        This new architecture also allows to start the buffering before obtaining the DRM license
     18        and it makes easier to manage dynamic change of the license or Key.
     19
     20        The decryptor is auto plugged by GStreamer playbin in PlaybackPipeline.
     21
     22        SVP: Secure Video Path also named trusted or protected video path, it is a memory which is
     23        protected by a hardware access control engine, it is not accessible to other unauthorised
     24        software or hardware components.
     25
     26        Tests:
     27            media/encrypted-media/clearKey/clearKey-cenc-audio-playback-mse.html
     28            media/encrypted-media/clearKey/clearKey-cenc-video-playback-mse.html
     29
     30        * platform/graphics/gstreamer/eme/WebKitCommonEncryptionDecryptorGStreamer.cpp:
     31        (webkitMediaCommonEncryptionDecryptSinkEventHandler):
     32        * platform/graphics/gstreamer/mse/AppendPipeline.cpp:
     33        (WebCore::dumpAppendState):
     34        (WebCore::AppendPipeline::AppendPipeline):
     35        (WebCore::AppendPipeline::handleNeedContextSyncMessage):
     36        (WebCore::AppendPipeline::handleAppsrcNeedDataReceived):
     37        (WebCore::AppendPipeline::setAppendState):
     38        (WebCore::AppendPipeline::parseDemuxerSrcPadCaps):
     39        (WebCore::AppendPipeline::appsinkNewSample):
     40        (WebCore::AppendPipeline::connectDemuxerSrcPadToAppsinkFromAnyThread):
     41        (WebCore::AppendPipeline::disconnectDemuxerSrcPadFromAppsinkFromAnyThread):
     42        (WebCore::appendPipelineElementMessageCallback): Deleted.
     43        (WebCore::AppendPipeline::handleElementMessage): Deleted.
     44        (WebCore::AppendPipeline::dispatchPendingDecryptionStructure): Deleted.
     45        (WebCore::AppendPipeline::dispatchDecryptionStructure): Deleted.
     46        * platform/graphics/gstreamer/mse/AppendPipeline.h:
     47        * platform/graphics/gstreamer/mse/MediaPlayerPrivateGStreamerMSE.cpp:
     48        (WebCore::MediaPlayerPrivateGStreamerMSE::attemptToDecryptWithInstance):
     49        * platform/graphics/gstreamer/mse/PlaybackPipeline.cpp:
     50
    1512018-04-27  Yacine Bandou  <yacine.bandou_ext@softathome.com>
    252

  • trunk/Source/WebCore/platform/graphics/gstreamer/eme/WebKitCommonEncryptionDecryptorGStreamer.cpp

    r231088 r231089  
    300300
    301301    switch (GST_EVENT_TYPE(event)) {
    302     case GST_EVENT_PROTECTION: {
    303         const char* systemId = nullptr;
    304 
    305         gst_event_parse_protection(event, &systemId, nullptr, nullptr);
    306         GST_TRACE_OBJECT(self, "received protection event for %s", systemId);
    307 
    308         if (!g_strcmp0(systemId, klass->protectionSystemId)) {
    309             GST_DEBUG_OBJECT(self, "sending protection event to the pipeline");
    310             gst_element_post_message(GST_ELEMENT(self),
    311                 gst_message_new_element(GST_OBJECT(self),
    312                     gst_structure_new("drm-key-needed", "event", GST_TYPE_EVENT, event, nullptr)));
    313         }
    314 
    315         gst_event_unref(event);
    316         result = TRUE;
    317         break;
    318     }
    319302    case GST_EVENT_CUSTOM_DOWNSTREAM_OOB: {
    320303        if (klass->handleKeyResponse(self, event)) {

  • trunk/Source/WebCore/platform/graphics/gstreamer/mse/AppendPipeline.cpp

    r230625 r231089  
    5656    case AppendPipeline::AppendState::Ongoing:
    5757        return "Ongoing";
    58     case AppendPipeline::AppendState::KeyNegotiation:
    59         return "KeyNegotiation";
    6058    case AppendPipeline::AppendState::DataStarve:
    6159        return "DataStarve";
    … …  
    9492}
    9593
    96 #if ENABLE(ENCRYPTED_MEDIA)
    97 static void appendPipelineElementMessageCallback(GstBus*, GstMessage* message, AppendPipeline* appendPipeline)
    98 {
    99     appendPipeline->handleElementMessage(message);
    100 }
    101 #endif
    102 
    10394static void appendPipelineStateChangeMessageCallback(GstBus*, GstMessage* message, AppendPipeline* appendPipeline)
    10495{
    … …  
    132123    g_signal_connect(m_bus.get(), "sync-message::need-context", G_CALLBACK(appendPipelineNeedContextMessageCallback), this);
    133124    g_signal_connect(m_bus.get(), "message::application", G_CALLBACK(appendPipelineApplicationMessageCallback), this);
    134 #if ENABLE(ENCRYPTED_MEDIA)
    135     g_signal_connect(m_bus.get(), "message::element", G_CALLBACK(appendPipelineElementMessageCallback), this);
    136 #endif
    137125    g_signal_connect(m_bus.get(), "message::state-changed", G_CALLBACK(appendPipelineStateChangeMessageCallback), this);
    138126
    … …  
    276264    gst_message_parse_context_type(message, &contextType);
    277265    GST_TRACE("context type: %s", contextType);
    278     if (!g_strcmp0(contextType, "drm-preferred-decryption-system-id")
    279         && m_appendState != AppendPipeline::AppendState::KeyNegotiation)
    280         setAppendState(AppendPipeline::AppendState::KeyNegotiation);
    281266
    282267    // MediaPlayerPrivateGStreamerBase will take care of setting up encryption.
    … …  
    329314    ASSERT_NOT_REACHED();
    330315}
    331 
    332 #if ENABLE(ENCRYPTED_MEDIA)
    333 void AppendPipeline::handleElementMessage(GstMessage* message)
    334 {
    335     ASSERT(WTF::isMainThread());
    336 
    337     const GstStructure* structure = gst_message_get_structure(message);
    338     GST_TRACE("%s message from %s", gst_structure_get_name(structure), GST_MESSAGE_SRC_NAME(message));
    339     if (m_playerPrivate && gst_structure_has_name(structure, "drm-key-needed")) {
    340         if (m_appendState != AppendPipeline::AppendState::KeyNegotiation)
    341             setAppendState(AppendPipeline::AppendState::KeyNegotiation);
    342 
    343         GST_DEBUG("sending drm-key-needed message from %s to the player", GST_MESSAGE_SRC_NAME(message));
    344         GRefPtr<GstEvent> event;
    345         gst_structure_get(structure, "event", GST_TYPE_EVENT, &event.outPtr(), nullptr);
    346         m_playerPrivate->handleProtectionEvent(event.get());
    347     }
    348 }
    349 #endif
    350316
    351317void AppendPipeline::handleStateChangeMessage(GstMessage* message)
    … …  
    374340    }
    375341
    376     ASSERT(m_appendState == AppendState::KeyNegotiation || m_appendState == AppendState::Ongoing || m_appendState == AppendState::Sampling);
     342    ASSERT(m_appendState == AppendState::Ongoing || m_appendState == AppendState::Sampling);
    377343    ASSERT(!m_appsrcNeedDataReceived);
    378344
    … …  
    432398    //           |         |            `->Aborting-->NotStarted
    433399    //           |         `->Sampling-···->Sampling-->LastSample-->NotStarted
    434     //           |         |                                     `->Aborting-->NotStarted
    435     //           |         `->KeyNegotiation-->Ongoing-->[...]
     400    //           |                                               `->Aborting-->NotStarted
    436401    //           `->Aborting-->NotStarted
    437402    AppendState oldAppendState = m_appendState;
    … …  
    469434        }
    470435        break;
    471     case AppendState::KeyNegotiation:
    472         switch (newAppendState) {
    473         case AppendState::Ongoing:
    474         case AppendState::Invalid:
    475             ok = true;
    476             break;
    477         default:
    478             break;
    479         }
    480         break;
    481436    case AppendState::Ongoing:
    482437        switch (newAppendState) {
    483         case AppendState::KeyNegotiation:
    484438        case AppendState::Sampling:
    485439        case AppendState::Invalid:
    … …  
    583537    m_demuxerSrcPadCaps = adoptGRef(demuxerSrcPadCaps);
    584538    m_streamType = WebCore::MediaSourceStreamTypeGStreamer::Unknown;
    585 #if ENABLE(ENCRYPTED_MEDIA)
    586     if (areEncryptedCaps(m_demuxerSrcPadCaps.get())) {
    587         // Any previous decryptor should have been removed from the pipeline by disconnectFromAppSinkFromStreamingThread()
    588         ASSERT(!m_decryptor);
    589         GstStructure* structure = gst_caps_get_structure(m_demuxerSrcPadCaps.get(), 0);
    590         m_decryptor = GStreamerEMEUtilities::createDecryptor(gst_structure_get_string(structure, "protection-system"));
    591         if (!m_decryptor) {
    592             GST_ERROR("decryptor not found for caps: %" GST_PTR_FORMAT, m_demuxerSrcPadCaps.get());
    593             return;
    594         }
    595     }
    596 #endif
     539
    597540    const char* originalMediaType = capsMediaType(m_demuxerSrcPadCaps.get());
    598541    if (!MediaPlayerPrivateGStreamerMSE::supportsCodec(originalMediaType)) {
    … …  
    674617        LockHolder locker(m_newSampleLock);
    675618
    676         // If we were in KeyNegotiation but samples are coming, assume we're already OnGoing
    677         if (m_appendState == AppendState::KeyNegotiation)
    678             setAppendState(AppendState::Ongoing);
    679 
    680619        // Ignore samples if we're not expecting them. Refuse processing if we're in Invalid state.
    681620        if (m_appendState != AppendState::Ongoing && m_appendState != AppendState::Sampling) {
    … …  
    1010949        }
    1011950
    1012 #if ENABLE(ENCRYPTED_MEDIA)
    1013         if (m_decryptor) {
    1014             gst_object_ref(m_decryptor.get());
    1015             gst_bin_add(GST_BIN(m_pipeline.get()), m_decryptor.get());
    1016             gst_element_sync_state_with_parent(m_decryptor.get());
    1017 
    1018             GRefPtr<GstPad> decryptorSinkPad = adoptGRef(gst_element_get_static_pad(m_decryptor.get(), "sink"));
    1019             GRefPtr<GstPad> decryptorSrcPad = adoptGRef(gst_element_get_static_pad(m_decryptor.get(), "src"));
    1020 
    1021             gst_pad_link(currentSrcPad.get(), decryptorSinkPad.get());
    1022             currentSrcPad = decryptorSrcPad;
    1023         }
    1024 #endif
    1025 
    1026951        gst_pad_link(currentSrcPad.get(), appsinkSinkPad.get());
    1027952
    1028953        gst_element_sync_state_with_parent(m_appsink.get());
    1029954
    1030 #if ENABLE(ENCRYPTED_MEDIA)
    1031         if (m_pendingDecryptionStructure)
    1032             dispatchPendingDecryptionStructure();
    1033 #endif
    1034955        gst_element_set_state(m_pipeline.get(), GST_STATE_PAUSED);
    1035956        gst_element_sync_state_with_parent(m_appsink.get());
    … …  
    11101031    GST_DEBUG("Disconnecting appsink");
    11111032
    1112 #if ENABLE(ENCRYPTED_MEDIA)
    1113     if (m_decryptor) {
    1114         gst_element_set_state(m_decryptor.get(), GST_STATE_NULL);
    1115         gst_bin_remove(GST_BIN(m_pipeline.get()), m_decryptor.get());
    1116         m_decryptor = nullptr;
    1117     }
    1118 #endif
    1119 
    11201033    if (m_parser) {
    11211034        gst_element_set_state(m_parser.get(), GST_STATE_NULL);
    … …  
    11261039    GST_DEBUG_BIN_TO_DOT_FILE_WITH_TS(GST_BIN(m_pipeline.get()), GST_DEBUG_GRAPH_SHOW_ALL, "pad-removed-after");
    11271040}
    1128 
    1129 #if ENABLE(ENCRYPTED_MEDIA)
    1130 void AppendPipeline::dispatchPendingDecryptionStructure()
    1131 {
    1132     ASSERT(m_decryptor);
    1133     ASSERT(m_pendingDecryptionStructure);
    1134     ASSERT(m_appendState == AppendState::KeyNegotiation);
    1135     GST_TRACE("dispatching key to append pipeline %p", this);
    1136 
    1137     // Release the m_pendingDecryptionStructure object since
    1138     // gst_event_new_custom() takes over ownership of it.
    1139     gst_element_send_event(m_pipeline.get(), gst_event_new_custom(GST_EVENT_CUSTOM_DOWNSTREAM_OOB, m_pendingDecryptionStructure.release()));
    1140 
    1141     setAppendState(AppendState::Ongoing);
    1142 }
    1143 
    1144 void AppendPipeline::dispatchDecryptionStructure(GUniquePtr<GstStructure>&& structure)
    1145 {
    1146     if (m_appendState == AppendState::KeyNegotiation) {
    1147         GST_TRACE("append pipeline %p in key negotiation", this);
    1148         m_pendingDecryptionStructure = WTFMove(structure);
    1149         if (m_decryptor)
    1150             dispatchPendingDecryptionStructure();
    1151         else
    1152             GST_TRACE("no decryptor yet, waiting for it");
    1153     } else
    1154         GST_TRACE("append pipeline %p not in key negotiation", this);
    1155 }
    1156 #endif
    11571041
    11581042static void appendPipelineAppsinkCapsChanged(GObject* appsinkPad, GParamSpec*, AppendPipeline* appendPipeline)

  • trunk/Source/WebCore/platform/graphics/gstreamer/mse/AppendPipeline.h

    r230625 r231089  
    4343class AppendPipeline : public ThreadSafeRefCounted<AppendPipeline> {
    4444public:
    45     enum class AppendState { Invalid, NotStarted, Ongoing, KeyNegotiation, DataStarve, Sampling, LastSample, Aborting };
     45    enum class AppendState { Invalid, NotStarted, Ongoing, DataStarve, Sampling, LastSample, Aborting };
    4646
    4747    AppendPipeline(Ref<MediaSourceClientGStreamerMSE>, Ref<SourceBufferPrivateGStreamer>, MediaPlayerPrivateGStreamerMSE&);
    … …  
    5151    void handleApplicationMessage(GstMessage*);
    5252    void handleStateChangeMessage(GstMessage*);
    53 #if ENABLE(ENCRYPTED_MEDIA)
    54     void handleElementMessage(GstMessage*);
    55 #endif
    5653
    5754    gint id();
    … …  
    6158    GstFlowReturn handleNewAppsinkSample(GstElement*);
    6259    GstFlowReturn pushNewBuffer(GstBuffer*);
    63 #if ENABLE(ENCRYPTED_MEDIA)
    64     void dispatchDecryptionStructure(GUniquePtr<GstStructure>&&);
    65 #endif
    6660
    6761    // Takes ownership of caps.
    … …  
    9993    void removeAppsrcDataLeavingProbe();
    10094    void setAppsrcDataLeavingProbe();
    101 #if ENABLE(ENCRYPTED_MEDIA)
    102     void dispatchPendingDecryptionStructure();
    103 #endif
    10495
    10596    Ref<MediaSourceClientGStreamerMSE> m_mediaSourceClient;
    … …  
    119110    GRefPtr<GstElement> m_demux;
    120111    GRefPtr<GstElement> m_parser; // Optional.
    121 #if ENABLE(ENCRYPTED_MEDIA)
    122     GRefPtr<GstElement> m_decryptor;
    123 #endif
    124112    // The demuxer has one src stream only, so only one appsink is needed and linked to it.
    125113    GRefPtr<GstElement> m_appsink;
    … …  
    156144
    157145    GRefPtr<GstBuffer> m_pendingBuffer;
    158 #if ENABLE(ENCRYPTED_MEDIA)
    159     GUniquePtr<GstStructure> m_pendingDecryptionStructure;
    160 #endif
    161146};
    162147

  • trunk/Source/WebCore/platform/graphics/gstreamer/mse/MediaPlayerPrivateGStreamerMSE.cpp

    r230625 r231089  
    956956        gst_structure_set_value(structure.get(), "key-values", &keyValueList);
    957957
    958         for (auto it : m_appendPipelinesMap)
    959             it.value->dispatchDecryptionStructure(GUniquePtr<GstStructure>(gst_structure_copy(structure.get())));
     958        gst_element_send_event(m_playbackPipeline->pipeline(), gst_event_new_custom(GST_EVENT_CUSTOM_DOWNSTREAM_OOB, gst_structure_copy(structure.get())));
    960959    }
    961960}

  • trunk/Source/WebCore/platform/graphics/gstreamer/mse/PlaybackPipeline.cpp

    r230909 r231089  
    181181    GUniquePtr<gchar> parserBinName(g_strdup_printf("streamparser%u", padId));
    182182
    183     if (!g_strcmp0(mediaType, "video/x-h264")) {
     183    if (areEncryptedCaps(caps)) {
     184        GST_DEBUG("It's encrypted content, parsers are not needed before decrypting the content");
     185    } else if (!g_strcmp0(mediaType, "video/x-h264")) {
    184186        GRefPtr<GstCaps> filterCaps = adoptGRef(gst_caps_new_simple("video/x-h264", "alignment", G_TYPE_STRING, "au", nullptr));
    185187        GstElement* capsfilter = gst_element_factory_make("capsfilter", nullptr);
Note: See TracChangeset for help on using the changeset viewer.