Changeset 261683 in webkit


Ignore:
Timestamp:
May 14, 2020 2:58:16 AM (4 years ago)
Author:
aboya@igalia.com
Message:

[GStreamer] Playbin3 track switch rework
​https://bugs.webkit.org/show_bug.cgi?id=211623

Reviewed by Philippe Normand.

This patch reworks how track selection and reporting of selected
tracks is done in the player.

The following found limitations and assumptions in current GStreamer
have informed this patch:

a) Although the API for playbin3 is designed to be able to accept any
number of tracks of any kind, this is not supported in practice.

b) The first track of each type is always selected. Even in playbin3
mode, looking for GST_STREAM_FLAG_SELECT is not a reliable method, as
in most cases the demuxer does not set it at all. [qtdemux never sets
it at all, and matroskademux only sets it in certain cases.]

c) Sending GST_EVENT_SELECT_STREAMS is only safe at certain moments.
It's not safe before pre-roll, after EOS or during the handling of
another SELECT_STREAMS.

d) Selecting text tracks with playbin APIs is not relevant. All text
tracks are already being picked by WebKitTextCombiner, unaffected by
playbin track selection.

e) Tracks requested in a GST_EVENT_SELECT_STREAMS are eventually
selected. On the other hand, looking at
GST_MESSAGE_STREAMS_SELECTED's content is not reliable, as this has
been seen to miss tracks depending on thread luck.

This patch takes the points above into account to rework how track
selection is handled in MediaPlayerPrivateGStreamer and fix the
following issues:

1) In playbin3 mode, no track was marked as selected initially,
because of reliance on GST_STREAM_FLAG_SELECT.

2) In playbin2 mode, sometimes tracks would not be initially marked as
selected. This occurred because of reliance on the "selected" property
in inputselector sinkpads, whose initialization is racy -- it can
occur after the track has been added and picked up by WebKit.

3) In playbin3 mode, the limitations explained before has been honored
to make track selection stable, delaying SELECT_STREAMS events until
they are safe to send.

This patch doesn't introduce significative behavior changes, rather
aiming for improving the stabilitity of the player. Existing tests
should provide enough coverage.

  • platform/graphics/gstreamer/AudioTrackPrivateGStreamer.cpp:

(WebCore::AudioTrackPrivateGStreamer::AudioTrackPrivateGStreamer):
(WebCore::AudioTrackPrivateGStreamer::setEnabled):

  • platform/graphics/gstreamer/AudioTrackPrivateGStreamer.h:
  • platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:

(WebCore::MediaPlayerPrivateGStreamer::notifyPlayerOfVideo):
(WebCore::MediaPlayerPrivateGStreamer::notifyPlayerOfAudio):
(WebCore::MediaPlayerPrivateGStreamer::updateEnabledVideoTrack):
(WebCore::MediaPlayerPrivateGStreamer::updateEnabledAudioTrack):
(WebCore::MediaPlayerPrivateGStreamer::playbin3SendSelectStreamsIfAppropriate):
(WebCore::MediaPlayerPrivateGStreamer::updateTracks):
(WebCore::MediaPlayerPrivateGStreamer::handleSyncMessage):
(WebCore::MediaPlayerPrivateGStreamer::handleMessage):
(WebCore::MediaPlayerPrivateGStreamer::didEnd):

  • platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.h:
  • platform/graphics/gstreamer/TrackPrivateBaseGStreamer.cpp:

(WebCore::TrackPrivateBaseGStreamer::TrackPrivateBaseGStreamer):

  • platform/graphics/gstreamer/TrackPrivateBaseGStreamer.h:
  • platform/graphics/gstreamer/VideoTrackPrivateGStreamer.cpp:

(WebCore::VideoTrackPrivateGStreamer::VideoTrackPrivateGStreamer):
(WebCore::VideoTrackPrivateGStreamer::setSelected):

  • platform/graphics/gstreamer/VideoTrackPrivateGStreamer.h:
Location:
trunk/Source/WebCore
Files:
9 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebCore/ChangeLog

    r261680 r261683  
     12020-05-14  Alicia Boya García  <aboya@igalia.com>
     2
     3        [GStreamer] Playbin3 track switch rework
     4        https://bugs.webkit.org/show_bug.cgi?id=211623
     5
     6        Reviewed by Philippe Normand.
     7
     8        This patch reworks how track selection and reporting of selected
     9        tracks is done in the player.
     10
     11        The following found limitations and assumptions in current GStreamer
     12        have informed this patch:
     13
     14        a) Although the API for playbin3 is designed to be able to accept any
     15        number of tracks of any kind, this is not supported in practice.
     16
     17        b) The first track of each type is always selected. Even in playbin3
     18        mode, looking for GST_STREAM_FLAG_SELECT is not a reliable method, as
     19        in most cases the demuxer does not set it at all. [qtdemux never sets
     20        it at all, and matroskademux only sets it in certain cases.]
     21
     22        c) Sending GST_EVENT_SELECT_STREAMS is only safe at certain moments.
     23        It's not safe before pre-roll, after EOS or during the handling of
     24        another SELECT_STREAMS.
     25
     26        d) Selecting text tracks with playbin APIs is not relevant. All text
     27        tracks are already being picked by WebKitTextCombiner, unaffected by
     28        playbin track selection.
     29
     30        e) Tracks requested in a GST_EVENT_SELECT_STREAMS are eventually
     31        selected. On the other hand,  looking at
     32        GST_MESSAGE_STREAMS_SELECTED's content is not reliable, as this has
     33        been seen to miss tracks depending on thread luck.
     34
     35        This patch takes the points above into account to rework how track
     36        selection is handled in MediaPlayerPrivateGStreamer and fix the
     37        following issues:
     38
     39        1) In playbin3 mode, no track was marked as selected initially,
     40        because of reliance on GST_STREAM_FLAG_SELECT.
     41
     42        2) In playbin2 mode, sometimes tracks would not be initially marked as
     43        selected. This occurred because of reliance on the "selected" property
     44        in inputselector sinkpads, whose initialization is racy -- it can
     45        occur after the track has been added and picked up by WebKit.
     46
     47        3) In playbin3 mode, the limitations explained before has been honored
     48        to make track selection stable, delaying SELECT_STREAMS events until
     49        they are safe to send.
     50
     51        This patch doesn't introduce significative behavior changes, rather
     52        aiming for improving the stabilitity of the player. Existing tests
     53        should provide enough coverage.
     54
     55        * platform/graphics/gstreamer/AudioTrackPrivateGStreamer.cpp:
     56        (WebCore::AudioTrackPrivateGStreamer::AudioTrackPrivateGStreamer):
     57        (WebCore::AudioTrackPrivateGStreamer::setEnabled):
     58        * platform/graphics/gstreamer/AudioTrackPrivateGStreamer.h:
     59        * platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:
     60        (WebCore::MediaPlayerPrivateGStreamer::notifyPlayerOfVideo):
     61        (WebCore::MediaPlayerPrivateGStreamer::notifyPlayerOfAudio):
     62        (WebCore::MediaPlayerPrivateGStreamer::updateEnabledVideoTrack):
     63        (WebCore::MediaPlayerPrivateGStreamer::updateEnabledAudioTrack):
     64        (WebCore::MediaPlayerPrivateGStreamer::playbin3SendSelectStreamsIfAppropriate):
     65        (WebCore::MediaPlayerPrivateGStreamer::updateTracks):
     66        (WebCore::MediaPlayerPrivateGStreamer::handleSyncMessage):
     67        (WebCore::MediaPlayerPrivateGStreamer::handleMessage):
     68        (WebCore::MediaPlayerPrivateGStreamer::didEnd):
     69        * platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.h:
     70        * platform/graphics/gstreamer/TrackPrivateBaseGStreamer.cpp:
     71        (WebCore::TrackPrivateBaseGStreamer::TrackPrivateBaseGStreamer):
     72        * platform/graphics/gstreamer/TrackPrivateBaseGStreamer.h:
     73        * platform/graphics/gstreamer/VideoTrackPrivateGStreamer.cpp:
     74        (WebCore::VideoTrackPrivateGStreamer::VideoTrackPrivateGStreamer):
     75        (WebCore::VideoTrackPrivateGStreamer::setSelected):
     76        * platform/graphics/gstreamer/VideoTrackPrivateGStreamer.h:
     77
    1782020-05-14  Philippe Normand  <pnormand@igalia.com>
    279

  • trunk/Source/WebCore/platform/graphics/gstreamer/AudioTrackPrivateGStreamer.cpp

    r246677 r261683  
    4141    // FIXME: Get a real ID from the tkhd atom.
    4242    m_id = "A" + String::number(index);
    43     notifyTrackOfActiveChanged();
    4443}
    4544
    … …  
    5756
    5857    m_id = gst_stream_get_stream_id(stream.get());
    59     if (gst_stream_get_stream_flags(stream.get()) & GST_STREAM_FLAG_SELECT)
    60         markAsActive();
    61 
    62     notifyTrackOfActiveChanged();
    6358}
    6459
    … …  
    7772}
    7873
    79 void AudioTrackPrivateGStreamer::markAsActive()
    80 {
    81     AudioTrackPrivate::setEnabled(true);
    82 }
    83 
    8474void AudioTrackPrivateGStreamer::setEnabled(bool enabled)
    8575{
    … …  
    8878    AudioTrackPrivate::setEnabled(enabled);
    8979
    90     if (enabled && m_player)
    91         m_player->enableTrack(TrackPrivateBaseGStreamer::TrackType::Audio, m_index);
     80    if (m_player)
     81        m_player->updateEnabledAudioTrack();
    9282}
    9383

  • trunk/Source/WebCore/platform/graphics/gstreamer/AudioTrackPrivateGStreamer.h

    r246677 r261683  
    5454
    5555    void setEnabled(bool) override;
    56     void markAsActive();
    5756    void setActive(bool enabled) override { setEnabled(enabled); }
    5857

  • trunk/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp

    r261680 r261683  
    106106        m_has##Type = true;                                             \
    107107        if (!useMediaSource) {                                          \
    108             RefPtr<Type##TrackPrivateGStreamer> track = Type##TrackPrivateGStreamer::create(makeWeakPtr(*this), i, stream); \
     108            RefPtr<Type##TrackPrivateGStreamer> track = Type##TrackPrivateGStreamer::create(makeWeakPtr(*this), type##TrackIndex++, stream); \
     109            if (!track->trackIndex()) {                                 \
     110                track->setActive(true);                                 \
     111                m_wanted##Type##StreamId = track->id();                 \
     112                m_requested##Type##StreamId = track->id();              \
     113            }                                                           \
    109114            m_##type##Tracks.add(track->id(), track);                   \
    110115            m_player->add##Type##Track(*track);                         \
    111             if (gst_stream_get_stream_flags(stream.get()) & GST_STREAM_FLAG_SELECT) \
    112                 m_current##Type##StreamId = String(gst_stream_get_stream_id(stream.get())); \
    113116        }                                                               \
    114117    } G_STMT_END
    … …  
    10751078
    10761079        auto track = VideoTrackPrivateGStreamer::create(makeWeakPtr(*this), i, pad);
     1080        if (!track->trackIndex())
     1081            track->setActive(true);
    10771082        ASSERT(streamId == track->id());
    10781083        m_videoTracks.add(streamId, track.copyRef());
    … …  
    11491154
    11501155        auto track = AudioTrackPrivateGStreamer::create(makeWeakPtr(*this), i, pad);
     1156        if (!track->trackIndex())
     1157            track->setActive(true);
    11511158        ASSERT(streamId == track->id());
    11521159        m_audioTracks.add(streamId, track);
    … …  
    14241431}
    14251432
    1426 void MediaPlayerPrivateGStreamer::enableTrack(TrackPrivateBaseGStreamer::TrackType trackType, unsigned index)
    1427 {
    1428     // FIXME: Remove isMediaSource() test below when fixing https://bugs.webkit.org/show_bug.cgi?id=182531.
    1429     if (isMediaSource()) {
    1430         GST_FIXME_OBJECT(m_pipeline.get(), "Audio/Video/Text track switching is not yet supported by the MSE backend.");
    1431         return;
    1432     }
    1433 
    1434     const char* propertyName;
    1435     const char* trackTypeAsString;
    1436     Vector<String> selectedStreams;
    1437     String selectedStreamId;
    1438 
    1439     GstStream* stream = nullptr;
    1440 
    1441     if (!m_isLegacyPlaybin) {
    1442         stream = gst_stream_collection_get_stream(m_streamCollection.get(), index);
    1443         if (!stream) {
    1444             GST_WARNING_OBJECT(pipeline(), "No stream to select at index %u", index);
    1445             return;
    1446         }
    1447         selectedStreamId = String::fromUTF8(gst_stream_get_stream_id(stream));
    1448         selectedStreams.append(selectedStreamId);
    1449     }
    1450 
    1451     switch (trackType) {
    1452     case TrackPrivateBaseGStreamer::TrackType::Audio:
    1453         propertyName = "current-audio";
    1454         trackTypeAsString = "audio";
    1455         if (!selectedStreamId.isEmpty() && selectedStreamId == m_currentAudioStreamId) {
    1456             GST_INFO_OBJECT(pipeline(), "%s stream: %s already selected, not doing anything.", trackTypeAsString, selectedStreamId.utf8().data());
    1457             return;
    1458         }
    1459 
    1460         if (!m_currentTextStreamId.isEmpty())
    1461             selectedStreams.append(m_currentTextStreamId);
    1462         if (!m_currentVideoStreamId.isEmpty())
    1463             selectedStreams.append(m_currentVideoStreamId);
    1464         break;
    1465     case TrackPrivateBaseGStreamer::TrackType::Video:
    1466         propertyName = "current-video";
    1467         trackTypeAsString = "video";
    1468         if (!selectedStreamId.isEmpty() && selectedStreamId == m_currentVideoStreamId) {
    1469             GST_INFO_OBJECT(pipeline(), "%s stream: %s already selected, not doing anything.", trackTypeAsString, selectedStreamId.utf8().data());
    1470             return;
    1471         }
    1472 
    1473         if (!m_currentAudioStreamId.isEmpty())
    1474             selectedStreams.append(m_currentAudioStreamId);
    1475         if (!m_currentTextStreamId.isEmpty())
    1476             selectedStreams.append(m_currentTextStreamId);
    1477         break;
    1478     case TrackPrivateBaseGStreamer::TrackType::Text:
    1479         propertyName = "current-text";
    1480         trackTypeAsString = "text";
    1481         if (!selectedStreamId.isEmpty() && selectedStreamId == m_currentTextStreamId) {
    1482             GST_INFO_OBJECT(pipeline(), "%s stream: %s already selected, not doing anything.", trackTypeAsString, selectedStreamId.utf8().data());
    1483             return;
    1484         }
    1485 
    1486         if (!m_currentAudioStreamId.isEmpty())
    1487             selectedStreams.append(m_currentAudioStreamId);
    1488         if (!m_currentVideoStreamId.isEmpty())
    1489             selectedStreams.append(m_currentVideoStreamId);
    1490         break;
    1491     case TrackPrivateBaseGStreamer::TrackType::Unknown:
    1492         FALLTHROUGH;
    1493     default:
    1494         ASSERT_NOT_REACHED();
    1495     }
    1496 
    1497     GST_INFO_OBJECT(pipeline(), "Enabling %s track with index: %u", trackTypeAsString, index);
    1498     if (m_isLegacyPlaybin)
    1499         g_object_set(m_pipeline.get(), propertyName, index, nullptr);
    1500     else {
    1501         GList* selectedStreamsList = nullptr;
    1502 
    1503         for (const auto& streamId : selectedStreams)
    1504             selectedStreamsList = g_list_append(selectedStreamsList, g_strdup(streamId.utf8().data()));
    1505 
    1506         // TODO: MSE GstStream API support: https://bugs.webkit.org/show_bug.cgi?id=182531
    1507         gst_element_send_event(m_pipeline.get(), gst_event_new_select_streams(selectedStreamsList));
    1508         g_list_free_full(selectedStreamsList, reinterpret_cast<GDestroyNotify>(g_free));
    1509     }
    1510 }
    1511 
    1512 void MediaPlayerPrivateGStreamer::updateTracks()
     1433void MediaPlayerPrivateGStreamer::updateEnabledVideoTrack()
     1434{
     1435    VideoTrackPrivateGStreamer* wantedTrack = nullptr;
     1436    for (auto& pair : m_videoTracks) {
     1437        VideoTrackPrivateGStreamer* track = pair.value.get();
     1438        if (track->selected()) {
     1439            wantedTrack = track;
     1440            break;
     1441        }
     1442    }
     1443
     1444    // No active track, no changes.
     1445    if (!wantedTrack)
     1446        return;
     1447
     1448    if (m_isLegacyPlaybin) {
     1449        GST_DEBUG_OBJECT(m_pipeline.get(), "Setting playbin2 current-video=%d", wantedTrack->trackIndex());
     1450        g_object_set(m_pipeline.get(), "current-video", wantedTrack->trackIndex(), nullptr);
     1451    } else {
     1452        m_wantedVideoStreamId = wantedTrack->id();
     1453        playbin3SendSelectStreamsIfAppropriate();
     1454    }
     1455}
     1456
     1457void MediaPlayerPrivateGStreamer::updateEnabledAudioTrack()
     1458{
     1459    AudioTrackPrivateGStreamer* wantedTrack = nullptr;
     1460    for (auto& pair : m_audioTracks) {
     1461        AudioTrackPrivateGStreamer* track = pair.value.get();
     1462        if (track->enabled()) {
     1463            wantedTrack = track;
     1464            break;
     1465        }
     1466    }
     1467
     1468    // No active track, no changes.
     1469    if (!wantedTrack)
     1470        return;
     1471
     1472    if (m_isLegacyPlaybin) {
     1473        GST_DEBUG_OBJECT(m_pipeline.get(), "Setting playbin2 current-audio=%d", wantedTrack->trackIndex());
     1474        g_object_set(m_pipeline.get(), "current-audio", wantedTrack->trackIndex(), nullptr);
     1475    } else {
     1476        m_wantedAudioStreamId = wantedTrack->id();
     1477        playbin3SendSelectStreamsIfAppropriate();
     1478    }
     1479}
     1480
     1481void MediaPlayerPrivateGStreamer::playbin3SendSelectStreamsIfAppropriate()
    15131482{
    15141483    ASSERT(!m_isLegacyPlaybin);
    15151484
     1485    bool haveDifferentStreamIds = (m_wantedAudioStreamId != m_currentAudioStreamId || m_wantedVideoStreamId != m_currentVideoStreamId);
     1486    bool shouldSendSelectStreams = !m_waitingForStreamsSelectedEvent && haveDifferentStreamIds && m_currentState == GST_STATE_PLAYING;
     1487    GST_DEBUG_OBJECT(m_pipeline.get(), "Checking if to send SELECT_STREAMS, m_waitingForStreamsSelectedEvent = %s, haveDifferentStreamIds = %s, m_currentState = %s... shouldSendSelectStreams = %s",
     1488        boolForPrinting(m_waitingForStreamsSelectedEvent), boolForPrinting(haveDifferentStreamIds), gst_element_state_get_name(m_currentState), boolForPrinting(shouldSendSelectStreams));
     1489    if (!shouldSendSelectStreams)
     1490        return;
     1491
     1492    GList* streams = nullptr;
     1493    if (!m_wantedVideoStreamId.isNull()) {
     1494        m_requestedVideoStreamId = m_wantedVideoStreamId;
     1495        streams = g_list_append(streams, g_strdup(m_wantedVideoStreamId.string().utf8().data()));
     1496    }
     1497    if (!m_wantedAudioStreamId.isNull()) {
     1498        m_requestedAudioStreamId = m_wantedAudioStreamId;
     1499        streams = g_list_append(streams, g_strdup(m_wantedAudioStreamId.string().utf8().data()));
     1500    }
     1501
     1502    if (!streams)
     1503        return;
     1504
     1505    m_waitingForStreamsSelectedEvent = true;
     1506    gst_element_send_event(m_pipeline.get(), gst_event_new_select_streams(streams));
     1507    g_list_free_full(streams, reinterpret_cast<GDestroyNotify>(g_free));
     1508}
     1509
     1510void MediaPlayerPrivateGStreamer::updateTracks(GRefPtr<GstStreamCollection>&& streamCollection)
     1511{
     1512    ASSERT(!m_isLegacyPlaybin);
     1513
    15161514    bool useMediaSource = isMediaSource();
    1517     unsigned length = gst_stream_collection_get_size(m_streamCollection.get());
     1515    unsigned length = gst_stream_collection_get_size(streamCollection.get());
     1516    GST_DEBUG_OBJECT(pipeline(), "Processing a stream collection with %u streams", length);
    15181517
    15191518    bool oldHasAudio = m_hasAudio;
    15201519    bool oldHasVideo = m_hasVideo;
     1520
    15211521    // New stream collections override previous ones.
    1522     clearTracks();
     1522    unsigned audioTrackIndex = 0;
     1523    unsigned videoTrackIndex = 0;
    15231524    unsigned textTrackIndex = 0;
    15241525    for (unsigned i = 0; i < length; i++) {
    1525         GRefPtr<GstStream> stream = gst_stream_collection_get_stream(m_streamCollection.get(), i);
     1526        GRefPtr<GstStream> stream = gst_stream_collection_get_stream(streamCollection.get(), i);
    15261527        String streamId(gst_stream_get_stream_id(stream.get()));
    15271528        GstStreamType type = gst_stream_get_stream_type(stream.get());
    15281529
    15291530        GST_DEBUG_OBJECT(pipeline(), "Inspecting %s track with ID %s", gst_stream_type_get_name(type), streamId.utf8().data());
     1531        if ((type & GST_STREAM_TYPE_AUDIO && m_audioTracks.contains(streamId)) || (type & GST_STREAM_TYPE_VIDEO && m_videoTracks.contains(streamId))
     1532            || (type & GST_STREAM_TYPE_TEXT && m_textTracks.contains(streamId)))
     1533        {
     1534            GST_DEBUG_OBJECT(pipeline(), "%s track with ID %s already exists, skipping", gst_stream_type_get_name(type), streamId.utf8().data());
     1535            continue;
     1536        }
     1537
    15301538        if (type & GST_STREAM_TYPE_AUDIO)
    15311539            CREATE_TRACK(audio, Audio);
    … …  
    15511559}
    15521560
    1553 void MediaPlayerPrivateGStreamer::clearTracks()
    1554 {
    1555 #if ENABLE(VIDEO_TRACK)
    1556     CLEAR_TRACKS(m_audioTracks, m_player->removeAudioTrack);
    1557     CLEAR_TRACKS(m_videoTracks, m_player->removeVideoTrack);
    1558     CLEAR_TRACKS(m_textTracks, m_player->removeTextTrack);
    1559 #endif // ENABLE(VIDEO_TRACK)
    1560 }
    1561 
    15621561void MediaPlayerPrivateGStreamer::videoChangedCallback(MediaPlayerPrivateGStreamer* player)
    15631562{
    … …  
    15891588        GRefPtr<GstStreamCollection> collection;
    15901589        gst_message_parse_stream_collection(message, &collection.outPtr());
     1590#ifndef GST_DISABLE_DEBUG
     1591        GST_DEBUG_OBJECT(pipeline(), "Received STREAM_COLLECTION message with upstream id \"%s\" defining the following streams:", gst_stream_collection_get_upstream_id(collection.get()));
     1592        unsigned numStreams = gst_stream_collection_get_size(collection.get());
     1593        for (unsigned i = 0; i < numStreams; i++) {
     1594            GstStream* stream = gst_stream_collection_get_stream(collection.get(), i);
     1595            GST_DEBUG_OBJECT(pipeline(), "#%u %s %s", i, gst_stream_type_get_name(gst_stream_get_stream_type(stream)), gst_stream_get_stream_id(stream));
     1596        }
     1597#endif
    15911598
    15921599        if (collection) {
    1593             m_streamCollection.swap(collection);
    1594             m_notifier->notify(MainThreadNotification::StreamCollectionChanged, [this] {
    1595                 this->updateTracks();
     1600            m_notifier->notify(MainThreadNotification::StreamCollectionChanged, [this, collection = WTFMove(collection)]() mutable {
     1601                this->updateTracks(WTFMove(collection));
    15961602            });
    15971603        }
     1604#ifndef GST_DISABLE_DEBUG
     1605    } else if (GST_MESSAGE_TYPE(message) == GST_MESSAGE_STREAMS_SELECTED && !m_isLegacyPlaybin) {
     1606        GST_DEBUG_OBJECT(pipeline(), "Received STREAMS_SELECTED message selecting the following streams:");
     1607        unsigned numStreams = gst_message_streams_selected_get_size(message);
     1608        for (unsigned i = 0; i < numStreams; i++) {
     1609            GstStream* stream = gst_message_streams_selected_get_stream(message, i);
     1610            GST_DEBUG_OBJECT(pipeline(), "#%u %s %s", i, gst_stream_type_get_name(gst_stream_get_stream_type(stream)), gst_stream_get_stream_id(stream));
     1611        }
     1612#endif
    15981613    }
    15991614
    … …  
    18911906            gst_element_state_get_name(currentState), '_', gst_element_state_get_name(newState)).utf8();
    18921907        GST_DEBUG_BIN_TO_DOT_FILE_WITH_TS(GST_BIN(m_pipeline.get()), GST_DEBUG_GRAPH_SHOW_ALL, dotFileName.data());
     1908
     1909        if (!m_isLegacyPlaybin && currentState == GST_STATE_PAUSED && newState == GST_STATE_PLAYING)
     1910            playbin3SendSelectStreamsIfAppropriate();
    18931911
    18941912        break;
    … …  
    20412059    }
    20422060    case GST_MESSAGE_STREAMS_SELECTED: {
    2043         GRefPtr<GstStreamCollection> collection;
    2044         gst_message_parse_streams_selected(message, &collection.outPtr());
    2045 
    2046         if (!collection)
     2061        if (m_isLegacyPlaybin)
    20472062            break;
    20482063
    2049         m_streamCollection.swap(collection);
    2050         m_currentAudioStreamId = "";
    2051         m_currentVideoStreamId = "";
    2052         m_currentTextStreamId = "";
    2053 
    2054         unsigned length = gst_message_streams_selected_get_size(message);
    2055         for (unsigned i = 0; i < length; i++) {
    2056             GRefPtr<GstStream> stream = gst_message_streams_selected_get_stream(message, i);
    2057             if (!stream)
    2058                 continue;
    2059 
    2060             GstStreamType type = gst_stream_get_stream_type(stream.get());
    2061             String streamId(gst_stream_get_stream_id(stream.get()));
    2062 
    2063             GST_DEBUG_OBJECT(pipeline(), "Selecting %s track with ID: %s", gst_stream_type_get_name(type), streamId.utf8().data());
    2064             // Playbin3 can send more than one selected stream of the same type
    2065             // but there's no priority or ordering system in place, so we assume
    2066             // the selected stream is the last one as reported by playbin3.
    2067             if (type & GST_STREAM_TYPE_AUDIO) {
    2068                 m_currentAudioStreamId = streamId;
    2069                 auto track = m_audioTracks.get(m_currentAudioStreamId);
    2070                 ASSERT(track);
    2071                 track->markAsActive();
    2072             } else if (type & GST_STREAM_TYPE_VIDEO) {
    2073                 m_currentVideoStreamId = streamId;
    2074                 auto track = m_videoTracks.get(m_currentVideoStreamId);
    2075                 ASSERT(track);
    2076                 track->markAsActive();
    2077             } else if (type & GST_STREAM_TYPE_TEXT)
    2078                 m_currentTextStreamId = streamId;
    2079             else
    2080                 GST_WARNING("Unknown stream type with stream-id %s", streamId.utf8().data());
    2081         }
     2064        GST_DEBUG_OBJECT(m_pipeline.get(), "Received STREAMS_SELECTED, setting m_waitingForStreamsSelectedEvent to false.");
     2065        m_waitingForStreamsSelectedEvent = false;
     2066
     2067        // Unfortunately, STREAMS_SELECTED messages from playbin3 are highly unreliable, often only including the audio
     2068        // stream or only the video stream when both are present and going to be played.
     2069        // Therefore, instead of reading the event data, we will just assume our previously requested selection was honored.
     2070        m_currentAudioStreamId = m_requestedAudioStreamId;
     2071        m_currentVideoStreamId = m_requestedVideoStreamId;
     2072
     2073        // It's possible the user made a track switch before the initial STREAMS_SELECED. Now it's a good moment to
     2074        // request it being attended. Note that it's not possible to send a SELECT_STREAMS before the first
     2075        // STREAMS_SELECTED message because at that point the pipeline is not compeletely constructed.
     2076        playbin3SendSelectStreamsIfAppropriate();
    20822077        break;
    20832078    }
    … …  
    26442639
    26452640    m_isEndReached = true;
     2641    // Now that playback has ended it's NOT a safe time to send a SELECT_STREAMS event. In fact, as of GStreamer 1.16,
     2642    // playbin3 will crash on a GStreamer assertion (combine->sinkpad being unexpectedly null) if we try. Instead, wait
     2643    // until we get the initial STREAMS_SELECTED message one more time.
     2644    m_waitingForStreamsSelectedEvent = true;
    26462645
    26472646    if (!m_player->isLooping()) {

  • trunk/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.h

    r258691 r261683  
    221221#endif
    222222
    223     void enableTrack(TrackPrivateBaseGStreamer::TrackType, unsigned index);
     223    void updateEnabledVideoTrack();
     224    void updateEnabledAudioTrack();
     225    void playbin3SendSelectStreamsIfAppropriate();
    224226
    225227    // Append pipeline interface
    … …  
    462464    void loadFull(const String& url, const String& pipelineName);
    463465
    464     void updateTracks();
    465     void clearTracks();
     466    void updateTracks(GRefPtr<GstStreamCollection>&&);
    466467
    467468#if ENABLE(ENCRYPTED_MEDIA)
    … …  
    509510    mutable Optional<Seconds> m_lastQueryTime;
    510511    bool m_isLegacyPlaybin;
    511     GRefPtr<GstStreamCollection> m_streamCollection;
    512512#if ENABLE(MEDIA_STREAM)
    513513    RefPtr<MediaStreamPrivate> m_streamPrivate;
    514514#endif
    515     String m_currentAudioStreamId;
    516     String m_currentVideoStreamId;
    517     String m_currentTextStreamId;
     515
     516    // playbin3 only:
     517    bool m_waitingForStreamsSelectedEvent { true };
     518    AtomString m_currentAudioStreamId; // Currently playing.
     519    AtomString m_currentVideoStreamId;
     520    AtomString m_wantedAudioStreamId; // Set in JavaScript.
     521    AtomString m_wantedVideoStreamId;
     522    AtomString m_requestedAudioStreamId; // Expected in the next STREAMS_SELECTED message.
     523    AtomString m_requestedVideoStreamId;
     524
    518525#if ENABLE(WEB_AUDIO)
    519526    std::unique_ptr<AudioSourceProviderGStreamer> m_audioSourceProvider;

  • trunk/Source/WebCore/platform/graphics/gstreamer/TrackPrivateBaseGStreamer.cpp

    r246677 r261683  
    5252    ASSERT(m_pad);
    5353
    54     g_signal_connect_swapped(m_pad.get(), "notify::active", G_CALLBACK(activeChangedCallback), this);
    5554    g_signal_connect_swapped(m_pad.get(), "notify::tags", G_CALLBACK(tagsChangedCallback), this);
    5655
    … …  
    9392}
    9493
    95 void TrackPrivateBaseGStreamer::activeChangedCallback(TrackPrivateBaseGStreamer* track)
    96 {
    97     track->m_notifier->notify(MainThreadNotification::ActiveChanged, [track] { track->notifyTrackOfActiveChanged(); });
    98 }
    99 
    10094void TrackPrivateBaseGStreamer::tagsChangedCallback(TrackPrivateBaseGStreamer* track)
    10195{
    … …  
    124118
    125119    m_notifier->notify(MainThreadNotification::TagsChanged, [this] { notifyTrackOfTagsChanged(); });
    126 }
    127 
    128 void TrackPrivateBaseGStreamer::notifyTrackOfActiveChanged()
    129 {
    130     if (!m_pad)
    131         return;
    132 
    133     gboolean active = false;
    134     if (g_object_class_find_property(G_OBJECT_GET_CLASS(m_pad.get()), "active"))
    135         g_object_get(m_pad.get(), "active", &active, nullptr);
    136 
    137     setActive(active);
    138120}
    139121

  • trunk/Source/WebCore/platform/graphics/gstreamer/TrackPrivateBaseGStreamer.h

    r246677 r261683  
    6666    TrackPrivateBaseGStreamer(TrackPrivateBase* owner, gint index, GRefPtr<GstStream>);
    6767
    68     void notifyTrackOfActiveChanged();
    6968    void notifyTrackOfTagsChanged();
    7069
    7170    enum MainThreadNotification {
    72         ActiveChanged = 1 << 0,
    7371        TagsChanged = 1 << 1,
    7472        NewSample = 1 << 2,

  • trunk/Source/WebCore/platform/graphics/gstreamer/VideoTrackPrivateGStreamer.cpp

    r246677 r261683  
    4141    // FIXME: Get a real ID from the tkhd atom.
    4242    m_id = "V" + String::number(index);
    43     notifyTrackOfActiveChanged();
    4443}
    4544
    … …  
    5756
    5857    m_id = gst_stream_get_stream_id(stream.get());
    59     if (gst_stream_get_stream_flags(stream.get()) & GST_STREAM_FLAG_SELECT)
    60         markAsActive();
    61     notifyTrackOfActiveChanged();
    6258}
    6359
    … …  
    7672}
    7773
    78 void VideoTrackPrivateGStreamer::markAsActive()
    79 {
    80     VideoTrackPrivate::setSelected(true);
    81 }
    82 
    8374void VideoTrackPrivateGStreamer::setSelected(bool selected)
    8475{
    … …  
    8778    VideoTrackPrivate::setSelected(selected);
    8879
    89     if (selected && m_player)
    90         m_player->enableTrack(TrackPrivateBaseGStreamer::TrackType::Video, m_index);
     80    if (m_player)
     81        m_player->updateEnabledVideoTrack();
    9182}
    9283

  • trunk/Source/WebCore/platform/graphics/gstreamer/VideoTrackPrivateGStreamer.h

    r246677 r261683  
    5454    void disconnect() override;
    5555
    56     void markAsActive();
    5756    void setSelected(bool) override;
    5857    void setActive(bool enabled) override { setSelected(enabled); }
Note: See TracChangeset for help on using the changeset viewer.