Changeset 291216 in webkit


Ignore:
Timestamp:
Mar 12, 2022 11:23:12 PM (4 months ago)
Author:
Jean-Yves Avenard
Message:

Safari produces scrambled output for some webm videos with vp8 codec.
​https://bugs.webkit.org/show_bug.cgi?id=236754
Source/ThirdParty/libwebrtc:

rdar://80869041

Reviewed by Eric Carlson.

  • Source/webrtc/sdk/WebKit/WebKitDecoderReceiver.cpp:

(webrtc::WebKitDecoderReceiver::decoderFailed): Tell CoreMedia if a frame
was silently dropped by the decoder.

Source/WebCore:

rdar://80869041

Reviewed by Eric Carlson.

The MediaFormatReader plugin and the MSE SourceBufferPrivate are using
a SampleMap to store all the media samples timing information: one sorted
by DTS order and the other in PTS order.
Those SampleMap use the sample's presentation time as unique key.
The VP8 codec can define hidden samples that are to be fed to the decoder
but will not decode into an actual image. Those samples are typically
packed together in the webm container in two or more consecutive blocks
with the same presentation time (similar behaviour can also be found in
webm where multiple frames may be stored with the same presentation
time).
When stored in the SampleMap, only the latest sample added would be kept
after overwriting the previous one with the same time.
To get around this issue, we pack all samples with the same presentation
time in a single MediaSamplesBlock so that they can be stored together in
the map without any losses.
Upon decoding, all those sub-samples will be retrieved and fed to the
decoder.

The CoreMedia MediaFormatReader backend however has a bug where it will
enter in an infinite if we return successive frames with the same
timestamp which will cause memory exhaustion and a crash.
To get around this, we make the grouped samples appear as discrete,
making each hidden sample have a duration of 1us followed by the
visible frames (which will see its duration shorten by 1us).

Tests: media/media-source/media-source-vp8-hiddenframes.html

media/media-vp8-hiddenframes.html

New tests had to be disabled due to bug 236755. We currently have no
way to guarantee which frame is currently displayed after either a seek
operation or reaching the end of playback.

  • platform/MediaSample.h:

(WebCore::MediaSamplesBlock::append):
(WebCore::MediaSamplesBlock::clear):

  • platform/graphics/cocoa/SourceBufferParserWebM.cpp:

(WebCore::WebMParser::parse):
(WebCore::WebMParser::OnFrame):
(WebCore::WebMParser::flushPendingVideoSamples):
(WebCore::WebMParser::VideoTrackData::resetCompletedFramesState):
(WebCore::WebMParser::VideoTrackData::consumeFrameData):
(WebCore::WebMParser::VideoTrackData::processPendingMediaSamples):
(WebCore::WebMParser::VideoTrackData::flushPendingSamples):
(WebCore::WebMParser::AudioTrackData::consumeFrameData):

  • platform/graphics/cocoa/SourceBufferParserWebM.h:

(WebCore::WebMParser::TrackData::consumeFrameData):
(WebCore::WebMParser::TrackData::resetCompletedFramesState):
(WebCore::WebMParser::TrackData::drainPendingSamples):

LayoutTests:

Reviewed by Eric Carlson.

VP8 files were generated such that alternative reference frames were used:
$ fmpeg -i dragon.webm -c:v libvpx -vf scale=320:-1 -auto-alt-ref 1 -arnr-maxframes 5 -arnr-strength 3 -pass 1 test-vp8-hiddenframes.webm
$ fmpeg -i dragon.webm -c:v libvpx -vf scale=320:-1 -auto-alt-ref 1 -arnr-maxframes 5 -arnr-strength 3 -pass 2 test-vp8-hiddenframes.webm

The command used to extract the last frame in png format was:
$ ffmpeg -sseof -3 -i test-vp8-hiddenframes.webm -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int -update 1 -q:v 1 test-vp8-hiddenframes.png

  • TestExpectations:
  • media/content/test-vp8-hiddenframes.png: Added.
  • media/content/test-vp8-hiddenframes.webm: Added.
  • media/media-source/media-source-vp8-hiddenframes-expected.html: Added.
  • media/media-source/media-source-vp8-hiddenframes.html: Added.
  • media/media-vp8-hiddenframes-expected.html: Added.
  • media/media-vp8-hiddenframes.html: Added.
  • media/utilities.js: Added.

(once):
(fetchWithXHR):
(loadSegment):

Location:
trunk
Files:
7 added
8 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r291198 r291216  
     12022-03-12  Jean-Yves Avenard  <jya@apple.com>
     2
     3        Safari produces scrambled output for some webm videos with vp8 codec.
     4        https://bugs.webkit.org/show_bug.cgi?id=236754
     5
     6        Reviewed by Eric Carlson.
     7
     8        VP8 files were generated such that alternative reference frames were used:
     9        $ fmpeg -i dragon.webm -c:v libvpx -vf scale=320:-1 -auto-alt-ref 1 -arnr-maxframes 5 -arnr-strength 3 -pass 1 test-vp8-hiddenframes.webm
     10        $ fmpeg -i dragon.webm -c:v libvpx -vf scale=320:-1 -auto-alt-ref 1 -arnr-maxframes 5 -arnr-strength 3 -pass 2 test-vp8-hiddenframes.webm
     11
     12        The command used to extract the last frame in png format was:
     13        $ ffmpeg -sseof -3 -i test-vp8-hiddenframes.webm -pred mixed -pix_fmt rgb24 -sws_flags +accurate_rnd+full_chroma_int -update 1 -q:v 1 test-vp8-hiddenframes.png
     14
     15        * TestExpectations:
     16        * media/content/test-vp8-hiddenframes.png: Added.
     17        * media/content/test-vp8-hiddenframes.webm: Added.
     18        * media/media-source/media-source-vp8-hiddenframes-expected.html: Added.
     19        * media/media-source/media-source-vp8-hiddenframes.html: Added.
     20        * media/media-vp8-hiddenframes-expected.html: Added.
     21        * media/media-vp8-hiddenframes.html: Added.
     22        * media/utilities.js: Added.
     23        (once):
     24        (fetchWithXHR):
     25        (loadSegment):
     26
    1272022-03-11  Nikolaos Mouchtaris  <nmouchtaris@apple.com>
    228

  • trunk/LayoutTests/TestExpectations

    r291036 r291216  
    6868fast/text/mac [ Skip ]
    6969scrollingcoordinator [ Skip ]
     70media/media-vp8-hiddenframes.html [ Skip ] # Requires MediaFormatReader and VP8 decoder
    7071media/ios [ Skip ]
    7172media/modern-media-controls/overflow-support [ Skip ]
    … …  
    47684769webkit.org/b/221973 media/media-webm-no-duration.html [ Skip ]
    47694770webkit.org/b/222493 media/media-source/media-source-webm-vp8-malformed-header.html [ Skip ]
     4771media/media-source/media-source-vp8-hiddenframes.html [ Skip ] # Requires VP8 decoder
    47704772
    47714773# WebXR - Missing modules.

  • trunk/Source/ThirdParty/libwebrtc/ChangeLog

    r291210 r291216  
     12022-03-12  Jean-Yves Avenard  <jya@apple.com>
     2
     3        Safari produces scrambled output for some webm videos with vp8 codec.
     4        https://bugs.webkit.org/show_bug.cgi?id=236754
     5        rdar://80869041
     6
     7        Reviewed by Eric Carlson.
     8
     9        * Source/webrtc/sdk/WebKit/WebKitDecoderReceiver.cpp:
     10        (webrtc::WebKitDecoderReceiver::decoderFailed): Tell CoreMedia if a frame
     11        was silently dropped by the decoder.
     12
    1132022-03-12  Tim Horton  <timothy_horton@apple.com>
    214

  • trunk/Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/WebKitDecoderReceiver.cpp

    r281868 r291216  
    152152        vtError = kVTVideoDecoderBadDataErr;
    153153
    154     VTDecoderSessionEmitDecodedFrame(m_session, m_currentFrame, vtError, 0, nullptr);
     154    VTDecoderSessionEmitDecodedFrame(m_session, m_currentFrame, vtError, vtError ? 0 : kVTDecodeInfo_FrameDropped, nullptr);
    155155    m_currentFrame = nullptr;
    156156

  • trunk/Source/WebCore/ChangeLog

    r291215 r291216  
     12022-03-12  Jean-Yves Avenard  <jya@apple.com>
     2
     3        Safari produces scrambled output for some webm videos with vp8 codec.
     4        https://bugs.webkit.org/show_bug.cgi?id=236754
     5        rdar://80869041
     6
     7        Reviewed by Eric Carlson.
     8
     9        The MediaFormatReader plugin and the MSE SourceBufferPrivate are using
     10        a SampleMap to store all the media samples timing information: one sorted
     11        by DTS order and the other in PTS order.
     12        Those SampleMap use the sample's presentation time as unique key.
     13        The VP8 codec can define hidden samples that are to be fed to the decoder
     14        but will not decode into an actual image. Those samples are typically
     15        packed together in the webm container in two or more consecutive blocks
     16        with the same presentation time (similar behaviour can also be found in
     17        webm where multiple frames may be stored with the same presentation
     18        time).
     19        When stored in the SampleMap, only the latest sample added would be kept
     20        after overwriting the previous one with the same time.
     21        To get around this issue, we pack all samples with the same presentation
     22        time in a single MediaSamplesBlock so that they can be stored together in
     23        the map without any losses.
     24        Upon decoding, all those sub-samples will be retrieved and fed to the
     25        decoder.
     26
     27        The CoreMedia MediaFormatReader backend however has a bug where it will
     28        enter in an infinite if we return successive frames with the same
     29        timestamp which will cause memory exhaustion and a crash.
     30        To get around this, we make the grouped samples appear as discrete,
     31        making each hidden sample have a duration of 1us followed by the
     32        visible frames (which will see its duration shorten by 1us).
     33
     34        Tests: media/media-source/media-source-vp8-hiddenframes.html
     35               media/media-vp8-hiddenframes.html
     36        New tests had to be disabled due to bug 236755. We currently have no
     37        way to guarantee which frame is currently displayed after either a seek
     38        operation or reaching the end of playback.
     39
     40        * platform/MediaSample.h:
     41        (WebCore::MediaSamplesBlock::append):
     42        (WebCore::MediaSamplesBlock::clear):
     43        * platform/graphics/cocoa/SourceBufferParserWebM.cpp:
     44        (WebCore::WebMParser::parse):
     45        (WebCore::WebMParser::OnFrame):
     46        (WebCore::WebMParser::flushPendingVideoSamples):
     47        (WebCore::WebMParser::VideoTrackData::resetCompletedFramesState):
     48        (WebCore::WebMParser::VideoTrackData::consumeFrameData):
     49        (WebCore::WebMParser::VideoTrackData::processPendingMediaSamples):
     50        (WebCore::WebMParser::VideoTrackData::flushPendingSamples):
     51        (WebCore::WebMParser::AudioTrackData::consumeFrameData):
     52        * platform/graphics/cocoa/SourceBufferParserWebM.h:
     53        (WebCore::WebMParser::TrackData::consumeFrameData):
     54        (WebCore::WebMParser::TrackData::resetCompletedFramesState):
     55        (WebCore::WebMParser::TrackData::drainPendingSamples):
     56
    1572022-03-12  Alan Bujtas  <zalan@apple.com>
    258

  • trunk/Source/WebCore/platform/MediaSample.h

    r291033 r291216  
    239239        MediaSample::SampleFlags flags;
    240240    };
     241    using SamplesVector = Vector<MediaSampleItem>;
    241242
    242243    void setInfo(RefPtr<const TrackInfo>&& info) { m_info = WTFMove(info); }
    … …  
    246247    TrackInfo::TrackType type() const { return m_info ? m_info->type() : TrackInfo::TrackType::Unknown; }
    247248    void append(MediaSampleItem&& item) { m_samples.append(WTFMove(item)); }
    248     void append(MediaSamplesBlock&& block) { m_samples.appendVector(std::exchange(block.m_samples, { })); }
     249    void append(MediaSamplesBlock&& block) { append(std::exchange(block.m_samples, { })); }
     250    void append(SamplesVector&& samples) { m_samples.appendVector(WTFMove(samples)); }
    249251    size_t size() const { return m_samples.size(); };
    250252    bool isEmpty() const { return m_samples.isEmpty(); }
    251253    void clear() { m_samples.clear(); }
    252     using SamplesVector = Vector<MediaSampleItem>;
    253254    SamplesVector takeSamples() { return std::exchange(m_samples, { }); }
    254255

  • trunk/Source/WebCore/platform/graphics/cocoa/SourceBufferParserWebM.cpp

    r291033 r291216  
    577577        if (m_status.ok() || m_status.code == Status::kEndOfFile || m_status.code == Status::kWouldBlock) {
    578578            m_reader->reclaimSegments();
     579
     580            // We always keep one sample queued in order to calculate the video sample's time, return it now.
     581            flushPendingVideoSamples();
    579582            return 0;
    580583        }
    … …  
    9971000    }
    9981001
    999     return trackData->consumeFrameData(*reader, metadata, bytesRemaining, MediaTime(block->timecode + m_currentTimecode, m_timescale), block->num_frames);
     1002    return trackData->consumeFrameData(*reader, metadata, bytesRemaining, MediaTime(block->timecode + m_currentTimecode, m_timescale));
    10001003}
    10011004
    … …  
    10471050}
    10481051
     1052void WebMParser::flushPendingVideoSamples()
     1053{
     1054    for (auto& track : m_tracks) {
     1055        if (track->trackType() == TrackInfo::TrackType::Video)
     1056            downcast<WebMParser::VideoTrackData>(track.get()).flushPendingSamples();
     1057    }
     1058}
     1059
    10491060void WebMParser::VideoTrackData::resetCompletedFramesState()
    10501061{
    1051     m_keyFrames.clear();
     1062    ASSERT(!m_pendingMediaSamples.size());
    10521063    TrackData::resetCompletedFramesState();
    10531064}
    10541065
    1055 webm::Status WebMParser::VideoTrackData::consumeFrameData(webm::Reader& reader, const FrameMetadata& metadata, uint64_t* bytesRemaining, const MediaTime& presentationTime, int)
     1066webm::Status WebMParser::VideoTrackData::consumeFrameData(webm::Reader& reader, const FrameMetadata& metadata, uint64_t* bytesRemaining, const MediaTime& presentationTime)
    10561067{
    10571068#if ENABLE(VP9)
    … …  
    10861097    }
    10871098
    1088     if (!m_completeMediaSamples.info())
    1089         m_completeMediaSamples.setInfo(formatDescription());
    1090     else if (formatDescription() && *formatDescription() != *m_completeMediaSamples.info())
    1091         drainPendingSamples();
    1092 
    1093     auto track = this->track();
    1094 
    1095     uint64_t duration = 0;
    1096     if (track.default_duration.is_present())
    1097         duration = track.default_duration.value() * presentationTime.timeScale() / k_us_in_seconds;
    1098 
    1099     m_completeMediaSamples.append({ presentationTime, presentationTime, MediaTime(duration, presentationTime.timeScale()), WTFMove(m_completeFrameData), isKey ? MediaSample::SampleFlags::IsSync : MediaSample::SampleFlags::None });
    1100 
    1101     drainPendingSamples();
     1099    processPendingMediaSamples(presentationTime);
     1100
     1101    m_pendingMediaSamples.append({ presentationTime, presentationTime, MediaTime::indefiniteTime(), WTFMove(m_completeFrameData), isKey ? MediaSample::SampleFlags::IsSync : MediaSample::SampleFlags::None });
    11021102
    11031103    ASSERT(!*bytesRemaining);
    … …  
    11111111}
    11121112
     1113void WebMParser::VideoTrackData::processPendingMediaSamples(const MediaTime& presentationTime)
     1114{
     1115    // WebM container doesn't contain information about duration; the end time of a frame is the start time of the next.
     1116    // Some frames however may have a duration of 0 which typically indicates that they should be decoded but not displayed.
     1117    // We group all the samples with the same presentation timestamp within the same final MediaSampleBlock.
     1118
     1119    if (!m_pendingMediaSamples.size())
     1120        return;
     1121    auto& lastSample = m_pendingMediaSamples.last();
     1122    lastSample.duration = presentationTime - lastSample.presentationTime;
     1123    if (presentationTime == lastSample.presentationTime)
     1124        return;
     1125
     1126    MediaTime timeOffset;
     1127    MediaTime durationOffset;
     1128    while (m_pendingMediaSamples.size()) {
     1129        auto sample = m_pendingMediaSamples.takeFirst();
     1130        if (timeOffset) {
     1131            sample.presentationTime += timeOffset;
     1132            sample.decodeTime += timeOffset;
     1133            auto usableOffset = std::min(durationOffset, sample.duration);
     1134            sample.duration -= usableOffset;
     1135            durationOffset -= usableOffset;
     1136        }
     1137        // The MediaFormatReader is unable to deal with samples having a duration of 0.
     1138        // We instead set those samples to have a 1us duration and shift the presentation/decode time
     1139        // of the following samples in the block by the same offset.
     1140        if (!sample.duration) {
     1141            sample.duration = MediaTime(1, k_us_in_seconds);
     1142            timeOffset += sample.duration;
     1143            durationOffset += sample.duration;
     1144        }
     1145        m_processedMediaSamples.append(WTFMove(sample));
     1146    }
     1147    m_lastDuration = m_processedMediaSamples.last().duration;
     1148    m_lastPresentationTime = presentationTime;
     1149    if (!m_processedMediaSamples.info())
     1150        m_processedMediaSamples.setInfo(formatDescription());
     1151    drainPendingSamples();
     1152}
     1153
     1154void WebMParser::VideoTrackData::flushPendingSamples()
     1155{
     1156    // We haven't been able to calculate the duration of the last sample as none will follow.
     1157    // We set its duration to the track's default duration, or if not known the time of the last sample processed.
     1158    if (!m_pendingMediaSamples.size())
     1159        return;
     1160    ASSERT(m_lastPresentationTime);
     1161    auto track = this->track();
     1162
     1163    MediaTime duration;
     1164    if (track.default_duration.is_present())
     1165        duration = MediaTime(track.default_duration.value() * m_lastPresentationTime->timeScale() / k_us_in_seconds, m_lastPresentationTime->timeScale());
     1166    else if (m_lastDuration)
     1167        duration = *m_lastDuration;
     1168    processPendingMediaSamples(*m_lastPresentationTime + duration);
     1169    m_lastPresentationTime.reset();
     1170    m_lastDuration.reset();
     1171}
     1172
    11131173void WebMParser::AudioTrackData::resetCompletedFramesState()
    11141174{
    … …  
    11171177}
    11181178
    1119 webm::Status WebMParser::AudioTrackData::consumeFrameData(webm::Reader& reader, const FrameMetadata& metadata, uint64_t* bytesRemaining, const MediaTime& presentationTime, int)
     1179webm::Status WebMParser::AudioTrackData::consumeFrameData(webm::Reader& reader, const FrameMetadata& metadata, uint64_t* bytesRemaining, const MediaTime& presentationTime)
    11201180{
    11211181    auto status = readFrameData(reader, metadata, bytesRemaining);
    … …  
    11801240    }
    11811241
    1182     if (!m_completeMediaSamples.info())
    1183         m_completeMediaSamples.setInfo(formatDescription());
    1184     else if (formatDescription() && *formatDescription() != *m_completeMediaSamples.info())
     1242    if (!m_processedMediaSamples.info())
     1243        m_processedMediaSamples.setInfo(formatDescription());
     1244    else if (formatDescription() && *formatDescription() != *m_processedMediaSamples.info())
    11851245        drainPendingSamples();
    11861246
    1187     m_completeMediaSamples.append({ presentationTime, MediaTime::invalidTime(), m_packetDuration, WTFMove(m_completeFrameData), MediaSample::SampleFlags::IsSync });
     1247    m_processedMediaSamples.append({ presentationTime, MediaTime::invalidTime(), m_packetDuration, WTFMove(m_completeFrameData), MediaSample::SampleFlags::IsSync });
    11881248
    11891249    drainPendingSamples();

  • trunk/Source/WebCore/platform/graphics/cocoa/SourceBufferParserWebM.h

    r291033 r291216  
    3838#include <webm/status.h>
    3939#include <webm/vp9_header_parser.h>
     40#include <wtf/Deque.h>
    4041#include <wtf/MediaTime.h>
    4142#include <wtf/UniqueRef.h>
    … …  
    147148        WebMParser& parser() const { return m_parser; }
    148149
    149         virtual webm::Status consumeFrameData(webm::Reader&, const webm::FrameMetadata&, uint64_t*, const MediaTime&, int)
     150        virtual webm::Status consumeFrameData(webm::Reader&, const webm::FrameMetadata&, uint64_t*, const MediaTime&)
    150151        {
    151152            ASSERT_NOT_REACHED();
    … …  
    156157        {
    157158            m_completeBlockBuffer = nullptr;
    158             m_completeMediaSamples = { };
     159            m_processedMediaSamples = { };
    159160        }
    160161
    … …  
    169170        void drainPendingSamples()
    170171        {
    171             if (!m_completeMediaSamples.size())
     172            if (!m_processedMediaSamples.size())
    172173                return;
    173             m_parser.provideMediaData(WTFMove(m_completeMediaSamples));
     174            m_parser.provideMediaData(WTFMove(m_processedMediaSamples));
    174175            resetCompletedFramesState();
    175176        }
    … …  
    178179        RefPtr<SharedBuffer> contiguousCompleteBlockBuffer(size_t offset, size_t length) const;
    179180        webm::Status readFrameData(webm::Reader&, const webm::FrameMetadata&, uint64_t* bytesRemaining);
    180         MediaSamplesBlock m_completeMediaSamples;
     181        MediaSamplesBlock m_processedMediaSamples;
    181182        bool m_useByteRange { false };
    182183        MediaSamplesBlock::MediaSampleDataType m_completeFrameData;
    … …  
    207208        }
    208209
     210        void flushPendingSamples();
     211
    209212    private:
    210213        const char* logClassName() const { return "VideoTrackData"; }
    211         webm::Status consumeFrameData(webm::Reader&, const webm::FrameMetadata&, uint64_t*, const MediaTime&, int) final;
     214        webm::Status consumeFrameData(webm::Reader&, const webm::FrameMetadata&, uint64_t*, const MediaTime&) final;
    212215        void resetCompletedFramesState() final;
     216        void processPendingMediaSamples(const MediaTime&);
     217        WTF::Deque<MediaSamplesBlock::MediaSampleItem> m_pendingMediaSamples;
     218        std::optional<MediaTime> m_lastDuration;
     219        std::optional<MediaTime> m_lastPresentationTime;
    213220
    214221#if ENABLE(VP9)
    215222        vp9_parser::Vp9HeaderParser m_headerParser;
    216         Vector<bool> m_keyFrames;
    217223#endif
    218224    };
    … …  
    231237
    232238    private:
    233         webm::Status consumeFrameData(webm::Reader&, const webm::FrameMetadata&, uint64_t*, const MediaTime&, int) final;
     239        webm::Status consumeFrameData(webm::Reader&, const webm::FrameMetadata&, uint64_t*, const MediaTime&) final;
    234240        void resetCompletedFramesState() final;
    235241        const char* logClassName() const { return "AudioTrackData"; }
    … …  
    245251    static bool isSupportedVideoCodec(StringView);
    246252    static bool isSupportedAudioCodec(StringView);
     253    void flushPendingVideoSamples();
    247254
    248255    // webm::Callback
Note: See TracChangeset for help on using the changeset viewer.