⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 280584 in webkit


Ignore:
Timestamp:
Aug 2, 2021, 9:21:47 PM (5 years ago)
Author:
Jean-Yves Avenard
Message:

Source/WebCore:
[WebAudio] webm; properly trim frames according to the codec delay information
​https://bugs.webkit.org/show_bug.cgi?id=228140
rdar://problem/80883882

Reviewed by Eric Carlson.

Add frame-perfect decoding for Opus and Vorbis in WebM container.
Unfortunately, due to a vorbis decoder bug, it is not possible to test that
vorbis decoding returns the right number of frames.
Amend Opus test to ensure the right amount of frames are returned.

  • platform/audio/cocoa/AudioFileReaderCocoa.cpp:

(WebCore::AudioFileReader::demuxWebMData const):
(WebCore::passthroughInputDataCallback): AudioToolbox internal logging could overflow
if we used packet descriptions with offsets, so we take the most commonly used scenario
where we have a single packet to decode at a time.
(WebCore::AudioFileReader::decodeWebMData const):

  • platform/graphics/cocoa/AudioTrackPrivateWebM.cpp:

(WebCore::AudioTrackPrivateWebM::codecDelay const):
(WebCore::AudioTrackPrivateWebM::setDiscardPadding):
(WebCore::AudioTrackPrivateWebM::discardPadding const):

  • platform/graphics/cocoa/AudioTrackPrivateWebM.h:
  • platform/graphics/cocoa/SourceBufferParserWebM.cpp:

(WebCore::SourceBufferParserWebM::OnElementEnd): Fly-by, ensure m_didParseInitializationDataCallback
only ever accessed on client thread.
(WebCore::SourceBufferParserWebM::OnBlockGroupEnd):

  • platform/graphics/cocoa/SourceBufferParserWebM.h: Add trimming data callback.

Only mark WEBCORE_EXPORT the methods that need it.
(WebCore::SourceBufferParserWebM::appendData):
(WebCore::SourceBufferParserWebM::setDidParseTrimmingDataCallback):

  • html/HTMLMediaElement.cpp:

(WebCore::HTMLMediaElement::canPlayType const): Make canPlayType(webm/audio; codecs=vorbis)
returns maybe if feature flag set.

LayoutTests:
​https://bugs.webkit.org/show_bug.cgi?id=228140
rdar://problem/80883882

Reviewed by Eric Carlson.

​https://bugs.webkit.org/show_bug.cgi?id=228140

  • webaudio/decode-audio-data-webm-opus-expected.txt:
  • webaudio/decode-audio-data-webm-opus.html:
Location:
trunk
Files:
10 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r280582 r280584  
     12021-08-02  Jean-Yves Avenard  <jya@apple.com>
     2
     3        https://bugs.webkit.org/show_bug.cgi?id=228140
     4        rdar://problem/80883882
     5
     6        Reviewed by Eric Carlson.
     7
     8        https://bugs.webkit.org/show_bug.cgi?id=228140
     9
     10        * webaudio/decode-audio-data-webm-opus-expected.txt:
     11        * webaudio/decode-audio-data-webm-opus.html:
     12
    1132021-08-02  Chris Dumez  <cdumez@apple.com>
    214

  • trunk/LayoutTests/webaudio/decode-audio-data-webm-opus-expected.txt

    r280416 r280584  
    55
    66PASS Successfully decoded content
     7PASS Decoding returned the right number of frames.
    78PASS successfullyParsed is true
    89

  • trunk/LayoutTests/webaudio/decode-audio-data-webm-opus.html

    r280416 r280584  
    1111window.jsTestIsAsync = true;
    1212
    13 var context = new window.AudioContext();
     13var context = new window.AudioContext({ sampleRate: 48000 });
    1414var request = new XMLHttpRequest();
    1515request.open("GET", 'resources/media/opus.webm', true);
    … …  
    1919    context.decodeAudioData(request.response, (buffer) => {
    2020        testPassed("Successfully decoded content");
     21        // File is exactly 1-0.0065s long @ 48000Hz, so 47688 frames.
     22        if (buffer.length === 47688)
     23          testPassed("Decoding returned the right number of frames.");
     24        else
     25          testFailed("Decoding returned the wrong number of frames: " + buffer.length);
    2126        finishJSTest();
    2227      }, () => {

  • trunk/Source/WebCore/ChangeLog

    r280582 r280584  
     12021-08-02  Jean-Yves Avenard  <jya@apple.com>
     2
     3        [WebAudio] webm; properly trim frames according to the codec delay information
     4        https://bugs.webkit.org/show_bug.cgi?id=228140
     5        rdar://problem/80883882
     6
     7        Reviewed by Eric Carlson.
     8
     9        Add frame-perfect decoding for Opus and Vorbis in WebM container.
     10        Unfortunately, due to a vorbis decoder bug, it is not possible to test that
     11        vorbis decoding returns the right number of frames.
     12        Amend Opus test to ensure the right amount of frames are returned.
     13
     14        * platform/audio/cocoa/AudioFileReaderCocoa.cpp:
     15        (WebCore::AudioFileReader::demuxWebMData const):
     16        (WebCore::passthroughInputDataCallback): AudioToolbox internal logging could overflow
     17        if we used packet descriptions with offsets, so we take the most commonly used scenario
     18        where we have a single packet to decode at a time.
     19        (WebCore::AudioFileReader::decodeWebMData const):
     20        * platform/graphics/cocoa/AudioTrackPrivateWebM.cpp:
     21        (WebCore::AudioTrackPrivateWebM::codecDelay const):
     22        (WebCore::AudioTrackPrivateWebM::setDiscardPadding):
     23        (WebCore::AudioTrackPrivateWebM::discardPadding const):
     24        * platform/graphics/cocoa/AudioTrackPrivateWebM.h:
     25        * platform/graphics/cocoa/SourceBufferParserWebM.cpp:
     26        (WebCore::SourceBufferParserWebM::OnElementEnd): Fly-by, ensure m_didParseInitializationDataCallback
     27        only ever accessed on client thread.
     28        (WebCore::SourceBufferParserWebM::OnBlockGroupEnd):
     29        * platform/graphics/cocoa/SourceBufferParserWebM.h: Add trimming data callback.
     30        Only mark WEBCORE_EXPORT the methods that need it.
     31        (WebCore::SourceBufferParserWebM::appendData):
     32        (WebCore::SourceBufferParserWebM::setDidParseTrimmingDataCallback):
     33        * html/HTMLMediaElement.cpp:
     34        (WebCore::HTMLMediaElement::canPlayType const): Make canPlayType(webm/audio; codecs=vorbis)
     35        returns maybe if feature flag set.
     36
     37
    1382021-08-02  Chris Dumez  <cdumez@apple.com>
    239

  • trunk/Source/WebCore/html/HTMLMediaElement.cpp

    r280531 r280584  
    10711071    // so that sites relying on this test to determine if webaudio use of opus or vorbis won't error.
    10721072    auto codecs = contentType.codecs();
    1073     if (support == MediaPlayer::SupportsType::IsSupported && ((codecs.contains("opus") && !webMWebAudioEnabled()) || codecs.contains("vorbis")))
     1073    if (support == MediaPlayer::SupportsType::IsSupported && ((codecs.contains("opus") || codecs.contains("vorbis")) && !webMWebAudioEnabled()))
    10741074        support = MediaPlayer::SupportsType::IsNotSupported;
    10751075#endif

  • trunk/Source/WebCore/platform/audio/cocoa/AudioFileReaderCocoa.cpp

    r280416 r280584  
    3636#include "AudioFileReader.h"
    3737#include "AudioSampleDataSource.h"
    38 #include "AudioTrackPrivate.h"
     38#include "AudioTrackPrivateWebM.h"
    3939#include "FloatConversion.h"
    4040#include "InbandTextTrackPrivate.h"
    … …  
    131131public:
    132132#if ENABLE(MEDIA_SOURCE)
    133     SourceBufferParserWebM::InitializationSegment m_initSegment;
     133    Ref<AudioTrackPrivateWebM> m_track;
    134134#endif
    135135    MediaTime m_duration;
    … …  
    185185    std::optional<uint64_t> audioTrackId;
    186186    MediaTime duration;
    187     SourceBufferParserWebM::InitializationSegment initSegment;
     187    RefPtr<AudioTrackPrivateWebM> track;
    188188    Vector<Ref<MediaSampleAVFObjC>> samples;
    189189    parser->setDidEncounterErrorDuringParsingCallback([&](uint64_t) {
    … …  
    195195                duration = init.duration;
    196196                audioTrackId = audioTrack.track->trackUID();
    197                 initSegment = WTFMove(init);
     197                track = static_pointer_cast<AudioTrackPrivateWebM>(audioTrack.track);
    198198                return;
    199199            }
    … …  
    208208        function();
    209209    });
     210    parser->setDidParseTrimmingDataCallback([&](uint64_t trackID, const MediaTime& discardPadding) {
     211        if (!audioTrackId || !track || trackID != *audioTrackId)
     212            return;
     213        track->setDiscardPadding(discardPadding);
     214    });
    210215    SourceBufferParser::Segment segment({ data, dataSize });
    211216    parser->appendData(WTFMove(segment));
    212     if (!audioTrackId)
     217    if (!track)
    213218        return nullptr;
    214219    parser->flushPendingAudioBuffers();
    215     return makeUnique<AudioFileReaderWebMData>(AudioFileReaderWebMData { WTFMove(initSegment), WTFMove(duration), WTFMove(samples) });
     220    return makeUnique<AudioFileReaderWebMData>(AudioFileReaderWebMData { track.releaseNonNull(), WTFMove(duration), WTFMove(samples) });
    216221}
    217222
    218223struct PassthroughUserData {
    219     UInt32 m_channels;
    220     UInt32 m_dataSize;
    221     const void* m_data;
     224    const UInt32 m_channels;
     225    const UInt32 m_dataSize;
     226    const char* m_data;
     227    const bool m_eos;
     228    const Vector<AudioStreamPacketDescription>& m_packets;
    222229    UInt32 m_index;
    223     Vector<AudioStreamPacketDescription>& m_packets;
     230    AudioStreamPacketDescription m_packet;
    224231};
    225232
    … …  
    237244    if (userData->m_index == userData->m_packets.size()) {
    238245        *numDataPackets = 0;
    239         return kNoMoreDataErr;
     246        return userData->m_eos ? noErr : kNoMoreDataErr;
     247    }
     248
     249    if (userData->m_index >= userData->m_packets.size()) {
     250        *numDataPackets = 0;
     251        return kAudioConverterErr_RequiresPacketDescriptionsError;
    240252    }
    241253
    242254    if (packetDesc) {
    243         if (userData->m_index >= userData->m_packets.size()) {
    244             *numDataPackets = 0;
    245             return kAudioConverterErr_RequiresPacketDescriptionsError;
    246         }
    247         *packetDesc = &userData->m_packets[userData->m_index];
     255        userData->m_packet = userData->m_packets[userData->m_index];
     256        userData->m_packet.mStartOffset = 0;
     257        *packetDesc = &userData->m_packet;
    248258    }
    249259
    250260    data->mBuffers[0].mNumberChannels = userData->m_channels;
    251     data->mBuffers[0].mDataByteSize = userData->m_dataSize;
    252     data->mBuffers[0].mData = const_cast<void*>(userData->m_data);
    253 
     261    data->mBuffers[0].mDataByteSize = userData->m_packets[userData->m_index].mDataByteSize;
     262    data->mBuffers[0].mData = const_cast<char*>(userData->m_data + userData->m_packets[userData->m_index].mStartOffset);
     263
     264    // Sanity check
     265    if (static_cast<char*>(data->mBuffers[0].mData) + data->mBuffers[0].mDataByteSize > userData->m_data + userData->m_dataSize) {
     266        RELEASE_LOG_FAULT(WebAudio, "Nonsensical data structure, aborting");
     267        return kAudioConverterErr_UnspecifiedError;
     268    }
    254269    *numDataPackets = 1;
    255270    userData->m_index++;
    … …  
    304319        PAL::AudioConverterSetProperty(converter, kAudioConverterDecompressionMagicCookie, magicCookieSize, magicCookie);
    305320
     321    AudioConverterPrimeInfo primeInfo = { UInt32(m_webmData->m_track->codecDelay().value_or(MediaTime()).toDouble() * outFormat.mSampleRate), 0 };
     322    INFO_LOG(LOGIDENTIFIER, "Will drop %u leading frames out of %llu", primeInfo.leadingFrames, numberOfFrames);
     323    PAL::AudioConverterSetProperty(converter, kAudioConverterPrimeInfo, sizeof(primeInfo), &primeInfo);
     324    UInt32 primeMethod = kConverterPrimeMethod_None;
     325    PAL::AudioConverterSetProperty(converter, kAudioConverterPrimeMethod, sizeof(primeMethod), &primeMethod);
     326
    306327    AudioBufferListHolder decodedBufferList(inFormat.mChannelsPerFrame);
    307328    if (!decodedBufferList) {
    … …  
    312333    size_t decodedFrames = 0;
    313334    OSStatus status;
    314     for (auto& sample : m_webmData->m_samples) {
     335    for (size_t i = 0; i < m_webmData->m_samples.size(); i++) {
     336        auto& sample = m_webmData->m_samples[i];
    315337        CMSampleBufferRef sampleBuffer = sample->sampleBuffer();
    316338        auto buffer = PAL::CMSampleBufferGetDataBuffer(sampleBuffer);
    … …  
    332354            return { };
    333355
    334         PassthroughUserData userData = { inFormat.mChannelsPerFrame, UInt32(srcSize), srcData, 0, descriptions };
     356        PassthroughUserData userData = { inFormat.mChannelsPerFrame, UInt32(srcSize), srcData, i == m_webmData->m_samples.size() - 1, descriptions, 0, { } };
    335357
    336358        do {
    … …  
    354376            }
    355377            decodedFrames += numFrames;
    356         } while (status != kNoMoreDataErr);
    357     }
    358     return decodedFrames;
     378        } while (status != kNoMoreDataErr && status != noErr);
     379    }
     380    size_t paddingFrames = m_webmData->m_track->discardPadding().value_or(MediaTime()).toDouble() * outFormat.mSampleRate;
     381    if (decodedFrames > paddingFrames)
     382        return decodedFrames - paddingFrames;
     383    return 0;
    359384}
    360385#endif

  • trunk/Source/WebCore/platform/graphics/cocoa/AudioTrackPrivateWebM.cpp

    r278253 r280584  
    8787}
    8888
     89std::optional<MediaTime> AudioTrackPrivateWebM::codecDelay() const
     90{
     91    if (!m_track.codec_delay.is_present())
     92        return { };
     93    constexpr uint32_t k_us_in_seconds = 1000000000;
     94    return MediaTime(m_track.codec_delay.value(), k_us_in_seconds);
     95}
     96
     97void AudioTrackPrivateWebM::setDiscardPadding(const MediaTime& discardPadding)
     98{
     99    m_discardPadding = discardPadding;
     100}
     101
     102std::optional<MediaTime> AudioTrackPrivateWebM::discardPadding() const
     103{
     104    if (m_discardPadding.isInvalid() || m_discardPadding < MediaTime())
     105        return { };
     106    return m_discardPadding;
     107}
     108
    89109}
    90110

  • trunk/Source/WebCore/platform/graphics/cocoa/AudioTrackPrivateWebM.h

    r278253 r280584  
    4444    std::optional<uint64_t> trackUID() const final;
    4545    std::optional<bool> defaultEnabled() const final;
     46    std::optional<MediaTime> codecDelay() const;
     47    void setDiscardPadding(const MediaTime&);
     48    std::optional<MediaTime> discardPadding() const;
    4649
    4750private:
    4851    AudioTrackPrivateWebM(webm::TrackEntry&&);
    4952    webm::TrackEntry m_track;
     53    MediaTime m_discardPadding { MediaTime::invalidTime() };
    5054    mutable AtomString m_trackID;
    5155    mutable AtomString m_label;

  • trunk/Source/WebCore/platform/graphics/cocoa/SourceBufferParserWebM.cpp

    r279904 r280584  
    265265constexpr CMVideoCodecType kCMVideoCodecType_VP9 { 'vp09' };
    266266
     267constexpr uint32_t k_us_in_seconds = 1000000000;
     268
    267269static bool isWebmParserAvailable()
    268270{
    … …  
    780782        }
    781783
    782         if (m_initializationSegmentEncountered && m_didParseInitializationDataCallback) {
     784        if (m_initializationSegmentEncountered) {
    783785            m_callOnClientThreadCallback([this, protectedThis = makeRef(*this), initializationSegment = WTFMove(*m_initializationSegment)]() mutable {
    784                 m_didParseInitializationDataCallback(WTFMove(initializationSegment));
     786                if (m_didParseInitializationDataCallback)
     787                    m_didParseInitializationDataCallback(WTFMove(initializationSegment));
    785788            });
    786789        }
    … …  
    848851
    849852    auto timecodeScale = info.timecode_scale.is_present() ? info.timecode_scale.value() : 1000000;
    850     m_timescale = 1000000000 / timecodeScale;
     853    m_timescale = k_us_in_seconds / timecodeScale;
    851854    m_initializationSegment->duration = info.duration.is_present() ? MediaTime(info.duration.value(), m_timescale) : MediaTime::indefiniteTime();
    852855
    … …  
    10241027{
    10251028    UNUSED_PARAM(metadata);
    1026     UNUSED_PARAM(blockGroup);
    10271029    INFO_LOG_IF_POSSIBLE(LOGIDENTIFIER);
     1030    if (blockGroup.block.is_present() && blockGroup.discard_padding.is_present()) {
     1031        auto trackNumber = blockGroup.block.value().track_number;
     1032        auto* trackData = trackDataForTrackNumber(trackNumber);
     1033        if (!trackData) {
     1034            ERROR_LOG_IF_POSSIBLE(LOGIDENTIFIER, "Ignoring unknown track number ", trackNumber);
     1035            return Status(Status::kOkCompleted);
     1036        }
     1037        if (trackData->track().track_uid.is_present() && blockGroup.discard_padding.value() > 0) {
     1038            m_callOnClientThreadCallback([this, protectedThis = makeRef(*this), trackID = trackData->track().track_uid.value(), padding = MediaTime(blockGroup.discard_padding.value(), k_us_in_seconds)]() {
     1039                if (m_didParseTrimmingDataCallback)
     1040                    m_didParseTrimmingDataCallback(trackID, padding);
     1041            });
     1042        }
     1043    }
    10281044    return Status(Status::kOkCompleted);
    10291045}
    … …  
    12131229    uint64_t duration = 0;
    12141230    if (track.default_duration.is_present())
    1215         duration = track.default_duration.value() * presentationTime.timescale / 1000000000;
     1231        duration = track.default_duration.value() * presentationTime.timescale / k_us_in_seconds;
    12161232
    12171233    CMSampleBufferRef rawSampleBuffer = nullptr;

  • trunk/Source/WebCore/platform/graphics/cocoa/SourceBufferParserWebM.h

    r279904 r280584  
    5656class MediaSampleAVFObjC;
    5757
    58 class WEBCORE_EXPORT SourceBufferParserWebM : public SourceBufferParser, private webm::Callback {
     58class SourceBufferParserWebM : public SourceBufferParser, private webm::Callback {
    5959    WTF_MAKE_FAST_ALLOCATED;
    6060public:
    … …  
    6464    static MediaPlayerEnums::SupportsType isContentTypeSupported(const ContentType&);
    6565    static const HashSet<String, ASCIICaseInsensitiveHash>& webmMIMETypes();
    66     static RefPtr<SourceBufferParserWebM> create(const ContentType&);
     66    WEBCORE_EXPORT static RefPtr<SourceBufferParserWebM> create(const ContentType&);
    6767
    6868    SourceBufferParserWebM();
    … …  
    7474
    7575    Type type() const { return Type::WebM; }
    76     void appendData(Segment&&, CompletionHandler<void()>&& = [] { }, AppendFlags = AppendFlags::None) final;
     76    WEBCORE_EXPORT void appendData(Segment&&, CompletionHandler<void()>&& = [] { }, AppendFlags = AppendFlags::None) final;
    7777    void flushPendingMediaData() final;
    7878    void setShouldProvideMediaDataForTrackID(bool, uint64_t) final;
    … …  
    8484    void setMinimumAudioSampleDuration(float);
    8585   
    86     void setLogger(const WTF::Logger&, const void* identifier) final;
     86    WEBCORE_EXPORT void setLogger(const WTF::Logger&, const void* identifier) final;
    8787
    8888    void provideMediaData(RetainPtr<CMSampleBufferRef>, uint64_t, std::optional<size_t> byteRangeOffset);
     89    using DidParseTrimmingDataCallback = WTF::Function<void(uint64_t trackID, const MediaTime& discardPadding)>;
     90    void setDidParseTrimmingDataCallback(DidParseTrimmingDataCallback&& callback)
     91    {
     92        m_didParseTrimmingDataCallback = WTFMove(callback);
     93    }
    8994
    9095    enum class ErrorCode : int32_t {
    … …  
    288293    const void* m_logIdentifier { nullptr };
    289294    uint64_t m_nextChildIdentifier { 0 };
     295    DidParseTrimmingDataCallback m_didParseTrimmingDataCallback;
    290296};
    291297
Note: See TracChangeset for help on using the changeset viewer.