Changeset 280416 in webkit


Ignore:
Timestamp:
Jul 28, 2021 9:18:16 PM (12 months ago)
Author:
Jean-Yves Avenard
Message:

[WebAudio] Add webm/opus container support
​https://bugs.webkit.org/show_bug.cgi?id=227110
<rdar://problem/79437067>

Reviewed by Jer Noble.

Source/WebCore:

Add support for webm in Web Audio.
As decoding needs to be performed in the web content process we
need to ensure that the decoders are available there. For security
reasons, com.apple.audio.AudioComponentRegistrar mach calls are blocked
which prevents registering a new audio component.
We instead register it in the UI process so that AudioComponentFetchServerRegistrations
SPI can forward the registration in the web process.
Unfortunately, for unknown reasons, this fails for Vorbis audio codec (bug 228139).

To demux the webm content, we use the existing SourceBufferParserWebM to extract all
audio packets. We then iterate over all CM Samples and decode them into PCM, performing
resampling as needed on the fly.
There are things left to fix.

  • Opus and Vorbis encoder delay isn't handled (bug 228140) which leaves audible

silence at the beginning of the destination buffer.

  • We can only decode mono and stereo content. Opus and Vorbis let you define a

custom channel ordering, this is particularly used with ambisonic audio (with opus 255 mode).
CoreMedia doesn't easily deal with such re-ordering as it appears to be essentially
designed for audible content. With more than 2 channels, decoding will either fail
or produce channels in the wrong order.

Test: webaudio/decode-audio-data-webm-opus.html

  • Headers.cmake: Export header so that we can register the decoder in the UI process.
  • WebCore.xcodeproj/project.pbxproj: same as above.
  • html/HTMLMediaElement.cpp:

(WebCore::HTMLMediaElement::canPlayType const): Make canPlayType(webm/audio; codecs=opus)
work if features is set.

  • page/RuntimeEnabledFeatures.h:

(WebCore::RuntimeEnabledFeatures::setWebMWebAudioEnabled):
(WebCore::RuntimeEnabledFeatures::webMWebAudioEnabled const):

  • platform/MediaSample.h: Fly-by fix: remove unnecessary WEBCORE_EXPORT
  • platform/audio/AudioBus.cpp:

(WebCore::AudioBus::setLength):

  • platform/audio/AudioBus.h: Add setLength method. Will change the reported length

without touching the underlying storage.

  • platform/audio/AudioChannel.h: Add setLength method. Will change the reported

length without touching the underlying storage.

  • platform/audio/cocoa/AudioFileReaderCocoa.cpp:

(WebCore::tryCreateAudioBufferList): Fix checkedInt use to catch potential overflow
(WebCore::AudioBufferListHolder::AudioBufferListHolder): Add convenience class that
free wrapped AudioBufferList object on exit.
(WebCore::AudioBufferListHolder::~AudioBufferListHolder):
(WebCore::AudioBufferListHolder::operator bool const):
(WebCore::AudioBufferListHolder::operator-> const):
(WebCore::AudioBufferListHolder::operator* const):
(WebCore::AudioBufferListHolder::isValid const):
(WebCore::AudioFileReader::AudioFileReader):
(WebCore::AudioFileReader::isMaybeWebM const): Method to check if the buffer
contains WebM data.
(WebCore::passthroughInputDataCallback): Required for AudioConverterFillComplexBuffer.
Will feed the decoder the data to decode, splitting the multiple packets found
in the CMSampleBuffer.
(WebCore::AudioFileReader::numberOfFrames const): Return the total number of
decoded frames in the packets.
(WebCore::AudioFileReader::fileDataFormat const): Create AudioStreamBasicDescription
related to the demuxed audio track. nullopt if unsupported content.
(WebCore::AudioFileReader::clientDataFormat const):
(WebCore::AudioFileReader::createBus):
(WebCore::AudioFileReader::logChannel const):

  • platform/audio/cocoa/AudioFileReaderCocoa.h:
  • platform/audio/cocoa/AudioSampleDataSource.h: Remove unused typedef.
  • platform/audio/cocoa/AudioSampleDataSource.mm:

(WebCore::AudioSampleDataSource::~AudioSampleDataSource): Fly-by fix, make it use
PAL softlinks
(WebCore::AudioSampleDataSource::setupConverter): Fly-by fix, make it use PAL
softlinks

  • platform/gamepad/mac/HIDGamepadProvider.mm: Fly-by fix, add missing header.
  • platform/graphics/TrackPrivateBase.cpp: Fly-by fix, add missing header.
  • platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h:

Fly-by fix required by webkit-prepare : don't tag the whole class as WEBCORE_EXPORT.
Move constructor and destructor function definitions out-of-line to prevent
linkage errors.

  • platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm:

(WebCore::MediaSampleAVFObjC::MediaSampleAVFObjC): Move code

  • platform/graphics/cocoa/WebCoreDecompressionSession.mm: Fly-by fix, add missing header.
  • platform/graphics/cocoa/WebMAudioUtilitiesCocoa.h: Export some symbols, add new

method definitions.

  • platform/graphics/cocoa/WebMAudioUtilitiesCocoa.mm:

(WebCore::registerDecoderFactory): Instead of attempting to register the decoder
by dl-opening the required library, first attempting to see if the decoder is
already available. Calling AudioComponentRegister in the web content process
when the GPU process is enabled will fail due to sandbox settings. Which would
incorrectly report the codec to not be available when it was.
(WebCore::createAudioFormatDescriptionForFormat):
(WebCore::isOpusDecoderAvailable): Fly-by fix, should the method be called
on non-mac platform it would have return the unitialized value of the available
static variable.
(WebCore::registerOpusDecoderIfNeeded): Added
(WebCore::isVorbisDecoderAvailable): Same as isOpusDecoderAvailable
(WebCore::registerVorbisDecoderIfNeeded): Added

Source/WebCore/PAL:

Add various AVFoundation (AVAudioConverter) related methods and
CoreMedia to access required CMSampleBuffer related methods.

  • pal/cf/CoreMediaSoftLink.cpp:
  • pal/cf/CoreMediaSoftLink.h:
  • pal/cocoa/AVFoundationSoftLink.h:
  • pal/cocoa/AVFoundationSoftLink.mm:

Source/WebKit:

Add default value for webm_webaudio feature flag for the different platforms.
Disabled by default on all platforms.
Fly-by fix: don't make webm parser dependent on VP9 support.

  • FeatureFlags/WebKit-appletvos.plist:
  • FeatureFlags/WebKit-ios.plist:
  • FeatureFlags/WebKit-macos.plist:
  • FeatureFlags/WebKit-watchos.plist:
  • GPUProcess/GPUProcess.h:
  • GPUProcess/GPUProcess.messages.in:
  • Shared/WebPreferencesDefaultValues.cpp:

(WebKit::defaultWebMWebAudioEnabled):

  • Shared/WebPreferencesDefaultValues.h:
  • UIProcess/Cocoa/WebProcessProxyCocoa.mm:

(WebKit::WebProcessProxy::sendAudioComponentRegistrations):

Source/WTF:

Add WebMWebAudioEnabled preference to make canPlayType return probably for mimetype:
"audio/webm; codecs=opus"
Disabled by default.

  • Scripts/Preferences/WebPreferencesExperimental.yaml:

LayoutTests:

  • platform/mac/TestExpectations: Only enable test on macOS Monterey
  • webaudio/decode-audio-data-basic.html: fix whitespace
  • webaudio/decode-audio-data-webm-opus-expected.txt: Added.
  • webaudio/decode-audio-data-webm-opus.html: Added.
  • webaudio/resources/media/opus.webm: Added.
  • webaudio/resources/media/vorbis.webm: Added.
Location:
trunk
Files:
4 added
43 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r280415 r280416  
     12021-07-28  Jean-Yves Avenard  <jya@apple.com>
     2
     3        [WebAudio] Add webm/opus container support
     4        https://bugs.webkit.org/show_bug.cgi?id=227110
     5        <rdar://problem/79437067>
     6
     7        Reviewed by Jer Noble.
     8
     9        * platform/mac/TestExpectations: Only enable test on macOS Monterey
     10        * webaudio/decode-audio-data-basic.html: fix whitespace
     11        * webaudio/decode-audio-data-webm-opus-expected.txt: Added.
     12        * webaudio/decode-audio-data-webm-opus.html: Added.
     13        * webaudio/resources/media/opus.webm: Added.
     14        * webaudio/resources/media/vorbis.webm: Added.
     15
    1162021-07-28  Sihui Liu  <sihui_liu@apple.com>
    217

  • trunk/LayoutTests/platform/mac/TestExpectations

    r280408 r280416  
    17561756[ Catalina Mojave BigSur ] media/media-source/media-webm-opus-partial.html [ Skip ]
    17571757[ Catalina Mojave BigSur ] media/media-source/media-webm-opus-partial-abort.html [ Skip ]
     1758[ Catalina Mojave BigSur ] webaudio/decode-audio-data-webm-opus.html [ Skip ]
    17581759
    17591760webkit.org/b/214422 imported/w3c/web-platform-tests/webaudio/the-audio-api/the-audiocontext-interface/suspend-after-construct.html [ Pass Failure ]

  • trunk/LayoutTests/webaudio/decode-audio-data-basic.html

    r267504 r280416  
    88<script>
    99description("Basic tests for decodeAudioData function.");
    10    
     10
    1111window.jsTestIsAsync = true;
    1212
    … …  
    3535    request.open("GET", decodeCaseArray[index].url, true);
    3636    request.responseType = "arraybuffer";
    37    
     37
    3838    request.onload = function() {
    3939        context.decodeAudioData(request.response, successCallback, errorCallback);
    40        
     40
    4141        function successCallback() {
    4242            if (decodeCaseArray[index].result)
    … …  
    4444            else
    4545                testFailed("The " + decodeCaseArray[index].url + " test: successCallback was not called.");
    46            
     46
    4747            runDecodeTest(++index);
    4848        }
    49        
     49
    5050        function errorCallback() {
    5151            if (decodeCaseArray[index].result)

  • trunk/Source/WTF/ChangeLog

    r280414 r280416  
     12021-07-28  Jean-Yves Avenard  <jya@apple.com>
     2
     3        [WebAudio] Add webm/opus container support
     4        https://bugs.webkit.org/show_bug.cgi?id=227110
     5        <rdar://problem/79437067>
     6
     7        Reviewed by Jer Noble.
     8
     9        Add WebMWebAudioEnabled preference to make canPlayType return probably for mimetype:
     10        "audio/webm; codecs=opus"
     11        Disabled by default.
     12
     13        * Scripts/Preferences/WebPreferencesExperimental.yaml:
     14
    1152021-07-28  Ryosuke Niwa  <rniwa@webkit.org>
    216

  • trunk/Source/WTF/Scripts/Preferences/WebPreferencesExperimental.yaml

    r280327 r280416  
    12621262      default: WebKit::defaultWebMParserEnabled()
    12631263
     1264WebMWebAudioEnabled:
     1265  type: bool
     1266  webcoreBinding: RuntimeEnabledFeatures
     1267  humanReadableName: "WebM Web Audio"
     1268  humanReadableDescription: "Enable WebM Web Audio"
     1269  condition: ENABLE(MEDIA_SOURCE)
     1270  defaultValue:
     1271    WebKitLegacy:
     1272      default: WebKit::defaultWebMWebAudioEnabled()
     1273    WebKit:
     1274      default: WebKit::defaultWebMWebAudioEnabled()
     1275
    12641276# FIXME: Is this implemented for WebKitLegacy? If not, this should be excluded from WebKitLegacy entirely.
    12651277WebRTCH264LowLatencyEncoderEnabled:

  • trunk/Source/WebCore/ChangeLog

    r280415 r280416  
     12021-07-28  Jean-Yves Avenard  <jya@apple.com>
     2
     3        [WebAudio] Add webm/opus container support
     4        https://bugs.webkit.org/show_bug.cgi?id=227110
     5        <rdar://problem/79437067>
     6
     7        Reviewed by Jer Noble.
     8
     9        Add support for webm in Web Audio.
     10        As decoding needs to be performed in the web content process we
     11        need to ensure that the decoders are available there. For security
     12        reasons, com.apple.audio.AudioComponentRegistrar mach calls are blocked
     13        which prevents registering a new audio component.
     14        We instead register it in the UI process so that AudioComponentFetchServerRegistrations
     15        SPI can forward the registration in the web process.
     16        Unfortunately, for unknown reasons, this fails for Vorbis audio codec (bug 228139).
     17
     18        To demux the webm content, we use the existing SourceBufferParserWebM to extract all
     19        audio packets. We then iterate over all CM Samples and decode them into PCM, performing
     20        resampling as needed on the fly.
     21        There are things left to fix.
     22        - Opus and Vorbis encoder delay isn't handled (bug 228140) which leaves audible
     23        silence at the beginning of the destination buffer.
     24        - We can only decode mono and stereo content. Opus and Vorbis let you define a
     25        custom channel ordering, this is particularly used with ambisonic audio (with opus 255 mode).
     26        CoreMedia doesn't easily deal with such re-ordering as it appears to be essentially
     27        designed for audible content. With more than 2 channels, decoding will either fail
     28        or produce channels in the wrong order.
     29
     30        Test: webaudio/decode-audio-data-webm-opus.html
     31
     32        * Headers.cmake: Export header so that we can register the decoder in the UI process.
     33        * WebCore.xcodeproj/project.pbxproj: same as above.
     34        * html/HTMLMediaElement.cpp:
     35        (WebCore::HTMLMediaElement::canPlayType const): Make canPlayType(webm/audio; codecs=opus)
     36        work if features is set.
     37        * page/RuntimeEnabledFeatures.h:
     38        (WebCore::RuntimeEnabledFeatures::setWebMWebAudioEnabled):
     39        (WebCore::RuntimeEnabledFeatures::webMWebAudioEnabled const):
     40        * platform/MediaSample.h: Fly-by fix: remove unnecessary WEBCORE_EXPORT
     41        * platform/audio/AudioBus.cpp:
     42        (WebCore::AudioBus::setLength):
     43        * platform/audio/AudioBus.h: Add setLength method. Will change the reported length
     44        without touching the underlying storage.
     45        * platform/audio/AudioChannel.h: Add setLength method. Will change the reported
     46        length without touching the underlying storage.
     47        * platform/audio/cocoa/AudioFileReaderCocoa.cpp:
     48        (WebCore::tryCreateAudioBufferList): Fix checkedInt use to catch potential overflow
     49        (WebCore::AudioBufferListHolder::AudioBufferListHolder): Add convenience class that
     50        free wrapped AudioBufferList object on exit.
     51        (WebCore::AudioBufferListHolder::~AudioBufferListHolder):
     52        (WebCore::AudioBufferListHolder::operator bool const):
     53        (WebCore::AudioBufferListHolder::operator-> const):
     54        (WebCore::AudioBufferListHolder::operator* const):
     55        (WebCore::AudioBufferListHolder::isValid const):
     56        (WebCore::AudioFileReader::AudioFileReader):
     57        (WebCore::AudioFileReader::isMaybeWebM const): Method to check if the buffer
     58        contains WebM data.
     59        (WebCore::passthroughInputDataCallback): Required for AudioConverterFillComplexBuffer.
     60        Will feed the decoder the data to decode, splitting the multiple packets found
     61        in the CMSampleBuffer.
     62        (WebCore::AudioFileReader::numberOfFrames const): Return the total number of
     63        decoded frames in the packets.
     64        (WebCore::AudioFileReader::fileDataFormat const): Create AudioStreamBasicDescription
     65        related to the demuxed audio track. nullopt if unsupported content.
     66        (WebCore::AudioFileReader::clientDataFormat const):
     67        (WebCore::AudioFileReader::createBus):
     68        (WebCore::AudioFileReader::logChannel const):
     69        * platform/audio/cocoa/AudioFileReaderCocoa.h:
     70        * platform/audio/cocoa/AudioSampleDataSource.h: Remove unused typedef.
     71        * platform/audio/cocoa/AudioSampleDataSource.mm:
     72        (WebCore::AudioSampleDataSource::~AudioSampleDataSource): Fly-by fix, make it use
     73        PAL softlinks
     74        (WebCore::AudioSampleDataSource::setupConverter): Fly-by fix, make it use PAL
     75        softlinks
     76        * platform/gamepad/mac/HIDGamepadProvider.mm: Fly-by fix, add missing header.
     77        * platform/graphics/TrackPrivateBase.cpp: Fly-by fix, add missing header.
     78        * platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h:
     79        Fly-by fix required by webkit-prepare : don't tag the whole class as WEBCORE_EXPORT.
     80        Move constructor and destructor function definitions out-of-line to prevent
     81        linkage errors.
     82        * platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm:
     83        (WebCore::MediaSampleAVFObjC::MediaSampleAVFObjC): Move code
     84        * platform/graphics/cocoa/WebCoreDecompressionSession.mm: Fly-by fix, add missing header.
     85        * platform/graphics/cocoa/WebMAudioUtilitiesCocoa.h: Export some symbols, add new
     86        method definitions.
     87        * platform/graphics/cocoa/WebMAudioUtilitiesCocoa.mm:
     88        (WebCore::registerDecoderFactory): Instead of attempting to register the decoder
     89        by dl-opening the required library, first attempting to see if the decoder is
     90        already available. Calling AudioComponentRegister in the web content process
     91        when the GPU process is enabled will fail due to sandbox settings. Which would
     92        incorrectly report the codec to not be available when it was.
     93        (WebCore::createAudioFormatDescriptionForFormat):
     94        (WebCore::isOpusDecoderAvailable): Fly-by fix, should the method be called
     95        on non-mac platform it would have return the unitialized value of the available
     96        static variable.
     97        (WebCore::registerOpusDecoderIfNeeded): Added
     98        (WebCore::isVorbisDecoderAvailable): Same as isOpusDecoderAvailable
     99        (WebCore::registerVorbisDecoderIfNeeded): Added
     100
    11012021-07-28  Sihui Liu  <sihui_liu@apple.com>
    2102

  • trunk/Source/WebCore/Headers.cmake

    r280415 r280416  
    13371337    platform/graphics/angle/GraphicsContextGLANGLEUtilities.h
    13381338
     1339    platform/graphics/cocoa/WebMAudioUtilitiesCocoa.h
     1340
    13391341    platform/graphics/displaylists/DisplayList.h
    13401342    platform/graphics/displaylists/DisplayListDrawGlyphsRecorder.h

  • trunk/Source/WebCore/PAL/ChangeLog

    r280384 r280416  
     12021-07-28  Jean-Yves Avenard  <jya@apple.com>
     2
     3        [WebAudio] Add webm/opus container support
     4        https://bugs.webkit.org/show_bug.cgi?id=227110
     5        <rdar://problem/79437067>
     6
     7        Reviewed by Jer Noble.
     8
     9        Add various AVFoundation (AVAudioConverter) related methods and
     10        CoreMedia to access required CMSampleBuffer related methods.
     11
     12        * pal/cf/CoreMediaSoftLink.cpp:
     13        * pal/cf/CoreMediaSoftLink.h:
     14        * pal/cocoa/AVFoundationSoftLink.h:
     15        * pal/cocoa/AVFoundationSoftLink.mm:
     16
    1172021-07-28  Alex Christensen  <achristensen@webkit.org>
    218

  • trunk/Source/WebCore/PAL/pal/cf/CoreMediaSoftLink.cpp

    r279786 r280416  
    3737#endif
    3838
     39#include <CoreAudio/CoreAudioTypes.h>
     40
    3941#if PLATFORM(IOS_FAMILY) || (PLATFORM(MAC) && __MAC_OS_X_VERSION_MAX_ALLOWED >= 101400)
    4042#define CMSAMPLEBUFFERCALL_NOESCAPE CF_NOESCAPE
    … …  
    4850SOFT_LINK_FUNCTION_FOR_SOURCE_WITH_EXPORT(PAL, CoreMedia, CMBlockBufferCopyDataBytes, OSStatus, (CMBlockBufferRef theSourceBuffer, size_t offsetToData, size_t dataLength, void* destination), (theSourceBuffer, offsetToData, dataLength, destination), PAL_EXPORT)
    4951SOFT_LINK_FUNCTION_FOR_SOURCE_WITH_EXPORT(PAL, CoreMedia, CMBlockBufferGetDataLength, size_t, (CMBlockBufferRef theBuffer), (theBuffer), PAL_EXPORT)
     52SOFT_LINK_FUNCTION_FOR_SOURCE_WITH_EXPORT(PAL, CoreMedia, CMBlockBufferIsRangeContiguous, Boolean, (CMBlockBufferRef theBuffer, size_t offset, size_t length), (theBuffer, offset, length), PAL_EXPORT)
    5053SOFT_LINK_FUNCTION_FOR_SOURCE_WITH_EXPORT(PAL, CoreMedia, CMBlockBufferReplaceDataBytes, OSStatus, (const void* sourceBytes, CMBlockBufferRef destinationBuffer, size_t offsetIntoDestination, size_t dataLength), (sourceBytes, destinationBuffer, offsetIntoDestination, dataLength), PAL_EXPORT)
    5154SOFT_LINK_FUNCTION_FOR_SOURCE_WITH_EXPORT(PAL, CoreMedia, CMFormatDescriptionGetExtensions, CFDictionaryRef, (CMFormatDescriptionRef desc), (desc), PAL_EXPORT)
    … …  
    5356SOFT_LINK_FUNCTION_FOR_SOURCE_WITH_EXPORT(PAL, CoreMedia, CMSampleBufferGetDataBuffer, CMBlockBufferRef, (CMSampleBufferRef sbuf), (sbuf), PAL_EXPORT)
    5457SOFT_LINK_FUNCTION_FOR_SOURCE_WITH_EXPORT(PAL, CoreMedia, CMSampleBufferGetFormatDescription, CMFormatDescriptionRef, (CMSampleBufferRef sbuf), (sbuf), PAL_EXPORT)
     58SOFT_LINK_FUNCTION_FOR_SOURCE_WITH_EXPORT(PAL, CoreMedia, CMSampleBufferGetAudioStreamPacketDescriptions, OSStatus, (CMSampleBufferRef sbuf, size_t packetDescriptionsSize, AudioStreamPacketDescription* packetDescriptionsOut, size_t* packetDescriptionsSizeNeededOut), (sbuf, packetDescriptionsSize, packetDescriptionsOut, packetDescriptionsSizeNeededOut), PAL_EXPORT)
    5559SOFT_LINK_FUNCTION_FOR_SOURCE_WITH_EXPORT(PAL, CoreMedia, CMSampleBufferGetSampleTimingInfo, OSStatus, (CMSampleBufferRef sbuf, CMItemIndex sampleIndex, CMSampleTimingInfo* timingInfoOut), (sbuf, sampleIndex, timingInfoOut), PAL_EXPORT)
    5660SOFT_LINK_FUNCTION_FOR_SOURCE_WITH_EXPORT(PAL, CoreMedia, CMSampleBufferDataIsReady, Boolean, (CMSampleBufferRef sbuf), (sbuf), PAL_EXPORT)
    … …  
    109113
    110114#if PLATFORM(COCOA)
     115SOFT_LINK_FUNCTION_FOR_SOURCE_WITH_EXPORT(PAL, CoreMedia, CMBlockBufferCreateContiguous, OSStatus, (CFAllocatorRef structureAllocator, CMBlockBufferRef sourceBuffer, CFAllocatorRef blockAllocator, const CMBlockBufferCustomBlockSource* customBlockSource, size_t offsetToData, size_t dataLength, CMBlockBufferFlags flags, CMBlockBufferRef* blockBufferOut), (structureAllocator, sourceBuffer, blockAllocator, customBlockSource, offsetToData, dataLength, flags, blockBufferOut), PAL_EXPORT)
    111116SOFT_LINK_FUNCTION_FOR_SOURCE_WITH_EXPORT(PAL, CoreMedia, CMFormatDescriptionGetMediaSubType, FourCharCode, (CMFormatDescriptionRef desc), (desc), PAL_EXPORT)
    112117SOFT_LINK_FUNCTION_FOR_SOURCE_WITH_EXPORT(PAL, CoreMedia, CMFormatDescriptionGetMediaType, CMMediaType, (CMFormatDescriptionRef desc), (desc), PAL_EXPORT)

  • trunk/Source/WebCore/PAL/pal/cf/CoreMediaSoftLink.h

    r279786 r280416  
    2929// FIXME: Should be USE(COREMEDIA), but this isn't currently defined on Windows.
    3030
     31#include <CoreAudio/CoreAudioTypes.h>
    3132#include <pal/spi/cf/CoreMediaSPI.h>
    3233#include <wtf/SoftLinking.h>
    … …  
    5051SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMBlockBufferGetDataLength, size_t, (CMBlockBufferRef theBuffer), (theBuffer))
    5152#define CMBlockBufferGetDataLength softLink_CoreMedia_CMBlockBufferGetDataLength
     53SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMBlockBufferIsRangeContiguous, Boolean, (CMBlockBufferRef theBuffer, size_t offset, size_t length), (theBuffer, offset, length))
     54#define CMBlockBufferIsRangeContiguous softLink_CoreMedia_CMBlockBufferIsRangeContiguous
    5255SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMBlockBufferReplaceDataBytes, OSStatus, (const void* sourceBytes, CMBlockBufferRef destinationBuffer, size_t offsetIntoDestination, size_t dataLength), (sourceBytes, destinationBuffer, offsetIntoDestination, dataLength))
    5356#define CMBlockBufferReplaceDataBytes softLink_CoreMedia_CMBlockBufferReplaceDataBytes
    … …  
    6063SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMSampleBufferGetFormatDescription, CMFormatDescriptionRef, (CMSampleBufferRef sbuf), (sbuf))
    6164#define CMSampleBufferGetFormatDescription softLink_CoreMedia_CMSampleBufferGetFormatDescription
     65SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMSampleBufferGetAudioStreamPacketDescriptions, OSStatus, (CMSampleBufferRef sbuf, size_t packetDescriptionsSize, AudioStreamPacketDescription* packetDescriptionsOut, size_t* packetDescriptionsSizeNeededOut), (sbuf, packetDescriptionsSize, packetDescriptionsOut, packetDescriptionsSizeNeededOut))
     66#define CMSampleBufferGetAudioStreamPacketDescriptions softLink_CoreMedia_CMSampleBufferGetAudioStreamPacketDescriptions
    6267SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMSampleBufferGetSampleTimingInfo, OSStatus, (CMSampleBufferRef sbuf, CMItemIndex sampleIndex, CMSampleTimingInfo* timingInfoOut), (sbuf, sampleIndex, timingInfoOut))
    6368#define CMSampleBufferGetSampleTimingInfo softLink_CoreMedia_CMSampleBufferGetSampleTimingInfo
    … …  
    169174#if PLATFORM(COCOA)
    170175
     176SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMBlockBufferCreateContiguous, OSStatus, (CFAllocatorRef structureAllocator, CMBlockBufferRef sourceBuffer, CFAllocatorRef blockAllocator, const CMBlockBufferCustomBlockSource* customBlockSource, size_t offsetToData, size_t dataLength, CMBlockBufferFlags flags, CMBlockBufferRef* blockBufferOut), (structureAllocator, sourceBuffer, blockAllocator, customBlockSource, offsetToData, dataLength, flags, blockBufferOut))
     177#define CMBlockBufferCreateContiguous softLink_CoreMedia_CMBlockBufferCreateContiguous
    171178SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMFormatDescriptionGetMediaSubType, FourCharCode, (CMFormatDescriptionRef desc), (desc))
    172179#define CMFormatDescriptionGetMediaSubType softLink_CoreMedia_CMFormatDescriptionGetMediaSubType

  • trunk/Source/WebCore/PAL/pal/cocoa/AVFoundationSoftLink.h

    r276828 r280416  
    325325#endif // HAVE(AVROUTEPICKERVIEW)
    326326
     327#if PLATFORM(COCOA)
     328SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAudioConverter)
     329SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAudioFormat)
     330SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAudioCompressedBuffer)
     331SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAudioPCMBuffer)
     332#endif // PLATFORM(COCOA)
     333
    327334#endif // USE(AVFOUNDATION)

  • trunk/Source/WebCore/PAL/pal/cocoa/AVFoundationSoftLink.mm

    r276828 r280416  
    231231SOFT_LINK_CONSTANT_MAY_FAIL_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVURLAssetHTTPCookiesKey, NSString *, PAL_EXPORT)
    232232SOFT_LINK_CONSTANT_MAY_FAIL_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVURLAssetRequiresCustomURLLoadingKey, NSString *, PAL_EXPORT)
    233 #endif
     233#endif // PLATFORM(IOS_FAMILY)
    234234
    235235#if PLATFORM(IOS_FAMILY) && !PLATFORM(WATCHOS) && !PLATFORM(APPLETV)
    … …  
    245245#endif // HAVE(AVROUTEPICKERVIEW)
    246246
     247#if PLATFORM(COCOA)
     248SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAudioConverter, PAL_EXPORT)
     249SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAudioFormat, PAL_EXPORT)
     250SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAudioCompressedBuffer, PAL_EXPORT)
     251SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAudioPCMBuffer, PAL_EXPORT)
     252#endif
     253
    247254#endif // USE(AVFOUNDATION)

  • trunk/Source/WebCore/WebCore.xcodeproj/project.pbxproj

    r280415 r280416  
    233233                07EE76EF1BEA619800F89133 /* MockRealtimeVideoSourceMac.h in Headers */ = {isa = PBXBuildFile; fileRef = 07EE76ED1BEA619800F89133 /* MockRealtimeVideoSourceMac.h */; };
    234234                07F4E93320B3587F002E3803 /* AVAssetMIMETypeCache.h in Headers */ = {isa = PBXBuildFile; fileRef = 07C8AD121D073D630087C5CE /* AVAssetMIMETypeCache.h */; settings = {ATTRIBUTES = (Private, ); }; };
    235                 07F5CFF52582A51D00662EF5 /* WebMAudioUtilitiesCocoa.h in Headers */ = {isa = PBXBuildFile; fileRef = 07F5CFF42582A4F800662EF5 /* WebMAudioUtilitiesCocoa.h */; };
     235                07F5CFF52582A51D00662EF5 /* WebMAudioUtilitiesCocoa.h in Headers */ = {isa = PBXBuildFile; fileRef = 07F5CFF42582A4F800662EF5 /* WebMAudioUtilitiesCocoa.h */; settings = {ATTRIBUTES = (Private, ); }; };
    236236                07F5CFF62582A52900662EF5 /* WebMAudioUtilitiesCocoa.mm in Sources */ = {isa = PBXBuildFile; fileRef = 07F5CFF22582A4F800662EF5 /* WebMAudioUtilitiesCocoa.mm */; };
    237237                07F876841AD580F900905849 /* MediaPlaybackTargetContext.h in Headers */ = {isa = PBXBuildFile; fileRef = 07F876831AD4A94500905849 /* MediaPlaybackTargetContext.h */; settings = {ATTRIBUTES = (Private, ); }; };

  • trunk/Source/WebCore/html/HTMLMediaElement.cpp

    r280330 r280416  
    10471047}
    10481048
     1049static inline bool webMWebAudioEnabled()
     1050{
     1051#if ENABLE(MEDIA_SOURCE)
     1052    return RuntimeEnabledFeatures::sharedFeatures().webMWebAudioEnabled();
     1053#else
     1054    return false;
     1055#endif
     1056}
     1057
    10491058String HTMLMediaElement::canPlayType(const String& mimeType) const
    10501059{
    … …  
    10601069    // so that sites relying on this test to determine if webaudio use of opus or vorbis won't error.
    10611070    auto codecs = contentType.codecs();
    1062     if (support == MediaPlayer::SupportsType::IsSupported && (codecs.contains("opus") || codecs.contains("vorbis")))
     1071    if (support == MediaPlayer::SupportsType::IsSupported && ((codecs.contains("opus") && !webMWebAudioEnabled()) || codecs.contains("vorbis")))
    10631072        support = MediaPlayer::SupportsType::IsNotSupported;
    10641073#endif

  • trunk/Source/WebCore/page/RuntimeEnabledFeatures.h

    r279974 r280416  
    236236    void setWebMParserEnabled(bool isEnabled) { m_webMParserEnabled = isEnabled; }
    237237    bool webMParserEnabled() const { return m_webMParserEnabled; }
     238    void setWebMWebAudioEnabled(bool isEnabled) { m_webMWebAudioEnabled = isEnabled; }
     239    bool webMWebAudioEnabled() const { return m_webMWebAudioEnabled; }
    238240#endif
    239241
    … …  
    372374#if ENABLE(MEDIA_SOURCE)
    373375    bool m_webMParserEnabled { false };
     376    bool m_webMWebAudioEnabled { false };
    374377#endif
    375378

  • trunk/Source/WebCore/platform/MediaSample.h

    r278253 r280416  
    5858};
    5959
    60 class WEBCORE_EXPORT MediaSample : public ThreadSafeRefCounted<MediaSample> {
     60class MediaSample : public ThreadSafeRefCounted<MediaSample> {
    6161public:
    6262    virtual ~MediaSample() = default;

  • trunk/Source/WebCore/platform/audio/AudioBus.cpp

    r278333 r280416  
    7474}
    7575
     76void AudioBus::setLength(size_t newLength)
     77{
     78    RELEASE_ASSERT_WITH_SECURITY_IMPLICATION(newLength <= m_length);
     79    if (newLength == m_length)
     80        return;
     81    for (unsigned i = 0; i < m_channels.size(); ++i)
     82        m_channels[i]->setLength(newLength);
     83    m_length = newLength;
     84}
     85
    7686void AudioBus::zero()
    7787{

  • trunk/Source/WebCore/platform/audio/AudioBus.h

    r278333 r280416  
    8484    size_t length() const { return m_length; }
    8585
     86    // Reduce the number of frames available without changing the allocated memory size.
     87    void setLength(size_t);
     88
    8689    // Sample-rate : 0.0 if unknown or "don't care"
    8790    float sampleRate() const { return m_sampleRate; }

  • trunk/Source/WebCore/platform/audio/AudioChannel.h

    r278333 r280416  
    7575    size_t length() const { return m_length; }
    7676
     77    // Set new length. Can only be set to a value lower than the current length.
     78    void setLength(size_t newLength)
     79    {
     80        RELEASE_ASSERT_WITH_SECURITY_IMPLICATION(newLength <= length());
     81        m_length = newLength;
     82    }
     83
    7784    // Direct access to PCM sample data. Non-const accessor clears silent flag.
    7885    float* mutableData()

  • trunk/Source/WebCore/platform/audio/cocoa/AudioFileReaderCocoa.cpp

    r279335 r280416  
    3535#include "AudioBus.h"
    3636#include "AudioFileReader.h"
     37#include "AudioSampleDataSource.h"
     38#include "AudioTrackPrivate.h"
    3739#include "FloatConversion.h"
     40#include "InbandTextTrackPrivate.h"
    3841#include "Logging.h"
     42#include "MediaSampleAVFObjC.h"
     43#include "VideoTrackPrivate.h"
     44#include "WebMAudioUtilitiesCocoa.h"
     45#include <AudioToolbox/AudioConverter.h>
    3946#include <AudioToolbox/ExtendedAudioFile.h>
    4047#include <CoreFoundation/CoreFoundation.h>
    41 #include <pal/cf/AudioToolboxSoftLink.h>
     48#include <SourceBufferParserWebM.h>
     49#include <limits>
    4250#include <wtf/CheckedArithmetic.h>
    4351#include <wtf/FastMalloc.h>
     52#include <wtf/Function.h>
    4453#include <wtf/RetainPtr.h>
     54#include <wtf/Scope.h>
     55#include <wtf/Vector.h>
     56#include <pal/cf/AudioToolboxSoftLink.h>
     57#include <pal/cf/CoreMediaSoftLink.h>
    4558
    4659namespace WebCore {
    … …  
    4861static WARN_UNUSED_RETURN AudioBufferList* tryCreateAudioBufferList(size_t numberOfBuffers)
    4962{
     63    if (!numberOfBuffers)
     64        return nullptr;
    5065    CheckedSize bufferListSize = sizeof(AudioBufferList) - sizeof(AudioBuffer);
    51     bufferListSize += numberOfBuffers * sizeof(AudioBuffer);
     66    bufferListSize += WTF::checkedProduct<size_t>(numberOfBuffers, sizeof(AudioBuffer));
    5267    if (bufferListSize.hasOverflowed())
    5368        return nullptr;
    … …  
    88103}
    89104
     105// On stack RAII class that will free the allocated AudioBufferList* as needed.
     106class AudioBufferListHolder {
     107public:
     108    explicit AudioBufferListHolder(size_t numberOfChannels)
     109        : m_bufferList(tryCreateAudioBufferList(numberOfChannels))
     110    {
     111    }
     112
     113    ~AudioBufferListHolder()
     114    {
     115        if (m_bufferList)
     116            destroyAudioBufferList(m_bufferList);
     117    }
     118
     119    explicit operator bool() const { return !!m_bufferList; }
     120    AudioBufferList* operator->() const { return m_bufferList; }
     121    operator AudioBufferList*() const { return m_bufferList; }
     122    AudioBufferList& operator*() const { ASSERT(m_bufferList); return *m_bufferList; }
     123    bool isValid() const { return validateAudioBufferList(m_bufferList); }
     124private:
     125    AudioBufferList* m_bufferList;
     126};
     127
     128class AudioFileReaderWebMData {
     129    WTF_MAKE_FAST_ALLOCATED;
     130
     131public:
     132#if ENABLE(MEDIA_SOURCE)
     133    SourceBufferParserWebM::InitializationSegment m_initSegment;
     134#endif
     135    MediaTime m_duration;
     136    Vector<Ref<MediaSampleAVFObjC>> m_samples;
     137};
     138
    90139AudioFileReader::AudioFileReader(const void* data, size_t dataSize)
    91140    : m_data(data)
    92141    , m_dataSize(dataSize)
    93 {
     142#if !RELEASE_LOG_DISABLED
     143    , m_logger(Logger::create(this))
     144    , m_logIdentifier(LoggerHelper::uniqueLogIdentifier())
     145#endif
     146{
     147#if ENABLE(MEDIA_SOURCE)
     148    if (isMaybeWebM(static_cast<const uint8_t*>(data), dataSize)) {
     149        m_webmData = demuxWebMData(static_cast<const uint8_t*>(data), dataSize);
     150        if (m_webmData)
     151            return;
     152    }
     153#endif
    94154    if (PAL::AudioFileOpenWithCallbacks(this, readProc, 0, getSizeProc, 0, 0, &m_audioFileID) != noErr)
    95155        return;
    … …  
    111171    m_audioFileID = 0;
    112172}
     173
     174#if ENABLE(MEDIA_SOURCE)
     175bool AudioFileReader::isMaybeWebM(const uint8_t* data, size_t dataSize) const
     176{
     177    // From https://mimesniff.spec.whatwg.org/#signature-for-webm
     178    return dataSize >= 4 && data[0] == 0x1A && data[1] == 0x45 && data[2] == 0xDF && data[3] == 0xA3;
     179}
     180
     181std::unique_ptr<AudioFileReaderWebMData> AudioFileReader::demuxWebMData(const uint8_t* data, size_t dataSize) const
     182{
     183    auto parser = adoptRef(new SourceBufferParserWebM());
     184    bool error = false;
     185    std::optional<uint64_t> audioTrackId;
     186    MediaTime duration;
     187    SourceBufferParserWebM::InitializationSegment initSegment;
     188    Vector<Ref<MediaSampleAVFObjC>> samples;
     189    parser->setDidEncounterErrorDuringParsingCallback([&](uint64_t) {
     190        error = true;
     191    });
     192    parser->setDidParseInitializationDataCallback([&](SourceBufferParserWebM::InitializationSegment&& init) {
     193        for (auto& audioTrack : init.audioTracks) {
     194            if (audioTrack.track && audioTrack.track->trackUID()) {
     195                duration = init.duration;
     196                audioTrackId = audioTrack.track->trackUID();
     197                initSegment = WTFMove(init);
     198                return;
     199            }
     200        }
     201    });
     202    parser->setDidProvideMediaDataCallback([&](Ref<MediaSample>&& sample, uint64_t trackID, const String&) {
     203        if (!audioTrackId || trackID != *audioTrackId)
     204            return;
     205        samples.append(static_reference_cast<MediaSampleAVFObjC>(WTFMove(sample)));
     206    });
     207    parser->setCallOnClientThreadCallback([](auto&& function) {
     208        function();
     209    });
     210    SourceBufferParser::Segment segment({ data, dataSize });
     211    parser->appendData(WTFMove(segment));
     212    if (!audioTrackId)
     213        return nullptr;
     214    parser->flushPendingAudioBuffers();
     215    return makeUnique<AudioFileReaderWebMData>(AudioFileReaderWebMData { WTFMove(initSegment), WTFMove(duration), WTFMove(samples) });
     216}
     217
     218struct PassthroughUserData {
     219    UInt32 m_channels;
     220    UInt32 m_dataSize;
     221    const void* m_data;
     222    UInt32 m_index;
     223    Vector<AudioStreamPacketDescription>& m_packets;
     224};
     225
     226// Error value we pass through the decoder to signal that nothing
     227// has gone wrong during decoding and we're done processing the packet.
     228const uint32_t kNoMoreDataErr = 'MOAR';
     229
     230static OSStatus passthroughInputDataCallback(AudioConverterRef, UInt32* numDataPackets, AudioBufferList* data, AudioStreamPacketDescription** packetDesc, void* inUserData)
     231{
     232    ASSERT(numDataPackets && data && inUserData);
     233    if (!numDataPackets || !data || !inUserData)
     234        return kAudioConverterErr_UnspecifiedError;
     235
     236    auto* userData = static_cast<PassthroughUserData*>(inUserData);
     237    if (userData->m_index == userData->m_packets.size()) {
     238        *numDataPackets = 0;
     239        return kNoMoreDataErr;
     240    }
     241
     242    if (packetDesc) {
     243        if (userData->m_index >= userData->m_packets.size()) {
     244            *numDataPackets = 0;
     245            return kAudioConverterErr_RequiresPacketDescriptionsError;
     246        }
     247        *packetDesc = &userData->m_packets[userData->m_index];
     248    }
     249
     250    data->mBuffers[0].mNumberChannels = userData->m_channels;
     251    data->mBuffers[0].mDataByteSize = userData->m_dataSize;
     252    data->mBuffers[0].mData = const_cast<void*>(userData->m_data);
     253
     254    *numDataPackets = 1;
     255    userData->m_index++;
     256
     257    return noErr;
     258}
     259
     260Vector<AudioStreamPacketDescription> AudioFileReader::getPacketDescriptions(CMSampleBufferRef sampleBuffer) const
     261{
     262    size_t packetDescriptionsSize;
     263    if (PAL::CMSampleBufferGetAudioStreamPacketDescriptions(sampleBuffer, 0, nullptr, &packetDescriptionsSize) != noErr) {
     264        RELEASE_LOG_FAULT(WebAudio, "Unable to get packet description list size");
     265        return { };
     266    }
     267    size_t numDescriptions = packetDescriptionsSize / sizeof(AudioStreamPacketDescription);
     268    if (!numDescriptions) {
     269        RELEASE_LOG_FAULT(WebAudio, "No packet description found.");
     270        return { };
     271    }
     272    Vector<AudioStreamPacketDescription> descriptions(numDescriptions);
     273    if (PAL::CMSampleBufferGetAudioStreamPacketDescriptions(sampleBuffer, packetDescriptionsSize, descriptions.data(), nullptr) != noErr) {
     274        RELEASE_LOG_FAULT(WebAudio, "Unable to get packet description list");
     275        return { };
     276    }
     277    auto numPackets = PAL::CMSampleBufferGetNumSamples(sampleBuffer);
     278    if (numDescriptions != size_t(numPackets)) {
     279        RELEASE_LOG_FAULT(WebAudio, "Unhandled CMSampleBuffer structure");
     280        return { };
     281    }
     282    return descriptions;
     283}
     284
     285std::optional<size_t> AudioFileReader::decodeWebMData(AudioBufferList& bufferList, size_t numberOfFrames, const AudioStreamBasicDescription& inFormat, const AudioStreamBasicDescription& outFormat) const
     286{
     287    AudioConverterRef converter;
     288    if (PAL::AudioConverterNew(&inFormat, &outFormat, &converter) != noErr) {
     289        RELEASE_LOG_FAULT(WebAudio, "Unable to create decoder");
     290        return { };
     291    }
     292    auto cleanup = makeScopeExit([&] {
     293        PAL::AudioConverterDispose(converter);
     294    });
     295    ASSERT(m_webmData && !m_webmData->m_samples.isEmpty() && m_webmData->m_samples[0]->sampleBuffer(), "Structure integrity was checked in numberOfFrames");
     296    auto formatDescription = PAL::CMSampleBufferGetFormatDescription(m_webmData->m_samples[0]->sampleBuffer());
     297    if (!formatDescription) {
     298        RELEASE_LOG_FAULT(WebAudio, "Unable to retrieve format description from first sample");
     299        return { };
     300    }
     301    size_t magicCookieSize = 0;
     302    const void* magicCookie = PAL::CMAudioFormatDescriptionGetMagicCookie(formatDescription, &magicCookieSize);
     303    if (magicCookie && magicCookieSize)
     304        PAL::AudioConverterSetProperty(converter, kAudioConverterDecompressionMagicCookie, magicCookieSize, magicCookie);
     305
     306    AudioBufferListHolder decodedBufferList(inFormat.mChannelsPerFrame);
     307    if (!decodedBufferList) {
     308        RELEASE_LOG_FAULT(WebAudio, "Unable to create decoder");
     309        return { };
     310    }
     311
     312    size_t decodedFrames = 0;
     313    OSStatus status;
     314    for (auto& sample : m_webmData->m_samples) {
     315        CMSampleBufferRef sampleBuffer = sample->sampleBuffer();
     316        auto buffer = PAL::CMSampleBufferGetDataBuffer(sampleBuffer);
     317        ASSERT(PAL::CMBlockBufferIsRangeContiguous(buffer, 0, 0));
     318        if (!PAL::CMBlockBufferIsRangeContiguous(buffer, 0, 0)) {
     319            RELEASE_LOG_FAULT(WebAudio, "Unable to read sample content (not contiguous)");
     320            return { };
     321        }
     322
     323        size_t srcSize = PAL::CMBlockBufferGetDataLength(buffer);
     324        char* srcData = nullptr;
     325        if (PAL::CMBlockBufferGetDataPointer(buffer, 0, nullptr, nullptr, &srcData) != noErr) {
     326            RELEASE_LOG_FAULT(WebAudio, "Unable to retrieve data");
     327            return { };
     328        }
     329
     330        auto descriptions = getPacketDescriptions(sampleBuffer);
     331        if (descriptions.isEmpty())
     332            return { };
     333
     334        PassthroughUserData userData = { inFormat.mChannelsPerFrame, UInt32(srcSize), srcData, 0, descriptions };
     335
     336        do {
     337            if (numberOfFrames < decodedFrames) {
     338                RELEASE_LOG_FAULT(WebAudio, "Decoded more frames than first calculated");
     339                return { };
     340            }
     341            // in: the max number of packets we can handle from the decoder.
     342            // out: the number of packets the decoder is actually returning.
     343            UInt32 numFrames = std::min<uint32_t>(std::numeric_limits<int32_t>::max() / sizeof(float), numberOfFrames - decodedFrames);
     344
     345            for (UInt32 i = 0; i < inFormat.mChannelsPerFrame; i++) {
     346                decodedBufferList->mBuffers[i].mNumberChannels = 1;
     347                decodedBufferList->mBuffers[i].mDataByteSize = numFrames * sizeof(float);
     348                decodedBufferList->mBuffers[i].mData = static_cast<float*>(bufferList.mBuffers[i].mData) + decodedFrames;
     349            }
     350            status = PAL::AudioConverterFillComplexBuffer(converter, passthroughInputDataCallback, &userData, &numFrames, decodedBufferList, nullptr);
     351            if (status && status != kNoMoreDataErr) {
     352                RELEASE_LOG_FAULT(WebAudio, "Error decoding data");
     353                return { };
     354            }
     355            decodedFrames += numFrames;
     356        } while (status != kNoMoreDataErr);
     357    }
     358    return decodedFrames;
     359}
     360#endif
    113361
    114362OSStatus AudioFileReader::readProc(void* clientData, SInt64 position, UInt32 requestCount, void* buffer, UInt32* actualCount)
    … …  
    137385}
    138386
    139 RefPtr<AudioBus> AudioFileReader::createBus(float sampleRate, bool mixToMono)
    140 {
    141     if (!m_extAudioFileRef)
    142         return nullptr;
    143 
    144     // Get file's data format
    145     UInt32 size = sizeof(m_fileDataFormat);
    146     if (PAL::ExtAudioFileGetProperty(m_extAudioFileRef, kExtAudioFileProperty_FileDataFormat, &size, &m_fileDataFormat) != noErr)
    147         return nullptr;
    148 
    149     size_t numberOfChannels = m_fileDataFormat.mChannelsPerFrame;
    150 
    151     // Number of frames
    152     SInt64 numberOfFrames64 = 0;
    153     size = sizeof(numberOfFrames64);
    154     if (PAL::ExtAudioFileGetProperty(m_extAudioFileRef, kExtAudioFileProperty_FileLengthFrames, &size, &numberOfFrames64) != noErr || numberOfFrames64 <= 0)
    155         return nullptr;
    156 
    157     double fileSampleRate = m_fileDataFormat.mSampleRate;
    158 
     387ssize_t AudioFileReader::numberOfFrames() const
     388{
     389    SInt64 numberOfFramesIn64 = 0;
     390
     391    if (!m_webmData) {
     392        if (!m_extAudioFileRef)
     393            return -1;
     394
     395        UInt32 size = sizeof(numberOfFramesIn64);
     396        if (PAL::ExtAudioFileGetProperty(m_extAudioFileRef, kExtAudioFileProperty_FileLengthFrames, &size, &numberOfFramesIn64) != noErr || numberOfFramesIn64 <= 0) {
     397            RELEASE_LOG_FAULT(WebAudio, "Unable to retrieve number of frames in content (unsupported?");
     398            return -1;
     399        }
     400
     401        return numberOfFramesIn64;
     402    }
     403#if ENABLE(MEDIA_SOURCE)
     404    if (m_webmData->m_samples.isEmpty()) {
     405        RELEASE_LOG_FAULT(WebAudio, "No sample demuxed from webm container");
     406        return -1;
     407    }
     408
     409    // Calculate the total number of decoded samples that were demuxed.
     410    // This code only handle the CMSampleBuffer as generated by the SourceBufferParserWebM
     411    // where a AudioStreamPacketDescriptions array is always provided even with
     412    // Constant bitrate and constant frames-per-packet audio.
     413    for (auto& sample : m_webmData->m_samples) {
     414        auto sampleBuffer = sample->sampleBuffer();
     415        if (!sampleBuffer) {
     416            RELEASE_LOG_FAULT(WebAudio, "Impossible memory corruption encountered");
     417            return -1;
     418        }
     419        const auto formatDescription = PAL::CMSampleBufferGetFormatDescription(sampleBuffer);
     420        if (!formatDescription) {
     421            RELEASE_LOG_FAULT(WebAudio, "Unable to retrieve format descriptiong from sample");
     422            return -1;
     423        }
     424        const AudioStreamBasicDescription* const asbd = PAL::CMAudioFormatDescriptionGetStreamBasicDescription(formatDescription);
     425        if (!asbd) {
     426            RELEASE_LOG_FAULT(WebAudio, "Unable to retrieve asbd from format description");
     427            return -1;
     428        }
     429
     430        auto descriptions = getPacketDescriptions(sampleBuffer);
     431        if (descriptions.isEmpty())
     432            return -1;
     433
     434        for (const auto& description : descriptions) {
     435            uint32_t fpp = description.mVariableFramesInPacket ? description.mVariableFramesInPacket : asbd->mFramesPerPacket;
     436            numberOfFramesIn64 += fpp;
     437        }
     438    }
     439    return numberOfFramesIn64;
     440#else
     441    return 0;
     442#endif
     443}
     444
     445std::optional<AudioStreamBasicDescription> AudioFileReader::fileDataFormat() const
     446{
     447    if (!m_webmData) {
     448        AudioStreamBasicDescription format;
     449        UInt32 size = sizeof(format);
     450        if (PAL::ExtAudioFileGetProperty(m_extAudioFileRef, kExtAudioFileProperty_FileDataFormat, &size, &format) != noErr)
     451            return { };
     452        return format;
     453    }
     454
     455    if (m_webmData->m_samples.isEmpty())
     456        return { };
     457
     458    CMFormatDescriptionRef formatDescription = PAL::CMSampleBufferGetFormatDescription(m_webmData->m_samples[0]->sampleBuffer());
     459    if (!formatDescription)
     460        return { };
     461
     462    const AudioStreamBasicDescription* const asbd = PAL::CMAudioFormatDescriptionGetStreamBasicDescription(formatDescription);
     463    return *asbd;
     464}
     465
     466AudioStreamBasicDescription AudioFileReader::clientDataFormat(const AudioStreamBasicDescription& inFormat, float sampleRate) const
     467{
    159468    // Make client format same number of channels as file format, but tweak a few things.
    160469    // Client format will be linear PCM (canonical), and potentially change sample-rate.
    161     m_clientDataFormat = m_fileDataFormat;
     470    AudioStreamBasicDescription outFormat = inFormat;
    162471
    163472    const int bytesPerFloat = sizeof(Float32);
    164473    const int bitsPerByte = 8;
    165     m_clientDataFormat.mFormatID = kAudioFormatLinearPCM;
    166     m_clientDataFormat.mFormatFlags = kAudioFormatFlagsNativeFloatPacked | kAudioFormatFlagIsNonInterleaved;
    167     m_clientDataFormat.mBytesPerPacket = bytesPerFloat;
    168     m_clientDataFormat.mFramesPerPacket = 1;
    169     m_clientDataFormat.mBytesPerFrame = bytesPerFloat;
    170     m_clientDataFormat.mChannelsPerFrame = numberOfChannels;
    171     m_clientDataFormat.mBitsPerChannel = bitsPerByte * bytesPerFloat;
     474    outFormat.mFormatID = kAudioFormatLinearPCM;
     475    outFormat.mFormatFlags = kAudioFormatFlagsNativeFloatPacked | kAudioFormatFlagIsNonInterleaved;
     476    outFormat.mBytesPerPacket = outFormat.mBytesPerFrame = bytesPerFloat;
     477    outFormat.mFramesPerPacket = 1;
     478    outFormat.mBitsPerChannel = bitsPerByte * bytesPerFloat;
    172479
    173480    if (sampleRate)
    174         m_clientDataFormat.mSampleRate = sampleRate;
    175 
    176     if (PAL::ExtAudioFileSetProperty(m_extAudioFileRef, kExtAudioFileProperty_ClientDataFormat, sizeof(AudioStreamBasicDescription), &m_clientDataFormat) != noErr)
    177         return nullptr;
    178 
    179     // Change numberOfFrames64 to destination sample-rate
    180     numberOfFrames64 = numberOfFrames64 * (m_clientDataFormat.mSampleRate / fileSampleRate);
    181     size_t numberOfFrames = static_cast<size_t>(numberOfFrames64);
    182 
     481        outFormat.mSampleRate = sampleRate;
     482
     483    return outFormat;
     484}
     485
     486RefPtr<AudioBus> AudioFileReader::createBus(float sampleRate, bool mixToMono)
     487{
     488    SInt64 numberOfFramesIn64 = numberOfFrames();
     489    if (numberOfFramesIn64 <= 0)
     490        return nullptr;
     491
     492    auto inFormat = fileDataFormat();
     493    if (!inFormat)
     494        return nullptr;
     495
     496    AudioStreamBasicDescription outFormat = clientDataFormat(*inFormat, sampleRate);
     497    size_t numberOfChannels = inFormat->mChannelsPerFrame;
     498    double fileSampleRate = inFormat->mSampleRate;
     499    SInt64 numberOfFramesOut64 = numberOfFramesIn64 * (outFormat.mSampleRate / fileSampleRate);
     500    size_t numberOfFrames = static_cast<size_t>(numberOfFramesOut64);
    183501    size_t busChannelCount = mixToMono ? 1 : numberOfChannels;
    184502
    185503    // Create AudioBus where we'll put the PCM audio data
    186504    auto audioBus = AudioBus::create(busChannelCount, numberOfFrames);
    187     audioBus->setSampleRate(narrowPrecisionToFloat(m_clientDataFormat.mSampleRate)); // save for later
     505    audioBus->setSampleRate(narrowPrecisionToFloat(outFormat.mSampleRate)); // save for later
     506
     507    AudioBufferListHolder bufferList(numberOfChannels);
     508    if (!bufferList) {
     509        RELEASE_LOG_FAULT(WebAudio, "tryCreateAudioBufferList(%ld) returned null", numberOfChannels);
     510        return nullptr;
     511    }
     512    const size_t bufferSize = numberOfFrames * sizeof(float);
    188513
    189514    // Only allocated in the mixToMono case; deallocated on destruction.
    190515    AudioFloatArray leftChannel;
    191516    AudioFloatArray rightChannel;
    192 
    193     AudioBufferList* bufferList = tryCreateAudioBufferList(numberOfChannels);
    194     if (!bufferList) {
    195         RELEASE_LOG_FAULT(WebAudio, "tryCreateAudioBufferList(%ld) returned null", numberOfChannels);
    196         return nullptr;
    197     }
    198     const size_t bufferSize = numberOfFrames * sizeof(float);
    199517
    200518    RELEASE_ASSERT(bufferList->mNumberBuffers == numberOfChannels);
    … …  
    223541    }
    224542
    225     if (!validateAudioBufferList(bufferList)) {
     543    if (!bufferList.isValid()) {
    226544        RELEASE_LOG_FAULT(WebAudio, "Generated buffer in AudioFileReader::createBus() did not pass validation");
    227545        ASSERT_NOT_REACHED();
    228         destroyAudioBufferList(bufferList);
    229         return nullptr;
    230     }
    231 
    232     // Read from the file (or in-memory version)
    233     UInt32 framesToRead = numberOfFrames;
    234     if (PAL::ExtAudioFileRead(m_extAudioFileRef, &framesToRead, bufferList) != noErr) {
    235         destroyAudioBufferList(bufferList);
    236         return nullptr;
     546        return nullptr;
     547    }
     548
     549    if (m_webmData) {
     550#if ENABLE(MEDIA_SOURCE)
     551        auto decodedFrames = decodeWebMData(*bufferList, numberOfFrames, *inFormat, outFormat);
     552        if (!decodedFrames)
     553            return nullptr;
     554        // The actual decoded number of frames may not match the number of frames calculated
     555        // while demuxing as frames can be trimmed. It will always be lower.
     556        audioBus->setLength(*decodedFrames);
     557        numberOfFrames = *decodedFrames;
     558#endif
     559    } else {
     560        if (PAL::ExtAudioFileSetProperty(m_extAudioFileRef, kExtAudioFileProperty_ClientDataFormat, sizeof(AudioStreamBasicDescription), &outFormat) != noErr)
     561            return nullptr;
     562
     563        // Read from the file (or in-memory version)
     564        UInt32 framesToRead = numberOfFrames;
     565        if (PAL::ExtAudioFileRead(m_extAudioFileRef, &framesToRead, bufferList) != noErr)
     566            return nullptr;
    237567    }
    238568
    … …  
    244574    }
    245575
    246     // Cleanup
    247     destroyAudioBufferList(bufferList);
    248 
    249576    return audioBus;
    250577}
    … …  
    256583}
    257584
     585#if !RELEASE_LOG_DISABLED
     586WTFLogChannel& AudioFileReader::logChannel() const
     587{
     588    return LogMedia;
     589}
     590#endif
     591
    258592} // WebCore
    259593

  • trunk/Source/WebCore/platform/audio/cocoa/AudioFileReaderCocoa.h

    r279335 r280416  
    3333
    3434#include <CoreAudio/CoreAudioTypes.h>
     35#include <optional>
     36#include <wtf/LoggerHelper.h>
    3537#include <wtf/RefPtr.h>
    3638
    3739using ExtAudioFileRef = struct OpaqueExtAudioFile*;
    3840using AudioFileID = struct OpaqueAudioFileID*;
     41typedef struct opaqueCMSampleBuffer* CMSampleBufferRef;
    3942
    4043namespace WebCore {
    4144
    4245class AudioBus;
     46class SourceBufferParserWebM;
     47class AudioFileReaderWebMData;
    4348
    4449// Wrapper class for AudioFile and ExtAudioFile CoreAudio APIs for reading files and in-memory versions of them...
    4550
    46 class AudioFileReader {
     51class AudioFileReader
     52#if !RELEASE_LOG_DISABLED
     53    : public LoggerHelper
     54#endif
     55{
    4756public:
    48     explicit AudioFileReader(const void* data, size_t dataSize);
     57    AudioFileReader(const void* data, size_t dataSize);
    4958    ~AudioFileReader();
    5059
    … …  
    5463    size_t dataSize() const { return m_dataSize; }
    5564
     65#if !RELEASE_LOG_DISABLED
     66    const Logger& logger() const final { return m_logger.get(); }
     67    const void* logIdentifier() const final { return m_logIdentifier; }
     68    WTFLogChannel& logChannel() const final;
     69    const char* logClassName() const final { return "AudioFileReaderCocoa"; }
     70#endif
     71
    5672private:
     73#if ENABLE(MEDIA_SOURCE)
     74    bool isMaybeWebM(const uint8_t* data, size_t dataSize) const;
     75    std::unique_ptr<AudioFileReaderWebMData> demuxWebMData(const uint8_t* data, size_t dataSize) const;
     76    Vector<AudioStreamPacketDescription> getPacketDescriptions(CMSampleBufferRef) const;
     77    std::optional<size_t> decodeWebMData(AudioBufferList&, size_t numberOfFrames, const AudioStreamBasicDescription& inFormat, const AudioStreamBasicDescription& outFormat) const;
     78#endif
    5779    static OSStatus readProc(void* clientData, SInt64 position, UInt32 requestCount, void* buffer, UInt32* actualCount);
    5880    static SInt64 getSizeProc(void* clientData);
     81    ssize_t numberOfFrames() const;
     82    std::optional<AudioStreamBasicDescription> fileDataFormat() const;
     83    AudioStreamBasicDescription clientDataFormat(const AudioStreamBasicDescription& inFormat, float sampleRate) const;
    5984
    6085    const void* m_data = { nullptr };
    … …  
    6489    ExtAudioFileRef m_extAudioFileRef = { nullptr };
    6590
    66     AudioStreamBasicDescription m_fileDataFormat;
    67     AudioStreamBasicDescription m_clientDataFormat;
     91    std::unique_ptr<AudioFileReaderWebMData> m_webmData;
     92
     93#if !RELEASE_LOG_DISABLED
     94    const Ref<Logger> m_logger;
     95    const void* m_logIdentifier;
     96#endif
     97
    6898};
    6999

  • trunk/Source/WebCore/platform/audio/cocoa/AudioSampleDataSource.h

    r278453 r280416  
    3434#include <wtf/text/WTFString.h>
    3535
    36 typedef const struct opaqueCMFormatDescription *CMFormatDescriptionRef;
    3736typedef struct opaqueCMSampleBuffer *CMSampleBufferRef;
    3837

  • trunk/Source/WebCore/platform/audio/cocoa/AudioSampleDataSource.mm

    r279335 r280416  
    4040#import <wtf/StringPrintStream.h>
    4141
     42#import <pal/cf/AudioToolboxSoftLink.h>
    4243#import <pal/cf/CoreMediaSoftLink.h>
    4344
    … …  
    6869{
    6970    if (m_converter)
    70         AudioConverterDispose(m_converter);
     71        PAL::AudioConverterDispose(m_converter);
    7172}
    7273
    … …  
    7677
    7778    if (m_converter) {
    78         AudioConverterDispose(m_converter);
     79        PAL::AudioConverterDispose(m_converter);
    7980        m_converter = nullptr;
    8081    }
    … …  
    8384        return 0;
    8485
    85     OSStatus err = AudioConverterNew(&m_inputDescription->streamDescription(), &m_outputDescription->streamDescription(), &m_converter);
     86    OSStatus err = PAL::AudioConverterNew(&m_inputDescription->streamDescription(), &m_outputDescription->streamDescription(), &m_converter);
    8687    if (err) {
    8788        RunLoop::main().dispatch([this, protectedThis = makeRefPtr(*this), err] {

  • trunk/Source/WebCore/platform/gamepad/mac/HIDGamepadProvider.mm

    r279480 r280416  
    2929#if ENABLE(GAMEPAD) && PLATFORM(MAC)
    3030
     31#import "GameControllerGamepadProvider.h"
    3132#import "GamepadProviderClient.h"
    3233#import "Logging.h"

  • trunk/Source/WebCore/platform/graphics/TrackPrivateBase.cpp

    r278253 r280416  
    3030
    3131#include "Logging.h"
    32 #include <wtf/LoggerHelper.h>
    3332
    3433namespace WebCore {

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h

    r278253 r280416  
    3535class PixelBuffer;
    3636
    37 class WEBCORE_EXPORT MediaSampleAVFObjC : public MediaSample {
     37class MediaSampleAVFObjC : public MediaSample {
    3838public:
    3939    static Ref<MediaSampleAVFObjC> create(CMSampleBufferRef sample, uint64_t trackID) { return adoptRef(*new MediaSampleAVFObjC(sample, trackID)); }
    … …  
    4545    static RetainPtr<CMSampleBufferRef> cloneSampleBufferAndSetAsDisplayImmediately(CMSampleBufferRef);
    4646
    47     RefPtr<JSC::Uint8ClampedArray> getRGBAImageData() const override;
     47    WEBCORE_EXPORT RefPtr<JSC::Uint8ClampedArray> getRGBAImageData() const override;
    4848
    4949    MediaTime presentationTime() const override;
    … …  
    6060    PlatformSample platformSample() override;
    6161    std::optional<ByteRange> byteRange() const override;
    62     void dump(PrintStream&) const override;
     62    WEBCORE_EXPORT void dump(PrintStream&) const override;
    6363    void offsetTimestampsBy(const MediaTime&) override;
    6464    void setTimestamps(const MediaTime&, const MediaTime&) override;
    65     bool isDivisable() const override;
    66     std::pair<RefPtr<MediaSample>, RefPtr<MediaSample>> divide(const MediaTime& presentationTime, UseEndTime) override;
    67     Ref<MediaSample> createNonDisplayingCopy() const override;
     65    WEBCORE_EXPORT bool isDivisable() const override;
     66    WEBCORE_EXPORT std::pair<RefPtr<MediaSample>, RefPtr<MediaSample>> divide(const MediaTime& presentationTime, UseEndTime) override;
     67    WEBCORE_EXPORT Ref<MediaSample> createNonDisplayingCopy() const override;
    6868
    6969    VideoRotation videoRotation() const override { return m_rotation; }
    7070    bool videoMirrored() const override { return m_mirrored; }
    71     uint32_t videoPixelFormat() const final;
     71    WEBCORE_EXPORT uint32_t videoPixelFormat() const final;
    7272
    7373    CMSampleBufferRef sampleBuffer() const { return m_sample.get(); }
    … …  
    7979
    8080protected:
    81     MediaSampleAVFObjC(RetainPtr<CMSampleBufferRef>&& sample)
    82         : m_sample(WTFMove(sample))
    83     {
    84     }
    85     MediaSampleAVFObjC(CMSampleBufferRef sample)
    86         : m_sample(sample)
    87     {
    88     }
    89     MediaSampleAVFObjC(CMSampleBufferRef sample, AtomString trackID)
    90         : m_sample(sample)
    91         , m_id(trackID)
    92     {
    93     }
    94     MediaSampleAVFObjC(CMSampleBufferRef sample, uint64_t trackID)
    95         : m_sample(sample)
    96         , m_id(AtomString::number(trackID))
    97     {
    98     }
    99     MediaSampleAVFObjC(CMSampleBufferRef sample, VideoRotation rotation, bool mirrored)
    100         : m_sample(sample)
    101         , m_rotation(rotation)
    102         , m_mirrored(mirrored)
    103     {
    104     }
     81    WEBCORE_EXPORT MediaSampleAVFObjC(RetainPtr<CMSampleBufferRef>&&);
     82    WEBCORE_EXPORT MediaSampleAVFObjC(CMSampleBufferRef);
     83    WEBCORE_EXPORT MediaSampleAVFObjC(CMSampleBufferRef, AtomString trackID);
     84    WEBCORE_EXPORT MediaSampleAVFObjC(CMSampleBufferRef, uint64_t trackID);
     85    WEBCORE_EXPORT MediaSampleAVFObjC(CMSampleBufferRef, VideoRotation, bool mirrored);
     86    WEBCORE_EXPORT virtual ~MediaSampleAVFObjC();
    10587
    10688    std::optional<MediaSample::ByteRange> byteRangeForAttachment(CFStringRef key) const;
    10789
    108     virtual ~MediaSampleAVFObjC() = default;
    10990    RetainPtr<CMSampleBufferRef> m_sample;
    11091    AtomString m_id;
    … …  
    11495
    11596} // namespace WebCore
    116 

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm

    r279335 r280416  
    4646namespace WebCore {
    4747
     48MediaSampleAVFObjC::MediaSampleAVFObjC(RetainPtr<CMSampleBufferRef>&& sample)
     49    : m_sample(WTFMove(sample))
     50{
     51}
     52MediaSampleAVFObjC::MediaSampleAVFObjC(CMSampleBufferRef sample)
     53    : m_sample(sample)
     54{
     55}
     56MediaSampleAVFObjC::MediaSampleAVFObjC(CMSampleBufferRef sample, AtomString trackID)
     57    : m_sample(sample)
     58    , m_id(trackID)
     59{
     60}
     61MediaSampleAVFObjC::MediaSampleAVFObjC(CMSampleBufferRef sample, uint64_t trackID)
     62    : m_sample(sample)
     63    , m_id(AtomString::number(trackID))
     64{
     65}
     66MediaSampleAVFObjC::MediaSampleAVFObjC(CMSampleBufferRef sample, VideoRotation rotation, bool mirrored)
     67    : m_sample(sample)
     68    , m_rotation(rotation)
     69    , m_mirrored(mirrored)
     70{
     71}
     72
     73MediaSampleAVFObjC::~MediaSampleAVFObjC() = default;
     74
    4875RefPtr<MediaSampleAVFObjC> MediaSampleAVFObjC::createImageSample(PixelBuffer&& pixelBuffer)
    4976{

  • trunk/Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.mm

    r279335 r280416  
    3232#import <CoreMedia/CMFormatDescription.h>
    3333#import <pal/avfoundation/MediaTimeAVFoundation.h>
    34 #import <pal/cf/CoreMediaSoftLink.h>
    3534#import <wtf/MainThread.h>
    3635#import <wtf/MediaTime.h>
    3736#import <wtf/MonotonicTime.h>
     37#import <wtf/RunLoop.h>
    3838#import <wtf/StringPrintStream.h>
    3939#import <wtf/Vector.h>
    … …  
    4343#import "CoreVideoSoftLink.h"
    4444#import "VideoToolboxSoftLink.h"
     45#import <pal/cf/CoreMediaSoftLink.h>
    4546
    4647// Equivalent to WTF_DECLARE_CF_TYPE_TRAIT(CMSampleBuffer);

  • trunk/Source/WebCore/platform/graphics/cocoa/WebMAudioUtilitiesCocoa.h

    r274902 r280416  
    3535namespace WebCore {
    3636
    37 bool isVorbisDecoderAvailable();
     37WEBCORE_EXPORT bool isVorbisDecoderAvailable();
     38WEBCORE_EXPORT bool registerVorbisDecoderIfNeeded();
    3839RetainPtr<CMFormatDescriptionRef> createVorbisAudioFormatDescription(size_t, const void*);
    3940
    … …  
    5354};
    5455
    55 bool isOpusDecoderAvailable();
     56WEBCORE_EXPORT bool isOpusDecoderAvailable();
     57WEBCORE_EXPORT bool registerOpusDecoderIfNeeded();
    5658bool parseOpusPrivateData(size_t privateDataSize, const void* privateData, size_t frameDataSize, const void* frameData, OpusCookieContents&);
    5759RetainPtr<CMFormatDescriptionRef> createOpusAudioFormatDescription(const OpusCookieContents&);

  • trunk/Source/WebCore/platform/graphics/cocoa/WebMAudioUtilitiesCocoa.mm

    r274902 r280416  
    3333#import "MediaUtilities.h"
    3434#import "PlatformMediaSessionManager.h"
     35#import <AudioToolbox/AudioCodec.h>
    3536#import <AudioToolbox/AudioComponent.h>
    3637#import <AudioToolbox/AudioFormat.h>
    … …  
    3940#import <wtf/FlipBytes.h>
    4041#import <wtf/Seconds.h>
    41 
    4242#if ENABLE(OPUS)
    4343#import <libwebrtc/opus_defines.h>
    4444#endif
     45#import <pal/cf/AudioToolboxSoftLink.h>
    4546
    4647namespace WebCore {
    … …  
    4950static bool registerDecoderFactory(const char* decoderName, OSType decoderType)
    5051{
     52    AudioComponentDescription desc { kAudioDecoderComponentType, decoderType, 'appl', kAudioComponentFlag_SandboxSafe, 0 };
     53    AudioComponent comp = PAL::AudioComponentFindNext(0, &desc);
     54    if (comp)
     55        return true; // Already registered.
     56
    5157    constexpr char audioComponentsDylib[] = "/System/Library/Components/AudioCodecs.component/Contents/MacOS/AudioCodecs";
    5258    void *handle = dlopen(audioComponentsDylib, RTLD_LAZY | RTLD_LOCAL);
    … …  
    5864        return false;
    5965
    60     AudioComponentDescription desc { 'adec', decoderType, 'appl', kAudioComponentFlag_SandboxSafe, 0 };
    6166    if (!AudioComponentRegister(&desc, CFSTR(""), 0, decoderFactory)) {
    6267        dlclose(handle);
    … …  
    7277    asbd.mFormatID = formatID;
    7378    uint32_t size = sizeof(asbd);
    74     auto error = AudioFormatGetProperty(kAudioFormatProperty_FormatInfo, magicCookie.size(), magicCookie.data(), &size, &asbd);
     79    auto error = PAL::AudioFormatGetProperty(kAudioFormatProperty_FormatInfo, magicCookie.size(), magicCookie.data(), &size, &asbd);
    7580    if (error) {
    7681        RELEASE_LOG_ERROR(Media, "createAudioFormatDescriptionForFormat failed with error %d (%.4s)", error, (char *)&error);
    … …  
    327332bool isOpusDecoderAvailable()
    328333{
    329     static bool available;
    330 
    331334#if ENABLE(OPUS) && PLATFORM(MAC)
    332335    if (!PlatformMediaSessionManager::opusDecoderEnabled())
    333336        return false;
    334337
     338    return registerOpusDecoderIfNeeded();
     339#else
     340    return false;
     341#endif
     342}
     343
     344bool registerOpusDecoderIfNeeded()
     345{
     346#if ENABLE(OPUS) && PLATFORM(MAC)
     347    static bool available;
     348
    335349    static dispatch_once_t onceToken;
    336350    dispatch_once(&onceToken, ^{
    337         available = registerDecoderFactory("ACOpusDecoderFactory", 'opus');
     351        available = registerDecoderFactory("ACOpusDecoderFactory", kAudioFormatOpus);
    338352    });
    339 #endif
    340353
    341354    return available;
     355#else
     356    return false;
     357#endif
    342358}
    343359
    … …  
    419435bool isVorbisDecoderAvailable()
    420436{
    421     static bool available;
    422 
    423437#if ENABLE(VORBIS) && PLATFORM(MAC)
    424438    if (!PlatformMediaSessionManager::vorbisDecoderEnabled())
    425439        return false;
     440
     441    return registerVorbisDecoderIfNeeded();
     442#else
     443    return false;
     444#endif
     445}
     446
     447bool registerVorbisDecoderIfNeeded()
     448{
     449#if ENABLE(VORBIS) && PLATFORM(MAC)
     450    static bool available;
    426451
    427452    static dispatch_once_t onceToken;
    … …  
    429454        available = registerDecoderFactory("ACVorbisDecoderFactory", 'vorb');
    430455    });
    431 #endif
    432456
    433457    return available;
     458#else
     459    return false;
     460#endif
    434461}
    435462

  • trunk/Source/WebKit/ChangeLog

    r280401 r280416  
     12021-07-28  Jean-Yves Avenard  <jya@apple.com>
     2
     3        [WebAudio] Add webm/opus container support
     4        https://bugs.webkit.org/show_bug.cgi?id=227110
     5        <rdar://problem/79437067>
     6
     7        Reviewed by Jer Noble.
     8
     9        Add default value for webm_webaudio feature flag for the different platforms.
     10        Disabled by default on all platforms.
     11        Fly-by fix: don't make webm parser dependent on VP9 support.
     12
     13        * FeatureFlags/WebKit-appletvos.plist:
     14        * FeatureFlags/WebKit-ios.plist:
     15        * FeatureFlags/WebKit-macos.plist:
     16        * FeatureFlags/WebKit-watchos.plist:
     17        * GPUProcess/GPUProcess.h:
     18        * GPUProcess/GPUProcess.messages.in:
     19        * Shared/WebPreferencesDefaultValues.cpp:
     20        (WebKit::defaultWebMWebAudioEnabled):
     21        * Shared/WebPreferencesDefaultValues.h:
     22        * UIProcess/Cocoa/WebProcessProxyCocoa.mm:
     23        (WebKit::WebProcessProxy::sendAudioComponentRegistrations):
     24
    1252021-07-28  Myles C. Maxfield  <mmaxfield@apple.com>
    226

  • trunk/Source/WebKit/FeatureFlags/WebKit-appletvos.plist

    r275239 r280416  
    8888        <true/>
    8989    </dict>
     90    <key>webm_webaudio</key>
     91    <dict>
     92        <key>Enabled</key>
     93        <true/>
     94    </dict>
    9095    <key>RB_full_manage_WK_jetsam</key>
    9196    <dict>

  • trunk/Source/WebKit/FeatureFlags/WebKit-ios.plist

    r275239 r280416  
    8888        <true/>
    8989    </dict>
     90    <key>webm_webaudio</key>
     91    <dict>
     92        <key>Enabled</key>
     93        <true/>
     94    </dict>
    9095    <key>RB_full_manage_WK_jetsam</key>
    9196    <dict>

  • trunk/Source/WebKit/FeatureFlags/WebKit-macos.plist

    r275239 r280416  
    8888        <true/>
    8989    </dict>
     90    <key>webm_webaudio</key>
     91    <dict>
     92        <key>Enabled</key>
     93        <true/>
     94    </dict>
    9095    <key>RB_full_manage_WK_jetsam</key>
    9196    <dict>

  • trunk/Source/WebKit/FeatureFlags/WebKit-watchos.plist

    r275239 r280416  
    8888        <true/>
    8989    </dict>
     90    <key>webm_webaudio</key>
     91    <dict>
     92        <key>Enabled</key>
     93        <true/>
     94    </dict>
    9095    <key>RB_full_manage_WK_jetsam</key>
    9196    <dict>

  • trunk/Source/WebKit/GPUProcess/GPUProcess.cpp

    r279926 r280416  
    452452#endif
    453453
    454 #if ENABLE(MEDIA_SOURCE) && ENABLE(VP9)
     454#if ENABLE(MEDIA_SOURCE)
    455455void GPUProcess::setWebMParserEnabled(bool enabled)
    456456{

  • trunk/Source/WebKit/GPUProcess/GPUProcess.h

    r279926 r280416  
    142142#endif
    143143
    144 #if ENABLE(MEDIA_SOURCE) && ENABLE(VP9)
     144#if ENABLE(MEDIA_SOURCE)
    145145    void setWebMParserEnabled(bool);
    146146#endif
    … …  
    195195    bool m_enableVP9SWDecoder { false };
    196196#endif
    197 #if ENABLE(MEDIA_SOURCE) && ENABLE(VP9)
     197#if ENABLE(MEDIA_SOURCE)
    198198    bool m_webMParserEnabled { false };
    199199#endif

  • trunk/Source/WebKit/GPUProcess/GPUProcess.messages.in

    r279912 r280416  
    5252#endif
    5353
    54 #if ENABLE(MEDIA_SOURCE) && ENABLE(VP9)
     54#if ENABLE(MEDIA_SOURCE)
    5555    SetWebMParserEnabled(bool enabled);
    5656#endif

  • trunk/Source/WebKit/Shared/WebPreferencesDefaultValues.cpp

    r280327 r280416  
    333333}
    334334
     335bool defaultWebMWebAudioEnabled()
     336{
     337#if HAVE(SYSTEM_FEATURE_FLAGS)
     338    return isFeatureFlagEnabled("webm_webaudio");
     339#endif
     340
     341    return false;
     342}
     343
    335344#endif // ENABLE(MEDIA_SOURCE)
    336345

  • trunk/Source/WebKit/Shared/WebPreferencesDefaultValues.h

    r280327 r280416  
    112112#if ENABLE(MEDIA_SOURCE)
    113113bool defaultWebMParserEnabled();
     114bool defaultWebMWebAudioEnabled();
    114115#endif
    115116

  • trunk/Source/WebKit/UIProcess/Cocoa/WebProcessProxyCocoa.mm

    r280258 r280416  
    3939#import "WebProcessPool.h"
    4040#import <WebCore/RuntimeApplicationChecks.h>
     41#import <WebCore/WebMAudioUtilitiesCocoa.h>
    4142#import <sys/sysctl.h>
    4243#import <wtf/NeverDestroyed.h>
    … …  
    296297    dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), [protectedThis = makeRef(*this)] () mutable {
    297298        CFDataRef registrations { nullptr };
     299
     300        WebCore::registerOpusDecoderIfNeeded();
     301        WebCore::registerVorbisDecoderIfNeeded();
    298302        if (noErr != AudioComponentFetchServerRegistrations(&registrations) || !registrations)
    299303            return;

  • trunk/Source/WebKitLegacy/mac/WebView/WebPreferencesDefaultValues.h

    r279974 r280416  
    8888#if ENABLE(MEDIA_SOURCE)
    8989bool defaultWebMParserEnabled();
     90bool defaultWebMWebAudioEnabled();
    9091#endif
    9192

  • trunk/Source/WebKitLegacy/mac/WebView/WebPreferencesDefaultValues.mm

    r279974 r280416  
    299299}
    300300
     301bool defaultWebMWebAudioEnabled()
     302{
     303#if HAVE(SYSTEM_FEATURE_FLAGS)
     304    return isFeatureFlagEnabled("webm_webaudio");
     305#endif
     306
     307    return false;
     308}
     309
    301310#endif // ENABLE(MEDIA_SOURCE)
    302311
Note: See TracChangeset for help on using the changeset viewer.