⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 277364 in webkit


Ignore:
Timestamp:
May 12, 2021, 1:57:38 AM (5 years ago)
Author:
youenn@apple.com
Message:

Introduce an internal unit to render audio MediaStreamTrack(s)
​https://bugs.webkit.org/show_bug.cgi?id=225601

Reviewed by Eric Carlson.

Source/WebCore:

Move files from platform/mediastream/mac to platform/mediastream/cocoa since they are used in both iOS and macOS.
Move Audio Unit functionality out of AudioMediaStreamTrackRendererUnit.
AudioMediaStreamTrackRendererUnit is responsible to manage sources and do the mixing.
To actually render audio, it will use an InternalUnit, which is currently done in process.

A future work will add support for a remote InternalUnit, that would run on GPUProcess.
To prepare for that, a callback allows customizing the Internal Unit creation.
We also change renderer start to be asynchronous, since it requires fetching the audio description from GPUProcess.

No change of behavior, covered by existing tests.

  • SourcesCocoa.txt:
  • WebCore.xcodeproj/project.pbxproj:
  • platform/audio/cocoa/AudioSampleBufferList.h:
  • platform/mediastream/AudioMediaStreamTrackRenderer.h:
  • platform/mediastream/AudioTrackPrivateMediaStream.cpp:

(WebCore::AudioTrackPrivateMediaStream::startRenderer):

  • platform/mediastream/cocoa/AudioMediaStreamTrackRendererCocoa.cpp: Renamed from Source/WebCore/platform/mediastream/mac/AudioMediaStreamTrackRendererCocoa.cpp.

(WebCore::AudioMediaStreamTrackRendererCocoa::start):

  • platform/mediastream/cocoa/AudioMediaStreamTrackRendererCocoa.h: Copied from Source/WebCore/platform/mediastream/mac/AudioMediaStreamTrackRendererCocoa.h.
  • platform/mediastream/cocoa/AudioMediaStreamTrackRendererInternalUnit.cpp: Renamed from Source/WebCore/platform/mediastream/mac/AudioMediaStreamTrackRendererUnit.cpp.

(WebCore::AudioMediaStreamTrackRendererInternalUnit::createLocalInternalUnit):
(WebCore::LocalAudioMediaStreamTrackRendererInternalUnit::LocalAudioMediaStreamTrackRendererInternalUnit):
(WebCore::LocalAudioMediaStreamTrackRendererInternalUnit::retrieveFormatDescription):
(WebCore::LocalAudioMediaStreamTrackRendererInternalUnit::setAudioOutputDevice):
(WebCore::LocalAudioMediaStreamTrackRendererInternalUnit::start):
(WebCore::LocalAudioMediaStreamTrackRendererInternalUnit::stop):
(WebCore::LocalAudioMediaStreamTrackRendererInternalUnit::createAudioUnitIfNeeded):
(WebCore::LocalAudioMediaStreamTrackRendererInternalUnit::renderingCallback):

  • platform/mediastream/cocoa/AudioMediaStreamTrackRendererInternalUnit.h: Renamed from Source/WebCore/platform/mediastream/mac/AudioMediaStreamTrackRendererCocoa.h.
  • platform/mediastream/cocoa/AudioMediaStreamTrackRendererUnit.cpp: Added.

(WebCore::getCreateInternalUnitFunction):
(WebCore::AudioMediaStreamTrackRendererUnit::setCreateInternalUnitFunction):
(WebCore::createInternalUnit):
(WebCore::AudioMediaStreamTrackRendererUnit::singleton):
(WebCore::AudioMediaStreamTrackRendererUnit::AudioMediaStreamTrackRendererUnit):
(WebCore::AudioMediaStreamTrackRendererUnit::~AudioMediaStreamTrackRendererUnit):
(WebCore::AudioMediaStreamTrackRendererUnit::setAudioOutputDevice):
(WebCore::AudioMediaStreamTrackRendererUnit::addSource):
(WebCore::AudioMediaStreamTrackRendererUnit::removeSource):
(WebCore::AudioMediaStreamTrackRendererUnit::start):
(WebCore::AudioMediaStreamTrackRendererUnit::stop):
(WebCore::AudioMediaStreamTrackRendererUnit::retrieveFormatDescription):
(WebCore::AudioMediaStreamTrackRendererUnit::render):

  • platform/mediastream/cocoa/AudioMediaStreamTrackRendererUnit.h: Renamed from Source/WebCore/platform/mediastream/mac/AudioMediaStreamTrackRendererUnit.h.

Source/WebKit:

Update according new WebCore API.

  • GPUProcess/webrtc/RemoteAudioMediaStreamTrackRenderer.cpp:

(WebKit::RemoteAudioMediaStreamTrackRenderer::start):

  • WebProcess/GPU/webrtc/AudioMediaStreamTrackRenderer.cpp:

(WebKit::AudioMediaStreamTrackRenderer::start):

  • WebProcess/GPU/webrtc/AudioMediaStreamTrackRenderer.h:
Location:
trunk/Source
Files:
2 added
10 edited
1 copied
4 moved

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebCore/ChangeLog

    r277361 r277364  
     12021-05-12  Youenn Fablet  <youenn@apple.com>
     2
     3        Introduce an internal unit to render audio MediaStreamTrack(s)
     4        https://bugs.webkit.org/show_bug.cgi?id=225601
     5
     6        Reviewed by Eric Carlson.
     7
     8        Move files from platform/mediastream/mac to platform/mediastream/cocoa since they are used in both iOS and macOS.
     9        Move Audio Unit functionality out of AudioMediaStreamTrackRendererUnit.
     10        AudioMediaStreamTrackRendererUnit is responsible to manage sources and do the mixing.
     11        To actually render audio, it will use an InternalUnit, which is currently done in process.
     12
     13        A future work will add support for a remote InternalUnit, that would run on GPUProcess.
     14        To prepare for that, a callback allows customizing the Internal Unit creation.
     15        We also change renderer start to be asynchronous, since it requires fetching the audio description from GPUProcess.
     16
     17        No change of behavior, covered by existing tests.
     18
     19        * SourcesCocoa.txt:
     20        * WebCore.xcodeproj/project.pbxproj:
     21        * platform/audio/cocoa/AudioSampleBufferList.h:
     22        * platform/mediastream/AudioMediaStreamTrackRenderer.h:
     23        * platform/mediastream/AudioTrackPrivateMediaStream.cpp:
     24        (WebCore::AudioTrackPrivateMediaStream::startRenderer):
     25        * platform/mediastream/cocoa/AudioMediaStreamTrackRendererCocoa.cpp: Renamed from Source/WebCore/platform/mediastream/mac/AudioMediaStreamTrackRendererCocoa.cpp.
     26        (WebCore::AudioMediaStreamTrackRendererCocoa::start):
     27        * platform/mediastream/cocoa/AudioMediaStreamTrackRendererCocoa.h: Copied from Source/WebCore/platform/mediastream/mac/AudioMediaStreamTrackRendererCocoa.h.
     28        * platform/mediastream/cocoa/AudioMediaStreamTrackRendererInternalUnit.cpp: Renamed from Source/WebCore/platform/mediastream/mac/AudioMediaStreamTrackRendererUnit.cpp.
     29        (WebCore::AudioMediaStreamTrackRendererInternalUnit::createLocalInternalUnit):
     30        (WebCore::LocalAudioMediaStreamTrackRendererInternalUnit::LocalAudioMediaStreamTrackRendererInternalUnit):
     31        (WebCore::LocalAudioMediaStreamTrackRendererInternalUnit::retrieveFormatDescription):
     32        (WebCore::LocalAudioMediaStreamTrackRendererInternalUnit::setAudioOutputDevice):
     33        (WebCore::LocalAudioMediaStreamTrackRendererInternalUnit::start):
     34        (WebCore::LocalAudioMediaStreamTrackRendererInternalUnit::stop):
     35        (WebCore::LocalAudioMediaStreamTrackRendererInternalUnit::createAudioUnitIfNeeded):
     36        (WebCore::LocalAudioMediaStreamTrackRendererInternalUnit::renderingCallback):
     37        * platform/mediastream/cocoa/AudioMediaStreamTrackRendererInternalUnit.h: Renamed from Source/WebCore/platform/mediastream/mac/AudioMediaStreamTrackRendererCocoa.h.
     38        * platform/mediastream/cocoa/AudioMediaStreamTrackRendererUnit.cpp: Added.
     39        (WebCore::getCreateInternalUnitFunction):
     40        (WebCore::AudioMediaStreamTrackRendererUnit::setCreateInternalUnitFunction):
     41        (WebCore::createInternalUnit):
     42        (WebCore::AudioMediaStreamTrackRendererUnit::singleton):
     43        (WebCore::AudioMediaStreamTrackRendererUnit::AudioMediaStreamTrackRendererUnit):
     44        (WebCore::AudioMediaStreamTrackRendererUnit::~AudioMediaStreamTrackRendererUnit):
     45        (WebCore::AudioMediaStreamTrackRendererUnit::setAudioOutputDevice):
     46        (WebCore::AudioMediaStreamTrackRendererUnit::addSource):
     47        (WebCore::AudioMediaStreamTrackRendererUnit::removeSource):
     48        (WebCore::AudioMediaStreamTrackRendererUnit::start):
     49        (WebCore::AudioMediaStreamTrackRendererUnit::stop):
     50        (WebCore::AudioMediaStreamTrackRendererUnit::retrieveFormatDescription):
     51        (WebCore::AudioMediaStreamTrackRendererUnit::render):
     52        * platform/mediastream/cocoa/AudioMediaStreamTrackRendererUnit.h: Renamed from Source/WebCore/platform/mediastream/mac/AudioMediaStreamTrackRendererUnit.h.
     53
    1542021-05-12  Youenn Fablet  <youenn@apple.com>
    255

  • trunk/Source/WebCore/SourcesCocoa.txt

    r277330 r277364  
    528528platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm
    529529platform/mediarecorder/cocoa/VideoSampleBufferCompressor.mm
     530platform/mediastream/cocoa/AudioMediaStreamTrackRendererCocoa.cpp
     531platform/mediastream/cocoa/AudioMediaStreamTrackRendererInternalUnit.cpp
     532platform/mediastream/cocoa/AudioMediaStreamTrackRendererUnit.cpp
    530533platform/mediastream/ios/AVAudioSessionCaptureDevice.mm
    531534platform/mediastream/ios/AVAudioSessionCaptureDeviceManager.mm @no-unify
    … …  
    534537platform/mediastream/mac/AVCaptureDeviceManager.mm @no-unify
    535538platform/mediastream/mac/AVVideoCaptureSource.mm @no-unify
    536 platform/mediastream/mac/AudioMediaStreamTrackRendererCocoa.cpp
    537 platform/mediastream/mac/AudioMediaStreamTrackRendererUnit.cpp
    538539platform/mediastream/mac/BaseAudioSharedUnit.cpp
    539540platform/mediastream/mac/CoreAudioCaptureDevice.cpp

  • trunk/Source/WebCore/WebCore.xcodeproj/project.pbxproj

    r277330 r277364  
    11761176                41BBA866257E373B00AC7F6D /* SFrameUtils.h in Headers */ = {isa = PBXBuildFile; fileRef = 41209E24257A2FBB00120ACA /* SFrameUtils.h */; settings = {ATTRIBUTES = (Private, ); }; };
    11771177                41BF204922BA7BE80004F812 /* RealtimeVideoSource.h in Headers */ = {isa = PBXBuildFile; fileRef = 41BF204022B947160004F812 /* RealtimeVideoSource.h */; settings = {ATTRIBUTES = (Private, ); }; };
     1178                41C3B8A12649B1E0004ED4DE /* AudioMediaStreamTrackRendererInternalUnit.h in Headers */ = {isa = PBXBuildFile; fileRef = 41C3B89F2649B1B7004ED4DE /* AudioMediaStreamTrackRendererInternalUnit.h */; settings = {ATTRIBUTES = (Private, ); }; };
     1179                41C3B8A22649B1E5004ED4DE /* AudioMediaStreamTrackRendererUnit.h in Headers */ = {isa = PBXBuildFile; fileRef = 41C3B89B2649B1B6004ED4DE /* AudioMediaStreamTrackRendererUnit.h */; settings = {ATTRIBUTES = (Private, ); }; };
    11781180                41C760B10EDE03D300C1655F /* ScriptState.h in Headers */ = {isa = PBXBuildFile; fileRef = 41C760B00EDE03D300C1655F /* ScriptState.h */; settings = {ATTRIBUTES = (Private, ); }; };
    11791181                41CB840125CAB7B30010E2B1 /* RealtimeIncomingVideoSourceCocoa.h in Headers */ = {isa = PBXBuildFile; fileRef = 41CB840025CAB7B00010E2B1 /* RealtimeIncomingVideoSourceCocoa.h */; settings = {ATTRIBUTES = (Private, ); }; };
    … …  
    76987700                41024FC823CF254F00FDF98E /* SampleBufferDisplayLayer.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = SampleBufferDisplayLayer.cpp; sourceTree = "<group>"; };
    76997701                4107908A1FC3E4F20061B27A /* ClientOrigin.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ClientOrigin.h; sourceTree = "<group>"; };
    7700                 41087F01248E59C800680E91 /* AudioMediaStreamTrackRendererUnit.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = AudioMediaStreamTrackRendererUnit.cpp; sourceTree = "<group>"; };
    7701                 41087F03248E59C900680E91 /* AudioMediaStreamTrackRendererUnit.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AudioMediaStreamTrackRendererUnit.h; sourceTree = "<group>"; };
    77027702                410938282347799A009428BA /* JSAbortControllerCustom.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = JSAbortControllerCustom.cpp; sourceTree = "<group>"; };
    77037703                4109382C2347850E009428BA /* JSAbortSignalCustom.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = JSAbortSignalCustom.cpp; sourceTree = "<group>"; };
    … …  
    78657865                4162A4551011464700DFF3ED /* JSDedicatedWorkerGlobalScope.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = JSDedicatedWorkerGlobalScope.cpp; sourceTree = "<group>"; };
    78667866                4162A4561011464700DFF3ED /* JSDedicatedWorkerGlobalScope.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JSDedicatedWorkerGlobalScope.h; sourceTree = "<group>"; };
    7867                 416A069223CCAA9300347109 /* AudioMediaStreamTrackRendererCocoa.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AudioMediaStreamTrackRendererCocoa.h; sourceTree = "<group>"; };
    7868                 416A069323CCAA9400347109 /* AudioMediaStreamTrackRendererCocoa.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = AudioMediaStreamTrackRendererCocoa.cpp; sourceTree = "<group>"; };
    78697867                416A069423CCAA9E00347109 /* AudioTrackPrivateMediaStream.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = AudioTrackPrivateMediaStream.cpp; sourceTree = "<group>"; };
    78707868                416A069523CCAA9E00347109 /* AudioMediaStreamTrackRenderer.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = AudioMediaStreamTrackRenderer.cpp; sourceTree = "<group>"; };
    … …  
    80248022                41BF204022B947160004F812 /* RealtimeVideoSource.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = RealtimeVideoSource.h; sourceTree = "<group>"; };
    80258023                41BF204222B947170004F812 /* RealtimeVideoSource.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = RealtimeVideoSource.cpp; sourceTree = "<group>"; };
     8024                41C3B89B2649B1B6004ED4DE /* AudioMediaStreamTrackRendererUnit.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = AudioMediaStreamTrackRendererUnit.h; sourceTree = "<group>"; };
     8025                41C3B89C2649B1B6004ED4DE /* AudioMediaStreamTrackRendererCocoa.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = AudioMediaStreamTrackRendererCocoa.h; sourceTree = "<group>"; };
     8026                41C3B89D2649B1B6004ED4DE /* AudioMediaStreamTrackRendererCocoa.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = AudioMediaStreamTrackRendererCocoa.cpp; sourceTree = "<group>"; };
     8027                41C3B89E2649B1B7004ED4DE /* AudioMediaStreamTrackRendererInternalUnit.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = AudioMediaStreamTrackRendererInternalUnit.cpp; sourceTree = "<group>"; };
     8028                41C3B89F2649B1B7004ED4DE /* AudioMediaStreamTrackRendererInternalUnit.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = AudioMediaStreamTrackRendererInternalUnit.h; sourceTree = "<group>"; };
     8029                41C3B8A02649B1B7004ED4DE /* AudioMediaStreamTrackRendererUnit.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = AudioMediaStreamTrackRendererUnit.cpp; sourceTree = "<group>"; };
    80268030                41C760B00EDE03D300C1655F /* ScriptState.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ScriptState.h; sourceTree = "<group>"; };
    80278031                41C7E1051E6A54360027B4DE /* CanvasCaptureMediaStreamTrack.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = CanvasCaptureMediaStreamTrack.cpp; sourceTree = "<group>"; };
    … …  
    1754917553                        isa = PBXGroup;
    1755017554                        children = (
     17555                                41C3B89A2649B18E004ED4DE /* cocoa */,
    1755117556                                CDC6751F1EAEA99600727C84 /* ios */,
    1755217557                                415747421E38698000E914D8 /* libwebrtc */,
    … …  
    1763317638                        isa = PBXGroup;
    1763417639                        children = (
    17635                                 416A069323CCAA9400347109 /* AudioMediaStreamTrackRendererCocoa.cpp */,
    17636                                 416A069223CCAA9300347109 /* AudioMediaStreamTrackRendererCocoa.h */,
    17637                                 41087F01248E59C800680E91 /* AudioMediaStreamTrackRendererUnit.cpp */,
    17638                                 41087F03248E59C900680E91 /* AudioMediaStreamTrackRendererUnit.h */,
    1763917640                                070363DA181A1CDC00C074A5 /* AVCaptureDeviceManager.h */,
    1764017641                                070363DB181A1CDC00C074A5 /* AVCaptureDeviceManager.mm */,
    … …  
    1987319874                        );
    1987419875                        name = Streams;
     19876                        sourceTree = "<group>";
     19877                };
     19878                41C3B89A2649B18E004ED4DE /* cocoa */ = {
     19879                        isa = PBXGroup;
     19880                        children = (
     19881                                41C3B89D2649B1B6004ED4DE /* AudioMediaStreamTrackRendererCocoa.cpp */,
     19882                                41C3B89C2649B1B6004ED4DE /* AudioMediaStreamTrackRendererCocoa.h */,
     19883                                41C3B89E2649B1B7004ED4DE /* AudioMediaStreamTrackRendererInternalUnit.cpp */,
     19884                                41C3B89F2649B1B7004ED4DE /* AudioMediaStreamTrackRendererInternalUnit.h */,
     19885                                41C3B8A02649B1B7004ED4DE /* AudioMediaStreamTrackRendererUnit.cpp */,
     19886                                41C3B89B2649B1B6004ED4DE /* AudioMediaStreamTrackRendererUnit.h */,
     19887                        );
     19888                        path = cocoa;
    1987519889                        sourceTree = "<group>";
    1987619890                };
    … …  
    3155331567                                FD31601012B0267600C1A359 /* AudioListener.h in Headers */,
    3155431568                                416A06A123CCAD0300347109 /* AudioMediaStreamTrackRenderer.h in Headers */,
     31569                                41C3B8A12649B1E0004ED4DE /* AudioMediaStreamTrackRendererInternalUnit.h in Headers */,
     31570                                41C3B8A22649B1E5004ED4DE /* AudioMediaStreamTrackRendererUnit.h in Headers */,
    3155531571                                FD31601312B0267600C1A359 /* AudioNode.h in Headers */,
    3155631572                                FD31601612B0267600C1A359 /* AudioNodeInput.h in Headers */,

  • trunk/Source/WebCore/platform/audio/cocoa/AudioSampleBufferList.h

    r268521 r277364  
    7373    void reset();
    7474
    75     static void zeroABL(AudioBufferList&, size_t);
     75    WEBCORE_EXPORT static void zeroABL(AudioBufferList&, size_t);
    7676    void zero();
    7777

  • trunk/Source/WebCore/platform/mediastream/AudioMediaStreamTrackRenderer.h

    r272213 r277364  
    4646    virtual ~AudioMediaStreamTrackRenderer() = default;
    4747
    48     virtual void start() = 0;
     48    virtual void start(CompletionHandler<void()>&&) = 0;
    4949    virtual void stop() = 0;
    5050    virtual void clear() = 0;

  • trunk/Source/WebCore/platform/mediastream/AudioTrackPrivateMediaStream.cpp

    r276197 r277364  
    160160{
    161161    ASSERT(isMainThread());
    162     if (m_isPlaying)
     162    if (m_isPlaying || !m_renderer)
    163163        return;
    164164
    165165    m_isPlaying = true;
    166     if (m_renderer)
    167         m_renderer->start();
    168     m_audioSource->addAudioSampleObserver(*this);
     166    m_renderer->start([protectedThis = makeRef(*this)] {
     167        if (protectedThis->m_isPlaying)
     168            protectedThis->m_audioSource->addAudioSampleObserver(protectedThis.get());
     169    });
    169170}
    170171

  • trunk/Source/WebCore/platform/mediastream/cocoa/AudioMediaStreamTrackRendererCocoa.cpp

    r277363 r277364  
    11/*
    2  * Copyright (C) 2017-2020 Apple Inc. All rights reserved.
     2 * Copyright (C) 2017-2021 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    4040AudioMediaStreamTrackRendererCocoa::~AudioMediaStreamTrackRendererCocoa() = default;
    4141
    42 void AudioMediaStreamTrackRendererCocoa::start()
     42void AudioMediaStreamTrackRendererCocoa::start(CompletionHandler<void()>&& callback)
    4343{
    4444    clear();
    4545
    46     if (auto* formatDescription = AudioMediaStreamTrackRendererUnit::singleton().formatDescription())
    47         m_outputDescription = makeUnique<CAAudioStreamDescription>(*formatDescription);
     46    AudioMediaStreamTrackRendererUnit::singleton().retrieveFormatDescription([weakThis = makeWeakPtr(this), callback = WTFMove(callback)](auto* formatDescription) mutable {
     47        if (weakThis)
     48            weakThis->m_outputDescription = makeUnique<CAAudioStreamDescription>(*formatDescription);
     49        callback();
     50    });
    4851}
    4952

  • trunk/Source/WebCore/platform/mediastream/cocoa/AudioMediaStreamTrackRendererCocoa.h

    r277363 r277364  
    11/*
    2  * Copyright (C) 2017 Apple Inc. All rights reserved.
     2 * Copyright (C) 2017-2021 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    5050    // AudioMediaStreamTrackRenderer
    5151    void pushSamples(const WTF::MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;
    52     void start() final;
     52    void start(CompletionHandler<void()>&&) final;
    5353    void stop() final;
    5454    void clear() final;

  • trunk/Source/WebCore/platform/mediastream/cocoa/AudioMediaStreamTrackRendererInternalUnit.cpp

    r277363 r277364  
    11/*
    2  * Copyright (C) 2020 Apple Inc. All rights reserved.
     2 * Copyright (C) 2021 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2525
    2626#include "config.h"
    27 #include "AudioMediaStreamTrackRendererUnit.h"
     27#include "AudioMediaStreamTrackRendererInternalUnit.h"
    2828
    2929#if ENABLE(MEDIA_STREAM)
    … …  
    4747namespace WebCore {
    4848
    49 AudioMediaStreamTrackRendererUnit& AudioMediaStreamTrackRendererUnit::singleton()
    50 {
    51     static NeverDestroyed<AudioMediaStreamTrackRendererUnit> registry;
    52     return registry;
    53 }
    54 
    55 AudioMediaStreamTrackRendererUnit::~AudioMediaStreamTrackRendererUnit()
    56 {
    57     stop();
    58 }
    59 
    60 void AudioMediaStreamTrackRendererUnit::setAudioOutputDevice(const String& deviceID)
     49class LocalAudioMediaStreamTrackRendererInternalUnit final : public AudioMediaStreamTrackRendererInternalUnit {
     50    WTF_MAKE_FAST_ALLOCATED;
     51public:
     52    explicit LocalAudioMediaStreamTrackRendererInternalUnit(RenderCallback&&);
     53
     54private:
     55    void createAudioUnitIfNeeded();
     56
     57    // AudioMediaStreamTrackRendererInternalUnit API.
     58    void start() final;
     59    void stop() final;
     60    void retrieveFormatDescription(CompletionHandler<void(const CAAudioStreamDescription*)>&&) final;
     61    void setAudioOutputDevice(const String&) final;
     62
     63    static OSStatus renderingCallback(void*, AudioUnitRenderActionFlags*, const AudioTimeStamp*, UInt32 inBusNumber, UInt32 sampleCount, AudioBufferList*);
     64
     65    RenderCallback m_renderCallback;
     66    std::unique_ptr<CAAudioStreamDescription> m_outputDescription;
     67    AudioComponentInstance m_remoteIOUnit { nullptr };
     68    bool m_isStarted { false };
     69#if PLATFORM(MAC)
     70    uint32_t m_deviceID { 0 };
     71#endif
     72};
     73
     74UniqueRef<AudioMediaStreamTrackRendererInternalUnit> AudioMediaStreamTrackRendererInternalUnit::createLocalInternalUnit(RenderCallback&& renderCallback)
     75{
     76    return makeUniqueRef<LocalAudioMediaStreamTrackRendererInternalUnit>(WTFMove(renderCallback));
     77}
     78
     79LocalAudioMediaStreamTrackRendererInternalUnit::LocalAudioMediaStreamTrackRendererInternalUnit(RenderCallback&& renderCallback)
     80    : m_renderCallback(WTFMove(renderCallback))
     81{
     82}
     83
     84void LocalAudioMediaStreamTrackRendererInternalUnit::retrieveFormatDescription(CompletionHandler<void(const CAAudioStreamDescription*)>&& callback)
     85{
     86    createAudioUnitIfNeeded();
     87    callback(m_outputDescription.get());
     88}
     89
     90void LocalAudioMediaStreamTrackRendererInternalUnit::setAudioOutputDevice(const String& deviceID)
    6191{
    6292#if PLATFORM(MAC)
    … …  
    6494
    6595    if (!device && !deviceID.isEmpty()) {
    66         RELEASE_LOG(WebRTC, "AudioMediaStreamTrackRendererUnit::setAudioOutputDeviceId - did not find device");
     96        RELEASE_LOG(WebRTC, "AudioMediaStreamTrackRendererInternalUnit::setAudioOutputDeviceId - did not find device");
    6797        return;
    6898    }
    … …  
    85115}
    86116
    87 void AudioMediaStreamTrackRendererUnit::addSource(Ref<AudioSampleDataSource>&& source)
    88 {
    89     RELEASE_LOG(WebRTC, "AudioMediaStreamTrackRendererUnit::addSource");
    90 
    91     {
    92         auto locker = holdLock(m_sourcesLock);
    93         ASSERT(!m_sources.contains(source.get()));
    94         m_sources.add(WTFMove(source));
    95         m_sourcesCopy = copyToVector(m_sources);
    96         m_shouldUpdateRenderSources = true;
    97     }
    98     start();
    99 }
    100 
    101 void AudioMediaStreamTrackRendererUnit::removeSource(AudioSampleDataSource& source)
    102 {
    103     RELEASE_LOG(WebRTC, "AudioMediaStreamTrackRendererUnit::removeSource");
    104 
    105     bool shouldStop = false;
    106     {
    107         auto locker = holdLock(m_sourcesLock);
    108         m_sources.remove(source);
    109         shouldStop = m_sources.isEmpty();
    110         m_sourcesCopy = copyToVector(m_sources);
    111         m_shouldUpdateRenderSources = true;
    112     }
    113     if (shouldStop)
    114         stop();
    115 }
    116 
    117 void AudioMediaStreamTrackRendererUnit::start()
    118 {
    119     RELEASE_LOG(WebRTC, "AudioMediaStreamTrackRendererUnit::start");
    120 
     117void LocalAudioMediaStreamTrackRendererInternalUnit::start()
     118{
    121119    if (m_isStarted)
    122120        return;
    … …  
    127125
    128126    if (auto error = AudioOutputUnitStart(m_remoteIOUnit)) {
    129         RELEASE_LOG_ERROR(WebRTC, "AudioMediaStreamTrackRendererUnit::start AudioOutputUnitStart failed, error = %d", error);
     127        RELEASE_LOG_ERROR(WebRTC, "AudioMediaStreamTrackRendererInternalUnit::start AudioOutputUnitStart failed, error = %d", error);
    130128        AudioComponentInstanceDispose(m_remoteIOUnit);
    131129        m_remoteIOUnit = nullptr;
     130        return;
    132131    }
    133132    m_isStarted = true;
    134     RELEASE_LOG(WebRTC, "AudioMediaStreamTrackRendererUnit is started");
    135 }
    136 
    137 void AudioMediaStreamTrackRendererUnit::stop()
    138 {
    139     RELEASE_LOG(WebRTC, "AudioMediaStreamTrackRendererUnit::stop");
    140 
     133    RELEASE_LOG(WebRTC, "AudioMediaStreamTrackRendererInternalUnit is started");
     134}
     135
     136void LocalAudioMediaStreamTrackRendererInternalUnit::stop()
     137{
    141138    if (!m_remoteIOUnit)
    142139        return;
    … …  
    151148}
    152149
    153 CAAudioStreamDescription* AudioMediaStreamTrackRendererUnit::formatDescription()
    154 {
    155     createAudioUnitIfNeeded();
    156     return m_outputDescription.get();
    157 }
    158 
    159 void AudioMediaStreamTrackRendererUnit::createAudioUnitIfNeeded()
     150void LocalAudioMediaStreamTrackRendererInternalUnit::createAudioUnitIfNeeded()
    160151{
    161152    ASSERT(!m_remoteIOUnit || m_outputDescription);
    … …  
    176167    ASSERT(ioComponent);
    177168    if (!ioComponent) {
    178         RELEASE_LOG_ERROR(WebRTC, "AudioMediaStreamTrackRendererUnit::createAudioUnit unable to find remote IO unit component");
     169        RELEASE_LOG_ERROR(WebRTC, "AudioMediaStreamTrackRendererInternalUnit::createAudioUnit unable to find remote IO unit component");
    179170        return;
    180171    }
    … …  
    182173    auto error = AudioComponentInstanceNew(ioComponent, &remoteIOUnit);
    183174    if (error) {
    184         RELEASE_LOG_ERROR(WebRTC, "AudioMediaStreamTrackRendererUnit::createAudioUnit unable to open vpio unit, error = %d", error);
     175        RELEASE_LOG_ERROR(WebRTC, "AudioMediaStreamTrackRendererInternalUnit::createAudioUnit unable to open vpio unit, error = %d", error);
    185176        return;
    186177    }
    … …  
    190181    error = AudioUnitSetProperty(remoteIOUnit, kAudioOutputUnitProperty_EnableIO, kAudioUnitScope_Output, 0, &param, sizeof(param));
    191182    if (error) {
    192         RELEASE_LOG_ERROR(WebRTC, "AudioMediaStreamTrackRendererUnit::createAudioUnit unable to enable vpio unit output, error = %d", error);
     183        RELEASE_LOG_ERROR(WebRTC, "AudioMediaStreamTrackRendererInternalUnit::createAudioUnit unable to enable vpio unit output, error = %d", error);
    193184        return;
    194185    }
    … …  
    199190        error = AudioUnitSetProperty(remoteIOUnit, kAudioOutputUnitProperty_CurrentDevice, kAudioUnitScope_Global, 0, &m_deviceID, sizeof(m_deviceID));
    200191        if (error) {
    201             RELEASE_LOG_ERROR(WebRTC, "AudioMediaStreamTrackRendererUnit::createAudioUnit unable to set unit device ID %d, error %d (%.4s)", (int)m_deviceID, (int)error, (char*)&error);
     192            RELEASE_LOG_ERROR(WebRTC, "AudioMediaStreamTrackRendererInternalUnit::createAudioUnit unable to set unit device ID %d, error %d (%.4s)", (int)m_deviceID, (int)error, (char*)&error);
    202193            return;
    203194        }
    … …  
    205196#endif
    206197
    207     AURenderCallbackStruct callback = { renderingCallback, this };
     198    AURenderCallbackStruct callback = { LocalAudioMediaStreamTrackRendererInternalUnit::renderingCallback, this };
    208199    error = AudioUnitSetProperty(remoteIOUnit, kAudioUnitProperty_SetRenderCallback, kAudioUnitScope_Global, 0, &callback, sizeof(callback));
    209200    if (error) {
    210         RELEASE_LOG_ERROR(WebRTC, "AudioMediaStreamTrackRendererUnit::createAudioUnit unable to set vpio unit speaker proc, error = %d", error);
     201        RELEASE_LOG_ERROR(WebRTC, "AudioMediaStreamTrackRendererInternalUnit::createAudioUnit unable to set vpio unit speaker proc, error = %d", error);
    211202        return;
    212203    }
    … …  
    215206    error  = AudioUnitGetProperty(remoteIOUnit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Input, 0, &outputDescription.streamDescription(), &size);
    216207    if (error) {
    217         RELEASE_LOG_ERROR(WebRTC, "AudioMediaStreamTrackRendererUnit::createAudioUnit unable to get input stream format, error = %d", error);
     208        RELEASE_LOG_ERROR(WebRTC, "AudioMediaStreamTrackRendererInternalUnit::createAudioUnit unable to get input stream format, error = %d", error);
    218209        return;
    219210    }
    … …  
    223214    error = AudioUnitSetProperty(remoteIOUnit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Input, 0, &outputDescription.streamDescription(), sizeof(outputDescription.streamDescription()));
    224215    if (error) {
    225         RELEASE_LOG_ERROR(WebRTC, "AudioMediaStreamTrackRendererUnit::createAudioUnit unable to set input stream format, error = %d", error);
     216        RELEASE_LOG_ERROR(WebRTC, "AudioMediaStreamTrackRendererInternalUnit::createAudioUnit unable to set input stream format, error = %d", error);
    226217        return;
    227218    }
    … …  
    229220    error = AudioUnitInitialize(remoteIOUnit);
    230221    if (error) {
    231         RELEASE_LOG_ERROR(WebRTC, "AudioMediaStreamTrackRendererUnit::createAudioUnit AudioUnitInitialize() failed, error = %d", error);
     222        RELEASE_LOG_ERROR(WebRTC, "AudioMediaStreamTrackRendererInternalUnit::createAudioUnit AudioUnitInitialize() failed, error = %d", error);
    232223        return;
    233224    }
    … …  
    237228}
    238229
    239 OSStatus AudioMediaStreamTrackRendererUnit::render(UInt32 sampleCount, AudioBufferList& ioData, UInt32 /*inBusNumber*/, const AudioTimeStamp& timeStamp, AudioUnitRenderActionFlags& actionFlags)
    240 {
    241     // For performance reasons, we forbid heap allocations while doing rendering on the audio thread.
    242     ForbidMallocUseForCurrentThreadScope forbidMallocUse;
    243 
    244     ASSERT(!isMainThread());
    245     if (m_shouldUpdateRenderSources) {
    246         auto locker = tryHoldLock(m_sourcesLock);
    247         if (!locker)
    248             return 0;
    249 
    250         m_renderSources = WTFMove(m_sourcesCopy);
    251         m_shouldUpdateRenderSources = false;
    252     }
    253 
    254     if (m_renderSources.isEmpty()) {
    255         if (m_outputDescription)
    256             AudioSampleBufferList::zeroABL(ioData, static_cast<size_t>(sampleCount * m_outputDescription->bytesPerFrame()));
    257         actionFlags = kAudioUnitRenderAction_OutputIsSilence;
    258         return 0;
    259     }
    260 
    261     // Mix all sources.
    262     bool isFirstSource = true;
    263     for (auto& source : m_renderSources) {
    264         source->pullSamples(ioData, static_cast<size_t>(sampleCount), timeStamp.mSampleTime, timeStamp.mHostTime, isFirstSource ? AudioSampleDataSource::Copy : AudioSampleDataSource::Mix);
    265         isFirstSource = false;
    266     }
    267     return 0;
    268 }
    269 
    270 OSStatus AudioMediaStreamTrackRendererUnit::renderingCallback(void* processor, AudioUnitRenderActionFlags* actionFlags, const AudioTimeStamp* timeStamp, UInt32 inBusNumber, UInt32 sampleCount, AudioBufferList* ioData)
    271 {
    272     return static_cast<AudioMediaStreamTrackRendererUnit*>(processor)->render(sampleCount, *ioData, inBusNumber, *timeStamp, *actionFlags);
     230OSStatus LocalAudioMediaStreamTrackRendererInternalUnit::renderingCallback(void* processor, AudioUnitRenderActionFlags* actionFlags, const AudioTimeStamp* timeStamp, UInt32, UInt32 sampleCount, AudioBufferList* ioData)
     231{
     232    return static_cast<LocalAudioMediaStreamTrackRendererInternalUnit*>(processor)->m_renderCallback(sampleCount, *ioData, timeStamp->mSampleTime, timeStamp->mHostTime, *actionFlags);
    273233}
    274234

  • trunk/Source/WebCore/platform/mediastream/cocoa/AudioMediaStreamTrackRendererInternalUnit.h

    r277363 r277364  
    11/*
    2  * Copyright (C) 2017 Apple Inc. All rights reserved.
     2 * Copyright (C) 2021 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2828#if ENABLE(MEDIA_STREAM)
    2929
    30 #include "AudioMediaStreamTrackRenderer.h"
    31 #include "Logging.h"
    32 #include <wtf/WeakPtr.h>
    33 
    3430#include <AudioToolbox/AudioToolbox.h>
    3531#include <CoreAudio/CoreAudioTypes.h>
    … …  
    3733namespace WebCore {
    3834
    39 class AudioSampleDataSource;
    40 class AudioSampleBufferList;
     35class AudioMediaStreamTrackRendererUnit;
    4136class CAAudioStreamDescription;
    4237
    43 class AudioMediaStreamTrackRendererCocoa : public AudioMediaStreamTrackRenderer, public CanMakeWeakPtr<AudioMediaStreamTrackRendererCocoa, WeakPtrFactoryInitialization::Eager> {
    44     WTF_MAKE_FAST_ALLOCATED;
     38class AudioMediaStreamTrackRendererInternalUnit {
    4539public:
    46     AudioMediaStreamTrackRendererCocoa();
    47     ~AudioMediaStreamTrackRendererCocoa();
     40    virtual ~AudioMediaStreamTrackRendererInternalUnit() = default;
    4841
    49 private:
    50     // AudioMediaStreamTrackRenderer
    51     void pushSamples(const WTF::MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;
    52     void start() final;
    53     void stop() final;
    54     void clear() final;
    55     void setVolume(float) final;
    56     void setAudioOutputDevice(const String&) final;
     42    using RenderCallback = Function<OSStatus(size_t sampleCount, AudioBufferList&, uint64_t sampleTime, double hostTime, AudioUnitRenderActionFlags&)>;
     43    WEBCORE_EXPORT static UniqueRef<AudioMediaStreamTrackRendererInternalUnit> createLocalInternalUnit(RenderCallback&&);
    5744
    58     std::unique_ptr<CAAudioStreamDescription> m_outputDescription;
    59     RefPtr<AudioSampleDataSource> m_dataSource;
    60     bool m_shouldReset { false };
     45    virtual void start() = 0;
     46    virtual void stop() = 0;
     47    virtual void retrieveFormatDescription(CompletionHandler<void(const CAAudioStreamDescription*)>&&) = 0;
     48    virtual void setAudioOutputDevice(const String&) = 0;
    6149};
    6250

  • trunk/Source/WebCore/platform/mediastream/cocoa/AudioMediaStreamTrackRendererUnit.h

    r277363 r277364  
    11/*
    2  * Copyright (C) 2020 Apple Inc. All rights reserved.
     2 * Copyright (C) 2020-2021 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2828#if ENABLE(MEDIA_STREAM)
    2929
    30 #include <AudioToolbox/AudioToolbox.h>
    31 #include <CoreAudio/CoreAudioTypes.h>
     30#include "AudioMediaStreamTrackRendererInternalUnit.h"
    3231#include <wtf/Forward.h>
    3332#include <wtf/HashSet.h>
     33#include <wtf/UniqueRef.h>
    3434#include <wtf/Vector.h>
    3535
    … …  
    3939class AudioSampleBufferList;
    4040class CAAudioStreamDescription;
     41class AudioMediaStreamTrackRendererInternalUnit;
    4142
    4243class AudioMediaStreamTrackRendererUnit {
    4344public:
    44     static AudioMediaStreamTrackRendererUnit& singleton();
     45    WEBCORE_EXPORT static AudioMediaStreamTrackRendererUnit& singleton();
    4546
    46     AudioMediaStreamTrackRendererUnit() = default;
     47    AudioMediaStreamTrackRendererUnit();
    4748    ~AudioMediaStreamTrackRendererUnit();
    4849
    49     static OSStatus renderingCallback(void*, AudioUnitRenderActionFlags*, const AudioTimeStamp*, UInt32 inBusNumber, UInt32 numberOfFrames, AudioBufferList*);
     50    using CreateInternalUnitFunction = Function<UniqueRef<AudioMediaStreamTrackRendererInternalUnit>(AudioMediaStreamTrackRendererInternalUnit::RenderCallback&&)>;
     51    WEBCORE_EXPORT static void setCreateInternalUnitFunction(CreateInternalUnitFunction&&);
    5052
    51     void start();
    52     void stop();
     53    WEBCORE_EXPORT void render(size_t sampleCount, AudioBufferList&, uint64_t sampleTime, double hostTime, AudioUnitRenderActionFlags&);
    5354
    5455    void setAudioOutputDevice(const String&);
    … …  
    5758    void removeSource(AudioSampleDataSource&);
    5859
    59     CAAudioStreamDescription* formatDescription();
     60    void retrieveFormatDescription(CompletionHandler<void(const CAAudioStreamDescription*)>&&);
    6061
    6162private:
     63    void start();
     64    void stop();
     65
    6266    void createAudioUnitIfNeeded();
    63     OSStatus render(UInt32 sampleCount, AudioBufferList&, UInt32 inBusNumber, const AudioTimeStamp&, AudioUnitRenderActionFlags&);
    6467
    65     AudioComponentInstance m_remoteIOUnit { nullptr };
    66     std::unique_ptr<CAAudioStreamDescription> m_outputDescription;
    6768    HashSet<Ref<AudioSampleDataSource>> m_sources;
    6869    Vector<Ref<AudioSampleDataSource>> m_sourcesCopy;
    … …  
    7071    bool m_shouldUpdateRenderSources { false };
    7172    Lock m_sourcesLock;
    72     bool m_isStarted { false };
    73 #if PLATFORM(MAC)
    74     uint32_t m_deviceID { 0 };
    75 #endif
     73    UniqueRef<AudioMediaStreamTrackRendererInternalUnit> m_internalUnit;
    7674};
    7775

  • trunk/Source/WebKit/ChangeLog

    r277362 r277364  
     12021-05-12  Youenn Fablet  <youenn@apple.com>
     2
     3        Introduce an internal unit to render audio MediaStreamTrack(s)
     4        https://bugs.webkit.org/show_bug.cgi?id=225601
     5
     6        Reviewed by Eric Carlson.
     7
     8        Update according new WebCore API.
     9
     10        * GPUProcess/webrtc/RemoteAudioMediaStreamTrackRenderer.cpp:
     11        (WebKit::RemoteAudioMediaStreamTrackRenderer::start):
     12        * WebProcess/GPU/webrtc/AudioMediaStreamTrackRenderer.cpp:
     13        (WebKit::AudioMediaStreamTrackRenderer::start):
     14        * WebProcess/GPU/webrtc/AudioMediaStreamTrackRenderer.h:
     15
    1162021-05-12  Martin Robinson  <mrobinson@igalia.com>
    217

  • trunk/Source/WebKit/GPUProcess/webrtc/RemoteAudioMediaStreamTrackRenderer.cpp

    r271041 r277364  
    7878void RemoteAudioMediaStreamTrackRenderer::start()
    7979{
    80     m_renderer->start();
     80    m_renderer->start([] { });
    8181}
    8282

  • trunk/Source/WebKit/WebProcess/GPU/webrtc/AudioMediaStreamTrackRenderer.cpp

    r276633 r277364  
    6565}
    6666
    67 void AudioMediaStreamTrackRenderer::start()
     67void AudioMediaStreamTrackRenderer::start(CompletionHandler<void()>&& callback)
    6868{
    6969    m_isPlaying = true;
    7070    m_connection->send(Messages::RemoteAudioMediaStreamTrackRenderer::Start { }, m_identifier);
     71    callback();
    7172}
    7273

  • trunk/Source/WebKit/WebProcess/GPU/webrtc/AudioMediaStreamTrackRenderer.h

    r272213 r277364  
    5555
    5656    // WebCore::AudioMediaStreamTrackRenderer
    57     void start() final;
     57    void start(CompletionHandler<void()>&&) final;
    5858    void stop() final;
    5959    void clear() final;
Note: See TracChangeset for help on using the changeset viewer.