Changeset 285027 in webkit


Ignore:
Timestamp:
Oct 29, 2021 6:36:03 AM (9 months ago)
Author:
youenn@apple.com
Message:

Ensure synchronized rendering of incoming audio tracks
​https://bugs.webkit.org/show_bug.cgi?id=232375

Reviewed by Eric Carlson.

WebRTC incoming audio tracks are guaranteed to be synchronized based on the audio module clock.
Previously, we were mixing all audio tracks together, but we were not handling synchronization of tracks that have the same clock.
This can cause some lower quality audio rendering.
To prevent this, all WebRTC incoming audio tracks are now mixed together in a single track that is then mixed with all other non synchronized tracks.
To do so we introduce IncomingAudioMediaStreamTrackRendererUnit which will receive all incoming audio tracks.
It then sends the mix to the actual AudioMediaStreamTrackRendererUnit.
To ensure tight synchronization, we introduce AudioSampleDataSource::pullAvailableSampleChunk which reads data using the provided timestamp relative to the input offset.
We do a refactoring to make it easy to either use IncomingAudioMediaStreamTrackRendererUnit or AudioMediaStreamTrackRendererUnit by having a base class called BaseAudioMediaStreamTrackRendererUnit.

Manually tested.

  • Modules/mediastream/libwebrtc/LibWebRTCPeerConnectionBackend.cpp:
  • Modules/mediastream/libwebrtc/LibWebRTCRtpReceiverBackend.cpp:
  • Modules/mediastream/libwebrtc/LibWebRTCRtpReceiverBackend.h:
  • SourcesCocoa.txt:
  • WebCore.xcodeproj/project.pbxproj:
  • loader/EmptyClients.cpp:
  • platform/audio/cocoa/AudioSampleDataSource.h:
  • platform/audio/cocoa/AudioSampleDataSource.mm:
  • platform/mediastream/AudioMediaStreamTrackRenderer.cpp:
  • platform/mediastream/AudioMediaStreamTrackRenderer.h:
  • platform/mediastream/AudioTrackPrivateMediaStream.cpp:
  • platform/mediastream/RealtimeIncomingAudioSource.cpp:
  • platform/mediastream/RealtimeIncomingAudioSource.h:
  • platform/mediastream/cocoa/AudioMediaStreamTrackRendererCocoa.cpp:
  • platform/mediastream/cocoa/AudioMediaStreamTrackRendererCocoa.h:
  • platform/mediastream/cocoa/AudioMediaStreamTrackRendererUnit.h:
  • platform/mediastream/cocoa/BaseAudioMediaStreamTrackRendererUnit.h: Added.
  • platform/mediastream/cocoa/IncomingAudioMediaStreamTrackRendererUnit.cpp: Added.
  • platform/mediastream/cocoa/IncomingAudioMediaStreamTrackRendererUnit.h: Added.
  • platform/mediastream/libwebrtc/LibWebRTCAudioModule.cpp:
  • platform/mediastream/libwebrtc/LibWebRTCAudioModule.h:
  • platform/mediastream/libwebrtc/LibWebRTCProvider.cpp:
  • platform/mediastream/libwebrtc/LibWebRTCProvider.h:
  • platform/mediastream/mac/RealtimeIncomingAudioSourceCocoa.cpp:
Location:
trunk/Source/WebCore
Files:
2 added
22 edited
1 copied

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebCore/ChangeLog

    r285024 r285027  
     12021-10-29  Youenn Fablet  <youenn@apple.com>
     2
     3        Ensure synchronized rendering of incoming audio tracks
     4        https://bugs.webkit.org/show_bug.cgi?id=232375
     5
     6        Reviewed by Eric Carlson.
     7
     8        WebRTC incoming audio tracks are guaranteed to be synchronized based on the audio module clock.
     9        Previously, we were mixing all audio tracks together, but we were not handling synchronization of tracks that have the same clock.
     10        This can cause some lower quality audio rendering.
     11        To prevent this, all WebRTC incoming audio tracks are now mixed together in a single track that is then mixed with all other non synchronized tracks.
     12        To do so we introduce IncomingAudioMediaStreamTrackRendererUnit which will receive all incoming audio tracks.
     13        It then sends the mix to the actual AudioMediaStreamTrackRendererUnit.
     14        To ensure tight synchronization, we introduce AudioSampleDataSource::pullAvailableSampleChunk which reads data using the provided timestamp relative to the input offset.
     15        We do a refactoring to make it easy to either use IncomingAudioMediaStreamTrackRendererUnit or AudioMediaStreamTrackRendererUnit by having a base class called BaseAudioMediaStreamTrackRendererUnit.
     16
     17        Manually tested.
     18
     19        * Modules/mediastream/libwebrtc/LibWebRTCPeerConnectionBackend.cpp:
     20        * Modules/mediastream/libwebrtc/LibWebRTCRtpReceiverBackend.cpp:
     21        * Modules/mediastream/libwebrtc/LibWebRTCRtpReceiverBackend.h:
     22        * SourcesCocoa.txt:
     23        * WebCore.xcodeproj/project.pbxproj:
     24        * loader/EmptyClients.cpp:
     25        * platform/audio/cocoa/AudioSampleDataSource.h:
     26        * platform/audio/cocoa/AudioSampleDataSource.mm:
     27        * platform/mediastream/AudioMediaStreamTrackRenderer.cpp:
     28        * platform/mediastream/AudioMediaStreamTrackRenderer.h:
     29        * platform/mediastream/AudioTrackPrivateMediaStream.cpp:
     30        * platform/mediastream/RealtimeIncomingAudioSource.cpp:
     31        * platform/mediastream/RealtimeIncomingAudioSource.h:
     32        * platform/mediastream/cocoa/AudioMediaStreamTrackRendererCocoa.cpp:
     33        * platform/mediastream/cocoa/AudioMediaStreamTrackRendererCocoa.h:
     34        * platform/mediastream/cocoa/AudioMediaStreamTrackRendererUnit.h:
     35        * platform/mediastream/cocoa/BaseAudioMediaStreamTrackRendererUnit.h: Added.
     36        * platform/mediastream/cocoa/IncomingAudioMediaStreamTrackRendererUnit.cpp: Added.
     37        * platform/mediastream/cocoa/IncomingAudioMediaStreamTrackRendererUnit.h: Added.
     38        * platform/mediastream/libwebrtc/LibWebRTCAudioModule.cpp:
     39        * platform/mediastream/libwebrtc/LibWebRTCAudioModule.h:
     40        * platform/mediastream/libwebrtc/LibWebRTCProvider.cpp:
     41        * platform/mediastream/libwebrtc/LibWebRTCProvider.h:
     42        * platform/mediastream/mac/RealtimeIncomingAudioSourceCocoa.cpp:
     43
    1442021-10-29  Youenn Fablet  <youenn@apple.com>
    245

  • trunk/Source/WebCore/Modules/mediastream/libwebrtc/LibWebRTCPeerConnectionBackend.cpp

    r284857 r285027  
    3030#include "Document.h"
    3131#include "IceCandidate.h"
     32#include "LibWebRTCAudioModule.h"
    3233#include "LibWebRTCDataChannelHandler.h"
    3334#include "LibWebRTCMediaEndpoint.h"
     35#include "LibWebRTCProvider.h"
    3436#include "LibWebRTCRtpReceiverBackend.h"
    3537#include "LibWebRTCRtpSenderBackend.h"
    … …  
    294296    auto& document = downcast<Document>(*m_peerConnection.scriptExecutionContext());
    295297
    296     auto source = backend->createSource();
     298    auto source = backend->createSource(document);
     299
    297300    // Remote source is initially muted and will be unmuted when receiving the first packet.
    298301    source->setMuted(true);

  • trunk/Source/WebCore/Modules/mediastream/libwebrtc/LibWebRTCRtpReceiverBackend.cpp

    r281225 r285027  
    2626#include "LibWebRTCRtpReceiverBackend.h"
    2727
     28#include "Document.h"
     29#include "LibWebRTCAudioModule.h"
    2830#include "LibWebRTCDtlsTransportBackend.h"
     31#include "LibWebRTCProvider.h"
    2932#include "LibWebRTCRtpReceiverTransformBackend.h"
    3033#include "LibWebRTCUtils.h"
     34#include "Page.h"
    3135#include "RTCRtpTransformBackend.h"
    3236#include "RealtimeIncomingAudioSource.h"
    … …  
    100104}
    101105
    102 Ref<RealtimeMediaSource> LibWebRTCRtpReceiverBackend::createSource()
     106Ref<RealtimeMediaSource> LibWebRTCRtpReceiverBackend::createSource(Document& document)
    103107{
    104108    auto rtcTrack = m_rtcReceiver->track();
    … …  
    109113    case cricket::MEDIA_TYPE_AUDIO: {
    110114        rtc::scoped_refptr<webrtc::AudioTrackInterface> audioTrack = static_cast<webrtc::AudioTrackInterface*>(rtcTrack.get());
    111         return RealtimeIncomingAudioSource::create(WTFMove(audioTrack), fromStdString(rtcTrack->id()));
     115        auto source = RealtimeIncomingAudioSource::create(WTFMove(audioTrack), fromStdString(rtcTrack->id()));
     116        if (document.page())
     117            source->setAudioModule(document.page()->libWebRTCProvider().audioModule());
     118        return source;
    112119    }
    113120    case cricket::MEDIA_TYPE_VIDEO: {

  • trunk/Source/WebCore/Modules/mediastream/libwebrtc/LibWebRTCRtpReceiverBackend.h

    r281225 r285027  
    3636
    3737namespace WebCore {
    38 
     38class Document;
    3939class RealtimeMediaSource;
    4040
    … …  
    4747    webrtc::RtpReceiverInterface* rtcReceiver() { return m_rtcReceiver.get(); }
    4848
    49     Ref<RealtimeMediaSource> createSource();
     49    Ref<RealtimeMediaSource> createSource(Document&);
    5050
    5151private:

  • trunk/Source/WebCore/SourcesCocoa.txt

    r284920 r285027  
    527527platform/mediastream/cocoa/AudioMediaStreamTrackRendererUnit.cpp
    528528platform/mediastream/cocoa/DisplayCaptureSourceCocoa.cpp
     529platform/mediastream/cocoa/IncomingAudioMediaStreamTrackRendererUnit.cpp
    529530platform/mediastream/ios/AVAudioSessionCaptureDevice.mm
    530531platform/mediastream/ios/AVAudioSessionCaptureDeviceManager.mm @no-unify

  • trunk/Source/WebCore/WebCore.xcodeproj/project.pbxproj

    r285012 r285027  
    11811181                4186BD40213EE3450001826F /* LibWebRTCRtpReceiverBackend.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 41D1A04A213EDDFE0063FB6B /* LibWebRTCRtpReceiverBackend.cpp */; };
    11821182                4186BD4E2140B9E80001826F /* LibWebRTCRtpTransceiverBackend.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 4186BD4D2140B9E80001826F /* LibWebRTCRtpTransceiverBackend.cpp */; };
     1183                41878FAD27294721002E1EDD /* BaseAudioMediaStreamTrackRendererUnit.h in Headers */ = {isa = PBXBuildFile; fileRef = 41878FA827282055002E1EDD /* BaseAudioMediaStreamTrackRendererUnit.h */; settings = {ATTRIBUTES = (Private, ); }; };
    11831184                41885B9311B6FDA6003383BB /* FormSubmission.h in Headers */ = {isa = PBXBuildFile; fileRef = 41885B9111B6FDA6003383BB /* FormSubmission.h */; settings = {ATTRIBUTES = (Private, ); }; };
    11841185                4188D55C26C54D6F004858C8 /* RTCDtlsTransportBackend.h in Headers */ = {isa = PBXBuildFile; fileRef = 4188D55A26C54D6F004858C8 /* RTCDtlsTransportBackend.h */; };
    … …  
    81598160                4186BD4B2140A8050001826F /* LibWebRTCRtpTransceiverBackend.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = LibWebRTCRtpTransceiverBackend.h; path = libwebrtc/LibWebRTCRtpTransceiverBackend.h; sourceTree = "<group>"; };
    81608161                4186BD4D2140B9E80001826F /* LibWebRTCRtpTransceiverBackend.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = LibWebRTCRtpTransceiverBackend.cpp; path = libwebrtc/LibWebRTCRtpTransceiverBackend.cpp; sourceTree = "<group>"; };
     8162                41878FA827282055002E1EDD /* BaseAudioMediaStreamTrackRendererUnit.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = BaseAudioMediaStreamTrackRendererUnit.h; sourceTree = "<group>"; };
     8163                41878FAA27282057002E1EDD /* IncomingAudioMediaStreamTrackRendererUnit.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = IncomingAudioMediaStreamTrackRendererUnit.h; sourceTree = "<group>"; };
     8164                41878FAB272934E6002E1EDD /* IncomingAudioMediaStreamTrackRendererUnit.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = IncomingAudioMediaStreamTrackRendererUnit.cpp; sourceTree = "<group>"; };
    81618165                418807DF24E4558300DDAF94 /* AbortAlgorithm.idl */ = {isa = PBXFileReference; lastKnownFileType = text; path = AbortAlgorithm.idl; sourceTree = "<group>"; };
    81628166                418807E124E458C300DDAF94 /* AbortAlgorithm.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = AbortAlgorithm.h; sourceTree = "<group>"; };
    … …  
    2017320177                                41C3B8A02649B1B7004ED4DE /* AudioMediaStreamTrackRendererUnit.cpp */,
    2017420178                                41C3B89B2649B1B6004ED4DE /* AudioMediaStreamTrackRendererUnit.h */,
     20179                                41878FA827282055002E1EDD /* BaseAudioMediaStreamTrackRendererUnit.h */,
    2017520180                                07BB1E6E27176CCA001DF289 /* DisplayCaptureSourceCocoa.cpp */,
    2017620181                                07BB1E6F27176CCA001DF289 /* DisplayCaptureSourceCocoa.h */,
     20182                                41878FAB272934E6002E1EDD /* IncomingAudioMediaStreamTrackRendererUnit.cpp */,
     20183                                41878FAA27282057002E1EDD /* IncomingAudioMediaStreamTrackRendererUnit.h */,
    2017720184                        );
    2017820185                        path = cocoa;
    … …  
    3192931936                                460BB6161D0A1BF000221812 /* Base64Utilities.h in Headers */,
    3193031937                                83198FBF24A160DD00420B05 /* BaseAudioContext.h in Headers */,
     31938                                41878FAD27294721002E1EDD /* BaseAudioMediaStreamTrackRendererUnit.h in Headers */,
    3193131939                                412DEF1F23A918A300D840F6 /* BaseAudioSharedUnit.h in Headers */,
    3193231940                                379E61CA126CA5C400B63E8D /* BaseButtonInputType.h in Headers */,

  • trunk/Source/WebCore/loader/EmptyClients.cpp

    r283179 r285027  
    5858#include "IDBConnectionToServer.h"
    5959#include "InspectorClient.h"
     60#include "LibWebRTCAudioModule.h"
    6061#include "LibWebRTCProvider.h"
    6162#include "MediaRecorderPrivate.h"

  • trunk/Source/WebCore/platform/audio/cocoa/AudioSampleDataSource.h

    r284862 r285027  
    6262
    6363    bool pullAvailableSamplesAsChunks(AudioBufferList&, size_t frameCount, uint64_t timeStamp, Function<void()>&&);
     64    bool pullAvailableSampleChunk(AudioBufferList&, size_t frameCount, uint64_t timeStamp, PullMode);
    6465
    6566    void setVolume(float volume) { m_volume = volume; }
    … …  
    7071
    7172    const CAAudioStreamDescription* inputDescription() const { return m_inputDescription ? &m_inputDescription.value() : nullptr; }
     73    const CAAudioStreamDescription* outputDescription() const { return m_outputDescription ? &m_outputDescription.value() : nullptr; }
    7274
    7375    void recomputeSampleOffset() { m_shouldComputeOutputSampleOffset = true; }
    … …  
    8789
    8890    void pushSamplesInternal(const AudioBufferList&, const MediaTime&, size_t frameCount);
     91    bool pullSamplesInternal(AudioBufferList&, size_t sampleCount, uint64_t timeStamp, PullMode);
    8992
    9093    std::optional<CAAudioStreamDescription> m_inputDescription;

  • trunk/Source/WebCore/platform/audio/cocoa/AudioSampleDataSource.mm

    r284862 r285027  
    287287    m_isFirstPull = false;
    288288
     289    return pullSamplesInternal(buffer, sampleCount, timeStamp, mode);
     290}
     291
     292bool AudioSampleDataSource::pullSamplesInternal(AudioBufferList& buffer, size_t sampleCount, uint64_t timeStamp, PullMode mode)
     293{
    289294    if (mode == Copy) {
    290295        m_ringBuffer->fetch(&buffer, sampleCount, timeStamp, CARingBuffer::Copy);
    … …  
    308313
    309314    return true;
     315}
     316
     317bool AudioSampleDataSource::pullAvailableSampleChunk(AudioBufferList& buffer, size_t sampleCount, uint64_t timeStamp, PullMode mode)
     318{
     319    ASSERT(buffer.mNumberBuffers == m_ringBuffer->channelCount());
     320    if (buffer.mNumberBuffers != m_ringBuffer->channelCount())
     321        return false;
     322
     323    if (m_muted)
     324        return false;
     325
     326    if (m_shouldComputeOutputSampleOffset) {
     327        m_shouldComputeOutputSampleOffset = false;
     328        m_outputSampleOffset = m_inputSampleOffset.timeValue() * m_outputDescription->sampleRate() / m_inputSampleOffset.timeScale();
     329    }
     330
     331    timeStamp += m_outputSampleOffset;
     332
     333    return pullSamplesInternal(buffer, sampleCount, timeStamp, mode);
    310334}
    311335

  • trunk/Source/WebCore/platform/mediastream/AudioMediaStreamTrackRenderer.cpp

    r262695 r285027  
    4141namespace WebCore {
    4242
    43 AudioMediaStreamTrackRenderer::RendererCreator AudioMediaStreamTrackRenderer::m_rendererCreator = nullptr;
    44 void AudioMediaStreamTrackRenderer::setCreator(RendererCreator creator)
     43std::unique_ptr<AudioMediaStreamTrackRenderer> AudioMediaStreamTrackRenderer::create(Init&& init)
    4544{
    46     m_rendererCreator = creator;
    47 }
    48 
    49 std::unique_ptr<AudioMediaStreamTrackRenderer> AudioMediaStreamTrackRenderer::create()
    50 {
    51     if (m_rendererCreator)
    52         return m_rendererCreator();
    53 
    5445#if PLATFORM(COCOA)
    55     return makeUnique<AudioMediaStreamTrackRendererCocoa>();
     46    return makeUnique<AudioMediaStreamTrackRendererCocoa>(WTFMove(init));
    5647#else
    5748    return nullptr;
    … …  
    5950}
    6051
     52AudioMediaStreamTrackRenderer::AudioMediaStreamTrackRenderer(Init&& init)
     53    : m_crashCallback(WTFMove(init.crashCallback))
     54#if USE(LIBWEBRTC)
     55    , m_audioModule(WTFMove(init.audioModule))
     56#endif
    6157#if !RELEASE_LOG_DISABLED
    62 void AudioMediaStreamTrackRenderer::setLogger(const Logger& logger, const void* identifier)
     58    , m_logger(init.logger)
     59    , m_logIdentifier(init.logIdentifier)
     60#endif
    6361{
    64     m_logger = &logger;
    65     m_logIdentifier = identifier;
    6662}
    6763
     64#if !RELEASE_LOG_DISABLED
    6865WTFLogChannel& AudioMediaStreamTrackRenderer::logChannel() const
    6966{

  • trunk/Source/WebCore/platform/mediastream/AudioMediaStreamTrackRenderer.h

    r277364 r285027  
    3838
    3939class AudioStreamDescription;
     40class LibWebRTCAudioModule;
    4041class PlatformAudioData;
    4142
    … …  
    4344    WTF_MAKE_FAST_ALLOCATED;
    4445public:
    45     static std::unique_ptr<AudioMediaStreamTrackRenderer> create();
     46    struct Init {
     47        Function<void()>&& crashCallback;
     48#if USE(LIBWEBRTC)
     49        RefPtr<LibWebRTCAudioModule> audioModule;
     50#endif
     51#if !RELEASE_LOG_DISABLED
     52        const Logger& logger;
     53        const void* logIdentifier;
     54#endif
     55    };
     56    static std::unique_ptr<AudioMediaStreamTrackRenderer> create(Init&&);
    4657    virtual ~AudioMediaStreamTrackRenderer() = default;
    4758
    … …  
    5768    virtual void setAudioOutputDevice(const String&);
    5869
    59 #if !RELEASE_LOG_DISABLED
    60     void setLogger(const Logger&, const void*);
    61 #endif
     70protected:
     71    explicit AudioMediaStreamTrackRenderer(Init&&);
    6272
    63     using RendererCreator = std::unique_ptr<AudioMediaStreamTrackRenderer> (*)();
    64     static void setCreator(RendererCreator);
    65 
    66     void setCrashCallback(Function<void()>&& callback) { m_crashCallback = WTFMove(callback); }
    67 
    68 protected:
    6973#if !RELEASE_LOG_DISABLED
    7074    const Logger& logger() const final;
    … …  
    7579#endif
    7680
     81#if USE(LIBWEBRTC)
     82    LibWebRTCAudioModule* audioModule();
     83#endif
     84
    7785    void crashed();
    7886
    7987private:
    80     static RendererCreator m_rendererCreator;
    81 
    8288    // Main thread writable members
    8389    float m_volume { 1 };
    8490    Function<void()> m_crashCallback;
    8591
     92#if USE(LIBWEBRTC)
     93    RefPtr<LibWebRTCAudioModule> m_audioModule;
     94#endif
     95
    8696#if !RELEASE_LOG_DISABLED
    87     RefPtr<const Logger> m_logger;
     97    Ref<const Logger> m_logger;
    8898    const void* m_logIdentifier;
    8999#endif
    … …  
    106116}
    107117
     118#if USE(LIBWEBRTC)
     119inline LibWebRTCAudioModule* AudioMediaStreamTrackRenderer::audioModule()
     120{
     121    return m_audioModule.get();
     122}
     123#endif
     124
    108125#if !RELEASE_LOG_DISABLED
    109126inline const Logger& AudioMediaStreamTrackRenderer::logger() const
    110127{
    111     return *m_logger;
     128    return m_logger.get();
    112129   
    113130}

  • trunk/Source/WebCore/platform/mediastream/AudioTrackPrivateMediaStream.cpp

    r284093 r285027  
    3030
    3131#include "AudioMediaStreamTrackRenderer.h"
     32#include "LibWebRTCAudioModule.h"
    3233#include "Logging.h"
     34#include "RealtimeIncomingAudioSource.h"
    3335
    3436namespace WebCore {
    … …  
    4951}
    5052
     53#if USE(LIBWEBRTC)
     54static RefPtr<LibWebRTCAudioModule> audioModuleFromSource(RealtimeMediaSource& source)
     55{
     56    if (!is<RealtimeIncomingAudioSource>(source))
     57        return nullptr;
     58    return downcast<RealtimeIncomingAudioSource>(source).audioModule();
     59}
     60#endif
     61
    5162std::unique_ptr<AudioMediaStreamTrackRenderer> AudioTrackPrivateMediaStream::createRenderer(AudioTrackPrivateMediaStream& stream)
    5263{
    53     auto renderer = AudioMediaStreamTrackRenderer::create();
    54     if (!renderer)
    55         return nullptr;
     64    auto& track = stream.m_streamTrack.get();
     65    return AudioMediaStreamTrackRenderer::create(AudioMediaStreamTrackRenderer::Init {
     66        [stream = WeakPtr { stream }] {
     67            if (stream)
     68                stream->createNewRenderer();
     69        }
     70#if USE(LIBWEBRTC)
     71        , audioModuleFromSource(stream.m_audioSource.get())
     72#endif
    5673#if !RELEASE_LOG_DISABLED
    57     auto& track = stream.m_streamTrack.get();
    58     renderer->setLogger(track.logger(), track.logIdentifier());
     74        , track.logger()
     75        , track.logIdentifier()
    5976#endif
    60     renderer->setCrashCallback([stream = WeakPtr { stream }] {
    61         if (stream)
    62             stream->createNewRenderer();
    6377    });
    64     return renderer;
    6578}
    6679

  • trunk/Source/WebCore/platform/mediastream/RealtimeIncomingAudioSource.cpp

    r282755 r285027  
    3535
    3636#include "LibWebRTCAudioFormat.h"
     37#include "LibWebRTCAudioModule.h"
    3738#include "Logging.h"
    3839
    … …  
    8182}
    8283
     84void RealtimeIncomingAudioSource::setAudioModule(RefPtr<LibWebRTCAudioModule>&& audioModule)
     85{
     86    ASSERT(!m_audioModule);
     87    m_audioModule = WTFMove(audioModule);
     88}
     89
    8390}
    8491

  • trunk/Source/WebCore/platform/mediastream/RealtimeIncomingAudioSource.h

    r276633 r285027  
    4646namespace WebCore {
    4747
     48class LibWebRTCAudioModule;
     49
    4850class RealtimeIncomingAudioSource
    4951    : public RealtimeMediaSource
    … …  
    5355public:
    5456    static Ref<RealtimeIncomingAudioSource> create(rtc::scoped_refptr<webrtc::AudioTrackInterface>&&, String&&);
     57
     58    void setAudioModule(RefPtr<LibWebRTCAudioModule>&&);
     59    LibWebRTCAudioModule* audioModule() { return m_audioModule.get(); }
    5560
    5661protected:
    … …  
    8085    RealtimeMediaSourceSettings m_currentSettings;
    8186    rtc::scoped_refptr<webrtc::AudioTrackInterface> m_audioTrack;
     87    RefPtr<LibWebRTCAudioModule> m_audioModule;
    8288
    8389#if !RELEASE_LOG_DISABLED

  • trunk/Source/WebCore/platform/mediastream/cocoa/AudioMediaStreamTrackRendererCocoa.cpp

    r285024 r285027  
    3737namespace WebCore {
    3838
    39 AudioMediaStreamTrackRendererCocoa::AudioMediaStreamTrackRendererCocoa()
    40     : m_resetObserver([this] { reset(); })
     39AudioMediaStreamTrackRendererCocoa::AudioMediaStreamTrackRendererCocoa(Init&& init)
     40    : AudioMediaStreamTrackRenderer(WTFMove(init))
     41    , m_resetObserver([this] { reset(); })
    4142{
    4243}
    … …  
    6061}
    6162
     63BaseAudioMediaStreamTrackRendererUnit& AudioMediaStreamTrackRendererCocoa::rendererUnit()
     64{
     65    if (auto* audioModule = this->audioModule())
     66        return audioModule->incomingAudioMediaStreamTrackRendererUnit();
     67    return AudioMediaStreamTrackRendererUnit::singleton();
     68}
     69
    6270void AudioMediaStreamTrackRendererCocoa::stop()
    6371{
    … …  
    6573
    6674    if (m_registeredDataSource)
    67         AudioMediaStreamTrackRendererUnit::singleton().removeSource(*m_registeredDataSource);
     75        rendererUnit().removeSource(*m_registeredDataSource);
    6876}
    6977
    … …  
    96104{
    97105    // FIXME: We should create a unit for ourselves here or use the default unit if deviceId is matching.
    98     AudioMediaStreamTrackRendererUnit::singleton().setAudioOutputDevice(deviceId);
     106    rendererUnit().setAudioOutputDevice(deviceId);
    99107    m_shouldRecreateDataSource = true;
    100108}
    … …  
    105113
    106114    if (m_registeredDataSource)
    107         AudioMediaStreamTrackRendererUnit::singleton().removeSource(*m_registeredDataSource);
     115        rendererUnit().removeSource(*m_registeredDataSource);
    108116
    109117    if (!m_outputDescription)
    … …  
    116124    m_registeredDataSource->setLogger(logger(), logIdentifier());
    117125    m_registeredDataSource->setVolume(volume());
    118     AudioMediaStreamTrackRendererUnit::singleton().addResetObserver(m_resetObserver);
    119     AudioMediaStreamTrackRendererUnit::singleton().addSource(*m_registeredDataSource);
     126    rendererUnit().addResetObserver(m_resetObserver);
     127    rendererUnit().addSource(*m_registeredDataSource);
    120128}
    121129

  • trunk/Source/WebCore/platform/mediastream/cocoa/AudioMediaStreamTrackRendererCocoa.h

    r285024 r285027  
    4040class AudioSampleDataSource;
    4141class AudioSampleBufferList;
     42class BaseAudioMediaStreamTrackRendererUnit;
    4243class CAAudioStreamDescription;
    4344
    … …  
    4546    WTF_MAKE_FAST_ALLOCATED;
    4647public:
    47     AudioMediaStreamTrackRendererCocoa();
     48    AudioMediaStreamTrackRendererCocoa(Init&&);
    4849    ~AudioMediaStreamTrackRendererCocoa();
    4950
    … …  
    6061    void setRegisteredDataSource(RefPtr<AudioSampleDataSource>&&);
    6162
     63    BaseAudioMediaStreamTrackRendererUnit& rendererUnit();
     64
    6265    std::unique_ptr<CAAudioStreamDescription> m_outputDescription;
    6366    RefPtr<AudioSampleDataSource> m_dataSource; // Used in background thread.

  • trunk/Source/WebCore/platform/mediastream/cocoa/AudioMediaStreamTrackRendererUnit.h

    r284674 r285027  
    2929
    3030#include "AudioMediaStreamTrackRendererInternalUnit.h"
    31 #include <wtf/Forward.h>
     31#include "BaseAudioMediaStreamTrackRendererUnit.h"
    3232#include <wtf/HashSet.h>
    3333#include <wtf/Lock.h>
    34 #include <wtf/Observer.h>
    3534#include <wtf/UniqueRef.h>
    3635#include <wtf/Vector.h>
    … …  
    4443class AudioMediaStreamTrackRendererInternalUnit;
    4544
    46 class AudioMediaStreamTrackRendererUnit {
     45class AudioMediaStreamTrackRendererUnit : public BaseAudioMediaStreamTrackRendererUnit {
    4746public:
    4847    WEBCORE_EXPORT static AudioMediaStreamTrackRendererUnit& singleton();
    … …  
    5756    void reset();
    5857
    59     void setAudioOutputDevice(const String&);
     58    void retrieveFormatDescription(CompletionHandler<void(const CAAudioStreamDescription*)>&&);
    6059
    61     void addSource(Ref<AudioSampleDataSource>&&);
    62     void removeSource(AudioSampleDataSource&);
    63 
    64     using ResetObserver = Observer<void()>;
    65     void addResetObserver(ResetObserver& observer) { m_resetObservers.add(observer); }
    66     void retrieveFormatDescription(CompletionHandler<void(const CAAudioStreamDescription*)>&&);
     60    // BaseAudioMediaStreamTrackRendererUnit
     61    void setAudioOutputDevice(const String&) final;
     62    void addSource(Ref<AudioSampleDataSource>&&) final;
     63    void removeSource(AudioSampleDataSource&) final;
     64    void addResetObserver(ResetObserver& observer) final { m_resetObservers.add(observer); }
    6765
    6866private:

  • trunk/Source/WebCore/platform/mediastream/cocoa/BaseAudioMediaStreamTrackRendererUnit.h

    r285026 r285027  
    11/*
    2  * Copyright (C) 2020 Apple Inc. All rights reserved.
     2 * Copyright (C) 2021 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2424 */
    2525
    26 #include "config.h"
    27 #include "AudioMediaStreamTrackRenderer.h"
     26#pragma once
    2827
    2928#if ENABLE(MEDIA_STREAM)
    3029
    31 #include "Logging.h"
    32 
    33 #if PLATFORM(COCOA)
    34 #include "AudioMediaStreamTrackRendererCocoa.h"
    35 #endif
    36 
    37 namespace WTF {
    38 class MediaTime;
    39 }
     30#include <wtf/Forward.h>
     31#include <wtf/Observer.h>
    4032
    4133namespace WebCore {
    4234
    43 AudioMediaStreamTrackRenderer::RendererCreator AudioMediaStreamTrackRenderer::m_rendererCreator = nullptr;
    44 void AudioMediaStreamTrackRenderer::setCreator(RendererCreator creator)
    45 {
    46     m_rendererCreator = creator;
    47 }
     35class AudioSampleDataSource;
    4836
    49 std::unique_ptr<AudioMediaStreamTrackRenderer> AudioMediaStreamTrackRenderer::create()
    50 {
    51     if (m_rendererCreator)
    52         return m_rendererCreator();
     37class BaseAudioMediaStreamTrackRendererUnit {
     38public:
     39    virtual ~BaseAudioMediaStreamTrackRendererUnit() = default;
    5340
    54 #if PLATFORM(COCOA)
    55     return makeUnique<AudioMediaStreamTrackRendererCocoa>();
    56 #else
    57     return nullptr;
    58 #endif
    59 }
     41    virtual void addSource(Ref<AudioSampleDataSource>&&) = 0;
     42    virtual void removeSource(AudioSampleDataSource&) = 0;
    6043
    61 #if !RELEASE_LOG_DISABLED
    62 void AudioMediaStreamTrackRenderer::setLogger(const Logger& logger, const void* identifier)
    63 {
    64     m_logger = &logger;
    65     m_logIdentifier = identifier;
    66 }
     44    virtual void setAudioOutputDevice(const String&) = 0;
    6745
    68 WTFLogChannel& AudioMediaStreamTrackRenderer::logChannel() const
    69 {
    70     return LogMedia;
    71 }
    72 #endif
     46    using ResetObserver = Observer<void()>;
     47    virtual void addResetObserver(ResetObserver&) = 0;
     48};
    7349
    7450}

  • trunk/Source/WebCore/platform/mediastream/libwebrtc/LibWebRTCAudioModule.cpp

    r284135 r285027  
    3333#include <wtf/FastMalloc.h>
    3434
     35#if PLATFORM(COCOA)
     36#include "IncomingAudioMediaStreamTrackRendererUnit.h"
     37#endif
     38
    3539namespace WebCore {
    3640
    … …  
    3842    : m_queue(WorkQueue::create("WebKitWebRTCAudioModule", WorkQueue::QOS::UserInteractive))
    3943    , m_logTimer(*this, &LibWebRTCAudioModule::logTimerFired)
     44{
     45}
     46
     47LibWebRTCAudioModule::~LibWebRTCAudioModule()
    4048{
    4149}
    … …  
    6371    m_queue->dispatch([this, protectedThis = rtc::scoped_refptr<webrtc::AudioDeviceModule>(this)] {
    6472        m_pollingTime = MonotonicTime::now();
     73#if PLATFORM(COCOA)
     74        m_currentAudioSampleCount = 0;
     75#endif
    6576        pollAudioData();
    6677    });
    … …  
    130141        char data[LibWebRTCAudioFormat::sampleByteSize * channels * LibWebRTCAudioFormat::chunkSampleCount];
    131142        m_audioTransport->PullRenderData(LibWebRTCAudioFormat::sampleByteSize * 8, LibWebRTCAudioFormat::sampleRate, channels, LibWebRTCAudioFormat::chunkSampleCount, data, &elapsedTime, &ntpTime);
     143#if PLATFORM(COCOA)
     144        if (m_isRenderingIncomingAudio)
     145            m_incomingAudioMediaStreamTrackRendererUnit->newAudioChunkPushed();
     146        m_currentAudioSampleCount += LibWebRTCAudioFormat::chunkSampleCount;
     147#endif
    132148    }
    133149}
     150
     151#if PLATFORM(COCOA)
     152BaseAudioMediaStreamTrackRendererUnit& LibWebRTCAudioModule::incomingAudioMediaStreamTrackRendererUnit()
     153{
     154    if (!m_incomingAudioMediaStreamTrackRendererUnit)
     155        m_incomingAudioMediaStreamTrackRendererUnit = makeUnique<IncomingAudioMediaStreamTrackRendererUnit>(*this);
     156    return *m_incomingAudioMediaStreamTrackRendererUnit;
     157}
     158#endif
    134159
    135160} // namespace WebCore

  • trunk/Source/WebCore/platform/mediastream/libwebrtc/LibWebRTCAudioModule.h

    r284860 r285027  
    4040
    4141namespace WebCore {
     42class BaseAudioMediaStreamTrackRendererUnit;
     43class IncomingAudioMediaStreamTrackRendererUnit;
    4244
    4345// LibWebRTCAudioModule is pulling streamed data to ensure audio data is passed to the audio track.
    … …  
    4648public:
    4749    LibWebRTCAudioModule();
     50    ~LibWebRTCAudioModule();
    4851
    4952    static constexpr unsigned PollSamplesCount = 1;
     53    void ref() { AddRef(); }
     54    void deref() { Release(); }
     55
     56#if PLATFORM(COCOA)
     57    void startIncomingAudioRendering() { m_isRenderingIncomingAudio = true; }
     58    void stopIncomingAudioRendering() { m_isRenderingIncomingAudio = false; }
     59    BaseAudioMediaStreamTrackRendererUnit& incomingAudioMediaStreamTrackRendererUnit();
     60    uint64_t currentAudioSampleCount() const { return m_currentAudioSampleCount; }
     61#endif
    5062
    5163private:
    … …  
    136148    Timer m_logTimer;
    137149    int m_timeSpent { 0 };
     150
     151#if PLATFORM(COCOA)
     152    uint64_t m_currentAudioSampleCount { 0 };
     153    bool m_isRenderingIncomingAudio { false };
     154    std::unique_ptr<IncomingAudioMediaStreamTrackRendererUnit> m_incomingAudioMediaStreamTrackRendererUnit;
     155#endif
    138156};
    139157

  • trunk/Source/WebCore/platform/mediastream/libwebrtc/LibWebRTCProvider.cpp

    r284085 r285027  
    7171#endif
    7272
     73#if USE(LIBWEBRTC)
     74LibWebRTCProvider::LibWebRTCProvider()
     75{
     76}
     77#endif
     78
     79LibWebRTCProvider::~LibWebRTCProvider()
     80{
     81}
     82
    7383#if !USE(LIBWEBRTC) || !PLATFORM(COCOA)
    7484void LibWebRTCProvider::registerWebKitVP9Decoder()
    … …  
    272282}
    273283
     284void LibWebRTCProvider::clearFactory()
     285{
     286    m_audioModule = nullptr;
     287    m_factory = nullptr;
     288}
     289
    274290rtc::scoped_refptr<webrtc::PeerConnectionFactoryInterface> LibWebRTCProvider::createPeerConnectionFactory(rtc::Thread* networkThread, rtc::Thread* signalingThread)
    275291{
    276     auto audioModule = rtc::scoped_refptr<webrtc::AudioDeviceModule>(new rtc::RefCountedObject<LibWebRTCAudioModule>());
     292    ASSERT(!m_audioModule);
     293    auto audioModule = rtc::scoped_refptr<LibWebRTCAudioModule>(new rtc::RefCountedObject<LibWebRTCAudioModule>());
     294    m_audioModule = audioModule.get();
    277295
    278296    return webrtc::CreatePeerConnectionFactory(networkThread, signalingThread, signalingThread, WTFMove(audioModule), webrtc::CreateBuiltinAudioEncoderFactory(), webrtc::CreateBuiltinAudioDecoderFactory(), createEncoderFactory(), createDecoderFactory(), nullptr, nullptr, nullptr);

  • trunk/Source/WebCore/platform/mediastream/libwebrtc/LibWebRTCProvider.h

    r284085 r285027  
    8080    static UniqueRef<LibWebRTCProvider> create();
    8181
    82     virtual ~LibWebRTCProvider() = default;
     82    virtual ~LibWebRTCProvider();
    8383
    8484    static bool webRTCAvailable();
    … …  
    104104
    105105    webrtc::PeerConnectionFactoryInterface* factory();
     106    LibWebRTCAudioModule* audioModule();
    106107
    107108    // FIXME: Make these methods not static.
    … …  
    135136    std::optional<RTCRtpCapabilities> senderCapabilities(const String& kind);
    136137
    137     void clearFactory() { m_factory = nullptr; }
     138    void clearFactory();
    138139
    139140    virtual void setLoggingLevel(WTFLogLevel);
    … …  
    151152
    152153protected:
    153     LibWebRTCProvider() = default;
     154    LibWebRTCProvider();
    154155
    155156    rtc::scoped_refptr<webrtc::PeerConnectionInterface> createPeerConnection(webrtc::PeerConnectionObserver&, rtc::NetworkManager&, rtc::PacketSocketFactory&, webrtc::PeerConnectionInterface::RTCConfiguration&&, std::unique_ptr<webrtc::AsyncResolverFactory>&&);
    … …  
    172173    bool m_useNetworkThreadWithSocketServer { true };
    173174
     175    RefPtr<LibWebRTCAudioModule> m_audioModule;
    174176    rtc::scoped_refptr<webrtc::PeerConnectionFactoryInterface> m_factory;
    175177    bool m_disableNonLocalhostConnections { false };
    … …  
    188190};
    189191
     192#if USE(LIBWEBRTC)
     193inline LibWebRTCAudioModule* LibWebRTCProvider::audioModule()
     194{
     195    return m_audioModule.get();
     196}
     197#endif
     198
    190199} // namespace WebCore

  • trunk/Source/WebCore/platform/mediastream/mac/RealtimeIncomingAudioSourceCocoa.cpp

    r279459 r285027  
    133133    }
    134134
    135     CMTime startTime = PAL::CMTimeMake(m_numberOfFrames, sampleRate);
     135    CMTime startTime = PAL::CMTimeMake(audioModule() ? audioModule()->currentAudioSampleCount() : m_numberOfFrames, LibWebRTCAudioFormat::sampleRate);
    136136    auto mediaTime = PAL::toMediaTime(startTime);
    137137    m_numberOfFrames += numberOfFrames;
Note: See TracChangeset for help on using the changeset viewer.