⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 259816 in webkit


Ignore:
Timestamp:
Apr 9, 2020, 12:44:21 PM (6 years ago)
Author:
youenn@apple.com
Message:

Introduce a RealtimeMediaSource observer dedicated to receiving audio samples
​https://bugs.webkit.org/show_bug.cgi?id=210180

Reviewed by Eric Carlson.

Source/WebCore:

RealtimeMediaSource::Observer was called on various threads which was making iterating through them difficult.
Observers were copied to a vector for iterating each time and locking was happening for each indivudual observer.

Instead of doing that, RealtimeMediaSource::Observer no longer gives access to audio samples, all its methods are called from the main thread.
AudioSampleObserver is introduced with a single method to receive audio samples.
RealtimeMediaSource keeps a set of AudioSampleObserver separated from its existing Observer hashset.
These observers can only be added/removed from the main thread while audio sample delivery usually happens from a background thread.
We no longer need to copy the audio sample observers in a vector and lock only once when iterating the audio sample observers.

This change requires to update MediaStreamTrackPrivate and RealtimeMediaSource observers.
This includes WebAudio, Audio Renderer and Media Recorder.
Each corresponding class is now registering itself as a MediaStreamTrackPrivate::Observer and when needed as a RealtimeMediaSource::Observer.
This allows removing the additional copy-in-vector/lock that was happening when iterating over MediaStreamTrackPrivate::Observers for audio samples.

This change also allows for a consumer to always observe the MediaStreamTrackPrivate but receive audio samples only when needed.
We change the AudioTrackPrivateMediaStream implementation so that, if the renderer is muted, not playing, the track is muted or not enabled,
it no longer receives audio samples.
This is especially useful in the typical case where a video element displays the locally captured stream (including a local audio track) but is muted
to prevent feedback loop.
For that reason, the player is now setting the muted state of the renderer instead of setting volume to 0.

Covered by existing test sets.

  • Modules/mediarecorder/MediaRecorder.cpp:

(WebCore::MediaRecorder::createMediaRecorderPrivate):

  • Modules/mediarecorder/MediaRecorder.h:
  • Modules/mediarecorder/MediaRecorderProvider.cpp:

(WebCore::MediaRecorderProvider::createMediaRecorderPrivate):

  • Modules/mediarecorder/MediaRecorderProvider.h:
  • loader/EmptyClients.cpp:
  • platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:

(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setVolume):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setMuted):

  • platform/mediarecorder/MediaRecorderPrivate.cpp:

(WebCore::MediaRecorderPrivate::selectTracks):

  • platform/mediarecorder/MediaRecorderPrivate.h:

(WebCore::MediaRecorderPrivate::setAudioSource):
(WebCore::MediaRecorderPrivate::~MediaRecorderPrivate):

  • platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp:

(WebCore::MediaRecorderPrivateAVFImpl::create):
(WebCore::MediaRecorderPrivateAVFImpl::~MediaRecorderPrivateAVFImpl):
(WebCore::MediaRecorderPrivateAVFImpl::audioSamplesAvailable):
(WebCore::MediaRecorderPrivateAVFImpl::stopRecording):
(WebCore::MediaRecorderPrivateAVFImpl::fetchData):

  • platform/mediarecorder/MediaRecorderPrivateAVFImpl.h:
  • platform/mediarecorder/MediaRecorderPrivateMock.cpp:

(WebCore::MediaRecorderPrivateMock::MediaRecorderPrivateMock):
(WebCore::MediaRecorderPrivateMock::~MediaRecorderPrivateMock):
(WebCore::MediaRecorderPrivateMock::stopRecording):
(WebCore::MediaRecorderPrivateMock::sampleBufferUpdated):
(WebCore::MediaRecorderPrivateMock::audioSamplesAvailable):
(WebCore::MediaRecorderPrivateMock::generateMockCounterString):
(WebCore::MediaRecorderPrivateMock::fetchData):

  • platform/mediarecorder/MediaRecorderPrivateMock.h:
  • platform/mediastream/AudioTrackPrivateMediaStream.cpp:

(WebCore::AudioTrackPrivateMediaStream::AudioTrackPrivateMediaStream):
(WebCore::AudioTrackPrivateMediaStream::clear):
(WebCore::AudioTrackPrivateMediaStream::play):
(WebCore::AudioTrackPrivateMediaStream::pause):
(WebCore::AudioTrackPrivateMediaStream::setMuted):
(WebCore::AudioTrackPrivateMediaStream::audioSamplesAvailable):
(WebCore::AudioTrackPrivateMediaStream::trackMutedChanged):
(WebCore::AudioTrackPrivateMediaStream::trackEnabledChanged):
(WebCore::AudioTrackPrivateMediaStream::trackEnded):
(WebCore::AudioTrackPrivateMediaStream::updateRenderer):
(WebCore::AudioTrackPrivateMediaStream::startRenderer):
(WebCore::AudioTrackPrivateMediaStream::stopRenderer):

  • platform/mediastream/AudioTrackPrivateMediaStream.h:
  • platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp:

(WebCore::webkitMediaStreamSrcFinalize):
(WebCore::webkitMediaStreamSrcChangeState):
(WebCore::webkitMediaStreamSrcSetupSrc):

  • platform/mediastream/MediaStreamPrivate.cpp:

(WebCore::MediaStreamPrivate::forEachTrack):

  • platform/mediastream/MediaStreamPrivate.h:
  • platform/mediastream/MediaStreamTrackPrivate.cpp:

(WebCore::MediaStreamTrackPrivate::forEachObserver const):
(WebCore::MediaStreamTrackPrivate::addObserver):
(WebCore::MediaStreamTrackPrivate::removeObserver):
(WebCore::MediaStreamTrackPrivate::hasStartedProducingAudioData):

  • platform/mediastream/MediaStreamTrackPrivate.h:
  • platform/mediastream/RealtimeMediaSource.cpp:

(WebCore::RealtimeMediaSource::addAudioSampleObserver):
(WebCore::RealtimeMediaSource::removeAudioSampleObserver):
(WebCore::RealtimeMediaSource::addObserver):
(WebCore::RealtimeMediaSource::removeObserver):
(WebCore::RealtimeMediaSource::forEachObserver const):
(WebCore::RealtimeMediaSource::audioSamplesAvailable):

  • platform/mediastream/RealtimeMediaSource.h:
  • platform/mediastream/RealtimeOutgoingAudioSource.cpp:

(WebCore::RealtimeOutgoingAudioSource::~RealtimeOutgoingAudioSource):
(WebCore::RealtimeOutgoingAudioSource::observeSource):
(WebCore::RealtimeOutgoingAudioSource::unobserveSource):

  • platform/mediastream/RealtimeOutgoingAudioSource.h:
  • platform/mediastream/WebAudioSourceProvider.h:
  • platform/mediastream/mac/WebAudioSourceProviderAVFObjC.h:
  • platform/mediastream/mac/WebAudioSourceProviderAVFObjC.mm:

(WebCore::WebAudioSourceProviderAVFObjC::WebAudioSourceProviderAVFObjC):
(WebCore::WebAudioSourceProviderAVFObjC::~WebAudioSourceProviderAVFObjC):
(WebCore::WebAudioSourceProviderAVFObjC::trackEnabledChanged):
(WebCore::WebAudioSourceProviderAVFObjC::audioSamplesAvailable):

  • testing/Internals.cpp:

(WebCore::Internals::~Internals):
(WebCore::createRecorderMockSource):
(WebCore::Internals::observeMediaStreamTrack):
(WebCore::Internals::videoSampleAvailable):

  • testing/Internals.h:

Source/WebKit:

Update SourceProxy to be an AudioSampleObserver.
Update MediaRecorder to only listen for audio samples if GPU remote writer is successfully created.
It also stops listening to audio samples as soon as stopping the recording.

  • UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp:

(WebKit::UserMediaCaptureManagerProxy::SourceProxy::SourceProxy):
(WebKit::UserMediaCaptureManagerProxy::SourceProxy::~SourceProxy):

  • WebProcess/GPU/webrtc/MediaRecorderPrivate.cpp:

(WebKit::MediaRecorderPrivate::MediaRecorderPrivate):
(WebKit::MediaRecorderPrivate::~MediaRecorderPrivate):
(WebKit::MediaRecorderPrivate::audioSamplesAvailable):
(WebKit::MediaRecorderPrivate::stopRecording):

  • WebProcess/GPU/webrtc/MediaRecorderPrivate.h:
  • WebProcess/GPU/webrtc/MediaRecorderProvider.cpp:

(WebKit::MediaRecorderProvider::createMediaRecorderPrivate):

  • WebProcess/GPU/webrtc/MediaRecorderProvider.h:
Location:
trunk/Source
Files:
35 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebCore/ChangeLog

    r259813 r259816  
     12020-04-09  Youenn Fablet  <youenn@apple.com>
     2
     3        Introduce a RealtimeMediaSource observer dedicated to receiving audio samples
     4        https://bugs.webkit.org/show_bug.cgi?id=210180
     5
     6        Reviewed by Eric Carlson.
     7
     8        RealtimeMediaSource::Observer was called on various threads which was making iterating through them difficult.
     9        Observers were copied to a vector for iterating each time and locking was happening for each indivudual observer.
     10
     11        Instead of doing that, RealtimeMediaSource::Observer no longer gives access to audio samples, all its methods are called from the main thread.
     12        AudioSampleObserver is introduced with a single method to receive audio samples.
     13        RealtimeMediaSource keeps a set of AudioSampleObserver separated from its existing Observer hashset.
     14        These observers can only be added/removed from the main thread while audio sample delivery usually happens from a background thread.
     15        We no longer need to copy the audio sample observers in a vector and lock only once when iterating the audio sample observers.
     16
     17        This change requires to update MediaStreamTrackPrivate and RealtimeMediaSource observers.
     18        This includes WebAudio, Audio Renderer and Media Recorder.
     19        Each corresponding class is now registering itself as a MediaStreamTrackPrivate::Observer and when needed as a RealtimeMediaSource::Observer.
     20        This allows removing the additional copy-in-vector/lock that was happening when iterating over MediaStreamTrackPrivate::Observers for audio samples.
     21
     22        This change also allows for a consumer to always observe the MediaStreamTrackPrivate but receive audio samples only when needed.
     23        We change the AudioTrackPrivateMediaStream implementation so that, if the renderer is muted, not playing, the track is muted or not enabled,
     24        it no longer receives audio samples.
     25        This is especially useful in the typical case where a video element displays the locally captured stream (including a local audio track) but is muted
     26        to prevent feedback loop.
     27        For that reason, the player is now setting the muted state of the renderer instead of setting volume to 0.
     28
     29        Covered by existing test sets.
     30
     31        * Modules/mediarecorder/MediaRecorder.cpp:
     32        (WebCore::MediaRecorder::createMediaRecorderPrivate):
     33        * Modules/mediarecorder/MediaRecorder.h:
     34        * Modules/mediarecorder/MediaRecorderProvider.cpp:
     35        (WebCore::MediaRecorderProvider::createMediaRecorderPrivate):
     36        * Modules/mediarecorder/MediaRecorderProvider.h:
     37        * loader/EmptyClients.cpp:
     38        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
     39        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setVolume):
     40        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setMuted):
     41        * platform/mediarecorder/MediaRecorderPrivate.cpp:
     42        (WebCore::MediaRecorderPrivate::selectTracks):
     43        * platform/mediarecorder/MediaRecorderPrivate.h:
     44        (WebCore::MediaRecorderPrivate::setAudioSource):
     45        (WebCore::MediaRecorderPrivate::~MediaRecorderPrivate):
     46        * platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp:
     47        (WebCore::MediaRecorderPrivateAVFImpl::create):
     48        (WebCore::MediaRecorderPrivateAVFImpl::~MediaRecorderPrivateAVFImpl):
     49        (WebCore::MediaRecorderPrivateAVFImpl::audioSamplesAvailable):
     50        (WebCore::MediaRecorderPrivateAVFImpl::stopRecording):
     51        (WebCore::MediaRecorderPrivateAVFImpl::fetchData):
     52        * platform/mediarecorder/MediaRecorderPrivateAVFImpl.h:
     53        * platform/mediarecorder/MediaRecorderPrivateMock.cpp:
     54        (WebCore::MediaRecorderPrivateMock::MediaRecorderPrivateMock):
     55        (WebCore::MediaRecorderPrivateMock::~MediaRecorderPrivateMock):
     56        (WebCore::MediaRecorderPrivateMock::stopRecording):
     57        (WebCore::MediaRecorderPrivateMock::sampleBufferUpdated):
     58        (WebCore::MediaRecorderPrivateMock::audioSamplesAvailable):
     59        (WebCore::MediaRecorderPrivateMock::generateMockCounterString):
     60        (WebCore::MediaRecorderPrivateMock::fetchData):
     61        * platform/mediarecorder/MediaRecorderPrivateMock.h:
     62        * platform/mediastream/AudioTrackPrivateMediaStream.cpp:
     63        (WebCore::AudioTrackPrivateMediaStream::AudioTrackPrivateMediaStream):
     64        (WebCore::AudioTrackPrivateMediaStream::clear):
     65        (WebCore::AudioTrackPrivateMediaStream::play):
     66        (WebCore::AudioTrackPrivateMediaStream::pause):
     67        (WebCore::AudioTrackPrivateMediaStream::setMuted):
     68        (WebCore::AudioTrackPrivateMediaStream::audioSamplesAvailable):
     69        (WebCore::AudioTrackPrivateMediaStream::trackMutedChanged):
     70        (WebCore::AudioTrackPrivateMediaStream::trackEnabledChanged):
     71        (WebCore::AudioTrackPrivateMediaStream::trackEnded):
     72        (WebCore::AudioTrackPrivateMediaStream::updateRenderer):
     73        (WebCore::AudioTrackPrivateMediaStream::startRenderer):
     74        (WebCore::AudioTrackPrivateMediaStream::stopRenderer):
     75        * platform/mediastream/AudioTrackPrivateMediaStream.h:
     76        * platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp:
     77        (WebCore::webkitMediaStreamSrcFinalize):
     78        (WebCore::webkitMediaStreamSrcChangeState):
     79        (WebCore::webkitMediaStreamSrcSetupSrc):
     80        * platform/mediastream/MediaStreamPrivate.cpp:
     81        (WebCore::MediaStreamPrivate::forEachTrack):
     82        * platform/mediastream/MediaStreamPrivate.h:
     83        * platform/mediastream/MediaStreamTrackPrivate.cpp:
     84        (WebCore::MediaStreamTrackPrivate::forEachObserver const):
     85        (WebCore::MediaStreamTrackPrivate::addObserver):
     86        (WebCore::MediaStreamTrackPrivate::removeObserver):
     87        (WebCore::MediaStreamTrackPrivate::hasStartedProducingAudioData):
     88        * platform/mediastream/MediaStreamTrackPrivate.h:
     89        * platform/mediastream/RealtimeMediaSource.cpp:
     90        (WebCore::RealtimeMediaSource::addAudioSampleObserver):
     91        (WebCore::RealtimeMediaSource::removeAudioSampleObserver):
     92        (WebCore::RealtimeMediaSource::addObserver):
     93        (WebCore::RealtimeMediaSource::removeObserver):
     94        (WebCore::RealtimeMediaSource::forEachObserver const):
     95        (WebCore::RealtimeMediaSource::audioSamplesAvailable):
     96        * platform/mediastream/RealtimeMediaSource.h:
     97        * platform/mediastream/RealtimeOutgoingAudioSource.cpp:
     98        (WebCore::RealtimeOutgoingAudioSource::~RealtimeOutgoingAudioSource):
     99        (WebCore::RealtimeOutgoingAudioSource::observeSource):
     100        (WebCore::RealtimeOutgoingAudioSource::unobserveSource):
     101        * platform/mediastream/RealtimeOutgoingAudioSource.h:
     102        * platform/mediastream/WebAudioSourceProvider.h:
     103        * platform/mediastream/mac/WebAudioSourceProviderAVFObjC.h:
     104        * platform/mediastream/mac/WebAudioSourceProviderAVFObjC.mm:
     105        (WebCore::WebAudioSourceProviderAVFObjC::WebAudioSourceProviderAVFObjC):
     106        (WebCore::WebAudioSourceProviderAVFObjC::~WebAudioSourceProviderAVFObjC):
     107        (WebCore::WebAudioSourceProviderAVFObjC::trackEnabledChanged):
     108        (WebCore::WebAudioSourceProviderAVFObjC::audioSamplesAvailable):
     109        * testing/Internals.cpp:
     110        (WebCore::Internals::~Internals):
     111        (WebCore::createRecorderMockSource):
     112        (WebCore::Internals::observeMediaStreamTrack):
     113        (WebCore::Internals::videoSampleAvailable):
     114        * testing/Internals.h:
     115
    11162020-04-09  Antti Koivisto  <antti@apple.com>
    2117

  • trunk/Source/WebCore/Modules/mediarecorder/MediaRecorder.cpp

    r259252 r259816  
    4545WTF_MAKE_ISO_ALLOCATED_IMPL(MediaRecorder);
    4646
    47 creatorFunction MediaRecorder::m_customCreator = nullptr;
     47MediaRecorder::CreatorFunction MediaRecorder::m_customCreator = nullptr;
    4848
    4949ExceptionOr<Ref<MediaRecorder>> MediaRecorder::create(Document& document, Ref<MediaStream>&& stream, Options&& options)
    … …  
    6060}
    6161
    62 void MediaRecorder::setCustomPrivateRecorderCreator(creatorFunction creator)
     62void MediaRecorder::setCustomPrivateRecorderCreator(CreatorFunction creator)
    6363{
    6464    m_customCreator = creator;
    6565}
    6666
    67 std::unique_ptr<MediaRecorderPrivate> MediaRecorder::createMediaRecorderPrivate(Document& document, const MediaStreamPrivate& stream)
     67std::unique_ptr<MediaRecorderPrivate> MediaRecorder::createMediaRecorderPrivate(Document& document, MediaStreamPrivate& stream)
    6868{
    6969    if (m_customCreator)
    70         return m_customCreator();
     70        return m_customCreator(stream);
    7171
    7272#if PLATFORM(COCOA)
    … …  
    228228}
    229229
    230 void MediaRecorder::audioSamplesAvailable(MediaStreamTrackPrivate& track, const MediaTime& mediaTime, const PlatformAudioData& audioData, const AudioStreamDescription& description, size_t sampleCount)
    231 {
    232     m_private->audioSamplesAvailable(track, mediaTime, audioData, description, sampleCount);
    233 }
    234 
    235230bool MediaRecorder::virtualHasPendingActivity() const
    236231{

  • trunk/Source/WebCore/Modules/mediarecorder/MediaRecorder.h

    r259252 r259816  
    3939class MediaRecorderPrivate;
    4040
    41 typedef std::unique_ptr<MediaRecorderPrivate>(*creatorFunction)();
    42 
    4341class MediaRecorder final
    4442    : public ActiveDOMObject
    … …  
    6361    static ExceptionOr<Ref<MediaRecorder>> create(Document&, Ref<MediaStream>&&, Options&& = { });
    6462   
    65     WEBCORE_EXPORT static void setCustomPrivateRecorderCreator(creatorFunction);
     63    using CreatorFunction = std::unique_ptr<MediaRecorderPrivate>(*)(MediaStreamPrivate&);
     64
     65    WEBCORE_EXPORT static void setCustomPrivateRecorderCreator(CreatorFunction);
    6666   
    6767    RecordingState state() const { return m_state; }
    … …  
    7979    MediaRecorder(Document&, Ref<MediaStream>&&, std::unique_ptr<MediaRecorderPrivate>&&, Options&& = { });
    8080
    81     static std::unique_ptr<MediaRecorderPrivate> createMediaRecorderPrivate(Document&, const MediaStreamPrivate&);
     81    static std::unique_ptr<MediaRecorderPrivate> createMediaRecorderPrivate(Document&, MediaStreamPrivate&);
    8282   
    8383    Document* document() const;
    … …  
    108108    void trackEnabledChanged(MediaStreamTrackPrivate&) final { };
    109109    void sampleBufferUpdated(MediaStreamTrackPrivate&, MediaSample&) final;
    110     void audioSamplesAvailable(MediaStreamTrackPrivate&, const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;
    111    
    112     static creatorFunction m_customCreator;
     110
     111    static CreatorFunction m_customCreator;
    113112   
    114113    Options m_options;

  • trunk/Source/WebCore/Modules/mediarecorder/MediaRecorderProvider.cpp

    r256493 r259816  
    3333namespace WebCore {
    3434
    35 std::unique_ptr<MediaRecorderPrivate> MediaRecorderProvider::createMediaRecorderPrivate(const MediaStreamPrivate& stream)
     35std::unique_ptr<MediaRecorderPrivate> MediaRecorderProvider::createMediaRecorderPrivate(MediaStreamPrivate& stream)
    3636{
    3737    return MediaRecorderPrivateAVFImpl::create(stream);

  • trunk/Source/WebCore/Modules/mediarecorder/MediaRecorderProvider.h

    r254194 r259816  
    3838
    3939#if ENABLE(MEDIA_STREAM) && PLATFORM(COCOA)
    40     virtual std::unique_ptr<MediaRecorderPrivate> createMediaRecorderPrivate(const MediaStreamPrivate&);
     40    virtual std::unique_ptr<MediaRecorderPrivate> createMediaRecorderPrivate(MediaStreamPrivate&);
    4141#endif
    4242

  • trunk/Source/WebCore/loader/EmptyClients.cpp

    r258628 r259816  
    583583private:
    584584#if ENABLE(MEDIA_STREAM) && PLATFORM(COCOA)
    585     std::unique_ptr<MediaRecorderPrivate> createMediaRecorderPrivate(const MediaStreamPrivate&) final { return nullptr; }
     585    std::unique_ptr<MediaRecorderPrivate> createMediaRecorderPrivate(MediaStreamPrivate&) final { return nullptr; }
    586586#endif
    587587};

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm

    r258503 r259816  
    561561    m_volume = volume;
    562562    for (const auto& track : m_audioTrackMap.values())
    563         track->setVolume(m_muted ? 0 : m_volume);
     563        track->setVolume(m_volume);
    564564}
    565565
    … …  
    572572    m_muted = muted;
    573573    for (const auto& track : m_audioTrackMap.values())
    574         track->setVolume(m_muted ? 0 : m_volume);
     574        track->setMuted(m_muted);
    575575}
    576576

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivate.cpp

    r256658 r259816  
    3333namespace WebCore {
    3434
    35 MediaRecorderPrivate::AudioVideoSelectedTracks MediaRecorderPrivate::selectTracks(const MediaStreamPrivate& stream)
     35MediaRecorderPrivate::AudioVideoSelectedTracks MediaRecorderPrivate::selectTracks(MediaStreamPrivate& stream)
    3636{
    3737    AudioVideoSelectedTracks selectedTracks;

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivate.h

    r256658 r259816  
    2828#include <wtf/Forward.h>
    2929#include "Exception.h"
     30#include "RealtimeMediaSource.h"
    3031
    3132#if ENABLE(MEDIA_STREAM)
    … …  
    4445class SharedBuffer;
    4546
    46 class MediaRecorderPrivate {
     47class MediaRecorderPrivate :
     48    public RealtimeMediaSource::AudioSampleObserver {
    4749public:
    48     virtual ~MediaRecorderPrivate() = default;
     50    ~MediaRecorderPrivate();
    4951
    5052    struct AudioVideoSelectedTracks {
    51         const MediaStreamTrackPrivate* audioTrack { nullptr };
    52         const MediaStreamTrackPrivate* videoTrack { nullptr };
     53        MediaStreamTrackPrivate* audioTrack { nullptr };
     54        MediaStreamTrackPrivate* videoTrack { nullptr };
    5355    };
    54     WEBCORE_EXPORT static AudioVideoSelectedTracks selectTracks(const MediaStreamPrivate&);
     56    WEBCORE_EXPORT static AudioVideoSelectedTracks selectTracks(MediaStreamPrivate&);
    5557
    5658    virtual void sampleBufferUpdated(const MediaStreamTrackPrivate&, MediaSample&) = 0;
    57     virtual void audioSamplesAvailable(const MediaStreamTrackPrivate&, const WTF::MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) = 0;
    5859
    59     virtual void fetchData(CompletionHandler<void(RefPtr<SharedBuffer>&&, const String& mimeType)>&&) = 0;
    60     virtual void stopRecording() { };
     60    using FetchDataCallback = CompletionHandler<void(RefPtr<SharedBuffer>&&, const String& mimeType)>;
     61    virtual void fetchData(FetchDataCallback&&) = 0;
     62    virtual void stopRecording() = 0;
    6163
    6264    using ErrorCallback = Function<void(Optional<Exception>&&)>;
    … …  
    6466
    6567protected:
     68    void setAudioSource(RefPtr<RealtimeMediaSource>&&);
     69
     70protected:
    6671    ErrorCallback m_errorCallback;
     72
     73private:
     74    RefPtr<RealtimeMediaSource> m_audioSource;
    6775};
     76
     77inline void MediaRecorderPrivate::setAudioSource(RefPtr<RealtimeMediaSource>&& audioSource)
     78{
     79    if (m_audioSource)
     80        m_audioSource->removeAudioSampleObserver(*this);
     81
     82    m_audioSource = WTFMove(audioSource);
     83
     84    if (m_audioSource)
     85        m_audioSource->addAudioSampleObserver(*this);
     86}
     87
     88inline MediaRecorderPrivate::~MediaRecorderPrivate()
     89{
     90    if (m_audioSource)
     91        m_audioSource->removeAudioSampleObserver(*this);
     92}
    6893
    6994} // namespace WebCore

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp

    r256658 r259816  
    3737namespace WebCore {
    3838
    39 std::unique_ptr<MediaRecorderPrivateAVFImpl> MediaRecorderPrivateAVFImpl::create(const MediaStreamPrivate& stream)
     39std::unique_ptr<MediaRecorderPrivateAVFImpl> MediaRecorderPrivateAVFImpl::create(MediaStreamPrivate& stream)
    4040{
    4141    // FIXME: we will need to implement support for multiple audio/video tracks
    … …  
    5252    if (selectedTracks.audioTrack)
    5353        audioTrackId = selectedTracks.audioTrack->id();
     54
    5455    String videoTrackId;
    5556    if (selectedTracks.videoTrack)
    5657        videoTrackId = selectedTracks.videoTrack->id();
    5758
    58     return makeUnique<MediaRecorderPrivateAVFImpl>(writer.releaseNonNull(), WTFMove(audioTrackId), WTFMove(videoTrackId));
     59    auto recorder = makeUnique<MediaRecorderPrivateAVFImpl>(writer.releaseNonNull(), WTFMove(audioTrackId), WTFMove(videoTrackId));
     60    if (selectedTracks.audioTrack)
     61        recorder->setAudioSource(&selectedTracks.audioTrack->source());
     62    return recorder;
    5963}
    6064
    … …  
    6670}
    6771
     72MediaRecorderPrivateAVFImpl::~MediaRecorderPrivateAVFImpl()
     73{
     74    setAudioSource(nullptr);
     75}
     76
    6877void MediaRecorderPrivateAVFImpl::sampleBufferUpdated(const MediaStreamTrackPrivate& track, MediaSample& sampleBuffer)
    6978{
    … …  
    7382}
    7483
    75 void MediaRecorderPrivateAVFImpl::audioSamplesAvailable(const MediaStreamTrackPrivate& track, const WTF::MediaTime& mediaTime, const PlatformAudioData& data, const AudioStreamDescription& description, size_t sampleCount)
     84void MediaRecorderPrivateAVFImpl::audioSamplesAvailable(const WTF::MediaTime& mediaTime, const PlatformAudioData& data, const AudioStreamDescription& description, size_t sampleCount)
    7685{
    77     if (track.id() != m_recordedAudioTrackID)
    78         return;
    7986    ASSERT(is<WebAudioBufferList>(data));
    8087    ASSERT(description.platformDescription().type == PlatformDescription::CAAudioStreamBasicType);
    … …  
    8491void MediaRecorderPrivateAVFImpl::stopRecording()
    8592{
     93    setAudioSource(nullptr);
    8694    m_writer->stopRecording();
    8795}
    8896
    89 void MediaRecorderPrivateAVFImpl::fetchData(CompletionHandler<void(RefPtr<SharedBuffer>&&, const String&)>&& completionHandler)
     97void MediaRecorderPrivateAVFImpl::fetchData(FetchDataCallback&& completionHandler)
    9098{
    9199    m_writer->fetchData([completionHandler = WTFMove(completionHandler), mimeType = mimeType()](auto&& buffer) mutable {

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateAVFImpl.h

    r256493 r259816  
    3434class MediaStreamPrivate;
    3535
    36 class MediaRecorderPrivateAVFImpl final : public MediaRecorderPrivate {
     36class MediaRecorderPrivateAVFImpl final
     37    : public MediaRecorderPrivate {
    3738    WTF_MAKE_FAST_ALLOCATED;
    3839public:
    39     static std::unique_ptr<MediaRecorderPrivateAVFImpl> create(const MediaStreamPrivate&);
     40    static std::unique_ptr<MediaRecorderPrivateAVFImpl> create(MediaStreamPrivate&);
     41    ~MediaRecorderPrivateAVFImpl();
    4042
    4143private:
    … …  
    4446    friend std::unique_ptr<MediaRecorderPrivateAVFImpl> std::make_unique<MediaRecorderPrivateAVFImpl>(Ref<MediaRecorderPrivateWriter>&&, String&&, String&&);
    4547
     48    // MediaRecorderPrivate
    4649    void sampleBufferUpdated(const MediaStreamTrackPrivate&, MediaSample&) final;
    47     void audioSamplesAvailable(const MediaStreamTrackPrivate&, const WTF::MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;
    48     void fetchData(CompletionHandler<void(RefPtr<SharedBuffer>&&, const String&)>&&) final;
     50    void fetchData(FetchDataCallback&&) final;
     51    void audioSamplesAvailable(const WTF::MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;
     52
    4953    const String& mimeType();
    5054    void stopRecording();
    51    
     55
    5256    Ref<MediaRecorderPrivateWriter> m_writer;
    5357    String m_recordedAudioTrackID;

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateMock.cpp

    r254194 r259816  
    3434namespace WebCore {
    3535
    36 void MediaRecorderPrivateMock::sampleBufferUpdated(const MediaStreamTrackPrivate& track, MediaSample&)
     36MediaRecorderPrivateMock::MediaRecorderPrivateMock(MediaStreamPrivate& stream)
    3737{
    38     generateMockString(track);
     38    auto selectedTracks = MediaRecorderPrivate::selectTracks(stream);
     39    if (selectedTracks.audioTrack) {
     40        m_audioTrackID = selectedTracks.audioTrack->id();
     41        setAudioSource(&selectedTracks.audioTrack->source());
     42    }
     43    if (selectedTracks.videoTrack)
     44        m_videoTrackID = selectedTracks.videoTrack->id();
    3945}
    4046
    41 void MediaRecorderPrivateMock::audioSamplesAvailable(const MediaStreamTrackPrivate& track, const WTF::MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t)
     47MediaRecorderPrivateMock::~MediaRecorderPrivateMock()
    4248{
    43     generateMockString(track);
     49    setAudioSource(nullptr);
    4450}
    4551
    46 void MediaRecorderPrivateMock::generateMockString(const MediaStreamTrackPrivate& track)
     52void MediaRecorderPrivateMock::stopRecording()
     53{
     54    setAudioSource(nullptr);
     55}
     56
     57void MediaRecorderPrivateMock::sampleBufferUpdated(const MediaStreamTrackPrivate&, MediaSample&)
    4758{
    4859    auto locker = holdLock(m_bufferLock);
    49     if (track.type() == RealtimeMediaSource::Type::Audio)
    50         m_buffer.append("Audio Track ID: ");
    51     else
    52         m_buffer.append("Video Track ID: ");
    53     m_buffer.append(track.id());
     60    m_buffer.append("Video Track ID: ");
     61    m_buffer.append(m_videoTrackID);
     62    generateMockCounterString();
     63}
     64
     65void MediaRecorderPrivateMock::audioSamplesAvailable(const WTF::MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t)
     66{
     67    auto locker = holdLock(m_bufferLock);
     68    m_buffer.append("Audio Track ID: ");
     69    m_buffer.append(m_audioTrackID);
     70    generateMockCounterString();
     71}
     72
     73void MediaRecorderPrivateMock::generateMockCounterString()
     74{
    5475    m_buffer.append(" Counter: ");
    5576    m_buffer.appendNumber(++m_counter);
    … …  
    5778}
    5879
    59 void MediaRecorderPrivateMock::fetchData(CompletionHandler<void(RefPtr<SharedBuffer>&&, const String&)>&& completionHandler)
     80void MediaRecorderPrivateMock::fetchData(FetchDataCallback&& completionHandler)
    6081{
    6182    auto locker = holdLock(m_bufferLock);

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateMock.h

    r254194 r259816  
    3535class MediaStreamTrackPrivate;
    3636
    37 class WEBCORE_EXPORT MediaRecorderPrivateMock final : public MediaRecorderPrivate {
     37class WEBCORE_EXPORT MediaRecorderPrivateMock final
     38    : public MediaRecorderPrivate {
     39public:
     40    explicit MediaRecorderPrivateMock(MediaStreamPrivate&);
     41    ~MediaRecorderPrivateMock();
     42
    3843private:
     44    // MediaRecorderPrivate
    3945    void sampleBufferUpdated(const MediaStreamTrackPrivate&, MediaSample&) final;
    40     void audioSamplesAvailable(const MediaStreamTrackPrivate&, const WTF::MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;
    41     void fetchData(CompletionHandler<void(RefPtr<SharedBuffer>&&, const String&)>&&) final;
     46    void fetchData(FetchDataCallback&&) final;
     47    void audioSamplesAvailable(const WTF::MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;
     48    void stopRecording() final;
     49
    4250    const String& mimeType();
    43    
    44     void generateMockString(const MediaStreamTrackPrivate&);
     51    void generateMockCounterString();
    4552
    4653    mutable Lock m_bufferLock;
    4754    StringBuilder m_buffer;
    4855    unsigned m_counter { 0 };
     56    String m_audioTrackID;
     57    String m_videoTrackID;
    4958};
    5059

  • trunk/Source/WebCore/platform/mediastream/AudioTrackPrivateMediaStream.cpp

    r258840 r259816  
    3636AudioTrackPrivateMediaStream::AudioTrackPrivateMediaStream(MediaStreamTrackPrivate& track)
    3737    : m_streamTrack(track)
     38    , m_audioSource(track.source())
    3839    , m_id(track.id())
    3940    , m_label(track.label())
    40     , m_timelineOffset(MediaTime::invalidTime())
    4141    , m_renderer { AudioMediaStreamTrackRenderer::create() }
    4242{
    … …  
    6363
    6464    m_isCleared = true;
    65     streamTrack().removeObserver(*this);
    66 
    67     m_renderer->clear();
    68 }
    69 
    70 void AudioTrackPrivateMediaStream::playInternal()
    71 {
    72     ASSERT(isMainThread());
    7365
    7466    if (m_isPlaying)
    75         return;
     67        m_audioSource->removeAudioSampleObserver(*this);
    7668
    77     m_isPlaying = true;
    78     m_autoPlay = false;
    79 
    80     m_renderer->start();
     69    streamTrack().removeObserver(*this);
     70    m_renderer->clear();
    8171}
    8272
    8373void AudioTrackPrivateMediaStream::play()
    8474{
    85     playInternal();
     75    m_shouldPlay = true;
     76    updateRenderer();
    8677}
    8778
    8879void AudioTrackPrivateMediaStream::pause()
    8980{
    90     ASSERT(isMainThread());
     81    m_shouldPlay = false;
     82    updateRenderer();
     83}
    9184
    92     if (!m_isPlaying)
    93         return;
    94 
    95     m_isPlaying = false;
    96     m_autoPlay = false;
    97 
    98     m_renderer->stop();
     85void AudioTrackPrivateMediaStream::setMuted(bool muted)
     86{
     87    m_muted = muted;
     88    updateRenderer();
    9989}
    10090
    … …  
    10292{
    10393    m_renderer->setVolume(volume);
     94    updateRenderer();
    10495}
    10596
    … …  
    110101
    111102// May get called on a background thread.
    112 void AudioTrackPrivateMediaStream::audioSamplesAvailable(MediaStreamTrackPrivate&, const MediaTime& sampleTime, const PlatformAudioData& audioData, const AudioStreamDescription& description, size_t sampleCount)
     103void AudioTrackPrivateMediaStream::audioSamplesAvailable(const MediaTime& sampleTime, const PlatformAudioData& audioData, const AudioStreamDescription& description, size_t sampleCount)
    113104{
    114     if (!m_isPlaying) {
    115         m_renderer->stop();
    116         return;
    117     }
    118 
    119105    m_renderer->pushSamples(sampleTime, audioData, description, sampleCount);
    120 
    121     if (m_autoPlay && !m_hasStartedAutoplay) {
    122         m_hasStartedAutoplay = true;
    123         callOnMainThread([this, protectedThis = makeRef(*this)] {
    124             if (m_autoPlay)
    125                 playInternal();
    126         });
    127     }
    128106}
    129107
    130108void AudioTrackPrivateMediaStream::trackMutedChanged(MediaStreamTrackPrivate&)
    131109{
    132     updateRendererMutedState();
     110    updateRenderer();
    133111}
    134112
    135113void AudioTrackPrivateMediaStream::trackEnabledChanged(MediaStreamTrackPrivate&)
    136114{
    137     updateRendererMutedState();
    138 }
    139 
    140 void AudioTrackPrivateMediaStream::updateRendererMutedState()
    141 {
    142     if (streamTrack().muted() || streamTrack().ended() || !streamTrack().enabled()) {
    143         pause();
    144         return;
    145     }
    146     play();
     115    updateRenderer();
    147116}
    148117
    149118void AudioTrackPrivateMediaStream::trackEnded(MediaStreamTrackPrivate&)
    150119{
    151     pause();
     120    updateRenderer();
     121}
     122
     123void AudioTrackPrivateMediaStream::updateRenderer()
     124{
     125    if (!m_shouldPlay || !volume() || m_muted || streamTrack().muted() || streamTrack().ended() || !streamTrack().enabled()) {
     126        stopRenderer();
     127        return;
     128    }
     129    startRenderer();
     130}
     131
     132void AudioTrackPrivateMediaStream::startRenderer()
     133{
     134    ASSERT(isMainThread());
     135    if (m_isPlaying)
     136        return;
     137
     138    m_isPlaying = true;
     139    m_audioSource->addAudioSampleObserver(*this);
     140    m_renderer->start();
     141}
     142
     143void AudioTrackPrivateMediaStream::stopRenderer()
     144{
     145    ASSERT(isMainThread());
     146    if (!m_isPlaying)
     147        return;
     148
     149    m_isPlaying = false;
     150    m_audioSource->removeAudioSampleObserver(*this);
     151    m_renderer->stop();
    152152}
    153153

  • trunk/Source/WebCore/platform/mediastream/AudioTrackPrivateMediaStream.h

    r254446 r259816  
    3535class AudioMediaStreamTrackRenderer;
    3636
    37 class AudioTrackPrivateMediaStream final : public AudioTrackPrivate, private MediaStreamTrackPrivate::Observer {
     37class AudioTrackPrivateMediaStream final
     38    : public AudioTrackPrivate
     39    , private MediaStreamTrackPrivate::Observer
     40    , private RealtimeMediaSource::AudioSampleObserver {
    3841    WTF_MAKE_NONCOPYABLE(AudioTrackPrivateMediaStream)
    3942public:
    … …  
    4851    MediaStreamTrackPrivate& streamTrack() { return m_streamTrack.get(); }
    4952
    50     MediaTime timelineOffset() const { return m_timelineOffset; }
    51     void setTimelineOffset(const MediaTime& offset) { m_timelineOffset = offset; }
    52 
    5353    void clear();
    5454
    … …  
    6060    float volume() const;
    6161
    62     void setMuted(bool muted) { m_muted = muted; }
     62    void setMuted(bool);
    6363    bool muted() const { return m_muted; }
    6464
    … …  
    7878
    7979    // MediaStreamTrackPrivate::Observer
    80     void audioSamplesAvailable(MediaStreamTrackPrivate&, const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;
    8180    void trackEnded(MediaStreamTrackPrivate&) final;
    8281    void trackMutedChanged(MediaStreamTrackPrivate&)  final;
    … …  
    8483    void trackSettingsChanged(MediaStreamTrackPrivate&) final { }
    8584
    86     void playInternal();
    87     void updateRendererMutedState();
     85    // RealtimeMediaSource::AudioSampleObserver
     86    void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;
     87
     88    void startRenderer();
     89    void stopRenderer();
     90    void updateRenderer();
    8891
    8992    // Main thread writable members
    9093    bool m_isPlaying { false };
    91     bool m_autoPlay { false };
     94    bool m_shouldPlay { false };
    9295    bool m_muted { false };
    9396    bool m_isCleared { false };
    9497
    9598    Ref<MediaStreamTrackPrivate> m_streamTrack;
     99    Ref<RealtimeMediaSource> m_audioSource;
    96100    AtomString m_id;
    97101    AtomString m_label;
    98102    int m_index { 0 };
    99     MediaTime m_timelineOffset;
    100103
    101104    // Audio thread members
    102105    std::unique_ptr<AudioMediaStreamTrackRenderer> m_renderer;
    103     bool m_hasStartedAutoplay { false };
    104106};
    105107

  • trunk/Source/WebCore/platform/mediastream/MediaStreamPrivate.cpp

    r258201 r259816  
    122122}
    123123
     124void MediaStreamPrivate::forEachTrack(const Function<void(MediaStreamTrackPrivate&)>& callback)
     125{
     126    for (auto& track : m_trackSet.values())
     127        callback(*track);
     128}
     129
    124130void MediaStreamPrivate::updateActiveState(NotifyClientOption notifyClientOption)
    125131{

  • trunk/Source/WebCore/platform/mediastream/MediaStreamPrivate.h

    r258201 r259816  
    8585    bool hasTracks() const { return !m_trackSet.isEmpty(); }
    8686    void forEachTrack(const Function<void(const MediaStreamTrackPrivate&)>&) const;
     87    void forEachTrack(const Function<void(MediaStreamTrackPrivate&)>&);
    8788    MediaStreamTrackPrivate* activeVideoTrack() { return m_activeVideoTrack; }
    8889

  • trunk/Source/WebCore/platform/mediastream/MediaStreamTrackPrivate.cpp

    r257039 r259816  
    7979}
    8080
    81 void MediaStreamTrackPrivate::forEachObserver(const WTF::Function<void(Observer&)>& apply) const
    82 {
    83     Vector<Observer*> observersCopy;
    84     {
    85         auto locker = holdLock(m_observersLock);
    86         observersCopy = copyToVector(m_observers);
    87     }
    88     for (auto* observer : observersCopy) {
    89         auto locker = holdLock(m_observersLock);
    90         // Make sure the observer has not been destroyed.
     81void MediaStreamTrackPrivate::forEachObserver(const Function<void(Observer&)>& apply) const
     82{
     83    ASSERT(isMainThread());
     84    for (auto* observer : copyToVector(m_observers)) {
    9185        if (!m_observers.contains(observer))
    9286            continue;
    … …  
    9791void MediaStreamTrackPrivate::addObserver(MediaStreamTrackPrivate::Observer& observer)
    9892{
    99     auto locker = holdLock(m_observersLock);
     93    ASSERT(isMainThread());
    10094    m_observers.add(&observer);
    10195}
    … …  
    10397void MediaStreamTrackPrivate::removeObserver(MediaStreamTrackPrivate::Observer& observer)
    10498{
    105     auto locker = holdLock(m_observersLock);
     99    ASSERT(isMainThread());
    106100    m_observers.remove(&observer);
    107101}
    … …  
    262256}
    263257
    264 // May get called on a background thread.
    265 void MediaStreamTrackPrivate::audioSamplesAvailable(const MediaTime& mediaTime, const PlatformAudioData& data, const AudioStreamDescription& description, size_t sampleCount)
    266 {
    267     if (!m_hasSentStartProducedData) {
    268         callOnMainThread([this, weakThis = makeWeakPtr(this)] {
    269             if (!weakThis)
    270                 return;
    271 
    272             if (!m_haveProducedData) {
    273                 m_haveProducedData = true;
    274                 updateReadyState();
    275             }
    276             m_hasSentStartProducedData = true;
    277         });
    278         return;
    279     }
    280 
    281     forEachObserver([&](auto& observer) {
    282         observer.audioSamplesAvailable(*this, mediaTime, data, description, sampleCount);
    283     });
     258void MediaStreamTrackPrivate::hasStartedProducingAudioData()
     259{
     260    if (m_haveProducedData)
     261        return;
     262    m_haveProducedData = true;
     263    updateReadyState();
    284264}
    285265

  • trunk/Source/WebCore/platform/mediastream/MediaStreamTrackPrivate.h

    r257039 r259816  
    6363        virtual void sampleBufferUpdated(MediaStreamTrackPrivate&, MediaSample&) { };
    6464        virtual void readyStateChanged(MediaStreamTrackPrivate&) { };
    65 
    66         // May get called on a background thread.
    67         virtual void audioSamplesAvailable(MediaStreamTrackPrivate&, const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) { };
    6865    };
    6966
    … …  
    10198
    10299    RealtimeMediaSource& source() { return m_source.get(); }
     100    const RealtimeMediaSource& source() const { return m_source.get(); }
    103101    WEBCORE_EXPORT RealtimeMediaSource::Type type() const;
    104102
    … …  
    133131    MediaStreamTrackPrivate(Ref<const Logger>&&, Ref<RealtimeMediaSource>&&, String&& id);
    134132
    135     // RealtimeMediaSourceObserver
     133    // RealtimeMediaSource::Observer
    136134    void sourceStarted() final;
    137135    void sourceStopped() final;
    … …  
    140138    bool preventSourceFromStopping() final;
    141139    void videoSampleAvailable(MediaSample&) final;
    142     void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;
    143140    void audioUnitWillStart() final;
     141    void hasStartedProducingAudioData() final;
    144142
    145143    void updateReadyState();

  • trunk/Source/WebCore/platform/mediastream/RealtimeMediaSource.cpp

    r259698 r259816  
    6161}
    6262
     63void RealtimeMediaSource::addAudioSampleObserver(RealtimeMediaSource::AudioSampleObserver& observer)
     64{
     65    ASSERT(isMainThread());
     66    auto locker = holdLock(m_audioSampleObserversLock);
     67    m_audioSampleObservers.add(&observer);
     68}
     69
     70void RealtimeMediaSource::removeAudioSampleObserver(RealtimeMediaSource::AudioSampleObserver& observer)
     71{
     72    ASSERT(isMainThread());
     73    auto locker = holdLock(m_audioSampleObserversLock);
     74    m_audioSampleObservers.remove(&observer);
     75}
     76
    6377void RealtimeMediaSource::addObserver(RealtimeMediaSource::Observer& observer)
    6478{
    65     auto locker = holdLock(m_observersLock);
     79    ASSERT(isMainThread());
    6680    m_observers.add(&observer);
    6781}
    … …  
    6983void RealtimeMediaSource::removeObserver(RealtimeMediaSource::Observer& observer)
    7084{
    71     auto locker = holdLock(m_observersLock);
     85    ASSERT(isMainThread());
    7286    m_observers.remove(&observer);
    7387    if (m_observers.isEmpty())
    … …  
    117131}
    118132
    119 void RealtimeMediaSource::forEachObserver(const WTF::Function<void(Observer&)>& apply) const
    120 {
    121     Vector<Observer*> observersCopy;
    122     {
    123         auto locker = holdLock(m_observersLock);
    124         observersCopy = copyToVector(m_observers);
    125     }
    126     for (auto* observer : observersCopy) {
    127         auto locker = holdLock(m_observersLock);
    128         // Make sure the observer has not been destroyed.
     133void RealtimeMediaSource::forEachObserver(const Function<void(Observer&)>& apply) const
     134{
     135    ASSERT(isMainThread());
     136    for (auto* observer : copyToVector(m_observers)) {
    129137        if (!m_observers.contains(observer))
    130138            continue;
    … …  
    183191void RealtimeMediaSource::audioSamplesAvailable(const MediaTime& time, const PlatformAudioData& audioData, const AudioStreamDescription& description, size_t numberOfFrames)
    184192{
    185     forEachObserver([&](auto& observer) {
    186         observer.audioSamplesAvailable(time, audioData, description, numberOfFrames);
    187     });
     193    if (!m_hasSentStartProducedAudioData) {
     194        callOnMainThread([this, weakThis = makeWeakPtr(this)] {
     195            if (!weakThis)
     196                return;
     197            if (m_hasSentStartProducedAudioData)
     198                return;
     199            m_hasSentStartProducedAudioData = true;
     200            forEachObserver([&](auto& observer) {
     201                observer.hasStartedProducingAudioData();
     202            });
     203        });
     204    }
     205
     206    auto locker = holdLock(m_audioSampleObserversLock);
     207    for (auto* observer : m_audioSampleObservers)
     208        observer->audioSamplesAvailable(time, audioData, description, numberOfFrames);
    188209}
    189210

  • trunk/Source/WebCore/platform/mediastream/RealtimeMediaSource.h

    r258504 r259816  
    7070class WEBCORE_EXPORT RealtimeMediaSource
    7171    : public ThreadSafeRefCounted<RealtimeMediaSource, WTF::DestructionThread::MainRunLoop>
    72     , public CanMakeWeakPtr<RealtimeMediaSource>
     72    , public CanMakeWeakPtr<RealtimeMediaSource, WeakPtrFactoryInitialization::Eager>
    7373#if !RELEASE_LOG_DISABLED
    7474    , protected LoggerHelper
    … …  
    9393        virtual void videoSampleAvailable(MediaSample&) { }
    9494
     95        virtual void hasStartedProducingAudioData() { }
     96    };
     97    class AudioSampleObserver {
     98    public:
     99        virtual ~AudioSampleObserver() = default;
     100
    95101        // May be called on a background thread.
    96         virtual void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t /*numberOfFrames*/) { }
     102        virtual void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t /*numberOfFrames*/) = 0;
    97103    };
    98104
    … …  
    131137    WEBCORE_EXPORT void addObserver(Observer&);
    132138    WEBCORE_EXPORT void removeObserver(Observer&);
     139
     140    WEBCORE_EXPORT void addAudioSampleObserver(AudioSampleObserver&);
     141    WEBCORE_EXPORT void removeAudioSampleObserver(AudioSampleObserver&);
    133142
    134143    const IntSize size() const;
    … …  
    254263    Type m_type;
    255264    String m_name;
    256     mutable RecursiveLock m_observersLock;
    257265    HashSet<Observer*> m_observers;
     266
     267    mutable RecursiveLock m_audioSampleObserversLock;
     268    HashSet<AudioSampleObserver*> m_audioSampleObservers;
     269
    258270    IntSize m_size;
    259271    IntSize m_intrinsicSize;
    … …  
    273285    bool m_captureDidFailed { false };
    274286    bool m_isEnded { false };
     287    bool m_hasSentStartProducedAudioData { false };
    275288};
    276289

  • trunk/Source/WebCore/platform/mediastream/RealtimeOutgoingAudioSource.cpp

    r254087 r259816  
    4646RealtimeOutgoingAudioSource::~RealtimeOutgoingAudioSource()
    4747{
    48 ASSERT(!m_audioSource->hasObserver(*this));
     48    ASSERT(!m_audioSource->hasObserver(*this));
    4949#if ASSERT_ENABLED
    5050    auto locker = holdLock(m_sinksLock);
    … …  
    5959    ASSERT(!m_audioSource->hasObserver(*this));
    6060    m_audioSource->addObserver(*this);
     61    m_audioSource->source().addAudioSampleObserver(*this);
    6162    initializeConverter();
    6263}
    … …  
    6465void RealtimeOutgoingAudioSource::unobserveSource()
    6566{
     67    m_audioSource->source().removeAudioSampleObserver(*this);
    6668    m_audioSource->removeObserver(*this);
    6769}

  • trunk/Source/WebCore/platform/mediastream/RealtimeOutgoingAudioSource.h

    r252987 r259816  
    5656    , public webrtc::AudioSourceInterface
    5757    , private MediaStreamTrackPrivate::Observer
     58    , private RealtimeMediaSource::AudioSampleObserver
    5859#if !RELEASE_LOG_DISABLED
    5960    , private LoggerHelper
    … …  
    109110    void sourceMutedChanged();
    110111    void sourceEnabledChanged();
    111     virtual void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) { };
    112112
    113113    virtual bool isReachingBufferedAudioDataHighLimit() { return false; };
    … …  
    119119    void trackMutedChanged(MediaStreamTrackPrivate&) final { sourceMutedChanged(); }
    120120    void trackEnabledChanged(MediaStreamTrackPrivate&) final { sourceEnabledChanged(); }
    121     void audioSamplesAvailable(MediaStreamTrackPrivate&, const MediaTime& mediaTime, const PlatformAudioData& data, const AudioStreamDescription& description, size_t sampleCount) { audioSamplesAvailable(mediaTime, data, description, sampleCount); }
    122121    void trackEnded(MediaStreamTrackPrivate&) final { }
    123122    void trackSettingsChanged(MediaStreamTrackPrivate&) final { }

  • trunk/Source/WebCore/platform/mediastream/WebAudioSourceProvider.h

    r218622 r259816  
    3333namespace WebCore {
    3434
    35 class WebAudioSourceProvider : public ThreadSafeRefCounted<WebAudioSourceProvider>, public AudioSourceProvider {
     35class WebAudioSourceProvider : public ThreadSafeRefCounted<WebAudioSourceProvider, WTF::DestructionThread::Main>, public AudioSourceProvider {
    3636};
    3737

  • trunk/Source/WebCore/platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp

    r254116 r259816  
    107107
    108108class WebKitMediaStreamTrackObserver
    109     : public MediaStreamTrackPrivate::Observer {
     109    : public MediaStreamTrackPrivate::Observer
     110    , public RealtimeMediaSource::AudioSampleObserver {
    110111    WTF_MAKE_FAST_ALLOCATED;
    111112public:
    … …  
    132133    }
    133134
    134     void audioSamplesAvailable(MediaStreamTrackPrivate&, const MediaTime&, const PlatformAudioData& audioData, const AudioStreamDescription&, size_t) final
     135    void audioSamplesAvailable(const MediaTime&, const PlatformAudioData& audioData, const AudioStreamDescription&, size_t) final
    135136    {
    136137        auto audiodata = static_cast<const GStreamerAudioData&>(audioData);
    … …  
    376377    GST_OBJECT_LOCK(self);
    377378    if (self->stream) {
    378         for (auto& track : self->stream->tracks())
     379        for (auto& track : self->stream->tracks()) {
     380            track->source().removeAudioSampleObserver(*self->mediaStreamTrackObserver.get());
    379381            track->removeObserver(*self->mediaStreamTrackObserver.get());
    380 
     382        }
    381383        self->stream->removeObserver(*self->mediaStreamObserver);
    382384        self->stream = nullptr;
    … …  
    397399        GST_OBJECT_LOCK(self);
    398400        if (self->stream) {
    399             for (auto& track : self->stream->tracks())
     401            for (auto& track : self->stream->tracks()) {
     402                track->source().removeAudioSampleObserver(*self->mediaStreamTrackObserver.get());
    400403                track->removeObserver(*self->mediaStreamTrackObserver.get());
    401         } else if (self->track)
     404            }
     405        } else if (self->track) {
     406            self->track->source().removeAudioSampleObserver(*self->mediaStreamTrackObserver.get());
    402407            self->track->removeObserver(*self->mediaStreamTrackObserver.get());
     408        }
    403409        GST_OBJECT_UNLOCK(self);
    404410    }
    … …  
    544550        webkitMediaStreamSrcAddPad(self, pad.get(), pad_template);
    545551
    546     if (observe_track)
     552    if (observe_track) {
    547553        track->addObserver(*self->mediaStreamTrackObserver.get());
    548 
     554        track->source().addAudioSampleObserver(*self->mediaStreamTrackObserver.get());
     555    }
    549556    gst_element_sync_state_with_parent(element);
    550557    return TRUE;

  • trunk/Source/WebCore/platform/mediastream/mac/WebAudioSourceProviderAVFObjC.h

    r239427 r259816  
    4747class CAAudioStreamDescription;
    4848
    49 class WEBCORE_EXPORT WebAudioSourceProviderAVFObjC final : public WebAudioSourceProvider, MediaStreamTrackPrivate::Observer {
     49class WEBCORE_EXPORT WebAudioSourceProviderAVFObjC final
     50    : public WebAudioSourceProvider
     51    , MediaStreamTrackPrivate::Observer
     52    , RealtimeMediaSource::AudioSampleObserver {
    5053public:
    5154    static Ref<WebAudioSourceProviderAVFObjC> create(MediaStreamTrackPrivate&);
    … …  
    6366
    6467    // MediaStreamTrackPrivate::Observer
    65     void audioSamplesAvailable(MediaStreamTrackPrivate&, const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;
    6668    void trackEnded(MediaStreamTrackPrivate&) final { }
    6769    void trackMutedChanged(MediaStreamTrackPrivate&) final { }
    6870    void trackSettingsChanged(MediaStreamTrackPrivate&) final { }
    69     void trackEnabledChanged(MediaStreamTrackPrivate&) final { }
     71    void trackEnabledChanged(MediaStreamTrackPrivate&) final;
     72
     73    // RealtimeMediaSource::AudioSampleObserver
     74    void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;
    7075
    7176    size_t m_listBufferSize { 0 };
    … …  
    7883    AudioSourceProviderClient* m_client { nullptr };
    7984    MediaStreamTrackPrivate* m_captureSource { nullptr };
     85    Ref<RealtimeMediaSource> m_source;
    8086    Lock m_mutex;
    8187    bool m_connected { false };
     88    bool m_enabled { true };
    8289};
    8390

  • trunk/Source/WebCore/platform/mediastream/mac/WebAudioSourceProviderAVFObjC.mm

    r257688 r259816  
    5555WebAudioSourceProviderAVFObjC::WebAudioSourceProviderAVFObjC(MediaStreamTrackPrivate& source)
    5656    : m_captureSource(&source)
     57    , m_source(source.source())
     58    , m_enabled(source.enabled())
    5759{
     60    m_source->addAudioSampleObserver(*this);
    5861}
    5962
    6063WebAudioSourceProviderAVFObjC::~WebAudioSourceProviderAVFObjC()
    6164{
     65    m_source->removeAudioSampleObserver(*this);
     66
    6267    auto locker = holdLock(m_mutex);
    6368
    … …  
    166171}
    167172
     173void WebAudioSourceProviderAVFObjC::trackEnabledChanged(MediaStreamTrackPrivate& track)
     174{
     175    m_enabled = track.enabled();
     176}
     177
    168178// May get called on a background thread.
    169 void WebAudioSourceProviderAVFObjC::audioSamplesAvailable(MediaStreamTrackPrivate& track, const MediaTime&, const PlatformAudioData& data, const AudioStreamDescription& description, size_t frameCount)
     179void WebAudioSourceProviderAVFObjC::audioSamplesAvailable(const MediaTime&, const PlatformAudioData& data, const AudioStreamDescription& description, size_t frameCount)
    170180{
    171     if (!track.enabled())
     181    if (!m_enabled || !m_connected)
    172182        return;
    173183

  • trunk/Source/WebCore/testing/Internals.cpp

    r259752 r259816  
    467467{
    468468#if ENABLE(MEDIA_STREAM)
    469     if (m_track)
    470         m_track->source().removeObserver(*this);
     469    if (m_trackSource) {
     470        m_trackSource->removeObserver(*this);
     471        m_trackSource->removeAudioSampleObserver(*this);
     472    }
    471473#endif
    472474}
    … …  
    16401642}
    16411643
    1642 static std::unique_ptr<MediaRecorderPrivate> createRecorderMockSource()
    1643 {
    1644     return std::unique_ptr<MediaRecorderPrivateMock>(new MediaRecorderPrivateMock);
     1644static std::unique_ptr<MediaRecorderPrivate> createRecorderMockSource(MediaStreamPrivate& stream)
     1645{
     1646    return std::unique_ptr<MediaRecorderPrivateMock>(new MediaRecorderPrivateMock(stream));
    16451647}
    16461648
    … …  
    49264928void Internals::observeMediaStreamTrack(MediaStreamTrack& track)
    49274929{
    4928     m_track = &track;
    4929     m_track->source().addObserver(*this);
     4930    m_trackSource = &track.source();
     4931    m_trackSource->addObserver(*this);
     4932    m_trackSource->addAudioSampleObserver(*this);
    49304933}
    49314934
    … …  
    49414944        return;
    49424945
    4943     auto& videoSettings = m_track->source().settings();
     4946    auto& videoSettings = m_trackSource->settings();
    49444947    if (!videoSettings.width() || !videoSettings.height())
    49454948        return;

  • trunk/Source/WebCore/testing/Internals.h

    r259575 r259816  
    124124#if ENABLE(MEDIA_STREAM)
    125125    , private RealtimeMediaSource::Observer
     126    , private RealtimeMediaSource::AudioSampleObserver
    126127#endif
    127128    {
    … …  
    970971    ExceptionOr<RenderedDocumentMarker*> markerAt(Node&, const String& markerType, unsigned index);
    971972
     973#if ENABLE(MEDIA_STREAM)
    972974    // RealtimeMediaSource::Observer API
    973 #if ENABLE(MEDIA_STREAM)
    974975    void videoSampleAvailable(MediaSample&) final;
     976    // RealtimeMediaSource::AudioSampleObserver API
    975977    void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final { m_trackAudioSampleCount++; }
    976978
    … …  
    978980    unsigned long m_trackVideoSampleCount { 0 };
    979981    unsigned long m_trackAudioSampleCount { 0 };
    980     RefPtr<MediaStreamTrack> m_track;
     982    RefPtr<RealtimeMediaSource> m_trackSource;
    981983    std::unique_ptr<TrackFramePromise> m_nextTrackFramePromise;
    982984#endif

  • trunk/Source/WebKit/ChangeLog

    r259814 r259816  
     12020-04-09  Youenn Fablet  <youenn@apple.com>
     2
     3        Introduce a RealtimeMediaSource observer dedicated to receiving audio samples
     4        https://bugs.webkit.org/show_bug.cgi?id=210180
     5
     6        Reviewed by Eric Carlson.
     7
     8        Update SourceProxy to be an AudioSampleObserver.
     9        Update MediaRecorder to only listen for audio samples if GPU remote writer is successfully created.
     10        It also stops listening to audio samples as soon as stopping the recording.
     11
     12        * UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp:
     13        (WebKit::UserMediaCaptureManagerProxy::SourceProxy::SourceProxy):
     14        (WebKit::UserMediaCaptureManagerProxy::SourceProxy::~SourceProxy):
     15        * WebProcess/GPU/webrtc/MediaRecorderPrivate.cpp:
     16        (WebKit::MediaRecorderPrivate::MediaRecorderPrivate):
     17        (WebKit::MediaRecorderPrivate::~MediaRecorderPrivate):
     18        (WebKit::MediaRecorderPrivate::audioSamplesAvailable):
     19        (WebKit::MediaRecorderPrivate::stopRecording):
     20        * WebProcess/GPU/webrtc/MediaRecorderPrivate.h:
     21        * WebProcess/GPU/webrtc/MediaRecorderProvider.cpp:
     22        (WebKit::MediaRecorderProvider::createMediaRecorderPrivate):
     23        * WebProcess/GPU/webrtc/MediaRecorderProvider.h:
     24
    1252020-04-09  David Kilzer  <ddkilzer@apple.com>
    226

  • trunk/Source/WebKit/UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp

    r259775 r259816  
    5151class UserMediaCaptureManagerProxy::SourceProxy
    5252    : public RealtimeMediaSource::Observer
     53    , public RealtimeMediaSource::AudioSampleObserver
    5354    , public SharedRingBufferStorage::Client {
    5455    WTF_MAKE_FAST_ALLOCATED;
    … …  
    6162    {
    6263        m_source->addObserver(*this);
     64        m_source->addAudioSampleObserver(*this);
    6365    }
    6466
    … …  
    6668    {
    6769        storage().invalidate();
     70        m_source->removeAudioSampleObserver(*this);
    6871        m_source->removeObserver(*this);
    6972    }

  • trunk/Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderPrivate.cpp

    r257606 r259816  
    4444using namespace WebCore;
    4545
    46 MediaRecorderPrivate::MediaRecorderPrivate(const MediaStreamPrivate& stream)
     46MediaRecorderPrivate::MediaRecorderPrivate(MediaStreamPrivate& stream)
    4747    : m_identifier(MediaRecorderIdentifier::generate())
    4848    , m_connection(WebProcess::singleton().ensureGPUProcessConnection().connection())
    … …  
    6565    }
    6666
    67     m_connection->sendWithAsyncReply(Messages::RemoteMediaRecorderManager::CreateRecorder { m_identifier, !!selectedTracks.audioTrack, width, height }, [this, weakThis = makeWeakPtr(this)](auto&& exception) {
    68         if (!weakThis || !exception)
     67    m_connection->sendWithAsyncReply(Messages::RemoteMediaRecorderManager::CreateRecorder { m_identifier, !!selectedTracks.audioTrack, width, height }, [this, weakThis = makeWeakPtr(this), audioTrack = makeRefPtr(selectedTracks.audioTrack)](auto&& exception) {
     68        if (!weakThis)
    6969            return;
    70         m_errorCallback(Exception { exception->code, WTFMove(exception->message) });
     70        if (exception)
     71            return m_errorCallback(Exception { exception->code, WTFMove(exception->message) });
     72        if (audioTrack)
     73            setAudioSource(&audioTrack->source());
    7174    }, 0);
    7275}
    … …  
    7477MediaRecorderPrivate::~MediaRecorderPrivate()
    7578{
     79    setAudioSource(nullptr);
    7680    m_connection->send(Messages::RemoteMediaRecorderManager::ReleaseRecorder { m_identifier }, 0);
    7781}
    … …  
    8993}
    9094
    91 void MediaRecorderPrivate::audioSamplesAvailable(const WebCore::MediaStreamTrackPrivate& track, const MediaTime& time, const PlatformAudioData& audioData, const AudioStreamDescription& description, size_t numberOfFrames)
     95void MediaRecorderPrivate::audioSamplesAvailable(const MediaTime& time, const PlatformAudioData& audioData, const AudioStreamDescription& description, size_t numberOfFrames)
    9296{
    93     if (track.id() != m_recordedAudioTrackID)
    94         return;
    95 
    9697    if (m_description != description) {
    9798        ASSERT(description.platformDescription().type == PlatformDescription::CAAudioStreamBasicType);
    … …  
    131132void MediaRecorderPrivate::stopRecording()
    132133{
     134    setAudioSource(nullptr);
    133135    m_connection->send(Messages::RemoteMediaRecorder::StopRecording { }, m_identifier);
    134136}

  • trunk/Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderPrivate.h

    r256493 r259816  
    4545namespace WebKit {
    4646
    47 class MediaRecorderPrivate final : public WebCore::MediaRecorderPrivate, public SharedRingBufferStorage::Client, public CanMakeWeakPtr<MediaRecorderPrivate> {
     47class MediaRecorderPrivate final
     48    : public WebCore::MediaRecorderPrivate
     49    , public SharedRingBufferStorage::Client
     50    , public CanMakeWeakPtr<MediaRecorderPrivate> {
    4851    WTF_MAKE_FAST_ALLOCATED;
    4952public:
    50     explicit MediaRecorderPrivate(const WebCore::MediaStreamPrivate&);
     53    explicit MediaRecorderPrivate(WebCore::MediaStreamPrivate&);
    5154    ~MediaRecorderPrivate();
    5255
    … …  
    5457    // WebCore::MediaRecorderPrivate
    5558    void sampleBufferUpdated(const WebCore::MediaStreamTrackPrivate&, WebCore::MediaSample&) final;
    56     void audioSamplesAvailable(const WebCore::MediaStreamTrackPrivate&, const WTF::MediaTime&, const WebCore::PlatformAudioData&, const WebCore::AudioStreamDescription&, size_t) final;
    5759    void fetchData(CompletionHandler<void(RefPtr<WebCore::SharedBuffer>&&, const String& mimeType)>&&) final;
    5860    void stopRecording() final;
     61    void audioSamplesAvailable(const WTF::MediaTime&, const WebCore::PlatformAudioData&, const WebCore::AudioStreamDescription&, size_t) final;
    5962
    6063    // SharedRingBufferStorage::Client

  • trunk/Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderProvider.cpp

    r257542 r259816  
    3535using namespace WebCore;
    3636
    37 std::unique_ptr<WebCore::MediaRecorderPrivate> MediaRecorderProvider::createMediaRecorderPrivate(const MediaStreamPrivate& stream)
     37std::unique_ptr<WebCore::MediaRecorderPrivate> MediaRecorderProvider::createMediaRecorderPrivate(MediaStreamPrivate& stream)
    3838{
    3939#if ENABLE(GPU_PROCESS)

  • trunk/Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderProvider.h

    r254194 r259816  
    3636private:
    3737#if ENABLE(MEDIA_STREAM) && PLATFORM(COCOA)
    38     std::unique_ptr<WebCore::MediaRecorderPrivate> createMediaRecorderPrivate(const WebCore::MediaStreamPrivate&) final;
     38    std::unique_ptr<WebCore::MediaRecorderPrivate> createMediaRecorderPrivate(WebCore::MediaStreamPrivate&) final;
    3939#endif
    4040};
Note: See TracChangeset for help on using the changeset viewer.