Changeset 261553 in webkit


Ignore:
Timestamp:
May 12, 2020 4:45:25 AM (4 years ago)
Author:
youenn@apple.com
Message:

Introduce a RealtimeMediaSource video sample observer
​https://bugs.webkit.org/show_bug.cgi?id=211718

Reviewed by Eric Carlson.

LayoutTests/imported/w3c:

  • web-platform-tests/mediacapture-streams/MediaStreamTrack-MediaElement-disabled-video-is-black.https-expected.txt:

Drive-by fix, since we are now receiving samples but not rendering them when track is not enabled, the loadeddata event is fired appropriately.
Update MediaPlayerPrivateMediaStreamAVFObjC::currentDisplayMode to return PaintItBlack in that case.

Source/WebCore:

We introduce an observer dedicated to video samples similarly to AudioSampleObserver for audio.
This will allow to move video frame processing out of the main thread progressively.
For now, we remove video sample observing from the track private and observers should now register directly to the realtime media source.
We update the various users, including MediaRecorder and the media player.
In both cases, they will only observe the video track to be played/recorded, which is both more efficient and simpler.

We introduced RealtimeMediaSource::Observer::hasStartedProducingData callback for MediaStreamTrackPrivate so
that the processing to do when the first samples are available can be done on the main thread.

MediaStreamTrackPrivate no longer filters out samples if it is not enabled.
As such, each consumer is now responsible to observe/unobserve the source when its track gets enabled/disabled similarly as for audio,
or do nothing if track is not enabled.

Similarly, RealtimeOutgoingVideoSourceCocoa will now only observe video samples when the track is enabled and not muted.

Covered by existing tests.

  • Modules/mediarecorder/MediaRecorder.cpp:

(WebCore::MediaRecorder::sampleBufferUpdated): Deleted.

  • Modules/mediarecorder/MediaRecorder.h:
  • platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
  • platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:

(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::~MediaPlayerPrivateMediaStreamAVFObjC):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::enqueueVideoSample):
We can renove the track check since we only observe the active video track.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::videoSampleAvailable):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::checkSelectedVideoTrack):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::sampleBufferUpdated): Deleted.

  • platform/mediarecorder/MediaRecorderPrivate.h:

(WebCore::MediaRecorderPrivate::setVideoSource):
(WebCore::MediaRecorderPrivate::~MediaRecorderPrivate):

  • platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp:

(WebCore::MediaRecorderPrivateAVFImpl::create):
(WebCore::MediaRecorderPrivateAVFImpl::~MediaRecorderPrivateAVFImpl):
(WebCore::MediaRecorderPrivateAVFImpl::videoSampleAvailable):
(WebCore::MediaRecorderPrivateAVFImpl::stopRecording):
(WebCore::MediaRecorderPrivateAVFImpl::sampleBufferUpdated): Deleted.

  • platform/mediarecorder/MediaRecorderPrivateAVFImpl.h:
  • platform/mediarecorder/MediaRecorderPrivateMock.cpp:

(WebCore::MediaRecorderPrivateMock::MediaRecorderPrivateMock):
(WebCore::MediaRecorderPrivateMock::~MediaRecorderPrivateMock):
(WebCore::MediaRecorderPrivateMock::stopRecording):
(WebCore::MediaRecorderPrivateMock::videoSampleAvailable):
(WebCore::MediaRecorderPrivateMock::sampleBufferUpdated): Deleted.

  • platform/mediarecorder/MediaRecorderPrivateMock.h:
  • platform/mediastream/MediaStreamTrackPrivate.cpp:

(WebCore::MediaStreamTrackPrivate::hasStartedProducingData):
(WebCore::MediaStreamTrackPrivate::updateReadyState):
(WebCore::MediaStreamTrackPrivate::videoSampleAvailable): Deleted.
(WebCore::MediaStreamTrackPrivate::hasStartedProducingAudioData): Deleted.

  • platform/mediastream/MediaStreamTrackPrivate.h:
  • platform/mediastream/RealtimeMediaSource.cpp:

(WebCore::RealtimeMediaSource::addVideoSampleObserver):
(WebCore::RealtimeMediaSource::removeVideoSampleObserver):
(WebCore::RealtimeMediaSource::updateHasStartedProducingData):
(WebCore::RealtimeMediaSource::videoSampleAvailable):
(WebCore::RealtimeMediaSource::audioSamplesAvailable):

  • platform/mediastream/RealtimeMediaSource.h:
  • platform/mediastream/RealtimeOutgoingVideoSource.cpp:

(WebCore::RealtimeOutgoingVideoSource::unobserveSource):
(WebCore::RealtimeOutgoingVideoSource::updateBlackFramesSending):

  • platform/mediastream/RealtimeOutgoingVideoSource.h:
  • platform/mediastream/RealtimeVideoSource.cpp:

(WebCore::RealtimeVideoSource::~RealtimeVideoSource):
(WebCore::RealtimeVideoSource::startProducingData):
(WebCore::RealtimeVideoSource::stopProducingData):
(WebCore::RealtimeVideoSource::videoSampleAvailable):

  • platform/mediastream/RealtimeVideoSource.h:
  • platform/mediastream/mac/AVVideoCaptureSource.mm:

(WebCore::AVVideoCaptureSource::captureOutputDidOutputSampleBufferFromConnection):

  • platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.h:
  • platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.mm:

(WebCore::RealtimeIncomingVideoSourceCocoa::processNewSample):

  • platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.cpp:

(WebCore::RealtimeOutgoingVideoSourceCocoa::videoSampleAvailable):
(WebCore::RealtimeOutgoingVideoSourceCocoa::sampleBufferUpdated): Deleted.

  • platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.h:
  • platform/mock/MockRealtimeVideoSource.cpp:
  • testing/Internals.cpp:

(WebCore::Internals::~Internals):
(WebCore::Internals::stopObservingRealtimeMediaSource):
(WebCore::Internals::observeMediaStreamTrack):

  • testing/Internals.h:

Source/WebKit:

  • UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp:

(WebKit::UserMediaCaptureManagerProxy::SourceProxy::SourceProxy):
(WebKit::UserMediaCaptureManagerProxy::SourceProxy::~SourceProxy):

  • WebProcess/GPU/webrtc/MediaRecorderPrivate.cpp:

(WebKit::MediaRecorderPrivate::MediaRecorderPrivate):
(WebKit::MediaRecorderPrivate::~MediaRecorderPrivate):
(WebKit::MediaRecorderPrivate::videoSampleAvailable):
(WebKit::MediaRecorderPrivate::stopRecording):
(WebKit::MediaRecorderPrivate::sampleBufferUpdated): Deleted.

  • WebProcess/GPU/webrtc/MediaRecorderPrivate.h:
Location:
trunk
Files:
33 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/imported/w3c/ChangeLog

    r261488 r261553  
     12020-05-12  Youenn Fablet  <youenn@apple.com>
     2
     3        Introduce a RealtimeMediaSource video sample observer
     4        https://bugs.webkit.org/show_bug.cgi?id=211718
     5
     6        Reviewed by Eric Carlson.
     7
     8        * web-platform-tests/mediacapture-streams/MediaStreamTrack-MediaElement-disabled-video-is-black.https-expected.txt:
     9        Drive-by fix, since we are now receiving samples but not rendering them when track is not enabled, the loadeddata event is fired appropriately.
     10        Update MediaPlayerPrivateMediaStreamAVFObjC::currentDisplayMode to return PaintItBlack in that case.
     11
    1122020-05-11  Antoine Quint  <graouts@apple.com>
    213

  • trunk/LayoutTests/imported/w3c/web-platform-tests/mediacapture-streams/MediaStreamTrack-MediaElement-disabled-video-is-black.https-expected.txt

    r252335 r261553  
    77
    88
    9 Harness Error (TIMEOUT), message = null
     9PASS Tests that a disabled video track in a MediaStream is rendered as blackness
    1010
    11 TIMEOUT Tests that a disabled video track in a MediaStream is rendered as blackness Test timed out
    12 

  • trunk/Source/WebCore/ChangeLog

    r261550 r261553  
     12020-05-12  Youenn Fablet  <youenn@apple.com>
     2
     3        Introduce a RealtimeMediaSource video sample observer
     4        https://bugs.webkit.org/show_bug.cgi?id=211718
     5
     6        Reviewed by Eric Carlson.
     7
     8        We introduce an observer dedicated to video samples similarly to AudioSampleObserver for audio.
     9        This will allow to move video frame processing out of the main thread progressively.
     10        For now, we remove video sample observing from the track private and observers should now register directly to the realtime media source.
     11        We update the various users, including MediaRecorder and the media player.
     12        In both cases, they will only observe the video track to be played/recorded, which is both more efficient and simpler.
     13
     14        We introduced RealtimeMediaSource::Observer::hasStartedProducingData callback for MediaStreamTrackPrivate so
     15        that the processing to do when the first samples are available can be done on the main thread.
     16
     17        MediaStreamTrackPrivate no longer filters out samples if it is not enabled.
     18        As such, each consumer is now responsible to observe/unobserve the source when its track gets enabled/disabled similarly as for audio,
     19        or do nothing if track is not enabled.
     20
     21        Similarly, RealtimeOutgoingVideoSourceCocoa will now only observe video samples when the track is enabled and not muted.
     22
     23        Covered by existing tests.
     24
     25        * Modules/mediarecorder/MediaRecorder.cpp:
     26        (WebCore::MediaRecorder::sampleBufferUpdated): Deleted.
     27        * Modules/mediarecorder/MediaRecorder.h:
     28        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
     29        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
     30        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::~MediaPlayerPrivateMediaStreamAVFObjC):
     31        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::enqueueVideoSample):
     32        We can renove the track check since we only observe the active video track.
     33        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::videoSampleAvailable):
     34        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::checkSelectedVideoTrack):
     35        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::sampleBufferUpdated): Deleted.
     36        * platform/mediarecorder/MediaRecorderPrivate.h:
     37        (WebCore::MediaRecorderPrivate::setVideoSource):
     38        (WebCore::MediaRecorderPrivate::~MediaRecorderPrivate):
     39        * platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp:
     40        (WebCore::MediaRecorderPrivateAVFImpl::create):
     41        (WebCore::MediaRecorderPrivateAVFImpl::~MediaRecorderPrivateAVFImpl):
     42        (WebCore::MediaRecorderPrivateAVFImpl::videoSampleAvailable):
     43        (WebCore::MediaRecorderPrivateAVFImpl::stopRecording):
     44        (WebCore::MediaRecorderPrivateAVFImpl::sampleBufferUpdated): Deleted.
     45        * platform/mediarecorder/MediaRecorderPrivateAVFImpl.h:
     46        * platform/mediarecorder/MediaRecorderPrivateMock.cpp:
     47        (WebCore::MediaRecorderPrivateMock::MediaRecorderPrivateMock):
     48        (WebCore::MediaRecorderPrivateMock::~MediaRecorderPrivateMock):
     49        (WebCore::MediaRecorderPrivateMock::stopRecording):
     50        (WebCore::MediaRecorderPrivateMock::videoSampleAvailable):
     51        (WebCore::MediaRecorderPrivateMock::sampleBufferUpdated): Deleted.
     52        * platform/mediarecorder/MediaRecorderPrivateMock.h:
     53        * platform/mediastream/MediaStreamTrackPrivate.cpp:
     54        (WebCore::MediaStreamTrackPrivate::hasStartedProducingData):
     55        (WebCore::MediaStreamTrackPrivate::updateReadyState):
     56        (WebCore::MediaStreamTrackPrivate::videoSampleAvailable): Deleted.
     57        (WebCore::MediaStreamTrackPrivate::hasStartedProducingAudioData): Deleted.
     58        * platform/mediastream/MediaStreamTrackPrivate.h:
     59        * platform/mediastream/RealtimeMediaSource.cpp:
     60        (WebCore::RealtimeMediaSource::addVideoSampleObserver):
     61        (WebCore::RealtimeMediaSource::removeVideoSampleObserver):
     62        (WebCore::RealtimeMediaSource::updateHasStartedProducingData):
     63        (WebCore::RealtimeMediaSource::videoSampleAvailable):
     64        (WebCore::RealtimeMediaSource::audioSamplesAvailable):
     65        * platform/mediastream/RealtimeMediaSource.h:
     66        * platform/mediastream/RealtimeOutgoingVideoSource.cpp:
     67        (WebCore::RealtimeOutgoingVideoSource::unobserveSource):
     68        (WebCore::RealtimeOutgoingVideoSource::updateBlackFramesSending):
     69        * platform/mediastream/RealtimeOutgoingVideoSource.h:
     70        * platform/mediastream/RealtimeVideoSource.cpp:
     71        (WebCore::RealtimeVideoSource::~RealtimeVideoSource):
     72        (WebCore::RealtimeVideoSource::startProducingData):
     73        (WebCore::RealtimeVideoSource::stopProducingData):
     74        (WebCore::RealtimeVideoSource::videoSampleAvailable):
     75        * platform/mediastream/RealtimeVideoSource.h:
     76        * platform/mediastream/mac/AVVideoCaptureSource.mm:
     77        (WebCore::AVVideoCaptureSource::captureOutputDidOutputSampleBufferFromConnection):
     78        * platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.h:
     79        * platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.mm:
     80        (WebCore::RealtimeIncomingVideoSourceCocoa::processNewSample):
     81        * platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.cpp:
     82        (WebCore::RealtimeOutgoingVideoSourceCocoa::videoSampleAvailable):
     83        (WebCore::RealtimeOutgoingVideoSourceCocoa::sampleBufferUpdated): Deleted.
     84        * platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.h:
     85        * platform/mock/MockRealtimeVideoSource.cpp:
     86        * testing/Internals.cpp:
     87        (WebCore::Internals::~Internals):
     88        (WebCore::Internals::stopObservingRealtimeMediaSource):
     89        (WebCore::Internals::observeMediaStreamTrack):
     90        * testing/Internals.h:
     91
    1922020-05-12  Philippe Normand  <pnormand@igalia.com>
    293

  • trunk/Source/WebCore/Modules/mediarecorder/MediaRecorder.cpp

    r260379 r261553  
    223223}
    224224
    225 void MediaRecorder::sampleBufferUpdated(MediaStreamTrackPrivate& track, MediaSample& mediaSample)
    226 {
    227     m_private->sampleBufferUpdated(track, mediaSample);
    228 }
    229 
    230225bool MediaRecorder::virtualHasPendingActivity() const
    231226{

  • trunk/Source/WebCore/Modules/mediarecorder/MediaRecorder.h

    r260379 r261553  
    109109    void trackSettingsChanged(MediaStreamTrackPrivate&) final { };
    110110    void trackEnabledChanged(MediaStreamTrackPrivate&) final { };
    111     void sampleBufferUpdated(MediaStreamTrackPrivate&, MediaSample&) final;
    112111
    113112    static CreatorFunction m_customCreator;

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h

    r260241 r261553  
    5252class VideoTrackPrivateMediaStream;
    5353
    54 class MediaPlayerPrivateMediaStreamAVFObjC final : public MediaPlayerPrivateInterface, private MediaStreamPrivate::Observer, public MediaStreamTrackPrivate::Observer, public SampleBufferDisplayLayer::Client
     54class MediaPlayerPrivateMediaStreamAVFObjC final
     55    : public MediaPlayerPrivateInterface
     56    , private MediaStreamPrivate::Observer
     57    , public MediaStreamTrackPrivate::Observer
     58    , public RealtimeMediaSource::VideoSampleObserver
     59    , public SampleBufferDisplayLayer::Client
    5560    , private LoggerHelper
    5661{
    … …  
    135140    MediaTime calculateTimelineOffset(const MediaSample&, double);
    136141   
    137     void enqueueVideoSample(MediaStreamTrackPrivate&, MediaSample&);
     142    void enqueueVideoSample(MediaSample&);
    138143    void enqueueCorrectedVideoSample(MediaSample&);
    139144    void requestNotificationWhenReadyForVideoData();
    … …  
    198203    void trackSettingsChanged(MediaStreamTrackPrivate&) override { };
    199204    void trackEnabledChanged(MediaStreamTrackPrivate&) override { };
    200     void sampleBufferUpdated(MediaStreamTrackPrivate&, MediaSample&) override;
    201205    void readyStateChanged(MediaStreamTrackPrivate&) override;
     206
     207    // RealtimeMediaSouce::VideoSampleObserver
     208    void videoSampleAvailable(MediaSample&) final;
    202209
    203210    RetainPtr<PlatformLayer> createVideoFullscreenLayer() override;

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm

    r261172 r261553  
    156156        track->pause();
    157157
    158     if (m_mediaStreamPrivate) {
     158    if (m_mediaStreamPrivate)
    159159        m_mediaStreamPrivate->removeObserver(*this);
    160160
    161         for (auto& track : m_audioTrackMap.values())
    162             track->streamTrack().removeObserver(*this);
    163 
    164         for (auto& track : m_videoTrackMap.values())
    165             track->streamTrack().removeObserver(*this);
    166     }
     161    for (auto& track : m_audioTrackMap.values())
     162        track->streamTrack().removeObserver(*this);
     163
     164    for (auto& track : m_videoTrackMap.values())
     165        track->streamTrack().removeObserver(*this);
     166
     167    if (m_activeVideoTrack)
     168        m_activeVideoTrack->source().removeVideoSampleObserver(*this);
    167169
    168170    [m_boundsChangeListener invalidate];
    … …  
    280282}
    281283
    282 void MediaPlayerPrivateMediaStreamAVFObjC::enqueueVideoSample(MediaStreamTrackPrivate& track, MediaSample& sample)
    283 {
    284     if (&track != activeVideoTrack())
    285         return;
    286 
     284void MediaPlayerPrivateMediaStreamAVFObjC::enqueueVideoSample(MediaSample& sample)
     285{
    287286    if (!m_imagePainter.mediaSample || m_displayMode != PausedImage) {
    288287        m_imagePainter.mediaSample = &sample;
    … …  
    295294        return;
    296295
    297     auto videoTrack = m_videoTrackMap.get(track.id());
     296    // FIXME: We should not query the map each time we get a sample.
     297    auto videoTrack = m_videoTrackMap.get(m_activeVideoTrack->id());
    298298    MediaTime timelineOffset = videoTrack->timelineOffset();
    299299    if (timelineOffset == MediaTime::invalidTime()) {
    … …  
    301301        videoTrack->setTimelineOffset(timelineOffset);
    302302
    303         INFO_LOG(LOGIDENTIFIER, "timeline offset for track ", track.id(), " set to ", timelineOffset);
     303        INFO_LOG(LOGIDENTIFIER, "timeline offset for track ", m_activeVideoTrack->id(), " set to ", timelineOffset);
    304304    }
    305305
    … …  
    472472MediaPlayerPrivateMediaStreamAVFObjC::DisplayMode MediaPlayerPrivateMediaStreamAVFObjC::currentDisplayMode() const
    473473{
    474     if (m_intrinsicSize.isEmpty() || !metaDataAvailable() || !m_sampleBufferDisplayLayer)
     474    if (m_intrinsicSize.isEmpty() || !metaDataAvailable())
    475475        return None;
    476476
    … …  
    736736}
    737737
    738 void MediaPlayerPrivateMediaStreamAVFObjC::sampleBufferUpdated(MediaStreamTrackPrivate& track, MediaSample& mediaSample)
    739 {
    740     ASSERT(track.id() == mediaSample.trackID());
    741     ASSERT(mediaSample.platformSample().type == PlatformSample::CMSampleBufferType);
    742     ASSERT(m_mediaStreamPrivate);
    743 
     738void MediaPlayerPrivateMediaStreamAVFObjC::videoSampleAvailable(MediaSample& mediaSample)
     739{
    744740    if (streamTime().toDouble() < 0)
    745741        return;
    746742
    747     switch (track.type()) {
    748     case RealtimeMediaSource::Type::None:
    749         // Do nothing.
    750         break;
    751     case RealtimeMediaSource::Type::Audio:
    752         break;
    753     case RealtimeMediaSource::Type::Video:
    754         if (&track == m_activeVideoTrack.get())
    755             enqueueVideoSample(track, mediaSample);
    756         break;
    757     }
     743    enqueueVideoSample(mediaSample);
    758744}
    759745
    … …  
    852838        auto oldVideoTrack = m_activeVideoTrack;
    853839        bool hideVideoLayer = true;
     840
    854841        m_activeVideoTrack = nullptr;
    855842        if (auto* activeVideoTrack = this->activeVideoTrack()) {
    … …  
    878865        m_pendingSelectedTrackCheck = false;
    879866        updateDisplayMode();
     867
     868        if (oldVideoTrack != m_activeVideoTrack) {
     869            if (oldVideoTrack)
     870                oldVideoTrack->source().removeVideoSampleObserver(*this);
     871            if (m_activeVideoTrack)
     872                m_activeVideoTrack->source().addVideoSampleObserver(*this);
     873        }
    880874    });
    881875}

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivate.h

    r259861 r261553  
    4545class SharedBuffer;
    4646
    47 class MediaRecorderPrivate :
    48     public RealtimeMediaSource::AudioSampleObserver {
     47class MediaRecorderPrivate
     48    : public RealtimeMediaSource::AudioSampleObserver
     49    , public RealtimeMediaSource::VideoSampleObserver {
    4950public:
    5051    ~MediaRecorderPrivate();
    … …  
    5556    };
    5657    WEBCORE_EXPORT static AudioVideoSelectedTracks selectTracks(MediaStreamPrivate&);
    57 
    58     virtual void sampleBufferUpdated(const MediaStreamTrackPrivate&, MediaSample&) = 0;
    5958
    6059    using FetchDataCallback = CompletionHandler<void(RefPtr<SharedBuffer>&&, const String& mimeType)>;
    … …  
    6766protected:
    6867    void setAudioSource(RefPtr<RealtimeMediaSource>&&);
     68    void setVideoSource(RefPtr<RealtimeMediaSource>&&);
    6969
    7070protected:
    … …  
    7373private:
    7474    RefPtr<RealtimeMediaSource> m_audioSource;
     75    RefPtr<RealtimeMediaSource> m_videoSource;
    7576};
    7677
    … …  
    8687}
    8788
     89inline void MediaRecorderPrivate::setVideoSource(RefPtr<RealtimeMediaSource>&& videoSource)
     90{
     91    if (m_videoSource)
     92        m_videoSource->removeVideoSampleObserver(*this);
     93
     94    m_videoSource = WTFMove(videoSource);
     95
     96    if (m_videoSource)
     97        m_videoSource->addVideoSampleObserver(*this);
     98}
     99
    88100inline MediaRecorderPrivate::~MediaRecorderPrivate()
    89101{
     102    // Subclasses should stop observing sonner than here. Otherwise they might be called from a background thread while half destroyed
     103    ASSERT(!m_audioSource);
     104    ASSERT(!m_videoSource);
    90105    if (m_audioSource)
    91106        m_audioSource->removeAudioSampleObserver(*this);
     107    if (m_videoSource)
     108        m_videoSource->removeVideoSampleObserver(*this);
    92109}
    93110

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp

    r259861 r261553  
    6060    if (selectedTracks.audioTrack)
    6161        recorder->setAudioSource(&selectedTracks.audioTrack->source());
     62    if (selectedTracks.videoTrack)
     63        recorder->setVideoSource(&selectedTracks.videoTrack->source());
    6264    return recorder;
    6365}
    … …  
    7375{
    7476    setAudioSource(nullptr);
     77    setVideoSource(nullptr);
    7578}
    7679
    77 void MediaRecorderPrivateAVFImpl::sampleBufferUpdated(const MediaStreamTrackPrivate& track, MediaSample& sampleBuffer)
     80void MediaRecorderPrivateAVFImpl::videoSampleAvailable(MediaSample& sampleBuffer)
    7881{
    79     if (track.id() != m_recordedVideoTrackID)
    80         return;
    8182    m_writer->appendVideoSampleBuffer(sampleBuffer.platformSample().sample.cmSampleBuffer);
    8283}
    … …  
    9293{
    9394    setAudioSource(nullptr);
     95    setVideoSource(nullptr);
    9496    m_writer->stopRecording();
    9597}

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateAVFImpl.h

    r259861 r261553  
    4747
    4848    // MediaRecorderPrivate
    49     void sampleBufferUpdated(const MediaStreamTrackPrivate&, MediaSample&) final;
     49    void videoSampleAvailable(MediaSample&) final;
    5050    void fetchData(FetchDataCallback&&) final;
    5151    void audioSamplesAvailable(const WTF::MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateMock.cpp

    r259861 r261553  
    4141        setAudioSource(&selectedTracks.audioTrack->source());
    4242    }
    43     if (selectedTracks.videoTrack)
     43    if (selectedTracks.videoTrack) {
    4444        m_videoTrackID = selectedTracks.videoTrack->id();
     45        setVideoSource(&selectedTracks.videoTrack->source());
     46    }
    4547}
    4648
    … …  
    4850{
    4951    setAudioSource(nullptr);
     52    setVideoSource(nullptr);
    5053}
    5154
    … …  
    5356{
    5457    setAudioSource(nullptr);
     58    setVideoSource(nullptr);
    5559}
    5660
    57 void MediaRecorderPrivateMock::sampleBufferUpdated(const MediaStreamTrackPrivate&, MediaSample&)
     61void MediaRecorderPrivateMock::videoSampleAvailable(MediaSample&)
    5862{
    5963    auto locker = holdLock(m_bufferLock);

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateMock.h

    r259861 r261553  
    4343private:
    4444    // MediaRecorderPrivate
    45     void sampleBufferUpdated(const MediaStreamTrackPrivate&, MediaSample&) final;
     45    void videoSampleAvailable(MediaSample&) final;
    4646    void fetchData(FetchDataCallback&&) final;
    4747    void audioSamplesAvailable(const WTF::MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;

  • trunk/Source/WebCore/platform/mediastream/MediaStreamTrackPrivate.cpp

    r261172 r261553  
    237237}
    238238
    239 void MediaStreamTrackPrivate::videoSampleAvailable(MediaSample& mediaSample)
    240 {
    241     ASSERT(isMainThread());
    242     if (!m_haveProducedData) {
    243         m_haveProducedData = true;
    244         updateReadyState();
    245     }
    246 
    247     if (!enabled())
    248         return;
    249 
    250     mediaSample.setTrackID(id());
    251     forEachObserver([&](auto& observer) {
    252         observer.sampleBufferUpdated(*this, mediaSample);
    253     });
    254 }
    255 
    256 void MediaStreamTrackPrivate::hasStartedProducingAudioData()
    257 {
    258     if (m_haveProducedData)
    259         return;
    260     m_haveProducedData = true;
     239void MediaStreamTrackPrivate::hasStartedProducingData()
     240{
     241    ASSERT(isMainThread());
     242    if (m_hasStartedProducingData)
     243        return;
     244    m_hasStartedProducingData = true;
    261245    updateReadyState();
    262246}
    … …  
    268252    if (m_isEnded)
    269253        state = ReadyState::Ended;
    270     else if (m_haveProducedData)
     254    else if (m_hasStartedProducingData)
    271255        state = ReadyState::Live;
    272256

  • trunk/Source/WebCore/platform/mediastream/MediaStreamTrackPrivate.h

    r260241 r261553  
    6060        virtual void trackSettingsChanged(MediaStreamTrackPrivate&) = 0;
    6161        virtual void trackEnabledChanged(MediaStreamTrackPrivate&) = 0;
    62         virtual void sampleBufferUpdated(MediaStreamTrackPrivate&, MediaSample&) { };
    6362        virtual void readyStateChanged(MediaStreamTrackPrivate&) { };
    6463    };
    … …  
    134133    void sourceSettingsChanged() final;
    135134    bool preventSourceFromStopping() final;
    136     void videoSampleAvailable(MediaSample&) final;
    137135    void audioUnitWillStart() final;
    138     void hasStartedProducingAudioData() final;
     136    void hasStartedProducingData() final;
    139137
    140138    void updateReadyState();
    … …  
    154152    bool m_isEnabled { true };
    155153    bool m_isEnded { false };
    156     bool m_haveProducedData { false };
    157     bool m_hasSentStartProducedData { false };
     154    bool m_hasStartedProducingData { false };
    158155    HintValue m_contentHint { HintValue::Empty };
    159156    RefPtr<WebAudioSourceProvider> m_audioSourceProvider;

  • trunk/Source/WebCore/platform/mediastream/RealtimeMediaSource.cpp

    r261373 r261553  
    7575}
    7676
     77void RealtimeMediaSource::addVideoSampleObserver(VideoSampleObserver& observer)
     78{
     79    ASSERT(isMainThread());
     80    auto locker = holdLock(m_videoSampleObserversLock);
     81    m_videoSampleObservers.add(&observer);
     82}
     83
     84void RealtimeMediaSource::removeVideoSampleObserver(VideoSampleObserver& observer)
     85{
     86    ASSERT(isMainThread());
     87    auto locker = holdLock(m_videoSampleObserversLock);
     88    m_videoSampleObservers.remove(&observer);
     89}
     90
    7791void RealtimeMediaSource::addObserver(Observer& observer)
    7892{
    … …  
    155169}
    156170
     171void RealtimeMediaSource::updateHasStartedProducingData()
     172{
     173    if (m_hasStartedProducingData)
     174        return;
     175
     176    callOnMainThread([this, weakThis = makeWeakPtr(this)] {
     177        if (!weakThis)
     178            return;
     179        if (m_hasStartedProducingData)
     180            return;
     181        m_hasStartedProducingData = true;
     182        forEachObserver([&](auto& observer) {
     183            observer.hasStartedProducingData();
     184        });
     185    });
     186}
     187
    157188void RealtimeMediaSource::videoSampleAvailable(MediaSample& mediaSample)
    158189{
     190    // FIXME: Migrate RealtimeMediaSource clients to non main thread processing.
     191    ASSERT(isMainThread());
    159192#if !RELEASE_LOG_DISABLED
    160193    ++m_frameCount;
    … …  
    171204#endif
    172205
    173     forEachObserver([&](auto& observer) {
    174         observer.videoSampleAvailable(mediaSample);
    175     });
     206    updateHasStartedProducingData();
     207
     208    auto locker = holdLock(m_videoSampleObserversLock);
     209    for (auto* observer : m_videoSampleObservers)
     210        observer->videoSampleAvailable(mediaSample);
    176211}
    177212
    178213void RealtimeMediaSource::audioSamplesAvailable(const MediaTime& time, const PlatformAudioData& audioData, const AudioStreamDescription& description, size_t numberOfFrames)
    179214{
    180     if (!m_hasSentStartProducedAudioData) {
    181         callOnMainThread([this, weakThis = makeWeakPtr(this)] {
    182             if (!weakThis)
    183                 return;
    184             if (m_hasSentStartProducedAudioData)
    185                 return;
    186             m_hasSentStartProducedAudioData = true;
    187             forEachObserver([&](auto& observer) {
    188                 observer.hasStartedProducingAudioData();
    189             });
    190         });
    191     }
     215    updateHasStartedProducingData();
    192216
    193217    auto locker = holdLock(m_audioSampleObserversLock);

  • trunk/Source/WebCore/platform/mediastream/RealtimeMediaSource.h

    r261373 r261553  
    9090        virtual bool preventSourceFromStopping() { return false; }
    9191
    92         // Called on the main thread.
    93         virtual void videoSampleAvailable(MediaSample&) { }
    94 
    95         virtual void hasStartedProducingAudioData() { }
     92        virtual void hasStartedProducingData() { }
    9693    };
    9794    class AudioSampleObserver {
    … …  
    10299        virtual void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t /*numberOfFrames*/) = 0;
    103100    };
     101    class VideoSampleObserver {
     102    public:
     103        virtual ~VideoSampleObserver() = default;
     104
     105        virtual void videoSampleAvailable(MediaSample&) { }
     106    };
    104107
    105108    virtual ~RealtimeMediaSource() = default;
    … …  
    139142    WEBCORE_EXPORT void addAudioSampleObserver(AudioSampleObserver&);
    140143    WEBCORE_EXPORT void removeAudioSampleObserver(AudioSampleObserver&);
     144
     145    WEBCORE_EXPORT void addVideoSampleObserver(VideoSampleObserver&);
     146    WEBCORE_EXPORT void removeVideoSampleObserver(VideoSampleObserver&);
    141147
    142148    const IntSize size() const;
    … …  
    251257    virtual void hasEnded() { }
    252258
     259    void updateHasStartedProducingData();
     260
    253261#if !RELEASE_LOG_DISABLED
    254262    RefPtr<const Logger> m_logger;
    … …  
    267275    mutable RecursiveLock m_audioSampleObserversLock;
    268276    HashSet<AudioSampleObserver*> m_audioSampleObservers;
     277
     278    mutable RecursiveLock m_videoSampleObserversLock;
     279    HashSet<VideoSampleObserver*> m_videoSampleObservers;
    269280
    270281    IntSize m_size;
    … …  
    285296    bool m_captureDidFailed { false };
    286297    bool m_isEnded { false };
    287     bool m_hasSentStartProducedAudioData { false };
     298    bool m_hasStartedProducingData { false };
    288299};
    289300

  • trunk/Source/WebCore/platform/mediastream/RealtimeOutgoingVideoSource.cpp

    r259289 r261553  
    7575{
    7676    m_videoSource->removeObserver(*this);
     77    m_videoSource->source().removeVideoSampleObserver(*this);
    7778}
    7879
    … …  
    115116{
    116117    if (!m_muted && m_enabled) {
     118        m_videoSource->source().addVideoSampleObserver(*this);
    117119        if (m_blackFrameTimer.isActive())
    118120            m_blackFrameTimer.stop();
    … …  
    120122    }
    121123
     124    m_videoSource->source().removeVideoSampleObserver(*this);
    122125    sendBlackFramesIfNeeded();
    123126}

  • trunk/Source/WebCore/platform/mediastream/RealtimeOutgoingVideoSource.h

    r260241 r261553  
    5252    , public webrtc::VideoTrackSourceInterface
    5353    , private MediaStreamTrackPrivate::Observer
     54    , private RealtimeMediaSource::VideoSampleObserver
    5455#if !RELEASE_LOG_DISABLED
    5556    , private LoggerHelper
    … …  
    129130    void trackEnabledChanged(MediaStreamTrackPrivate&) final { sourceEnabledChanged(); }
    130131    void trackSettingsChanged(MediaStreamTrackPrivate&) final { initializeFromSource(); }
    131     void sampleBufferUpdated(MediaStreamTrackPrivate&, MediaSample&) override { }
    132132    void trackEnded(MediaStreamTrackPrivate&) final { }
     133
     134    // RealtimeMediaSource::VideoSampleObserver API
     135    void videoSampleAvailable(MediaSample&) override { }
    133136
    134137    Ref<MediaStreamTrackPrivate> m_videoSource;

  • trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.cpp

    r260364 r261553  
    4747RealtimeVideoSource::~RealtimeVideoSource()
    4848{
     49    m_source->removeVideoSampleObserver(*this);
    4950    m_source->removeObserver(*this);
    5051}
    … …  
    5354{
    5455    m_source->start();
     56    m_source->addVideoSampleObserver(*this);
    5557}
    5658
    5759void RealtimeVideoSource::stopProducingData()
    5860{
     61    m_source->removeVideoSampleObserver(*this);
    5962    m_source->stop();
    6063}
    … …  
    167170void RealtimeVideoSource::videoSampleAvailable(MediaSample& sample)
    168171{
    169     if (!isProducingData())
    170         return;
    171 
    172172    if (m_frameDecimation > 1 && ++m_frameDecimationCounter % m_frameDecimation)
    173173        return;

  • trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.h

    r261373 r261553  
    3434class ImageTransferSessionVT;
    3535
    36 class RealtimeVideoSource final : public RealtimeMediaSource, public RealtimeMediaSource::Observer {
     36class RealtimeVideoSource final
     37    : public RealtimeMediaSource
     38    , public RealtimeMediaSource::Observer
     39    , public RealtimeMediaSource::VideoSampleObserver {
    3740public:
    3841    static Ref<RealtimeVideoSource> create(Ref<RealtimeVideoCaptureSource>&& source) { return adoptRef(*new RealtimeVideoSource(WTFMove(source))); }
    … …  
    5962    bool isSameAs(RealtimeMediaSource& source) const final { return this == &source || m_source.ptr() == &source; }
    6063
    61     // Observer
     64    // RealtimeMediaSource::Observer
    6265    void sourceMutedChanged() final;
    6366    void sourceSettingsChanged() final;
    6467    void sourceStopped() final;
    6568    bool preventSourceFromStopping() final;
     69
     70    // RealtimeMediaSource::VideoSampleObserver
    6671    void videoSampleAvailable(MediaSample&) final;
    6772

  • trunk/Source/WebCore/platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp

    r260937 r261553  
    108108class WebKitMediaStreamTrackObserver
    109109    : public MediaStreamTrackPrivate::Observer
    110     , public RealtimeMediaSource::AudioSampleObserver {
     110    , public RealtimeMediaSource::AudioSampleObserver
     111    , public RealtimeMediaSource::VideoSampleObserver {
    111112    WTF_MAKE_FAST_ALLOCATED;
    112113public:
    … …  
    131132    void readyStateChanged(MediaStreamTrackPrivate&) final { };
    132133
    133     void sampleBufferUpdated(MediaStreamTrackPrivate&, MediaSample& sample) final
     134    void videoSampleAvailable(MediaSample& sample) final
    134135    {
    135136        if (!m_enabled)
    … …  
    388389        for (auto& track : self->stream->tracks()) {
    389390            track->source().removeAudioSampleObserver(*self->mediaStreamTrackObserver.get());
     391            track->source().removeVideoSampleObserver(*self->mediaStreamTrackObserver.get());
    390392            track->removeObserver(*self->mediaStreamTrackObserver.get());
    391393        }
    … …  
    410412            for (auto& track : self->stream->tracks()) {
    411413                track->source().removeAudioSampleObserver(*self->mediaStreamTrackObserver.get());
     414                track->source().removeVideoSampleObserver(*self->mediaStreamTrackObserver.get());
    412415                track->removeObserver(*self->mediaStreamTrackObserver.get());
    413416            }
    414417        } else if (self->track) {
    415418            self->track->source().removeAudioSampleObserver(*self->mediaStreamTrackObserver.get());
     419            self->track->source().removeVideoSampleObserver(*self->mediaStreamTrackObserver.get());
    416420            self->track->removeObserver(*self->mediaStreamTrackObserver.get());
    417421        }
    … …  
    561565    if (observe_track) {
    562566        track->addObserver(*self->mediaStreamTrackObserver.get());
    563         track->source().addAudioSampleObserver(*self->mediaStreamTrackObserver.get());
     567        auto& source = track->source();
     568        switch (source.type()) {
     569        case RealtimeMediaSource::Type::Audio:
     570            source.addAudioSampleObserver(*self->mediaStreamTrackObserver.get());
     571            break;
     572        case RealtimeMediaSource::Type::Video:
     573            source.addVideoSampleObserver(*self->mediaStreamTrackObserver.get());
     574            break;
     575        case RealtimeMediaSource::Type::None:
     576            ASSERT_NOT_REACHED();
     577        }
    564578    }
    565579    gst_element_sync_state_with_parent(element);

  • trunk/Source/WebCore/platform/mediastream/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.cpp

    r239921 r261553  
    5151}
    5252
    53 void RealtimeOutgoingVideoSourceLibWebRTC::sampleBufferUpdated(MediaStreamTrackPrivate&, MediaSample& sample)
     53void RealtimeOutgoingVideoSourceLibWebRTC::videoSampleAvailable(MediaSample& sample)
    5454{
    55     if (isSilenced())
    56         return;
    57 
    5855    switch (sample.videoRotation()) {
    5956    case MediaSample::VideoRotation::None:

  • trunk/Source/WebCore/platform/mediastream/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.h

    r237861 r261553  
    4242    rtc::scoped_refptr<webrtc::VideoFrameBuffer> createBlackFrame(size_t, size_t) final;
    4343
    44     // MediaStreamTrackPrivate::Observer API
    45     void sampleBufferUpdated(MediaStreamTrackPrivate&, MediaSample&) final;
     44    // RealtimeMediaSource::VideoSampleObserver API
     45    void videoSampleAvailable(MediaSample&) final;
    4646};
    4747

  • trunk/Source/WebCore/platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.h

    r236445 r261553  
    5151    void OnFrame(const webrtc::VideoFrame&) final;
    5252
    53     RetainPtr<CMSampleBufferRef> m_buffer;
    5453    RetainPtr<CVPixelBufferRef> m_blackFrame;
    5554    int m_blackFrameWidth { 0 };

  • trunk/Source/WebCore/platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.mm

    r260366 r261553  
    201201void RealtimeIncomingVideoSourceCocoa::processNewSample(CMSampleBufferRef sample, unsigned width, unsigned height, MediaSample::VideoRotation rotation)
    202202{
    203     m_buffer = sample;
    204203    auto size = this->size();
    205204    if (WTF::safeCast<int>(width) != size.width() || WTF::safeCast<int>(height) != size.height())

  • trunk/Source/WebCore/platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.cpp

    r246759 r261553  
    6363}
    6464
    65 void RealtimeOutgoingVideoSourceCocoa::sampleBufferUpdated(MediaStreamTrackPrivate&, MediaSample& sample)
     65void RealtimeOutgoingVideoSourceCocoa::videoSampleAvailable(MediaSample& sample)
    6666{
    67     if (isSilenced())
    68         return;
    69 
    7067#if !RELEASE_LOG_DISABLED
    7168    if (!(++m_numberOfFrames % 60))

  • trunk/Source/WebCore/platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.h

    r246759 r261553  
    4545    rtc::scoped_refptr<webrtc::VideoFrameBuffer> createBlackFrame(size_t width, size_t height) final;
    4646
    47     // MediaStreamTrackPrivate::Observer API
    48     void sampleBufferUpdated(MediaStreamTrackPrivate&, MediaSample&) final;
     47    // RealtimeMediaSource::VideoSampleObserver API
     48    void videoSampleAvailable(MediaSample&) final;
    4949
    5050    RetainPtr<CVPixelBufferRef> convertToYUV(CVPixelBufferRef);

  • trunk/Source/WebCore/testing/Internals.cpp

    r261500 r261553  
    477477{
    478478#if ENABLE(MEDIA_STREAM)
    479     if (m_trackSource) {
    480         m_trackSource->removeObserver(*this);
    481         m_trackSource->removeAudioSampleObserver(*this);
    482     }
     479    stopObservingRealtimeMediaSource();
    483480#endif
    484481}
    … …  
    50405037}
    50415038
     5039void Internals::stopObservingRealtimeMediaSource()
     5040{
     5041    if (!m_trackSource)
     5042        return;
     5043
     5044    switch (m_trackSource->type()) {
     5045    case RealtimeMediaSource::Type::Audio:
     5046        m_trackSource->removeAudioSampleObserver(*this);
     5047        break;
     5048    case RealtimeMediaSource::Type::Video:
     5049        m_trackSource->removeVideoSampleObserver(*this);
     5050        break;
     5051    case RealtimeMediaSource::Type::None:
     5052        ASSERT_NOT_REACHED();
     5053    }
     5054    m_trackSource->removeObserver(*this);
     5055
     5056    m_trackSource = nullptr;
     5057    m_trackAudioSampleCount = 0;
     5058    m_trackVideoSampleCount = 0;
     5059}
     5060
    50425061void Internals::observeMediaStreamTrack(MediaStreamTrack& track)
    50435062{
    5044     if (m_trackSource) {
    5045         m_trackSource->removeObserver(*this);
    5046         m_trackSource->removeAudioSampleObserver(*this);
    5047 
    5048         m_trackAudioSampleCount = 0;
    5049         m_trackVideoSampleCount = 0;
    5050     }
     5063    stopObservingRealtimeMediaSource();
    50515064
    50525065    m_trackSource = &track.source();
    50535066    m_trackSource->addObserver(*this);
    5054     m_trackSource->addAudioSampleObserver(*this);
     5067    switch (m_trackSource->type()) {
     5068    case RealtimeMediaSource::Type::Audio:
     5069        m_trackSource->addAudioSampleObserver(*this);
     5070        break;
     5071    case RealtimeMediaSource::Type::Video:
     5072        m_trackSource->addVideoSampleObserver(*this);
     5073        break;
     5074    case RealtimeMediaSource::Type::None:
     5075        ASSERT_NOT_REACHED();
     5076    }
    50555077}
    50565078

  • trunk/Source/WebCore/testing/Internals.h

    r261479 r261553  
    131131    , private RealtimeMediaSource::Observer
    132132    , private RealtimeMediaSource::AudioSampleObserver
     133    , private RealtimeMediaSource::VideoSampleObserver
    133134#endif
    134135    {
    … …  
    764765
    765766#if ENABLE(MEDIA_STREAM)
     767    void stopObservingRealtimeMediaSource();
     768
    766769    void setMockAudioTrackChannelNumber(MediaStreamTrack&, unsigned short);
    767770    void setCameraMediaStreamTrackOrientation(MediaStreamTrack&, int orientation);

  • trunk/Source/WebKit/ChangeLog

    r261549 r261553  
     12020-05-12  Youenn Fablet  <youenn@apple.com>
     2
     3        Introduce a RealtimeMediaSource video sample observer
     4        https://bugs.webkit.org/show_bug.cgi?id=211718
     5
     6        Reviewed by Eric Carlson.
     7
     8        * UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp:
     9        (WebKit::UserMediaCaptureManagerProxy::SourceProxy::SourceProxy):
     10        (WebKit::UserMediaCaptureManagerProxy::SourceProxy::~SourceProxy):
     11        * WebProcess/GPU/webrtc/MediaRecorderPrivate.cpp:
     12        (WebKit::MediaRecorderPrivate::MediaRecorderPrivate):
     13        (WebKit::MediaRecorderPrivate::~MediaRecorderPrivate):
     14        (WebKit::MediaRecorderPrivate::videoSampleAvailable):
     15        (WebKit::MediaRecorderPrivate::stopRecording):
     16        (WebKit::MediaRecorderPrivate::sampleBufferUpdated): Deleted.
     17        * WebProcess/GPU/webrtc/MediaRecorderPrivate.h:
     18
    1192020-05-12  Mark Lam  <mark.lam@apple.com>
    220

  • trunk/Source/WebKit/UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp

    r261375 r261553  
    5151
    5252class UserMediaCaptureManagerProxy::SourceProxy
    53     : public RealtimeMediaSource::Observer
    54     , public RealtimeMediaSource::AudioSampleObserver
     53    : private RealtimeMediaSource::Observer
     54    , private RealtimeMediaSource::AudioSampleObserver
     55    , private RealtimeMediaSource::VideoSampleObserver
    5556    , public SharedRingBufferStorage::Client {
    5657    WTF_MAKE_FAST_ALLOCATED;
    … …  
    6364    {
    6465        m_source->addObserver(*this);
    65         m_source->addAudioSampleObserver(*this);
     66        switch (m_source->type()) {
     67        case RealtimeMediaSource::Type::Audio:
     68            m_source->addAudioSampleObserver(*this);
     69            break;
     70        case RealtimeMediaSource::Type::Video:
     71            m_source->addVideoSampleObserver(*this);
     72            break;
     73        case RealtimeMediaSource::Type::None:
     74            ASSERT_NOT_REACHED();
     75        }
    6676    }
    6777
    … …  
    6979    {
    7080        storage().invalidate();
    71         m_source->removeAudioSampleObserver(*this);
     81
     82        switch (m_source->type()) {
     83        case RealtimeMediaSource::Type::Audio:
     84            m_source->removeAudioSampleObserver(*this);
     85            break;
     86        case RealtimeMediaSource::Type::Video:
     87            m_source->removeVideoSampleObserver(*this);
     88            break;
     89        case RealtimeMediaSource::Type::None:
     90            ASSERT_NOT_REACHED();
     91        }
    7292        m_source->removeObserver(*this);
    7393    }

  • trunk/Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderPrivate.cpp

    r261133 r261553  
    6565    }
    6666
    67     m_connection->sendWithAsyncReply(Messages::RemoteMediaRecorderManager::CreateRecorder { m_identifier, !!selectedTracks.audioTrack, width, height }, [this, weakThis = makeWeakPtr(this), audioTrack = makeRefPtr(selectedTracks.audioTrack)](auto&& exception) {
     67    m_connection->sendWithAsyncReply(Messages::RemoteMediaRecorderManager::CreateRecorder { m_identifier, !!selectedTracks.audioTrack, width, height }, [this, weakThis = makeWeakPtr(this), audioTrack = makeRefPtr(selectedTracks.audioTrack), videoTrack = makeRefPtr(selectedTracks.videoTrack)](auto&& exception) {
    6868        if (!weakThis)
    6969            return;
    … …  
    7272        if (audioTrack)
    7373            setAudioSource(&audioTrack->source());
     74        if (videoTrack)
     75            setVideoSource(&videoTrack->source());
    7476    }, 0);
    7577}
    … …  
    7880{
    7981    setAudioSource(nullptr);
     82    setVideoSource(nullptr);
    8083    m_connection->send(Messages::RemoteMediaRecorderManager::ReleaseRecorder { m_identifier }, 0);
    8184}
    8285
    83 void MediaRecorderPrivate::sampleBufferUpdated(const WebCore::MediaStreamTrackPrivate& track, WebCore::MediaSample& sample)
     86void MediaRecorderPrivate::videoSampleAvailable(MediaSample& sample)
    8487{
    85     if (track.id() != m_recordedVideoTrackID)
    86         return;
    8788    if (auto remoteSample = RemoteVideoSample::create(sample))
    8889        m_connection->send(Messages::RemoteMediaRecorder::VideoSampleAvailable { WTFMove(*remoteSample) }, m_identifier);
    … …  
    129130{
    130131    setAudioSource(nullptr);
     132    setVideoSource(nullptr);
    131133    m_connection->send(Messages::RemoteMediaRecorder::StopRecording { }, m_identifier);
    132134}

  • trunk/Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderPrivate.h

    r259861 r261553  
    5656private:
    5757    // WebCore::MediaRecorderPrivate
    58     void sampleBufferUpdated(const WebCore::MediaStreamTrackPrivate&, WebCore::MediaSample&) final;
     58    void videoSampleAvailable(WebCore::MediaSample&) final;
    5959    void fetchData(CompletionHandler<void(RefPtr<WebCore::SharedBuffer>&&, const String& mimeType)>&&) final;
    6060    void stopRecording() final;
Note: See TracChangeset for help on using the changeset viewer.