Changeset 288435 in webkit

Timestamp:

Jan 24, 2022 4:41:47 AM (6 months ago)

Author:

youenn@apple.com

Message:

MediaStream canvas.captureStream() playback to LocalSampleBufferDisplayLayer is flaky
​https://bugs.webkit.org/show_bug.cgi?id=230623
<rdar://problem/83668578>

Reviewed by Kimmo Kinnunen.

There are a few issues when creating the video element, setting the srcObject and playing it right away.
The first thing is that we try to render the video element while thinking that it might be video accelerated
(it is always the case for MediaStreamTrack based tracks), but the video element is asynchronously updating whether it is accelerated.
Following what other MediaPlayerPrivate are doing, when MediaPlayerPrivateMediaStreamAVFObjC has set up its video layers,
we schedule a call to renderingModeChanged which will trigger a relayout.

We also ensure, when the layer is fully initialized to set all values (bounds, hide states) as otherwise, this might be lost on GPUProcess
which currently does not register a RemoteSampleBufferDisplayLayer at the time the corresponding IPC message is received.

The additional issue is in the case the MediaStreamTrack source only generates one frame for some time.
In that case, and if the frame is enqueueud while setting up the video layers, we were not rendering this frame.
To fix this, when the video layers are ready, we reenqueue the last video frame as neeeded.

Manually tested.

• platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
• platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:

Location:

trunk/Source/WebCore

Files:

• ChangeLog (modified) (1 diff)
• platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h (modified) (2 diffs)
• platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm (modified) (6 diffs)

trunk/Source/WebCore/ChangeLog

@@ +1 @@
+2022-01-24  Youenn Fablet  <youenn@apple.com>
+
+        MediaStream canvas.captureStream() playback to LocalSampleBufferDisplayLayer is flaky
+        https://bugs.webkit.org/show_bug.cgi?id=230623
+        <rdar://problem/83668578>
+
+        Reviewed by Kimmo Kinnunen.
+
+        There are a few issues when creating the video element, setting the srcObject and playing it right away.
+        The first thing is that we try to render the video element while thinking that it might be video accelerated
+        (it is always the case for MediaStreamTrack based tracks), but the video element is asynchronously updating whether it is accelerated.
+        Following what other MediaPlayerPrivate are doing, when MediaPlayerPrivateMediaStreamAVFObjC has set up its video layers,
+        we schedule a call to renderingModeChanged which will trigger a relayout.
+
+        We also ensure, when the layer is fully initialized to set all values (bounds, hide states) as otherwise, this might be lost on GPUProcess
+        which currently does not register a RemoteSampleBufferDisplayLayer at the time the corresponding IPC message is received.
+
+        The additional issue is in the case the MediaStreamTrack source only generates one frame for some time.
+        In that case, and if the frame is enqueueud while setting up the video layers, we were not rendering this frame.
+        To fix this, when the video layers are ready, we reenqueue the last video frame as neeeded.
+
+        Manually tested.
+
+        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
+        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
+
 2022-01-24  Kimmo Kinnunen  <kkinnunen@apple.com>
 

trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h

@@ -139 +139 @@
     void processNewVideoSample(MediaSample&, VideoSampleMetadata, Seconds);
     void enqueueVideoSample(MediaSample&);
+    void reenqueueCurrentVideoSampleIfNeeded();
     void requestNotificationWhenReadyForVideoData();
 
@@ -169 +170 @@
     void updateTracks();
     void updateRenderingMode();
+    void scheduleRenderingModeChanged();
     void checkSelectedVideoTrack();
     void updateDisplayLayer();

trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm

@@ -278 +278 @@
 }
 
+void MediaPlayerPrivateMediaStreamAVFObjC::reenqueueCurrentVideoSampleIfNeeded()
+{
+    if (!m_currentVideoSampleLock.tryLock())
+        return;
+    Locker locker { AdoptLock, m_currentVideoSampleLock };
+
+    if (!m_currentVideoSample && !m_imagePainter.mediaSample)
+        return;
+
+    enqueueVideoSample(m_currentVideoSample ? *m_currentVideoSample : *m_imagePainter.mediaSample);
+}
+
 void MediaPlayerPrivateMediaStreamAVFObjC::processNewVideoSample(MediaSample& sample, VideoSampleMetadata metadata, Seconds presentationTime)
 {
@@ -399 +411 @@
     }
 
-    updateRenderingMode();
+    scheduleRenderingModeChanged();
+
+    m_sampleBufferDisplayLayer->updateBoundsAndPosition(m_sampleBufferDisplayLayer->rootLayer().bounds, m_videoRotation);
+    m_sampleBufferDisplayLayer->updateDisplayMode(m_displayMode < PausedImage, hideRootLayer());
     m_shouldUpdateDisplayLayer = true;
 
@@ -417 +432 @@
         m_sampleBufferDisplayLayer = nullptr;
 
-    updateRenderingMode();
+    scheduleRenderingModeChanged();
     
     m_videoLayerManager->didDestroyVideoLayer();
@@ -552 +567 @@
         m_sampleBufferDisplayLayer->play();
     updateDisplayMode();
+    reenqueueCurrentVideoSampleIfNeeded();
 
     scheduleDeferredTask([this] {
@@ -623 +639 @@
     m_isPageVisible = isVisible;
     flushRenderers();
+    reenqueueCurrentVideoSampleIfNeeded();
 }
 
@@ -712 +729 @@
 void MediaPlayerPrivateMediaStreamAVFObjC::updateRenderingMode()
 {
-    if (!updateDisplayMode())
-        return;
-
+    if (updateDisplayMode())
+        scheduleRenderingModeChanged();
+}
+
+void MediaPlayerPrivateMediaStreamAVFObjC::scheduleRenderingModeChanged()
+{
     scheduleDeferredTask([this] {
         if (m_player)
             m_player->renderingModeChanged();
+        reenqueueCurrentVideoSampleIfNeeded();
     });
-
 }