Changeset 268398 in webkit

Timestamp:

Oct 13, 2020, 10:06:16 AM (6 years ago)

Author:

youenn@apple.com

Message:

Webcam video from navigator.mediaDevices.getUserMedia() to 2D canvas fails on Safari on iPhone
​https://bugs.webkit.org/show_bug.cgi?id=217578
<rdar://problem/70183875>

Reviewed by Eric Carlson.

It is unneeded for MediaStream video backends to enqueue samples to the display layer if the video element is hidden.
In iOS, the samples may never be flushed which might cause capture failing after enqueuing too many camera samples.
To avoid that, we no longer enqueue samples to the display layer when the player is not visible.
In case of canvas painting, other backends need to be made visible for canvas painting to work.
For MediaStream backend, we do not need that since we always keep the last sample.
For that reason, we keep the backend as not visible even if canvas happens.
We do so by introducing a setVisibleForCanvas that is a no-op for MediaStream backend and similar to setVisible for other backends.
For good measure, we now flush the MediaStream display layer whenever visibility changed.

Manually tested.

• html/HTMLVideoElement.cpp:

(WebCore::HTMLVideoElement::paintCurrentFrameInContext):

• platform/graphics/MediaPlayer.cpp:

(WebCore::MediaPlayer::setVisibleForCanvas):

• platform/graphics/MediaPlayer.h:
• platform/graphics/MediaPlayerPrivate.h:

(WebCore::MediaPlayerPrivateInterface::setVisibleForCanvas):

• platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
• platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:

(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::enqueueVideoSample):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setVisible):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setVisibleForCanvas):

Location:

trunk/Source/WebCore

Files:

• ChangeLog (modified) (1 diff)
• html/HTMLVideoElement.cpp (modified) (1 diff)
• platform/graphics/MediaPlayer.cpp (modified) (1 diff)
• platform/graphics/MediaPlayer.h (modified) (1 diff)
• platform/graphics/MediaPlayerPrivate.h (modified) (1 diff)
• platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h (modified) (1 diff)
• platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm (modified) (2 diffs)

trunk/Source/WebCore/ChangeLog

@@ +1 @@
+2020-10-13  Youenn Fablet  <youenn@apple.com>
+
+        Webcam video from navigator.mediaDevices.getUserMedia() to 2D canvas fails on Safari on iPhone
+        https://bugs.webkit.org/show_bug.cgi?id=217578
+        <rdar://problem/70183875>
+
+        Reviewed by Eric Carlson.
+
+        It is unneeded for MediaStream video backends to enqueue samples to the display layer if the video element is hidden.
+        In iOS, the samples may never be flushed which might cause capture failing after enqueuing too many camera samples.
+        To avoid that, we no longer enqueue samples to the display layer when the player is not visible.
+        In case of canvas painting, other backends need to be made visible for canvas painting to work.
+        For MediaStream backend, we do not need that since we always keep the last sample.
+        For that reason, we keep the backend as not visible even if canvas happens.
+        We do so by introducing a setVisibleForCanvas that is a no-op for MediaStream backend and similar to setVisible for other backends.
+        For good measure, we now flush the MediaStream display layer whenever visibility changed.
+ 
+        Manually tested.
+
+        * html/HTMLVideoElement.cpp:
+        (WebCore::HTMLVideoElement::paintCurrentFrameInContext):
+        * platform/graphics/MediaPlayer.cpp:
+        (WebCore::MediaPlayer::setVisibleForCanvas):
+        * platform/graphics/MediaPlayer.h:
+        * platform/graphics/MediaPlayerPrivate.h:
+        (WebCore::MediaPlayerPrivateInterface::setVisibleForCanvas):
+        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
+        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
+        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::enqueueVideoSample):
+        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setVisible):
+        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setVisibleForCanvas):
+
 2020-10-13  Youenn Fablet  <youenn@apple.com>
 

trunk/Source/WebCore/html/HTMLVideoElement.cpp

@@ -305 +305 @@
         return;
     
-    player->setVisible(true); // Make player visible or it won't draw.
+    player->setVisibleForCanvas(true); // Make player visible or it won't draw.
     context.paintFrameForMedia(*player, destRect);
 }

trunk/Source/WebCore/platform/graphics/MediaPlayer.cpp

@@ -963 +963 @@
 }
 
+void MediaPlayer::setVisibleForCanvas(bool visible)
+{
+    m_visible = visible;
+    m_private->setVisibleForCanvas(visible);
+}
+
 MediaPlayer::Preload MediaPlayer::preload() const
 {

trunk/Source/WebCore/platform/graphics/MediaPlayer.h

@@ -349 +349 @@
     bool visible() const;
     void setVisible(bool);
+    void setVisibleForCanvas(bool);
 
     void prepareToPlay();

trunk/Source/WebCore/platform/graphics/MediaPlayerPrivate.h

@@ -97 +97 @@
 
     virtual void setVisible(bool) = 0;
+    virtual void setVisibleForCanvas(bool visible) { setVisible(visible); }
 
     virtual float duration() const { return 0; }

trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h

@@ -125 +125 @@
 
     void setVisible(bool) final;
+    void setVisibleForCanvas(bool) final;
 
     MediaTime durationMediaTime() const override;

trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm

@@ -253 +253 @@
 void MediaPlayerPrivateMediaStreamAVFObjC::enqueueVideoSample(MediaSample& sample)
 {
+    if (!m_visible)
+        return;
+
     auto locker = tryHoldLock(m_sampleBufferDisplayLayerLock);
     if (!locker)
@@ -594 +597 @@
 
     m_visible = visible;
-    if (m_visible)
-        flushRenderers();
+    flushRenderers();
+}
+
+void MediaPlayerPrivateMediaStreamAVFObjC::setVisibleForCanvas(bool)
+{
 }