Changeset 268837 in webkit

Timestamp:

Oct 21, 2020, 3:16:06 PM (6 years ago)

Author:

Alan Coon

Message:

Cherry-pick r268398. rdar://problem/70541902

Webcam video from navigator.mediaDevices.getUserMedia() to 2D canvas fails on Safari on iPhone
​https://bugs.webkit.org/show_bug.cgi?id=217578
<rdar://problem/70183875>

Reviewed by Eric Carlson.

It is unneeded for MediaStream video backends to enqueue samples to the display layer if the video element is hidden.
In iOS, the samples may never be flushed which might cause capture failing after enqueuing too many camera samples.
To avoid that, we no longer enqueue samples to the display layer when the player is not visible.
In case of canvas painting, other backends need to be made visible for canvas painting to work.
For MediaStream backend, we do not need that since we always keep the last sample.
For that reason, we keep the backend as not visible even if canvas happens.
We do so by introducing a setVisibleForCanvas that is a no-op for MediaStream backend and similar to setVisible for other backends.
For good measure, we now flush the MediaStream display layer whenever visibility changed.

Manually tested.

• html/HTMLVideoElement.cpp: (WebCore::HTMLVideoElement::paintCurrentFrameInContext):
• platform/graphics/MediaPlayer.cpp: (WebCore::MediaPlayer::setVisibleForCanvas):
• platform/graphics/MediaPlayer.h:
• platform/graphics/MediaPlayerPrivate.h: (WebCore::MediaPlayerPrivateInterface::setVisibleForCanvas):
• platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
• platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm: (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::enqueueVideoSample): (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setVisible): (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setVisibleForCanvas):

git-svn-id: ​https://svn.webkit.org/repository/webkit/trunk@268398 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Location:

branches/safari-610-branch/Source/WebCore

Files:

• ChangeLog (modified) (1 diff)
• html/HTMLVideoElement.cpp (modified) (1 diff)
• platform/graphics/MediaPlayer.cpp (modified) (1 diff)
• platform/graphics/MediaPlayer.h (modified) (1 diff)
• platform/graphics/MediaPlayerPrivate.h (modified) (1 diff)
• platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h (modified) (1 diff)
• platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm (modified) (2 diffs)

branches/safari-610-branch/Source/WebCore/ChangeLog

@@ +1 @@
+2020-10-21  Russell Epstein  <repstein@apple.com>
+
+        Cherry-pick r268398. rdar://problem/70541902
+
+    Webcam video from navigator.mediaDevices.getUserMedia() to 2D canvas fails on Safari on iPhone
+    https://bugs.webkit.org/show_bug.cgi?id=217578
+    <rdar://problem/70183875>
+    
+    Reviewed by Eric Carlson.
+    
+    It is unneeded for MediaStream video backends to enqueue samples to the display layer if the video element is hidden.
+    In iOS, the samples may never be flushed which might cause capture failing after enqueuing too many camera samples.
+    To avoid that, we no longer enqueue samples to the display layer when the player is not visible.
+    In case of canvas painting, other backends need to be made visible for canvas painting to work.
+    For MediaStream backend, we do not need that since we always keep the last sample.
+    For that reason, we keep the backend as not visible even if canvas happens.
+    We do so by introducing a setVisibleForCanvas that is a no-op for MediaStream backend and similar to setVisible for other backends.
+    For good measure, we now flush the MediaStream display layer whenever visibility changed.
+    
+    Manually tested.
+    
+    * html/HTMLVideoElement.cpp:
+    (WebCore::HTMLVideoElement::paintCurrentFrameInContext):
+    * platform/graphics/MediaPlayer.cpp:
+    (WebCore::MediaPlayer::setVisibleForCanvas):
+    * platform/graphics/MediaPlayer.h:
+    * platform/graphics/MediaPlayerPrivate.h:
+    (WebCore::MediaPlayerPrivateInterface::setVisibleForCanvas):
+    * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
+    * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
+    (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::enqueueVideoSample):
+    (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setVisible):
+    (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setVisibleForCanvas):
+    
+    git-svn-id: https://svn.webkit.org/repository/webkit/trunk@268398 268f45cc-cd09-0410-ab3c-d52691b4dbfc
+
+    2020-10-13  Youenn Fablet  <youenn@apple.com>
+
+            Webcam video from navigator.mediaDevices.getUserMedia() to 2D canvas fails on Safari on iPhone
+            https://bugs.webkit.org/show_bug.cgi?id=217578
+            <rdar://problem/70183875>
+
+            Reviewed by Eric Carlson.
+
+            It is unneeded for MediaStream video backends to enqueue samples to the display layer if the video element is hidden.
+            In iOS, the samples may never be flushed which might cause capture failing after enqueuing too many camera samples.
+            To avoid that, we no longer enqueue samples to the display layer when the player is not visible.
+            In case of canvas painting, other backends need to be made visible for canvas painting to work.
+            For MediaStream backend, we do not need that since we always keep the last sample.
+            For that reason, we keep the backend as not visible even if canvas happens.
+            We do so by introducing a setVisibleForCanvas that is a no-op for MediaStream backend and similar to setVisible for other backends.
+            For good measure, we now flush the MediaStream display layer whenever visibility changed.
+
+            Manually tested.
+
+            * html/HTMLVideoElement.cpp:
+            (WebCore::HTMLVideoElement::paintCurrentFrameInContext):
+            * platform/graphics/MediaPlayer.cpp:
+            (WebCore::MediaPlayer::setVisibleForCanvas):
+            * platform/graphics/MediaPlayer.h:
+            * platform/graphics/MediaPlayerPrivate.h:
+            (WebCore::MediaPlayerPrivateInterface::setVisibleForCanvas):
+            * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
+            * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
+            (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::enqueueVideoSample):
+            (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setVisible):
+            (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setVisibleForCanvas):
+
 2020-10-21  Russell Epstein  <repstein@apple.com>
 

branches/safari-610-branch/Source/WebCore/html/HTMLVideoElement.cpp

@@ -297 +297 @@
         return;
     
-    player->setVisible(true); // Make player visible or it won't draw.
+    player->setVisibleForCanvas(true); // Make player visible or it won't draw.
     player->paintCurrentFrameInContext(context, destRect);
 }

branches/safari-610-branch/Source/WebCore/platform/graphics/MediaPlayer.cpp

@@ -952 +952 @@
 }
 
+void MediaPlayer::setVisibleForCanvas(bool visible)
+{
+    m_visible = visible;
+    m_private->setVisibleForCanvas(visible);
+}
+
 MediaPlayer::Preload MediaPlayer::preload() const
 {

branches/safari-610-branch/Source/WebCore/platform/graphics/MediaPlayer.h

@@ -345 +345 @@
     bool visible() const;
     void setVisible(bool);
+    void setVisibleForCanvas(bool);
 
     void prepareToPlay();

branches/safari-610-branch/Source/WebCore/platform/graphics/MediaPlayerPrivate.h

@@ -96 +96 @@
 
     virtual void setVisible(bool) = 0;
+    virtual void setVisibleForCanvas(bool visible) { setVisible(visible); }
 
     virtual float duration() const { return 0; }

branches/safari-610-branch/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h

@@ -125 +125 @@
 
     void setVisible(bool) final;
+    void setVisibleForCanvas(bool) final;
 
     MediaTime durationMediaTime() const override;

branches/safari-610-branch/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm

@@ -253 +253 @@
 void MediaPlayerPrivateMediaStreamAVFObjC::enqueueVideoSample(MediaSample& sample)
 {
+    if (!m_visible)
+        return;
+
     auto locker = tryHoldLock(m_sampleBufferDisplayLayerLock);
     if (!locker)
@@ -594 +597 @@
 
     m_visible = visible;
-    if (m_visible)
-        flushRenderers();
+    flushRenderers();
+}
+
+void MediaPlayerPrivateMediaStreamAVFObjC::setVisibleForCanvas(bool)
+{
 }