⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 280720 in webkit


Ignore:
Timestamp:
Aug 6, 2021, 2:15:49 AM (5 years ago)
Author:
youenn@apple.com
Message:

MediaPlayerPrivateMediaStreamAVFObjC should skip enqueuing frames when not visible
​https://bugs.webkit.org/show_bug.cgi?id=228816
<rdar://81077972>

Reviewed by Eric Carlson.

Source/WebCore:

Do not create layers when not needed.
This prevents enqueuing frames in a AVSampleBufferDisplayLayer that will only buffer them, thus breaking camera capture/video decoding.
We do this by not calling ensureLayers when getting a new track. Instead we react upon player renderingCanBeAccelerated value.

It is also unnecessary and takes CPU cycles to enqueue frames when the video element is not visible.
HTMLMediaElement and RenderVideo thus pass to MediaPlayer a new flag telling whether the video element is visible in the view port.
MediaPlayerPrivateMediaStreamAVFObjC will then skip enqueueing frames if not needed.

Add getter and internals API to cover these changes.

Test: fast/mediastream/mediaPlayer-visibility.html

  • html/HTMLMediaElement.cpp:

(WebCore::HTMLMediaElement::isVisibleInViewportChanged):

  • platform/graphics/MediaPlayer.cpp:

(WebCore::MediaPlayer::loadWithNextMediaEngine):
(WebCore::MediaPlayer::setVisibleInViewport):

  • platform/graphics/MediaPlayer.h:
  • platform/graphics/MediaPlayerPrivate.h:

(WebCore::MediaPlayerPrivateInterface::setVisibleInViewport):

  • platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
  • platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:

(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::enqueueVideoSample):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setVisible):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setVisibleInViewport):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::checkSelectedVideoTrack):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::paintCurrentFrameInContext):

  • rendering/RenderVideo.cpp:

(WebCore::RenderVideo::updatePlayer):

  • testing/Internals.cpp:

(WebCore::Internals::isPlayerVisibleInViewport const):

  • testing/Internals.h:
  • testing/Internals.idl:

LayoutTests:

  • fast/mediastream/mediaPlayer-visibility-expected.txt: Added.
  • fast/mediastream/mediaPlayer-visibility.html: Added.
Location:
trunk
Files:
2 added
12 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r280719 r280720  
     12021-08-06  Youenn Fablet  <youenn@apple.com>
     2
     3        MediaPlayerPrivateMediaStreamAVFObjC should skip enqueuing frames when not visible
     4        https://bugs.webkit.org/show_bug.cgi?id=228816
     5        <rdar://81077972>
     6
     7        Reviewed by Eric Carlson.
     8
     9        * fast/mediastream/mediaPlayer-visibility-expected.txt: Added.
     10        * fast/mediastream/mediaPlayer-visibility.html: Added.
     11
    1122021-08-05  Ryan Haddad  <ryanhaddad@apple.com>
    213

  • trunk/Source/WebCore/ChangeLog

    r280718 r280720  
     12021-08-06  Youenn Fablet  <youenn@apple.com>
     2
     3        MediaPlayerPrivateMediaStreamAVFObjC should skip enqueuing frames when not visible
     4        https://bugs.webkit.org/show_bug.cgi?id=228816
     5        <rdar://81077972>
     6
     7        Reviewed by Eric Carlson.
     8
     9        Do not create layers when not needed.
     10        This prevents enqueuing frames in a AVSampleBufferDisplayLayer that will only buffer them, thus breaking camera capture/video decoding.
     11        We do this by not calling ensureLayers when getting a new track. Instead we react upon player renderingCanBeAccelerated value.
     12
     13        It is also unnecessary and takes CPU cycles to enqueue frames when the video element is not visible.
     14        HTMLMediaElement and RenderVideo thus pass to MediaPlayer a new flag telling whether the video element is visible in the view port.
     15        MediaPlayerPrivateMediaStreamAVFObjC will then skip enqueueing frames if not needed.
     16
     17        Add getter and internals API to cover these changes.
     18
     19        Test: fast/mediastream/mediaPlayer-visibility.html
     20
     21        * html/HTMLMediaElement.cpp:
     22        (WebCore::HTMLMediaElement::isVisibleInViewportChanged):
     23        * platform/graphics/MediaPlayer.cpp:
     24        (WebCore::MediaPlayer::loadWithNextMediaEngine):
     25        (WebCore::MediaPlayer::setVisibleInViewport):
     26        * platform/graphics/MediaPlayer.h:
     27        * platform/graphics/MediaPlayerPrivate.h:
     28        (WebCore::MediaPlayerPrivateInterface::setVisibleInViewport):
     29        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
     30        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
     31        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::enqueueVideoSample):
     32        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setVisible):
     33        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setVisibleInViewport):
     34        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::checkSelectedVideoTrack):
     35        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::paintCurrentFrameInContext):
     36        * rendering/RenderVideo.cpp:
     37        (WebCore::RenderVideo::updatePlayer):
     38        * testing/Internals.cpp:
     39        (WebCore::Internals::isPlayerVisibleInViewport const):
     40        * testing/Internals.h:
     41        * testing/Internals.idl:
     42
    1432021-08-05  Cameron McCormack  <heycam@apple.com>
    244

  • trunk/Source/WebCore/html/HTMLMediaElement.cpp

    r280664 r280720  
    67456745    m_player->setMuted(effectiveMuted());
    67466746    m_player->setVisible(!m_elementIsHidden);
     6747    m_player->setVisibleInViewport(isVisibleInViewport());
    67476748    schedulePlaybackControlsManagerUpdate();
    67486749
    … …  
    80728073void HTMLMediaElement::isVisibleInViewportChanged()
    80738074{
     8075    if (m_player)
     8076        m_player->setVisibleInViewport(isVisibleInViewport());
    80748077    queueTaskKeepingObjectAlive(*this, TaskSource::MediaElement, [this] {
    80758078        if (isContextStopped())

  • trunk/Source/WebCore/platform/graphics/MediaPlayer.cpp

    r280308 r280720  
    576576            if (m_visible)
    577577                m_private->setVisible(m_visible);
     578            if (m_visibleInViewport)
     579                m_private->setVisibleInViewport(m_visibleInViewport);
    578580            m_private->prepareForPlayback(m_privateBrowsing, m_preload, m_preservesPitch, m_shouldPrepareToRender);
    579581        }
    … …  
    10101012{
    10111013    m_private->setVisibleForCanvas(visible);
     1014}
     1015
     1016void MediaPlayer::setVisibleInViewport(bool visible)
     1017{
     1018    m_visibleInViewport = visible;
     1019    m_private->setVisibleInViewport(visible);
    10121020}
    10131021

  • trunk/Source/WebCore/platform/graphics/MediaPlayer.h

    r280308 r280720  
    361361    void setVisibleForCanvas(bool);
    362362
     363    void setVisibleInViewport(bool);
     364    bool isVisibleInViewport() const { return m_visibleInViewport; }
     365
    363366    void prepareToPlay();
    364367    void play();
    … …  
    700703    double m_volume { 1 };
    701704    bool m_visible { false };
     705    bool m_visibleInViewport { false };
    702706    bool m_muted { false };
    703707    bool m_preservesPitch { true };

  • trunk/Source/WebCore/platform/graphics/MediaPlayerPrivate.h

    r280308 r280720  
    9999    virtual void setVisible(bool) = 0;
    100100    virtual void setVisibleForCanvas(bool visible) { setVisible(visible); }
     101    virtual void setVisibleInViewport(bool) { }
    101102
    102103    virtual float duration() const { return 0; }

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h

    r278253 r280720  
    123123    void setVisible(bool) final;
    124124    void setVisibleForCanvas(bool) final;
     125    void setVisibleInViewport(bool) final;
    125126
    126127    MediaTime durationMediaTime() const override;
    … …  
    144145    bool metaDataAvailable() const { return m_mediaStreamPrivate && m_readyState >= MediaPlayer::ReadyState::HaveMetadata; }
    145146
    146     void acceleratedRenderingStateChanged() override;
     147    void acceleratedRenderingStateChanged() final { updateLayersAsNeeded(); }
    147148    bool supportsAcceleratedRendering() const override { return true; }
    148149
    … …  
    171172
    172173    void layersAreInitialized(IntSize, bool);
     174    void updateLayersAsNeeded();
    173175
    174176    enum DisplayMode {
    … …  
    273275    bool m_ended { false };
    274276    bool m_hasEverEnqueuedVideoFrame { false };
    275     bool m_visible { false };
     277    bool m_isPageVisible { false };
     278    bool m_isVisibleInViewPort { false };
    276279    bool m_haveSeenMetadata { false };
    277280    bool m_waitingForFirstImage { false };

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm

    r280518 r280720  
    253253void MediaPlayerPrivateMediaStreamAVFObjC::enqueueVideoSample(MediaSample& sample)
    254254{
    255     if (!m_visible)
     255    if (!m_isPageVisible || !m_isVisibleInViewPort)
    256256        return;
    257257
    … …  
    615615}
    616616
    617 void MediaPlayerPrivateMediaStreamAVFObjC::setVisible(bool visible)
    618 {
    619     if (m_visible == visible)
    620         return;
    621 
    622     m_visible = visible;
     617void MediaPlayerPrivateMediaStreamAVFObjC::setVisible(bool isVisible)
     618{
     619    if (m_isPageVisible == isVisible)
     620        return;
     621
     622    m_isPageVisible = isVisible;
    623623    flushRenderers();
    624624}
    … …  
    626626void MediaPlayerPrivateMediaStreamAVFObjC::setVisibleForCanvas(bool)
    627627{
     628}
     629
     630void MediaPlayerPrivateMediaStreamAVFObjC::setVisibleInViewport(bool isVisible)
     631{
     632    m_isVisibleInViewPort = isVisible;
    628633}
    629634
    … …  
    862867            m_waitingForFirstImage = true;
    863868    }
    864     ensureLayers();
     869
     870    updateLayersAsNeeded();
     871
    865872    if (m_sampleBufferDisplayLayer) {
    866873        if (!m_activeVideoTrack)
    … …  
    9991006}
    10001007
    1001 void MediaPlayerPrivateMediaStreamAVFObjC::acceleratedRenderingStateChanged()
     1008void MediaPlayerPrivateMediaStreamAVFObjC::updateLayersAsNeeded()
    10021009{
    10031010    if (m_player->renderingCanBeAccelerated())

  • trunk/Source/WebCore/rendering/RenderVideo.cpp

    r277321 r280720  
    283283    mediaPlayer->setSize(IntSize(videoBounds.width(), videoBounds.height()));
    284284    mediaPlayer->setVisible(!videoElement().elementIsHidden());
     285    mediaPlayer->setVisibleInViewport(videoElement().isVisibleInViewport());
    285286    mediaPlayer->setShouldMaintainAspectRatio(style().objectFit() != ObjectFit::Fill);
    286287}

  • trunk/Source/WebCore/testing/Internals.cpp

    r280715 r280720  
    44214421    return element.isDisablingSleep();
    44224422}
     4423
     4424bool Internals::isPlayerVisibleInViewport(HTMLMediaElement& element) const
     4425{
     4426    auto player = element.player();
     4427    return player && player->isVisibleInViewport();
     4428}
     4429
    44234430#endif // ENABLE(VIDEO)
    44244431

  • trunk/Source/WebCore/testing/Internals.h

    r280715 r280720  
    714714    void activeAudioRouteDidChange(bool shouldPause);
    715715    bool elementIsBlockingDisplaySleep(HTMLMediaElement&) const;
     716    bool isPlayerVisibleInViewport(HTMLMediaElement&) const;
    716717#endif
    717718

  • trunk/Source/WebCore/testing/Internals.idl

    r280715 r280720  
    774774    [Conditional=VIDEO] undefined simulateSystemWake();
    775775    [Conditional=VIDEO] boolean elementIsBlockingDisplaySleep(HTMLMediaElement element);
     776    [Conditional=VIDEO] boolean isPlayerVisibleInViewport(HTMLMediaElement element);
    776777
    777778    MockPageOverlay installMockPageOverlay(PageOverlayType type);
Note: See TracChangeset for help on using the changeset viewer.