⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 285993 in webkit


Ignore:
Timestamp:
Nov 18, 2021, 2:34:22 AM (5 years ago)
Author:
youenn@apple.com
Message:

MediaPlayerAVFoundation should support rvfc
​https://bugs.webkit.org/show_bug.cgi?id=231945
LayoutTests/imported/w3c:

<rdar://problem/84657372>

Reviewed by Eric Carlson.

Transforming tests from async to promise based so that there is only one video playing at a time, to make it work smoothly in iOS.

  • web-platform-tests/video-rvfc/request-video-frame-callback-dom-expected.txt:
  • web-platform-tests/video-rvfc/request-video-frame-callback-dom.html:
  • web-platform-tests/video-rvfc/request-video-frame-callback-parallel-expected.txt:
  • web-platform-tests/video-rvfc/request-video-frame-callback-parallel.html:
  • web-platform-tests/video-rvfc/request-video-frame-callback-repeating-expected.txt:
  • web-platform-tests/video-rvfc/request-video-frame-callback-repeating.html:
  • web-platform-tests/video-rvfc/request-video-frame-callback-expected.txt:
  • web-platform-tests/video-rvfc/request-video-frame-callback.html:

Source/WebCore:

<rdar://problem/84657372>

Reviewed by Eric Carlson.

Add API to start/stop gathering metadata.
This goes from video element down to media player private.
Conversely, add a callback to get the metadata, including the pixel buffer.

Implement support in MediaPlayerPrivateAVFoundationObjC by adding a specific timeline observer.
We reuse the same video output and the same pixel buffer as when painting the video element in the canvas.
To make sure painting in a canvas does not have side effects with gathering video metadata, we do not update the pixel buffer
when painting on a canvas in case video metadata is gathered.
Instead, we consider that pixel buffer is as up to date as possible in that case.
To make it work in WK1, we implement the synchronous video frame callback getter.

Round the millisecond now time given to the callback, like done for RAF.

Covered by enabled tests.

  • WebCore.xcodeproj/project.pbxproj:
  • html/HTMLMediaElement.h:
  • html/HTMLVideoElement.cpp:
  • html/HTMLVideoElement.h:
  • platform/VideoFrameMetadata.h:
  • platform/cocoa/CVPixelBufferRef.h: Added.
  • platform/graphics/MediaPlayer.cpp:
  • platform/graphics/MediaPlayer.h:
  • platform/graphics/MediaPlayerPrivate.h:
  • platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.h:
  • platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:
  • platform/graphics/cv/PixelBufferConformerCV.h:

Source/WebKit:

<rdar://problem/84657372>

Reviewed by Eric Carlson.

Add IPC messaging to start/stop metadata gathering from WebProcess to GPUProcess.
Add IPC messaging to push metadata to WebProcess, including sending the corresponding CVPixelBuffer for the current metadata.
MediaPlayerPrivateRemote will then serve that buffer when being requested the current image.
This ensures that painting on a canvas does not mark the current buffer as acquired, which would then skip the video frame metadata gathering.
This also ensures the same buffer is used if a video element is painted several times during the same rvfc callback.
This has the side benefit of removing the synchronous IPC to paint the video element in a canvas.

  • GPUProcess/media/RemoteMediaPlayerProxy.cpp:
  • GPUProcess/media/RemoteMediaPlayerProxy.h:
  • GPUProcess/media/RemoteMediaPlayerProxy.messages.in:
  • GPUProcess/media/cocoa/RemoteMediaPlayerProxyCocoa.mm:
  • Scripts/webkit/messages.py:
  • WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp:
  • WebProcess/GPU/media/MediaPlayerPrivateRemote.h:
  • WebProcess/GPU/media/MediaPlayerPrivateRemote.messages.in:
  • WebProcess/GPU/media/cocoa/MediaPlayerPrivateRemoteCocoa.mm:

Tools:

Reviewed by Eric Carlson.

Enable rvfc by default in test runner.

  • DumpRenderTree/mac/DumpRenderTree.mm:
  • WebKitTestRunner/TestController.cpp:

LayoutTests:

Reviewed by Eric Carlson.

Unskipped some tests.

  • platform/ios-wk2/TestExpectations:
  • platform/mac-wk1/TestExpectations:
  • platform/mac-wk2/TestExpectations:
Location:
trunk
Files:
37 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r285989 r285993  
     12021-11-18  Youenn Fablet  <youenn@apple.com>
     2
     3        MediaPlayerAVFoundation should support rvfc
     4        https://bugs.webkit.org/show_bug.cgi?id=231945
     5
     6        Reviewed by Eric Carlson.
     7
     8        Unskipped some tests.
     9
     10        * platform/ios-wk2/TestExpectations:
     11        * platform/mac-wk1/TestExpectations:
     12        * platform/mac-wk2/TestExpectations:
     13
    1142021-11-18  Sergio Villar Senin  <svillar@igalia.com>
    215

  • trunk/LayoutTests/imported/w3c/ChangeLog

    r285989 r285993  
     12021-11-18  Youenn Fablet  <youenn@apple.com>
     2
     3        MediaPlayerAVFoundation should support rvfc
     4        https://bugs.webkit.org/show_bug.cgi?id=231945
     5        <rdar://problem/84657372>
     6
     7        Reviewed by Eric Carlson.
     8
     9        Transforming tests from async to promise based so that there is only one video playing at a time, to make it work smoothly in iOS.
     10
     11        * web-platform-tests/video-rvfc/request-video-frame-callback-dom-expected.txt:
     12        * web-platform-tests/video-rvfc/request-video-frame-callback-dom.html:
     13        * web-platform-tests/video-rvfc/request-video-frame-callback-parallel-expected.txt:
     14        * web-platform-tests/video-rvfc/request-video-frame-callback-parallel.html:
     15        * web-platform-tests/video-rvfc/request-video-frame-callback-repeating-expected.txt:
     16        * web-platform-tests/video-rvfc/request-video-frame-callback-repeating.html:
     17        * web-platform-tests/video-rvfc/request-video-frame-callback-expected.txt:
     18        * web-platform-tests/video-rvfc/request-video-frame-callback.html:
     19
    1202021-11-18  Sergio Villar Senin  <svillar@igalia.com>
    221

  • trunk/LayoutTests/imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-dom-expected.txt

    r284528 r285993  
    11
    2 Harness Error (TIMEOUT), message = null
     2PASS Test a video outside of the DOM can still use video.rVFC.
     3PASS Test video.rVFC works with "display:none".
     4PASS Test video.rVFC works with "visibility:hidden".
    35
    4 TIMEOUT Test a video outside of the DOM can still use video.rVFC. Test timed out
    5 TIMEOUT Test video.rVFC works with "display:none". Test timed out
    6 TIMEOUT Test video.rVFC works with "visibility:hidden". Test timed out
    7 

  • trunk/LayoutTests/imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-dom.html

    r284528 r285993  
    1414}
    1515
    16 async_test(function(t) {
     16promise_test(async function(t) {
     17    let done;
     18    const promise = new Promise(resolve => done = resolve);
     19
    1720    let video = document.createElement('video');
    1821
    19     video.requestVideoFrameCallback(t.step_func_done());
     22    video.requestVideoFrameCallback(done);
    2023    video.src = testVideo.url;
    21     video.play();
     24    await video.play();
    2225
     26    return promise;
    2327}, 'Test a video outside of the DOM can still use video.rVFC.');
    2428
    2529function rvfcStyleTest(applyStyle, description) {
    26     async_test(function(t) {
     30    promise_test(async function(t) {
     31      let done;
     32      const promise = new Promise(resolve => done = resolve);
     33
    2734      let video = document.createElement('video');
    2835      document.body.appendChild(video);
    … …  
    3239        t.step_func( _ => {
    3340          // Make sure we can receive more than one callback.
    34           video.requestVideoFrameCallback(t.step_func_done());
     41          video.requestVideoFrameCallback(done);
    3542        })
    3643      );
    3744
    3845      video.src = testVideo.url;
    39       video.play();
     46      await video.play();
     47
     48      return promise;
    4049    }, description);
    4150}

  • trunk/LayoutTests/imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-expected.txt

    r284528 r285993  
    11
    22
    3 Harness Error (TIMEOUT), message = null
     3PASS Test we can register a video.rVFC callback.
     4PASS Test video.rVFC callbacks run before window.rAF callbacks.
     5PASS Test we can cancel a video.rVFC request.
     6PASS Test invalid calls to the video.rVFC API.
     7PASS Test video.rVFC does not stop when switching sources.
    48
    5 TIMEOUT Test we can register a video.rVFC callback. Test timed out
    6 TIMEOUT Test video.rVFC callbacks run before window.rAF callbacks. Test timed out
    7 TIMEOUT Test we can cancel a video.rVFC request. Test timed out
    8 PASS Test invalid calls to the video.rVFC API.
    9 TIMEOUT Test video.rVFC does not stop when switching sources. Test timed out
    10 

  • trunk/LayoutTests/imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-parallel-expected.txt

    r284528 r285993  
    11
    22
    3 Harness Error (TIMEOUT), message = null
     3PASS Test callbacks get the same information.
     4PASS Test we can cancel callbacks from callbacks.
    45
    5 TIMEOUT Test callbacks get the same information. Test timed out
    6 TIMEOUT Test we can cancel callbacks from callbacks. Test timed out
    7 

  • trunk/LayoutTests/imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-parallel.html

    r284528 r285993  
    88<script>
    99
    10 async_test(function(t) {
     10promise_test(async function(t) {
     11    let done;
     12    const promise = new Promise(resolve => done = resolve);
     13
    1114    let video = document.createElement('video');
    1215    document.body.appendChild(video);
    … …  
    2023    }));
    2124
    22     video.requestVideoFrameCallback(t.step_func_done((time, metadata) => {
     25    video.requestVideoFrameCallback(t.step_func((time, metadata) => {
    2326      assert_equals(firstTime, time);
    2427      assert_object_equals(firstMetadata, metadata);
     28      done();
    2529    }));
    2630
    … …  
    2832    video.play();
    2933
     34    return promise;
    3035}, 'Test callbacks get the same information.');
    3136
    32 async_test(function(t) {
     37promise_test(async function(t) {
     38    let done;
     39    const promise = new Promise(resolve => done = resolve);
     40
    3341    let video = document.createElement('video');
    3442    document.body.appendChild(video);
    … …  
    4755
    4856    // NOTE: This callback should be executed last.
    49     video.requestVideoFrameCallback(
    50       t.step_func_done()
    51     );
     57    video.requestVideoFrameCallback(done);
    5258
    5359    video.src = getVideoURI('/media/movie_5');
    5460    video.play();
     61
     62    return promise;
    5563}, 'Test we can cancel callbacks from callbacks.');
    5664</script>

  • trunk/LayoutTests/imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-repeating-expected.txt

    r284528 r285993  
    11
    22
    3 Harness Error (TIMEOUT), message = null
     3PASS Test new callbacks are only called on the next frame.
     4PASS Test chaining calls to video.rVFC, and verify the required parameters.
    45
    5 TIMEOUT Test new callbacks are only called on the next frame. Test timed out
    6 TIMEOUT Test chaining calls to video.rVFC, and verify the required parameters. Test timed out
    7 

  • trunk/LayoutTests/imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-repeating.html

    r284528 r285993  
    88<script>
    99
    10 async_test(function(t) {
     10promise_test(async function(t) {
     11    let done;
     12    const promise = new Promise(resolve => done = resolve);
     13
    1114    let video = document.createElement('video');
    1215    document.body.appendChild(video);
    … …  
    2528      // Queue up a second callback, and make sure it's called at the same time
    2629      // as the one we just queued up.
    27       video.requestVideoFrameCallback(t.step_func_done((time) => {
     30      video.requestVideoFrameCallback(t.step_func((time) => {
    2831        assert_equals(time, secondTime, "Callbacks queued together should be called at the same time");
     32        done();
    2933      }))
    3034
    … …  
    3236
    3337    video.src = getVideoURI('/media/movie_5');
    34     video.play();
     38    await video.play();
    3539
     40    return promise;
    3641}, 'Test new callbacks are only called on the next frame.');
    3742
    38 async_test(function(t) {
     43promise_test(async function(t) {
     44    let done;
     45    const promise = new Promise(resolve => done = resolve);
     46
    3947    let video = document.createElement('video');
    4048    document.body.appendChild(video);
    … …  
    6573
    6674      if (++currentCallNumber > maxNumberOfCalls) {
    67         t.done()
     75        done()
    6876      } else {
    6977        video.requestVideoFrameCallback(t.step_func(repeatingCallback));
    … …  
    7482
    7583    video.src = getVideoURI('/media/movie_5');
    76     video.play();
     84    await video.play();
    7785
     86    return promise;
    7887}, 'Test chaining calls to video.rVFC, and verify the required parameters.');
    7988</script>

  • trunk/LayoutTests/imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback.html

    r284528 r285993  
    1919}
    2020
    21 async_test(function(t) {
     21promise_test(async function(t) {
     22    let done;
     23    const promise = new Promise(resolve => done = resolve);
     24
    2225    let video = document.createElement('video');
    2326    document.body.appendChild(video);
    2427
    2528    let id = video.requestVideoFrameCallback(
    26       t.step_func_done((time, metadata) => {
     29      t.step_func((time, metadata) => {
    2730        assert_true(time > 0);
    2831        assert_equals(metadata.height, testVideo.height);
    2932        assert_equals(metadata.width, testVideo.width);
     33        done();
    3034      })
    3135    );
    … …  
    3438
    3539    video.src = testVideo.url;
    36     video.play();
     40    await video.play();
    3741
     42    return promise;
    3843}, 'Test we can register a video.rVFC callback.');
    3944
    40 async_test(function(t) {
     45promise_test(async function(t) {
     46    let done;
     47    const promise = new Promise(resolve => done = resolve);
     48
    4149    let video = document.createElement('video');
    4250    document.body.appendChild(video);
    … …  
    4654        // Queue a call to window.rAF, and make sure it is executed within the
    4755        // same turn of the event loop (with the same 'time' parameter).
    48         window.requestAnimationFrame( t.step_func_done( window_now => {
     56        window.requestAnimationFrame( t.step_func( window_now => {
    4957          assert_equals(video_now, window_now);
     58          done();
    5059        }));
    5160      })
    … …  
    5362
    5463    video.src = testVideo.url;
    55     video.play();
     64    await video.play();
    5665
     66    return promise;
    5767}, 'Test video.rVFC callbacks run before window.rAF callbacks.');
    5868
    5969
    60 async_test(function(t) {
     70promise_test(async function(t) {
     71    let done;
     72    const promise = new Promise(resolve => done = resolve);
     73
    6174    let video = document.createElement('video');
    6275    document.body.appendChild(video);
    … …  
    7588        // give it some more time and really make sure it doesn't, by going
    7689        // throught the event loop once more.
    77         t.step_timeout(() => { t.done(); }, 0);
     90        t.step_timeout(() => { done(); }, 0);
    7891      })
    7992    );
    8093
    8194    video.src = testVideo.url;
    82     video.play();
     95    await video.play();
     96
     97    return promise;
    8398}, 'Test we can cancel a video.rVFC request.');
    8499
    … …  
    106121promise_test(async function(t) {
    107122    let video = document.createElement('video');
     123    video.autoplay = true;
    108124    document.body.appendChild(video);
    109125

  • trunk/LayoutTests/platform/ios-wk2/TestExpectations

    r285565 r285993  
    144144fast/mediastream/getUserMedia-rvfc.html [ Pass ]
    145145webrtc/peerConnection-rvfc.html [ Pass ]
    146 # Timing out tests until we add regular video streaming backend support.
     146# Timing out tests until we add XR session.
    147147imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-before-xr-session.https.html [ Skip ]
    148 imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-dom.html [ Skip ]
    149148imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-during-xr-session.https.html [ Skip ]
    150 imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-parallel.html [ Skip ]
    151 imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-repeating.html [ Skip ]
    152 imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback.html [ Skip ]
    153149
    154150#//////////////////////////////////////////////////////////////////////////////////////////

  • trunk/LayoutTests/platform/mac-wk1/TestExpectations

    r285936 r285993  
    3333# rdar://problem/61793884 : Enabling this test for WK1 only.
    3434fast/shapes/shape-outside-floats/shape-outside-imagedata-overflow.html [ Pass ]
     35
     36imported/w3c/web-platform-tests/video-rvfc [ Pass ]
     37# Timing out tests until we add XR session.
     38imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-before-xr-session.https.html [ Skip ]
     39imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-during-xr-session.https.html [ Skip ]
    3540
    3641#//////////////////////////////////////////////////////////////////////////////////////////

  • trunk/LayoutTests/platform/mac-wk2/TestExpectations

    r285928 r285993  
    123123fast/mediastream/getUserMedia-rvfc.html [ Pass ]
    124124webrtc/peerConnection-rvfc.html [ Pass ]
    125 # Timing out tests until we add regular video streaming backend support.
     125# Timing out tests until we add XR session.
    126126imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-before-xr-session.https.html [ Skip ]
    127 imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-dom.html [ Skip ]
    128127imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-during-xr-session.https.html [ Skip ]
    129 imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-parallel.html [ Skip ]
    130 imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-repeating.html [ Skip ]
    131 imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback.html [ Skip ]
    132128
    133129#//////////////////////////////////////////////////////////////////////////////////////////

  • trunk/Source/WebCore/ChangeLog

    r285992 r285993  
     12021-11-18  Youenn Fablet  <youenn@apple.com>
     2
     3        MediaPlayerAVFoundation should support rvfc
     4        https://bugs.webkit.org/show_bug.cgi?id=231945
     5        <rdar://problem/84657372>
     6
     7        Reviewed by Eric Carlson.
     8
     9        Add API to start/stop gathering metadata.
     10        This goes from video element down to media player private.
     11        Conversely, add a callback to get the metadata, including the pixel buffer.
     12
     13        Implement support in MediaPlayerPrivateAVFoundationObjC by adding a specific timeline observer.
     14        We reuse the same video output and the same pixel buffer as when painting the video element in the canvas.
     15        To make sure painting in a canvas does not have side effects with gathering video metadata, we do not update the pixel buffer
     16        when painting on a canvas in case video metadata is gathered.
     17        Instead, we consider that pixel buffer is as up to date as possible in that case.
     18        To make it work in WK1, we implement the synchronous video frame callback getter.
     19
     20        Round the millisecond now time given to the callback, like done for RAF.
     21
     22        Covered by enabled tests.
     23
     24        * WebCore.xcodeproj/project.pbxproj:
     25        * html/HTMLMediaElement.h:
     26        * html/HTMLVideoElement.cpp:
     27        * html/HTMLVideoElement.h:
     28        * platform/VideoFrameMetadata.h:
     29        * platform/cocoa/CVPixelBufferRef.h: Added.
     30        * platform/graphics/MediaPlayer.cpp:
     31        * platform/graphics/MediaPlayer.h:
     32        * platform/graphics/MediaPlayerPrivate.h:
     33        * platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.h:
     34        * platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:
     35        * platform/graphics/cv/PixelBufferConformerCV.h:
     36
    1372021-11-18  Chris Lord  <clord@igalia.com>
    238

  • trunk/Source/WebCore/html/HTMLMediaElement.h

    r285571 r285993  
    642642    bool isChangingVideoFullscreenMode() const { return m_changingVideoFullscreenMode; }
    643643
     644protected:
     645    void mediaPlayerEngineUpdated() override;
     646
    644647private:
    645648    friend class Internals;
    … …  
    693696    void mediaPlayerRenderingModeChanged() final;
    694697    bool mediaPlayerAcceleratedCompositingEnabled() final;
    695     void mediaPlayerEngineUpdated() final;
    696698    void mediaPlayerWillInitializeMediaEngine() final;
    697699    void mediaPlayerDidInitializeMediaEngine() final;

  • trunk/Source/WebCore/html/HTMLVideoElement.cpp

    r285984 r285993  
    590590unsigned HTMLVideoElement::requestVideoFrameCallback(Ref<VideoFrameRequestCallback>&& callback)
    591591{
     592    if (m_videoFrameRequests.isEmpty() && player())
     593        player()->startVideoFrameMetadataGathering();
     594
    592595    auto identifier = ++m_nextVideoFrameRequestIndex;
    593596    m_videoFrameRequests.append(makeUniqueRef<VideoFrameRequest>(identifier, WTFMove(callback)));
    … …  
    609612    }
    610613    m_videoFrameRequests.remove(index);
     614
     615    if (m_videoFrameRequests.isEmpty() && player())
     616        player()->stopVideoFrameMetadataGathering();
    611617}
    612618
    … …  
    641647        auto& request = m_videoFrameRequests[index];
    642648        if (!request->cancelled) {
    643             request->callback->handleEvent(now.value(), *videoFrameMetadata);
     649            request->callback->handleEvent(std::round(now.milliseconds()), *videoFrameMetadata);
    644650            request->cancelled = true;
    645651        }
    … …  
    648654
    649655    m_videoFrameRequests.removeAllMatching([](auto& callback) { return callback->cancelled; });
    650 }
    651 
    652 }
    653 
    654 #endif
     656
     657    if (m_videoFrameRequests.isEmpty() && player())
     658        player()->stopVideoFrameMetadataGathering();
     659}
     660
     661void HTMLVideoElement::mediaPlayerEngineUpdated()
     662{
     663    HTMLMediaElement::mediaPlayerEngineUpdated();
     664    if (!m_videoFrameRequests.isEmpty() && player())
     665        player()->startVideoFrameMetadataGathering();
     666}
     667
     668}
     669
     670#endif

  • trunk/Source/WebCore/html/HTMLVideoElement.h

    r284528 r285993  
    142142    PlatformMediaSession::MediaType presentationType() const final { return PlatformMediaSession::MediaType::Video; }
    143143
     144    void mediaPlayerEngineUpdated() final;
     145
    144146    std::unique_ptr<HTMLImageLoader> m_imageLoader;
    145147

  • trunk/Source/WebCore/platform/VideoFrameMetadata.h

    r284528 r285993  
    4444    std::optional<double> receiveTime;
    4545    std::optional<unsigned> rtpTimestamp;
     46
     47    template<class Encoder> void encode(Encoder&) const;
     48    template<class Decoder> static std::optional<VideoFrameMetadata> decode(Decoder&);
    4649};
     50
     51template<class Encoder>
     52inline void VideoFrameMetadata::encode(Encoder& encoder) const
     53{
     54    encoder << presentationTime << expectedDisplayTime << width << height << mediaTime << presentedFrames << processingDuration << captureTime << receiveTime << rtpTimestamp;
     55}
     56
     57template<class Decoder>
     58inline std::optional<VideoFrameMetadata> VideoFrameMetadata::decode(Decoder& decoder)
     59{
     60    std::optional<double> presentationTime;
     61    decoder >> presentationTime;
     62    if (!presentationTime)
     63        return std::nullopt;
     64
     65    std::optional<double> expectedDisplayTime;
     66    decoder >> expectedDisplayTime;
     67    if (!expectedDisplayTime)
     68        return std::nullopt;
     69
     70    std::optional<unsigned> width;
     71    decoder >> width;
     72    if (!width)
     73        return std::nullopt;
     74
     75    std::optional<unsigned> height;
     76    decoder >> height;
     77    if (!height)
     78        return std::nullopt;
     79
     80    std::optional<double> mediaTime;
     81    decoder >> mediaTime;
     82    if (!mediaTime)
     83        return std::nullopt;
     84
     85    std::optional<unsigned> presentedFrames;
     86    decoder >> presentedFrames;
     87    if (!presentedFrames)
     88        return std::nullopt;
     89
     90    std::optional<std::optional<double>> processingDuration;
     91    decoder >> processingDuration;
     92    if (!processingDuration)
     93        return std::nullopt;
     94
     95    std::optional<std::optional<double>> captureTime;
     96    decoder >> captureTime;
     97    if (!captureTime)
     98        return std::nullopt;
     99
     100    std::optional<std::optional<double>> receiveTime;
     101    decoder >> receiveTime;
     102    if (!receiveTime)
     103        return std::nullopt;
     104
     105    std::optional<std::optional<unsigned>> rtpTimestamp;
     106    decoder >> rtpTimestamp;
     107    if (!rtpTimestamp)
     108        return std::nullopt;
     109
     110    return VideoFrameMetadata { *presentationTime, *expectedDisplayTime, *width, *height, *mediaTime, *presentedFrames, *processingDuration, *captureTime, *receiveTime, *rtpTimestamp };
     111}
    47112
    48113}

  • trunk/Source/WebCore/platform/graphics/MediaPlayer.cpp

    r284857 r285993  
    591591            if (m_visibleInViewport)
    592592                m_private->setVisibleInViewport(m_visibleInViewport);
     593            if (m_isGatheringVideoFrameMetadata)
     594                m_private->startVideoFrameMetadataGathering();
    593595            m_private->prepareForPlayback(m_privateBrowsing, m_preload, m_preservesPitch, m_shouldPrepareToRender);
    594596        }
    … …  
    17211723    return m_private->videoFrameMetadata();
    17221724}
     1725
     1726void MediaPlayer::startVideoFrameMetadataGathering()
     1727{
     1728    m_isGatheringVideoFrameMetadata = true;
     1729    m_private->startVideoFrameMetadataGathering();
     1730}
     1731
     1732void MediaPlayer::stopVideoFrameMetadataGathering()
     1733{
     1734    m_isGatheringVideoFrameMetadata = false;
     1735    m_private->stopVideoFrameMetadataGathering();
     1736}
     1737
     1738#if PLATFORM(COCOA)
     1739void MediaPlayer::onNewVideoFrameMetadata(VideoFrameMetadata&& metadata, RetainPtr<CVPixelBufferRef>&& buffer)
     1740{
     1741    client().mediaPlayerOnNewVideoFrameMetadata(WTFMove(metadata), WTFMove(buffer));
     1742}
     1743#endif
    17231744
    17241745String MediaPlayer::elementId() const

  • trunk/Source/WebCore/platform/graphics/MediaPlayer.h

    r284857 r285993  
    287287    virtual void mediaPlayerQueueTaskOnEventLoop(Function<void()>&& task) { callOnMainThread(WTFMove(task)); }
    288288
     289#if PLATFORM(COCOA)
     290    virtual void mediaPlayerOnNewVideoFrameMetadata(VideoFrameMetadata&&, RetainPtr<CVPixelBufferRef>&&) { }
     291#endif
     292
    289293#if !RELEASE_LOG_DISABLED
    290294    virtual const void* mediaPlayerLogIdentifier() { return nullptr; }
    … …  
    584588    void removeTextTrack(InbandTextTrackPrivate&);
    585589    void removeVideoTrack(VideoTrackPrivate&);
     590
     591#if PLATFORM(COCOA)
     592    void onNewVideoFrameMetadata(VideoFrameMetadata&&, RetainPtr<CVPixelBufferRef>&&);
     593#endif
    586594
    587595    bool requiresTextTrackRepresentation() const;
    … …  
    682690
    683691    std::optional<VideoFrameMetadata> videoFrameMetadata();
     692    void startVideoFrameMetadataGathering();
     693    void stopVideoFrameMetadataGathering();
    684694
    685695private:
    … …  
    726736    bool m_shouldContinueAfterKeyNeeded { false };
    727737#endif
     738    bool m_isGatheringVideoFrameMetadata { false };
    728739};
    729740

  • trunk/Source/WebCore/platform/graphics/MediaPlayerPrivate.h

    r284857 r285993  
    324324
    325325    virtual std::optional<VideoFrameMetadata> videoFrameMetadata() { return { }; }
     326    virtual void startVideoFrameMetadataGathering() { }
     327    virtual void stopVideoFrameMetadataGathering() { }
    326328};
    327329

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.h

    r285330 r285993  
    2929
    3030#include "MediaPlayerPrivateAVFoundation.h"
     31#include <CoreMedia/CMTime.h>
    3132#include <wtf/Function.h>
    3233#include <wtf/HashMap.h>
    … …  
    342343    bool pauseAtHostTime(const MonotonicTime&) final;
    343344    bool haveBeenAskedToPaint() const { return m_haveBeenAskedToPaint; }
     345
     346    void startVideoFrameMetadataGathering() final;
     347    void stopVideoFrameMetadataGathering() final;
     348    std::optional<VideoFrameMetadata> videoFrameMetadata() final { return std::exchange(m_videoFrameMetadata, { }); }
     349    void checkNewVideoFrameMetadata(CMTime);
    344350
    345351    RetainPtr<AVURLAsset> m_avAsset;
    … …  
    452458    bool m_waitForVideoOutputMediaDataWillChangeTimedOut { false };
    453459    bool m_haveBeenAskedToPaint { false };
     460    uint64_t m_sampleCount { 0 };
     461    RetainPtr<id> m_videoFrameMetadataGatheringObserver;
     462    bool m_isGatheringVideoFrameMetadata { false };
     463    std::optional<VideoFrameMetadata> m_videoFrameMetadata;
    454464};
    455465

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm

    r285330 r285993  
    537537        m_currentTimeObserver = nil;
    538538
     539        if (m_videoFrameMetadataGatheringObserver) {
     540            [m_avPlayer removeTimeObserver:m_videoFrameMetadataGatheringObserver.get()];
     541            m_videoFrameMetadataGatheringObserver = nil;
     542        }
     543
    539544        m_avPlayer = nil;
    540545    }
    … …  
    11091114    }];
    11101115
     1116    if (m_isGatheringVideoFrameMetadata)
     1117        startVideoFrameMetadataGathering();
    11111118    setDelayCallbacks(false);
    11121119}
    … …  
    14321439{
    14331440    return m_cachedTimeControlStatus == AVPlayerTimeControlStatusPaused;
     1441}
     1442
     1443void MediaPlayerPrivateAVFoundationObjC::startVideoFrameMetadataGathering()
     1444{
     1445    ASSERT(!m_videoFrameMetadataGatheringObserver || m_avPlayer);
     1446    m_isGatheringVideoFrameMetadata = true;
     1447
     1448    // FIXME: We should use a CADisplayLink to get updates on rendering, for now we emulate with addPeriodicTimeObserverForInterval.
     1449    m_videoFrameMetadataGatheringObserver = [m_avPlayer addPeriodicTimeObserverForInterval:PAL::CMTimeMake(1, 60) queue:dispatch_get_main_queue() usingBlock:[weakThis = WeakPtr { *this }](CMTime currentTime) {
     1450        ensureOnMainThread([weakThis, currentTime] {
     1451            if (weakThis)
     1452                weakThis->checkNewVideoFrameMetadata(currentTime);
     1453        });
     1454    }];
     1455}
     1456
     1457void MediaPlayerPrivateAVFoundationObjC::checkNewVideoFrameMetadata(CMTime currentTime)
     1458{
     1459    if (!updateLastPixelBuffer())
     1460        return;
     1461
     1462    VideoFrameMetadata metadata;
     1463    metadata.width = m_cachedPresentationSize.width();
     1464    metadata.height = m_cachedPresentationSize.height();
     1465    metadata.presentedFrames = ++m_sampleCount;
     1466    metadata.mediaTime = PAL::CMTimeGetSeconds(currentTime);
     1467    // FIXME: presentationTime and expectedDisplayTime might not always have the same value, we should try getting more precise values.
     1468    metadata.presentationTime = MonotonicTime::now().secondsSinceEpoch().seconds();
     1469    metadata.expectedDisplayTime = metadata.presentationTime;
     1470
     1471    m_videoFrameMetadata = metadata;
     1472    player()->onNewVideoFrameMetadata(WTFMove(metadata), m_lastPixelBuffer.get());
     1473}
     1474
     1475void MediaPlayerPrivateAVFoundationObjC::stopVideoFrameMetadataGathering()
     1476{
     1477    m_isGatheringVideoFrameMetadata = false;
     1478    m_videoFrameMetadata = { };
     1479
     1480    if (m_videoFrameMetadataGatheringObserver) {
     1481        [m_avPlayer removeTimeObserver:m_videoFrameMetadataGatheringObserver.get()];
     1482        m_videoFrameMetadataGatheringObserver = nil;
     1483    }
    14341484}
    14351485
    … …  
    25492599    // for the requested time has already been retrieved. In this case, the last valid image (if any)
    25502600    // should be displayed.
    2551     if (!updateLastPixelBuffer() && (m_lastImage || !m_lastPixelBuffer))
     2601    if ((m_isGatheringVideoFrameMetadata || !updateLastPixelBuffer()) && (m_lastImage || !m_lastPixelBuffer))
    25522602        return;
    25532603
    … …  
    25842634RetainPtr<CVPixelBufferRef> MediaPlayerPrivateAVFoundationObjC::pixelBufferForCurrentTime()
    25852635{
    2586     updateLastPixelBuffer();
    2587     if (!m_lastPixelBuffer)
    2588         return nullptr;
     2636    if (!m_isGatheringVideoFrameMetadata)
     2637        updateLastPixelBuffer();
    25892638
    25902639    return m_lastPixelBuffer;

  • trunk/Source/WebCore/platform/graphics/cv/PixelBufferConformerCV.h

    r269371 r285993  
    3939    WEBCORE_EXPORT PixelBufferConformerCV(CFDictionaryRef attributes);
    4040    WEBCORE_EXPORT RetainPtr<CVPixelBufferRef> convert(CVPixelBufferRef);
    41     RetainPtr<CGImageRef> createImageFromPixelBuffer(CVPixelBufferRef);
     41    WEBCORE_EXPORT RetainPtr<CGImageRef> createImageFromPixelBuffer(CVPixelBufferRef);
    4242
    4343private:

  • trunk/Source/WebKit/ChangeLog

    r285992 r285993  
     12021-11-18  Youenn Fablet  <youenn@apple.com>
     2
     3        MediaPlayerAVFoundation should support rvfc
     4        https://bugs.webkit.org/show_bug.cgi?id=231945
     5        <rdar://problem/84657372>
     6
     7        Reviewed by Eric Carlson.
     8
     9        Add IPC messaging to start/stop metadata gathering from WebProcess to GPUProcess.
     10        Add IPC messaging to push metadata to WebProcess, including sending the corresponding CVPixelBuffer for the current metadata.
     11        MediaPlayerPrivateRemote will then serve that buffer when being requested the current image.
     12        This ensures that painting on a canvas does not mark the current buffer as acquired, which would then skip the video frame metadata gathering.
     13        This also ensures the same buffer is used if a video element is painted several times during the same rvfc callback.
     14        This has the side benefit of removing the synchronous IPC to paint the video element in a canvas.
     15
     16        * GPUProcess/media/RemoteMediaPlayerProxy.cpp:
     17        * GPUProcess/media/RemoteMediaPlayerProxy.h:
     18        * GPUProcess/media/RemoteMediaPlayerProxy.messages.in:
     19        * GPUProcess/media/cocoa/RemoteMediaPlayerProxyCocoa.mm:
     20        * Scripts/webkit/messages.py:
     21        * WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp:
     22        * WebProcess/GPU/media/MediaPlayerPrivateRemote.h:
     23        * WebProcess/GPU/media/MediaPlayerPrivateRemote.messages.in:
     24        * WebProcess/GPU/media/cocoa/MediaPlayerPrivateRemoteCocoa.mm:
     25
    1262021-11-18  Chris Lord  <clord@igalia.com>
    227

  • trunk/Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.cpp

    r284294 r285993  
    10991099}
    11001100
     1101void RemoteMediaPlayerProxy::startVideoFrameMetadataGathering()
     1102{
     1103    if (m_player)
     1104        m_player->startVideoFrameMetadataGathering();
     1105}
     1106
     1107void RemoteMediaPlayerProxy::stopVideoFrameMetadataGathering()
     1108{
     1109    if (m_player)
     1110        m_player->startVideoFrameMetadataGathering();
     1111}
     1112
    11011113#if !RELEASE_LOG_DISABLED
    11021114WTFLogChannel& RemoteMediaPlayerProxy::logChannel() const

  • trunk/Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.h

    r284439 r285993  
    308308    void pauseAtHostTime(MonotonicTime);
    309309
     310    void startVideoFrameMetadataGathering();
     311    void stopVideoFrameMetadataGathering();
     312#if PLATFORM(COCOA)
     313    void mediaPlayerOnNewVideoFrameMetadata(WebCore::VideoFrameMetadata&&, RetainPtr<CVPixelBufferRef>&&);
     314#endif
     315
    310316    bool mediaPlayerPausedOrStalled() const;
    311317    void currentTimeChanged(const MediaTime&);

  • trunk/Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.messages.in

    r284439 r285993  
    133133    PlayAtHostTime(MonotonicTime time)
    134134    PauseAtHostTime(MonotonicTime time)
     135
     136    StartVideoFrameMetadataGathering()
     137    StopVideoFrameMetadataGathering()
    135138}
    136139

  • trunk/Source/WebKit/GPUProcess/media/cocoa/RemoteMediaPlayerProxyCocoa.mm

    r284439 r285993  
    3131#import "LayerHostingContext.h"
    3232#import "MediaPlayerPrivateRemoteMessages.h"
     33#import "WebCoreArgumentCoders.h"
    3334#import <QuartzCore/QuartzCore.h>
    3435#import <WebCore/FloatSize.h>
    … …  
    8283    m_videoInlineSize = size;
    8384    setVideoInlineSizeIfPossible(*m_inlineLayerHostingContext, size);
     85}
     86
     87void RemoteMediaPlayerProxy::mediaPlayerOnNewVideoFrameMetadata(VideoFrameMetadata&& metadata, RetainPtr<CVPixelBufferRef>&& buffer)
     88{
     89    m_webProcessConnection->send(Messages::MediaPlayerPrivateRemote::PushVideoFrameMetadata(metadata, buffer), m_id);
    8490}
    8591

  • trunk/Source/WebKit/Scripts/webkit/messages.py

    r285981 r285993  
    372372        '"GestureTypes.h"': ["PLATFORM(IOS_FAMILY)"],
    373373        '"WCLayerTreeHostIdentifier.h"': ["USE(GRAPHICS_LAYER_WC)"],
     374        '<WebCore/CVUtilities.h>': ["PLATFORM(COCOA)", ],
     375        '<WebCore/DataDetectorType.h>': ["ENABLE(DATA_DETECTION)"],
    374376        '<WebCore/MediaPlaybackTargetContext.h>': ["ENABLE(WIRELESS_PLAYBACK_TARGET)"],
    375         '<WebCore/DataDetectorType.h>': ["ENABLE(DATA_DETECTION)"],
    376377    }
    377378    if not header in conditions:
    … …  
    680681
    681682    special_cases = {
     683        'CVPixelBufferRef': ['<WebCore/CVUtilities.h>'],
    682684        'IPC::Semaphore': ['"IPCSemaphore.h"'],
    683685        'Inspector::ExtensionError': ['"InspectorExtensionTypes.h"'],

  • trunk/Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp

    r284741 r285993  
    8181
    8282#if PLATFORM(COCOA)
     83#import <WebCore/PixelBufferConformerCV.h>
    8384#import <WebCore/VideoLayerManagerObjC.h>
    8485#endif
    … …  
    13781379}
    13791380
     1381std::optional<VideoFrameMetadata> MediaPlayerPrivateRemote::videoFrameMetadata()
     1382{
     1383    auto videoFrameMetadata = std::exchange(m_videoFrameMetadata, { });
     1384    return videoFrameMetadata;
     1385}
     1386
     1387void MediaPlayerPrivateRemote::startVideoFrameMetadataGathering()
     1388{
     1389    m_isGatheringVideoFrameMetadata = true;
     1390    connection().send(Messages::RemoteMediaPlayerProxy::StartVideoFrameMetadataGathering(), m_id);
     1391}
     1392
     1393void MediaPlayerPrivateRemote::stopVideoFrameMetadataGathering()
     1394{
     1395    m_isGatheringVideoFrameMetadata = false;
     1396#if PLATFORM(COCOA)
     1397    m_pixelBufferGatheredWithVideoFrameMetadata = nullptr;
     1398#endif
     1399    connection().send(Messages::RemoteMediaPlayerProxy::StopVideoFrameMetadataGathering(), m_id);
     1400}
     1401
    13801402void MediaPlayerPrivateRemote::requestResource(RemoteMediaResourceIdentifier remoteMediaResourceIdentifier, WebCore::ResourceRequest&& request, WebCore::PlatformMediaResourceLoader::LoadOptions options, CompletionHandler<void()>&& completionHandler)
    13811403{

  • trunk/Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.h

    r284741 r285993  
    4040#include <WebCore/MediaPlayerPrivate.h>
    4141#include <WebCore/SecurityOriginData.h>
     42#include <WebCore/VideoFrameMetadata.h>
    4243#include <wtf/LoggerHelper.h>
    4344#include <wtf/MediaTime.h>
    … …  
    5758class SerializedPlatformDataCueValue;
    5859class VideoLayerManager;
     60
     61#if PLATFORM(COCOA)
     62class PixelBufferConformerCV;
     63#endif
    5964}
    6065
    … …  
    400405    bool pauseAtHostTime(const MonotonicTime&) final;
    401406    void updateConfiguration(RemoteMediaPlayerConfiguration&&);
     407
     408    std::optional<WebCore::VideoFrameMetadata> videoFrameMetadata() final;
     409    void startVideoFrameMetadataGathering() final;
     410    void stopVideoFrameMetadataGathering() final;
     411
     412#if PLATFORM(COCOA)
     413    void pushVideoFrameMetadata(WebCore::VideoFrameMetadata&&, RetainPtr<CVPixelBufferRef>&&);
     414#endif
    402415
    403416    WeakPtr<WebCore::MediaPlayer> m_player;
    … …  
    451464    RetainPtr<CVPixelBufferRef> m_pixelBufferForCurrentTime;
    452465#endif
     466#if PLATFORM(COCOA)
     467    RetainPtr<CVPixelBufferRef> m_pixelBufferGatheredWithVideoFrameMetadata;
     468    std::unique_ptr<WebCore::PixelBufferConformerCV> m_pixelBufferConformer;
     469#endif
     470    std::optional<WebCore::VideoFrameMetadata> m_videoFrameMetadata;
     471    bool m_isGatheringVideoFrameMetadata { false };
    453472};
    454473

  • trunk/Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.messages.in

    r278917 r285993  
    9292    GetRawCookies(URL url) -> (Vector<WebCore::Cookie> result) Async
    9393#endif
     94
     95#if PLATFORM(COCOA)
     96    PushVideoFrameMetadata(struct WebCore::VideoFrameMetadata metadata, RetainPtr<CVPixelBufferRef> buffer);
     97#endif
    9498}
    9599

  • trunk/Source/WebKit/WebProcess/GPU/media/cocoa/MediaPlayerPrivateRemoteCocoa.mm

    r284439 r285993  
    3434#import <WebCore/ColorSpaceCG.h>
    3535#import <WebCore/IOSurface.h>
     36#import <WebCore/PixelBufferConformerCV.h>
    3637#import <pal/spi/cocoa/QuartzCoreSPI.h>
    3738#import <wtf/MachSendRight.h>
     39
     40#import <WebCore/CoreVideoSoftLink.h>
    3841
    3942namespace WebKit {
    … …  
    4750#endif
    4851
     52void MediaPlayerPrivateRemote::pushVideoFrameMetadata(WebCore::VideoFrameMetadata&& videoFrameMetadata, RetainPtr<CVPixelBufferRef>&& buffer)
     53{
     54    if (!m_isGatheringVideoFrameMetadata)
     55        return;
     56    m_videoFrameMetadata = WTFMove(videoFrameMetadata);
     57    m_pixelBufferGatheredWithVideoFrameMetadata = WTFMove(buffer);
     58}
     59
    4960RefPtr<NativeImage> MediaPlayerPrivateRemote::nativeImageForCurrentTime()
    5061{
     62    if (m_pixelBufferGatheredWithVideoFrameMetadata) {
     63        if (!m_pixelBufferConformer)
     64            m_pixelBufferConformer = makeUnique<PixelBufferConformerCV>((__bridge CFDictionaryRef)@{ (__bridge NSString *)kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_32BGRA) });
     65        ASSERT(m_pixelBufferConformer);
     66        if (!m_pixelBufferConformer)
     67            return nullptr;
     68        return NativeImage::create(m_pixelBufferConformer->createImageFromPixelBuffer(m_pixelBufferGatheredWithVideoFrameMetadata.get()));
     69    }
     70
    5171    std::optional<MachSendRight> sendRight;
    5272    auto colorSpace = DestinationColorSpace::SRGB();

  • trunk/Tools/ChangeLog

    r285988 r285993  
     12021-11-18  Youenn Fablet  <youenn@apple.com>
     2
     3        MediaPlayerAVFoundation should support rvfc
     4        https://bugs.webkit.org/show_bug.cgi?id=231945
     5
     6        Reviewed by Eric Carlson.
     7
     8        Enable rvfc by default in test runner.
     9
     10        * DumpRenderTree/mac/DumpRenderTree.mm:
     11        * WebKitTestRunner/TestController.cpp:
     12
    1132021-11-18  Carlos Garcia Campos  <cgarcia@igalia.com>
    214

  • trunk/Tools/DumpRenderTree/mac/DumpRenderTree.mm

    r284528 r285993  
    864864
    865865        if (enableAllExperimentalFeatures) {
    866             for (WebFeature *feature in [WebPreferences _experimentalFeatures]) {
    867                 // FIXME: We disable rvfc by default. Enable it when the video backend support is good enough.
    868                 auto enabled = [feature.name isEqual:@"RequestVideoFrameCallback"] ? NO : YES;
    869                 [preferences _setEnabled:enabled forFeature:feature];
    870             }
     866            for (WebFeature *feature in [WebPreferences _experimentalFeatures])
     867                [preferences _setEnabled:YES forFeature:feature];
    871868        }
    872869

  • trunk/Tools/WebKitTestRunner/TestController.cpp

    r285168 r285993  
    906906            WKPreferencesEnableAllExperimentalFeatures(preferences);
    907907
    908         // FIXME: We disable rvfc by default. Enable it when the video backend support is good enough.
    909         WKPreferencesSetRequestVideoFrameCallbackEnabled(preferences, false);
    910 
    911908        WKPreferencesResetAllInternalDebugFeatures(preferences);
    912909
Note: See TracChangeset for help on using the changeset viewer.