Changeset 285984 in webkit


Ignore:
Timestamp:
Nov 17, 2021 11:46:05 PM (8 months ago)
Author:
youenn@apple.com
Message:

Add support for more rvfc metadata
​https://bugs.webkit.org/show_bug.cgi?id=233185

Reviewed by Eric Carlson.

LayoutTests/imported/w3c:

  • web-platform-tests/video-rvfc/request-video-frame-callback-webrtc.https-expected.txt:

Source/WebCore:

Introduce VideoSampleMetadata that allows a RealtimeMediaSource to provide metadata for each video frame.
Fill VideoSampleMetadata for each source (capture and WebRTC)
Make use of VideoSampleMetadata in MediaPlayerPrivateMediaStreamAVFObjC to fill VideoFrameMetadata.
Enhance MediaPlayerPrivateAVFoundationObjC to fill in the newly exposed metadata in VideoFrameMetadata.
Update HTMLVideoElement to process the timestamp values as needed.

Covered by existing tests.

  • Headers.cmake:
  • Modules/mediastream/CanvasCaptureMediaStreamTrack.cpp:
  • WebCore.xcodeproj/project.pbxproj:
  • html/HTMLVideoElement.cpp:
  • html/VideoFrameMetadata.idl:
  • platform/VideoSampleMetadata.h: Added.
  • platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
  • platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
  • platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp:
  • platform/mediarecorder/MediaRecorderPrivateAVFImpl.h:
  • platform/mediarecorder/MediaRecorderPrivateMock.cpp:
  • platform/mediarecorder/MediaRecorderPrivateMock.h:
  • platform/mediastream/RealtimeIncomingVideoSource.cpp:
  • platform/mediastream/RealtimeIncomingVideoSource.h:
  • platform/mediastream/RealtimeMediaSource.cpp:
  • platform/mediastream/RealtimeMediaSource.h:
  • platform/mediastream/RealtimeOutgoingVideoSource.h:
  • platform/mediastream/RealtimeVideoCaptureSource.cpp:
  • platform/mediastream/RealtimeVideoCaptureSource.h:
  • platform/mediastream/RealtimeVideoSource.cpp:
  • platform/mediastream/RealtimeVideoSource.h:
  • platform/mediastream/cocoa/DisplayCaptureSourceCocoa.cpp:
  • platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp:
  • platform/mediastream/gstreamer/GStreamerVideoCaptureSource.cpp:
  • platform/mediastream/gstreamer/MockRealtimeVideoSourceGStreamer.cpp:
  • platform/mediastream/gstreamer/MockRealtimeVideoSourceGStreamer.h:
  • platform/mediastream/libwebrtc/gstreamer/RealtimeIncomingVideoSourceLibWebRTC.cpp:
  • platform/mediastream/libwebrtc/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.cpp:
  • platform/mediastream/libwebrtc/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.h:
  • platform/mediastream/mac/AVVideoCaptureSource.mm:
  • platform/mediastream/mac/MockRealtimeVideoSourceMac.mm:
  • platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.mm:
  • platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.cpp:
  • platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.h:
  • testing/Internals.cpp:
  • testing/Internals.h:

Source/WebKit:

Make sure to receive frame metadats, send it through IPC to WebProcess and pipe it to media source observers.

  • UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp:
  • WebProcess/GPU/webrtc/MediaRecorderPrivate.cpp:
  • WebProcess/GPU/webrtc/MediaRecorderPrivate.h:
  • WebProcess/cocoa/RemoteCaptureSampleManager.cpp:
  • WebProcess/cocoa/RemoteCaptureSampleManager.h:
  • WebProcess/cocoa/RemoteCaptureSampleManager.messages.in:
  • WebProcess/cocoa/RemoteRealtimeDisplaySource.h:
  • WebProcess/cocoa/RemoteRealtimeVideoSource.cpp:
  • WebProcess/cocoa/RemoteRealtimeVideoSource.h:

LayoutTests:

  • platform/ios-wk2/TestExpectations:
  • platform/mac-wk1/TestExpectations:
  • platform/mac-wk2/TestExpectations:
Location:
trunk
Files:
1 added
49 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r285982 r285984  
     12021-11-17  Youenn Fablet  <youenn@apple.com>
     2
     3        Add support for more rvfc metadata
     4        https://bugs.webkit.org/show_bug.cgi?id=233185
     5
     6        Reviewed by Eric Carlson.
     7
     8        * platform/ios-wk2/TestExpectations:
     9        * platform/mac-wk1/TestExpectations:
     10        * platform/mac-wk2/TestExpectations:
     11
    1122021-11-17  Ricky Mondello  <rmondello@apple.com>
    213

  • trunk/LayoutTests/imported/w3c/ChangeLog

    r285954 r285984  
     12021-11-17  Youenn Fablet  <youenn@apple.com>
     2
     3        Add support for more rvfc metadata
     4        https://bugs.webkit.org/show_bug.cgi?id=233185
     5
     6        Reviewed by Eric Carlson.
     7
     8        * web-platform-tests/video-rvfc/request-video-frame-callback-webrtc.https-expected.txt:
     9
    1102021-11-17  Chris Dumez  <cdumez@apple.com>
    211

  • trunk/LayoutTests/imported/w3c/web-platform-tests/video-rvfc/request-video-frame-callback-webrtc.https-expected.txt

    r284528 r285984  
    11
    2 FAIL Test video.requestVideoFrameCallback() parameters for WebRTC applications. assert_greater_than: expected a number but got a "undefined"
     2PASS Test video.requestVideoFrameCallback() parameters for WebRTC applications.
    33
    44

  • trunk/Source/WebCore/ChangeLog

    r285982 r285984  
     12021-11-17  Youenn Fablet  <youenn@apple.com>
     2
     3        Add support for more rvfc metadata
     4        https://bugs.webkit.org/show_bug.cgi?id=233185
     5
     6        Reviewed by Eric Carlson.
     7
     8        Introduce VideoSampleMetadata that allows a RealtimeMediaSource to provide metadata for each video frame.
     9        Fill VideoSampleMetadata for each source (capture and WebRTC)
     10        Make use of VideoSampleMetadata in MediaPlayerPrivateMediaStreamAVFObjC to fill VideoFrameMetadata.
     11        Enhance MediaPlayerPrivateAVFoundationObjC to fill in the newly exposed metadata in VideoFrameMetadata.
     12        Update HTMLVideoElement to process the timestamp values as needed.
     13
     14        Covered by existing tests.
     15
     16        * Headers.cmake:
     17        * Modules/mediastream/CanvasCaptureMediaStreamTrack.cpp:
     18        * WebCore.xcodeproj/project.pbxproj:
     19        * html/HTMLVideoElement.cpp:
     20        * html/VideoFrameMetadata.idl:
     21        * platform/VideoSampleMetadata.h: Added.
     22        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
     23        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
     24        * platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp:
     25        * platform/mediarecorder/MediaRecorderPrivateAVFImpl.h:
     26        * platform/mediarecorder/MediaRecorderPrivateMock.cpp:
     27        * platform/mediarecorder/MediaRecorderPrivateMock.h:
     28        * platform/mediastream/RealtimeIncomingVideoSource.cpp:
     29        * platform/mediastream/RealtimeIncomingVideoSource.h:
     30        * platform/mediastream/RealtimeMediaSource.cpp:
     31        * platform/mediastream/RealtimeMediaSource.h:
     32        * platform/mediastream/RealtimeOutgoingVideoSource.h:
     33        * platform/mediastream/RealtimeVideoCaptureSource.cpp:
     34        * platform/mediastream/RealtimeVideoCaptureSource.h:
     35        * platform/mediastream/RealtimeVideoSource.cpp:
     36        * platform/mediastream/RealtimeVideoSource.h:
     37        * platform/mediastream/cocoa/DisplayCaptureSourceCocoa.cpp:
     38        * platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp:
     39        * platform/mediastream/gstreamer/GStreamerVideoCaptureSource.cpp:
     40        * platform/mediastream/gstreamer/MockRealtimeVideoSourceGStreamer.cpp:
     41        * platform/mediastream/gstreamer/MockRealtimeVideoSourceGStreamer.h:
     42        * platform/mediastream/libwebrtc/gstreamer/RealtimeIncomingVideoSourceLibWebRTC.cpp:
     43        * platform/mediastream/libwebrtc/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.cpp:
     44        * platform/mediastream/libwebrtc/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.h:
     45        * platform/mediastream/mac/AVVideoCaptureSource.mm:
     46        * platform/mediastream/mac/MockRealtimeVideoSourceMac.mm:
     47        * platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.mm:
     48        * platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.cpp:
     49        * platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.h:
     50        * testing/Internals.cpp:
     51        * testing/Internals.h:
     52
    1532021-11-17  Ricky Mondello  <rmondello@apple.com>
    254

  • trunk/Source/WebCore/Headers.cmake

    r285949 r285984  
    11911191    platform/ValidationBubble.h
    11921192    platform/VideoFrameMetadata.h
     1193    platform/VideoSampleMetadata.h
    11931194    platform/WebGLStateTracker.h
    11941195    platform/Widget.h

  • trunk/Source/WebCore/Modules/mediastream/CanvasCaptureMediaStreamTrack.cpp

    r283489 r285984  
    197197        return;
    198198
    199     videoSampleAvailable(*sample);
     199    VideoSampleMetadata metadata;
     200    metadata.captureTime = MonotonicTime::now().secondsSinceEpoch();
     201    videoSampleAvailable(*sample, metadata);
    200202}
    201203

  • trunk/Source/WebCore/WebCore.xcodeproj/project.pbxproj

    r285949 r285984  
    11191119                415864A023BF7CBF00A0A61E /* RealtimeVideoUtilities.h in Headers */ = {isa = PBXBuildFile; fileRef = 41D1938F2152C561006F14CA /* RealtimeVideoUtilities.h */; settings = {ATTRIBUTES = (Private, ); }; };
    11201120                415CDAF51E6B8F8B004F11EE /* CanvasCaptureMediaStreamTrack.h in Headers */ = {isa = PBXBuildFile; fileRef = 41C7E1061E6A54360027B4DE /* CanvasCaptureMediaStreamTrack.h */; };
     1121                416049332743B0E800A86FA0 /* VideoSampleMetadata.h in Headers */ = {isa = PBXBuildFile; fileRef = 416049312743B0E700A86FA0 /* VideoSampleMetadata.h */; settings = {ATTRIBUTES = (Private, ); }; };
    11211122                41614A791DA64241004AD06F /* HTTPHeaderValues.h in Headers */ = {isa = PBXBuildFile; fileRef = 41614A771DA64236004AD06F /* HTTPHeaderValues.h */; settings = {ATTRIBUTES = (Private, ); }; };
    11221123                4161E2D51FE48DC500EC2E96 /* FetchLoader.h in Headers */ = {isa = PBXBuildFile; fileRef = 4147E2B51C89912600A7E715 /* FetchLoader.h */; settings = {ATTRIBUTES = (Private, ); }; };
    … …  
    83418342                415E1BB62150152A0022DA96 /* RTCRtpSynchronizationSource.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = RTCRtpSynchronizationSource.h; sourceTree = "<group>"; };
    83428343                415E1BB7215015300022DA96 /* RTCRtpSynchronizationSource.idl */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text; path = RTCRtpSynchronizationSource.idl; sourceTree = "<group>"; };
     8344                416049312743B0E700A86FA0 /* VideoSampleMetadata.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = VideoSampleMetadata.h; sourceTree = "<group>"; };
    83438345                41614A761DA64236004AD06F /* HTTPHeaderValues.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = HTTPHeaderValues.cpp; sourceTree = "<group>"; };
    83448346                41614A771DA64236004AD06F /* HTTPHeaderValues.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = HTTPHeaderValues.h; sourceTree = "<group>"; };
    … …  
    2915929161                                83C45B8D1DC2B67C008871BA /* ValidationBubble.h */,
    2916029162                                41DEEFB42719BA1900CB8D74 /* VideoFrameMetadata.h */,
     29163                                416049312743B0E700A86FA0 /* VideoSampleMetadata.h */,
    2916129164                                515F79511CFCA3C700CCED93 /* WebCoreCrossThreadCopier.cpp */,
    2916229165                                515F79521CFCA3C700CCED93 /* WebCoreCrossThreadCopier.h */,
    … …  
    3673436737                                075033A8252BD36800F70CE3 /* VideoPlaybackQualityMetrics.h in Headers */,
    3673536738                                0757B13E214AE79900794B0D /* VideoPreset.h in Headers */,
     36739                                416049332743B0E800A86FA0 /* VideoSampleMetadata.h in Headers */,
    3673636740                                CDC939A81E9BDFB100BB768D /* VideoToolboxSoftLink.h in Headers */,
    3673736741                                BE88E0DF1715D2A200658D98 /* VideoTrack.h in Headers */,

  • trunk/Source/WebCore/html/HTMLVideoElement.cpp

    r284528 r285984  
    4242#include "Logging.h"
    4343#include "Page.h"
     44#include "Performance.h"
    4445#include "PictureInPictureSupport.h"
    4546#include "RenderImage.h"
    … …  
    610611}
    611612
     613static void processVideoFrameMetadataTimestamps(VideoFrameMetadata& metadata, Performance& performance)
     614{
     615    metadata.presentationTime = performance.relativeTimeFromTimeOriginInReducedResolution(MonotonicTime::fromRawSeconds(metadata.presentationTime));
     616    metadata.expectedDisplayTime = performance.relativeTimeFromTimeOriginInReducedResolution(MonotonicTime::fromRawSeconds(metadata.expectedDisplayTime));
     617    if (metadata.captureTime)
     618        metadata.captureTime = performance.relativeTimeFromTimeOriginInReducedResolution(MonotonicTime::fromRawSeconds(*metadata.captureTime));
     619    if (metadata.receiveTime)
     620        metadata.receiveTime = performance.relativeTimeFromTimeOriginInReducedResolution(MonotonicTime::fromRawSeconds(*metadata.receiveTime));
     621}
     622
    612623void HTMLVideoElement::serviceRequestVideoFrameCallbacks(ReducedResolutionSeconds now)
    613624{
    … …  
    616627
    617628    auto videoFrameMetadata = player()->videoFrameMetadata();
    618     if (!videoFrameMetadata)
    619         return;
     629    if (!videoFrameMetadata || !document().domWindow())
     630        return;
     631
     632    processVideoFrameMetadataTimestamps(*videoFrameMetadata, document().domWindow()->performance());
    620633
    621634    Ref protectedThis { *this };

  • trunk/Source/WebCore/html/VideoFrameMetadata.idl

    r284528 r285984  
    3030    JSGenerateToJSObject,
    3131] dictionary VideoFrameMetadata {
    32     // FIXME: expose presentationTime and expectedDisplayTime.
    33     // required DOMHighResTimeStamp presentationTime;
    34     // required DOMHighResTimeStamp expectedDisplayTime;
     32    required DOMHighResTimeStamp presentationTime;
     33    required DOMHighResTimeStamp expectedDisplayTime;
    3534
    3635    required unsigned long width;

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h

    r284857 r285984  
    137137    void flushRenderers();
    138138
    139     void processNewVideoSample(MediaSample&, bool hasChangedOrientation);
     139    void processNewVideoSample(MediaSample&, bool hasChangedOrientation, VideoSampleMetadata, Seconds);
    140140    void enqueueVideoSample(MediaSample&);
    141141    void requestNotificationWhenReadyForVideoData();
    … …  
    209209
    210210    // RealtimeMediaSouce::VideoSampleObserver
    211     void videoSampleAvailable(MediaSample&) final;
     211    void videoSampleAvailable(MediaSample&, VideoSampleMetadata) final;
    212212
    213213    RetainPtr<PlatformLayer> createVideoFullscreenLayer() override;
    … …  
    284284    uint64_t m_sampleCount { 0 };
    285285    uint64_t m_lastVideoFrameMetadataSampleCount { 0 };
     286    Seconds m_presentationTime { 0 };
     287    VideoSampleMetadata m_sampleMetadata;
    286288};
    287289   

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm

    r284528 r285984  
    245245}
    246246
    247 void MediaPlayerPrivateMediaStreamAVFObjC::videoSampleAvailable(MediaSample& sample)
    248 {
    249     processNewVideoSample(sample, sample.videoRotation() != m_videoRotation || sample.videoMirrored() != m_videoMirrored);
     247void MediaPlayerPrivateMediaStreamAVFObjC::videoSampleAvailable(MediaSample& sample, VideoSampleMetadata metadata)
     248{
     249    auto presentationTime = MonotonicTime::now().secondsSinceEpoch();
     250    processNewVideoSample(sample, sample.videoRotation() != m_videoRotation || sample.videoMirrored() != m_videoMirrored, metadata, presentationTime);
    250251    enqueueVideoSample(sample);
    251252}
    … …  
    277278}
    278279
    279 void MediaPlayerPrivateMediaStreamAVFObjC::processNewVideoSample(MediaSample& sample, bool hasChangedOrientation)
     280void MediaPlayerPrivateMediaStreamAVFObjC::processNewVideoSample(MediaSample& sample, bool hasChangedOrientation, VideoSampleMetadata metadata, Seconds presentationTime)
    280281{
    281282    if (!isMainThread()) {
    … …  
    284285            m_currentVideoSample = &sample;
    285286        }
    286         callOnMainThread([weakThis = WeakPtr { *this }, hasChangedOrientation]() mutable {
     287        callOnMainThread([weakThis = WeakPtr { *this }, hasChangedOrientation, metadata, presentationTime]() mutable {
    287288            if (!weakThis)
    288289                return;
    … …  
    299300                return;
    300301
    301             weakThis->processNewVideoSample(*sample, hasChangedOrientation);
     302            weakThis->processNewVideoSample(*sample, hasChangedOrientation, metadata, presentationTime);
    302303        });
    303304        return;
    … …  
    317318    }
    318319
     320    m_presentationTime = presentationTime;
     321    m_sampleMetadata = metadata;
    319322    ++m_sampleCount;
    320323
    … …  
    11071110    metadata.height = m_intrinsicSize.height();
    11081111    metadata.presentedFrames = m_sampleCount;
    1109 
    1110     // FIXME: It would be good to expose more video frame metadata.
     1112    metadata.presentationTime = m_presentationTime.seconds();
     1113    metadata.expectedDisplayTime = m_presentationTime.seconds();
     1114    metadata.processingDuration = m_sampleMetadata.processingDuration;
     1115    if (m_sampleMetadata.captureTime)
     1116        metadata.captureTime = m_sampleMetadata.captureTime->seconds();
     1117    if (m_sampleMetadata.receiveTime)
     1118        metadata.receiveTime = m_sampleMetadata.receiveTime->seconds();
     1119    metadata.rtpTimestamp = m_sampleMetadata.rtpTimestamp;
    11111120
    11121121    return metadata;

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp

    r284857 r285984  
    8484}
    8585
    86 void MediaRecorderPrivateAVFImpl::videoSampleAvailable(MediaSample& sampleBuffer)
     86void MediaRecorderPrivateAVFImpl::videoSampleAvailable(MediaSample& sampleBuffer, VideoSampleMetadata)
    8787{
    8888    if (shouldMuteVideo()) {

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateAVFImpl.h

    r284857 r285984  
    5050
    5151    // MediaRecorderPrivate
    52     void videoSampleAvailable(MediaSample&) final;
     52    void videoSampleAvailable(MediaSample&, VideoSampleMetadata) final;
    5353    void fetchData(FetchDataCallback&&) final;
    5454    void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateMock.cpp

    r284857 r285984  
    6868}
    6969
    70 void MediaRecorderPrivateMock::videoSampleAvailable(MediaSample&)
     70void MediaRecorderPrivateMock::videoSampleAvailable(MediaSample&, VideoSampleMetadata)
    7171{
    7272    Locker locker { m_bufferLock };

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateMock.h

    r284857 r285984  
    4343private:
    4444    // MediaRecorderPrivate
    45     void videoSampleAvailable(MediaSample&) final;
     45    void videoSampleAvailable(MediaSample&, VideoSampleMetadata) final;
    4646    void fetchData(FetchDataCallback&&) final;
    4747    void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;

  • trunk/Source/WebCore/platform/mediastream/RealtimeIncomingVideoSource.cpp

    r282755 r285984  
    105105}
    106106
     107VideoSampleMetadata RealtimeIncomingVideoSource::metadataFromVideoFrame(const webrtc::VideoFrame& frame)
     108{
     109    VideoSampleMetadata metadata;
     110    if (frame.ntp_time_ms() > 0)
     111        metadata.captureTime = Seconds::fromMilliseconds(frame.ntp_time_ms());
     112    if (isInBounds<uint32_t>(frame.timestamp()))
     113        metadata.rtpTimestamp = frame.timestamp();
     114    auto lastPacketTimestamp = std::max_element(frame.packet_infos().cbegin(), frame.packet_infos().cend(), [](const auto& a, const auto& b) {
     115        return a.receive_time() < b.receive_time();
     116    });
     117    metadata.receiveTime = Seconds::fromMicroseconds(lastPacketTimestamp->receive_time().us());
     118    return metadata;
     119}
     120
    107121} // namespace WebCore
    108122

  • trunk/Source/WebCore/platform/mediastream/RealtimeIncomingVideoSource.h

    r278253 r285984  
    6363    const char* logClassName() const final { return "RealtimeIncomingVideoSource"; }
    6464#endif
    65    
     65
     66    static VideoSampleMetadata metadataFromVideoFrame(const webrtc::VideoFrame&);
     67
    6668private:
    6769    // RealtimeMediaSource API

  • trunk/Source/WebCore/platform/mediastream/RealtimeMediaSource.cpp

    r283033 r285984  
    192192}
    193193
    194 void RealtimeMediaSource::videoSampleAvailable(MediaSample& mediaSample)
     194void RealtimeMediaSource::videoSampleAvailable(MediaSample& mediaSample, VideoSampleMetadata metadata)
    195195{
    196196#if !RELEASE_LOG_DISABLED
    … …  
    212212    Locker locker { m_videoSampleObserversLock };
    213213    for (auto* observer : m_videoSampleObservers)
    214         observer->videoSampleAvailable(mediaSample);
     214        observer->videoSampleAvailable(mediaSample, metadata);
    215215}
    216216

  • trunk/Source/WebCore/platform/mediastream/RealtimeMediaSource.h

    r284857 r285984  
    4343#include "RealtimeMediaSourceCapabilities.h"
    4444#include "RealtimeMediaSourceFactory.h"
     45#include "VideoSampleMetadata.h"
    4546#include <wtf/CompletionHandler.h>
    4647#include <wtf/Lock.h>
    … …  
    103104
    104105        // May be called on a background thread.
    105         virtual void videoSampleAvailable(MediaSample&) = 0;
     106        virtual void videoSampleAvailable(MediaSample&, VideoSampleMetadata) = 0;
    106107    };
    107108
    … …  
    245246    void initializeEchoCancellation(bool echoCancellation) { m_echoCancellation = echoCancellation; }
    246247
    247     void videoSampleAvailable(MediaSample&);
     248    void videoSampleAvailable(MediaSample&, VideoSampleMetadata);
    248249    void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t);
    249250

  • trunk/Source/WebCore/platform/mediastream/RealtimeOutgoingVideoSource.h

    r278340 r285984  
    136136
    137137    // RealtimeMediaSource::VideoSampleObserver API
    138     void videoSampleAvailable(MediaSample&) override { }
     138    void videoSampleAvailable(MediaSample&, VideoSampleMetadata) override { }
    139139
    140140    Ref<MediaStreamTrackPrivate> m_videoSource;

  • trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.cpp

    r279940 r285984  
    387387}
    388388
    389 void RealtimeVideoCaptureSource::dispatchMediaSampleToObservers(MediaSample& sample)
     389void RealtimeVideoCaptureSource::dispatchMediaSampleToObservers(MediaSample& sample, WebCore::VideoSampleMetadata metadata)
    390390{
    391391    MediaTime sampleTime = sample.presentationTime();
    … …  
    401401        m_observedFrameRate = (m_observedFrameTimeStamps.size() / interval);
    402402
    403     videoSampleAvailable(sample);
     403    videoSampleAvailable(sample, metadata);
    404404}
    405405

  • trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.h

    r281305 r285984  
    7373    void updateCapabilities(RealtimeMediaSourceCapabilities&);
    7474
    75     void dispatchMediaSampleToObservers(MediaSample&);
     75    void dispatchMediaSampleToObservers(MediaSample&, WebCore::VideoSampleMetadata);
    7676    const Vector<IntSize>& standardVideoSizes();
    7777

  • trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.cpp

    r282755 r285984  
    180180#endif
    181181
    182 void RealtimeVideoSource::videoSampleAvailable(MediaSample& sample)
     182void RealtimeVideoSource::videoSampleAvailable(MediaSample& sample, VideoSampleMetadata metadata)
    183183{
    184184    if (m_frameDecimation > 1 && ++m_frameDecimationCounter % m_frameDecimation)
    … …  
    194194    if (!size.isEmpty() && size != expandedIntSize(sample.presentationSize())) {
    195195        if (auto mediaSample = adaptVideoSample(sample)) {
    196             RealtimeMediaSource::videoSampleAvailable(*mediaSample);
     196            RealtimeMediaSource::videoSampleAvailable(*mediaSample, metadata);
    197197            return;
    198198        }
    … …  
    200200#endif
    201201
    202     RealtimeMediaSource::videoSampleAvailable(sample);
     202    RealtimeMediaSource::videoSampleAvailable(sample, metadata);
    203203}
    204204

  • trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.h

    r278253 r285984  
    7373
    7474    // RealtimeMediaSource::VideoSampleObserver
    75     void videoSampleAvailable(MediaSample&) final;
     75    void videoSampleAvailable(MediaSample&, VideoSampleMetadata) final;
    7676
    7777#if PLATFORM(COCOA)

  • trunk/Source/WebCore/platform/mediastream/cocoa/DisplayCaptureSourceCocoa.cpp

    r284512 r285984  
    221221        m_imageTransferSession = ImageTransferSessionVT::create(preferedPixelBufferFormat());
    222222
    223     auto sampleTime = MediaTime::createWithDouble((elapsedTime() + 100_ms).seconds());
     223    auto elapsedTime = this->elapsedTime();
     224    auto sampleTime = MediaTime::createWithDouble((elapsedTime + 100_ms).seconds());
    224225
    225226    auto frame = m_capturer->generateFrame();
    … …  
    283284    }
    284285
    285     videoSampleAvailable(*sample.get());
     286    VideoSampleMetadata metadata;
     287    metadata.captureTime = MonotonicTime::now().secondsSinceEpoch();
     288    videoSampleAvailable(*sample.get(), metadata);
    286289}
    287290

  • trunk/Source/WebCore/platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp

    r284368 r285984  
    263263    }
    264264
    265     void videoSampleAvailable(MediaSample& sample) final
     265    void videoSampleAvailable(MediaSample& sample, VideoSampleMetadata) final
    266266    {
    267267        if (!m_parent)

  • trunk/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoCaptureSource.cpp

    r281305 r285984  
    196196        return;
    197197
    198     dispatchMediaSampleToObservers(WTFMove(sample));
     198    dispatchMediaSampleToObservers(WTFMove(sample), { });
    199199}
    200200

  • trunk/Source/WebCore/platform/mediastream/gstreamer/MockRealtimeVideoSourceGStreamer.cpp

    r285071 r285984  
    8989}
    9090
    91 void MockDisplayCaptureSourceGStreamer::videoSampleAvailable(MediaSample& sample)
     91void MockDisplayCaptureSourceGStreamer::videoSampleAvailable(MediaSample& sample, VideoSampleMetadata metadata)
    9292{
    93     RealtimeMediaSource::videoSampleAvailable(sample);
     93    RealtimeMediaSource::videoSampleAvailable(sample, metadata);
    9494}
    9595
    … …  
    155155    auto sample = MediaSampleGStreamer::createImageSample(WTFMove(*pixelBuffer), size(), frameRate(), sampleRotation());
    156156    sample->offsetTimestampsBy(MediaTime::createWithDouble((elapsedTime() + 100_ms).seconds()));
    157     dispatchMediaSampleToObservers(sample.get());
     157    dispatchMediaSampleToObservers(sample.get(), { });
    158158}
    159159

  • trunk/Source/WebCore/platform/mediastream/gstreamer/MockRealtimeVideoSourceGStreamer.h

    r281491 r285984  
    4949protected:
    5050    // RealtimeMediaSource::VideoSampleObserver
    51     void videoSampleAvailable(MediaSample&) final;
     51    void videoSampleAvailable(MediaSample&, VideoSampleMetadata) final;
    5252
    5353private:

  • trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/RealtimeIncomingVideoSourceLibWebRTC.cpp

    r282860 r285984  
    6262        auto gstSample = GStreamerSampleFromLibWebRTCVideoFrame(frame);
    6363        auto sample = MediaSampleGStreamer::create(WTFMove(gstSample), { }, { });
    64         protectedThis->videoSampleAvailable(sample.get());
     64        protectedThis->videoSampleAvailable(sample.get(), { });
    6565    });
    6666}

  • trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.cpp

    r279982 r285984  
    5151}
    5252
    53 void RealtimeOutgoingVideoSourceLibWebRTC::videoSampleAvailable(MediaSample& sample)
     53void RealtimeOutgoingVideoSourceLibWebRTC::videoSampleAvailable(MediaSample& sample, VideoSampleMetadata)
    5454{
    5555    switch (sample.videoRotation()) {

  • trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.h

    r279982 r285984  
    4343
    4444    // RealtimeMediaSource::VideoSampleObserver API
    45     void videoSampleAvailable(MediaSample&) final;
     45    void videoSampleAvailable(MediaSample&, VideoSampleMetadata) final;
    4646};
    4747

  • trunk/Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.mm

    r285068 r285984  
    571571    m_buffer = &sample.get();
    572572    setIntrinsicSize(expandedIntSize(sample->presentationSize()));
    573     dispatchMediaSampleToObservers(WTFMove(sample));
     573    VideoSampleMetadata metadata;
     574    metadata.captureTime = MonotonicTime::now().secondsSinceEpoch();
     575    dispatchMediaSampleToObservers(WTFMove(sample), metadata);
    574576}
    575577

  • trunk/Source/WebCore/platform/mediastream/mac/MockRealtimeVideoSourceMac.mm

    r284135 r285984  
    100100        return;
    101101
    102     m_workQueue->dispatch([this, protectedThis = Ref { *this }, sampleBuffer = WTFMove(sampleBuffer)]() mutable {
    103         dispatchMediaSampleToObservers(*sampleBuffer);
     102    auto captureTime = MonotonicTime::now().secondsSinceEpoch();
     103    m_workQueue->dispatch([this, protectedThis = Ref { *this }, sampleBuffer = WTFMove(sampleBuffer), captureTime]() mutable {
     104        VideoSampleMetadata metadata;
     105        metadata.captureTime = captureTime;
     106        dispatchMediaSampleToObservers(*sampleBuffer, metadata);
    104107    });
    105108}

  • trunk/Source/WebCore/platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.mm

    r283036 r285984  
    197197
    198198    setIntrinsicSize(IntSize(width, height));
    199     videoSampleAvailable(MediaSampleAVFObjC::create(sample.get(), rotation));
     199
     200    videoSampleAvailable(MediaSampleAVFObjC::create(sample.get(), rotation), metadataFromVideoFrame(frame));
    200201}
    201202

  • trunk/Source/WebCore/platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.cpp

    r279335 r285984  
    6262}
    6363
    64 void RealtimeOutgoingVideoSourceCocoa::videoSampleAvailable(MediaSample& sample)
     64void RealtimeOutgoingVideoSourceCocoa::videoSampleAvailable(MediaSample& sample, VideoSampleMetadata)
    6565{
    6666#if !RELEASE_LOG_DISABLED

  • trunk/Source/WebCore/platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.h

    r261553 r285984  
    4646
    4747    // RealtimeMediaSource::VideoSampleObserver API
    48     void videoSampleAvailable(MediaSample&) final;
     48    void videoSampleAvailable(MediaSample&, VideoSampleMetadata) final;
    4949
    5050    RetainPtr<CVPixelBufferRef> convertToYUV(CVPixelBufferRef);

  • trunk/Source/WebCore/testing/Internals.cpp

    r285973 r285984  
    54785478}
    54795479
    5480 void Internals::videoSampleAvailable(MediaSample& sample)
     5480void Internals::videoSampleAvailable(MediaSample& sample, VideoSampleMetadata)
    54815481{
    54825482    callOnMainThread([this, weakThis = WeakPtr { *this }, sample = Ref { sample }] {

  • trunk/Source/WebCore/testing/Internals.h

    r285610 r285984  
    12201220#if ENABLE(MEDIA_STREAM)
    12211221    // RealtimeMediaSource::Observer API
    1222     void videoSampleAvailable(MediaSample&) final;
     1222    void videoSampleAvailable(MediaSample&, VideoSampleMetadata) final;
    12231223    // RealtimeMediaSource::AudioSampleObserver API
    12241224    void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final { m_trackAudioSampleCount++; }

  • trunk/Source/WebKit/ChangeLog

    r285981 r285984  
     12021-11-17  Youenn Fablet  <youenn@apple.com>
     2
     3        Add support for more rvfc metadata
     4        https://bugs.webkit.org/show_bug.cgi?id=233185
     5
     6        Reviewed by Eric Carlson.
     7
     8        Make sure to receive frame metadats, send it through IPC to WebProcess and pipe it to media source observers.
     9
     10        * UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp:
     11        * WebProcess/GPU/webrtc/MediaRecorderPrivate.cpp:
     12        * WebProcess/GPU/webrtc/MediaRecorderPrivate.h:
     13        * WebProcess/cocoa/RemoteCaptureSampleManager.cpp:
     14        * WebProcess/cocoa/RemoteCaptureSampleManager.h:
     15        * WebProcess/cocoa/RemoteCaptureSampleManager.messages.in:
     16        * WebProcess/cocoa/RemoteRealtimeDisplaySource.h:
     17        * WebProcess/cocoa/RemoteRealtimeVideoSource.cpp:
     18        * WebProcess/cocoa/RemoteRealtimeVideoSource.h:
     19
    1202021-11-17  Myles C. Maxfield  <mmaxfield@apple.com>
    221

  • trunk/Source/WebKit/UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp

    r284220 r285984  
    180180    }
    181181
    182     void videoSampleAvailable(MediaSample& sample) final
     182    void videoSampleAvailable(MediaSample& sample, VideoSampleMetadata metadata) final
    183183    {
    184184        std::unique_ptr<RemoteVideoSample> remoteSample;
    … …  
    193193                remoteSample->setOwnershipIdentity(*m_webProcessIdentityToken);
    194194#endif
    195             m_connection->send(Messages::RemoteCaptureSampleManager::VideoSampleAvailable(m_id, WTFMove(*remoteSample)), 0);
     195            m_connection->send(Messages::RemoteCaptureSampleManager::VideoSampleAvailable(m_id, WTFMove(*remoteSample), metadata), 0);
    196196        }
    197197    }

  • trunk/Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderPrivate.cpp

    r284095 r285984  
    9090}
    9191
    92 void MediaRecorderPrivate::videoSampleAvailable(MediaSample& sample)
     92void MediaRecorderPrivate::videoSampleAvailable(MediaSample& sample, VideoSampleMetadata)
    9393{
    9494    std::unique_ptr<RemoteVideoSample> remoteSample;

  • trunk/Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderPrivate.h

    r283431 r285984  
    5757private:
    5858    // WebCore::MediaRecorderPrivate
    59     void videoSampleAvailable(WebCore::MediaSample&) final;
     59    void videoSampleAvailable(WebCore::MediaSample&, WebCore::VideoSampleMetadata) final;
    6060    void fetchData(CompletionHandler<void(RefPtr<WebCore::SharedBuffer>&&, const String& mimeType, double)>&&) final;
    6161    void stopRecording(CompletionHandler<void()>&&) final;

  • trunk/Source/WebKit/WebProcess/cocoa/RemoteCaptureSampleManager.cpp

    r285520 r285984  
    151151}
    152152
    153 void RemoteCaptureSampleManager::videoSampleAvailable(RealtimeMediaSourceIdentifier identifier, RemoteVideoSample&& sample)
     153void RemoteCaptureSampleManager::videoSampleAvailable(RealtimeMediaSourceIdentifier identifier, RemoteVideoSample&& sample, VideoSampleMetadata metadata)
    154154{
    155155    ASSERT(!WTF::isMainRunLoop());
    … …  
    160160        return;
    161161    }
    162     iterator->value->videoSampleAvailable(WTFMove(sample));
     162    iterator->value->videoSampleAvailable(WTFMove(sample), metadata);
    163163}
    164164
    … …  
    233233}
    234234
    235 void RemoteCaptureSampleManager::RemoteVideo::videoSampleAvailable(RemoteVideoSample&& remoteSample)
     235void RemoteCaptureSampleManager::RemoteVideo::videoSampleAvailable(RemoteVideoSample&& remoteSample, VideoSampleMetadata metadata)
    236236{
    237237    if (!m_imageTransferSession || m_imageTransferSession->pixelFormat() != remoteSample.videoFormat())
    … …  
    249249    }
    250250    switchOn(m_source, [&](Ref<RemoteRealtimeVideoSource>& source) {
    251         source->videoSampleAvailable(*sampleRef, remoteSample.size());
     251        source->videoSampleAvailable(*sampleRef, remoteSample.size(), metadata);
    252252    }, [&](Ref<RemoteRealtimeDisplaySource>& source) {
    253         source->remoteVideoSampleAvailable(*sampleRef);
     253        source->remoteVideoSampleAvailable(*sampleRef, metadata);
    254254    });
    255255}

  • trunk/Source/WebKit/WebProcess/cocoa/RemoteCaptureSampleManager.h

    r284444 r285984  
    7171    void audioStorageChanged(WebCore::RealtimeMediaSourceIdentifier, const SharedMemory::IPCHandle&, const WebCore::CAAudioStreamDescription&, uint64_t numberOfFrames, IPC::Semaphore&&, const MediaTime&, size_t frameSampleSize);
    7272    void audioSamplesAvailable(WebCore::RealtimeMediaSourceIdentifier, MediaTime, uint64_t numberOfFrames);
    73     void videoSampleAvailable(WebCore::RealtimeMediaSourceIdentifier, WebCore::RemoteVideoSample&&);
     73    void videoSampleAvailable(WebCore::RealtimeMediaSourceIdentifier, WebCore::RemoteVideoSample&&, WebCore::VideoSampleMetadata);
    7474
    7575    void setConnection(IPC::Connection*);
    … …  
    106106        explicit RemoteVideo(Source&&);
    107107
    108         void videoSampleAvailable(WebCore::RemoteVideoSample&&);
     108        void videoSampleAvailable(WebCore::RemoteVideoSample&&, WebCore::VideoSampleMetadata);
    109109
    110110    private:

  • trunk/Source/WebKit/WebProcess/cocoa/RemoteCaptureSampleManager.messages.in

    r277256 r285984  
    2626messages -> RemoteCaptureSampleManager NotRefCounted {
    2727    AudioStorageChanged(WebCore::RealtimeMediaSourceIdentifier id, WebKit::SharedMemory::IPCHandle storageHandle, WebCore::CAAudioStreamDescription description, uint64_t numberOfFrames, IPC::Semaphore captureSemaphore, MediaTime mediaTime, size_t frameChunkSize);
    28     VideoSampleAvailable(WebCore::RealtimeMediaSourceIdentifier id, WebCore::RemoteVideoSample sample)
     28    VideoSampleAvailable(WebCore::RealtimeMediaSourceIdentifier id, WebCore::RemoteVideoSample sample, struct WebCore::VideoSampleMetadata metadata)
    2929}
    3030

  • trunk/Source/WebKit/WebProcess/cocoa/RemoteRealtimeDisplaySource.h

    r284444 r285984  
    6262    void captureFailed() final;
    6363
    64     void remoteVideoSampleAvailable(WebCore::MediaSample& sample) { videoSampleAvailable(sample); }
     64    void remoteVideoSampleAvailable(WebCore::MediaSample& sample, WebCore::VideoSampleMetadata metadata) { videoSampleAvailable(sample, metadata); }
    6565    void sourceMutedChanged(bool value) { notifyMutedChange(value); }
    6666

  • trunk/Source/WebKit/WebProcess/cocoa/RemoteRealtimeVideoSource.cpp

    r284444 r285984  
    113113}
    114114
    115 void RemoteRealtimeVideoSource::videoSampleAvailable(MediaSample& sample, IntSize sampleSize)
     115void RemoteRealtimeVideoSource::videoSampleAvailable(MediaSample& sample, IntSize sampleSize, VideoSampleMetadata metadata)
    116116{
    117117    ASSERT(type() == Type::Video);
    … …  
    132132        });
    133133    }
    134     dispatchMediaSampleToObservers(sample);
     134    dispatchMediaSampleToObservers(sample, metadata);
    135135}
    136136

  • trunk/Source/WebKit/WebProcess/cocoa/RemoteRealtimeVideoSource.h

    r283033 r285984  
    6767    void captureFailed() final;
    6868
    69     void videoSampleAvailable(WebCore::MediaSample&, WebCore::IntSize);
     69    void videoSampleAvailable(WebCore::MediaSample&, WebCore::IntSize, WebCore::VideoSampleMetadata);
    7070    void sourceMutedChanged(bool value) { notifyMutedChange(value); }
    7171
Note: See TracChangeset for help on using the changeset viewer.