⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 291611 in webkit


Ignore:
Timestamp:
Mar 22, 2022, 3:36:52 AM (5 years ago)
Author:
youenn@apple.com
Message:

Move RealtimeMediaSource::VideoSampleObserver::videoSampleAvailable to VideoFrame
​https://bugs.webkit.org/show_bug.cgi?id=237884

Reviewed by Eric Carlson.

Source/WebCore:

Refactoring to do the following:

  • Renaming of VideoSampleObserver to VideoFrameObserver, videoSampleAvailable to videoFrameAvailable...
  • videoFrameAvailable takes a VideoFrame instead of a MediaSample
  • Use VideoFrame::rotation and VideoFrame::isMirrored.
  • Use VideoFrame::Rotation instead of MediaSample::VideoRotation

Update code according this renaming.
Move from using MediaSample to VideoFrame where it makes sense.

Prepare for removal of MediaSample as parent class of VideoFrame.

  • Modules/mediastream/CanvasCaptureMediaStreamTrack.cpp:
  • html/canvas/WebGLRenderingContextBase.cpp:
  • platform/MediaSample.h:
  • platform/VideoFrame.cpp:
  • platform/VideoFrame.h:
  • platform/graphics/avfoundation/SampleBufferDisplayLayer.h:
  • platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.h:
  • platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.mm:
  • platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:
  • platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm:
  • platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
  • platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
  • platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h:
  • platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm:
  • platform/graphics/cocoa/GraphicsContextGLCocoa.mm:
  • platform/graphics/cv/ImageRotationSessionVT.h:
  • platform/graphics/cv/ImageRotationSessionVT.mm:
  • platform/graphics/cv/ImageTransferSessionVT.h:
  • platform/graphics/cv/ImageTransferSessionVT.mm:
  • platform/graphics/cv/VideoFrameCV.h:
  • platform/graphics/cv/VideoFrameCV.mm:
  • platform/graphics/gstreamer/VideoFrameGStreamer.cpp:
  • platform/graphics/gstreamer/VideoFrameGStreamer.h:
  • platform/mediarecorder/MediaRecorderPrivate.h:
  • platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp:
  • platform/mediarecorder/MediaRecorderPrivateAVFImpl.h:
  • platform/mediarecorder/MediaRecorderPrivateMock.cpp:
  • platform/mediarecorder/MediaRecorderPrivateMock.h:
  • platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.h:
  • platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm:
  • platform/mediastream/RealtimeMediaSource.cpp:
  • platform/mediastream/RealtimeMediaSource.h:
  • platform/mediastream/RealtimeOutgoingVideoSource.cpp:
  • platform/mediastream/RealtimeOutgoingVideoSource.h:
  • platform/mediastream/RealtimeVideoCaptureSource.cpp:
  • platform/mediastream/RealtimeVideoCaptureSource.h:
  • platform/mediastream/RealtimeVideoSource.cpp:
  • platform/mediastream/RealtimeVideoSource.h:
  • platform/mediastream/VideoPreset.h:
  • platform/mediastream/cocoa/DisplayCaptureSourceCocoa.cpp:
  • platform/mediastream/cocoa/DisplayCaptureSourceCocoa.h:
  • platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp:
  • platform/mediastream/gstreamer/MockRealtimeVideoSourceGStreamer.cpp:
  • platform/mediastream/gstreamer/MockRealtimeVideoSourceGStreamer.h:
  • platform/mediastream/ios/ReplayKitCaptureSource.mm:
  • platform/mediastream/libwebrtc/VideoFrameLibWebRTC.cpp:
  • platform/mediastream/libwebrtc/VideoFrameLibWebRTC.h:
  • platform/mediastream/libwebrtc/gstreamer/RealtimeIncomingVideoSourceLibWebRTC.cpp:
  • platform/mediastream/libwebrtc/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.cpp:
  • platform/mediastream/libwebrtc/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.h:
  • platform/mediastream/mac/AVVideoCaptureSource.h:
  • platform/mediastream/mac/AVVideoCaptureSource.mm:
  • platform/mediastream/mac/MockRealtimeVideoSourceMac.mm:
  • platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.h:
  • platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.mm:
  • platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.cpp:
  • platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.h:
  • platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.mm:
  • platform/mock/MockRealtimeVideoSource.cpp:
  • platform/mock/MockRealtimeVideoSource.h:
  • testing/Internals.cpp:
  • testing/Internals.h:

Source/WebKit:

Move from using MediaSample to VideoFrame directly.

  • GPUProcess/graphics/RemoteGraphicsContextGLCocoa.cpp:
  • GPUProcess/media/RemoteVideoFrameObjectHeap.cpp:
  • GPUProcess/media/RemoteVideoFrameObjectHeap.h:
  • GPUProcess/webrtc/LibWebRTCCodecsProxy.mm:
  • GPUProcess/webrtc/RemoteMediaRecorder.cpp:
  • GPUProcess/webrtc/RemoteMediaRecorder.h:
  • GPUProcess/webrtc/RemoteMediaRecorder.messages.in:
  • GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.cpp:
  • GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.h:
  • GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.messages.in:
  • UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp:
  • WebProcess/GPU/media/RemoteVideoFrameProxy.cpp:
  • WebProcess/GPU/media/RemoteVideoFrameProxy.h:
  • WebProcess/GPU/webrtc/LibWebRTCCodecs.cpp:
  • WebProcess/GPU/webrtc/MediaRecorderPrivate.cpp:
  • WebProcess/GPU/webrtc/MediaRecorderPrivate.h:
  • WebProcess/GPU/webrtc/SampleBufferDisplayLayer.cpp:
  • WebProcess/GPU/webrtc/SampleBufferDisplayLayer.h:
  • WebProcess/GPU/webrtc/SharedVideoFrame.cpp:
  • WebProcess/GPU/webrtc/SharedVideoFrame.h:
  • WebProcess/cocoa/RemoteCaptureSampleManager.cpp:
  • WebProcess/cocoa/RemoteCaptureSampleManager.h:
  • WebProcess/cocoa/RemoteCaptureSampleManager.messages.in:
  • WebProcess/cocoa/RemoteRealtimeVideoSource.h:
Location:
trunk/Source
Files:
91 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebCore/ChangeLog

    r291610 r291611  
     12022-03-22  Youenn Fablet  <youenn@apple.com>
     2
     3        Move RealtimeMediaSource::VideoSampleObserver::videoSampleAvailable to VideoFrame
     4        https://bugs.webkit.org/show_bug.cgi?id=237884
     5
     6        Reviewed by Eric Carlson.
     7
     8        Refactoring to do the following:
     9        - Renaming of VideoSampleObserver to VideoFrameObserver, videoSampleAvailable to videoFrameAvailable...
     10        - videoFrameAvailable takes a VideoFrame instead of a MediaSample
     11        - Use VideoFrame::rotation and VideoFrame::isMirrored.
     12        - Use VideoFrame::Rotation instead of MediaSample::VideoRotation
     13        Update code according this renaming.
     14        Move from using MediaSample to VideoFrame where it makes sense.
     15
     16        Prepare for removal of MediaSample as parent class of VideoFrame.
     17
     18        * Modules/mediastream/CanvasCaptureMediaStreamTrack.cpp:
     19        * html/canvas/WebGLRenderingContextBase.cpp:
     20        * platform/MediaSample.h:
     21        * platform/VideoFrame.cpp:
     22        * platform/VideoFrame.h:
     23        * platform/graphics/avfoundation/SampleBufferDisplayLayer.h:
     24        * platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.h:
     25        * platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.mm:
     26        * platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:
     27        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm:
     28        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
     29        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
     30        * platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h:
     31        * platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm:
     32        * platform/graphics/cocoa/GraphicsContextGLCocoa.mm:
     33        * platform/graphics/cv/ImageRotationSessionVT.h:
     34        * platform/graphics/cv/ImageRotationSessionVT.mm:
     35        * platform/graphics/cv/ImageTransferSessionVT.h:
     36        * platform/graphics/cv/ImageTransferSessionVT.mm:
     37        * platform/graphics/cv/VideoFrameCV.h:
     38        * platform/graphics/cv/VideoFrameCV.mm:
     39        * platform/graphics/gstreamer/VideoFrameGStreamer.cpp:
     40        * platform/graphics/gstreamer/VideoFrameGStreamer.h:
     41        * platform/mediarecorder/MediaRecorderPrivate.h:
     42        * platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp:
     43        * platform/mediarecorder/MediaRecorderPrivateAVFImpl.h:
     44        * platform/mediarecorder/MediaRecorderPrivateMock.cpp:
     45        * platform/mediarecorder/MediaRecorderPrivateMock.h:
     46        * platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.h:
     47        * platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm:
     48        * platform/mediastream/RealtimeMediaSource.cpp:
     49        * platform/mediastream/RealtimeMediaSource.h:
     50        * platform/mediastream/RealtimeOutgoingVideoSource.cpp:
     51        * platform/mediastream/RealtimeOutgoingVideoSource.h:
     52        * platform/mediastream/RealtimeVideoCaptureSource.cpp:
     53        * platform/mediastream/RealtimeVideoCaptureSource.h:
     54        * platform/mediastream/RealtimeVideoSource.cpp:
     55        * platform/mediastream/RealtimeVideoSource.h:
     56        * platform/mediastream/VideoPreset.h:
     57        * platform/mediastream/cocoa/DisplayCaptureSourceCocoa.cpp:
     58        * platform/mediastream/cocoa/DisplayCaptureSourceCocoa.h:
     59        * platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp:
     60        * platform/mediastream/gstreamer/MockRealtimeVideoSourceGStreamer.cpp:
     61        * platform/mediastream/gstreamer/MockRealtimeVideoSourceGStreamer.h:
     62        * platform/mediastream/ios/ReplayKitCaptureSource.mm:
     63        * platform/mediastream/libwebrtc/VideoFrameLibWebRTC.cpp:
     64        * platform/mediastream/libwebrtc/VideoFrameLibWebRTC.h:
     65        * platform/mediastream/libwebrtc/gstreamer/RealtimeIncomingVideoSourceLibWebRTC.cpp:
     66        * platform/mediastream/libwebrtc/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.cpp:
     67        * platform/mediastream/libwebrtc/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.h:
     68        * platform/mediastream/mac/AVVideoCaptureSource.h:
     69        * platform/mediastream/mac/AVVideoCaptureSource.mm:
     70        * platform/mediastream/mac/MockRealtimeVideoSourceMac.mm:
     71        * platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.h:
     72        * platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.mm:
     73        * platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.cpp:
     74        * platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.h:
     75        * platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.mm:
     76        * platform/mock/MockRealtimeVideoSource.cpp:
     77        * platform/mock/MockRealtimeVideoSource.h:
     78        * testing/Internals.cpp:
     79        * testing/Internals.h:
     80
    1812022-03-22  Zan Dobersek  <zdobersek@igalia.com>
    282

  • trunk/Source/WebCore/Modules/mediastream/CanvasCaptureMediaStreamTrack.cpp

    r291468 r291611  
    200200    VideoFrameTimeMetadata metadata;
    201201    metadata.captureTime = MonotonicTime::now().secondsSinceEpoch();
    202     videoSampleAvailable(*videoFrame, metadata);
     202    videoFrameAvailable(*videoFrame, metadata);
    203203}
    204204

  • trunk/Source/WebCore/html/canvas/WebGLRenderingContextBase.cpp

    r291477 r291611  
    136136#if ENABLE(OFFSCREEN_CANVAS)
    137137#include "OffscreenCanvas.h"
    138 #endif
    139 
    140 #if ENABLE(MEDIA_STREAM)
    141 #include "MediaSample.h"
    142138#endif
    143139

  • trunk/Source/WebCore/platform/MediaSample.h

    r291216 r291611  
    118118        Left = 270,
    119119    };
    120     virtual VideoRotation videoRotation() const { return VideoRotation::None; }
    121     virtual bool videoMirrored() const { return false; }
    122     virtual uint32_t videoPixelFormat() const { return 0; }
     120    virtual VideoRotation rotation() const { return VideoRotation::None; }
     121    virtual bool isMirrored() const { return false; }
     122    virtual uint32_t pixelFormat() const { return 0; }
    123123#if PLATFORM(COCOA)
    124124    virtual CVPixelBufferRef pixelBuffer() const { return nullptr; };

  • trunk/Source/WebCore/platform/VideoFrame.cpp

    r291078 r291611  
    4949}
    5050
    51 MediaSample::VideoRotation VideoFrame::videoRotation() const
     51VideoFrame::Rotation VideoFrame::rotation() const
    5252{
    5353    return m_rotation;
    5454}
    5555
    56 bool VideoFrame::videoMirrored() const
     56bool VideoFrame::isMirrored() const
    5757{
    5858    return m_isMirrored;
    … …  
    131131        return nullptr;
    132132
    133     return VideoFrameCV::create(presentationTime(), videoMirrored(), videoRotation(), buffer);
     133    return VideoFrameCV::create(presentationTime(), isMirrored(), rotation(), buffer);
    134134}
    135135#endif

  • trunk/Source/WebCore/platform/VideoFrame.h

    r291078 r291611  
    4343    WEBCORE_EXPORT ~VideoFrame();
    4444
     45    using Rotation = MediaSample::VideoRotation;
     46
    4547    // WebCore::MediaSample overrides.
    4648    WEBCORE_EXPORT MediaTime presentationTime() const final;
    47     WEBCORE_EXPORT VideoRotation videoRotation() const final;
    48     WEBCORE_EXPORT bool videoMirrored() const final;
     49    WEBCORE_EXPORT Rotation rotation() const final;
     50    WEBCORE_EXPORT bool isMirrored() const final;
    4951    // FIXME: When VideoFrame is not MediaSample, these will not be needed.
    5052    WEBCORE_EXPORT WebCore::PlatformSample platformSample() const final;

  • trunk/Source/WebCore/platform/graphics/avfoundation/SampleBufferDisplayLayer.h

    r284093 r291611  
    2626#pragma once
    2727
    28 #include "MediaSample.h"
    2928#include "PlatformLayer.h"
     29#include "VideoFrame.h"
    3030#include <wtf/CompletionHandler.h>
    3131#include <wtf/WeakPtr.h>
    … …  
    3737namespace WebCore {
    3838class IntSize;
    39 class MediaSample;
     39class VideoFrame;
    4040
    4141class SampleBufferDisplayLayer {
    … …  
    6262
    6363    virtual void updateAffineTransform(CGAffineTransform) = 0;
    64     virtual void updateBoundsAndPosition(CGRect, MediaSample::VideoRotation) = 0;
     64    virtual void updateBoundsAndPosition(CGRect, VideoFrame::Rotation) = 0;
    6565
    6666    virtual void flush() = 0;
    … …  
    7070    virtual void pause() = 0;
    7171
    72     virtual void enqueueSample(MediaSample&) = 0;
    73     virtual void clearEnqueuedSamples() = 0;
     72    virtual void enqueueVideoFrame(VideoFrame&) = 0;
     73    virtual void clearVideoFrames() = 0;
    7474
    7575    virtual PlatformLayer* rootLayer() = 0;

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.h

    r291421 r291611  
    4848    ~LocalSampleBufferDisplayLayer();
    4949
     50    void enqueueBuffer(CVPixelBufferRef, MediaTime);
     51
    5052    // API used by WebAVSampleBufferStatusChangeListener
    5153    void layerStatusDidChange();
    … …  
    5759    PlatformLayer* displayLayer();
    5860
    59     PlatformLayer* rootLayer() final;
    60 
    6161    enum class ShouldUpdateRootLayer { No, Yes };
    62     void updateRootLayerBoundsAndPosition(CGRect, MediaSample::VideoRotation, ShouldUpdateRootLayer);
     62    void updateRootLayerBoundsAndPosition(CGRect, VideoFrame::Rotation, ShouldUpdateRootLayer);
    6363    void updateRootLayerAffineTransform(CGAffineTransform);
    6464
     65    // SampleBufferDisplayLayer.
     66    PlatformLayer* rootLayer() final;
    6567    void initialize(bool hideRootLayer, IntSize, CompletionHandler<void(bool didSucceed)>&&) final;
    6668#if !RELEASE_LOG_DISABLED
    … …  
    7274
    7375    void updateAffineTransform(CGAffineTransform)  final;
    74     void updateBoundsAndPosition(CGRect, MediaSample::VideoRotation) final;
     76    void updateBoundsAndPosition(CGRect, VideoFrame::Rotation) final;
    7577
    7678    void flush() final;
    … …  
    8082    void pause() final;
    8183
    82     void enqueue(CVPixelBufferRef, MediaTime);
    83     void enqueueSample(MediaSample&) final;
    84     void clearEnqueuedSamples() final;
     84    void enqueueVideoFrame(VideoFrame&) final;
     85    void clearVideoFrames() final;
    8586    void setRenderPolicy(RenderPolicy) final;
    8687
    8788private:
    88     void removeOldSamplesFromPendingQueue();
    89     void addSampleToPendingQueue(MediaSample&);
     89
     90    void removeOldVideoFramesFromPendingQueue();
     91    void addVideoFrameToPendingQueue(Ref<VideoFrame>&&);
    9092    void requestNotificationWhenReadyForVideoData();
    91     void enqueueSampleBuffer(MediaSample&);
    92     void setRootLayerBoundsAndPositions(CGRect, MediaSample::VideoRotation);
     93    void setRootLayerBoundsAndPositions(CGRect, VideoFrame::Rotation);
    9394
    9495#if !RELEASE_LOG_DISABLED
    … …  
    105106
    106107    // Only accessed through m_processingQueue or if m_processingQueue is null.
    107     using PendingSampleQueue = Deque<Ref<MediaSample>>;
    108     PendingSampleQueue m_pendingVideoSampleQueue;
     108    using PendingSampleQueue = Deque<Ref<VideoFrame>>;
     109    PendingSampleQueue m_pendingVideoFrameQueue;
    109110   
    110111    bool m_paused { false };

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.mm

    r291421 r291611  
    3232#import "IntSize.h"
    3333#import "Logging.h"
    34 #import "MediaSampleAVFObjC.h"
    3534#import "MediaUtilities.h"
    3635
    … …  
    4342#import <wtf/cf/TypeCastsCF.h>
    4443
     44#import <pal/avfoundation/MediaTimeAVFoundation.h>
    4545#import <pal/spi/cocoa/AVFoundationSPI.h>
    46 
     46#import <pal/cf/CoreMediaSoftLink.h>
    4747#import <pal/cocoa/AVFoundationSoftLink.h>
    4848
    … …  
    199199    [m_statusChangeListener stop];
    200200
    201     m_pendingVideoSampleQueue.clear();
     201    m_pendingVideoFrameQueue.clear();
    202202
    203203    [m_sampleBufferDisplayLayer stopRequestingMediaData];
    … …  
    266266}
    267267
    268 void LocalSampleBufferDisplayLayer::updateBoundsAndPosition(CGRect bounds, MediaSample::VideoRotation rotation)
     268void LocalSampleBufferDisplayLayer::updateBoundsAndPosition(CGRect bounds, VideoFrame::Rotation rotation)
    269269{
    270270    updateRootLayerBoundsAndPosition(bounds, rotation, ShouldUpdateRootLayer::No);
    271271}
    272272
    273 void LocalSampleBufferDisplayLayer::setRootLayerBoundsAndPositions(CGRect bounds, MediaSample::VideoRotation rotation)
     273void LocalSampleBufferDisplayLayer::setRootLayerBoundsAndPositions(CGRect bounds, VideoFrame::Rotation rotation)
    274274{
    275275    CGPoint position = { bounds.size.width / 2, bounds.size.height / 2};
    276     if (rotation == MediaSample::VideoRotation::Right || rotation == MediaSample::VideoRotation::Left)
     276    if (rotation == VideoFrame::Rotation::Right || rotation == VideoFrame::Rotation::Left)
    277277        std::swap(bounds.size.width, bounds.size.height);
    278278
    … …  
    281281}
    282282
    283 void LocalSampleBufferDisplayLayer::updateRootLayerBoundsAndPosition(CGRect bounds, MediaSample::VideoRotation rotation, ShouldUpdateRootLayer shouldUpdateRootLayer)
     283void LocalSampleBufferDisplayLayer::updateRootLayerBoundsAndPosition(CGRect bounds, VideoFrame::Rotation rotation, ShouldUpdateRootLayer shouldUpdateRootLayer)
    284284{
    285285    runWithoutAnimations([&] {
    … …  
    287287            setRootLayerBoundsAndPositions(bounds, rotation);
    288288
    289         if (rotation == MediaSample::VideoRotation::Right || rotation == MediaSample::VideoRotation::Left)
     289        if (rotation == VideoFrame::Rotation::Right || rotation == VideoFrame::Rotation::Left)
    290290            std::swap(bounds.size.width, bounds.size.height);
    291291
    … …  
    313313static const double rendererLatency = 0.02;
    314314
    315 void LocalSampleBufferDisplayLayer::enqueueSample(MediaSample& sample)
     315void LocalSampleBufferDisplayLayer::enqueueVideoFrame(VideoFrame& videoFrame)
    316316{
    317317    if (m_paused) {
    … …  
    322322    }
    323323
    324     m_processingQueue->dispatch([this, sample = Ref { sample }] {
     324    m_processingQueue->dispatch([this, videoFrame = Ref { videoFrame }]() mutable {
    325325        if (![m_sampleBufferDisplayLayer isReadyForMoreMediaData]) {
    326326            RELEASE_LOG(WebRTC, "LocalSampleBufferDisplayLayer::enqueueSample (%{public}s) not ready for more media data", m_logIdentifier.utf8().data());
    327             addSampleToPendingQueue(sample);
     327            addVideoFrameToPendingQueue(WTFMove(videoFrame));
    328328            requestNotificationWhenReadyForVideoData();
    329329            return;
    330330        }
    331         enqueueSampleBuffer(sample);
    332     });
    333 }
    334 
    335 void LocalSampleBufferDisplayLayer::enqueueSampleBuffer(MediaSample& sample)
    336 {
    337     enqueue(sample.pixelBuffer(), sample.presentationTime());
     331        enqueueBuffer(videoFrame->pixelBuffer(), videoFrame->presentationTime());
     332    });
    338333}
    339334
    … …  
    347342}
    348343
    349 void LocalSampleBufferDisplayLayer::enqueue(CVPixelBufferRef pixelBuffer, MediaTime presentationTime)
     344void LocalSampleBufferDisplayLayer::enqueueBuffer(CVPixelBufferRef pixelBuffer, MediaTime presentationTime)
    350345{
    351346    ASSERT(!isMainThread());
    … …  
    362357    if (!(m_frameRateMonitor.frameCount() % frameCountPerLog)) {
    363358        if (auto* metrics = [m_sampleBufferDisplayLayer videoPerformanceMetrics])
    364             RELEASE_LOG(WebRTC, "LocalSampleBufferDisplayLayer (%{public}s) metrics, total=%lu, dropped=%lu, corrupted=%lu, display-composited=%lu, non-display-composited=%lu (pending=%lu)", m_logIdentifier.utf8().data(), metrics.totalNumberOfVideoFrames, metrics.numberOfDroppedVideoFrames, metrics.numberOfCorruptedVideoFrames, metrics.numberOfDisplayCompositedVideoFrames, metrics.numberOfNonDisplayCompositedVideoFrames, m_pendingVideoSampleQueue.size());
     359            RELEASE_LOG(WebRTC, "LocalSampleBufferDisplayLayer (%{public}s) metrics, total=%lu, dropped=%lu, corrupted=%lu, display-composited=%lu, non-display-composited=%lu (pending=%lu)", m_logIdentifier.utf8().data(), metrics.totalNumberOfVideoFrames, metrics.numberOfDroppedVideoFrames, metrics.numberOfCorruptedVideoFrames, metrics.numberOfDisplayCompositedVideoFrames, metrics.numberOfNonDisplayCompositedVideoFrames, m_pendingVideoFrameQueue.size());
    365360    }
    366361    m_frameRateMonitor.update();
    … …  
    374369        if (!weakThis)
    375370            return;
    376         RELEASE_LOG(WebRTC, "LocalSampleBufferDisplayLayer::enqueueSampleBuffer (%{public}s) at %f, previous frame was at %f, observed frame rate is %f, delay since last frame is %f ms, frame count is %lu", weakThis->m_logIdentifier.utf8().data(), frameTime, lastFrameTime, observedFrameRate, (frameTime - lastFrameTime) * 1000, frameCount);
    377     });
    378 }
    379 #endif
    380 
    381 void LocalSampleBufferDisplayLayer::removeOldSamplesFromPendingQueue()
     371        RELEASE_LOG(WebRTC, "LocalSampleBufferDisplayLayer::enqueueVideoFrame (%{public}s) at %f, previous frame was at %f, observed frame rate is %f, delay since last frame is %f ms, frame count is %lu", weakThis->m_logIdentifier.utf8().data(), frameTime, lastFrameTime, observedFrameRate, (frameTime - lastFrameTime) * 1000, frameCount);
     372    });
     373}
     374#endif
     375
     376void LocalSampleBufferDisplayLayer::removeOldVideoFramesFromPendingQueue()
    382377{
    383378    ASSERT(!isMainThread());
    384379
    385     if (m_pendingVideoSampleQueue.isEmpty())
     380    if (m_pendingVideoFrameQueue.isEmpty())
    386381        return;
    387382
    388383    if (m_renderPolicy == RenderPolicy::Immediately) {
    389         m_pendingVideoSampleQueue.clear();
     384        m_pendingVideoFrameQueue.clear();
    390385        return;
    391386    }
    392387
    393388    auto now = MediaTime::createWithDouble(MonotonicTime::now().secondsSinceEpoch().value());
    394     while (!m_pendingVideoSampleQueue.isEmpty()) {
    395         auto presentationTime = m_pendingVideoSampleQueue.first()->presentationTime();
     389    while (!m_pendingVideoFrameQueue.isEmpty()) {
     390        auto presentationTime = m_pendingVideoFrameQueue.first()->presentationTime();
    396391        if (presentationTime.isValid() && presentationTime > now)
    397392            break;
    398         m_pendingVideoSampleQueue.removeFirst();
    399     }
    400 }
    401 
    402 void LocalSampleBufferDisplayLayer::addSampleToPendingQueue(MediaSample& sample)
     393        m_pendingVideoFrameQueue.removeFirst();
     394    }
     395}
     396
     397void LocalSampleBufferDisplayLayer::addVideoFrameToPendingQueue(Ref<VideoFrame>&& videoFrame)
    403398{
    404399    ASSERT(!isMainThread());
    405400
    406     removeOldSamplesFromPendingQueue();
    407     m_pendingVideoSampleQueue.append(sample);
    408 }
    409 
    410 void LocalSampleBufferDisplayLayer::clearEnqueuedSamples()
     401    removeOldVideoFramesFromPendingQueue();
     402    m_pendingVideoFrameQueue.append(WTFMove(videoFrame));
     403}
     404
     405void LocalSampleBufferDisplayLayer::clearVideoFrames()
    411406{
    412407    m_processingQueue->dispatch([this] {
    413         m_pendingVideoSampleQueue.clear();
     408        m_pendingVideoFrameQueue.clear();
    414409    });
    415410}
    … …  
    425420            [m_sampleBufferDisplayLayer stopRequestingMediaData];
    426421
    427             while (!m_pendingVideoSampleQueue.isEmpty()) {
     422            while (!m_pendingVideoFrameQueue.isEmpty()) {
    428423                if (![m_sampleBufferDisplayLayer isReadyForMoreMediaData]) {
    429424                    requestNotificationWhenReadyForVideoData();
    430425                    return;
    431426                }
    432                 auto sample = m_pendingVideoSampleQueue.takeFirst();
    433                 enqueueSampleBuffer(sample);
     427                auto videoFrame = m_pendingVideoFrameQueue.takeFirst();
     428                enqueueBuffer(videoFrame->pixelBuffer(), videoFrame->presentationTime());
    434429            }
    435430        });

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm

    r291326 r291611  
    27552755    if (!m_lastPixelBuffer)
    27562756        return nullptr;
    2757     return VideoFrameCV::create(currentMediaTime(), false, VideoFrameCV::VideoRotation::None, RetainPtr { m_lastPixelBuffer });
     2757    return VideoFrameCV::create(currentMediaTime(), false, VideoFrame::Rotation::None, RetainPtr { m_lastPixelBuffer });
    27582758}
    27592759

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm

    r290590 r291611  
    748748    if (!m_lastPixelBuffer)
    749749        return nullptr;
    750     return VideoFrameCV::create(currentMediaTime(), false, VideoFrameCV::VideoRotation::None, RetainPtr { m_lastPixelBuffer });
     750    return VideoFrameCV::create(currentMediaTime(), false, VideoFrame::Rotation::None, RetainPtr { m_lastPixelBuffer });
    751751}
    752752

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h

    r291269 r291611  
    2929
    3030#include "MediaPlayerPrivate.h"
    31 #include "MediaSample.h"
    3231#include "MediaStreamPrivate.h"
    3332#include "SampleBufferDisplayLayer.h"
     33#include "VideoFrame.h"
    3434#include <wtf/Deque.h>
    3535#include <wtf/Forward.h>
    … …  
    5353    , private MediaStreamPrivate::Observer
    5454    , public MediaStreamTrackPrivate::Observer
    55     , public RealtimeMediaSource::VideoSampleObserver
     55    , public RealtimeMediaSource::VideoFrameObserver
    5656    , public SampleBufferDisplayLayer::Client
    5757    , private LoggerHelper
    … …  
    137137    void flushRenderers();
    138138
    139     void processNewVideoSample(MediaSample&, VideoFrameTimeMetadata, Seconds);
    140     void enqueueVideoSample(MediaSample&);
    141     void reenqueueCurrentVideoSampleIfNeeded();
     139    void processNewVideoFrame(VideoFrame&, VideoFrameTimeMetadata, Seconds);
     140    void enqueueVideoFrame(VideoFrame&);
     141    void reenqueueCurrentVideoFrameIfNeeded();
    142142    void requestNotificationWhenReadyForVideoData();
    143143
    … …  
    212212    void readyStateChanged(MediaStreamTrackPrivate&) override;
    213213
    214     // RealtimeMediaSouce::VideoSampleObserver
    215     void videoSampleAvailable(MediaSample&, VideoFrameTimeMetadata) final;
     214    // RealtimeMediaSouce::VideoFrameObserver
     215    void videoFrameAvailable(VideoFrame&, VideoFrameTimeMetadata) final;
    216216
    217217    RetainPtr<PlatformLayer> createVideoFullscreenLayer() override;
    … …  
    239239
    240240        RefPtr<NativeImage> cgImage;
    241         RefPtr<MediaSample> mediaSample;
     241        RefPtr<VideoFrame> videoFrame;
    242242        std::unique_ptr<PixelBufferConformerCV> pixelBufferConformer;
    243243    };
    … …  
    261261    bool m_canEnqueueDisplayLayer { false };
    262262    // Used on sample thread.
    263     MediaSample::VideoRotation m_videoRotation { MediaSample::VideoRotation::None };
     263    VideoFrame::Rotation m_videoRotation { VideoFrame::Rotation::None };
    264264    bool m_videoMirrored { false };
    265265
    … …  
    273273    RetainPtr<WebRootSampleBufferBoundsChangeListener> m_boundsChangeListener;
    274274
    275     Lock m_currentVideoSampleLock;
    276     RefPtr<MediaSample> m_currentVideoSample WTF_GUARDED_BY_LOCK(m_currentVideoSampleLock);
     275    Lock m_currentVideoFrameLock;
     276    RefPtr<VideoFrame> m_currentVideoFrame WTF_GUARDED_BY_LOCK(m_currentVideoFrameLock);
    277277
    278278    bool m_playing { false };

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm

    r291269 r291611  
    3434#import "Logging.h"
    3535#import "MediaPlayer.h"
    36 #import "MediaSampleAVFObjC.h"
    3736#import "MediaSessionManagerCocoa.h"
    3837#import "MediaStreamPrivate.h"
    … …  
    161160
    162161    if (m_activeVideoTrack)
    163         m_activeVideoTrack->streamTrack().source().removeVideoSampleObserver(*this);
     162        m_activeVideoTrack->streamTrack().source().removeVideoFrameObserver(*this);
    164163
    165164    [m_boundsChangeListener invalidate];
    … …  
    230229#pragma mark AVSampleBuffer Methods
    231230
    232 static inline CGAffineTransform videoTransformationMatrix(MediaSample& sample)
    233 {
    234     auto size = sample.presentationSize();
     231static inline CGAffineTransform videoTransformationMatrix(VideoFrame& videoFrame)
     232{
     233    auto size = videoFrame.presentationSize();
    235234    size_t width = static_cast<size_t>(size.width());
    236235    size_t height = static_cast<size_t>(size.height());
    … …  
    238237        return CGAffineTransformIdentity;
    239238
    240     auto videoTransform = CGAffineTransformMakeRotation(static_cast<int>(sample.videoRotation()) * M_PI / 180);
    241     if (sample.videoMirrored())
     239    auto videoTransform = CGAffineTransformMakeRotation(static_cast<int>(videoFrame.rotation()) * M_PI / 180);
     240    if (videoFrame.isMirrored())
    242241        videoTransform = CGAffineTransformScale(videoTransform, -1, 1);
    243242
    … …  
    245244}
    246245
    247 void MediaPlayerPrivateMediaStreamAVFObjC::videoSampleAvailable(MediaSample& sample, VideoFrameTimeMetadata metadata)
     246void MediaPlayerPrivateMediaStreamAVFObjC::videoFrameAvailable(VideoFrame& videoFrame, VideoFrameTimeMetadata metadata)
    248247{
    249248    auto presentationTime = MonotonicTime::now().secondsSinceEpoch();
    250     processNewVideoSample(sample, metadata, presentationTime);
    251     enqueueVideoSample(sample);
    252 }
    253 
    254 void MediaPlayerPrivateMediaStreamAVFObjC::enqueueVideoSample(MediaSample& sample)
     249    processNewVideoFrame(videoFrame, metadata, presentationTime);
     250    enqueueVideoFrame(videoFrame);
     251}
     252
     253void MediaPlayerPrivateMediaStreamAVFObjC::enqueueVideoFrame(VideoFrame& videoFrame)
    255254{
    256255    if (!m_isPageVisible || !m_isVisibleInViewPort)
    … …  
    264263        return;
    265264
    266     if (sample.videoRotation() != m_videoRotation || sample.videoMirrored() != m_videoMirrored) {
    267         m_videoRotation = sample.videoRotation();
    268         m_videoMirrored = sample.videoMirrored();
     265    if (videoFrame.rotation() != m_videoRotation || videoFrame.isMirrored() != m_videoMirrored) {
     266        m_videoRotation = videoFrame.rotation();
     267        m_videoMirrored = videoFrame.isMirrored();
    269268        m_shouldUpdateDisplayLayer = true;
    270269    }
    271270    if (m_shouldUpdateDisplayLayer) {
    272         m_sampleBufferDisplayLayer->updateAffineTransform(videoTransformationMatrix(sample));
     271        m_sampleBufferDisplayLayer->updateAffineTransform(videoTransformationMatrix(videoFrame));
    273272        m_sampleBufferDisplayLayer->updateBoundsAndPosition(m_sampleBufferDisplayLayer->rootLayer().bounds, m_videoRotation);
    274273        m_shouldUpdateDisplayLayer = false;
    275274    }
    276275
    277     m_sampleBufferDisplayLayer->enqueueSample(sample);
    278 }
    279 
    280 void MediaPlayerPrivateMediaStreamAVFObjC::reenqueueCurrentVideoSampleIfNeeded()
    281 {
    282     if (!m_currentVideoSampleLock.tryLock())
    283         return;
    284     Locker locker { AdoptLock, m_currentVideoSampleLock };
    285 
    286     if (!m_currentVideoSample && !m_imagePainter.mediaSample)
    287         return;
    288 
    289     enqueueVideoSample(m_currentVideoSample ? *m_currentVideoSample : *m_imagePainter.mediaSample);
    290 }
    291 
    292 void MediaPlayerPrivateMediaStreamAVFObjC::processNewVideoSample(MediaSample& sample, VideoFrameTimeMetadata metadata, Seconds presentationTime)
     276    m_sampleBufferDisplayLayer->enqueueVideoFrame(videoFrame);
     277}
     278
     279void MediaPlayerPrivateMediaStreamAVFObjC::reenqueueCurrentVideoFrameIfNeeded()
     280{
     281    if (!m_currentVideoFrameLock.tryLock())
     282        return;
     283    Locker locker { AdoptLock, m_currentVideoFrameLock };
     284
     285    if (!m_currentVideoFrame && !m_imagePainter.videoFrame)
     286        return;
     287
     288    enqueueVideoFrame(m_currentVideoFrame ? *m_currentVideoFrame : *m_imagePainter.videoFrame);
     289}
     290
     291void MediaPlayerPrivateMediaStreamAVFObjC::processNewVideoFrame(VideoFrame& videoFrame, VideoFrameTimeMetadata metadata, Seconds presentationTime)
    293292{
    294293    if (!isMainThread()) {
    295294        {
    296             Locker locker { m_currentVideoSampleLock };
    297             m_currentVideoSample = &sample;
     295            Locker locker { m_currentVideoFrameLock };
     296            m_currentVideoFrame = &videoFrame;
    298297        }
    299298        callOnMainThread([weakThis = WeakPtr { *this }, metadata, presentationTime]() mutable {
    … …  
    301300                return;
    302301
    303             RefPtr<MediaSample> sample;
     302            RefPtr<VideoFrame> videoFrame;
    304303            {
    305                 Locker locker { weakThis->m_currentVideoSampleLock };
    306                 sample = WTFMove(weakThis->m_currentVideoSample);
     304                Locker locker { weakThis->m_currentVideoFrameLock };
     305                videoFrame = WTFMove(weakThis->m_currentVideoFrame);
    307306            }
    308             if (!sample)
     307            if (!videoFrame)
    309308                return;
    310309
    311             weakThis->processNewVideoSample(*sample, metadata, presentationTime);
     310            weakThis->processNewVideoFrame(*videoFrame, metadata, presentationTime);
    312311        });
    313312        return;
    … …  
    317316        return;
    318317
    319     if (!m_imagePainter.mediaSample || m_displayMode != PausedImage) {
    320         m_imagePainter.mediaSample = &sample;
     318    if (!m_imagePainter.videoFrame || m_displayMode != PausedImage) {
     319        m_imagePainter.videoFrame = &videoFrame;
    321320        m_imagePainter.cgImage = nullptr;
    322321        if (m_readyState < MediaPlayer::ReadyState::HaveEnoughData)
    … …  
    325324
    326325    m_presentationTime = presentationTime;
    327     m_videoFrameSize = sample.presentationSize();
    328     if (sample.videoRotation() == MediaSample::VideoRotation::Left || sample.videoRotation() == MediaSample::VideoRotation::Right)
     326    m_videoFrameSize = videoFrame.presentationSize();
     327    if (videoFrame.rotation() == VideoFrame::Rotation::Left || videoFrame.rotation() == VideoFrame::Rotation::Right)
    329328        m_videoFrameSize = { m_videoFrameSize.height(), m_videoFrameSize.width() };
    330329    m_sampleMetadata = metadata;
    … …  
    361360    if (m_sampleBufferDisplayLayer && m_sampleBufferDisplayLayer->didFail()) {
    362361        flushRenderers();
    363         if (m_imagePainter.mediaSample)
    364             enqueueVideoSample(*m_imagePainter.mediaSample);
     362        if (m_imagePainter.videoFrame)
     363            enqueueVideoFrame(*m_imagePainter.videoFrame);
    365364        updateDisplayMode();
    366365    }
    … …  
    570569        m_sampleBufferDisplayLayer->play();
    571570    updateDisplayMode();
    572     reenqueueCurrentVideoSampleIfNeeded();
     571    reenqueueCurrentVideoFrameIfNeeded();
    573572
    574573    scheduleDeferredTask([this] {
    … …  
    642641    m_isPageVisible = isVisible;
    643642    flushRenderers();
    644     reenqueueCurrentVideoSampleIfNeeded();
     643    reenqueueCurrentVideoFrameIfNeeded();
    645644}
    646645
    … …  
    682681        return MediaPlayer::ReadyState::HaveNothing;
    683682
    684     bool waitingForImage = activeVideoTrack() && !m_imagePainter.mediaSample;
     683    bool waitingForImage = activeVideoTrack() && !m_imagePainter.videoFrame;
    685684    if (waitingForImage && (!m_haveSeenMetadata || m_waitingForFirstImage))
    686685        return MediaPlayer::ReadyState::HaveNothing;
    … …  
    741740        if (m_player)
    742741            m_player->renderingModeChanged();
    743         reenqueueCurrentVideoSampleIfNeeded();
     742        reenqueueCurrentVideoFrameIfNeeded();
    744743    });
    745744}
    … …  
    895894    if (m_sampleBufferDisplayLayer) {
    896895        if (!m_activeVideoTrack)
    897             m_sampleBufferDisplayLayer->clearEnqueuedSamples();
     896            m_sampleBufferDisplayLayer->clearVideoFrames();
    898897        m_sampleBufferDisplayLayer->updateDisplayMode(hideVideoLayer || m_displayMode < PausedImage, hideRootLayer());
    899898    }
    … …  
    903902    if (oldVideoTrack != m_activeVideoTrack) {
    904903        if (oldVideoTrack)
    905             oldVideoTrack->streamTrack().source().removeVideoSampleObserver(*this);
     904            oldVideoTrack->streamTrack().source().removeVideoFrameObserver(*this);
    906905        if (m_activeVideoTrack) {
    907906            if (m_sampleBufferDisplayLayer && m_activeVideoTrack->streamTrack().source().isCaptureSource())
    908907                m_sampleBufferDisplayLayer->setRenderPolicy(SampleBufferDisplayLayer::RenderPolicy::Immediately);
    909             m_activeVideoTrack->streamTrack().source().addVideoSampleObserver(*this);
     908            m_activeVideoTrack->streamTrack().source().addVideoFrameObserver(*this);
    910909            ALWAYS_LOG(LOGIDENTIFIER, "observing video source ", m_activeVideoTrack->streamTrack().logIdentifier());
    911910        }
    … …  
    988987void MediaPlayerPrivateMediaStreamAVFObjC::updateCurrentFrameImage()
    989988{
    990     if (m_imagePainter.cgImage || !m_imagePainter.mediaSample)
     989    if (m_imagePainter.cgImage || !m_imagePainter.videoFrame)
    991990        return;
    992991
    … …  
    998997        return;
    999998
    1000     if (auto pixelBuffer = m_imagePainter.mediaSample->pixelBuffer())
     999    if (auto pixelBuffer = m_imagePainter.videoFrame->pixelBuffer())
    10011000        m_imagePainter.cgImage = NativeImage::create(m_imagePainter.pixelBufferConformer->createImageFromPixelBuffer(pixelBuffer));
    10021001}
    … …  
    10071006        return;
    10081007
    1009     if (m_displayMode != PaintItBlack && m_imagePainter.mediaSample)
     1008    if (m_displayMode != PaintItBlack && m_imagePainter.videoFrame)
    10101009        updateCurrentFrameImage();
    10111010
    … …  
    10161015    }
    10171016
    1018     if (!m_imagePainter.cgImage || !m_imagePainter.mediaSample)
     1017    if (!m_imagePainter.cgImage || !m_imagePainter.videoFrame)
    10191018        return;
    10201019
    10211020    auto& image = m_imagePainter.cgImage;
    10221021    FloatRect imageRect { FloatPoint::zero(), image->size() };
    1023     AffineTransform videoTransform = videoTransformationMatrix(*m_imagePainter.mediaSample);
     1022    AffineTransform videoTransform = videoTransformationMatrix(*m_imagePainter.videoFrame);
    10241023    FloatRect transformedDestRect = valueOrDefault(videoTransform.inverse()).mapRect(destRect);
    10251024    context.concatCTM(videoTransform);
    … …  
    10331032    if (m_displayMode == PaintItBlack)
    10341033        return nullptr;
    1035     if (!m_imagePainter.mediaSample)
    1036         return nullptr;
    1037     if (is<VideoFrame>(*m_imagePainter.mediaSample))
    1038         return &downcast<VideoFrame>(*m_imagePainter.mediaSample);
    1039     ASSERT(is<MediaSampleAVFObjC>(*m_imagePainter.mediaSample));
    1040     return downcast<MediaSampleAVFObjC>(*m_imagePainter.mediaSample).videoFrame();
     1034    return m_imagePainter.videoFrame;
    10411035}
    10421036
    … …  
    11121106{
    11131107    cgImage = nullptr;
    1114     mediaSample = nullptr;
     1108    videoFrame = nullptr;
    11151109    pixelBufferConformer = nullptr;
    11161110}

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h

    r291550 r291611  
    6666    WEBCORE_EXPORT Ref<MediaSample> createNonDisplayingCopy() const override;
    6767
    68     VideoRotation videoRotation() const override { return m_rotation; }
    69     bool videoMirrored() const override { return m_mirrored; }
    70     WEBCORE_EXPORT uint32_t videoPixelFormat() const final;
     68    VideoRotation rotation() const override { return m_rotation; }
     69    bool isMirrored() const override { return m_mirrored; }
     70    WEBCORE_EXPORT uint32_t pixelFormat() const final;
    7171    WEBCORE_EXPORT CVPixelBufferRef pixelBuffer() const final;
    7272    WEBCORE_EXPORT void setOwnershipIdentity(const ProcessIdentity&) final;

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm

    r291550 r291611  
    111111}
    112112
    113 uint32_t MediaSampleAVFObjC::videoPixelFormat() const
     113uint32_t MediaSampleAVFObjC::pixelFormat() const
    114114{
    115115    auto pixelBuffer = static_cast<CVPixelBufferRef>(PAL::CMSampleBufferGetImageBuffer(m_sample.get()));

  • trunk/Source/WebCore/platform/graphics/cocoa/GraphicsContextGLCocoa.mm

    r291566 r291611  
    802802    if (m_resourceOwner)
    803803        setOwnershipIdentityForCVPixelBuffer(mediaSamplePixelBuffer.get(), m_resourceOwner);
    804     return VideoFrameCV::create({ }, false, VideoFrame::VideoRotation::None, WTFMove(mediaSamplePixelBuffer));
     804    return VideoFrameCV::create({ }, false, VideoFrame::Rotation::None, WTFMove(mediaSamplePixelBuffer));
    805805}
    806806#endif

  • trunk/Source/WebCore/platform/graphics/cv/ImageRotationSessionVT.h

    r289365 r291611  
    3535namespace WebCore {
    3636
    37 class MediaSample;
     37class VideoFrame;
    3838
    3939class ImageRotationSessionVT final {
    … …  
    6161
    6262    RetainPtr<CVPixelBufferRef> rotate(CVPixelBufferRef);
    63     WEBCORE_EXPORT RetainPtr<CVPixelBufferRef> rotate(MediaSample&, const RotationProperties&, IsCGImageCompatible);
     63    WEBCORE_EXPORT RetainPtr<CVPixelBufferRef> rotate(VideoFrame&, const RotationProperties&, IsCGImageCompatible);
    6464
    6565private:

  • trunk/Source/WebCore/platform/graphics/cv/ImageRotationSessionVT.mm

    r291550 r291611  
    3030#import "CVUtilities.h"
    3131#import "Logging.h"
    32 #import "MediaSample.h"
     32#import "VideoFrame.h"
    3333
    3434#import "CoreVideoSoftLink.h"
    … …  
    128128}
    129129
    130 RetainPtr<CVPixelBufferRef> ImageRotationSessionVT::rotate(MediaSample& videoFrame, const RotationProperties& rotation, IsCGImageCompatible cgImageCompatible)
     130RetainPtr<CVPixelBufferRef> ImageRotationSessionVT::rotate(VideoFrame& videoFrame, const RotationProperties& rotation, IsCGImageCompatible cgImageCompatible)
    131131{
    132132    auto pixelBuffer = videoFrame.pixelBuffer();

  • trunk/Source/WebCore/platform/graphics/cv/ImageTransferSessionVT.h

    r291550 r291611  
    4646    }
    4747
    48     RefPtr<MediaSample> convertVideoFrame(MediaSample&, const IntSize&);
    49     RefPtr<MediaSample> createMediaSample(CGImageRef, const MediaTime&, const IntSize&, MediaSample::VideoRotation = MediaSample::VideoRotation::None, bool mirrored = false);
    50     RefPtr<MediaSample> createMediaSample(CMSampleBufferRef, const MediaTime&, const IntSize&, MediaSample::VideoRotation = MediaSample::VideoRotation::None, bool mirrored = false);
     48    RefPtr<VideoFrame> convertVideoFrame(VideoFrame&, const IntSize&);
     49    RefPtr<VideoFrame> createVideoFrame(CGImageRef, const MediaTime&, const IntSize&, VideoFrame::Rotation = VideoFrame::Rotation::None, bool mirrored = false);
     50    RefPtr<VideoFrame> createVideoFrame(CMSampleBufferRef, const MediaTime&, const IntSize&, VideoFrame::Rotation = VideoFrame::Rotation::None, bool mirrored = false);
    5151
    5252#if !PLATFORM(MACCATALYST)
    53     WEBCORE_EXPORT RefPtr<MediaSample> createMediaSample(IOSurfaceRef, const MediaTime&, const IntSize&, MediaSample::VideoRotation = MediaSample::VideoRotation::None, bool mirrored = false);
     53    WEBCORE_EXPORT RefPtr<VideoFrame> createVideoFrame(IOSurfaceRef, const MediaTime&, const IntSize&, VideoFrame::Rotation = VideoFrame::Rotation::None, bool mirrored = false);
    5454#endif
    5555

  • trunk/Source/WebCore/platform/graphics/cv/ImageTransferSessionVT.mm

    r291550 r291611  
    256256#endif
    257257
    258 RefPtr<MediaSample> ImageTransferSessionVT::convertVideoFrame(MediaSample& videoFrame, const IntSize& size)
     258RefPtr<VideoFrame> ImageTransferSessionVT::convertVideoFrame(VideoFrame& videoFrame, const IntSize& size)
    259259{
    260260    if (size == expandedIntSize(videoFrame.presentationSize()))
    … …  
    265265        return nullptr;
    266266
    267     return VideoFrameCV::create(videoFrame.presentationTime(), videoFrame.videoMirrored(), videoFrame.videoRotation(), WTFMove(resizedBuffer));
     267    return VideoFrameCV::create(videoFrame.presentationTime(), videoFrame.isMirrored(), videoFrame.rotation(), WTFMove(resizedBuffer));
    268268}
    269269
    270270#if !PLATFORM(MACCATALYST)
    271 RefPtr<MediaSample> ImageTransferSessionVT::createMediaSample(IOSurfaceRef surface, const MediaTime& sampleTime, const IntSize& size, MediaSample::VideoRotation rotation, bool mirrored)
     271RefPtr<VideoFrame> ImageTransferSessionVT::createVideoFrame(IOSurfaceRef surface, const MediaTime& sampleTime, const IntSize& size, VideoFrame::Rotation rotation, bool mirrored)
    272272{
    273273    auto sampleBuffer = createCMSampleBuffer(surface, sampleTime, size);
    … …  
    279279#endif
    280280
    281 RefPtr<MediaSample> ImageTransferSessionVT::createMediaSample(CGImageRef image, const MediaTime& sampleTime, const IntSize& size, MediaSample::VideoRotation rotation, bool mirrored)
     281RefPtr<VideoFrame> ImageTransferSessionVT::createVideoFrame(CGImageRef image, const MediaTime& sampleTime, const IntSize& size, VideoFrame::Rotation rotation, bool mirrored)
    282282{
    283283    auto sampleBuffer = createCMSampleBuffer(image, sampleTime, size);
    … …  
    288288}
    289289
    290 RefPtr<MediaSample> ImageTransferSessionVT::createMediaSample(CMSampleBufferRef buffer, const MediaTime& sampleTime, const IntSize& size, MediaSample::VideoRotation rotation, bool mirrored)
     290RefPtr<VideoFrame> ImageTransferSessionVT::createVideoFrame(CMSampleBufferRef buffer, const MediaTime& sampleTime, const IntSize& size, VideoFrame::Rotation rotation, bool mirrored)
    291291{
    292292    auto sampleBuffer = convertCMSampleBuffer(buffer, size, &sampleTime);

  • trunk/Source/WebCore/platform/graphics/cv/VideoFrameCV.h

    r291550 r291611  
    3232#include <wtf/RetainPtr.h>
    3333
    34 using CVPixelBufferRef = struct __CVBuffer*;
     34using CMSampleBufferRef = struct opaqueCMSampleBuffer*;
    3535
    3636namespace WebCore {
    … …  
    4040class VideoFrameCV : public VideoFrame {
    4141public:
    42     WEBCORE_EXPORT static Ref<VideoFrameCV> create(MediaTime presentationTime, bool isMirrored, VideoRotation, RetainPtr<CVPixelBufferRef>&&);
    43     WEBCORE_EXPORT static Ref<VideoFrameCV> create(CMSampleBufferRef, bool isMirrored, VideoRotation);
     42    WEBCORE_EXPORT static Ref<VideoFrameCV> create(MediaTime presentationTime, bool isMirrored, Rotation, RetainPtr<CVPixelBufferRef>&&);
     43    WEBCORE_EXPORT static Ref<VideoFrameCV> create(CMSampleBufferRef, bool isMirrored, Rotation);
    4444    static RefPtr<VideoFrameCV> createFromPixelBuffer(PixelBuffer&&);
    4545    WEBCORE_EXPORT ~VideoFrameCV();
    … …  
    5353    // VideoFrame overrides.
    5454    WEBCORE_EXPORT WebCore::FloatSize presentationSize() const final;
    55     WEBCORE_EXPORT uint32_t videoPixelFormat() const final;
     55    WEBCORE_EXPORT uint32_t pixelFormat() const final;
    5656    WEBCORE_EXPORT void setOwnershipIdentity(const ProcessIdentity&) final;
    5757    bool isCV() const final { return true; }
    58     WEBCORE_EXPORT RefPtr<WebCore::VideoFrameCV> asVideoFrameCV() final;
    5958
    6059private:
    61     WEBCORE_EXPORT VideoFrameCV(MediaTime presentationTime, bool isMirrored, VideoRotation, RetainPtr<CVPixelBufferRef>&&);
     60    WEBCORE_EXPORT VideoFrameCV(MediaTime presentationTime, bool isMirrored, Rotation, RetainPtr<CVPixelBufferRef>&&);
    6261
    6362    const RetainPtr<CVPixelBufferRef> m_pixelBuffer;
    … …  
    6665template<typename Encoder> void VideoFrameCV::encode(Encoder& encoder) const
    6766{
    68     encoder << m_presentationTime << m_isMirrored << m_rotation << m_pixelBuffer;
     67    encoder << presentationTime() << isMirrored() << rotation() << m_pixelBuffer;
    6968}
    7069
    … …  
    7372    auto presentationTime = decoder.template decode<MediaTime>();
    7473    auto isMirrored = decoder.template decode<bool>();
    75     auto rotation = decoder.template decode<VideoRotation>();
     74    auto rotation = decoder.template decode<Rotation>();
    7675    auto pixelBuffer = decoder.template decode<RetainPtr<CVPixelBufferRef>>();
    7776    if (!decoder.isValid() || !*pixelBuffer)
    … …  
    8483SPECIALIZE_TYPE_TRAITS_BEGIN(WebCore::VideoFrameCV)
    8584    static bool isType(const WebCore::VideoFrame& videoFrame) { return videoFrame.isCV(); }
    86     static bool isType(const WebCore::MediaSample& mediaSample) { return is<WebCore::VideoFrame>(mediaSample) && is<WebCore::VideoFrameCV>(downcast<WebCore::VideoFrame>(mediaSample)); }
    8785SPECIALIZE_TYPE_TRAITS_END()
    8886

  • trunk/Source/WebCore/platform/graphics/cv/VideoFrameCV.mm

    r291550 r291611  
    3535namespace WebCore {
    3636
    37 Ref<VideoFrameCV> VideoFrameCV::create(CMSampleBufferRef sampleBuffer, bool isMirrored, VideoRotation rotation)
     37Ref<VideoFrameCV> VideoFrameCV::create(CMSampleBufferRef sampleBuffer, bool isMirrored, Rotation rotation)
    3838{
    3939    auto pixelBuffer = static_cast<CVPixelBufferRef>(PAL::CMSampleBufferGetImageBuffer(sampleBuffer));
    … …  
    4545}
    4646
    47 Ref<VideoFrameCV> VideoFrameCV::create(MediaTime presentationTime, bool isMirrored, VideoRotation rotation, RetainPtr<CVPixelBufferRef>&& pixelBuffer)
     47Ref<VideoFrameCV> VideoFrameCV::create(MediaTime presentationTime, bool isMirrored, Rotation rotation, RetainPtr<CVPixelBufferRef>&& pixelBuffer)
    4848{
    4949    ASSERT(pixelBuffer);
    … …  
    7474    }
    7575    ASSERT_UNUSED(status, !status);
    76     return create({ }, false, VideoRotation::None, WTFMove(cvPixelBuffer));
     76    return create({ }, false, Rotation::None, WTFMove(cvPixelBuffer));
    7777}
    7878
    79 VideoFrameCV::VideoFrameCV(MediaTime presentationTime, bool isMirrored, VideoRotation rotation, RetainPtr<CVPixelBufferRef>&& pixelBuffer)
     79VideoFrameCV::VideoFrameCV(MediaTime presentationTime, bool isMirrored, Rotation rotation, RetainPtr<CVPixelBufferRef>&& pixelBuffer)
    8080    : VideoFrame(presentationTime, isMirrored, rotation)
    8181    , m_pixelBuffer(WTFMove(pixelBuffer))
    … …  
    9090}
    9191
    92 uint32_t VideoFrameCV::videoPixelFormat() const
     92uint32_t VideoFrameCV::pixelFormat() const
    9393{
    9494    return CVPixelBufferGetPixelFormatType(m_pixelBuffer.get());
    … …  
    103103}
    104104
    105 RefPtr<WebCore::VideoFrameCV> VideoFrameCV::asVideoFrameCV()
    106 {
    107     return this;
    108 }
    109 
    110105ImageOrientation VideoFrameCV::orientation() const
    111106{
    112107    // Sample transform first flips x-coordinates, then rotates.
    113     switch (m_rotation) {
    114     case MediaSample::VideoRotation::None:
    115         return m_isMirrored ? ImageOrientation::OriginTopRight : ImageOrientation::OriginTopLeft;
    116     case MediaSample::VideoRotation::Right:
    117         return m_isMirrored ? ImageOrientation::OriginRightBottom : ImageOrientation::OriginRightTop;
    118     case MediaSample::VideoRotation::UpsideDown:
    119         return m_isMirrored ? ImageOrientation::OriginBottomLeft : ImageOrientation::OriginBottomRight;
    120     case MediaSample::VideoRotation::Left:
    121         return m_isMirrored ? ImageOrientation::OriginLeftTop : ImageOrientation::OriginLeftBottom;
     108    switch (rotation()) {
     109    case VideoFrame::Rotation::None:
     110        return isMirrored() ? ImageOrientation::OriginTopRight : ImageOrientation::OriginTopLeft;
     111    case VideoFrame::Rotation::Right:
     112        return isMirrored() ? ImageOrientation::OriginRightBottom : ImageOrientation::OriginRightTop;
     113    case VideoFrame::Rotation::UpsideDown:
     114        return isMirrored() ? ImageOrientation::OriginBottomLeft : ImageOrientation::OriginBottomRight;
     115    case VideoFrame::Rotation::Left:
     116        return isMirrored() ? ImageOrientation::OriginLeftTop : ImageOrientation::OriginLeftBottom;
    122117    }
    123118}

  • trunk/Source/WebCore/platform/graphics/gstreamer/VideoFrameGStreamer.cpp

    r291357 r291611  
    3131namespace WebCore {
    3232
    33 Ref<VideoFrameGStreamer> VideoFrameGStreamer::createFromPixelBuffer(PixelBuffer&& pixelBuffer, const MediaTime& presentationTime, const IntSize& destinationSize, double frameRate, VideoRotation videoRotation, bool videoMirrored, std::optional<VideoFrameTimeMetadata>&& metadata)
     33Ref<VideoFrameGStreamer> VideoFrameGStreamer::createFromPixelBuffer(PixelBuffer&& pixelBuffer, const MediaTime& presentationTime, const IntSize& destinationSize, double frameRate, Rotation videoRotation, bool videoMirrored, std::optional<VideoFrameTimeMetadata>&& metadata)
    3434{
    3535    ensureGStreamerInitialized();
    … …  
    8686}
    8787
    88 VideoFrameGStreamer::VideoFrameGStreamer(GRefPtr<GstSample>&& sample, const FloatSize& presentationSize, const MediaTime& presentationTime, VideoRotation videoRotation, bool videoMirrored, std::optional<VideoFrameTimeMetadata>&& metadata)
     88VideoFrameGStreamer::VideoFrameGStreamer(GRefPtr<GstSample>&& sample, const FloatSize& presentationSize, const MediaTime& presentationTime, Rotation videoRotation, bool videoMirrored, std::optional<VideoFrameTimeMetadata>&& metadata)
    8989    : VideoFrame(presentationTime, videoMirrored, videoRotation)
    9090    , m_sample(WTFMove(sample))
    … …  
    9999}
    100100
    101 VideoFrameGStreamer::VideoFrameGStreamer(const GRefPtr<GstSample>& sample, const MediaTime& presentationTime, VideoRotation videoRotation)
     101VideoFrameGStreamer::VideoFrameGStreamer(const GRefPtr<GstSample>& sample, const MediaTime& presentationTime, Rotation videoRotation)
    102102    : VideoFrame(presentationTime, false, videoRotation)
    103103    , m_sample(sample)

  • trunk/Source/WebCore/platform/graphics/gstreamer/VideoFrameGStreamer.h

    r291357 r291611  
    3131class VideoFrameGStreamer final : public VideoFrame {
    3232public:
    33     static Ref<VideoFrameGStreamer> create(GRefPtr<GstSample>&& sample, const FloatSize& presentationSize, const MediaTime& presentationTime = MediaTime::invalidTime(), VideoRotation videoRotation = VideoRotation::None, bool videoMirrored = false, std::optional<VideoFrameTimeMetadata>&& metadata = std::nullopt)
     33    static Ref<VideoFrameGStreamer> create(GRefPtr<GstSample>&& sample, const FloatSize& presentationSize, const MediaTime& presentationTime = MediaTime::invalidTime(), Rotation videoRotation = Rotation::None, bool videoMirrored = false, std::optional<VideoFrameTimeMetadata>&& metadata = std::nullopt)
    3434    {
    3535        return adoptRef(*new VideoFrameGStreamer(WTFMove(sample), presentationSize, presentationTime, videoRotation, videoMirrored, WTFMove(metadata)));
    3636    }
    3737
    38     static Ref<VideoFrameGStreamer> createWrappedSample(const GRefPtr<GstSample>& sample, const MediaTime& presentationTime, VideoRotation videoRotation = VideoRotation::None)
     38    static Ref<VideoFrameGStreamer> createWrappedSample(const GRefPtr<GstSample>& sample, const MediaTime& presentationTime, Rotation videoRotation = Rotation::None)
    3939    {
    4040        return adoptRef(*new VideoFrameGStreamer(sample, presentationTime, videoRotation));
    4141    }
    4242
    43     static Ref<VideoFrameGStreamer> createFromPixelBuffer(PixelBuffer&&, const MediaTime& presentationTime = MediaTime::invalidTime(), const IntSize& destinationSize = { }, double frameRate = 1, VideoRotation videoRotation = VideoRotation::None, bool videoMirrored = false, std::optional<VideoFrameTimeMetadata>&& metadata = std::nullopt);
     43    static Ref<VideoFrameGStreamer> createFromPixelBuffer(PixelBuffer&&, const MediaTime& presentationTime = MediaTime::invalidTime(), const IntSize& destinationSize = { }, double frameRate = 1, Rotation videoRotation = Rotation::None, bool videoMirrored = false, std::optional<VideoFrameTimeMetadata>&& metadata = std::nullopt);
    4444
    4545    GstSample* sample() const { return m_sample.get(); }
    4646
    4747private:
    48     VideoFrameGStreamer(GRefPtr<GstSample>&&, const FloatSize& presentationSize, const MediaTime& presentationTime = MediaTime::invalidTime(), VideoRotation = VideoRotation::None, bool videoMirrored = false, std::optional<VideoFrameTimeMetadata>&& = std::nullopt);
    49     VideoFrameGStreamer(const GRefPtr<GstSample>&, const MediaTime& presentationTime, VideoRotation = VideoRotation::None);
     48    VideoFrameGStreamer(GRefPtr<GstSample>&&, const FloatSize& presentationSize, const MediaTime& presentationTime = MediaTime::invalidTime(), Rotation = Rotation::None, bool videoMirrored = false, std::optional<VideoFrameTimeMetadata>&& = std::nullopt);
     49    VideoFrameGStreamer(const GRefPtr<GstSample>&, const MediaTime& presentationTime, Rotation = Rotation::None);
    5050
    5151    FloatSize presentationSize() const final { return m_presentationSize; }
    5252    RefPtr<JSC::Uint8ClampedArray> getRGBAImageData() const final;
     53    uint32_t pixelFormat() const final { return 0; }
    5354
    5455    GRefPtr<GstSample> m_sample;

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivate.h

    r289894 r291611  
    5050class MediaRecorderPrivate
    5151    : public RealtimeMediaSource::AudioSampleObserver
    52     , public RealtimeMediaSource::VideoSampleObserver {
     52    , public RealtimeMediaSource::VideoFrameObserver {
    5353public:
    5454    ~MediaRecorderPrivate();
    … …  
    117117{
    118118    if (m_videoSource)
    119         m_videoSource->removeVideoSampleObserver(*this);
     119        m_videoSource->removeVideoFrameObserver(*this);
    120120
    121121    m_videoSource = WTFMove(videoSource);
    122122
    123123    if (m_videoSource)
    124         m_videoSource->addVideoSampleObserver(*this);
     124        m_videoSource->addVideoFrameObserver(*this);
    125125}
    126126
    … …  
    133133        m_audioSource->removeAudioSampleObserver(*this);
    134134    if (m_videoSource)
    135         m_videoSource->removeVideoSampleObserver(*this);
     135        m_videoSource->removeVideoFrameObserver(*this);
    136136}
    137137

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp

    r291421 r291611  
    8484}
    8585
    86 void MediaRecorderPrivateAVFImpl::videoSampleAvailable(MediaSample& sample, VideoFrameTimeMetadata)
     86void MediaRecorderPrivateAVFImpl::videoFrameAvailable(VideoFrame& videoFrame, VideoFrameTimeMetadata)
    8787{
    8888    if (shouldMuteVideo()) {
    8989        if (!m_blackFrame) {
    90             auto size = sample.presentationSize();
    91             m_blackFrame = VideoFrameCV::create(sample.presentationTime(), sample.videoMirrored(), sample.videoRotation(), createBlackPixelBuffer(size.width(), size.height()));
     90            auto size = videoFrame.presentationSize();
     91            m_blackFrame = VideoFrameCV::create(videoFrame.presentationTime(), videoFrame.isMirrored(), videoFrame.rotation(), createBlackPixelBuffer(size.width(), size.height()));
    9292        }
    9393        m_writer->appendVideoFrame(*m_blackFrame);
    … …  
    9696
    9797    m_blackFrame = nullptr;
    98     m_writer->appendVideoFrame(VideoFrameCV::create(sample.presentationTime(), sample.videoMirrored(), sample.videoRotation(), sample.pixelBuffer()));
     98    m_writer->appendVideoFrame(videoFrame);
    9999}
    100100

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateAVFImpl.h

    r291421 r291611  
    5050
    5151    // MediaRecorderPrivate
    52     void videoSampleAvailable(MediaSample&, VideoFrameTimeMetadata) final;
     52    void videoFrameAvailable(VideoFrame&, VideoFrameTimeMetadata) final;
    5353    void fetchData(FetchDataCallback&&) final;
    5454    void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateGStreamer.h

    r291269 r291611  
    5252
    5353private:
    54     void videoSampleAvailable(MediaSample&, VideoFrameTimeMetadata) final { };
     54    void videoFrameAvailable(VideoFrame&, VideoFrameTimeMetadata) final { };
    5555    void audioSamplesAvailable(const WTF::MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final { };
    5656

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateMock.cpp

    r291269 r291611  
    6868}
    6969
    70 void MediaRecorderPrivateMock::videoSampleAvailable(MediaSample&, VideoFrameTimeMetadata)
     70void MediaRecorderPrivateMock::videoFrameAvailable(VideoFrame&, VideoFrameTimeMetadata)
    7171{
    7272    Locker locker { m_bufferLock };

  • trunk/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateMock.h

    r291269 r291611  
    4343private:
    4444    // MediaRecorderPrivate
    45     void videoSampleAvailable(MediaSample&, VideoFrameTimeMetadata) final;
     45    void videoFrameAvailable(VideoFrame&, VideoFrameTimeMetadata) final;
    4646    void fetchData(FetchDataCallback&&) final;
    4747    void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;

  • trunk/Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.h

    r291421 r291611  
    133133
    134134    RetainPtr<WebAVAssetWriterDelegate> m_writerDelegate;
    135     Deque<RetainPtr<CMSampleBufferRef>> m_pendingVideoSampleQueue;
     135    Deque<RetainPtr<CMSampleBufferRef>> m_pendingVideoFrameQueue;
    136136    Deque<RetainPtr<CMSampleBufferRef>> m_pendingAudioSampleQueue;
    137137

  • trunk/Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm

    r291421 r291611  
    133133{
    134134    m_pendingAudioSampleQueue.clear();
    135     m_pendingVideoSampleQueue.clear();
     135    m_pendingVideoFrameQueue.clear();
    136136    if (m_writer) {
    137137        [m_writer cancelWriting];
    … …  
    284284
    285285    if (m_isFlushingSamples) {
    286         m_pendingVideoSampleQueue.append(WTFMove(buffer));
     286        m_pendingVideoFrameQueue.append(WTFMove(buffer));
    287287        return true;
    288288    }
    289289
    290     while (!m_pendingVideoSampleQueue.isEmpty() && [m_videoAssetWriterInput isReadyForMoreMediaData])
    291         appendCompressedVideoSampleBuffer(m_pendingVideoSampleQueue.takeFirst().get());
     290    while (!m_pendingVideoFrameQueue.isEmpty() && [m_videoAssetWriterInput isReadyForMoreMediaData])
     291        appendCompressedVideoSampleBuffer(m_pendingVideoFrameQueue.takeFirst().get());
    292292
    293293    if (![m_videoAssetWriterInput isReadyForMoreMediaData]) {
    294         m_pendingVideoSampleQueue.append(WTFMove(buffer));
     294        m_pendingVideoFrameQueue.append(WTFMove(buffer));
    295295        return true;
    296296    }
    … …  
    335335{
    336336    bool hasPendingAudioSamples = !m_pendingAudioSampleQueue.isEmpty();
    337     bool hasPendingVideoSamples = !m_pendingVideoSampleQueue.isEmpty();
     337    bool hasPendingVideoSamples = !m_pendingVideoFrameQueue.isEmpty();
    338338
    339339    if (m_hasEncodedVideoSamples) {
    … …  
    350350    ASSERT(!m_isFlushingSamples);
    351351    m_isFlushingSamples = true;
    352     auto block = makeBlockPtr([this, weakThis = WeakPtr { *this }, hasPendingAudioSamples, hasPendingVideoSamples, audioSampleQueue = WTFMove(m_pendingAudioSampleQueue), videoSampleQueue = WTFMove(m_pendingVideoSampleQueue), callback = WTFMove(callback)]() mutable {
     352    auto block = makeBlockPtr([this, weakThis = WeakPtr { *this }, hasPendingAudioSamples, hasPendingVideoSamples, audioSampleQueue = WTFMove(m_pendingAudioSampleQueue), videoSampleQueue = WTFMove(m_pendingVideoFrameQueue), callback = WTFMove(callback)]() mutable {
    353353        if (!weakThis) {
    354354            callback();
    … …  
    387387        m_firstVideoFrame = true;
    388388        m_resumedVideoTime = PAL::CMClockGetTime(PAL::CMClockGetHostTimeClock());
    389         if (frame.videoRotation() != MediaSample::VideoRotation::None || frame.videoMirrored()) {
    390             m_videoTransform = CGAffineTransformMakeRotation(static_cast<int>(frame.videoRotation()) * M_PI / 180);
    391             if (frame.videoMirrored())
     389        if (frame.rotation() != VideoFrame::Rotation::None || frame.isMirrored()) {
     390            m_videoTransform = CGAffineTransformMakeRotation(static_cast<int>(frame.rotation()) * M_PI / 180);
     391            if (frame.isMirrored())
    392392                m_videoTransform = CGAffineTransformScale(*m_videoTransform, -1, 1);
    393393        }

  • trunk/Source/WebCore/platform/mediastream/RealtimeMediaSource.cpp

    r291269 r291611  
    7676}
    7777
    78 void RealtimeMediaSource::addVideoSampleObserver(VideoSampleObserver& observer)
     78void RealtimeMediaSource::addVideoFrameObserver(VideoFrameObserver& observer)
    7979{
    8080    ASSERT(isMainThread());
    81     Locker locker { m_videoSampleObserversLock };
    82     m_videoSampleObservers.add(&observer);
    83 }
    84 
    85 void RealtimeMediaSource::removeVideoSampleObserver(VideoSampleObserver& observer)
     81    Locker locker { m_VideoFrameObserversLock };
     82    m_VideoFrameObservers.add(&observer);
     83}
     84
     85void RealtimeMediaSource::removeVideoFrameObserver(VideoFrameObserver& observer)
    8686{
    8787    ASSERT(isMainThread());
    88     Locker locker { m_videoSampleObserversLock };
    89     m_videoSampleObservers.remove(&observer);
     88    Locker locker { m_VideoFrameObserversLock };
     89    m_VideoFrameObservers.remove(&observer);
    9090}
    9191
    … …  
    193193}
    194194
    195 void RealtimeMediaSource::videoSampleAvailable(MediaSample& mediaSample, VideoFrameTimeMetadata metadata)
     195void RealtimeMediaSource::videoFrameAvailable(VideoFrame& videoFrame, VideoFrameTimeMetadata metadata)
    196196{
    197197#if !RELEASE_LOG_DISABLED
    … …  
    211211    updateHasStartedProducingData();
    212212
    213     Locker locker { m_videoSampleObserversLock };
    214     for (auto* observer : m_videoSampleObservers)
    215         observer->videoSampleAvailable(mediaSample, metadata);
     213    Locker locker { m_VideoFrameObserversLock };
     214    for (auto* observer : m_VideoFrameObservers)
     215        observer->videoFrameAvailable(videoFrame, metadata);
    216216}
    217217

  • trunk/Source/WebCore/platform/mediastream/RealtimeMediaSource.h

    r291269 r291611  
    3939#include "Image.h"
    4040#include "MediaConstraints.h"
    41 #include "MediaSample.h"
    4241#include "PlatformLayer.h"
    4342#include "RealtimeMediaSourceCapabilities.h"
    4443#include "RealtimeMediaSourceFactory.h"
     44#include "VideoFrame.h"
    4545#include "VideoFrameTimeMetadata.h"
    4646#include <wtf/CompletionHandler.h>
    … …  
    9999        virtual void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t /*numberOfFrames*/) = 0;
    100100    };
    101     class VideoSampleObserver {
     101    class VideoFrameObserver {
    102102    public:
    103         virtual ~VideoSampleObserver() = default;
     103        virtual ~VideoFrameObserver() = default;
    104104
    105105        // May be called on a background thread.
    106         virtual void videoSampleAvailable(MediaSample&, VideoFrameTimeMetadata) = 0;
     106        virtual void videoFrameAvailable(VideoFrame&, VideoFrameTimeMetadata) = 0;
    107107    };
    108108
    … …  
    148148    WEBCORE_EXPORT void removeAudioSampleObserver(AudioSampleObserver&);
    149149
    150     WEBCORE_EXPORT void addVideoSampleObserver(VideoSampleObserver&);
    151     WEBCORE_EXPORT void removeVideoSampleObserver(VideoSampleObserver&);
     150    WEBCORE_EXPORT void addVideoFrameObserver(VideoFrameObserver&);
     151    WEBCORE_EXPORT void removeVideoFrameObserver(VideoFrameObserver&);
    152152
    153153    const IntSize size() const;
    … …  
    251251    void initializeEchoCancellation(bool echoCancellation) { m_echoCancellation = echoCancellation; }
    252252
    253     void videoSampleAvailable(MediaSample&, VideoFrameTimeMetadata);
     253    void videoFrameAvailable(VideoFrame&, VideoFrameTimeMetadata);
    254254    void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t);
    255255
    … …  
    290290    HashSet<AudioSampleObserver*> m_audioSampleObservers WTF_GUARDED_BY_LOCK(m_audioSampleObserversLock);
    291291
    292     mutable Lock m_videoSampleObserversLock;
    293     HashSet<VideoSampleObserver*> m_videoSampleObservers WTF_GUARDED_BY_LOCK(m_videoSampleObserversLock);
     292    mutable Lock m_VideoFrameObserversLock;
     293    HashSet<VideoFrameObserver*> m_VideoFrameObservers WTF_GUARDED_BY_LOCK(m_VideoFrameObserversLock);
    294294
    295295    // Set on the main thread from constraints.

  • trunk/Source/WebCore/platform/mediastream/RealtimeOutgoingVideoSource.cpp

    r286644 r291611  
    7676{
    7777    m_videoSource->removeObserver(*this);
    78     m_videoSource->source().removeVideoSampleObserver(*this);
     78    m_videoSource->source().removeVideoFrameObserver(*this);
    7979}
    8080
    … …  
    115115{
    116116    if (!m_muted && m_enabled) {
    117         m_videoSource->source().addVideoSampleObserver(*this);
     117        m_videoSource->source().addVideoFrameObserver(*this);
    118118        if (m_blackFrameTimer.isActive())
    119119            m_blackFrameTimer.stop();
    … …  
    121121    }
    122122
    123     m_videoSource->source().removeVideoSampleObserver(*this);
     123    m_videoSource->source().removeVideoFrameObserver(*this);
    124124    sendBlackFramesIfNeeded();
    125125}

  • trunk/Source/WebCore/platform/mediastream/RealtimeOutgoingVideoSource.h

    r291269 r291611  
    5151    , public webrtc::VideoTrackSourceInterface
    5252    , private MediaStreamTrackPrivate::Observer
    53     , private RealtimeMediaSource::VideoSampleObserver
     53    , private RealtimeMediaSource::VideoFrameObserver
    5454#if !RELEASE_LOG_DISABLED
    5555    , private LoggerHelper
    … …  
    135135    void trackEnded(MediaStreamTrackPrivate&) final { }
    136136
    137     // RealtimeMediaSource::VideoSampleObserver API
    138     void videoSampleAvailable(MediaSample&, VideoFrameTimeMetadata) override { }
     137    // RealtimeMediaSource::VideoFrameObserver API
     138    void videoFrameAvailable(VideoFrame&, VideoFrameTimeMetadata) override { }
    139139
    140140    Ref<MediaStreamTrackPrivate> m_videoSource;

  • trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.cpp

    r291269 r291611  
    366366}
    367367
    368 void RealtimeVideoCaptureSource::dispatchMediaSampleToObservers(MediaSample& sample, WebCore::VideoFrameTimeMetadata metadata)
    369 {
    370     MediaTime sampleTime = sample.presentationTime();
     368void RealtimeVideoCaptureSource::dispatchVideoFrameToObservers(VideoFrame& videoFrame, WebCore::VideoFrameTimeMetadata metadata)
     369{
     370    MediaTime sampleTime = videoFrame.presentationTime();
    371371
    372372    auto frameTime = sampleTime.toDouble();
    … …  
    380380        m_observedFrameRate = (m_observedFrameTimeStamps.size() / interval);
    381381
    382     videoSampleAvailable(sample, metadata);
     382    videoFrameAvailable(videoFrame, metadata);
    383383}
    384384

  • trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.h

    r291269 r291611  
    2929
    3030#include "ImageBuffer.h"
    31 #include "MediaSample.h"
    3231#include "RealtimeMediaSource.h"
    3332#include "VideoPreset.h"
    … …  
    4746    bool supportsSizeAndFrameRate(std::optional<int> width, std::optional<int> height, std::optional<double>) override;
    4847    virtual void generatePresets() = 0;
    49     virtual MediaSample::VideoRotation sampleRotation() const { return MediaSample::VideoRotation::None; }
     48    virtual VideoFrame::Rotation videoFrameRotation() const { return VideoFrame::Rotation::None; }
    5049
    5150    double observedFrameRate() const { return m_observedFrameRate; }
    … …  
    7473    void updateCapabilities(RealtimeMediaSourceCapabilities&);
    7574
    76     void dispatchMediaSampleToObservers(MediaSample&, WebCore::VideoFrameTimeMetadata);
     75    void dispatchVideoFrameToObservers(VideoFrame&, VideoFrameTimeMetadata);
    7776
    7877    static Span<const IntSize> standardVideoSizes();

  • trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.cpp

    r291550 r291611  
    5252RealtimeVideoSource::~RealtimeVideoSource()
    5353{
    54     m_source->removeVideoSampleObserver(*this);
     54    m_source->removeVideoFrameObserver(*this);
    5555    m_source->removeObserver(*this);
    5656}
    … …  
    7070{
    7171    m_source->start();
    72     m_source->addVideoSampleObserver(*this);
     72    m_source->addVideoFrameObserver(*this);
    7373}
    7474
    7575void RealtimeVideoSource::stopProducingData()
    7676{
    77     m_source->removeVideoSampleObserver(*this);
     77    m_source->removeVideoFrameObserver(*this);
    7878    m_source->stop();
    7979}
    … …  
    8181void RealtimeVideoSource::endProducingData()
    8282{
    83     m_source->removeVideoSampleObserver(*this);
     83    m_source->removeVideoFrameObserver(*this);
    8484    m_source->requestToEnd(*this);
    8585}
    … …  
    126126void RealtimeVideoSource::sourceSettingsChanged()
    127127{
    128     auto rotation = m_source->sampleRotation();
     128    auto rotation = m_source->videoFrameRotation();
    129129    auto size = this->size();
    130130    if (size.isEmpty())
    131131        size = m_source->size();
    132132
    133     if (rotation == MediaSample::VideoRotation::Left || rotation == MediaSample::VideoRotation::Right)
     133    if (rotation == VideoFrame::Rotation::Left || rotation == VideoFrame::Rotation::Right)
    134134        size = size.transposedSize();
    135135
    … …  
    173173
    174174#if PLATFORM(COCOA)
    175 RefPtr<MediaSample> RealtimeVideoSource::adaptVideoFrame(MediaSample& videoFrame)
    176 {
    177     if (!m_imageTransferSession || m_imageTransferSession->pixelFormat() != videoFrame.videoPixelFormat())
    178         m_imageTransferSession = ImageTransferSessionVT::create(videoFrame.videoPixelFormat(), m_shouldUseIOSurface);
     175RefPtr<VideoFrame> RealtimeVideoSource::adaptVideoFrame(VideoFrame& videoFrame)
     176{
     177    if (!m_imageTransferSession || m_imageTransferSession->pixelFormat() != videoFrame.pixelFormat())
     178        m_imageTransferSession = ImageTransferSessionVT::create(videoFrame.pixelFormat(), m_shouldUseIOSurface);
    179179
    180180    ASSERT(m_imageTransferSession);
    … …  
    189189#endif
    190190
    191 void RealtimeVideoSource::videoSampleAvailable(MediaSample& videoFrame, VideoFrameTimeMetadata metadata)
     191void RealtimeVideoSource::videoFrameAvailable(VideoFrame& videoFrame, VideoFrameTimeMetadata metadata)
    192192{
    193193    if (m_frameDecimation > 1 && ++m_frameDecimationCounter % m_frameDecimation)
    … …  
    203203    if (!size.isEmpty() && size != expandedIntSize(videoFrame.presentationSize())) {
    204204        if (auto newVideoFrame = adaptVideoFrame(videoFrame)) {
    205             RealtimeMediaSource::videoSampleAvailable(*newVideoFrame, metadata);
     205            RealtimeMediaSource::videoFrameAvailable(*newVideoFrame, metadata);
    206206            return;
    207207        }
    … …  
    209209#endif
    210210
    211     RealtimeMediaSource::videoSampleAvailable(videoFrame, metadata);
     211    RealtimeMediaSource::videoFrameAvailable(videoFrame, metadata);
    212212}
    213213

  • trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.h

    r291550 r291611  
    3737    : public RealtimeMediaSource
    3838    , public RealtimeMediaSource::Observer
    39     , public RealtimeMediaSource::VideoSampleObserver {
     39    , public RealtimeMediaSource::VideoFrameObserver {
    4040public:
    4141    static Ref<RealtimeVideoSource> create(Ref<RealtimeVideoCaptureSource>&& source, bool shouldUseIOSurface = true) { return adoptRef(*new RealtimeVideoSource(WTFMove(source), shouldUseIOSurface)); }
    … …  
    7272    bool preventSourceFromStopping() final;
    7373
    74     // RealtimeMediaSource::VideoSampleObserver
    75     void videoSampleAvailable(MediaSample&, VideoFrameTimeMetadata) final;
     74    // RealtimeMediaSource::VideoFrameObserver
     75    void videoFrameAvailable(VideoFrame&, VideoFrameTimeMetadata) final;
    7676
    7777#if PLATFORM(COCOA)
    78     RefPtr<MediaSample> adaptVideoFrame(MediaSample&);
     78    RefPtr<VideoFrame> adaptVideoFrame(VideoFrame&);
    7979#endif
    8080

  • trunk/Source/WebCore/platform/mediastream/VideoPreset.h

    r278253 r291611  
    2929
    3030#include "ImageBuffer.h"
    31 #include "MediaSample.h"
    3231#include "RealtimeMediaSource.h"
    3332#include <wtf/Lock.h>

  • trunk/Source/WebCore/platform/mediastream/cocoa/DisplayCaptureSourceCocoa.cpp

    r291269 r291611  
    294294    }
    295295
    296     auto sample = WTF::switchOn(frame,
    297         [this, sampleTime](RetainPtr<IOSurfaceRef>& surface) -> RefPtr<MediaSample> {
     296    auto videoFrame = WTF::switchOn(frame,
     297        [this, sampleTime](RetainPtr<IOSurfaceRef>& surface) -> RefPtr<VideoFrame> {
    298298            if (!surface)
    299299                return nullptr;
    300300
    301             return m_imageTransferSession->createMediaSample(surface.get(), sampleTime, size());
     301            return m_imageTransferSession->createVideoFrame(surface.get(), sampleTime, size());
    302302        },
    303         [this, sampleTime](RefPtr<NativeImage>& image) -> RefPtr<MediaSample> {
     303        [this, sampleTime](RefPtr<NativeImage>& image) -> RefPtr<VideoFrame> {
    304304            if (!image)
    305305                return nullptr;
    306306
    307             return m_imageTransferSession->createMediaSample(image->platformImage().get(), sampleTime, size());
     307            return m_imageTransferSession->createVideoFrame(image->platformImage().get(), sampleTime, size());
    308308        },
    309         [this, sampleTime](RetainPtr<CMSampleBufferRef>& sample) -> RefPtr<MediaSample> {
     309        [this, sampleTime](RetainPtr<CMSampleBufferRef>& sample) -> RefPtr<VideoFrame> {
    310310            if (!sample)
    311311                return nullptr;
    312312
    313             return m_imageTransferSession->createMediaSample(sample.get(), sampleTime, size());
     313            return m_imageTransferSession->createVideoFrame(sample.get(), sampleTime, size());
    314314        }
    315315    );
    316316
    317     if (!sample) {
     317    if (!videoFrame) {
    318318        ASSERT_NOT_REACHED();
    319319        return;
    … …  
    322322    VideoFrameTimeMetadata metadata;
    323323    metadata.captureTime = MonotonicTime::now().secondsSinceEpoch();
    324     videoSampleAvailable(*sample.get(), metadata);
     324    videoFrameAvailable(*videoFrame.get(), metadata);
    325325}
    326326

  • trunk/Source/WebCore/platform/mediastream/cocoa/DisplayCaptureSourceCocoa.h

    r291034 r291611  
    3030#include "CaptureDevice.h"
    3131#include "RealtimeMediaSource.h"
     32#include "RealtimeMediaSourceSettings.h"
    3233#include <wtf/Observer.h>
    3334#include <wtf/RefCounted.h>
    … …  
    3738#include <wtf/text/WTFString.h>
    3839
    39 
    40 typedef struct CGImage *CGImageRef;
    41 typedef struct __CVBuffer *CVPixelBufferRef;
    42 typedef struct __IOSurface *IOSurfaceRef;
     40using CGImageRef = CGImage*;
     41using CVPixelBufferRef = struct __CVBuffer*;
     42using IOSurfaceRef = struct __IOSurface*;
     43using CMSampleBufferRef = struct opaqueCMSampleBuffer*;
    4344
    4445namespace WTF {

  • trunk/Source/WebCore/platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp

    r291483 r291611  
    129129class InternalSource final : public MediaStreamTrackPrivate::Observer,
    130130    public RealtimeMediaSource::AudioSampleObserver,
    131     public RealtimeMediaSource::VideoSampleObserver {
     131    public RealtimeMediaSource::VideoFrameObserver {
    132132    WTF_MAKE_FAST_ALLOCATED;
    133133public:
    … …  
    182182            m_track.source().addAudioSampleObserver(*this);
    183183        else if (m_track.hasVideo())
    184             m_track.source().addVideoSampleObserver(*this);
     184            m_track.source().addVideoFrameObserver(*this);
    185185        m_isObserving = true;
    186186    }
    … …  
    197197            m_track.source().removeAudioSampleObserver(*this);
    198198        else if (m_track.hasVideo())
    199             m_track.source().removeVideoSampleObserver(*this);
     199            m_track.source().removeVideoFrameObserver(*this);
    200200        m_track.removeObserver(*this);
    201201    }
    … …  
    273273    }
    274274
    275     void videoSampleAvailable(MediaSample& sample, VideoFrameTimeMetadata) final
     275    void videoFrameAvailable(VideoFrame& videoFrame, VideoFrameTimeMetadata) final
    276276    {
    277277        if (!m_parent || !m_isObserving)
    278278            return;
    279279
    280         auto sampleSize = sample.presentationSize();
    281         IntSize captureSize(sampleSize.width(), sampleSize.height());
     280        auto videoFrameSize = videoFrame.presentationSize();
     281        IntSize captureSize(videoFrameSize.width(), videoFrameSize.height());
    282282
    283283        auto settings = m_track.settings();
    … …  
    290290            m_configuredSize.setHeight(captureSize.height());
    291291
    292         auto videoRotation = sample.videoRotation();
    293         bool videoMirrored = sample.videoMirrored();
     292        auto videoRotation = videoFrame.rotation();
     293        bool videoMirrored = videoFrame.isMirrored();
    294294        if (m_videoRotation != videoRotation || m_videoMirrored != videoMirrored) {
    295295            m_videoRotation = videoRotation;
    … …  
    302302        }
    303303
    304         auto* videoFrame = static_cast<VideoFrameGStreamer*>(&sample);
    305         auto* gstSample = videoFrame->sample();
     304        auto* gstSample = static_cast<VideoFrameGStreamer*>(&videoFrame)->sample();
    306305        if (!m_configuredSize.isEmpty() && m_lastKnownSize != m_configuredSize) {
    307306            m_lastKnownSize = m_configuredSize;
    … …  
    385384    IntSize m_lastKnownSize;
    386385    GRefPtr<GstSample> m_blackFrame;
    387     MediaSample::VideoRotation m_videoRotation { MediaSample::VideoRotation::None };
     386    VideoFrame::Rotation m_videoRotation { VideoFrame::Rotation::None };
    388387    bool m_videoMirrored { false };
    389388};

  • trunk/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoCaptureSource.cpp

    r291357 r291611  
    196196        return;
    197197
    198     dispatchMediaSampleToObservers(WTFMove(videoFrame), { });
     198    dispatchVideoFrameToObservers(WTFMove(videoFrame), { });
    199199}
    200200

  • trunk/Source/WebCore/platform/mediastream/gstreamer/MockRealtimeVideoSourceGStreamer.cpp

    r291357 r291611  
    7070    , m_deviceType(deviceType)
    7171{
    72     m_source->addVideoSampleObserver(*this);
     72    m_source->addVideoFrameObserver(*this);
    7373}
    7474
    7575MockDisplayCaptureSourceGStreamer::~MockDisplayCaptureSourceGStreamer()
    7676{
    77     m_source->removeVideoSampleObserver(*this);
     77    m_source->removeVideoFrameObserver(*this);
    7878}
    7979
    8080void MockDisplayCaptureSourceGStreamer::stopProducingData()
    8181{
    82     m_source->removeVideoSampleObserver(*this);
     82    m_source->removeVideoFrameObserver(*this);
    8383    m_source->stop();
    8484}
    … …  
    8787{
    8888    RealtimeMediaSource::requestToEnd(callingObserver);
    89     m_source->removeVideoSampleObserver(*this);
     89    m_source->removeVideoFrameObserver(*this);
    9090    m_source->requestToEnd(callingObserver);
    9191}
    … …  
    9797}
    9898
    99 void MockDisplayCaptureSourceGStreamer::videoSampleAvailable(MediaSample& sample, VideoFrameTimeMetadata metadata)
     99void MockDisplayCaptureSourceGStreamer::videoFrameAvailable(VideoFrame& videoFrame, VideoFrameTimeMetadata metadata)
    100100{
    101     RealtimeMediaSource::videoSampleAvailable(sample, metadata);
     101    RealtimeMediaSource::videoFrameAvailable(videoFrame, metadata);
    102102}
    103103
    … …  
    166166    metadata->captureTime = MonotonicTime::now().secondsSinceEpoch();
    167167    auto presentationTime = MediaTime::createWithDouble((elapsedTime() + 100_ms).seconds());
    168     auto videoFrame = VideoFrameGStreamer::createFromPixelBuffer(WTFMove(*pixelBuffer), presentationTime, size(), frameRate(), sampleRotation(), false, WTFMove(metadata));
    169     dispatchMediaSampleToObservers(videoFrame.get(), { });
     168    auto videoFrame = VideoFrameGStreamer::createFromPixelBuffer(WTFMove(*pixelBuffer), presentationTime, size(), frameRate(), videoFrameRotation(), false, WTFMove(metadata));
     169    dispatchVideoFrameToObservers(videoFrame.get(), { });
    170170}
    171171

  • trunk/Source/WebCore/platform/mediastream/gstreamer/MockRealtimeVideoSourceGStreamer.h

    r291269 r291611  
    4141};
    4242
    43 class MockDisplayCaptureSourceGStreamer final : public RealtimeMediaSource, RealtimeMediaSource::VideoSampleObserver {
     43class MockDisplayCaptureSourceGStreamer final : public RealtimeMediaSource, RealtimeMediaSource::VideoFrameObserver {
    4444public:
    4545    static CaptureSourceOrError create(const CaptureDevice&, String&&, const MediaConstraints*);
    … …  
    5050
    5151protected:
    52     // RealtimeMediaSource::VideoSampleObserver
    53     void videoSampleAvailable(MediaSample&, VideoFrameTimeMetadata) final;
     52    // RealtimeMediaSource::VideoFrameObserver
     53    void videoFrameAvailable(VideoFrame&, VideoFrameTimeMetadata) final;
    5454
    5555private:

  • trunk/Source/WebCore/platform/mediastream/gstreamer/RealtimeIncomingVideoSourceGStreamer.cpp

    r291483 r291611  
    101101{
    102102    auto* buffer = gst_sample_get_buffer(gstSample.get());
    103     videoSampleAvailable(VideoFrameGStreamer::createWrappedSample(gstSample, fromGstClockTime(GST_BUFFER_PTS(buffer))), { });
     103    videoFrameAvailable(VideoFrameGStreamer::createWrappedSample(gstSample, fromGstClockTime(GST_BUFFER_PTS(buffer))), { });
    104104}
    105105

  • trunk/Source/WebCore/platform/mediastream/ios/ReplayKitCaptureSource.mm

    r288950 r291611  
    157157
    158158    auto completionHandler = makeBlockPtr([this, weakThis = WeakPtr { *this }, identifier](NSError * _Nullable error) {
    159         // FIXME: It should be safe to call `videoSampleAvailable` from any thread. Test this and get rid of this main thread hop.
     159        // FIXME: It should be safe to call `videoFrameAvailable` from any thread. Test this and get rid of this main thread hop.
    160160        RunLoop::main().dispatch([this, weakThis, error = retainPtr(error), identifier]() mutable {
    161161            if (!weakThis || !error)

  • trunk/Source/WebCore/platform/mediastream/libwebrtc/VideoFrameLibWebRTC.cpp

    r291051 r291611  
    3434namespace WebCore {
    3535
    36 Ref<VideoFrameLibWebRTC> VideoFrameLibWebRTC::create(MediaTime presentationTime, bool isMirrored, VideoRotation rotation, rtc::scoped_refptr<webrtc::VideoFrameBuffer>&& buffer, ConversionCallback&& conversionCallback)
     36Ref<VideoFrameLibWebRTC> VideoFrameLibWebRTC::create(MediaTime presentationTime, bool isMirrored, Rotation rotation, rtc::scoped_refptr<webrtc::VideoFrameBuffer>&& buffer, ConversionCallback&& conversionCallback)
    3737{
    3838    return adoptRef(*new VideoFrameLibWebRTC(presentationTime, isMirrored, rotation, WTFMove(buffer), WTFMove(conversionCallback)));
    3939}
    4040
    41 VideoFrameLibWebRTC::VideoFrameLibWebRTC(MediaTime presentationTime, bool isMirrored, VideoRotation rotation, rtc::scoped_refptr<webrtc::VideoFrameBuffer>&& buffer, ConversionCallback&& conversionCallback)
     41VideoFrameLibWebRTC::VideoFrameLibWebRTC(MediaTime presentationTime, bool isMirrored, Rotation rotation, rtc::scoped_refptr<webrtc::VideoFrameBuffer>&& buffer, ConversionCallback&& conversionCallback)
    4242    : VideoFrame(presentationTime, isMirrored, rotation)
    4343    , m_buffer(WTFMove(buffer))

  • trunk/Source/WebCore/platform/mediastream/libwebrtc/VideoFrameLibWebRTC.h

    r291051 r291611  
    4343public:
    4444    using ConversionCallback = Function<RetainPtr<CVPixelBufferRef>(webrtc::VideoFrameBuffer&)>;
    45     static Ref<VideoFrameLibWebRTC> create(MediaTime, bool isMirrored, VideoRotation, rtc::scoped_refptr<webrtc::VideoFrameBuffer>&&, ConversionCallback&&);
     45    static Ref<VideoFrameLibWebRTC> create(MediaTime, bool isMirrored, Rotation, rtc::scoped_refptr<webrtc::VideoFrameBuffer>&&, ConversionCallback&&);
    4646
    4747    rtc::scoped_refptr<webrtc::VideoFrameBuffer> buffer() { return m_buffer; }
    4848
    4949private:
    50     VideoFrameLibWebRTC(MediaTime, bool isMirrored, VideoRotation, rtc::scoped_refptr<webrtc::VideoFrameBuffer>&&, ConversionCallback&&);
     50    VideoFrameLibWebRTC(MediaTime, bool isMirrored, Rotation, rtc::scoped_refptr<webrtc::VideoFrameBuffer>&&, ConversionCallback&&);
    5151
    5252    // VideoFrame
    5353    FloatSize presentationSize() const final { return m_size; }
    54     uint32_t videoPixelFormat() const final { return m_videoPixelFormat; }
     54    uint32_t pixelFormat() const final { return m_videoPixelFormat; }
    5555    CVPixelBufferRef pixelBuffer() const final;
    5656
    … …  
    6868SPECIALIZE_TYPE_TRAITS_BEGIN(WebCore::VideoFrameLibWebRTC)
    6969    static bool isType(const WebCore::VideoFrame& videoFrame) { return videoFrame.isLibWebRTC(); }
    70     static bool isType(const WebCore::MediaSample& mediaSample) { return is<WebCore::VideoFrame>(mediaSample) && is<WebCore::VideoFrameLibWebRTC>(downcast<WebCore::VideoFrame>(mediaSample)); }
    7170SPECIALIZE_TYPE_TRAITS_END()
    7271

  • trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/RealtimeIncomingVideoSourceLibWebRTC.cpp

    r291357 r291611  
    6262    if (frame.video_frame_buffer()->type() == webrtc::VideoFrameBuffer::Type::kNative) {
    6363        auto* framebuffer = static_cast<GStreamerVideoFrameLibWebRTC*>(frame.video_frame_buffer().get());
    64         videoSampleAvailable(VideoFrameGStreamer::createWrappedSample(framebuffer->getSample(), presentationTime, static_cast<MediaSample::VideoRotation>(frame.rotation())), { });
     64        videoFrameeAvailable(VideoFrameGStreamer::createWrappedSample(framebuffer->getSample(), presentationTime, static_cast<VideoFrame::Rotation>(frame.rotation())), { });
    6565    } else {
    6666        auto gstSample = convertLibWebRTCVideoFrameToGStreamerSample(frame);
    6767        auto metadata = std::make_optional(metadataFromVideoFrame(frame));
    68         videoSampleAvailable(VideoFrameGStreamer::create(WTFMove(gstSample), { }, presentationTime, static_cast<MediaSample::VideoRotation>(frame.rotation()), false, WTFMove(metadata)), { });
     68        videoFrameAvailable(VideoFrameGStreamer::create(WTFMove(gstSample), { }, presentationTime, static_cast<VideoFrame::Rotation>(frame.rotation()), false, WTFMove(metadata)), { });
    6969    }
    7070}

  • trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.cpp

    r291357 r291611  
    5151}
    5252
    53 void RealtimeOutgoingVideoSourceLibWebRTC::videoSampleAvailable(MediaSample& sample, VideoFrameTimeMetadata)
     53void RealtimeOutgoingVideoSourceLibWebRTC::videoFrameAvailable(VideoFrame& videoFrame, VideoFrameTimeMetadata)
    5454{
    55     switch (sample.videoRotation()) {
    56     case MediaSample::VideoRotation::None:
     55    switch (videoFrame.rotation()) {
     56    case VideoFrame::Rotation::None:
    5757        m_currentRotation = webrtc::kVideoRotation_0;
    5858        break;
    59     case MediaSample::VideoRotation::UpsideDown:
     59    case VideoFrame::Rotation::UpsideDown:
    6060        m_currentRotation = webrtc::kVideoRotation_180;
    6161        break;
    62     case MediaSample::VideoRotation::Right:
     62    case VideoFrame::Rotation::Right:
    6363        m_currentRotation = webrtc::kVideoRotation_90;
    6464        break;
    65     case MediaSample::VideoRotation::Left:
     65    case VideoFrame::Rotation::Left:
    6666        m_currentRotation = webrtc::kVideoRotation_270;
    6767        break;

  • trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.h

    r291269 r291611  
    4242    rtc::scoped_refptr<webrtc::VideoFrameBuffer> createBlackFrame(size_t, size_t) final;
    4343
    44     // RealtimeMediaSource::VideoSampleObserver API
    45     void videoSampleAvailable(MediaSample&, VideoFrameTimeMetadata) final;
     44    // RealtimeMediaSource::VideoFrameObserver API
     45    void videoFrameAvailable(VideoFrame&, VideoFrameTimeMetadata) final;
    4646};
    4747

  • trunk/Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.h

    r291034 r291611  
    9292    bool interrupted() const final;
    9393
    94     MediaSample::VideoRotation sampleRotation() const final { return m_sampleRotation; }
     94    VideoFrame::Rotation videoFrameRotation() const final { return m_videoFrameRotation; }
    9595    void setFrameRateWithPreset(double, RefPtr<VideoPreset>) final;
    9696    bool prefersPreset(VideoPreset&) final;
    … …  
    100100    void setSessionSizeAndFrameRate();
    101101    bool setPreset(NSString*);
    102     void computeSampleRotation();
     102    void computeVideoFrameRotation();
    103103    AVFrameRateRange* frameDurationForFrameRate(double);
    104104
    … …  
    119119    void verifyIsCapturing();
    120120
    121     RefPtr<MediaSample> m_buffer;
     121    RefPtr<VideoFrame> m_buffer;
    122122    RetainPtr<AVCaptureVideoDataOutput> m_videoOutput;
    123123    std::unique_ptr<ImageTransferSessionVT> m_imageTransferSession;
    … …  
    125125    int m_sensorOrientation { 0 };
    126126    int m_deviceOrientation { 0 };
    127     MediaSample::VideoRotation m_sampleRotation { MediaSample::VideoRotation::None };
     127    VideoFrame::Rotation m_videoFrameRotation { VideoFrame::Rotation::None };
    128128
    129129    std::optional<RealtimeMediaSourceSettings> m_currentSettings;

  • trunk/Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.mm

    r291550 r291611  
    252252
    253253    auto size = this->size();
    254     if (m_sampleRotation == MediaSample::VideoRotation::Left || m_sampleRotation == MediaSample::VideoRotation::Right)
     254    if (m_videoFrameRotation == VideoFrame::Rotation::Left || m_videoFrameRotation == VideoFrame::Rotation::Right)
    255255        size = size.transposedSize();
    256256   
    … …  
    497497
    498498    m_sensorOrientation = sensorOrientationFromVideoOutput(m_videoOutput.get());
    499     computeSampleRotation();
     499    computeVideoFrameRotation();
    500500
    501501    return true;
    … …  
    522522    ASSERT(orientation == 0 || orientation == 90 || orientation == -90 || orientation == 180);
    523523    m_deviceOrientation = orientation;
    524     computeSampleRotation();
    525     ALWAYS_LOG_IF(loggerPtr(), LOGIDENTIFIER, "rotation = ", m_sampleRotation, ", orientation = ", m_deviceOrientation);
    526 }
    527 
    528 void AVVideoCaptureSource::computeSampleRotation()
     524    computeVideoFrameRotation();
     525    ALWAYS_LOG_IF(loggerPtr(), LOGIDENTIFIER, "rotation = ", m_videoFrameRotation, ", orientation = ", m_deviceOrientation);
     526}
     527
     528void AVVideoCaptureSource::computeVideoFrameRotation()
    529529{
    530530    bool frontCamera = [device() position] == AVCaptureDevicePositionFront;
    531     MediaSample::VideoRotation sampleRotation;
     531    VideoFrame::Rotation videoFrameRotation;
    532532    switch (m_sensorOrientation - m_deviceOrientation) {
    533533    case 0:
    534         sampleRotation = MediaSample::VideoRotation::None;
     534        videoFrameRotation = VideoFrame::Rotation::None;
    535535        break;
    536536    case 180:
    537537    case -180:
    538         sampleRotation = MediaSample::VideoRotation::UpsideDown;
     538        videoFrameRotation = VideoFrame::Rotation::UpsideDown;
    539539        break;
    540540    case 90:
    541541    case -270:
    542         sampleRotation = frontCamera ? MediaSample::VideoRotation::Left : MediaSample::VideoRotation::Right;
     542        videoFrameRotation = frontCamera ? VideoFrame::Rotation::Left : VideoFrame::Rotation::Right;
    543543        break;
    544544    case -90:
    545545    case 270:
    546         sampleRotation = frontCamera ? MediaSample::VideoRotation::Right : MediaSample::VideoRotation::Left;
     546        videoFrameRotation = frontCamera ? VideoFrame::Rotation::Right : VideoFrame::Rotation::Left;
    547547        break;
    548548    default:
    549549        ASSERT_NOT_REACHED();
    550         sampleRotation = MediaSample::VideoRotation::None;
    551     }
    552     if (sampleRotation == m_sampleRotation)
    553         return;
    554 
    555     m_sampleRotation = sampleRotation;
     550        videoFrameRotation = VideoFrame::Rotation::None;
     551    }
     552    if (videoFrameRotation == m_videoFrameRotation)
     553        return;
     554
     555    m_videoFrameRotation = videoFrameRotation;
    556556    notifySettingsDidChangeObservers({ RealtimeMediaSourceSettings::Flag::Width, RealtimeMediaSourceSettings::Flag::Height });
    557557}
    … …  
    562562        return;
    563563
    564     auto videoFrame = VideoFrameCV::create(sampleBuffer, [captureConnection isVideoMirrored], m_sampleRotation);
     564    auto videoFrame = VideoFrameCV::create(sampleBuffer, [captureConnection isVideoMirrored], m_videoFrameRotation);
    565565    m_buffer = &videoFrame.get();
    566566    setIntrinsicSize(expandedIntSize(videoFrame->presentationSize()));
    567567    VideoFrameTimeMetadata metadata;
    568568    metadata.captureTime = MonotonicTime::now().secondsSinceEpoch();
    569     dispatchMediaSampleToObservers(WTFMove(videoFrame), metadata);
     569    dispatchVideoFrameToObservers(WTFMove(videoFrame), metadata);
    570570}
    571571

  • trunk/Source/WebCore/platform/mediastream/mac/MockRealtimeVideoSourceMac.mm

    r291550 r291611  
    9494        platformImage = nativeImage->platformImage();
    9595
    96     auto sampleTime = MediaTime::createWithDouble((elapsedTime() + 100_ms).seconds());
    97     auto sampleBuffer = m_imageTransferSession->createMediaSample(platformImage.get(), sampleTime, size(), sampleRotation());
    98     if (!sampleBuffer)
     96    auto presentationTime = MediaTime::createWithDouble((elapsedTime() + 100_ms).seconds());
     97    auto videoFrame = m_imageTransferSession->createVideoFrame(platformImage.get(), presentationTime, size(), videoFrameRotation());
     98    if (!videoFrame)
    9999        return;
    100100
    101101    auto captureTime = MonotonicTime::now().secondsSinceEpoch();
    102     m_workQueue->dispatch([this, protectedThis = Ref { *this }, sampleBuffer = WTFMove(sampleBuffer), captureTime]() mutable {
     102    m_workQueue->dispatch([this, protectedThis = Ref { *this }, videoFrame = WTFMove(videoFrame), captureTime]() mutable {
    103103        VideoFrameTimeMetadata metadata;
    104104        metadata.captureTime = captureTime;
    105         dispatchMediaSampleToObservers(*sampleBuffer, metadata);
     105        dispatchVideoFrameToObservers(*videoFrame, metadata);
    106106    });
    107107}

  • trunk/Source/WebCore/platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.h

    r291550 r291611  
    5050    RetainPtr<CVPixelBufferRef> pixelBufferFromVideoFrame(const webrtc::VideoFrame&);
    5151    CVPixelBufferPoolRef pixelBufferPool(size_t width, size_t height, webrtc::BufferType);
    52     RefPtr<VideoFrame> toVideoFrame(const webrtc::VideoFrame&, MediaSample::VideoRotation);
    53     Ref<VideoFrame> createVideoSampleFromCVPixelBuffer(CVPixelBufferRef, MediaSample::VideoRotation, int64_t);
     52    RefPtr<VideoFrame> toVideoFrame(const webrtc::VideoFrame&, VideoFrame::Rotation);
     53    Ref<VideoFrame> createVideoSampleFromCVPixelBuffer(CVPixelBufferRef, VideoFrame::Rotation, int64_t);
    5454
    5555    // rtc::VideoSinkInterface

  • trunk/Source/WebCore/platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.mm

    r291550 r291611  
    9292}
    9393
    94 Ref<VideoFrame> RealtimeIncomingVideoSourceCocoa::createVideoSampleFromCVPixelBuffer(CVPixelBufferRef pixelBuffer, MediaSample::VideoRotation rotation, int64_t timeStamp)
     94Ref<VideoFrame> RealtimeIncomingVideoSourceCocoa::createVideoSampleFromCVPixelBuffer(CVPixelBufferRef pixelBuffer, VideoFrame::Rotation rotation, int64_t timeStamp)
    9595{
    9696    return VideoFrameCV::create(MediaTime(timeStamp, 1000000), false, rotation, pixelBuffer);
    9797}
    9898
    99 RefPtr<VideoFrame> RealtimeIncomingVideoSourceCocoa::toVideoFrame(const webrtc::VideoFrame& frame, MediaSample::VideoRotation rotation)
     99RefPtr<VideoFrame> RealtimeIncomingVideoSourceCocoa::toVideoFrame(const webrtc::VideoFrame& frame, VideoFrame::Rotation rotation)
    100100{
    101101    if (muted()) {
    … …  
    145145    unsigned height = webrtcVideoFrame.height();
    146146
    147     MediaSample::VideoRotation rotation;
     147    VideoFrame::Rotation rotation;
    148148    switch (webrtcVideoFrame.rotation()) {
    149149    case webrtc::kVideoRotation_0:
    150         rotation = MediaSample::VideoRotation::None;
     150        rotation = VideoFrame::Rotation::None;
    151151        break;
    152152    case webrtc::kVideoRotation_180:
    153         rotation = MediaSample::VideoRotation::UpsideDown;
     153        rotation = VideoFrame::Rotation::UpsideDown;
    154154        break;
    155155    case webrtc::kVideoRotation_90:
    156         rotation = MediaSample::VideoRotation::Right;
     156        rotation = VideoFrame::Rotation::Right;
    157157        std::swap(width, height);
    158158        break;
    159159    case webrtc::kVideoRotation_270:
    160         rotation = MediaSample::VideoRotation::Left;
     160        rotation = VideoFrame::Rotation::Left;
    161161        std::swap(width, height);
    162162        break;
    … …  
    172172
    173173    setIntrinsicSize(IntSize(width, height));
    174     videoSampleAvailable(*videoFrame, metadataFromVideoFrame(webrtcVideoFrame));
     174    videoFrameAvailable(*videoFrame, metadataFromVideoFrame(webrtcVideoFrame));
    175175}
    176176

  • trunk/Source/WebCore/platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.cpp

    r291269 r291611  
    6464}
    6565
    66 void RealtimeOutgoingVideoSourceCocoa::videoSampleAvailable(MediaSample& sample, VideoFrameTimeMetadata)
     66void RealtimeOutgoingVideoSourceCocoa::videoFrameAvailable(VideoFrame& videoFrame, VideoFrameTimeMetadata)
    6767{
    6868#if !RELEASE_LOG_DISABLED
    … …  
    7171#endif
    7272
    73     switch (sample.videoRotation()) {
    74     case MediaSample::VideoRotation::None:
     73    switch (videoFrame.rotation()) {
     74    case VideoFrame::Rotation::None:
    7575        m_currentRotation = webrtc::kVideoRotation_0;
    7676        break;
    77     case MediaSample::VideoRotation::UpsideDown:
     77    case VideoFrame::Rotation::UpsideDown:
    7878        m_currentRotation = webrtc::kVideoRotation_180;
    7979        break;
    80     case MediaSample::VideoRotation::Right:
     80    case VideoFrame::Rotation::Right:
    8181        m_currentRotation = webrtc::kVideoRotation_90;
    8282        break;
    83     case MediaSample::VideoRotation::Left:
     83    case VideoFrame::Rotation::Left:
    8484        m_currentRotation = webrtc::kVideoRotation_270;
    8585        break;
    … …  
    8787
    8888    bool shouldApplyRotation = m_shouldApplyRotation && m_currentRotation != webrtc::kVideoRotation_0;
    89     if (!shouldApplyRotation && is<VideoFrame>(sample)) {
    90         if (downcast<VideoFrame>(sample).isRemoteProxy()) {
    91             Ref videoFrame { downcast<VideoFrame>(sample) };
    92             auto size = sample.presentationSize();
    93             sendFrame(webrtc::toWebRTCVideoFrameBuffer(&videoFrame.leakRef(),
     89    if (!shouldApplyRotation) {
     90        if (videoFrame.isRemoteProxy()) {
     91            Ref remoteVideoFrame { videoFrame };
     92            auto size = videoFrame.presentationSize();
     93            sendFrame(webrtc::toWebRTCVideoFrameBuffer(&remoteVideoFrame.leakRef(),
    9494                [](auto* pointer) { return static_cast<VideoFrame*>(pointer)->pixelBuffer(); },
    9595                [](auto* pointer) { static_cast<VideoFrame*>(pointer)->deref(); },
    … …  
    9797            return;
    9898        }
    99         if (downcast<VideoFrame>(sample).isLibWebRTC()) {
    100             sendFrame(downcast<VideoFrameLibWebRTC>(sample).buffer());
     99        if (videoFrame.isLibWebRTC()) {
     100            sendFrame(downcast<VideoFrameLibWebRTC>(videoFrame).buffer());
    101101            return;
    102102        }
    103103    }
    104104
    105     auto pixelBuffer = sample.pixelBuffer();
     105    auto pixelBuffer = videoFrame.pixelBuffer();
    106106    auto pixelFormatType = CVPixelBufferGetPixelFormatType(pixelBuffer);
    107107

  • trunk/Source/WebCore/platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.h

    r291269 r291611  
    4545    rtc::scoped_refptr<webrtc::VideoFrameBuffer> createBlackFrame(size_t width, size_t height) final;
    4646
    47     // RealtimeMediaSource::VideoSampleObserver API
    48     void videoSampleAvailable(MediaSample&, VideoFrameTimeMetadata) final;
     47    // RealtimeMediaSource::VideoFrameObserver API
     48    void videoFrameAvailable(VideoFrame&, VideoFrameTimeMetadata) final;
    4949
    5050    RetainPtr<CVPixelBufferRef> convertToYUV(CVPixelBufferRef);

  • trunk/Source/WebCore/platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.mm

    r289365 r291611  
    3232#import "ImageRotationSessionVT.h"
    3333#import "Logging.h"
    34 #import "MediaSample.h"
    3534#import "PixelBufferConformerCV.h"
    3635#import "RealtimeVideoUtilities.h"

  • trunk/Source/WebCore/platform/mock/MockRealtimeVideoSource.cpp

    r291034 r291611  
    181181    auto size = this->size();
    182182    if (mockCamera()) {
    183         if (m_deviceOrientation == MediaSample::VideoRotation::Left || m_deviceOrientation == MediaSample::VideoRotation::Right)
     183        if (m_deviceOrientation == VideoFrame::Rotation::Left || m_deviceOrientation == VideoFrame::Rotation::Right)
    184184            size = size.transposedSize();
    185185    }
    … …  
    497497    switch (orientation) {
    498498    case 0:
    499         m_deviceOrientation = MediaSample::VideoRotation::None;
     499        m_deviceOrientation = VideoFrame::Rotation::None;
    500500        break;
    501501    case 90:
    502         m_deviceOrientation = MediaSample::VideoRotation::Right;
     502        m_deviceOrientation = VideoFrame::Rotation::Right;
    503503        break;
    504504    case -90:
    505         m_deviceOrientation = MediaSample::VideoRotation::Left;
     505        m_deviceOrientation = VideoFrame::Rotation::Left;
    506506        break;
    507507    case 180:
    508         m_deviceOrientation = MediaSample::VideoRotation::UpsideDown;
     508        m_deviceOrientation = VideoFrame::Rotation::UpsideDown;
    509509        break;
    510510    default:

  • trunk/Source/WebCore/platform/mock/MockRealtimeVideoSource.h

    r291034 r291611  
    6363    Seconds elapsedTime();
    6464    void settingsDidChange(OptionSet<RealtimeMediaSourceSettings::Flag>) override;
    65     MediaSample::VideoRotation sampleRotation() const final { return m_deviceOrientation; }
     65    VideoFrame::Rotation videoFrameRotation() const final { return m_deviceOrientation; }
    6666    void generatePresets() override;
    6767
    … …  
    123123    MockMediaDevice m_device;
    124124    RefPtr<VideoPreset> m_preset;
    125     MediaSample::VideoRotation m_deviceOrientation { MediaSample::VideoRotation::None };
     125    VideoFrame::Rotation m_deviceOrientation { VideoFrame::Rotation::None };
    126126};
    127127

  • trunk/Source/WebCore/testing/Internals.cpp

    r291517 r291611  
    55265526    case RealtimeMediaSource::Type::Screen:
    55275527    case RealtimeMediaSource::Type::Window:
    5528         m_trackSource->removeVideoSampleObserver(*this);
     5528        m_trackSource->removeVideoFrameObserver(*this);
    55295529        break;
    55305530    case RealtimeMediaSource::Type::None:
    … …  
    55535553    case RealtimeMediaSource::Type::Screen:
    55545554    case RealtimeMediaSource::Type::Window:
    5555         m_trackSource->addVideoSampleObserver(*this);
     5555        m_trackSource->addVideoFrameObserver(*this);
    55565556        break;
    55575557    case RealtimeMediaSource::Type::None:
    … …  
    55705570}
    55715571
    5572 void Internals::videoSampleAvailable(MediaSample& sample, VideoFrameTimeMetadata)
    5573 {
    5574     callOnMainThread([this, weakThis = WeakPtr { *this }, sample = Ref { sample }] {
     5572void Internals::videoFrameAvailable(VideoFrame& videoFrame, VideoFrameTimeMetadata)
     5573{
     5574    callOnMainThread([this, weakThis = WeakPtr { *this }, videoFrame = Ref { videoFrame }] {
    55755575        if (!weakThis)
    55765576            return;
    55775577        m_trackVideoSampleCount++;
    5578         m_trackVideoRotation = static_cast<int>(sample->videoRotation());
     5578        m_trackVideoRotation = static_cast<int>(videoFrame->rotation());
    55795579        if (!m_nextTrackFramePromise)
    55805580            return;
    … …  
    55845584            return;
    55855585
    5586         auto rgba = sample->getRGBAImageData();
     5586        auto rgba = videoFrame->getRGBAImageData();
    55875587        if (!rgba)
    55885588            return;

  • trunk/Source/WebCore/testing/Internals.h

    r291517 r291611  
    166166    , public RealtimeMediaSource::Observer
    167167    , private RealtimeMediaSource::AudioSampleObserver
    168     , private RealtimeMediaSource::VideoSampleObserver
     168    , private RealtimeMediaSource::VideoFrameObserver
    169169#endif
    170170    {
    … …  
    12971297#if ENABLE(MEDIA_STREAM)
    12981298    // RealtimeMediaSource::Observer API
    1299     void videoSampleAvailable(MediaSample&, VideoFrameTimeMetadata) final;
     1299    void videoFrameAvailable(VideoFrame&, VideoFrameTimeMetadata) final;
    13001300    // RealtimeMediaSource::AudioSampleObserver API
    13011301    void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final { m_trackAudioSampleCount++; }

  • trunk/Source/WebKit/ChangeLog

    r291608 r291611  
     12022-03-22  Youenn Fablet  <youenn@apple.com>
     2
     3        Move RealtimeMediaSource::VideoSampleObserver::videoSampleAvailable to VideoFrame
     4        https://bugs.webkit.org/show_bug.cgi?id=237884
     5
     6        Reviewed by Eric Carlson.
     7
     8        Move from using MediaSample to VideoFrame directly.
     9
     10        * GPUProcess/graphics/RemoteGraphicsContextGLCocoa.cpp:
     11        * GPUProcess/media/RemoteVideoFrameObjectHeap.cpp:
     12        * GPUProcess/media/RemoteVideoFrameObjectHeap.h:
     13        * GPUProcess/webrtc/LibWebRTCCodecsProxy.mm:
     14        * GPUProcess/webrtc/RemoteMediaRecorder.cpp:
     15        * GPUProcess/webrtc/RemoteMediaRecorder.h:
     16        * GPUProcess/webrtc/RemoteMediaRecorder.messages.in:
     17        * GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.cpp:
     18        * GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.h:
     19        * GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.messages.in:
     20        * UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp:
     21        * WebProcess/GPU/media/RemoteVideoFrameProxy.cpp:
     22        * WebProcess/GPU/media/RemoteVideoFrameProxy.h:
     23        * WebProcess/GPU/webrtc/LibWebRTCCodecs.cpp:
     24        * WebProcess/GPU/webrtc/MediaRecorderPrivate.cpp:
     25        * WebProcess/GPU/webrtc/MediaRecorderPrivate.h:
     26        * WebProcess/GPU/webrtc/SampleBufferDisplayLayer.cpp:
     27        * WebProcess/GPU/webrtc/SampleBufferDisplayLayer.h:
     28        * WebProcess/GPU/webrtc/SharedVideoFrame.cpp:
     29        * WebProcess/GPU/webrtc/SharedVideoFrame.h:
     30        * WebProcess/cocoa/RemoteCaptureSampleManager.cpp:
     31        * WebProcess/cocoa/RemoteCaptureSampleManager.h:
     32        * WebProcess/cocoa/RemoteCaptureSampleManager.messages.in:
     33        * WebProcess/cocoa/RemoteRealtimeVideoSource.h:
     34
    1352022-03-22  Diego Pino Garcia  <dpino@igalia.com>
    236

  • trunk/Source/WebKit/GPUProcess/graphics/RemoteGraphicsContextGLCocoa.cpp

    r290513 r291611  
    5757        return;
    5858    }
    59     // Note: This extra casting is needed since VideoFrames are still MediaSamples.
    60     RefPtr<WebCore::VideoFrameCV> videoFrameCV;
    61     if (is<WebCore::VideoFrameCV>(*videoFrame))
    62         videoFrameCV = &downcast<WebCore::VideoFrameCV>(*videoFrame);
    63     else if (is<WebCore::MediaSampleAVFObjC>(*videoFrame))
    64         videoFrameCV = downcast<WebCore::MediaSampleAVFObjC>(*videoFrame).videoFrame();
    65     else {
     59
     60    auto videoFrameCV = videoFrame->asVideoFrameCV();
     61    if (!videoFrameCV) {
    6662        ASSERT_NOT_REACHED(); // Programming error, not a IPC attack.
    6763        completionHandler(false);

  • trunk/Source/WebKit/GPUProcess/media/RemoteVideoFrameObjectHeap.cpp

    r290824 r291611  
    3434
    3535#if PLATFORM(COCOA)
    36 #include <WebCore/MediaSampleAVFObjC.h>
    3736#include <WebCore/VideoFrameCV.h>
    3837#include <pal/cf/CoreMediaSoftLink.h>
    … …  
    8584}
    8685
    87 RemoteVideoFrameProxy::Properties RemoteVideoFrameObjectHeap::add(Ref<WebCore::MediaSample>&& frame)
     86RemoteVideoFrameProxy::Properties RemoteVideoFrameObjectHeap::add(Ref<WebCore::VideoFrame>&& frame)
    8887{
    8988    auto write = RemoteVideoFrameWriteReference::generateForAdd();

  • trunk/Source/WebKit/GPUProcess/media/RemoteVideoFrameObjectHeap.h

    r290824 r291611  
    3030#include "RemoteVideoFrameProxy.h"
    3131#include "ThreadSafeObjectHeap.h"
    32 #include <WebCore/MediaSample.h>
     32#include <WebCore/VideoFrame.h>
    3333
    3434#if PLATFORM(COCOA)
    … …  
    4545
    4646    void close();
    47     RemoteVideoFrameProxy::Properties add(Ref<WebCore::MediaSample>&&);
    48     RefPtr<WebCore::MediaSample> get(RemoteVideoFrameReadReference&& read) { return m_heap.retire(WTFMove(read), 0_s); }
     47    RemoteVideoFrameProxy::Properties add(Ref<WebCore::VideoFrame>&&);
     48    RefPtr<WebCore::VideoFrame> get(RemoteVideoFrameReadReference&& read) { return m_heap.retire(WTFMove(read), 0_s); }
    4949
    5050    void stopListeningForIPC(Ref<RemoteVideoFrameObjectHeap>&&) { close(); }
    … …  
    6565
    6666    const Ref<IPC::Connection> m_connection;
    67     ThreadSafeObjectHeap<RemoteVideoFrameIdentifier, RefPtr<WebCore::MediaSample>> m_heap;
     67    ThreadSafeObjectHeap<RemoteVideoFrameIdentifier, RefPtr<WebCore::VideoFrame>> m_heap;
    6868#if PLATFORM(COCOA)
    6969    SharedVideoFrameWriter m_sharedVideoFrameWriter;

  • trunk/Source/WebKit/GPUProcess/webrtc/LibWebRTCCodecsProxy.mm

    r291550 r291611  
    9898        videoFrameObjectHeap = m_videoFrameObjectHeap.ptr();
    9999    return [identifier, connection = m_connection, resourceOwner = m_resourceOwner, videoFrameObjectHeap = WTFMove(videoFrameObjectHeap)] (CVPixelBufferRef pixelBuffer, uint32_t timeStampNs, uint32_t timeStamp) mutable {
    100         auto videoFrame = WebCore::VideoFrameCV::create(MediaTime(timeStampNs, 1), false, WebCore::MediaSample::VideoRotation::None, pixelBuffer);
     100        auto videoFrame = WebCore::VideoFrameCV::create(MediaTime(timeStampNs, 1), false, WebCore::VideoFrame::Rotation::None, pixelBuffer);
    101101        if (resourceOwner)
    102102            videoFrame->setOwnershipIdentity(resourceOwner);
    … …  
    216216}
    217217
    218 static inline webrtc::VideoRotation toWebRTCVideoRotation(WebCore::MediaSample::VideoRotation rotation)
     218static inline webrtc::VideoRotation toWebRTCVideoRotation(WebCore::VideoFrame::Rotation rotation)
    219219{
    220220    switch (rotation) {
    221     case WebCore::MediaSample::VideoRotation::None:
     221    case WebCore::VideoFrame::Rotation::None:
    222222        return webrtc::kVideoRotation_0;
    223     case WebCore::MediaSample::VideoRotation::UpsideDown:
     223    case WebCore::VideoFrame::Rotation::UpsideDown:
    224224        return webrtc::kVideoRotation_180;
    225     case WebCore::MediaSample::VideoRotation::Right:
     225    case WebCore::VideoFrame::Rotation::Right:
    226226        return webrtc::kVideoRotation_90;
    227     case WebCore::MediaSample::VideoRotation::Left:
     227    case WebCore::VideoFrame::Rotation::Left:
    228228        return webrtc::kVideoRotation_270;
    229229    }

  • trunk/Source/WebKit/GPUProcess/webrtc/RemoteMediaRecorder.cpp

    r291550 r291611  
    8787}
    8888
    89 void RemoteMediaRecorder::videoSampleAvailable(SharedVideoFrame&& sharedVideoFrame)
     89void RemoteMediaRecorder::videoFrameAvailable(SharedVideoFrame&& sharedVideoFrame)
    9090{
    9191    if (auto frame = m_sharedVideoFrameReader.read(WTFMove(sharedVideoFrame)))

  • trunk/Source/WebKit/GPUProcess/webrtc/RemoteMediaRecorder.h

    r290721 r291611  
    7272    void audioSamplesStorageChanged(const SharedMemory::IPCHandle&, const WebCore::CAAudioStreamDescription&, uint64_t numberOfFrames);
    7373    void audioSamplesAvailable(MediaTime, uint64_t numberOfFrames);
    74     void videoSampleAvailable(SharedVideoFrame&&);
     74    void videoFrameAvailable(SharedVideoFrame&&);
    7575    void fetchData(CompletionHandler<void(IPC::DataReference&&, double)>&&);
    7676    void stopRecording(CompletionHandler<void()>&&);

  • trunk/Source/WebKit/GPUProcess/webrtc/RemoteMediaRecorder.messages.in

    r290903 r291611  
    2727    AudioSamplesStorageChanged(WebKit::SharedMemory::IPCHandle storageHandle, WebCore::CAAudioStreamDescription description, uint64_t numberOfFrames)
    2828    AudioSamplesAvailable(MediaTime time, uint64_t numberOfFrames)
    29     VideoSampleAvailable(struct WebKit::SharedVideoFrame frame)
     29    VideoFrameAvailable(struct WebKit::SharedVideoFrame frame)
    3030    FetchData() -> (IPC::DataReference buffer, double timeCode)
    3131    StopRecording() -> ()

  • trunk/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.cpp

    r291550 r291611  
    9292}
    9393
    94 void RemoteSampleBufferDisplayLayer::updateBoundsAndPosition(CGRect bounds, WebCore::MediaSample::VideoRotation rotation)
     94void RemoteSampleBufferDisplayLayer::updateBoundsAndPosition(CGRect bounds, WebCore::VideoFrame::Rotation rotation)
    9595{
    9696    m_sampleBufferDisplayLayer->updateRootLayerBoundsAndPosition(bounds, rotation, LocalSampleBufferDisplayLayer::ShouldUpdateRootLayer::Yes);
    … …  
    117117}
    118118
    119 void RemoteSampleBufferDisplayLayer::enqueue(SharedVideoFrame&& frame)
     119void RemoteSampleBufferDisplayLayer::enqueueVideoFrame(SharedVideoFrame&& frame)
    120120{
    121121    if (auto videoFrame = m_sharedVideoFrameReader.read(WTFMove(frame)))
    122         m_sampleBufferDisplayLayer->enqueue(videoFrame->pixelBuffer(), videoFrame->presentationTime());
     122        m_sampleBufferDisplayLayer->enqueueBuffer(videoFrame->pixelBuffer(), videoFrame->presentationTime());
    123123}
    124124
    125 void RemoteSampleBufferDisplayLayer::clearEnqueuedSamples()
     125void RemoteSampleBufferDisplayLayer::clearVideoFrames()
    126126{
    127     m_sampleBufferDisplayLayer->clearEnqueuedSamples();
     127    m_sampleBufferDisplayLayer->clearVideoFrames();
    128128}
    129129

  • trunk/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.h

    r290721 r291611  
    7171    void updateDisplayMode(bool hideDisplayLayer, bool hideRootLayer);
    7272    void updateAffineTransform(CGAffineTransform);
    73     void updateBoundsAndPosition(CGRect, WebCore::MediaSample::VideoRotation);
     73    void updateBoundsAndPosition(CGRect, WebCore::VideoFrame::Rotation);
    7474    void flush();
    7575    void flushAndRemoveImage();
    7676    void play();
    7777    void pause();
    78     void enqueue(SharedVideoFrame&&);
    79     void clearEnqueuedSamples();
     78    void enqueueVideoFrame(SharedVideoFrame&&);
     79    void clearVideoFrames();
    8080    void setSharedVideoFrameSemaphore(IPC::Semaphore&&);
    8181    void setSharedVideoFrameMemory(const SharedMemory::IPCHandle&);

  • trunk/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.messages.in

    r290358 r291611  
    3030    UpdateDisplayMode(bool hideDisplayLayer, bool hideRootLayer)
    3131    UpdateAffineTransform(CGAffineTransform transform)
    32     UpdateBoundsAndPosition(CGRect bounds, WebCore::MediaSample::VideoRotation rotation)
     32    UpdateBoundsAndPosition(CGRect bounds, WebCore::VideoFrame::Rotation rotation)
    3333    Flush()
    3434    FlushAndRemoveImage()
    35     Enqueue(struct WebKit::SharedVideoFrame frame)
    36     ClearEnqueuedSamples()
     35    EnqueueVideoFrame(struct WebKit::SharedVideoFrame frame)
     36    ClearVideoFrames()
    3737    Play()
    3838    Pause()

  • trunk/Source/WebKit/UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp

    r291550 r291611  
    5656    : private RealtimeMediaSource::Observer
    5757    , private RealtimeMediaSource::AudioSampleObserver
    58     , private RealtimeMediaSource::VideoSampleObserver {
     58    , private RealtimeMediaSource::VideoFrameObserver {
    5959    WTF_MAKE_FAST_ALLOCATED;
    6060public:
    … …  
    7575        case RealtimeMediaSource::Type::Screen:
    7676        case RealtimeMediaSource::Type::Window:
    77             m_source->addVideoSampleObserver(*this);
     77            m_source->addVideoFrameObserver(*this);
    7878            break;
    7979        case RealtimeMediaSource::Type::None:
    … …  
    9292        case RealtimeMediaSource::Type::Screen:
    9393        case RealtimeMediaSource::Type::Window:
    94             m_source->removeVideoSampleObserver(*this);
     94            m_source->removeVideoFrameObserver(*this);
    9595            break;
    9696        case RealtimeMediaSource::Type::None:
    … …  
    179179    }
    180180
    181     RefPtr<MediaSample> rotateVideoFrameIfNeeded(MediaSample& sample)
    182     {
    183         if (m_shouldApplyRotation && sample.videoRotation() != MediaSample::VideoRotation::None) {
    184             auto pixelBuffer = rotatePixelBuffer(sample);
    185             return VideoFrameCV::create(sample.presentationTime(), sample.videoMirrored(), MediaSample::VideoRotation::None, WTFMove(pixelBuffer));
    186         }
    187         return &sample;
    188     }
    189 
    190     void videoSampleAvailable(MediaSample& frame, VideoFrameTimeMetadata metadata) final
     181    RefPtr<VideoFrame> rotateVideoFrameIfNeeded(VideoFrame& frame)
     182    {
     183        if (m_shouldApplyRotation && frame.rotation() != VideoFrame::Rotation::None) {
     184            auto pixelBuffer = rotatePixelBuffer(frame);
     185            return VideoFrameCV::create(frame.presentationTime(), frame.isMirrored(), VideoFrame::Rotation::None, WTFMove(pixelBuffer));
     186        }
     187        return &frame;
     188    }
     189
     190    void videoFrameAvailable(VideoFrame& frame, VideoFrameTimeMetadata metadata) final
    191191    {
    192192        auto videoFrame = rotateVideoFrameIfNeeded(frame);
    … …  
    196196            videoFrame->setOwnershipIdentity(m_resourceOwner);
    197197        if (!m_videoFrameObjectHeap) {
    198             m_connection->send(Messages::RemoteCaptureSampleManager::VideoFrameAvailableCV(m_id, videoFrame->pixelBuffer(), videoFrame->videoRotation(), videoFrame->videoMirrored(), videoFrame->presentationTime(), metadata), 0);
     198            m_connection->send(Messages::RemoteCaptureSampleManager::VideoFrameAvailableCV(m_id, videoFrame->pixelBuffer(), videoFrame->rotation(), videoFrame->isMirrored(), videoFrame->presentationTime(), metadata), 0);
    199199            return;
    200200        }
    … …  
    203203    }
    204204
    205     RetainPtr<CVPixelBufferRef> rotatePixelBuffer(MediaSample& sample)
     205    RetainPtr<CVPixelBufferRef> rotatePixelBuffer(VideoFrame& videoFrame)
    206206    {
    207207        if (!m_rotationSession)
    … …  
    209209
    210210        ImageRotationSessionVT::RotationProperties rotation;
    211         switch (sample.videoRotation()) {
    212         case MediaSample::VideoRotation::None:
     211        switch (videoFrame.rotation()) {
     212        case VideoFrame::Rotation::None:
    213213            ASSERT_NOT_REACHED();
    214214            rotation.angle = 0;
    215215            break;
    216         case MediaSample::VideoRotation::UpsideDown:
     216        case VideoFrame::Rotation::UpsideDown:
    217217            rotation.angle = 180;
    218218            break;
    219         case MediaSample::VideoRotation::Right:
     219        case VideoFrame::Rotation::Right:
    220220            rotation.angle = 90;
    221221            break;
    222         case MediaSample::VideoRotation::Left:
     222        case VideoFrame::Rotation::Left:
    223223            rotation.angle = 270;
    224224            break;
    225225        }
    226         return m_rotationSession->rotate(sample, rotation, ImageRotationSessionVT::IsCGImageCompatible::No);
     226        return m_rotationSession->rotate(videoFrame, rotation, ImageRotationSessionVT::IsCGImageCompatible::No);
    227227    }
    228228

  • trunk/Source/WebKit/WebProcess/GPU/media/RemoteVideoFrameProxy.cpp

    r291051 r291611  
    4343namespace WebKit {
    4444
    45 RemoteVideoFrameProxy::Properties RemoteVideoFrameProxy::properties(WebKit::RemoteVideoFrameReference&& reference, const WebCore::MediaSample& mediaSample)
     45RemoteVideoFrameProxy::Properties RemoteVideoFrameProxy::properties(WebKit::RemoteVideoFrameReference&& reference, const WebCore::VideoFrame& videoFrame)
    4646{
    4747    return {
    4848        WTFMove(reference),
    49         mediaSample.presentationTime(),
    50         mediaSample.videoMirrored(),
    51         mediaSample.videoRotation(),
    52         expandedIntSize(mediaSample.presentationSize()),
    53         mediaSample.videoPixelFormat()
     49        videoFrame.presentationTime(),
     50        videoFrame.isMirrored(),
     51        videoFrame.rotation(),
     52        expandedIntSize(videoFrame.presentationSize()),
     53        videoFrame.pixelFormat()
    5454    };
    5555}
    … …  
    9595}
    9696
    97 uint32_t RemoteVideoFrameProxy::videoPixelFormat() const
     97uint32_t RemoteVideoFrameProxy::pixelFormat() const
    9898{
    9999    return m_pixelFormat;

  • trunk/Source/WebKit/WebProcess/GPU/media/RemoteVideoFrameProxy.h

    r291051 r291611  
    6060        MediaTime presentationTime;
    6161        bool isMirrored { false };
    62         VideoRotation rotation { VideoRotation::None };
     62        Rotation rotation { Rotation::None };
    6363        WebCore::IntSize size;
    6464        uint32_t pixelFormat { 0 };
    … …  
    6868    };
    6969
    70     static Properties properties(WebKit::RemoteVideoFrameReference&&, const WebCore::MediaSample&);
     70    static Properties properties(WebKit::RemoteVideoFrameReference&&, const WebCore::VideoFrame&);
    7171
    7272    static Ref<RemoteVideoFrameProxy> create(IPC::Connection&, RemoteVideoFrameObjectHeapProxy&, Properties&&);
    … …  
    8585    // WebCore::VideoFrame overrides.
    8686    WebCore::FloatSize presentationSize() const final { return m_size; }
    87     uint32_t videoPixelFormat() const final;
     87    uint32_t pixelFormat() const final;
    8888    bool isRemoteProxy() const final { return true; }
    8989#if PLATFORM(COCOA)
    … …  
    117117    auto presentationTime = decoder.template decode<MediaTime>();
    118118    auto isMirrored = decoder.template decode<bool>();
    119     auto videoRotation = decoder.template decode<VideoRotation>();
     119    auto rotation = decoder.template decode<Rotation>();
    120120    auto size = decoder.template decode<WebCore::IntSize>();
    121121    auto pixelFormat = decoder.template decode<uint32_t>();
    122122    if (!decoder.isValid())
    123123        return std::nullopt;
    124     return Properties { WTFMove(*reference), WTFMove(*presentationTime), *isMirrored, *videoRotation, *size, *pixelFormat };
     124    return Properties { WTFMove(*reference), WTFMove(*presentationTime), *isMirrored, *rotation, *size, *pixelFormat };
    125125}
    126126
    … …  
    130130
    131131SPECIALIZE_TYPE_TRAITS_BEGIN(WebKit::RemoteVideoFrameProxy)
    132     static bool isType(const WebCore::MediaSample& mediaSample) { return is<WebCore::VideoFrame>(mediaSample) && downcast<WebCore::VideoFrame>(mediaSample).isRemoteProxy(); }
     132    static bool isType(const WebCore::VideoFrame& videoFrame) { return videoFrame.isRemoteProxy(); }
    133133SPECIALIZE_TYPE_TRAITS_END()
    134134

  • trunk/Source/WebKit/WebProcess/GPU/webrtc/LibWebRTCCodecs.cpp

    r291095 r291611  
    103103}
    104104
    105 static inline MediaSample::VideoRotation toMediaSampleVideoRotation(webrtc::VideoRotation rotation)
     105static inline VideoFrame::Rotation toVideoRotation(webrtc::VideoRotation rotation)
    106106{
    107107    switch (rotation) {
    108108    case webrtc::kVideoRotation_0:
    109         return MediaSample::VideoRotation::None;
     109        return VideoFrame::Rotation::None;
    110110    case webrtc::kVideoRotation_180:
    111         return MediaSample::VideoRotation::UpsideDown;
     111        return VideoFrame::Rotation::UpsideDown;
    112112    case webrtc::kVideoRotation_90:
    113         return MediaSample::VideoRotation::Right;
     113        return VideoFrame::Rotation::Right;
    114114    case webrtc::kVideoRotation_270:
    115         return MediaSample::VideoRotation::Left;
     115        return VideoFrame::Rotation::Left;
    116116    }
    117117    ASSERT_NOT_REACHED();
    118     return MediaSample::VideoRotation::None;
     118    return VideoFrame::Rotation::None;
    119119}
    120120
    … …  
    495495        return WEBRTC_VIDEO_CODEC_ERROR;
    496496
    497     SharedVideoFrame sharedVideoFrame { MediaTime(frame.timestamp_us() * 1000, 1000000), false, toMediaSampleVideoRotation(frame.rotation()), WTFMove(*buffer) };
     497    SharedVideoFrame sharedVideoFrame { MediaTime(frame.timestamp_us() * 1000, 1000000), false, toVideoRotation(frame.rotation()), WTFMove(*buffer) };
    498498    encoder.connection->send(Messages::LibWebRTCCodecsProxy::EncodeFrame { encoder.identifier, sharedVideoFrame, frame.timestamp(), shouldEncodeAsKeyFrame }, 0);
    499499    return WEBRTC_VIDEO_CODEC_OK;

  • trunk/Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderPrivate.cpp

    r291269 r291611  
    9393}
    9494
    95 void MediaRecorderPrivate::videoSampleAvailable(MediaSample& sample, VideoFrameTimeMetadata)
     95void MediaRecorderPrivate::videoFrameAvailable(VideoFrame& videoFrame, VideoFrameTimeMetadata)
    9696{
    9797    if (shouldMuteVideo()) {
    9898        if (!m_blackFrameSize) {
    99             auto size = sample.presentationSize();
     99            auto size = videoFrame.presentationSize();
    100100            m_blackFrameSize = WebCore::IntSize { static_cast<int>(size.width()), static_cast<int>(size.height()) };
    101101        }
    102         SharedVideoFrame sharedVideoFrame { sample.presentationTime(), sample.videoMirrored(), sample.videoRotation(), *m_blackFrameSize };
    103         m_connection->send(Messages::RemoteMediaRecorder::VideoSampleAvailable { sharedVideoFrame }, m_identifier);
     102        SharedVideoFrame sharedVideoFrame { videoFrame.presentationTime(), videoFrame.isMirrored(), videoFrame.rotation(), *m_blackFrameSize };
     103        m_connection->send(Messages::RemoteMediaRecorder::VideoFrameAvailable { sharedVideoFrame }, m_identifier);
    104104        return;
    105105    }
    106106
    107107    m_blackFrameSize = { };
    108     auto sharedVideoFrame = m_sharedVideoFrameWriter.write(sample,
     108    auto sharedVideoFrame = m_sharedVideoFrameWriter.write(videoFrame,
    109109        [this](auto& semaphore) { m_connection->send(Messages::RemoteMediaRecorder::SetSharedVideoFrameSemaphore { semaphore }, m_identifier); },
    110110        [this](auto& handle) { m_connection->send(Messages::RemoteMediaRecorder::SetSharedVideoFrameMemory { handle }, m_identifier); }
    111111    );
    112112    if (sharedVideoFrame)
    113         m_connection->send(Messages::RemoteMediaRecorder::VideoSampleAvailable { WTFMove(*sharedVideoFrame) }, m_identifier);
     113        m_connection->send(Messages::RemoteMediaRecorder::VideoFrameAvailable { WTFMove(*sharedVideoFrame) }, m_identifier);
    114114}
    115115

  • trunk/Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderPrivate.h

    r291269 r291611  
    5757private:
    5858    // WebCore::MediaRecorderPrivate
    59     void videoSampleAvailable(WebCore::MediaSample&, WebCore::VideoFrameTimeMetadata) final;
     59    void videoFrameAvailable(WebCore::VideoFrame&, WebCore::VideoFrameTimeMetadata) final;
    6060    void fetchData(CompletionHandler<void(RefPtr<WebCore::FragmentedSharedBuffer>&&, const String& mimeType, double)>&&) final;
    6161    void stopRecording(CompletionHandler<void()>&&) final;

  • trunk/Source/WebKit/WebProcess/GPU/webrtc/SampleBufferDisplayLayer.cpp

    r290721 r291611  
    101101}
    102102
    103 void SampleBufferDisplayLayer::updateBoundsAndPosition(CGRect bounds, MediaSample::VideoRotation rotation)
     103void SampleBufferDisplayLayer::updateBoundsAndPosition(CGRect bounds, VideoFrame::Rotation rotation)
    104104{
    105105    m_connection->send(Messages::RemoteSampleBufferDisplayLayer::UpdateBoundsAndPosition { bounds, rotation }, m_identifier);
    … …  
    128128}
    129129
    130 void SampleBufferDisplayLayer::enqueueSample(MediaSample& sample)
     130void SampleBufferDisplayLayer::enqueueVideoFrame(VideoFrame& videoFrame)
    131131{
    132132    if (m_paused)
    133133        return;
    134134
    135     auto sharedVideoFrame = m_sharedVideoFrameWriter.write(sample,
     135    auto sharedVideoFrame = m_sharedVideoFrameWriter.write(videoFrame,
    136136        [this](auto& semaphore) { m_connection->send(Messages::RemoteSampleBufferDisplayLayer::SetSharedVideoFrameSemaphore { semaphore }, m_identifier); },
    137137        [this](auto& handle) { m_connection->send(Messages::RemoteSampleBufferDisplayLayer::SetSharedVideoFrameMemory { handle }, m_identifier); }
    … …  
    140140        return;
    141141
    142     m_connection->send(Messages::RemoteSampleBufferDisplayLayer::Enqueue { *sharedVideoFrame }, m_identifier);
     142    m_connection->send(Messages::RemoteSampleBufferDisplayLayer::EnqueueVideoFrame { *sharedVideoFrame }, m_identifier);
    143143}
    144144
    145 void SampleBufferDisplayLayer::clearEnqueuedSamples()
     145void SampleBufferDisplayLayer::clearVideoFrames()
    146146{
    147     m_connection->send(Messages::RemoteSampleBufferDisplayLayer::ClearEnqueuedSamples { }, m_identifier);
     147    m_connection->send(Messages::RemoteSampleBufferDisplayLayer::ClearVideoFrames { }, m_identifier);
    148148}
    149149

  • trunk/Source/WebKit/WebProcess/GPU/webrtc/SampleBufferDisplayLayer.h

    r290358 r291611  
    6363    void updateDisplayMode(bool hideDisplayLayer, bool hideRootLayer) final;
    6464    void updateAffineTransform(CGAffineTransform) final;
    65     void updateBoundsAndPosition(CGRect, WebCore::MediaSample::VideoRotation) final;
     65    void updateBoundsAndPosition(CGRect, WebCore::VideoFrame::Rotation) final;
    6666    void flush() final;
    6767    void flushAndRemoveImage() final;
    6868    void play() final;
    6969    void pause() final;
    70     void enqueueSample(WebCore::MediaSample&) final;
    71     void clearEnqueuedSamples() final;
     70    void enqueueVideoFrame(WebCore::VideoFrame&) final;
     71    void clearVideoFrames() final;
    7272    PlatformLayer* rootLayer() final;
    7373

  • trunk/Source/WebKit/WebProcess/GPU/webrtc/SharedVideoFrame.cpp

    r291421 r291611  
    9494}
    9595
    96 std::optional<SharedVideoFrame> SharedVideoFrameWriter::write(MediaSample& frame, const Function<void(IPC::Semaphore&)>& newSemaphoreCallback, const Function<void(const SharedMemory::IPCHandle&)>& newMemoryCallback)
     96std::optional<SharedVideoFrame> SharedVideoFrameWriter::write(VideoFrame& frame, const Function<void(IPC::Semaphore&)>& newSemaphoreCallback, const Function<void(const SharedMemory::IPCHandle&)>& newMemoryCallback)
    9797{
    9898    auto buffer = writeBuffer(frame, newSemaphoreCallback, newMemoryCallback);
    9999    if (!buffer)
    100100        return { };
    101     return SharedVideoFrame { frame.presentationTime(), frame.videoMirrored(), frame.videoRotation(), WTFMove(*buffer) };
    102 }
    103 
    104 std::optional<SharedVideoFrame::Buffer> SharedVideoFrameWriter::writeBuffer(MediaSample& frame, const Function<void(IPC::Semaphore&)>& newSemaphoreCallback, const Function<void(const SharedMemory::IPCHandle&)>& newMemoryCallback)
     101    return SharedVideoFrame { frame.presentationTime(), frame.isMirrored(), frame.rotation(), WTFMove(*buffer) };
     102}
     103
     104std::optional<SharedVideoFrame::Buffer> SharedVideoFrameWriter::writeBuffer(VideoFrame& frame, const Function<void(IPC::Semaphore&)>& newSemaphoreCallback, const Function<void(const SharedMemory::IPCHandle&)>& newMemoryCallback)
    105105{
    106106    if (is<RemoteVideoFrameProxy>(frame))

  • trunk/Source/WebKit/WebProcess/GPU/webrtc/SharedVideoFrame.h

    r291421 r291611  
    5656    MediaTime time;
    5757    bool mirrored { false };
    58     WebCore::MediaSample::VideoRotation rotation { WebCore::MediaSample::VideoRotation::None };
     58    WebCore::VideoFrame::Rotation rotation { WebCore::VideoFrame::Rotation::None };
    5959    using Buffer = std::variant<std::nullptr_t, RemoteVideoFrameReadReference, MachSendRight, WebCore::IntSize>;
    6060    Buffer buffer;
    … …  
    6969    SharedVideoFrameWriter();
    7070
    71     std::optional<SharedVideoFrame> write(WebCore::MediaSample&, const Function<void(IPC::Semaphore&)>&, const Function<void(const SharedMemory::IPCHandle&)>&);
     71    std::optional<SharedVideoFrame> write(WebCore::VideoFrame&, const Function<void(IPC::Semaphore&)>&, const Function<void(const SharedMemory::IPCHandle&)>&);
    7272    std::optional<SharedVideoFrame::Buffer> writeBuffer(CVPixelBufferRef, const Function<void(IPC::Semaphore&)>&, const Function<void(const SharedMemory::IPCHandle&)>&, bool canSendIOSurface = true);
    7373#if USE(LIBWEBRTC)
    … …  
    8181    bool prepareWriting(const WebCore::SharedVideoFrameInfo&, const Function<void(IPC::Semaphore&)>&, const Function<void(const SharedMemory::IPCHandle&)>&);
    8282
    83     std::optional<SharedVideoFrame::Buffer> writeBuffer(WebCore::MediaSample&, const Function<void(IPC::Semaphore&)>&, const Function<void(const SharedMemory::IPCHandle&)>&);
     83    std::optional<SharedVideoFrame::Buffer> writeBuffer(WebCore::VideoFrame&, const Function<void(IPC::Semaphore&)>&, const Function<void(const SharedMemory::IPCHandle&)>&);
    8484#if USE(LIBWEBRTC)
    8585    std::optional<SharedVideoFrame::Buffer> writeBuffer(webrtc::VideoFrameBuffer&, const Function<void(IPC::Semaphore&)>&, const Function<void(const SharedMemory::IPCHandle&)>&);

  • trunk/Source/WebKit/WebProcess/cocoa/RemoteCaptureSampleManager.cpp

    r291269 r291611  
    160160        return;
    161161    }
    162     auto videoFrameSize = videoFrame->size();
    163     iterator->value->videoFrameAvailable(WTFMove(videoFrame), videoFrameSize, metadata);
    164 }
    165 
    166 void RemoteCaptureSampleManager::videoFrameAvailableCV(RealtimeMediaSourceIdentifier identifier, RetainPtr<CVPixelBufferRef>&& pixelBuffer, WebCore::MediaSample::VideoRotation rotation, bool mirrored, MediaTime presentationTime, WebCore::VideoFrameTimeMetadata metadata)
     162    iterator->value->remoteVideoFrameAvailable(WTFMove(videoFrame), metadata);
     163}
     164
     165void RemoteCaptureSampleManager::videoFrameAvailableCV(RealtimeMediaSourceIdentifier identifier, RetainPtr<CVPixelBufferRef>&& pixelBuffer, WebCore::VideoFrame::Rotation rotation, bool mirrored, MediaTime presentationTime, WebCore::VideoFrameTimeMetadata metadata)
    167166{
    168167    ASSERT(!WTF::isMainRunLoop());
    … …  
    174173
    175174    auto videoFrame = VideoFrameCV::create(presentationTime, mirrored, rotation, WTFMove(pixelBuffer));
    176     auto size = videoFrame->presentationSize();
    177     iterator->value->videoFrameAvailable(videoFrame.get(), { static_cast<int>(size.width()), static_cast<int>(size.height()) }, metadata);
     175    iterator->value->remoteVideoFrameAvailable(videoFrame.get(), metadata);
    178176}
    179177

  • trunk/Source/WebKit/WebProcess/cocoa/RemoteCaptureSampleManager.h

    r291269 r291611  
    7575    void videoFrameAvailable(WebCore::RealtimeMediaSourceIdentifier, RemoteVideoFrameProxy::Properties&&, WebCore::VideoFrameTimeMetadata);
    7676    // FIXME: Will be removed once RemoteVideoFrameProxy providers are the only ones sending data.
    77     void videoFrameAvailableCV(WebCore::RealtimeMediaSourceIdentifier, RetainPtr<CVPixelBufferRef>&&, WebCore::MediaSample::VideoRotation, bool mirrored, MediaTime, WebCore::VideoFrameTimeMetadata);
     77    void videoFrameAvailableCV(WebCore::RealtimeMediaSourceIdentifier, RetainPtr<CVPixelBufferRef>&&, WebCore::VideoFrame::Rotation, bool mirrored, MediaTime, WebCore::VideoFrameTimeMetadata);
    7878
    7979    void setConnection(IPC::Connection*);

  • trunk/Source/WebKit/WebProcess/cocoa/RemoteCaptureSampleManager.messages.in

    r291269 r291611  
    2727    AudioStorageChanged(WebCore::RealtimeMediaSourceIdentifier id, WebKit::SharedMemory::IPCHandle storageHandle, WebCore::CAAudioStreamDescription description, uint64_t numberOfFrames, IPC::Semaphore captureSemaphore, MediaTime mediaTime, size_t frameChunkSize);
    2828    VideoFrameAvailable(WebCore::RealtimeMediaSourceIdentifier id, WebKit::RemoteVideoFrameProxy::Properties sample, struct WebCore::VideoFrameTimeMetadata metadata)
    29     VideoFrameAvailableCV(WebCore::RealtimeMediaSourceIdentifier id, RetainPtr<CVPixelBufferRef> pixelBuffer, WebCore::MediaSample::VideoRotation rotation, bool mirrored, MediaTime presentationTime, struct WebCore::VideoFrameTimeMetadata metadata)
     29    VideoFrameAvailableCV(WebCore::RealtimeMediaSourceIdentifier id, RetainPtr<CVPixelBufferRef> pixelBuffer, WebCore::VideoFrame::Rotation rotation, bool mirrored, MediaTime presentationTime, struct WebCore::VideoFrameTimeMetadata metadata)
    3030
    3131}

  • trunk/Source/WebKit/WebProcess/cocoa/RemoteRealtimeVideoSource.h

    r291269 r291611  
    5959    void captureStopped(bool didFail);
    6060
    61     void videoFrameAvailable(WebCore::VideoFrame& frame, WebCore::IntSize, WebCore::VideoFrameTimeMetadata metadata) { videoSampleAvailable(frame, metadata); }
     61    void remoteVideoFrameAvailable(WebCore::VideoFrame& frame, WebCore::VideoFrameTimeMetadata metadata) { videoFrameAvailable(frame, metadata); }
    6262    void sourceMutedChanged(bool value, bool interrupted);
    6363
Note: See TracChangeset for help on using the changeset viewer.