Changeset 254817 in webkit


Ignore:
Timestamp:
Jan 20, 2020 2:39:00 AM (4 years ago)
Author:
youenn@apple.com
Message:

Add support for MediaStream video track rendering in GPUProcess
https://bugs.webkit.org/show_bug.cgi?id=206286

Reviewed by Eric Carlson.

Source/WebCore:

Add a way for WebKit layer to create its own specific SampleBufferDisplayLayer.
WebKit layer can set a function pointer that will be used to create SampleBufferDisplayLayer.
If none is provided, LocalSampleBufferDisplayLayer is used.

Update MediaPlayerPrivateMediaStreamAVFObjC to monitor itself its root layer bounds change.
Conversely, LocalSampleBufferDisplayLayer no longer monitors its root layer bounds change.
This removes the need for SampleBufferDisplayLayer::Client to be notified of bounds change.
We add an API to update the root layer bounds of a LocalSampleBufferDisplayLayer as well,
this API being used by GPUProcess to mirror changes done to WebProcess root layer.

WebRTC incoming video tracks do not have presentation timestamps so we mark them as kCMSampleAttachmentKey_DisplayImmediately.
This is currently lost when doing IPC from WebProcess to NetworkProcess. Add this information for every enqueued sample in LocalSampleBufferDisplayLayer.
We might want in the future to serialize all attachments of the sample.

Manually tested.

  • SourcesCocoa.txt:
  • WebCore.xcodeproj/project.pbxproj:
  • platform/graphics/avfoundation/SampleBufferDisplayLayer.cpp: Added.

(WebCore::SampleBufferDisplayLayer::setCreator):
(WebCore::SampleBufferDisplayLayer::create):

  • platform/graphics/avfoundation/SampleBufferDisplayLayer.h:
  • platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.h:
  • platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.mm:

(-[WebAVSampleBufferStatusChangeListener invalidate]):
(-[WebAVSampleBufferStatusChangeListener start]):
(-[WebAVSampleBufferStatusChangeListener stop]):
(-[WebAVSampleBufferStatusChangeListener observeValueForKeyPath:ofObject:change:context:]):
(WebCore::LocalSampleBufferDisplayLayer::LocalSampleBufferDisplayLayer):
(WebCore::LocalSampleBufferDisplayLayer::~LocalSampleBufferDisplayLayer):
(WebCore::LocalSampleBufferDisplayLayer::updateBoundsAndPosition):
(WebCore::LocalSampleBufferDisplayLayer::enqueueSample):

  • platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
  • platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:

(-[WebRootSampleBufferBoundsChangeListener initWithParent:]):
(-[WebRootSampleBufferBoundsChangeListener dealloc]):
(-[WebRootSampleBufferBoundsChangeListener invalidate]):
(-[WebRootSampleBufferBoundsChangeListener start]):
(-[WebRootSampleBufferBoundsChangeListener stop]):
(-[WebRootSampleBufferBoundsChangeListener observeValueForKeyPath:ofObject:change:context:]):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::MediaPlayerPrivateMediaStreamAVFObjC):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::~MediaPlayerPrivateMediaStreamAVFObjC):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::ensureLayers):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::rootLayer const):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::updateDisplayLayer):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::rootLayerBoundsDidChange):

  • platform/graphics/cg/PDFDocumentImage.cpp:

Unified build fix.

Source/WebKit:

Implement WebProcess SampleBufferDisplayLayer and GPUProcess RemoteSampleBufferDisplayLayer.
RemoteSampleBufferDisplayLayer is a wrapper around LocalSampleBufferDisplayLayer.

A synchronous IPC is used at creation time of the layer to make sure we have a shared layer synchronously.
Future refactoring should allow to remove that constraint as done in regular video playing.

Override SampleBufferDisplayLayer create function pointer when rendering is to be done in GPUProcess.

  • DerivedSources-input.xcfilelist:
  • DerivedSources-output.xcfilelist:
  • DerivedSources.make:
  • GPUProcess/GPUConnectionToWebProcess.cpp:

(WebKit::GPUConnectionToWebProcess::sampleBufferDisplayLayerManager):
(WebKit::GPUConnectionToWebProcess::didReceiveMessage):
(WebKit::GPUConnectionToWebProcess::didReceiveSyncMessage):

  • GPUProcess/GPUConnectionToWebProcess.h:
  • GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.cpp: Added.

(WebKit::RemoteSampleBufferDisplayLayer::create):
(WebKit::RemoteSampleBufferDisplayLayer::RemoteSampleBufferDisplayLayer):
(WebKit::RemoteSampleBufferDisplayLayer::~RemoteSampleBufferDisplayLayer):
(WebKit::RemoteSampleBufferDisplayLayer::contextID):
(WebKit::RemoteSampleBufferDisplayLayer::bounds const):
(WebKit::RemoteSampleBufferDisplayLayer::updateDisplayMode):
(WebKit::RemoteSampleBufferDisplayLayer::updateAffineTransform):
(WebKit::RemoteSampleBufferDisplayLayer::updateBoundsAndPosition):
(WebKit::RemoteSampleBufferDisplayLayer::flush):
(WebKit::RemoteSampleBufferDisplayLayer::flushAndRemoveImage):
(WebKit::RemoteSampleBufferDisplayLayer::enqueueSample):
(WebKit::RemoteSampleBufferDisplayLayer::clearEnqueuedSamples):
(WebKit::RemoteSampleBufferDisplayLayer::messageSenderConnection const):
(WebKit::RemoteSampleBufferDisplayLayer::sampleBufferDisplayLayerStatusDidChange):
(WebKit::RemoteSampleBufferDisplayLayer::streamTime const):

  • GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.h: Added.
  • GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.messages.in: Added.
  • GPUProcess/webrtc/RemoteSampleBufferDisplayLayerManager.cpp: Added.

(WebKit::RemoteSampleBufferDisplayLayerManager::RemoteSampleBufferDisplayLayerManager):
(WebKit::RemoteSampleBufferDisplayLayerManager::didReceiveLayerMessage):
(WebKit::RemoteSampleBufferDisplayLayerManager::createLayer):
(WebKit::RemoteSampleBufferDisplayLayerManager::releaseLayer):

  • GPUProcess/webrtc/RemoteSampleBufferDisplayLayerManager.h: Added.
  • GPUProcess/webrtc/RemoteSampleBufferDisplayLayerManager.messages.in: Added.
  • Scripts/webkit/messages.py:
  • Shared/WebCoreArgumentCoders.h:
  • Shared/mac/WebCoreArgumentCodersMac.mm:

(IPC::ArgumentCoder<CGRect>::encode):
(IPC::ArgumentCoder<CGRect>::decode):
(IPC::ArgumentCoder<CGSize>::encode):
(IPC::ArgumentCoder<CGSize>::decode):
(IPC::ArgumentCoder<CGPoint>::encode):
(IPC::ArgumentCoder<CGPoint>::decode):
(IPC::ArgumentCoder<CGAffineTransform>::encode):
(IPC::ArgumentCoder<CGAffineTransform>::decode):

  • Sources.txt:
  • SourcesCocoa.txt:
  • WebKit.xcodeproj/project.pbxproj:
  • WebProcess/GPU/GPUProcessConnection.cpp:

(WebKit::GPUProcessConnection::sampleBufferDisplayLayerManager):
(WebKit::GPUProcessConnection::didReceiveMessage):

  • WebProcess/GPU/GPUProcessConnection.h:
  • WebProcess/GPU/media/RemoteMediaPlayerManager.cpp:

(WebKit::RemoteMediaPlayerManager::updatePreferences):

  • WebProcess/GPU/webrtc/SampleBufferDisplayLayer.cpp: Added.

(WebKit::SampleBufferDisplayLayer::create):
(WebKit::SampleBufferDisplayLayer::SampleBufferDisplayLayer):
(WebKit::SampleBufferDisplayLayer::~SampleBufferDisplayLayer):
(WebKit::SampleBufferDisplayLayer::didFail const):
(WebKit::SampleBufferDisplayLayer::updateDisplayMode):
(WebKit::SampleBufferDisplayLayer::bounds const):
(WebKit::SampleBufferDisplayLayer::updateAffineTransform):
(WebKit::SampleBufferDisplayLayer::updateBoundsAndPosition):
(WebKit::SampleBufferDisplayLayer::flush):
(WebKit::SampleBufferDisplayLayer::flushAndRemoveImage):
(WebKit::SampleBufferDisplayLayer::enqueueSample):
(WebKit::SampleBufferDisplayLayer::clearEnqueuedSamples):
(WebKit::SampleBufferDisplayLayer::rootLayer):
(WebKit::SampleBufferDisplayLayer::setDidFail):
(WebKit::SampleBufferDisplayLayer::boundsChanged):

  • WebProcess/GPU/webrtc/SampleBufferDisplayLayer.h: Added.
  • WebProcess/GPU/webrtc/SampleBufferDisplayLayer.messages.in: Added.
  • WebProcess/GPU/webrtc/SampleBufferDisplayLayerIdentifier.h: Added.
  • WebProcess/GPU/webrtc/SampleBufferDisplayLayerManager.cpp: Added.

(WebKit::SampleBufferDisplayLayerManager::didReceiveLayerMessage):
(WebKit::SampleBufferDisplayLayerManager::createLayer):
(WebKit::SampleBufferDisplayLayerManager::addLayer):
(WebKit::SampleBufferDisplayLayerManager::removeLayer):

  • WebProcess/GPU/webrtc/SampleBufferDisplayLayerManager.h: Added.
  • WebProcess/WebProcess.h:
Location:
trunk/Source
Files:
10 added
26 edited
3 copied

Legend:

Unmodified
Added
Removed
  • trunk/Source/WebCore/ChangeLog

    r254814 r254817  
     12020-01-20  youenn fablet  <youenn@apple.com>
     2
     3        Add support for MediaStream video track rendering in GPUProcess
     4        https://bugs.webkit.org/show_bug.cgi?id=206286
     5
     6        Reviewed by Eric Carlson.
     7
     8        Add a way for WebKit layer to create its own specific SampleBufferDisplayLayer.
     9        WebKit layer can set a function pointer that will be used to create SampleBufferDisplayLayer.
     10        If none is provided, LocalSampleBufferDisplayLayer is used.
     11
     12        Update MediaPlayerPrivateMediaStreamAVFObjC to monitor itself its root layer bounds change.
     13        Conversely, LocalSampleBufferDisplayLayer no longer monitors its root layer bounds change.
     14        This removes the need for SampleBufferDisplayLayer::Client to be notified of bounds change.
     15        We add an API to update the root layer bounds of a LocalSampleBufferDisplayLayer as well,
     16        this API being used by GPUProcess to mirror changes done to WebProcess root layer.
     17
     18        WebRTC incoming video tracks do not have presentation timestamps so we mark them as kCMSampleAttachmentKey_DisplayImmediately.
     19        This is currently lost when doing IPC from WebProcess to NetworkProcess. Add this information for every enqueued sample in LocalSampleBufferDisplayLayer.
     20        We might want in the future to serialize all attachments of the sample.
     21
     22        Manually tested.
     23
     24        * SourcesCocoa.txt:
     25        * WebCore.xcodeproj/project.pbxproj:
     26        * platform/graphics/avfoundation/SampleBufferDisplayLayer.cpp: Added.
     27        (WebCore::SampleBufferDisplayLayer::setCreator):
     28        (WebCore::SampleBufferDisplayLayer::create):
     29        * platform/graphics/avfoundation/SampleBufferDisplayLayer.h:
     30        * platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.h:
     31        * platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.mm:
     32        (-[WebAVSampleBufferStatusChangeListener invalidate]):
     33        (-[WebAVSampleBufferStatusChangeListener start]):
     34        (-[WebAVSampleBufferStatusChangeListener stop]):
     35        (-[WebAVSampleBufferStatusChangeListener observeValueForKeyPath:ofObject:change:context:]):
     36        (WebCore::LocalSampleBufferDisplayLayer::LocalSampleBufferDisplayLayer):
     37        (WebCore::LocalSampleBufferDisplayLayer::~LocalSampleBufferDisplayLayer):
     38        (WebCore::LocalSampleBufferDisplayLayer::updateBoundsAndPosition):
     39        (WebCore::LocalSampleBufferDisplayLayer::enqueueSample):
     40        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
     41        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
     42        (-[WebRootSampleBufferBoundsChangeListener initWithParent:]):
     43        (-[WebRootSampleBufferBoundsChangeListener dealloc]):
     44        (-[WebRootSampleBufferBoundsChangeListener invalidate]):
     45        (-[WebRootSampleBufferBoundsChangeListener start]):
     46        (-[WebRootSampleBufferBoundsChangeListener stop]):
     47        (-[WebRootSampleBufferBoundsChangeListener observeValueForKeyPath:ofObject:change:context:]):
     48        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::MediaPlayerPrivateMediaStreamAVFObjC):
     49        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::~MediaPlayerPrivateMediaStreamAVFObjC):
     50        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::ensureLayers):
     51        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::rootLayer const):
     52        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::updateDisplayLayer):
     53        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::rootLayerBoundsDidChange):
     54        * platform/graphics/cg/PDFDocumentImage.cpp:
     55        Unified build fix.
     56
    1572020-01-20  youenn fablet  <youenn@apple.com>
    258
  • trunk/Source/WebCore/SourcesCocoa.txt

    r254813 r254817  
    231231platform/graphics/angle/TemporaryANGLESetting.cpp @no-unify
    232232
     233platform/graphics/avfoundation/SampleBufferDisplayLayer.cpp
    233234platform/graphics/avfoundation/AVTrackPrivateAVFObjCImpl.mm @no-unify
    234235platform/graphics/avfoundation/AudioSourceProviderAVFObjC.mm @no-unify
  • trunk/Source/WebCore/WebCore.xcodeproj/project.pbxproj

    r254813 r254817  
    72867286                3FFFF9AB159D9B060020BBD5 /* ViewportStyleResolver.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ViewportStyleResolver.cpp; sourceTree = "<group>"; };
    72877287                3FFFF9AC159D9B060020BBD5 /* ViewportStyleResolver.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ViewportStyleResolver.h; sourceTree = "<group>"; };
     7288                41024FC823CF254F00FDF98E /* SampleBufferDisplayLayer.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = SampleBufferDisplayLayer.cpp; sourceTree = "<group>"; };
    72887289                410626A72280A22A006D1B59 /* JSAudioNodeCustom.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = JSAudioNodeCustom.cpp; sourceTree = "<group>"; };
    72897290                4107908A1FC3E4F20061B27A /* ClientOrigin.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ClientOrigin.h; sourceTree = "<group>"; };
     
    1634916350                                CDBEAEAB19D92B6C00BEBA88 /* MediaSelectionGroupAVFObjC.h */,
    1635016351                                CDBEAEAA19D92B6C00BEBA88 /* MediaSelectionGroupAVFObjC.mm */,
     16352                                41024FC823CF254F00FDF98E /* SampleBufferDisplayLayer.cpp */,
    1635116353                                414598BE23C8AAB8002B9CC8 /* SampleBufferDisplayLayer.h */,
    1635216354                                CD336F6317FA0A4D00DDDCD0 /* VideoTrackPrivateAVF.h */,
  • trunk/Source/WebCore/platform/graphics/avfoundation/SampleBufferDisplayLayer.cpp

    r254816 r254817  
    2424 */
    2525
    26 #pragma once
     26#include "config.h"
     27#include "SampleBufferDisplayLayer.h"
    2728
    28 #include "PlatformLayer.h"
    29 #include <wtf/WeakPtr.h>
     29#include "IntSize.h"
     30#include "LocalSampleBufferDisplayLayer.h"
    3031
    31 namespace WTF {
    32 class MediaTime;
     32namespace WebCore {
     33
     34SampleBufferDisplayLayer::LayerCreator SampleBufferDisplayLayer::m_layerCreator = nullptr;
     35void SampleBufferDisplayLayer::setCreator(LayerCreator creator)
     36{
     37    m_layerCreator = creator;
    3338}
    3439
    35 namespace WebCore {
    36 class IntSize;
    37 class MediaSample;
     40std::unique_ptr<SampleBufferDisplayLayer> SampleBufferDisplayLayer::create(Client& client, bool hideRootLayer, IntSize size)
     41{
     42    if (m_layerCreator)
     43        return m_layerCreator(client, hideRootLayer, size);
    3844
    39 class SampleBufferDisplayLayer {
    40 public:
    41     class Client : public CanMakeWeakPtr<Client> {
    42     public:
    43         virtual ~Client() = default;
    44         virtual void sampleBufferDisplayLayerStatusDidChange(SampleBufferDisplayLayer&) = 0;
    45         virtual void sampleBufferDisplayLayerBoundsDidChange(SampleBufferDisplayLayer&) = 0;
    46         virtual WTF::MediaTime streamTime() const = 0;
    47     };
    48 
    49     explicit SampleBufferDisplayLayer(Client&);
    50     virtual ~SampleBufferDisplayLayer() = default;
    51 
    52     virtual bool didFail() const = 0;
    53 
    54     virtual void updateDisplayMode(bool hideDisplayLayer, bool hideRootLayer) = 0;
    55 
    56     virtual CGRect bounds() const = 0;
    57     virtual void updateAffineTransform(CGAffineTransform) = 0;
    58     virtual void updateBoundsAndPosition(CGRect, CGPoint) = 0;
    59 
    60     virtual void flush() = 0;
    61     virtual void flushAndRemoveImage() = 0;
    62 
    63     virtual void enqueueSample(MediaSample&) = 0;
    64     virtual void clearEnqueuedSamples() = 0;
    65 
    66     virtual PlatformLayer* rootLayer() = 0;
    67 
    68 protected:
    69     WeakPtr<Client> m_client;
    70 };
    71 
    72 inline SampleBufferDisplayLayer::SampleBufferDisplayLayer(Client& client)
    73     : m_client(makeWeakPtr(client))
    74 {
     45#if ENABLE(MEDIA_STREAM) && USE(AVFOUNDATION)
     46    return LocalSampleBufferDisplayLayer::create(client, hideRootLayer, size);
     47#else
     48    return nullptr;
     49#endif
    7550}
    7651
  • trunk/Source/WebCore/platform/graphics/avfoundation/SampleBufferDisplayLayer.h

    r254813 r254817  
    4343        virtual ~Client() = default;
    4444        virtual void sampleBufferDisplayLayerStatusDidChange(SampleBufferDisplayLayer&) = 0;
    45         virtual void sampleBufferDisplayLayerBoundsDidChange(SampleBufferDisplayLayer&) = 0;
    4645        virtual WTF::MediaTime streamTime() const = 0;
    4746    };
    4847
    49     explicit SampleBufferDisplayLayer(Client&);
     48    WEBCORE_EXPORT static std::unique_ptr<SampleBufferDisplayLayer> create(Client&, bool hideRootLayer, IntSize);
     49    using LayerCreator = std::unique_ptr<SampleBufferDisplayLayer> (*)(Client&, bool hideRootLayer, IntSize);
     50    WEBCORE_EXPORT static void setCreator(LayerCreator);
     51
    5052    virtual ~SampleBufferDisplayLayer() = default;
    5153
     
    6769
    6870protected:
     71    explicit SampleBufferDisplayLayer(Client&);
     72
    6973    WeakPtr<Client> m_client;
     74
     75private:
     76    static LayerCreator m_layerCreator;
    7077};
    7178
  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.h

    r254813 r254817  
    4141    WTF_MAKE_FAST_ALLOCATED;
    4242public:
    43     static std::unique_ptr<SampleBufferDisplayLayer> create(Client&, bool hideRootLayer, IntSize);
     43    static std::unique_ptr<LocalSampleBufferDisplayLayer> create(Client&, bool hideRootLayer, IntSize);
    4444
    4545    LocalSampleBufferDisplayLayer(RetainPtr<AVSampleBufferDisplayLayer>&&, Client&, bool hideRootLayer, IntSize);
     
    5555    PlatformLayer* rootLayer() final;
    5656
    57 private:
     57    void updateRootLayerBoundsAndPosition(CGRect, CGPoint);
     58
    5859    bool didFail() const final;
    5960
     
    7071    void clearEnqueuedSamples() final;
    7172
    72     void ensureLayers();
    73 
     73private:
    7474    void removeOldSamplesFromPendingQueue();
    7575    void addSampleToPendingQueue(MediaSample&);
  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.mm

    r254813 r254817  
    3838
    3939#import <wtf/MainThread.h>
     40#import <wtf/cf/TypeCastsCF.h>
    4041
    4142#import <pal/cocoa/AVFoundationSoftLink.h>
     
    8586    [_parent->displayLayer() addObserver:self forKeyPath:@"status" options:NSKeyValueObservingOptionNew context:nil];
    8687    [_parent->displayLayer() addObserver:self forKeyPath:@"error" options:NSKeyValueObservingOptionNew context:nil];
    87     [_parent->rootLayer() addObserver:self forKeyPath:@"bounds" options:NSKeyValueObservingOptionNew context:nil];
    8888}
    8989
     
    9797        [_parent->displayLayer() removeObserver:self forKeyPath:@"error"];
    9898    }
    99     if (_parent->rootLayer())
    100         [_parent->rootLayer() removeObserver:self forKeyPath:@"bounds"];
    10199}
    102100
     
    105103    UNUSED_PARAM(context);
    106104    UNUSED_PARAM(keyPath);
     105    UNUSED_PARAM(change);
    107106    ASSERT(_parent);
    108107
     
    133132        }
    134133    }
    135 
    136     if ([[change valueForKey:NSKeyValueChangeNotificationIsPriorKey] boolValue])
    137         return;
    138 
    139     if ((CALayer *)object == _parent->rootLayer()) {
    140         if ([keyPath isEqualToString:@"bounds"]) {
    141             if (!_parent)
    142                 return;
    143 
    144             if (isMainThread()) {
    145                 _parent->rootLayerBoundsDidChange();
    146                 return;
    147             }
    148 
    149             callOnMainThread([protectedSelf = RetainPtr<WebAVSampleBufferStatusChangeListener>(self)] {
    150                 if (!protectedSelf->_parent)
    151                     return;
    152 
    153                 protectedSelf->_parent->rootLayerBoundsDidChange();
    154             });
    155         }
    156     }
    157 
    158134}
    159135@end
     
    170146}
    171147
    172 std::unique_ptr<SampleBufferDisplayLayer> LocalSampleBufferDisplayLayer::create(Client& client, bool hideRootLayer, IntSize size)
     148std::unique_ptr<LocalSampleBufferDisplayLayer> LocalSampleBufferDisplayLayer::create(Client& client, bool hideRootLayer, IntSize size)
    173149{
    174150    auto sampleBufferDisplayLayer = adoptNS([PAL::allocAVSampleBufferDisplayLayerInstance() init]);
     
    236212}
    237213
    238 void LocalSampleBufferDisplayLayer::rootLayerBoundsDidChange()
    239 {
    240     ASSERT(isMainThread());
    241     if (!m_client)
    242         return;
    243     m_client->sampleBufferDisplayLayerBoundsDidChange(*this);
    244 }
    245 
    246214PlatformLayer* LocalSampleBufferDisplayLayer::displayLayer()
    247215{
     
    285253{
    286254    runWithoutAnimations([&] {
     255        m_sampleBufferDisplayLayer.get().bounds = videoBounds;
     256        m_sampleBufferDisplayLayer.get().position = position;
     257    });
     258}
     259
     260void LocalSampleBufferDisplayLayer::updateRootLayerBoundsAndPosition(CGRect videoBounds, CGPoint position)
     261{
     262    runWithoutAnimations([&] {
     263        m_rootLayer.get().bounds = videoBounds;
     264        m_rootLayer.get().position = position;
    287265        m_sampleBufferDisplayLayer.get().bounds = videoBounds;
    288266        m_sampleBufferDisplayLayer.get().position = position;
  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h

    r254813 r254817  
    3737
    3838OBJC_CLASS AVSampleBufferDisplayLayer;
    39 OBJC_CLASS WebAVSampleBufferStatusChangeListener;
     39OBJC_CLASS WebRootSampleBufferBoundsChangeListener;
    4040
    4141namespace PAL {
     
    8282    WTFLogChannel& logChannel() const final;
    8383#endif
     84
     85    PlatformLayer* rootLayer() const;
     86    void rootLayerBoundsDidChange();
    8487
    8588private:
     
    245248    // SampleBufferDisplayLayer::Client
    246249    void sampleBufferDisplayLayerStatusDidChange(SampleBufferDisplayLayer&) final;
    247     void sampleBufferDisplayLayerBoundsDidChange(SampleBufferDisplayLayer&) final;
    248250
    249251#if !RELEASE_LOG_DISABLED
     
    251253    const void* m_logIdentifier;
    252254#endif
     255
     256    RetainPtr<WebRootSampleBufferBoundsChangeListener> m_boundsChangeListener;
    253257
    254258    bool m_videoMirrored { false };
  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm

    r254813 r254817  
    4949#import <pal/cocoa/AVFoundationSoftLink.h>
    5050
     51@interface WebRootSampleBufferBoundsChangeListener : NSObject {
     52    WeakPtr<WebCore::MediaPlayerPrivateMediaStreamAVFObjC> _parent;
     53}
     54
     55- (id)initWithParent:(WebCore::MediaPlayerPrivateMediaStreamAVFObjC*)callback;
     56- (void)invalidate;
     57- (void)begin;
     58- (void)stop;
     59@end
     60
     61@implementation WebRootSampleBufferBoundsChangeListener
     62
     63- (id)initWithParent:(WebCore::MediaPlayerPrivateMediaStreamAVFObjC*)parent
     64{
     65    if (!(self = [super init]))
     66        return nil;
     67
     68    _parent = makeWeakPtr(parent);
     69
     70    return self;
     71}
     72
     73- (void)dealloc
     74{
     75    [self invalidate];
     76    [super dealloc];
     77}
     78
     79- (void)invalidate
     80{
     81    [self stop];
     82    _parent = nullptr;
     83}
     84
     85- (void)begin
     86{
     87    ASSERT(_parent);
     88    ASSERT(_parent->rootLayer());
     89
     90    [_parent->rootLayer() addObserver:self forKeyPath:@"bounds" options:NSKeyValueObservingOptionNew context:nil];
     91}
     92
     93- (void)stop
     94{
     95    if (!_parent)
     96        return;
     97
     98    if (_parent->rootLayer())
     99        [_parent->rootLayer() removeObserver:self forKeyPath:@"bounds"];
     100}
     101
     102- (void)observeValueForKeyPath:(NSString *)keyPath ofObject:(id)object change:(NSDictionary *)change context:(void *)context
     103{
     104    UNUSED_PARAM(context);
     105    UNUSED_PARAM(keyPath);
     106    ASSERT(_parent);
     107
     108    if (!_parent)
     109        return;
     110
     111    if ([[change valueForKey:NSKeyValueChangeNotificationIsPriorKey] boolValue])
     112        return;
     113
     114    if ((CALayer *)object == _parent->rootLayer()) {
     115        if ([keyPath isEqualToString:@"bounds"]) {
     116            if (!_parent)
     117                return;
     118
     119            if (isMainThread()) {
     120                _parent->rootLayerBoundsDidChange();
     121                return;
     122            }
     123
     124            callOnMainThread([protectedSelf = RetainPtr<WebRootSampleBufferBoundsChangeListener>(self)] {
     125                if (!protectedSelf->_parent)
     126                    return;
     127
     128                protectedSelf->_parent->rootLayerBoundsDidChange();
     129            });
     130        }
     131    }
     132
     133}
     134@end
     135
    51136namespace WebCore {
    52137using namespace PAL;
     
    65150    , m_logIdentifier(player->mediaPlayerLogIdentifier())
    66151#endif
     152    , m_boundsChangeListener(adoptNS([[WebRootSampleBufferBoundsChangeListener alloc] initWithParent:this]))
    67153{
    68154    INFO_LOG(LOGIDENTIFIER);
     
    82168            track->removeObserver(*this);
    83169    }
     170
     171    [m_boundsChangeListener invalidate];
    84172
    85173    destroyLayers();
     
    285373
    286374    auto size = snappedIntRect(m_player->playerContentBoxRect()).size();
    287     m_sampleBufferDisplayLayer = LocalSampleBufferDisplayLayer::create(*this, hideRootLayer(), size);
     375    m_sampleBufferDisplayLayer = SampleBufferDisplayLayer::create(*this, hideRootLayer(), size);
    288376
    289377    if (!m_sampleBufferDisplayLayer) {
     
    296384
    297385    m_videoFullscreenLayerManager->setVideoLayer(m_sampleBufferDisplayLayer->rootLayer(), size);
     386
     387    [m_boundsChangeListener begin];
    298388}
    299389
     
    364454{
    365455    INFO_LOG(LOGIDENTIFIER);
     456}
     457
     458PlatformLayer* MediaPlayerPrivateMediaStreamAVFObjC::rootLayer() const
     459{
     460    return m_sampleBufferDisplayLayer ? m_sampleBufferDisplayLayer->rootLayer() : nullptr;
    366461}
    367462
     
    9491044        return;
    9501045
    951     auto bounds = m_sampleBufferDisplayLayer->bounds();
     1046    auto bounds = rootLayer().bounds;
    9521047    auto videoBounds = bounds;
    9531048    if (m_videoRotation == MediaSample::VideoRotation::Right || m_videoRotation == MediaSample::VideoRotation::Left)
     
    9571052}
    9581053
    959 void MediaPlayerPrivateMediaStreamAVFObjC::sampleBufferDisplayLayerBoundsDidChange(SampleBufferDisplayLayer&)
     1054void MediaPlayerPrivateMediaStreamAVFObjC::rootLayerBoundsDidChange()
    9601055{
    9611056    updateDisplayLayer();
  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h

    r254778 r254817  
    3939    static Ref<MediaSampleAVFObjC> create(CMSampleBufferRef sample, VideoRotation rotation = VideoRotation::None, bool mirrored = false) { return adoptRef(*new MediaSampleAVFObjC(sample, rotation, mirrored)); }
    4040    static RefPtr<MediaSampleAVFObjC> createImageSample(Vector<uint8_t>&&, unsigned long width, unsigned long height);
     41
     42    WEBCORE_EXPORT static void setAsDisplayImmediately(MediaSample&);
    4143
    4244    RefPtr<JSC::Uint8ClampedArray> getRGBAImageData() const override;
  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm

    r254778 r254817  
    312312}
    313313
     314void MediaSampleAVFObjC::setAsDisplayImmediately(MediaSample& sample)
     315{
     316    CFArrayRef attachmentsArray = CMSampleBufferGetSampleAttachmentsArray(sample.platformSample().sample.cmSampleBuffer, true);
     317    for (CFIndex i = 0; i < CFArrayGetCount(attachmentsArray); ++i) {
     318        CFMutableDictionaryRef attachments = checked_cf_cast<CFMutableDictionaryRef>(CFArrayGetValueAtIndex(attachmentsArray, i));
     319        CFDictionarySetValue(attachments, kCMSampleAttachmentKey_DisplayImmediately, kCFBooleanTrue);
     320    }
     321}
     322
    314323String MediaSampleAVFObjC::toJSONString() const
    315324{
  • trunk/Source/WebCore/platform/graphics/cg/PDFDocumentImage.cpp

    r254807 r254817  
    4343#include <CoreGraphics/CGContext.h>
    4444#include <CoreGraphics/CGPDFDocument.h>
     45#include <pal/spi/cg/CoreGraphicsSPI.h>
    4546#include <wtf/MathExtras.h>
    4647#include <wtf/RAMSize.h>
  • trunk/Source/WebKit/ChangeLog

    r254809 r254817  
     12020-01-20  youenn fablet  <youenn@apple.com>
     2
     3        Add support for MediaStream video track rendering in GPUProcess
     4        https://bugs.webkit.org/show_bug.cgi?id=206286
     5
     6        Reviewed by Eric Carlson.
     7
     8        Implement WebProcess SampleBufferDisplayLayer and GPUProcess RemoteSampleBufferDisplayLayer.
     9        RemoteSampleBufferDisplayLayer is a wrapper around LocalSampleBufferDisplayLayer.
     10
     11        A synchronous IPC is used at creation time of the layer to make sure we have a shared layer synchronously.
     12        Future refactoring should allow to remove that constraint as done in regular video playing.
     13
     14        Override SampleBufferDisplayLayer create function pointer when rendering is to be done in GPUProcess.
     15
     16        * DerivedSources-input.xcfilelist:
     17        * DerivedSources-output.xcfilelist:
     18        * DerivedSources.make:
     19        * GPUProcess/GPUConnectionToWebProcess.cpp:
     20        (WebKit::GPUConnectionToWebProcess::sampleBufferDisplayLayerManager):
     21        (WebKit::GPUConnectionToWebProcess::didReceiveMessage):
     22        (WebKit::GPUConnectionToWebProcess::didReceiveSyncMessage):
     23        * GPUProcess/GPUConnectionToWebProcess.h:
     24        * GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.cpp: Added.
     25        (WebKit::RemoteSampleBufferDisplayLayer::create):
     26        (WebKit::RemoteSampleBufferDisplayLayer::RemoteSampleBufferDisplayLayer):
     27        (WebKit::RemoteSampleBufferDisplayLayer::~RemoteSampleBufferDisplayLayer):
     28        (WebKit::RemoteSampleBufferDisplayLayer::contextID):
     29        (WebKit::RemoteSampleBufferDisplayLayer::bounds const):
     30        (WebKit::RemoteSampleBufferDisplayLayer::updateDisplayMode):
     31        (WebKit::RemoteSampleBufferDisplayLayer::updateAffineTransform):
     32        (WebKit::RemoteSampleBufferDisplayLayer::updateBoundsAndPosition):
     33        (WebKit::RemoteSampleBufferDisplayLayer::flush):
     34        (WebKit::RemoteSampleBufferDisplayLayer::flushAndRemoveImage):
     35        (WebKit::RemoteSampleBufferDisplayLayer::enqueueSample):
     36        (WebKit::RemoteSampleBufferDisplayLayer::clearEnqueuedSamples):
     37        (WebKit::RemoteSampleBufferDisplayLayer::messageSenderConnection const):
     38        (WebKit::RemoteSampleBufferDisplayLayer::sampleBufferDisplayLayerStatusDidChange):
     39        (WebKit::RemoteSampleBufferDisplayLayer::streamTime const):
     40        * GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.h: Added.
     41        * GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.messages.in: Added.
     42        * GPUProcess/webrtc/RemoteSampleBufferDisplayLayerManager.cpp: Added.
     43        (WebKit::RemoteSampleBufferDisplayLayerManager::RemoteSampleBufferDisplayLayerManager):
     44        (WebKit::RemoteSampleBufferDisplayLayerManager::didReceiveLayerMessage):
     45        (WebKit::RemoteSampleBufferDisplayLayerManager::createLayer):
     46        (WebKit::RemoteSampleBufferDisplayLayerManager::releaseLayer):
     47        * GPUProcess/webrtc/RemoteSampleBufferDisplayLayerManager.h: Added.
     48        * GPUProcess/webrtc/RemoteSampleBufferDisplayLayerManager.messages.in: Added.
     49        * Scripts/webkit/messages.py:
     50        * Shared/WebCoreArgumentCoders.h:
     51        * Shared/mac/WebCoreArgumentCodersMac.mm:
     52        (IPC::ArgumentCoder<CGRect>::encode):
     53        (IPC::ArgumentCoder<CGRect>::decode):
     54        (IPC::ArgumentCoder<CGSize>::encode):
     55        (IPC::ArgumentCoder<CGSize>::decode):
     56        (IPC::ArgumentCoder<CGPoint>::encode):
     57        (IPC::ArgumentCoder<CGPoint>::decode):
     58        (IPC::ArgumentCoder<CGAffineTransform>::encode):
     59        (IPC::ArgumentCoder<CGAffineTransform>::decode):
     60        * Sources.txt:
     61        * SourcesCocoa.txt:
     62        * WebKit.xcodeproj/project.pbxproj:
     63        * WebProcess/GPU/GPUProcessConnection.cpp:
     64        (WebKit::GPUProcessConnection::sampleBufferDisplayLayerManager):
     65        (WebKit::GPUProcessConnection::didReceiveMessage):
     66        * WebProcess/GPU/GPUProcessConnection.h:
     67        * WebProcess/GPU/media/RemoteMediaPlayerManager.cpp:
     68        (WebKit::RemoteMediaPlayerManager::updatePreferences):
     69        * WebProcess/GPU/webrtc/SampleBufferDisplayLayer.cpp: Added.
     70        (WebKit::SampleBufferDisplayLayer::create):
     71        (WebKit::SampleBufferDisplayLayer::SampleBufferDisplayLayer):
     72        (WebKit::SampleBufferDisplayLayer::~SampleBufferDisplayLayer):
     73        (WebKit::SampleBufferDisplayLayer::didFail const):
     74        (WebKit::SampleBufferDisplayLayer::updateDisplayMode):
     75        (WebKit::SampleBufferDisplayLayer::bounds const):
     76        (WebKit::SampleBufferDisplayLayer::updateAffineTransform):
     77        (WebKit::SampleBufferDisplayLayer::updateBoundsAndPosition):
     78        (WebKit::SampleBufferDisplayLayer::flush):
     79        (WebKit::SampleBufferDisplayLayer::flushAndRemoveImage):
     80        (WebKit::SampleBufferDisplayLayer::enqueueSample):
     81        (WebKit::SampleBufferDisplayLayer::clearEnqueuedSamples):
     82        (WebKit::SampleBufferDisplayLayer::rootLayer):
     83        (WebKit::SampleBufferDisplayLayer::setDidFail):
     84        (WebKit::SampleBufferDisplayLayer::boundsChanged):
     85        * WebProcess/GPU/webrtc/SampleBufferDisplayLayer.h: Added.
     86        * WebProcess/GPU/webrtc/SampleBufferDisplayLayer.messages.in: Added.
     87        * WebProcess/GPU/webrtc/SampleBufferDisplayLayerIdentifier.h: Added.
     88        * WebProcess/GPU/webrtc/SampleBufferDisplayLayerManager.cpp: Added.
     89        (WebKit::SampleBufferDisplayLayerManager::didReceiveLayerMessage):
     90        (WebKit::SampleBufferDisplayLayerManager::createLayer):
     91        (WebKit::SampleBufferDisplayLayerManager::addLayer):
     92        (WebKit::SampleBufferDisplayLayerManager::removeLayer):
     93        * WebProcess/GPU/webrtc/SampleBufferDisplayLayerManager.h: Added.
     94        * WebProcess/WebProcess.h:
     95
    1962020-01-19  Eric Carlson  <eric.carlson@apple.com>
    297
  • trunk/Source/WebKit/DerivedSources-input.xcfilelist

    r254809 r254817  
    2525$(PROJECT_DIR)/GPUProcess/webrtc/RemoteMediaRecorder.messages.in
    2626$(PROJECT_DIR)/GPUProcess/webrtc/RemoteMediaRecorderManager.messages.in
     27$(PROJECT_DIR)/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.messages.in
     28$(PROJECT_DIR)/GPUProcess/webrtc/RemoteSampleBufferDisplayLayerManager.messages.in
    2729$(PROJECT_DIR)/NetworkProcess/Cookies/WebCookieManager.messages.in
    2830$(PROJECT_DIR)/NetworkProcess/CustomProtocols/LegacyCustomProtocolManager.messages.in
     
    110112$(PROJECT_DIR)/WebProcess/GPU/webrtc/LibWebRTCCodecs.messages.in
    111113$(PROJECT_DIR)/WebProcess/GPU/webrtc/LibWebRTCRemoteCodecs.messages.in
     114$(PROJECT_DIR)/WebProcess/GPU/webrtc/SampleBufferDisplayLayer.messages.in
    112115$(PROJECT_DIR)/WebProcess/Geolocation/WebGeolocationManager.messages.in
    113116$(PROJECT_DIR)/WebProcess/Network/NetworkProcessConnection.messages.in
  • trunk/Source/WebKit/DerivedSources-output.xcfilelist

    r254809 r254817  
    155155$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteObjectRegistryMessages.h
    156156$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteObjectRegistryMessagesReplies.h
     157$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteSampleBufferDisplayLayerManagerMessageReceiver.cpp
     158$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteSampleBufferDisplayLayerManagerMessages.h
     159$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteSampleBufferDisplayLayerManagerMessagesReplies.h
     160$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteSampleBufferDisplayLayerMessageReceiver.cpp
     161$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteSampleBufferDisplayLayerMessages.h
     162$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteSampleBufferDisplayLayerMessagesReplies.h
    157163$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteScrollingCoordinatorMessageReceiver.cpp
    158164$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteScrollingCoordinatorMessages.h
     
    164170$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteWebInspectorUIMessages.h
    165171$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteWebInspectorUIMessagesReplies.h
     172$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/SampleBufferDisplayLayerMessageReceiver.cpp
     173$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/SampleBufferDisplayLayerMessages.h
     174$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/SampleBufferDisplayLayerMessagesReplies.h
    166175$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/SecItemShimProxyMessageReceiver.cpp
    167176$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/SecItemShimProxyMessages.h
  • trunk/Source/WebKit/DerivedSources.make

    r254809 r254817  
    152152    RemoteMediaResourceManager \
    153153    RemoteObjectRegistry \
     154    RemoteSampleBufferDisplayLayer \
     155    RemoteSampleBufferDisplayLayerManager \
    154156    RemoteScrollingCoordinator \
    155157    RemoteWebInspectorProxy \
    156158    RemoteWebInspectorUI \
     159    SampleBufferDisplayLayer \
    157160    SecItemShimProxy \
    158161    ServiceWorkerFetchTask \
  • trunk/Source/WebKit/GPUProcess/GPUConnectionToWebProcess.cpp

    r254778 r254817  
    5050#include "RemoteMediaResourceManager.h"
    5151#include "RemoteMediaResourceManagerMessages.h"
     52#include "RemoteSampleBufferDisplayLayerManager.h"
     53#include "RemoteSampleBufferDisplayLayerManagerMessages.h"
     54#include "RemoteSampleBufferDisplayLayerMessages.h"
    5255#include "RemoteScrollingCoordinatorTransaction.h"
    5356#include "UserMediaCaptureManagerProxy.h"
     
    163166    return *m_audioTrackRendererManager;
    164167}
     168
     169RemoteSampleBufferDisplayLayerManager& GPUConnectionToWebProcess::sampleBufferDisplayLayerManager()
     170{
     171    if (!m_sampleBufferDisplayLayerManager)
     172        m_sampleBufferDisplayLayerManager = makeUnique<RemoteSampleBufferDisplayLayerManager>(m_connection.copyRef());
     173
     174    return *m_sampleBufferDisplayLayerManager;
     175}
    165176#endif
    166177#endif
     
    210221        return;
    211222    }
     223    if (decoder.messageReceiverName() == Messages::RemoteSampleBufferDisplayLayerManager::messageReceiverName()) {
     224        sampleBufferDisplayLayerManager().didReceiveMessageFromWebProcess(connection, decoder);
     225        return;
     226    }
     227    if (decoder.messageReceiverName() == Messages::RemoteSampleBufferDisplayLayer::messageReceiverName()) {
     228        sampleBufferDisplayLayerManager().didReceiveLayerMessage(connection, decoder);
     229        return;
     230    }
    212231#endif
    213232#endif
     
    232251        return;
    233252    }
     253#if PLATFORM(COCOA) && ENABLE(VIDEO_TRACK)
     254    if (decoder.messageReceiverName() == Messages::RemoteSampleBufferDisplayLayerManager::messageReceiverName()) {
     255        sampleBufferDisplayLayerManager().didReceiveSyncMessageFromWebProcess(connection, decoder, replyEncoder);
     256        return;
     257    }
     258#endif
    234259#endif
    235260
  • trunk/Source/WebKit/GPUProcess/GPUConnectionToWebProcess.h

    r254778 r254817  
    4545class RemoteMediaRecorderManager;
    4646class RemoteMediaResourceManager;
     47class RemoteSampleBufferDisplayLayerManager;
    4748class UserMediaCaptureManagerProxy;
    4849
     
    7778#if ENABLE(VIDEO_TRACK)
    7879    RemoteAudioMediaStreamTrackRendererManager& audioTrackRendererManager();
     80    RemoteSampleBufferDisplayLayerManager& sampleBufferDisplayLayerManager();
    7981#endif
    8082#endif
     
    99101#if ENABLE(VIDEO_TRACK)
    100102    std::unique_ptr<RemoteAudioMediaStreamTrackRendererManager> m_audioTrackRendererManager;
     103    std::unique_ptr<RemoteSampleBufferDisplayLayerManager> m_sampleBufferDisplayLayerManager;
    101104#endif
    102105#endif
  • trunk/Source/WebKit/Scripts/webkit/messages.py

    r254778 r254817  
    226226        'WebKit::RTCDecoderIdentifier',
    227227        'WebKit::RTCEncoderIdentifier',
     228        'WebKit::SampleBufferDisplayLayerIdentifier',
    228229        'WebKit::StorageAreaIdentifier',
    229230        'WebKit::StorageAreaImplIdentifier',
  • trunk/Source/WebKit/Shared/WebCoreArgumentCoders.h

    r254778 r254817  
    447447    static Optional<WebCore::KeypressCommand> decode(Decoder&);
    448448};
     449
     450template<> struct ArgumentCoder<CGPoint> {
     451    static void encode(Encoder&, CGPoint);
     452    static Optional<CGPoint> decode(Decoder&);
     453};
     454
     455template<> struct ArgumentCoder<CGSize> {
     456    static void encode(Encoder&, CGSize);
     457    static Optional<CGSize> decode(Decoder&);
     458};
     459
     460template<> struct ArgumentCoder<CGRect> {
     461    static void encode(Encoder&, CGRect);
     462    static Optional<CGRect> decode(Decoder&);
     463};
     464
     465template<> struct ArgumentCoder<CGAffineTransform> {
     466    static void encode(Encoder&, CGAffineTransform);
     467    static Optional<CGAffineTransform> decode(Decoder&);
     468};
    449469#endif
    450470
  • trunk/Source/WebKit/Shared/mac/WebCoreArgumentCodersMac.mm

    r254778 r254817  
    549549}
    550550
     551void ArgumentCoder<CGRect>::encode(Encoder& encoder, CGRect rect)
     552{
     553    encoder << rect.origin << rect.size;
     554}
     555
     556Optional<CGRect> ArgumentCoder<CGRect>::decode(Decoder& decoder)
     557{
     558    Optional<CGPoint> origin;
     559    decoder >> origin;
     560    if (!origin)
     561        return { };
     562
     563    Optional<CGSize> size;
     564    decoder >> size;
     565    if (!size)
     566        return { };
     567
     568    return CGRect { *origin, *size };
     569}
     570
     571void ArgumentCoder<CGSize>::encode(Encoder& encoder, CGSize size)
     572{
     573    encoder << size.width << size.height;
     574}
     575
     576Optional<CGSize> ArgumentCoder<CGSize>::decode(Decoder& decoder)
     577{
     578    CGSize size;
     579    if (!decoder.decode(size.width))
     580        return { };
     581    if (!decoder.decode(size.height))
     582        return { };
     583    return size;
     584}
     585
     586void ArgumentCoder<CGPoint>::encode(Encoder& encoder, CGPoint point)
     587{
     588    encoder << point.x << point.y;
     589}
     590
     591Optional<CGPoint> ArgumentCoder<CGPoint>::decode(Decoder& decoder)
     592{
     593    CGPoint point;
     594    if (!decoder.decode(point.x))
     595        return { };
     596    if (!decoder.decode(point.y))
     597        return { };
     598    return point;
     599}
     600
     601void ArgumentCoder<CGAffineTransform>::encode(Encoder& encoder, CGAffineTransform transform)
     602{
     603    encoder << transform.a << transform.b << transform.c << transform.d << transform.tx << transform.ty;
     604}
     605
     606Optional<CGAffineTransform> ArgumentCoder<CGAffineTransform>::decode(Decoder& decoder)
     607{
     608    CGAffineTransform transform;
     609    if (!decoder.decode(transform.a))
     610        return { };
     611    if (!decoder.decode(transform.b))
     612        return { };
     613    if (!decoder.decode(transform.c))
     614        return { };
     615    if (!decoder.decode(transform.d))
     616        return { };
     617    if (!decoder.decode(transform.tx))
     618        return { };
     619    if (!decoder.decode(transform.ty))
     620        return { };
     621    return transform;
     622}
     623
    551624#if ENABLE(CONTENT_FILTERING)
    552625
  • trunk/Source/WebKit/Sources.txt

    r254778 r254817  
    2929GPUProcess/webrtc/RemoteMediaRecorder.cpp
    3030GPUProcess/webrtc/RemoteMediaRecorderManager.cpp
     31GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.cpp
     32GPUProcess/webrtc/RemoteSampleBufferDisplayLayerManager.cpp
    3133GPUProcess/media/RemoteAudioTrackProxy.cpp
    3234GPUProcess/media/RemoteMediaPlayerManagerProxy.cpp
     
    530532WebProcess/GPU/webrtc/MediaRecorderProvider.cpp
    531533WebProcess/GPU/webrtc/AudioMediaStreamTrackRenderer.cpp
     534WebProcess/GPU/webrtc/SampleBufferDisplayLayer.cpp
     535WebProcess/GPU/webrtc/SampleBufferDisplayLayerManager.cpp
    532536
    533537WebProcess/Network/NetworkProcessConnection.cpp
  • trunk/Source/WebKit/SourcesCocoa.txt

    r254778 r254817  
    645645RemoteMediaRecorderMessageReceiver.cpp
    646646RemoteMediaRecorderManagerMessageReceiver.cpp
     647RemoteSampleBufferDisplayLayerManagerMessageReceiver.cpp
     648RemoteSampleBufferDisplayLayerMessageReceiver.cpp
     649SampleBufferDisplayLayerMessageReceiver.cpp
    647650ServiceWorkerFetchTaskMessageReceiver.cpp
    648651TextCheckingControllerProxyMessageReceiver.cpp
  • trunk/Source/WebKit/WebKit.xcodeproj/project.pbxproj

    r254809 r254817  
    33043304                3F915C0F1F564DED00183CE9 /* WKFullScreenWindowControllerIOS.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; name = WKFullScreenWindowControllerIOS.h; path = ios/fullscreen/WKFullScreenWindowControllerIOS.h; sourceTree = "<group>"; };
    33053305                3FB08E421F60B240005E5312 /* iOS.xcassets */ = {isa = PBXFileReference; lastKnownFileType = folder.assetcatalog; path = iOS.xcassets; sourceTree = "<group>"; };
     3306                41024FC023CF104500FDF98E /* SampleBufferDisplayLayerManager.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = SampleBufferDisplayLayerManager.h; sourceTree = "<group>"; };
     3307                41024FC123CF104500FDF98E /* SampleBufferDisplayLayerManager.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = SampleBufferDisplayLayerManager.cpp; sourceTree = "<group>"; };
    33063308                4103FBA12061BDB700C2EAF8 /* NetworkLoadChecker.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = NetworkLoadChecker.h; sourceTree = "<group>"; };
    33073309                4103FBA22061BDB800C2EAF8 /* NetworkLoadChecker.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = NetworkLoadChecker.cpp; sourceTree = "<group>"; };
     
    34353437                41FCD6AE23CCC26100C62567 /* AudioMediaStreamTrackRenderer.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = AudioMediaStreamTrackRenderer.cpp; sourceTree = "<group>"; };
    34363438                41FCD6AF23CCC26100C62567 /* AudioMediaStreamTrackRendererIdentifier.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AudioMediaStreamTrackRendererIdentifier.h; sourceTree = "<group>"; };
     3439                41FCD6B323CDC60400C62567 /* SampleBufferDisplayLayer.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = SampleBufferDisplayLayer.h; sourceTree = "<group>"; };
     3440                41FCD6B423CDCAC200C62567 /* RemoteSampleBufferDisplayLayer.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = RemoteSampleBufferDisplayLayer.h; sourceTree = "<group>"; };
     3441                41FCD6B523CDE1C600C62567 /* RemoteSampleBufferDisplayLayer.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = RemoteSampleBufferDisplayLayer.cpp; sourceTree = "<group>"; };
     3442                41FCD6B623CDE1C600C62567 /* RemoteSampleBufferDisplayLayer.messages.in */ = {isa = PBXFileReference; lastKnownFileType = text; path = RemoteSampleBufferDisplayLayer.messages.in; sourceTree = "<group>"; };
     3443                41FCD6B723CDE53D00C62567 /* SampleBufferDisplayLayer.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = SampleBufferDisplayLayer.cpp; sourceTree = "<group>"; };
     3444                41FCD6B823CDE53E00C62567 /* SampleBufferDisplayLayer.messages.in */ = {isa = PBXFileReference; lastKnownFileType = text; path = SampleBufferDisplayLayer.messages.in; sourceTree = "<group>"; };
     3445                41FCD6BC23CE031800C62567 /* SampleBufferDisplayLayerIdentifier.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = SampleBufferDisplayLayerIdentifier.h; sourceTree = "<group>"; };
     3446                41FCD6BD23CE043F00C62567 /* RemoteSampleBufferDisplayLayerManager.messages.in */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text; path = RemoteSampleBufferDisplayLayerManager.messages.in; sourceTree = "<group>"; };
     3447                41FCD6BE23CE044000C62567 /* RemoteSampleBufferDisplayLayerManager.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = RemoteSampleBufferDisplayLayerManager.cpp; sourceTree = "<group>"; };
     3448                41FCD6BF23CE044000C62567 /* RemoteSampleBufferDisplayLayerManager.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = RemoteSampleBufferDisplayLayerManager.h; sourceTree = "<group>"; };
    34373449                4450AEBF1DC3FAE5009943F2 /* SharedMemoryCocoa.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = SharedMemoryCocoa.cpp; sourceTree = "<group>"; };
    34383450                4459984122833E6000E61373 /* SyntheticEditingCommandType.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = SyntheticEditingCommandType.h; sourceTree = "<group>"; };
     
    72437255                                4172198C23B612E800AE5686 /* RTCDecoderIdentifier.h */,
    72447256                                4158649A23BE092400A0A61E /* RTCEncoderIdentifier.h */,
     7257                                41FCD6B723CDE53D00C62567 /* SampleBufferDisplayLayer.cpp */,
     7258                                41FCD6B323CDC60400C62567 /* SampleBufferDisplayLayer.h */,
     7259                                41FCD6B823CDE53E00C62567 /* SampleBufferDisplayLayer.messages.in */,
     7260                                41FCD6BC23CE031800C62567 /* SampleBufferDisplayLayerIdentifier.h */,
     7261                                41024FC123CF104500FDF98E /* SampleBufferDisplayLayerManager.cpp */,
     7262                                41024FC023CF104500FDF98E /* SampleBufferDisplayLayerManager.h */,
    72457263                        );
    72467264                        path = webrtc;
     
    72657283                                4176E89923C36D8E003E83FE /* RemoteMediaRecorderManager.h */,
    72667284                                4176E89B23C37222003E83FE /* RemoteMediaRecorderManager.messages.in */,
     7285                                41FCD6B523CDE1C600C62567 /* RemoteSampleBufferDisplayLayer.cpp */,
     7286                                41FCD6B423CDCAC200C62567 /* RemoteSampleBufferDisplayLayer.h */,
     7287                                41FCD6B623CDE1C600C62567 /* RemoteSampleBufferDisplayLayer.messages.in */,
     7288                                41FCD6BE23CE044000C62567 /* RemoteSampleBufferDisplayLayerManager.cpp */,
     7289                                41FCD6BF23CE044000C62567 /* RemoteSampleBufferDisplayLayerManager.h */,
     7290                                41FCD6BD23CE043F00C62567 /* RemoteSampleBufferDisplayLayerManager.messages.in */,
    72677291                        );
    72687292                        path = webrtc;
  • trunk/Source/WebKit/WebProcess/GPU/GPUProcessConnection.cpp

    r254809 r254817  
    3636#include "RemoteMediaPlayerManager.h"
    3737#include "RemoteMediaPlayerManagerMessages.h"
     38#include "SampleBufferDisplayLayerMessages.h"
    3839#include "UserMediaCaptureManager.h"
    3940#include "UserMediaCaptureManagerMessages.h"
     
    6667}
    6768
     69#if PLATFORM(COCOA) && ENABLE(VIDEO_TRACK) && ENABLE(MEDIA_STREAM)
     70SampleBufferDisplayLayerManager& GPUProcessConnection::sampleBufferDisplayLayerManager()
     71{
     72    if (!m_sampleBufferDisplayLayerManager)
     73        m_sampleBufferDisplayLayerManager = makeUnique<SampleBufferDisplayLayerManager>();
     74    return *m_sampleBufferDisplayLayerManager;
     75}
     76#endif
     77
    6878void GPUProcessConnection::didReceiveMessage(IPC::Connection& connection, IPC::Decoder& decoder)
    6979{
     
    7989        return;
    8090    }
    81 #endif
    82    
     91#if PLATFORM(COCOA) && ENABLE(VIDEO_TRACK)
     92    if (decoder.messageReceiverName() == Messages::SampleBufferDisplayLayer::messageReceiverName()) {
     93        sampleBufferDisplayLayerManager().didReceiveLayerMessage(connection, decoder);
     94        return;
     95    }
     96#endif // PLATFORM(COCOA) && ENABLE(VIDEO_TRACK)
     97#endif // ENABLE(MEDIA_STREAM)
    8398#if USE(LIBWEBRTC) && PLATFORM(COCOA)
    8499    if (decoder.messageReceiverName() == Messages::LibWebRTCCodecs::messageReceiverName()) {
  • trunk/Source/WebKit/WebProcess/GPU/GPUProcessConnection.h

    r254778 r254817  
    2929
    3030#include "Connection.h"
     31#include "SampleBufferDisplayLayerManager.h"
    3132#include <wtf/RefCounted.h>
    3233#include <wtf/text/WTFString.h>
     
    5253    Optional<audit_token_t> auditToken() const { return m_auditToken; }
    5354#endif
     55#if PLATFORM(COCOA) && ENABLE(VIDEO_TRACK) && ENABLE(MEDIA_STREAM)
     56    SampleBufferDisplayLayerManager& sampleBufferDisplayLayerManager();
     57#endif
    5458
    5559private:
     
    6771    Optional<audit_token_t> m_auditToken;
    6872#endif
     73#if PLATFORM(COCOA) && ENABLE(VIDEO_TRACK) && ENABLE(MEDIA_STREAM)
     74    std::unique_ptr<SampleBufferDisplayLayerManager> m_sampleBufferDisplayLayerManager;
     75#endif
    6976};
    7077
  • trunk/Source/WebKit/WebProcess/GPU/media/RemoteMediaPlayerManager.cpp

    r254809 r254817  
    254254
    255255#if PLATFORM(COCOA) && ENABLE(VIDEO_TRACK) && ENABLE(MEDIA_STREAM)
    256     if (settings.useGPUProcessForMedia())
     256    if (settings.useGPUProcessForMedia()) {
    257257        WebCore::AudioMediaStreamTrackRenderer::setCreator(WebKit::AudioMediaStreamTrackRenderer::create);
     258        WebCore::SampleBufferDisplayLayer::setCreator([](auto& client, bool hideRootLayer, auto size) {
     259            return WebProcess::singleton().ensureGPUProcessConnection().sampleBufferDisplayLayerManager().createLayer(client, hideRootLayer, size);
     260        });
     261    }
    258262#endif
    259263}
  • trunk/Source/WebKit/WebProcess/GPU/webrtc/SampleBufferDisplayLayer.h

    r254816 r254817  
    2626#pragma once
    2727
    28 #if ENABLE(MEDIA_STREAM) && USE(AVFOUNDATION)
     28#if PLATFORM(COCOA) && ENABLE(GPU_PROCESS) && ENABLE(VIDEO_TRACK) && ENABLE(MEDIA_STREAM)
    2929
    30 #include "SampleBufferDisplayLayer.h"
    31 #include <wtf/Deque.h>
    32 #include <wtf/Forward.h>
    33 #include <wtf/RetainPtr.h>
     30#include "MessageReceiver.h"
     31#include "SampleBufferDisplayLayerIdentifier.h"
     32#include <WebCore/SampleBufferDisplayLayer.h>
     33#include <wtf/WeakPtr.h>
    3434
    35 OBJC_CLASS AVSampleBufferDisplayLayer;
    36 OBJC_CLASS WebAVSampleBufferStatusChangeListener;
     35namespace WebKit {
    3736
    38 namespace WebCore {
     37class SampleBufferDisplayLayerManager;
    3938
    40 class WEBCORE_EXPORT LocalSampleBufferDisplayLayer final : public SampleBufferDisplayLayer, public CanMakeWeakPtr<LocalSampleBufferDisplayLayer> {
    41     WTF_MAKE_FAST_ALLOCATED;
     39class SampleBufferDisplayLayer final : public WebCore::SampleBufferDisplayLayer, public IPC::MessageReceiver, public CanMakeWeakPtr<SampleBufferDisplayLayer> {
    4240public:
    43     static std::unique_ptr<SampleBufferDisplayLayer> create(Client&, bool hideRootLayer, IntSize);
     41    static std::unique_ptr<SampleBufferDisplayLayer> create(SampleBufferDisplayLayerManager&, Client&, bool hideRootLayer, WebCore::IntSize);
     42    ~SampleBufferDisplayLayer();
    4443
    45     LocalSampleBufferDisplayLayer(RetainPtr<AVSampleBufferDisplayLayer>&&, Client&, bool hideRootLayer, IntSize);
    46     ~LocalSampleBufferDisplayLayer();
     44    SampleBufferDisplayLayerIdentifier identifier() const { return m_identifier; }
    4745
    48     // API used by WebAVSampleBufferStatusChangeListener
    49     void layerStatusDidChange();
    50     void layerErrorDidChange();
    51     void rootLayerBoundsDidChange();
    52 
    53     PlatformLayer* displayLayer();
    54 
    55     PlatformLayer* rootLayer() final;
     46    void didReceiveMessage(IPC::Connection&, IPC::Decoder&) final;
    5647
    5748private:
     49    SampleBufferDisplayLayer(SampleBufferDisplayLayerManager&, Client&, bool hideRootLayer, WebCore::IntSize);
     50
     51    // WebCore::SampleBufferDisplayLayer
    5852    bool didFail() const final;
    59 
    6053    void updateDisplayMode(bool hideDisplayLayer, bool hideRootLayer) final;
    61 
    6254    CGRect bounds() const final;
    63     void updateAffineTransform(CGAffineTransform)  final;
     55    void updateAffineTransform(CGAffineTransform) final;
    6456    void updateBoundsAndPosition(CGRect, CGPoint) final;
    65 
    6657    void flush() final;
    6758    void flushAndRemoveImage() final;
     59    void enqueueSample(WebCore::MediaSample&) final;
     60    void clearEnqueuedSamples() final;
     61    PlatformLayer* rootLayer() final;
     62   
     63    void setDidFail(bool);
    6864
    69     void enqueueSample(MediaSample&) final;
    70     void clearEnqueuedSamples() final;
     65    WeakPtr<SampleBufferDisplayLayerManager> m_manager;
     66    Ref<IPC::Connection> m_connection;
     67    SampleBufferDisplayLayerIdentifier m_identifier;
    7168
    72     void ensureLayers();
    73 
    74     void removeOldSamplesFromPendingQueue();
    75     void addSampleToPendingQueue(MediaSample&);
    76     void requestNotificationWhenReadyForVideoData();
    77 
    78 private:
    79     RetainPtr<WebAVSampleBufferStatusChangeListener> m_statusChangeListener;
    80     RetainPtr<AVSampleBufferDisplayLayer> m_sampleBufferDisplayLayer;
    81     RetainPtr<PlatformLayer> m_rootLayer;
    82 
    83     using PendingSampleQueue = Deque<Ref<MediaSample>>;
    84     PendingSampleQueue m_pendingVideoSampleQueue;
     69    RetainPtr<PlatformLayer> m_videoLayer;
     70    CGRect m_bounds;
     71    bool m_didFail { false };
    8572};
    8673
    8774}
    8875
    89 #endif // ENABLE(MEDIA_STREAM) && USE(AVFOUNDATION)
     76#endif // PLATFORM(COCOA) && ENABLE(GPU_PROCESS) && ENABLE(VIDEO_TRACK) && ENABLE(MEDIA_STREAM)
  • trunk/Source/WebKit/WebProcess/GPU/webrtc/SampleBufferDisplayLayerManager.h

    r254816 r254817  
    2626#pragma once
    2727
    28 #include "PlatformLayer.h"
    29 #include <wtf/WeakPtr.h>
     28#if PLATFORM(COCOA) && ENABLE(GPU_PROCESS) && ENABLE(VIDEO_TRACK) && ENABLE(MEDIA_STREAM)
    3029
    31 namespace WTF {
    32 class MediaTime;
     30#include "SampleBufferDisplayLayer.h"
     31#include <wtf/HashMap.h>
     32
     33namespace WebKit {
     34
     35class SampleBufferDisplayLayerManager : public CanMakeWeakPtr<SampleBufferDisplayLayerManager> {
     36    WTF_MAKE_FAST_ALLOCATED;
     37public:
     38    SampleBufferDisplayLayerManager() = default;
     39    ~SampleBufferDisplayLayerManager() = default;
     40
     41    void addLayer(SampleBufferDisplayLayer&);
     42    void removeLayer(SampleBufferDisplayLayer&);
     43
     44    void didReceiveLayerMessage(IPC::Connection&, IPC::Decoder&);
     45    std::unique_ptr<WebCore::SampleBufferDisplayLayer> createLayer(SampleBufferDisplayLayer::Client&, bool hideRootLayer, WebCore::IntSize);
     46
     47private:
     48    HashMap<SampleBufferDisplayLayerIdentifier, WeakPtr<SampleBufferDisplayLayer>> m_layers;
     49};
     50
    3351}
    3452
    35 namespace WebCore {
    36 class IntSize;
    37 class MediaSample;
    38 
    39 class SampleBufferDisplayLayer {
    40 public:
    41     class Client : public CanMakeWeakPtr<Client> {
    42     public:
    43         virtual ~Client() = default;
    44         virtual void sampleBufferDisplayLayerStatusDidChange(SampleBufferDisplayLayer&) = 0;
    45         virtual void sampleBufferDisplayLayerBoundsDidChange(SampleBufferDisplayLayer&) = 0;
    46         virtual WTF::MediaTime streamTime() const = 0;
    47     };
    48 
    49     explicit SampleBufferDisplayLayer(Client&);
    50     virtual ~SampleBufferDisplayLayer() = default;
    51 
    52     virtual bool didFail() const = 0;
    53 
    54     virtual void updateDisplayMode(bool hideDisplayLayer, bool hideRootLayer) = 0;
    55 
    56     virtual CGRect bounds() const = 0;
    57     virtual void updateAffineTransform(CGAffineTransform) = 0;
    58     virtual void updateBoundsAndPosition(CGRect, CGPoint) = 0;
    59 
    60     virtual void flush() = 0;
    61     virtual void flushAndRemoveImage() = 0;
    62 
    63     virtual void enqueueSample(MediaSample&) = 0;
    64     virtual void clearEnqueuedSamples() = 0;
    65 
    66     virtual PlatformLayer* rootLayer() = 0;
    67 
    68 protected:
    69     WeakPtr<Client> m_client;
    70 };
    71 
    72 inline SampleBufferDisplayLayer::SampleBufferDisplayLayer(Client& client)
    73     : m_client(makeWeakPtr(client))
    74 {
    75 }
    76 
    77 }
     53#endif // PLATFORM(COCOA) && ENABLE(GPU_PROCESS) && ENABLE(VIDEO_TRACK) && ENABLE(MEDIA_STREAM)
Note: See TracChangeset for help on using the changeset viewer.