⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 254564 in webkit


Ignore:
Timestamp:
Jan 15, 2020, 6:24:12 AM (7 years ago)
Author:
youenn@apple.com
Message:

Introduce an abstract SampleBufferDisplayLayer
​https://bugs.webkit.org/show_bug.cgi?id=206066

Reviewed by Eric Carlson.

Move use of display layers in MediaPlayerPrivateMediaStreamAVFObjC to a new class LocalSampleBufferDisplayLayer
that implements an interface named SampleBufferDisplayLayer.
A future patch will implement this interface by IPCing to GPUProcess.
We move both layers and handling of the sample queue to LocalSampleBufferDisplayLayer.

Contrary to previously, we do not call again enqueueVideoSample in case we enqueued a sample for later use in the display layer.
Instead, we directly render it, which should not change much since this is a realtime track and in the future the buffer will be in GPUProcess anyway.

  • SourcesCocoa.txt:
  • WebCore.xcodeproj/project.pbxproj:
  • platform/graphics/avfoundation/SampleBufferDisplayLayer.h: Added.

(WebCore::SampleBufferDisplayLayer::SampleBufferDisplayLayer):

  • platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.h: Added.
  • platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.mm: Added.

(-[WebAVSampleBufferStatusChangeListener initWithParent:]):
(-[WebAVSampleBufferStatusChangeListener dealloc]):
(-[WebAVSampleBufferStatusChangeListener invalidate]):
(-[WebAVSampleBufferStatusChangeListener beginObservingLayers]):
(-[WebAVSampleBufferStatusChangeListener stopObservingLayers]):
(-[WebAVSampleBufferStatusChangeListener observeValueForKeyPath:ofObject:change:context:]):
(WebCore::runWithoutAnimations):
(WebCore::LocalSampleBufferDisplayLayer::LocalSampleBufferDisplayLayer):
(WebCore::LocalSampleBufferDisplayLayer::~LocalSampleBufferDisplayLayer):
(WebCore::LocalSampleBufferDisplayLayer::layerStatusDidChange):
(WebCore::LocalSampleBufferDisplayLayer::layerErrorDidChange):
(WebCore::LocalSampleBufferDisplayLayer::rootLayerBoundsDidChange):
(WebCore::LocalSampleBufferDisplayLayer::displayLayer):
(WebCore::LocalSampleBufferDisplayLayer::rootLayer):
(WebCore::LocalSampleBufferDisplayLayer::didFail const):
(WebCore::LocalSampleBufferDisplayLayer::updateDisplayMode):
(WebCore::LocalSampleBufferDisplayLayer::bounds const):
(WebCore::LocalSampleBufferDisplayLayer::updateAffineTransform):
(WebCore::LocalSampleBufferDisplayLayer::updateBoundsAndPosition):
(WebCore::LocalSampleBufferDisplayLayer::ensureLayers):
(WebCore::LocalSampleBufferDisplayLayer::flush):
(WebCore::LocalSampleBufferDisplayLayer::flushAndRemoveImage):
(WebCore::LocalSampleBufferDisplayLayer::enqueueSample):
(WebCore::LocalSampleBufferDisplayLayer::removeOldSamplesFromPendingQueue):
(WebCore::LocalSampleBufferDisplayLayer::addSampleToPendingQueue):
(WebCore::LocalSampleBufferDisplayLayer::clearEnqueuedSamples):
(WebCore::LocalSampleBufferDisplayLayer::requestNotificationWhenReadyForVideoData):

  • platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
  • platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:

(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::MediaPlayerPrivateMediaStreamAVFObjC):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::~MediaPlayerPrivateMediaStreamAVFObjC):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::enqueueCorrectedVideoSample):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::sampleBufferDisplayLayerStatusDidChange):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::applicationDidBecomeActive):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::flushRenderers):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::ensureLayers):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::destroyLayers):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::platformLayer const):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::displayLayer):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::updateDisplayMode):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::checkSelectedVideoTrack):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setBufferingPolicy):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::updateDisplayLayer):
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::sampleBufferDisplayLayerBoundsDidChange):

Location:
trunk/Source/WebCore
Files:
3 added
5 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebCore/ChangeLog

    r254562 r254564  
     12020-01-15  youenn fablet  <youenn@apple.com>
     2
     3        Introduce an abstract SampleBufferDisplayLayer
     4        https://bugs.webkit.org/show_bug.cgi?id=206066
     5
     6        Reviewed by Eric Carlson.
     7
     8        Move use of display layers in MediaPlayerPrivateMediaStreamAVFObjC to a new class LocalSampleBufferDisplayLayer
     9        that implements an interface named SampleBufferDisplayLayer.
     10        A future patch will implement this interface by IPCing to GPUProcess.
     11        We move both layers and handling of the sample queue to LocalSampleBufferDisplayLayer.
     12
     13        Contrary to previously, we do not call again enqueueVideoSample in case we enqueued a sample for later use in the display layer.
     14        Instead, we directly render it, which should not change much since this is a realtime track and in the future the buffer will be in GPUProcess anyway.
     15
     16        * SourcesCocoa.txt:
     17        * WebCore.xcodeproj/project.pbxproj:
     18        * platform/graphics/avfoundation/SampleBufferDisplayLayer.h: Added.
     19        (WebCore::SampleBufferDisplayLayer::SampleBufferDisplayLayer):
     20        * platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.h: Added.
     21        * platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.mm: Added.
     22        (-[WebAVSampleBufferStatusChangeListener initWithParent:]):
     23        (-[WebAVSampleBufferStatusChangeListener dealloc]):
     24        (-[WebAVSampleBufferStatusChangeListener invalidate]):
     25        (-[WebAVSampleBufferStatusChangeListener beginObservingLayers]):
     26        (-[WebAVSampleBufferStatusChangeListener stopObservingLayers]):
     27        (-[WebAVSampleBufferStatusChangeListener observeValueForKeyPath:ofObject:change:context:]):
     28        (WebCore::runWithoutAnimations):
     29        (WebCore::LocalSampleBufferDisplayLayer::LocalSampleBufferDisplayLayer):
     30        (WebCore::LocalSampleBufferDisplayLayer::~LocalSampleBufferDisplayLayer):
     31        (WebCore::LocalSampleBufferDisplayLayer::layerStatusDidChange):
     32        (WebCore::LocalSampleBufferDisplayLayer::layerErrorDidChange):
     33        (WebCore::LocalSampleBufferDisplayLayer::rootLayerBoundsDidChange):
     34        (WebCore::LocalSampleBufferDisplayLayer::displayLayer):
     35        (WebCore::LocalSampleBufferDisplayLayer::rootLayer):
     36        (WebCore::LocalSampleBufferDisplayLayer::didFail const):
     37        (WebCore::LocalSampleBufferDisplayLayer::updateDisplayMode):
     38        (WebCore::LocalSampleBufferDisplayLayer::bounds const):
     39        (WebCore::LocalSampleBufferDisplayLayer::updateAffineTransform):
     40        (WebCore::LocalSampleBufferDisplayLayer::updateBoundsAndPosition):
     41        (WebCore::LocalSampleBufferDisplayLayer::ensureLayers):
     42        (WebCore::LocalSampleBufferDisplayLayer::flush):
     43        (WebCore::LocalSampleBufferDisplayLayer::flushAndRemoveImage):
     44        (WebCore::LocalSampleBufferDisplayLayer::enqueueSample):
     45        (WebCore::LocalSampleBufferDisplayLayer::removeOldSamplesFromPendingQueue):
     46        (WebCore::LocalSampleBufferDisplayLayer::addSampleToPendingQueue):
     47        (WebCore::LocalSampleBufferDisplayLayer::clearEnqueuedSamples):
     48        (WebCore::LocalSampleBufferDisplayLayer::requestNotificationWhenReadyForVideoData):
     49        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
     50        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
     51        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::MediaPlayerPrivateMediaStreamAVFObjC):
     52        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::~MediaPlayerPrivateMediaStreamAVFObjC):
     53        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::enqueueCorrectedVideoSample):
     54        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::sampleBufferDisplayLayerStatusDidChange):
     55        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::applicationDidBecomeActive):
     56        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::flushRenderers):
     57        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::ensureLayers):
     58        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::destroyLayers):
     59        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::platformLayer const):
     60        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::displayLayer):
     61        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::updateDisplayMode):
     62        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::checkSelectedVideoTrack):
     63        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setBufferingPolicy):
     64        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::updateDisplayLayer):
     65        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::sampleBufferDisplayLayerBoundsDidChange):
     66
    1672020-01-15  youenn fablet  <youenn@apple.com>
    268

  • trunk/Source/WebCore/SourcesCocoa.txt

    r254446 r254564  
    254254platform/graphics/avfoundation/objc/InbandTextTrackPrivateAVFObjC.mm @no-unify
    255255platform/graphics/avfoundation/objc/InbandTextTrackPrivateLegacyAVFObjC.mm @no-unify
     256platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.mm @no-unify
    256257platform/graphics/avfoundation/objc/MediaPlaybackTargetPickerMac.mm @no-unify
    257258platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm @no-unify

  • trunk/Source/WebCore/WebCore.xcodeproj/project.pbxproj

    r254551 r254564  
    10681068                413CCD4A20DE034F0065A21A /* MockMediaDevice.h in Headers */ = {isa = PBXBuildFile; fileRef = 413CCD4820DE013C0065A21A /* MockMediaDevice.h */; settings = {ATTRIBUTES = (Private, ); }; };
    10691069                413E00791DB0E4F2002341D2 /* MemoryRelease.h in Headers */ = {isa = PBXBuildFile; fileRef = 413E00781DB0E4DE002341D2 /* MemoryRelease.h */; settings = {ATTRIBUTES = (Private, ); }; };
     1070                414598C223C8D177002B9CC8 /* LocalSampleBufferDisplayLayer.mm in Sources */ = {isa = PBXBuildFile; fileRef = 414598C123C8AD79002B9CC8 /* LocalSampleBufferDisplayLayer.mm */; };
    10701071                414B82051D6DF0E50077EBE3 /* StructuredClone.h in Headers */ = {isa = PBXBuildFile; fileRef = 414B82031D6DF0D90077EBE3 /* StructuredClone.h */; };
    10711072                414DEDE71F9FE91E0047C40D /* EmptyFrameLoaderClient.h in Headers */ = {isa = PBXBuildFile; fileRef = 414DEDE51F9FE9150047C40D /* EmptyFrameLoaderClient.h */; settings = {ATTRIBUTES = (Private, ); }; };
    … …  
    11541155                41F1D21F0EF35C2A00DA8753 /* ScriptCachedFrameData.h in Headers */ = {isa = PBXBuildFile; fileRef = 41F1D21D0EF35C2A00DA8753 /* ScriptCachedFrameData.h */; settings = {ATTRIBUTES = (Private, ); }; };
    11551156                41FABD2D1F4DFE4A006A6C97 /* DOMCacheEngine.h in Headers */ = {isa = PBXBuildFile; fileRef = 41FABD2B1F4DFE42006A6C97 /* DOMCacheEngine.h */; settings = {ATTRIBUTES = (Private, ); }; };
     1157                41FCD6B923CE015500C62567 /* SampleBufferDisplayLayer.h in Headers */ = {isa = PBXBuildFile; fileRef = 414598BE23C8AAB8002B9CC8 /* SampleBufferDisplayLayer.h */; settings = {ATTRIBUTES = (Private, ); }; };
     1158                41FCD6BB23CE027700C62567 /* LocalSampleBufferDisplayLayer.h in Headers */ = {isa = PBXBuildFile; fileRef = 414598C023C8AD78002B9CC8 /* LocalSampleBufferDisplayLayer.h */; settings = {ATTRIBUTES = (Private, ); }; };
    11561159                427DA71D13735DFA007C57FB /* JSServiceWorkerInternals.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 427DA71B13735DFA007C57FB /* JSServiceWorkerInternals.cpp */; };
    11571160                427DA71E13735DFA007C57FB /* JSServiceWorkerInternals.h in Headers */ = {isa = PBXBuildFile; fileRef = 427DA71C13735DFA007C57FB /* JSServiceWorkerInternals.h */; };
    … …  
    73477350                413E007B1DB0E707002341D2 /* MemoryReleaseCocoa.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = MemoryReleaseCocoa.mm; sourceTree = "<group>"; };
    73487351                413FC4CD1FD1DD8C00541C4B /* ServiceWorkerClientQueryOptions.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ServiceWorkerClientQueryOptions.h; sourceTree = "<group>"; };
     7352                414598BE23C8AAB8002B9CC8 /* SampleBufferDisplayLayer.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = SampleBufferDisplayLayer.h; sourceTree = "<group>"; };
     7353                414598C023C8AD78002B9CC8 /* LocalSampleBufferDisplayLayer.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = LocalSampleBufferDisplayLayer.h; sourceTree = "<group>"; };
     7354                414598C123C8AD79002B9CC8 /* LocalSampleBufferDisplayLayer.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = LocalSampleBufferDisplayLayer.mm; sourceTree = "<group>"; };
    73497355                4147E2B21C88337F00A7E715 /* FetchBodyOwner.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = FetchBodyOwner.h; sourceTree = "<group>"; };
    73507356                4147E2B31C89912600A7E715 /* FetchBodyOwner.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = FetchBodyOwner.cpp; sourceTree = "<group>"; };
    … …  
    1632616332                                CDBEAEAB19D92B6C00BEBA88 /* MediaSelectionGroupAVFObjC.h */,
    1632716333                                CDBEAEAA19D92B6C00BEBA88 /* MediaSelectionGroupAVFObjC.mm */,
     16334                                414598BE23C8AAB8002B9CC8 /* SampleBufferDisplayLayer.h */,
    1632816335                                CD336F6317FA0A4D00DDDCD0 /* VideoTrackPrivateAVF.h */,
    1632916336                                0709D7901AE5557E004E42F8 /* WebMediaSessionManagerMac.cpp */,
    … …  
    2674926756                                07367DDD172CA67F00D861B9 /* InbandTextTrackPrivateLegacyAVFObjC.h */,
    2675026757                                07367DDE172CA67F00D861B9 /* InbandTextTrackPrivateLegacyAVFObjC.mm */,
     26758                                414598C023C8AD78002B9CC8 /* LocalSampleBufferDisplayLayer.h */,
     26759                                414598C123C8AD79002B9CC8 /* LocalSampleBufferDisplayLayer.mm */,
    2675126760                                078E43DB1ABB6F6F001C2FA6 /* MediaPlaybackTargetPickerMac.h */,
    2675226761                                078E43DC1ABB6F6F001C2FA6 /* MediaPlaybackTargetPickerMac.mm */,
    … …  
    3139831407                                A516E8B7136E04DB0076C3C0 /* LocalizedDateCache.h in Headers */,
    3139931408                                935207BE09BD410A00F2038D /* LocalizedStrings.h in Headers */,
     31409                                41FCD6BB23CE027700C62567 /* LocalSampleBufferDisplayLayer.h in Headers */,
    3140031410                                BCE1C41B0D982980003B02F2 /* Location.h in Headers */,
    3140131411                                6B4D412D23983F88002494C2 /* LoggedInStatus.h in Headers */,
    … …  
    3218232192                                7C52229E1E1DAE47002CB8F7 /* RuntimeEnabledFeatures.h in Headers */,
    3218332193                                CE7A6C28208537E200FA2B46 /* SameSiteInfo.h in Headers */,
     32194                                41FCD6B923CE015500C62567 /* SampleBufferDisplayLayer.h in Headers */,
    3218432195                                CDD7089718359F6F002B3DC6 /* SampleMap.h in Headers */,
    3218532196                                49E911CB0EF86D47009D0CAF /* ScaleTransformOperation.h in Headers */,
    … …  
    3365133662                                4186BD3E213EE3400001826F /* LibWebRTCUtils.cpp in Sources */,
    3365233663                                9759E93E14EF1CF80026A2DD /* LoadableTextTrack.cpp in Sources */,
     33664                                414598C223C8D177002B9CC8 /* LocalSampleBufferDisplayLayer.mm in Sources */,
    3365333665                                FABE72FE1059C21100D999DD /* MathMLNames.cpp in Sources */,
    3365433666                                2D9BF7051DBFBB24007A7D99 /* MediaEncryptedEvent.cpp in Sources */,

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h

    r254465 r254564  
    3131#include "MediaSample.h"
    3232#include "MediaStreamPrivate.h"
     33#include "SampleBufferDisplayLayer.h"
    3334#include <wtf/Deque.h>
    3435#include <wtf/Forward.h>
    3536#include <wtf/LoggerHelper.h>
    36 #include <wtf/WeakPtr.h>
    3737
    3838OBJC_CLASS AVSampleBufferDisplayLayer;
    … …  
    5252class VideoTrackPrivateMediaStream;
    5353
    54 class MediaPlayerPrivateMediaStreamAVFObjC final : public CanMakeWeakPtr<MediaPlayerPrivateMediaStreamAVFObjC>, public MediaPlayerPrivateInterface, private MediaStreamPrivate::Observer, private MediaStreamTrackPrivate::Observer
     54class MediaPlayerPrivateMediaStreamAVFObjC final : public MediaPlayerPrivateInterface, private MediaStreamPrivate::Observer, private MediaStreamTrackPrivate::Observer, public SampleBufferDisplayLayer::Client
    5555#if !RELEASE_LOG_DISABLED
    5656    , private LoggerHelper
    … …  
    7575    void ensureLayers();
    7676    void destroyLayers();
    77 
    78     void layerStatusDidChange(AVSampleBufferDisplayLayer*);
    79     void layerErrorDidChange(AVSampleBufferDisplayLayer*);
    80     void backgroundLayerBoundsChanged();
    81 
    82     PlatformLayer* displayLayer();
    83     PlatformLayer* backgroundLayer();
    8477
    8578#if !RELEASE_LOG_DISABLED
    … …  
    137130
    138131    void flushRenderers();
    139 
    140     using PendingSampleQueue = Deque<Ref<MediaSample>>;
    141     void addSampleToPendingQueue(PendingSampleQueue&, MediaSample&);
    142     void removeOldSamplesFromPendingQueue(PendingSampleQueue&);
    143132
    144133    MediaTime calculateTimelineOffset(const MediaSample&, double);
    … …  
    213202    void setVideoFullscreenFrame(FloatRect) override;
    214203
    215     MediaTime streamTime() const;
     204    MediaTime streamTime() const final;
    216205
    217206    AudioSourceProvider* audioSourceProvider() final;
    … …  
    221210    void applicationDidBecomeActive() final;
    222211
    223     bool hideBackgroundLayer() const { return (!m_activeVideoTrack || m_waitingForFirstImage) && m_displayMode != PaintItBlack; }
     212    bool hideRootLayer() const { return (!m_activeVideoTrack || m_waitingForFirstImage) && m_displayMode != PaintItBlack; }
    224213
    225214    MediaPlayer* m_player { nullptr };
    226215    RefPtr<MediaStreamPrivate> m_mediaStreamPrivate;
    227216    RefPtr<MediaStreamTrackPrivate> m_activeVideoTrack;
    228     RetainPtr<WebAVSampleBufferStatusChangeListener> m_statusChangeListener;
    229     RetainPtr<AVSampleBufferDisplayLayer> m_sampleBufferDisplayLayer;
    230     RetainPtr<PlatformLayer> m_backgroundLayer;
    231217    std::unique_ptr<PAL::Clock> m_clock;
    232218
    … …  
    245231    HashMap<String, RefPtr<AudioTrackPrivateMediaStream>> m_audioTrackMap;
    246232    HashMap<String, RefPtr<VideoTrackPrivateMediaStream>> m_videoTrackMap;
    247     PendingSampleQueue m_pendingVideoSampleQueue;
    248233
    249234    MediaPlayer::NetworkState m_networkState { MediaPlayer::NetworkState::Empty };
    … …  
    255240    MediaSample::VideoRotation m_videoRotation { MediaSample::VideoRotation::None };
    256241    CGAffineTransform m_videoTransform;
     242    std::unique_ptr<SampleBufferDisplayLayer> m_sampleBufferDisplayLayer;
    257243    std::unique_ptr<VideoFullscreenLayerManagerObjC> m_videoFullscreenLayerManager;
     244
     245    // SampleBufferDisplayLayer::Client
     246    void sampleBufferDisplayLayerStatusDidChange(SampleBufferDisplayLayer&) final;
     247    void sampleBufferDisplayLayerBoundsDidChange(SampleBufferDisplayLayer&) final;
    258248
    259249#if !RELEASE_LOG_DISABLED

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm

    r254465 r254564  
    11/*
    2  * Copyright (C) 2015-2017 Apple Inc. All rights reserved.
     2 * Copyright (C) 2015-2020 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3232#import "GraphicsContextCG.h"
    3333#import "Logging.h"
     34#import "LocalSampleBufferDisplayLayer.h"
    3435#import "MediaStreamPrivate.h"
    3536#import "PixelBufferConformerCV.h"
    3637#import "VideoFullscreenLayerManagerObjC.h"
    3738#import "VideoTrackPrivateMediaStream.h"
    38 #import <AVFoundation/AVSampleBufferDisplayLayer.h>
    3939#import <CoreGraphics/CGAffineTransform.h>
    40 #import <QuartzCore/CALayer.h>
    41 #import <QuartzCore/CATransaction.h>
    4240#import <objc_runtime.h>
    4341#import <pal/avfoundation/MediaTimeAVFoundation.h>
    4442#import <pal/spi/mac/AVFoundationSPI.h>
    4543#import <pal/system/Clock.h>
    46 #import <wtf/Deque.h>
    47 #import <wtf/Function.h>
    4844#import <wtf/MainThread.h>
    4945#import <wtf/NeverDestroyed.h>
    … …  
    5349#import <pal/cocoa/AVFoundationSoftLink.h>
    5450
    55 using namespace WebCore;
    56 
    57 @interface WebAVSampleBufferStatusChangeListener : NSObject {
    58     MediaPlayerPrivateMediaStreamAVFObjC* _parent;
    59 }
    60 
    61 - (id)initWithParent:(MediaPlayerPrivateMediaStreamAVFObjC*)callback;
    62 - (void)invalidate;
    63 - (void)beginObservingLayers;
    64 - (void)stopObservingLayers;
    65 @end
    66 
    67 @implementation WebAVSampleBufferStatusChangeListener
    68 
    69 - (id)initWithParent:(MediaPlayerPrivateMediaStreamAVFObjC*)parent
    70 {
    71     if (!(self = [super init]))
    72         return nil;
    73 
    74     _parent = parent;
    75 
    76     return self;
    77 }
    78 
    79 - (void)dealloc
    80 {
    81     [self invalidate];
    82     [super dealloc];
    83 }
    84 
    85 - (void)invalidate
    86 {
    87     [self stopObservingLayers];
    88     _parent = nullptr;
    89 }
    90 
    91 - (void)beginObservingLayers
    92 {
    93     ASSERT(_parent);
    94     ASSERT(_parent->displayLayer());
    95     ASSERT(_parent->backgroundLayer());
    96 
    97     [_parent->displayLayer() addObserver:self forKeyPath:@"status" options:NSKeyValueObservingOptionNew context:nil];
    98     [_parent->displayLayer() addObserver:self forKeyPath:@"error" options:NSKeyValueObservingOptionNew context:nil];
    99     [_parent->backgroundLayer() addObserver:self forKeyPath:@"bounds" options:NSKeyValueObservingOptionNew context:nil];
    100 }
    101 
    102 - (void)stopObservingLayers
    103 {
    104     if (!_parent)
    105         return;
    106 
    107     if (_parent->displayLayer()) {
    108         [_parent->displayLayer() removeObserver:self forKeyPath:@"status"];
    109         [_parent->displayLayer() removeObserver:self forKeyPath:@"error"];
    110     }
    111     if (_parent->backgroundLayer())
    112         [_parent->backgroundLayer() removeObserver:self forKeyPath:@"bounds"];
    113 }
    114 
    115 - (void)observeValueForKeyPath:(NSString *)keyPath ofObject:(id)object change:(NSDictionary *)change context:(void *)context
    116 {
    117     UNUSED_PARAM(context);
    118     UNUSED_PARAM(keyPath);
    119     ASSERT(_parent);
    120 
    121     if (!_parent)
    122         return;
    123 
    124     if ([object isKindOfClass:PAL::getAVSampleBufferDisplayLayerClass()]) {
    125         RetainPtr<AVSampleBufferDisplayLayer> layer = (AVSampleBufferDisplayLayer *)object;
    126         ASSERT(layer.get() == _parent->displayLayer());
    127 
    128         if ([keyPath isEqualToString:@"status"]) {
    129             RetainPtr<NSNumber> status = [change valueForKey:NSKeyValueChangeNewKey];
    130             callOnMainThread([protectedSelf = RetainPtr<WebAVSampleBufferStatusChangeListener>(self), layer = WTFMove(layer), status = WTFMove(status)] {
    131                 if (!protectedSelf->_parent)
    132                     return;
    133 
    134                 protectedSelf->_parent->layerStatusDidChange(layer.get());
    135             });
    136             return;
    137         }
    138 
    139         if ([keyPath isEqualToString:@"error"]) {
    140             RetainPtr<NSNumber> status = [change valueForKey:NSKeyValueChangeNewKey];
    141             callOnMainThread([protectedSelf = RetainPtr<WebAVSampleBufferStatusChangeListener>(self), layer = WTFMove(layer), status = WTFMove(status)] {
    142                 if (!protectedSelf->_parent)
    143                     return;
    144 
    145                 protectedSelf->_parent->layerErrorDidChange(layer.get());
    146             });
    147             return;
    148         }
    149     }
    150 
    151     if ([[change valueForKey:NSKeyValueChangeNotificationIsPriorKey] boolValue])
    152         return;
    153 
    154     if ((CALayer *)object == _parent->backgroundLayer()) {
    155         if ([keyPath isEqualToString:@"bounds"]) {
    156             if (!_parent)
    157                 return;
    158 
    159             if (isMainThread()) {
    160                 _parent->backgroundLayerBoundsChanged();
    161                 return;
    162             }
    163 
    164             callOnMainThread([protectedSelf = RetainPtr<WebAVSampleBufferStatusChangeListener>(self)] {
    165                 if (!protectedSelf->_parent)
    166                     return;
    167 
    168                 protectedSelf->_parent->backgroundLayerBoundsChanged();
    169             });
    170         }
    171     }
    172 
    173 }
    174 @end
    175 
    17651namespace WebCore {
    17752using namespace PAL;
    … …  
    18459MediaPlayerPrivateMediaStreamAVFObjC::MediaPlayerPrivateMediaStreamAVFObjC(MediaPlayer* player)
    18560    : m_player(player)
    186     , m_statusChangeListener(adoptNS([[WebAVSampleBufferStatusChangeListener alloc] initWithParent:this]))
    18761    , m_clock(PAL::Clock::create())
    18862    , m_videoFullscreenLayerManager(makeUnique<VideoFullscreenLayerManagerObjC>())
    … …  
    19973    INFO_LOG(LOGIDENTIFIER);
    20074
    201     [m_statusChangeListener invalidate];
    202 
    20375    for (const auto& track : m_audioTrackMap.values())
    20476        track->pause();
    … …  
    269141#pragma mark -
    270142#pragma mark AVSampleBuffer Methods
    271 
    272 void MediaPlayerPrivateMediaStreamAVFObjC::removeOldSamplesFromPendingQueue(PendingSampleQueue& queue)
    273 {
    274     if (queue.isEmpty())
    275         return;
    276 
    277     auto decodeTime = queue.first()->decodeTime();
    278     if (!decodeTime.isValid() || decodeTime < MediaTime::zeroTime()) {
    279         while (queue.size() > 5)
    280             queue.removeFirst();
    281 
    282         return;
    283     }
    284 
    285     MediaTime now = streamTime();
    286     while (!queue.isEmpty()) {
    287         if (queue.first()->decodeTime() > now)
    288             break;
    289         queue.removeFirst();
    290     }
    291 }
    292 
    293 void MediaPlayerPrivateMediaStreamAVFObjC::addSampleToPendingQueue(PendingSampleQueue& queue, MediaSample& sample)
    294 {
    295     removeOldSamplesFromPendingQueue(queue);
    296     queue.append(sample);
    297 }
    298143
    299144MediaTime MediaPlayerPrivateMediaStreamAVFObjC::calculateTimelineOffset(const MediaSample& sample, double latency)
    … …  
    331176}
    332177
    333 static void runWithoutAnimations(const WTF::Function<void()>& function)
    334 {
    335     [CATransaction begin];
    336     [CATransaction setAnimationDuration:0];
    337     [CATransaction setDisableActions:YES];
    338     function();
    339     [CATransaction commit];
    340 }
    341 
    342178void MediaPlayerPrivateMediaStreamAVFObjC::enqueueCorrectedVideoSample(MediaSample& sample)
    343179{
    344180    if (m_sampleBufferDisplayLayer) {
    345         if ([m_sampleBufferDisplayLayer status] == AVQueuedSampleBufferRenderingStatusFailed)
     181        if (m_sampleBufferDisplayLayer->didFail())
    346182            return;
    347183
    … …  
    349185            m_videoRotation = sample.videoRotation();
    350186            m_videoMirrored = sample.videoMirrored();
    351             runWithoutAnimations([this, &sample] {
    352                 m_sampleBufferDisplayLayer.get().affineTransform = videoTransformationMatrix(sample, true);
    353                 updateDisplayLayer();
    354             });
     187            m_sampleBufferDisplayLayer->updateAffineTransform(videoTransformationMatrix(sample, true));
     188            updateDisplayLayer();
    355189        }
    356 
    357         if (![m_sampleBufferDisplayLayer isReadyForMoreMediaData]) {
    358             addSampleToPendingQueue(m_pendingVideoSampleQueue, sample);
    359             requestNotificationWhenReadyForVideoData();
    360             return;
    361         }
    362 
    363         [m_sampleBufferDisplayLayer enqueueSampleBuffer:sample.platformSample().sample.cmSampleBuffer];
     190        m_sampleBufferDisplayLayer->enqueueSample(sample);
    364191    }
    365192
    … …  
    416243}
    417244
    418 void MediaPlayerPrivateMediaStreamAVFObjC::requestNotificationWhenReadyForVideoData()
    419 {
    420     auto weakThis = makeWeakPtr(*this);
    421     [m_sampleBufferDisplayLayer requestMediaDataWhenReadyOnQueue:dispatch_get_main_queue() usingBlock:^ {
    422         if (!weakThis)
    423             return;
    424 
    425         [m_sampleBufferDisplayLayer stopRequestingMediaData];
    426 
    427         if (!m_activeVideoTrack) {
    428             m_pendingVideoSampleQueue.clear();
    429             return;
    430         }
    431 
    432         while (!m_pendingVideoSampleQueue.isEmpty()) {
    433             if (![m_sampleBufferDisplayLayer isReadyForMoreMediaData]) {
    434                 requestNotificationWhenReadyForVideoData();
    435                 return;
    436             }
    437 
    438             auto sample = m_pendingVideoSampleQueue.takeFirst();
    439             enqueueVideoSample(*m_activeVideoTrack.get(), sample.get());
    440         }
    441     }];
    442 }
    443 
    444245AudioSourceProvider* MediaPlayerPrivateMediaStreamAVFObjC::audioSourceProvider()
    445246{
    … …  
    448249}
    449250
    450 void MediaPlayerPrivateMediaStreamAVFObjC::layerErrorDidChange(AVSampleBufferDisplayLayer* layer)
    451 {
     251void MediaPlayerPrivateMediaStreamAVFObjC::sampleBufferDisplayLayerStatusDidChange(SampleBufferDisplayLayer& layer)
     252{
     253    ASSERT(&layer == m_sampleBufferDisplayLayer.get());
    452254    UNUSED_PARAM(layer);
    453     ERROR_LOG(LOGIDENTIFIER, "error = ", [[layer.error localizedDescription] UTF8String]);
    454 }
    455 
    456 void MediaPlayerPrivateMediaStreamAVFObjC::layerStatusDidChange(AVSampleBufferDisplayLayer* layer)
    457 {
    458     ALWAYS_LOG(LOGIDENTIFIER, "status = ", (int)layer.status);
    459 
    460     if (layer.status != AVQueuedSampleBufferRenderingStatusRendering)
    461         return;
    462     if (!m_sampleBufferDisplayLayer || !m_activeVideoTrack || layer != m_sampleBufferDisplayLayer)
    463         return;
    464 
    465     auto track = m_videoTrackMap.get(m_activeVideoTrack->id());
    466     if (track)
     255    if (!m_activeVideoTrack)
     256        return;
     257
     258    if (auto track = m_videoTrackMap.get(m_activeVideoTrack->id()))
    467259        track->setTimelineOffset(MediaTime::invalidTime());
    468260}
    … …  
    470262void MediaPlayerPrivateMediaStreamAVFObjC::applicationDidBecomeActive()
    471263{
    472     if (m_sampleBufferDisplayLayer && [m_sampleBufferDisplayLayer status] == AVQueuedSampleBufferRenderingStatusFailed) {
     264    if (m_sampleBufferDisplayLayer && m_sampleBufferDisplayLayer->didFail()) {
    473265        flushRenderers();
    474266        if (m_imagePainter.mediaSample)
    … …  
    481273{
    482274    if (m_sampleBufferDisplayLayer)
    483         [m_sampleBufferDisplayLayer flush];
     275        m_sampleBufferDisplayLayer->flush();
    484276}
    485277
    … …  
    492284        return;
    493285
    494     m_sampleBufferDisplayLayer = adoptNS([PAL::allocAVSampleBufferDisplayLayerInstance() init]);
     286    auto size = snappedIntRect(m_player->playerContentBoxRect()).size();
     287    m_sampleBufferDisplayLayer = LocalSampleBufferDisplayLayer::create(*this, hideRootLayer(), size);
     288
    495289    if (!m_sampleBufferDisplayLayer) {
    496         ERROR_LOG(LOGIDENTIFIER, "+[AVSampleBufferDisplayLayer alloc] failed.");
    497         return;
    498     }
    499 
    500     m_sampleBufferDisplayLayer.get().backgroundColor = cachedCGColor(Color::black);
    501     m_sampleBufferDisplayLayer.get().anchorPoint = { .5, .5 };
    502     m_sampleBufferDisplayLayer.get().needsDisplayOnBoundsChange = YES;
    503     m_sampleBufferDisplayLayer.get().videoGravity = AVLayerVideoGravityResizeAspectFill;
    504 
    505     m_backgroundLayer = adoptNS([[CALayer alloc] init]);
    506     m_backgroundLayer.get().hidden = hideBackgroundLayer();
    507 
    508     m_backgroundLayer.get().backgroundColor = cachedCGColor(Color::black);
    509     m_backgroundLayer.get().needsDisplayOnBoundsChange = YES;
    510 
    511     auto size = snappedIntRect(m_player->playerContentBoxRect()).size();
    512     m_backgroundLayer.get().bounds = CGRectMake(0, 0, size.width(), size.height());
    513 
    514     [m_statusChangeListener beginObservingLayers];
    515 
    516     [m_backgroundLayer addSublayer:m_sampleBufferDisplayLayer.get()];
    517 
    518 #ifndef NDEBUG
    519     [m_sampleBufferDisplayLayer setName:@"MediaPlayerPrivateMediaStreamAVFObjC AVSampleBufferDisplayLayer"];
    520     [m_backgroundLayer setName:@"MediaPlayerPrivateMediaStreamAVFObjC AVSampleBufferDisplayLayer parent"];
    521 #endif
     290        ERROR_LOG(LOGIDENTIFIER, "Creating the SampleBufferDisplayLayer failed.");
     291        return;
     292    }
    522293
    523294    updateRenderingMode();
    524295    updateDisplayLayer();
    525296
    526     m_videoFullscreenLayerManager->setVideoLayer(m_backgroundLayer.get(), size);
     297    m_videoFullscreenLayerManager->setVideoLayer(m_sampleBufferDisplayLayer->rootLayer(), size);
    527298}
    528299
    529300void MediaPlayerPrivateMediaStreamAVFObjC::destroyLayers()
    530301{
    531     [m_statusChangeListener stopObservingLayers];
    532     if (m_sampleBufferDisplayLayer) {
    533         m_pendingVideoSampleQueue.clear();
    534         [m_sampleBufferDisplayLayer stopRequestingMediaData];
    535         [m_sampleBufferDisplayLayer flush];
     302    if (m_sampleBufferDisplayLayer)
    536303        m_sampleBufferDisplayLayer = nullptr;
    537     }
    538     m_backgroundLayer = nullptr;
    539304
    540305    updateRenderingMode();
    … …  
    603368PlatformLayer* MediaPlayerPrivateMediaStreamAVFObjC::platformLayer() const
    604369{
    605     if (!m_backgroundLayer || m_displayMode == None)
     370    if (!m_sampleBufferDisplayLayer || !m_sampleBufferDisplayLayer->rootLayer() || m_displayMode == None)
    606371        return nullptr;
    607372
    608373    return m_videoFullscreenLayerManager->videoInlineLayer();
    609 }
    610 
    611 PlatformLayer* MediaPlayerPrivateMediaStreamAVFObjC::displayLayer()
    612 {
    613     return m_sampleBufferDisplayLayer.get();
    614 }
    615 
    616 PlatformLayer* MediaPlayerPrivateMediaStreamAVFObjC::backgroundLayer()
    617 {
    618     return m_backgroundLayer.get();
    619374}
    620375
    … …  
    654409    m_displayMode = displayMode;
    655410
    656     auto hidden = m_displayMode < PausedImage;
    657     if (m_sampleBufferDisplayLayer && m_sampleBufferDisplayLayer.get().hidden != hidden) {
    658         runWithoutAnimations([this, hidden] {
    659             m_sampleBufferDisplayLayer.get().hidden = hidden;
    660         });
    661     }
    662     hidden = hideBackgroundLayer();
    663     if (m_backgroundLayer && m_backgroundLayer.get().hidden != hidden) {
    664         runWithoutAnimations([this, hidden] {
    665             m_backgroundLayer.get().hidden = hidden;
    666         });
    667     }
     411    if (m_sampleBufferDisplayLayer)
     412        m_sampleBufferDisplayLayer->updateDisplayMode(m_displayMode < PausedImage, hideRootLayer());
    668413
    669414    return true;
    … …  
    1024769        }
    1025770        ensureLayers();
    1026         m_sampleBufferDisplayLayer.get().hidden = hideVideoLayer || m_displayMode < PausedImage;
    1027         m_backgroundLayer.get().hidden = hideBackgroundLayer();
     771        if (m_sampleBufferDisplayLayer) {
     772            if (!m_activeVideoTrack)
     773                m_sampleBufferDisplayLayer->clearEnqueuedSamples();
     774            m_sampleBufferDisplayLayer->updateDisplayMode(hideVideoLayer || m_displayMode < PausedImage, hideRootLayer());
     775        }
    1028776
    1029777        m_pendingSelectedTrackCheck = false;
    … …  
    1174922void MediaPlayerPrivateMediaStreamAVFObjC::setBufferingPolicy(MediaPlayer::BufferingPolicy policy)
    1175923{
    1176     if (policy != MediaPlayer::BufferingPolicy::Default)
    1177         [m_sampleBufferDisplayLayer flushAndRemoveImage];
     924    if (policy != MediaPlayer::BufferingPolicy::Default && m_sampleBufferDisplayLayer)
     925        m_sampleBufferDisplayLayer->flushAndRemoveImage();
    1178926}
    1179927
    … …  
    1198946void MediaPlayerPrivateMediaStreamAVFObjC::updateDisplayLayer()
    1199947{
    1200     if (!m_backgroundLayer || !m_sampleBufferDisplayLayer)
    1201         return;
    1202 
    1203     auto backgroundBounds = m_backgroundLayer.get().bounds;
    1204     auto videoBounds = backgroundBounds;
     948    if (!m_sampleBufferDisplayLayer)
     949        return;
     950
     951    auto bounds = m_sampleBufferDisplayLayer->bounds();
     952    auto videoBounds = bounds;
    1205953    if (m_videoRotation == MediaSample::VideoRotation::Right || m_videoRotation == MediaSample::VideoRotation::Left)
    1206954        std::swap(videoBounds.size.width, videoBounds.size.height);
    1207955
    1208     m_sampleBufferDisplayLayer.get().bounds = videoBounds;
    1209     m_sampleBufferDisplayLayer.get().position = { backgroundBounds.size.width / 2, backgroundBounds.size.height / 2};
    1210 }
    1211 
    1212 void MediaPlayerPrivateMediaStreamAVFObjC::backgroundLayerBoundsChanged()
    1213 {
    1214     runWithoutAnimations([this] {
    1215         updateDisplayLayer();
    1216     });
     956    m_sampleBufferDisplayLayer->updateBoundsAndPosition(videoBounds, { bounds.size.width / 2, bounds.size.height / 2});
     957}
     958
     959void MediaPlayerPrivateMediaStreamAVFObjC::sampleBufferDisplayLayerBoundsDidChange(SampleBufferDisplayLayer&)
     960{
     961    updateDisplayLayer();
    1217962}
    1218963
Note: See TracChangeset for help on using the changeset viewer.