Changeset 206025 in webkit


Ignore:
Timestamp:
Sep 16, 2016 9:21:58 AM (8 years ago)
Author:
jer.noble@apple.com
Message:

[media-source] Only fire 'resize' events when new sized frame is displayed, not parsed.
​https://bugs.webkit.org/show_bug.cgi?id=162039

Reviewed by Eric Carlson.

Fixes tests: imported/w3c/web-platform-tests/media-source/mediasource-config-change-mp4-av-framesize.html

imported/w3c/web-platform-tests/media-source/mediasource-config-change-mp4-v-framesize.html

Currently, the SourceBufferPrivateAVFObjC will signal a size change as soon as the sample is
parsed during appendData(). This is incorrect, as the intrinsic size of the video element
should be based on when the sample is displayed, and it causes some W3C test cases to fail.
Set a boundary time observer on the player's synchronizer at each sample's presentation time
where that sample signals a change in intrinsic size. Flush those observers whenever the
samples are flushed un-displayed (typically, during a seek). Because the observer callback
may have already been issued (but not executed) at a flush, use a separate weak pointer
factory, and invalidate all outstanding size change observer weak pointers when flushing.

  • Modules/mediasource/SourceBuffer.cpp:

(WebCore::SourceBuffer::reenqueueMediaForTime): When re-enqueing, enqueue the next decodable

sample, even if it doesn't include the current time. This handles cases where the current
time is 0, and the first video sample starts at 0.033.

  • platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h:
  • platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm:

(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::MediaPlayerPrivateMediaSourceAVFObjC):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::~MediaPlayerPrivateMediaSourceAVFObjC):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::naturalSize):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::sizeWillChangeAtTime):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::flushPendingSizeChanges):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::sizeChanged): Deleted.

  • platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.h:
  • platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.mm:

(WebCore::SourceBufferPrivateAVFObjC::processCodedFrame):
(WebCore::SourceBufferPrivateAVFObjC::flushAndEnqueueNonDisplayingSamples):
(WebCore::SourceBufferPrivateAVFObjC::enqueueSample):

Location:
trunk/Source/WebCore
Files:
6 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebCore/ChangeLog

    r206024 r206025  
     12016-09-16  Jer Noble  <jer.noble@apple.com>
     2
     3        [media-source] Only fire 'resize' events when new sized frame is displayed, not parsed.
     4        https://bugs.webkit.org/show_bug.cgi?id=162039
     5
     6        Reviewed by Eric Carlson.
     7
     8        Fixes tests: imported/w3c/web-platform-tests/media-source/mediasource-config-change-mp4-av-framesize.html
     9                     imported/w3c/web-platform-tests/media-source/mediasource-config-change-mp4-v-framesize.html
     10
     11        Currently, the SourceBufferPrivateAVFObjC will signal a size change as soon as the sample is
     12        parsed during appendData(). This is incorrect, as the intrinsic size of the video element
     13        should be based on when the sample is displayed, and it causes some W3C test cases to fail.
     14        Set a boundary time observer on the player's synchronizer at each sample's presentation time
     15        where that sample signals a change in intrinsic size. Flush those observers whenever the
     16        samples are flushed un-displayed (typically, during a seek). Because the observer callback
     17        may have already been issued (but not executed) at a flush, use a separate weak pointer
     18        factory, and invalidate all outstanding size change observer weak pointers when flushing.
     19
     20        * Modules/mediasource/SourceBuffer.cpp:
     21        (WebCore::SourceBuffer::reenqueueMediaForTime): When re-enqueing, enqueue the next decodable
     22            sample, even if it doesn't include the current time. This handles cases where the current
     23            time is 0, and the first video sample starts at 0.033.
     24        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h:
     25        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm:
     26        (WebCore::MediaPlayerPrivateMediaSourceAVFObjC::MediaPlayerPrivateMediaSourceAVFObjC):
     27        (WebCore::MediaPlayerPrivateMediaSourceAVFObjC::~MediaPlayerPrivateMediaSourceAVFObjC):
     28        (WebCore::MediaPlayerPrivateMediaSourceAVFObjC::naturalSize):
     29        (WebCore::MediaPlayerPrivateMediaSourceAVFObjC::sizeWillChangeAtTime):
     30        (WebCore::MediaPlayerPrivateMediaSourceAVFObjC::flushPendingSizeChanges):
     31        (WebCore::MediaPlayerPrivateMediaSourceAVFObjC::sizeChanged): Deleted.
     32        * platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.h:
     33        * platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.mm:
     34        (WebCore::SourceBufferPrivateAVFObjC::processCodedFrame):
     35        (WebCore::SourceBufferPrivateAVFObjC::flushAndEnqueueNonDisplayingSamples):
     36        (WebCore::SourceBufferPrivateAVFObjC::enqueueSample):
     37
    1382016-09-16  Alex Christensen  <achristensen@webkit.org>
    239

  • trunk/Source/WebCore/Modules/mediasource/SourceBuffer.cpp

    r206023 r206025  
    18551855{
    18561856    // Find the sample which contains the current presentation time.
    1857     auto currentSamplePTSIterator = trackBuffer.samples.presentationOrder().findSampleContainingPresentationTime(time);
     1857    auto currentSamplePTSIterator = trackBuffer.samples.presentationOrder().findSampleOnOrAfterPresentationTime(time);
    18581858
    18591859    if (currentSamplePTSIterator == trackBuffer.samples.presentationOrder().end()) {

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h

    r205938 r206025  
    8484
    8585    void effectiveRateChanged();
    86     void sizeChanged();
     86    void sizeWillChangeAtTime(const MediaTime&, const FloatSize&);
     87    void flushPendingSizeChanges();
    8788    void characteristicsChanged();
    8889
    … …  
    211212    MediaPlayer* m_player;
    212213    WeakPtrFactory<MediaPlayerPrivateMediaSourceAVFObjC> m_weakPtrFactory;
     214    WeakPtrFactory<MediaPlayerPrivateMediaSourceAVFObjC> m_sizeChangeObserverWeakPtrFactory;
    213215    RefPtr<MediaSourcePrivateAVFObjC> m_mediaSourcePrivate;
    214216    RetainPtr<AVAsset> m_asset;
    … …  
    219221    RetainPtr<id> m_durationObserver;
    220222    RetainPtr<AVStreamSession> m_streamSession;
     223    Deque<RetainPtr<id>> m_sizeChangeObservers;
    221224    Timer m_seekTimer;
    222225    CDMSessionMediaSourceAVFObjC* m_session;
    … …  
    224227    MediaPlayer::ReadyState m_readyState;
    225228    MediaTime m_lastSeekTime;
     229    FloatSize m_naturalSize;
    226230    double m_rate;
    227231    bool m_playing;

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm

    r205938 r206025  
    4444#import <QuartzCore/CALayer.h>
    4545#import <objc_runtime.h>
     46#import <wtf/Deque.h>
    4647#import <wtf/MainThread.h>
    4748#import <wtf/NeverDestroyed.h>
    … …  
    142143    : m_player(player)
    143144    , m_weakPtrFactory(this)
     145    , m_sizeChangeObserverWeakPtrFactory(this)
    144146    , m_synchronizer(adoptNS([allocAVSampleBufferRenderSynchronizerInstance() init]))
    145147    , m_seekTimer(*this, &MediaPlayerPrivateMediaSourceAVFObjC::seekInternal)
    … …  
    196198    if (m_durationObserver)
    197199        [m_synchronizer removeTimeObserver:m_durationObserver.get()];
     200    flushPendingSizeChanges();
    198201
    199202    m_seekTimer.stop();
    … …  
    361364FloatSize MediaPlayerPrivateMediaSourceAVFObjC::naturalSize() const
    362365{
    363     if (!m_mediaSourcePrivate)
    364         return FloatSize();
    365 
    366     return m_mediaSourcePrivate->naturalSize();
     366    return m_naturalSize;
    367367}
    368368
    … …  
    683683}
    684684
    685 void MediaPlayerPrivateMediaSourceAVFObjC::sizeChanged()
    686 {
    687     m_player->sizeChanged();
     685void MediaPlayerPrivateMediaSourceAVFObjC::sizeWillChangeAtTime(const MediaTime& time, const FloatSize& size)
     686{
     687    auto weakThis = m_sizeChangeObserverWeakPtrFactory.createWeakPtr();
     688    NSArray* times = @[[NSValue valueWithCMTime:toCMTime(time)]];
     689    RetainPtr<id> observer = [m_synchronizer addBoundaryTimeObserverForTimes:times queue:dispatch_get_main_queue() usingBlock:[weakThis, size] {
     690        if (!weakThis)
     691            return;
     692        RetainPtr<id> observer = weakThis->m_sizeChangeObservers.takeFirst();
     693        weakThis->m_naturalSize = size;
     694        weakThis->m_player->sizeChanged();
     695    }];
     696    m_sizeChangeObservers.append(WTFMove(observer));
     697}
     698
     699void MediaPlayerPrivateMediaSourceAVFObjC::flushPendingSizeChanges()
     700{
     701    while (!m_sizeChangeObservers.isEmpty()) {
     702        RetainPtr<id> observer = m_sizeChangeObservers.takeFirst();
     703        [m_synchronizer removeTimeObserver:observer.get()];
     704    }
     705    m_sizeChangeObserverWeakPtrFactory.revokeAll();
    688706}
    689707

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.h

    r203362 r206025  
    155155
    156156    FloatSize m_cachedSize;
     157    FloatSize m_currentSize;
    157158    bool m_parsingSucceeded;
    158159    int m_enabledVideoTrackID;

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.mm

    r204239 r206025  
    552552bool SourceBufferPrivateAVFObjC::processCodedFrame(int trackID, CMSampleBufferRef sampleBuffer, const String&)
    553553{
    554     if (trackID == m_enabledVideoTrackID) {
    555         CMFormatDescriptionRef formatDescription = CMSampleBufferGetFormatDescription(sampleBuffer);
    556         FloatSize formatSize = FloatSize(CMVideoFormatDescriptionGetPresentationDimensions(formatDescription, true, true));
    557         if (formatSize != m_cachedSize) {
    558             LOG(MediaSource, "SourceBufferPrivateAVFObjC::processCodedFrame(%p) - size change detected: {width=%lf, height=%lf}", formatSize.width(), formatSize.height());
    559             m_cachedSize = formatSize;
    560             if (m_mediaSource)
    561                 m_mediaSource->player()->sizeChanged();
    562         }
    563     } else if (!m_audioRenderers.contains(trackID)) {
     554    if (trackID != m_enabledVideoTrackID && !m_audioRenderers.contains(trackID)) {
    564555        // FIXME(125161): We don't handle text tracks, and passing this sample up to SourceBuffer
    565556        // will just confuse its state. Drop this sample until we can handle text tracks properly.
    566557        return false;
    567558    }
    568 
    569559
    570560    if (m_client) {
    … …  
    958948    }
    959949
    960     if (m_mediaSource)
     950    if (m_mediaSource) {
    961951        m_mediaSource->player()->setHasAvailableVideoFrame(false);
     952        m_mediaSource->player()->flushPendingSizeChanges();
     953    }
    962954}
    963955
    … …  
    977969
    978970    if (trackID == m_enabledVideoTrackID) {
     971        CMFormatDescriptionRef formatDescription = CMSampleBufferGetFormatDescription(platformSample.sample.cmSampleBuffer);
     972        FloatSize formatSize = FloatSize(CMVideoFormatDescriptionGetPresentationDimensions(formatDescription, true, true));
     973        if (formatSize != m_cachedSize) {
     974            LOG(MediaSource, "SourceBufferPrivateAVFObjC::enqueueSample(%p) - size change detected: {width=%lf, height=%lf}", formatSize.width(), formatSize.height());
     975            m_cachedSize = formatSize;
     976            if (m_mediaSource)
     977                m_mediaSource->player()->sizeWillChangeAtTime(mediaSample->presentationTime(), formatSize);
     978        }
     979
    979980        [m_displayLayer enqueueSampleBuffer:platformSample.sample.cmSampleBuffer];
    980981        if (m_mediaSource)
Note: See TracChangeset for help on using the changeset viewer.