Changeset 260245 in webkit


Ignore:
Timestamp:
Apr 17, 2020 7:22:23 AM (4 years ago)
Author:
youenn@apple.com
Message:

Safari doesn't apply frameRate limit when request stream from Camera
​https://bugs.webkit.org/show_bug.cgi?id=210186
<rdar://problem/61452794>

Reviewed by Eric Carlson.

Source/WebCore:

Add support to RealtimeVideoSource to decimate the video samples based on the observed frame rate of its capture source.
This allows supporting two tracks using the same capture device, one track being low frame rate and the other one high frame rate.

Clean-up refactoring to make RealtimeVideoSource directly inherit from RealtimeVideoCaptureSource.
Migrate size and format of frame adaptation from RealtimeVideoCaptureSource to RealtimeVideoSource.
Fix mock capture source to update its frame rate when asked by applyConstraints.

Tests: fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html

fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html
fast/mediastream/mediastreamtrack-video-frameRate-decreasing.html
fast/mediastream/mediastreamtrack-video-frameRate-increasing.html

  • platform/mediastream/RealtimeVideoCaptureSource.cpp:

(WebCore::RealtimeVideoCaptureSource::dispatchMediaSampleToObservers):
(WebCore::RealtimeVideoCaptureSource::clientUpdatedSizeAndFrameRate):

  • platform/mediastream/RealtimeVideoCaptureSource.h:

(WebCore::RealtimeVideoCaptureSource::observedFrameRate const):

  • platform/mediastream/RealtimeVideoSource.cpp:

(WebCore::RealtimeVideoSource::RealtimeVideoSource):
(WebCore::m_source):
(WebCore::RealtimeVideoSource::adaptVideoSample):
(WebCore::RealtimeVideoSource::videoSampleAvailable):

  • platform/mediastream/RealtimeVideoSource.h:
  • platform/mock/MockRealtimeVideoSource.cpp:

(WebCore::MockRealtimeVideoSource::setFrameRateWithPreset):

  • testing/Internals.cpp:

(WebCore::Internals::observeMediaStreamTrack):

LayoutTests:

  • fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing-expected.txt: Added.
  • fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html: Added.
  • fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing-expected.txt: Added.
  • fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html: Added.
  • fast/mediastream/mediastreamtrack-video-framerate-decreasing-expected.txt: added.
  • fast/mediastream/mediastreamtrack-video-framerate-decreasing.html: added.
  • fast/mediastream/mediastreamtrack-video-framerate-increasing-expected.txt: added.
  • fast/mediastream/mediastreamtrack-video-framerate-increasing.html: added.
  • webrtc/routines.js:
Location:
trunk
Files:
8 added
9 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r260243 r260245  
     12020-04-17  Youenn Fablet  <youenn@apple.com>
     2
     3        Safari doesn't apply frameRate limit when request stream from Camera
     4        https://bugs.webkit.org/show_bug.cgi?id=210186
     5        <rdar://problem/61452794>
     6
     7        Reviewed by Eric Carlson.
     8
     9        * fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing-expected.txt: Added.
     10        * fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html: Added.
     11        * fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing-expected.txt: Added.
     12        * fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html: Added.
     13        * fast/mediastream/mediastreamtrack-video-framerate-decreasing-expected.txt: added.
     14        * fast/mediastream/mediastreamtrack-video-framerate-decreasing.html: added.
     15        * fast/mediastream/mediastreamtrack-video-framerate-increasing-expected.txt: added.
     16        * fast/mediastream/mediastreamtrack-video-framerate-increasing.html: added.
     17        * webrtc/routines.js:
     18
    1192020-04-17  Alexey Shvayka  <shvaikalesh@gmail.com>
    220

  • trunk/LayoutTests/webrtc/routines.js

    r252681 r260245  
    226226    return stats;
    227227}
     228
     229function getReceivedTrackStats(connection)
     230{
     231    return connection.getStats().then((report) => {
     232        var stats;
     233        report.forEach((statItem) => {
     234            if (statItem.type === "track") {
     235                stats = statItem;
     236            }
     237        });
     238        return stats;
     239    });
     240}
     241
     242async function computeFrameRate(stream, video)
     243{
     244    if (window.internals) {
     245        internals.observeMediaStreamTrack(stream.getVideoTracks()[0]);
     246        await new Promise(resolve => setTimeout(resolve, 1000));
     247        return internals.trackVideoSampleCount;
     248    }
     249
     250    let connection;
     251    video.srcObject = await new Promise((resolve, reject) => {
     252        createConnections((firstConnection) => {
     253            firstConnection.addTrack(stream.getVideoTracks()[0], stream);
     254        }, (secondConnection) => {
     255            connection = secondConnection;
     256            secondConnection.ontrack = (trackEvent) => {
     257                resolve(trackEvent.streams[0]);
     258            };
     259        });
     260        setTimeout(() => reject("Test timed out"), 5000);
     261    });
     262
     263    await video.play();
     264
     265    const stats1 = await getReceivedTrackStats(connection);
     266    await new Promise(resolve => setTimeout(resolve, 1000));
     267    const stats2 = await getReceivedTrackStats(connection);
     268    return (stats2.framesReceived - stats1.framesReceived) * 1000 / (stats2.timestamp - stats1.timestamp);
     269}

  • trunk/Source/WebCore/ChangeLog

    r260243 r260245  
     12020-04-17  Youenn Fablet  <youenn@apple.com>
     2
     3        Safari doesn't apply frameRate limit when request stream from Camera
     4        https://bugs.webkit.org/show_bug.cgi?id=210186
     5        <rdar://problem/61452794>
     6
     7        Reviewed by Eric Carlson.
     8
     9        Add support to RealtimeVideoSource to decimate the video samples based on the observed frame rate of its capture source.
     10        This allows supporting two tracks using the same capture device, one track being low frame rate and the other one high frame rate.
     11
     12        Clean-up refactoring to make RealtimeVideoSource directly inherit from RealtimeVideoCaptureSource.
     13        Migrate size and format of frame adaptation from RealtimeVideoCaptureSource to RealtimeVideoSource.
     14        Fix mock capture source to update its frame rate when asked by applyConstraints.
     15
     16        Tests: fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html
     17               fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html
     18               fast/mediastream/mediastreamtrack-video-frameRate-decreasing.html
     19               fast/mediastream/mediastreamtrack-video-frameRate-increasing.html
     20
     21        * platform/mediastream/RealtimeVideoCaptureSource.cpp:
     22        (WebCore::RealtimeVideoCaptureSource::dispatchMediaSampleToObservers):
     23        (WebCore::RealtimeVideoCaptureSource::clientUpdatedSizeAndFrameRate):
     24        * platform/mediastream/RealtimeVideoCaptureSource.h:
     25        (WebCore::RealtimeVideoCaptureSource::observedFrameRate const):
     26        * platform/mediastream/RealtimeVideoSource.cpp:
     27        (WebCore::RealtimeVideoSource::RealtimeVideoSource):
     28        (WebCore::m_source):
     29        (WebCore::RealtimeVideoSource::adaptVideoSample):
     30        (WebCore::RealtimeVideoSource::videoSampleAvailable):
     31        * platform/mediastream/RealtimeVideoSource.h:
     32        * platform/mock/MockRealtimeVideoSource.cpp:
     33        (WebCore::MockRealtimeVideoSource::setFrameRateWithPreset):
     34        * testing/Internals.cpp:
     35        (WebCore::Internals::observeMediaStreamTrack):
     36
    1372020-04-17  Alexey Shvayka  <shvaikalesh@gmail.com>
    238

  • trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.cpp

    r258215 r260245  
    3434#include "RemoteVideoSample.h"
    3535#include <wtf/JSONValues.h>
    36 
    37 #if PLATFORM(COCOA)
    38 #include "ImageTransferSessionVT.h"
    39 #endif
    4036
    4137namespace WebCore {
    … …  
    377373}
    378374
    379 RefPtr<MediaSample> RealtimeVideoCaptureSource::adaptVideoSample(MediaSample& sample)
     375void RealtimeVideoCaptureSource::dispatchMediaSampleToObservers(MediaSample& sample)
    380376{
    381377    MediaTime sampleTime = sample.presentationTime();
    … …  
    391387        m_observedFrameRate = (m_observedFrameTimeStamps.size() / interval);
    392388
    393     auto mediaSample = makeRefPtr(&sample);
    394 
    395 #if PLATFORM(COCOA)
    396     auto size = this->size();
    397     if (!size.isEmpty() && size != expandedIntSize(sample.presentationSize())) {
    398 
    399         if (!m_imageTransferSession || m_imageTransferSession->pixelFormat() != sample.videoPixelFormat())
    400             m_imageTransferSession = ImageTransferSessionVT::create(sample.videoPixelFormat());
    401 
    402         ASSERT(m_imageTransferSession);
    403         if (m_imageTransferSession) {
    404             mediaSample = m_imageTransferSession->convertMediaSample(sample, size);
    405             if (!mediaSample) {
    406                 ASSERT_NOT_REACHED();
    407                 return nullptr;
    408             }
    409         }
    410     }
    411 #endif
    412 
    413     return mediaSample.releaseNonNull();
    414 }
    415 
    416 void RealtimeVideoCaptureSource::dispatchMediaSampleToObservers(MediaSample& sample)
    417 {
    418     if (auto mediaSample = adaptVideoSample(sample))
    419         videoSampleAvailable(*mediaSample);
     389    videoSampleAvailable(sample);
    420390}
    421391
    … …  
    428398    if (height && *height < static_cast<int>(settings.height()))
    429399        height = { };
    430 
    431     // FIXME: handle frameRate potential increase.
    432     if (!width && !height)
     400    if (frameRate && *frameRate < static_cast<double>(settings.frameRate()))
     401        frameRate = { };
     402
     403    if (!width && !height && !frameRate)
    433404        return;
    434405

  • trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.h

    r246644 r260245  
    4949    virtual MediaSample::VideoRotation sampleRotation() const { return MediaSample::VideoRotation::None; }
    5050
     51    double observedFrameRate() const { return m_observedFrameRate; }
     52
    5153protected:
    5254    RealtimeVideoCaptureSource(String&& name, String&& id, String&& hashSalt);
    … …  
    7072    void setDefaultSize(const IntSize& size) { m_defaultSize = size; }
    7173
    72     double observedFrameRate() const { return m_observedFrameRate; }
    73 
    7474    void dispatchMediaSampleToObservers(MediaSample&);
    7575    const Vector<IntSize>& standardVideoSizes();
    76     RefPtr<MediaSample> adaptVideoSample(MediaSample&);
    7776
    7877private:
    … …  
    9493    double m_observedFrameRate { 0 };
    9594    IntSize m_defaultSize;
    96 #if PLATFORM(COCOA)
    97     std::unique_ptr<ImageTransferSessionVT> m_imageTransferSession;
    98 #endif
    9995};
    10096

  • trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.cpp

    r257919 r260245  
    2929#if ENABLE(MEDIA_STREAM)
    3030
     31#if PLATFORM(COCOA)
     32#include "ImageTransferSessionVT.h"
     33#endif
     34
    3135namespace WebCore {
    3236
    3337RealtimeVideoSource::RealtimeVideoSource(Ref<RealtimeVideoCaptureSource>&& source)
    34     : RealtimeVideoCaptureSource(String { source->name() }, String { source->persistentID() }, String { source->deviceIDHashSalt() })
     38    : RealtimeMediaSource(Type::Video, String { source->name() }, String { source->persistentID() }, String { source->deviceIDHashSalt() })
    3539    , m_source(WTFMove(source))
    3640{
    … …  
    3842    m_currentSettings = m_source->settings();
    3943    setSize(m_source->size());
     44    setFrameRate(m_source->frameRate());
    4045}
    4146
    … …  
    143148}
    144149
     150#if PLATFORM(COCOA)
     151RefPtr<MediaSample> RealtimeVideoSource::adaptVideoSample(MediaSample& sample)
     152{
     153    if (!m_imageTransferSession || m_imageTransferSession->pixelFormat() != sample.videoPixelFormat())
     154        m_imageTransferSession = ImageTransferSessionVT::create(sample.videoPixelFormat());
     155
     156    ASSERT(m_imageTransferSession);
     157    if (!m_imageTransferSession)
     158        return nullptr;
     159
     160    auto mediaSample = m_imageTransferSession->convertMediaSample(sample, size());
     161    ASSERT(mediaSample);
     162
     163    return mediaSample;
     164}
     165#endif
     166
    145167void RealtimeVideoSource::videoSampleAvailable(MediaSample& sample)
    146168{
    … …  
    148170        return;
    149171
    150     if (auto mediaSample = adaptVideoSample(sample))
    151         RealtimeMediaSource::videoSampleAvailable(*mediaSample);
     172    if (m_frameDecimation > 1 && ++m_frameDecimationCounter % m_frameDecimation)
     173        return;
     174
     175    m_frameDecimation = static_cast<size_t>(m_source->observedFrameRate() / frameRate());
     176    if (!m_frameDecimation)
     177        m_frameDecimation = 1;
     178
     179#if PLATFORM(COCOA)
     180    auto size = this->size();
     181    if (!size.isEmpty() && size != expandedIntSize(sample.presentationSize())) {
     182        if (auto mediaSample = adaptVideoSample(sample)) {
     183            RealtimeMediaSource::videoSampleAvailable(*mediaSample);
     184            return;
     185        }
     186    }
     187#endif
     188
     189    RealtimeMediaSource::videoSampleAvailable(sample);
    152190}
    153191

  • trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.h

    r258698 r260245  
    3232namespace WebCore {
    3333
    34 // FIXME: Make RealtimeVideoSource derive from RealtimeMediaSource directly.
    35 class RealtimeVideoSource final : public RealtimeVideoCaptureSource, public RealtimeMediaSource::Observer {
     34class ImageTransferSessionVT;
     35
     36class RealtimeVideoSource final : public RealtimeMediaSource, public RealtimeMediaSource::Observer {
    3637public:
    3738    static Ref<RealtimeVideoSource> create(Ref<RealtimeVideoCaptureSource>&& source) { return adoptRef(*new RealtimeVideoSource(WTFMove(source))); }
    … …  
    5253    const RealtimeMediaSourceCapabilities& capabilities() final { return m_source->capabilities(); }
    5354    const RealtimeMediaSourceSettings& settings() final { return m_currentSettings; }
    54     void generatePresets() final { m_source->generatePresets(); }
    5555    bool isCaptureSource() const final { return m_source->isCaptureSource(); }
    5656    CaptureDevice::DeviceType deviceType() const final { return m_source->deviceType(); }
    … …  
    6565    void videoSampleAvailable(MediaSample&) final;
    6666
     67#if PLATFORM(COCOA)
     68    RefPtr<MediaSample> adaptVideoSample(MediaSample&);
     69#endif
     70
    6771#if !RELEASE_LOG_DISABLED
    6872    void setLogger(const Logger&, const void*) final;
    … …  
    7175    Ref<RealtimeVideoCaptureSource> m_source;
    7276    RealtimeMediaSourceSettings m_currentSettings;
     77#if PLATFORM(COCOA)
     78    std::unique_ptr<ImageTransferSessionVT> m_imageTransferSession;
     79#endif
     80    size_t m_frameDecimation { 1 };
     81    size_t m_frameDecimationCounter { 0 };
    7382#if !RELEASE_LOG_DISABLED
    7483    uint64_t m_cloneCounter { 0 };

  • trunk/Source/WebCore/platform/mock/MockRealtimeVideoSource.cpp

    r259845 r260245  
    188188}
    189189
    190 void MockRealtimeVideoSource::setFrameRateWithPreset(double, RefPtr<VideoPreset> preset)
     190void MockRealtimeVideoSource::setFrameRateWithPreset(double frameRate, RefPtr<VideoPreset> preset)
    191191{
    192192    m_preset = WTFMove(preset);
    193193    if (m_preset)
    194194        setIntrinsicSize(m_preset->size);
     195    if (isProducingData())
     196        m_emitFrameTimer.startRepeating(1_s / frameRate);
    195197}
    196198

  • trunk/Source/WebCore/testing/Internals.cpp

    r260190 r260245  
    50035003void Internals::observeMediaStreamTrack(MediaStreamTrack& track)
    50045004{
     5005    if (m_trackSource) {
     5006        m_trackSource->removeObserver(*this);
     5007        m_trackSource->removeAudioSampleObserver(*this);
     5008
     5009        m_trackAudioSampleCount = 0;
     5010        m_trackVideoSampleCount = 0;
     5011    }
     5012
    50055013    m_trackSource = &track.source();
    50065014    m_trackSource->addObserver(*this);
Note: See TracChangeset for help on using the changeset viewer.