⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 253275 in webkit


Ignore:
Timestamp:
Dec 8, 2019, 11:10:02 AM (7 years ago)
Author:
youenn@apple.com
Message:

Support different resolutions for video tracks captured from UIProcess
​https://bugs.webkit.org/show_bug.cgi?id=204945

Reviewed by Eric Carlson.

Source/WebCore:

Remove whether the a source is remote or not.
Previously we were doing the distinction as resizing would happen in WebProcess and not in UIProcess.
We are now moving away from doing resizing in WebProcess.
Covered by existing tests.

  • platform/mediastream/RealtimeMediaSource.h:
  • platform/mediastream/RealtimeMediaSourceFactory.h:
  • platform/mediastream/RealtimeVideoCaptureSource.cpp:

(WebCore::RealtimeVideoCaptureSource::adaptVideoSample):
For remote tracks, we also update the size for observers.

  • platform/mediastream/RealtimeVideoSource.cpp:

(WebCore::m_source):
(WebCore::RealtimeVideoSource::sourceSettingsChanged):
Update computation of size based on rotation.
This makes sure we have resizing done right.

  • platform/mediastream/RealtimeVideoSource.h:
  • platform/mediastream/mac/DisplayCaptureSourceCocoa.cpp:

(WebCore::DisplayCaptureSourceCocoa::emitFrame):

Source/WebKit:

Change video capture in uiprocess to an experimental flag.
Add support for remote video source cloning by sending some IPC to UIProcess and cloning the source there.
This triggers IPC for both cloning and cloned sources but this allows to do resizing in UIProcess instead of WebProcess.
We thus disable video capture resizing in WebProcess.

  • Shared/WebPreferences.yaml:
  • UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp:

(WebKit::UserMediaCaptureManagerProxy::SourceProxy::start):
(WebKit::UserMediaCaptureManagerProxy::SourceProxy::stop):
(WebKit::UserMediaCaptureManagerProxy::SourceProxy::requestToEnd):
(WebKit::UserMediaCaptureManagerProxy::SourceProxy::preventSourceFromStopping):
(WebKit::UserMediaCaptureManagerProxy::createMediaSourceForCaptureDeviceWithConstraints):
(WebKit::UserMediaCaptureManagerProxy::startProducingData):
(WebKit::UserMediaCaptureManagerProxy::stopProducingData):
(WebKit::UserMediaCaptureManagerProxy::clone):
(WebKit::UserMediaCaptureManagerProxy::requestToEnd):

  • UIProcess/Cocoa/UserMediaCaptureManagerProxy.h:
  • UIProcess/Cocoa/UserMediaCaptureManagerProxy.messages.in:
  • WebProcess/cocoa/UserMediaCaptureManager.cpp:

(WebKit::UserMediaCaptureManager::Source::sourceID const):
(WebKit::UserMediaCaptureManager::Source::settings const):
(WebKit::UserMediaCaptureManager::Source::remoteVideoSampleAvailable):
(WebKit::UserMediaCaptureManager::Source::requestToEnd):
(WebKit::UserMediaCaptureManager::Source::stopBeingObserved):
(WebKit::UserMediaCaptureManager::cloneSource):
(WebKit::UserMediaCaptureManager::cloneVideoSource):
(WebKit::UserMediaCaptureManager::requestToEnd):
(WebKit::UserMediaCaptureManager::VideoFactory::setVideoCapturePageState):

  • WebProcess/cocoa/UserMediaCaptureManager.h:

LayoutTests:

  • fast/mediastream/mediastreamtrack-video-clone-expected.txt:
  • fast/mediastream/mediastreamtrack-video-clone.html:
Location:
trunk
Files:
17 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r253267 r253275  
     12019-12-08  youenn fablet  <youenn@apple.com>
     2
     3        Support different resolutions for video tracks captured from UIProcess
     4        https://bugs.webkit.org/show_bug.cgi?id=204945
     5
     6        Reviewed by Eric Carlson.
     7
     8        * fast/mediastream/mediastreamtrack-video-clone-expected.txt:
     9        * fast/mediastream/mediastreamtrack-video-clone.html:
     10
    1112019-12-07  Wenson Hsieh  <wenson_hsieh@apple.com>
    212

  • trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-clone-expected.txt

    r247391 r253275  
    1    
     1    
    22
    33PASS Setup for width test

  • trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-clone.html

    r253033 r253275  
    1 <!-- webkit-test-runner [ enableCaptureVideoInUIProcess=false ] -->
    21<!DOCTYPE html>
    32<html>

  • trunk/Source/WebCore/ChangeLog

    r253270 r253275  
     12019-12-08  youenn fablet  <youenn@apple.com>
     2
     3        Support different resolutions for video tracks captured from UIProcess
     4        https://bugs.webkit.org/show_bug.cgi?id=204945
     5
     6        Reviewed by Eric Carlson.
     7
     8        Remove whether the a source is remote or not.
     9        Previously we were doing the distinction as resizing would happen in WebProcess and not in UIProcess.
     10        We are now moving away from doing resizing in WebProcess.
     11        Covered by existing tests.
     12
     13        * platform/mediastream/RealtimeMediaSource.h:
     14        * platform/mediastream/RealtimeMediaSourceFactory.h:
     15        * platform/mediastream/RealtimeVideoCaptureSource.cpp:
     16        (WebCore::RealtimeVideoCaptureSource::adaptVideoSample):
     17        For remote tracks, we also update the size for observers.
     18        * platform/mediastream/RealtimeVideoSource.cpp:
     19        (WebCore::m_source):
     20        (WebCore::RealtimeVideoSource::sourceSettingsChanged):
     21        Update computation of size based on rotation.
     22        This makes sure we have resizing done right.
     23        * platform/mediastream/RealtimeVideoSource.h:
     24        * platform/mediastream/mac/DisplayCaptureSourceCocoa.cpp:
     25        (WebCore::DisplayCaptureSourceCocoa::emitFrame):
     26
    1272019-12-08  Antti Koivisto  <antti@apple.com>
    228

  • trunk/Source/WebCore/platform/mediastream/RealtimeMediaSource.h

    r248046 r253275  
    184184    virtual bool isIncomingVideoSource() const { return false; }
    185185
    186     void setIsRemote(bool isRemote) { m_isRemote = isRemote; }
    187     bool isRemote() const { return m_isRemote; }
    188 
    189186#if !RELEASE_LOG_DISABLED
    190187    void setLogger(const Logger&, const void*);
    … …  
    269266    bool m_interrupted { false };
    270267    bool m_captureDidFailed { false };
    271     bool m_isRemote { false };
    272268    bool m_isEnded { false };
    273269};

  • trunk/Source/WebCore/platform/mediastream/RealtimeMediaSourceFactory.h

    r253033 r253275  
    4343    WEBCORE_EXPORT void setActiveSource(RealtimeMediaSource&);
    4444    WEBCORE_EXPORT void unsetActiveSource(RealtimeMediaSource&);
     45
     46protected:
    4547    RealtimeMediaSource* activeSource() { return m_activeSource; }
    4648

  • trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.cpp

    r246644 r253275  
    396396
    397397#if PLATFORM(COCOA)
    398     if (!isRemote()) {
    399         auto size = this->size();
    400         if (!size.isEmpty() && size != expandedIntSize(sample.presentationSize())) {
    401 
    402             if (!m_imageTransferSession || m_imageTransferSession->pixelFormat() != sample.videoPixelFormat())
    403                 m_imageTransferSession = ImageTransferSessionVT::create(sample.videoPixelFormat());
    404 
    405             if (m_imageTransferSession) {
    406                 mediaSample = m_imageTransferSession->convertMediaSample(sample, size);
    407                 if (!mediaSample) {
    408                     ASSERT_NOT_REACHED();
    409                     return nullptr;
    410                 }
     398    auto size = this->size();
     399    if (!size.isEmpty() && size != expandedIntSize(sample.presentationSize())) {
     400
     401        if (!m_imageTransferSession || m_imageTransferSession->pixelFormat() != sample.videoPixelFormat())
     402            m_imageTransferSession = ImageTransferSessionVT::create(sample.videoPixelFormat());
     403
     404        ASSERT(m_imageTransferSession);
     405        if (m_imageTransferSession) {
     406            mediaSample = m_imageTransferSession->convertMediaSample(sample, size);
     407            if (!mediaSample) {
     408                ASSERT_NOT_REACHED();
     409                return nullptr;
    411410            }
    412411        }

  • trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.cpp

    r247391 r253275  
    3737    m_source->addObserver(*this);
    3838    m_currentSettings = m_source->settings();
     39    m_rotation = m_source->sampleRotation();
     40    setSize(m_source->size());
    3941}
    4042
    … …  
    9294void RealtimeVideoSource::sourceSettingsChanged()
    9395{
    94     auto rotation = m_source->sampleRotation();
    9596    auto size = this->size();
    9697    if (size.isEmpty())
    9798        size = m_source->size();
    98     if (rotation == MediaSample::VideoRotation::Left || rotation == MediaSample::VideoRotation::Right)
    99         size = size.transposedSize();
     99
     100    auto rotation = m_source->sampleRotation();
     101    if (m_rotation != rotation) {
     102        bool shouldTransposeSize = false;
     103        switch (m_rotation) {
     104        case MediaSample::VideoRotation::None:
     105        case MediaSample::VideoRotation::UpsideDown:
     106            shouldTransposeSize = rotation == MediaSample::VideoRotation::Left || rotation == MediaSample::VideoRotation::Right;
     107            break;
     108        case MediaSample::VideoRotation::Left:
     109        case MediaSample::VideoRotation::Right:
     110            shouldTransposeSize = rotation == MediaSample::VideoRotation::None || rotation == MediaSample::VideoRotation::UpsideDown;
     111        }
     112        m_rotation = rotation;
     113        if (shouldTransposeSize) {
     114            size = size.transposedSize();
     115            setSize(size);
     116        }
     117    }
     118
    100119    m_currentSettings.setWidth(size.width());
    101120    m_currentSettings.setHeight(size.height());

  • trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.h

    r247391 r253275  
    6565    void videoSampleAvailable(MediaSample&) final;
    6666
     67    MediaSample::VideoRotation m_rotation { MediaSample::VideoRotation::None };
    6768    Ref<RealtimeVideoCaptureSource> m_source;
    6869    RealtimeMediaSourceSettings m_currentSettings;

  • trunk/Source/WebCore/platform/mediastream/mac/DisplayCaptureSourceCocoa.cpp

    r239163 r253275  
    187187    setIntrinsicSize(imageSize);
    188188
    189     auto mediaSampleSize = isRemote() ? imageSize : frameSize();
     189    auto mediaSampleSize = frameSize();
    190190
    191191    RefPtr<MediaSample> sample = WTF::switchOn(frame,

  • trunk/Source/WebKit/ChangeLog

    r253274 r253275  
     12019-12-08  youenn fablet  <youenn@apple.com>
     2
     3        Support different resolutions for video tracks captured from UIProcess
     4        https://bugs.webkit.org/show_bug.cgi?id=204945
     5
     6        Reviewed by Eric Carlson.
     7
     8        Change video capture in uiprocess to an experimental flag.
     9        Add support for remote video source cloning by sending some IPC to UIProcess and cloning the source there.
     10        This triggers IPC for both cloning and cloned sources but this allows to do resizing in UIProcess instead of WebProcess.
     11        We thus disable video capture resizing in WebProcess.
     12
     13        * Shared/WebPreferences.yaml:
     14        * UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp:
     15        (WebKit::UserMediaCaptureManagerProxy::SourceProxy::start):
     16        (WebKit::UserMediaCaptureManagerProxy::SourceProxy::stop):
     17        (WebKit::UserMediaCaptureManagerProxy::SourceProxy::requestToEnd):
     18        (WebKit::UserMediaCaptureManagerProxy::SourceProxy::preventSourceFromStopping):
     19        (WebKit::UserMediaCaptureManagerProxy::createMediaSourceForCaptureDeviceWithConstraints):
     20        (WebKit::UserMediaCaptureManagerProxy::startProducingData):
     21        (WebKit::UserMediaCaptureManagerProxy::stopProducingData):
     22        (WebKit::UserMediaCaptureManagerProxy::clone):
     23        (WebKit::UserMediaCaptureManagerProxy::requestToEnd):
     24        * UIProcess/Cocoa/UserMediaCaptureManagerProxy.h:
     25        * UIProcess/Cocoa/UserMediaCaptureManagerProxy.messages.in:
     26        * WebProcess/cocoa/UserMediaCaptureManager.cpp:
     27        (WebKit::UserMediaCaptureManager::Source::sourceID const):
     28        (WebKit::UserMediaCaptureManager::Source::settings const):
     29        (WebKit::UserMediaCaptureManager::Source::remoteVideoSampleAvailable):
     30        (WebKit::UserMediaCaptureManager::Source::requestToEnd):
     31        (WebKit::UserMediaCaptureManager::Source::stopBeingObserved):
     32        (WebKit::UserMediaCaptureManager::cloneSource):
     33        (WebKit::UserMediaCaptureManager::cloneVideoSource):
     34        (WebKit::UserMediaCaptureManager::requestToEnd):
     35        (WebKit::UserMediaCaptureManager::VideoFactory::setVideoCapturePageState):
     36        * WebProcess/cocoa/UserMediaCaptureManager.h:
     37
    1382019-12-08  youenn fablet  <youenn@apple.com>
    239

  • trunk/Source/WebKit/Shared/WebPreferences.yaml

    r253213 r253275  
    14991499  webcoreName: genericCueAPIEnabled
    15001500
     1501CaptureVideoInUIProcessEnabled:
     1502  type: bool
     1503  defaultValue: false
     1504  humanReadableName: "Capture video in UIProcess"
     1505  humanReadableDescription: "Enable video capture in UIProcess"
     1506  category: experimental
     1507  webcoreBinding: none
     1508  condition: ENABLE(MEDIA_STREAM)
     1509
     1510
    15011511# For internal features:
    15021512# The type should be boolean.
    … …  
    16781688  humanReadableName: "Capture audio in UIProcess"
    16791689  humanReadableDescription: "Enable audio capture in UIProcess"
    1680   category: internal
    1681   webcoreBinding: none
    1682   condition: ENABLE(MEDIA_STREAM)
    1683 
    1684 CaptureVideoInUIProcessEnabled:
    1685   type: bool
    1686   defaultValue: false
    1687   humanReadableName: "Capture video in UIProcess"
    1688   humanReadableDescription: "Enable video capture in UIProcess"
    16891690  category: internal
    16901691  webcoreBinding: none

  • trunk/Source/WebKit/UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp

    r253045 r253275  
    6969    int64_t numberOfFrames() { return m_numberOfFrames; }
    7070
     71    void start()
     72    {
     73        m_isEnded = false;
     74        m_source->start();
     75    }
     76
     77    void stop()
     78    {
     79        m_isEnded = true;
     80        m_source->stop();
     81    }
     82
     83    void requestToEnd()
     84    {
     85        m_isEnded = true;
     86        m_source->requestToEnd(*this);
     87    }
     88
     89private:
    7190    void sourceStopped() final {
    7291        if (m_source->captureDidFail()) {
    … …  
    122141    }
    123142
    124 protected:
     143    bool preventSourceFromStopping()
     144    {
     145        // Do not allow the source to stop if we are still using it.
     146        return !m_isEnded;
     147    }
     148
    125149    uint64_t m_id;
    126150    Ref<IPC::Connection> m_connection;
    … …  
    129153    CAAudioStreamDescription m_description { };
    130154    int64_t m_numberOfFrames { 0 };
     155    bool m_isEnded { false };
    131156};
    132157
    … …  
    170195    if (sourceOrError) {
    171196        auto source = sourceOrError.source();
    172         source->setIsRemote(true);
    173197        settings = source->settings();
    174198        ASSERT(!m_proxies.contains(id));
    … …  
    183207    MESSAGE_CHECK_CONTEXTID(id);
    184208    if (auto* proxy = m_proxies.get(id))
    185         proxy->source().start();
     209        proxy->start();
    186210}
    187211
    … …  
    190214    MESSAGE_CHECK_CONTEXTID(id);
    191215    if (auto* proxy = m_proxies.get(id))
    192         proxy->source().stop();
     216        proxy->stop();
    193217}
    194218
    … …  
    230254}
    231255
     256void UserMediaCaptureManagerProxy::clone(uint64_t clonedID, uint64_t newSourceID)
     257{
     258    ASSERT(m_proxies.contains(clonedID));
     259    ASSERT(!m_proxies.contains(newSourceID));
     260    if (auto* proxy = m_proxies.get(clonedID))
     261        m_proxies.add(newSourceID, makeUnique<SourceProxy>(newSourceID, *m_process.connection(), proxy->source().clone()));
     262}
     263
     264void UserMediaCaptureManagerProxy::requestToEnd(uint64_t sourceID)
     265{
     266    if (auto* proxy = m_proxies.get(sourceID))
     267        proxy->requestToEnd();
     268}
     269
    232270void UserMediaCaptureManagerProxy::clear()
    233271{

  • trunk/Source/WebKit/UIProcess/Cocoa/UserMediaCaptureManagerProxy.h

    r253045 r253275  
    6262    void setMuted(uint64_t, bool);
    6363    void applyConstraints(uint64_t, const WebCore::MediaConstraints&);
     64    void clone(uint64_t clonedID, uint64_t cloneID);
     65    void requestToEnd(uint64_t);
    6466
    6567    class SourceProxy;

  • trunk/Source/WebKit/UIProcess/Cocoa/UserMediaCaptureManagerProxy.messages.in

    r252655 r253275  
    3232    SetMuted(uint64_t id, bool muted)
    3333    ApplyConstraints(uint64_t id, struct WebCore::MediaConstraints constraints)
     34    Clone(uint64_t clonedID, uint64_t cloneID)
     35    RequestToEnd(uint64_t sourceID)
    3436}
    3537

  • trunk/Source/WebKit/WebProcess/cocoa/UserMediaCaptureManager.cpp

    r253055 r253275  
    111111    }
    112112
     113    Ref<RealtimeMediaSource> clone() final
     114    {
     115        return m_manager.cloneSource(*this);
     116    }
     117
     118    uint64_t sourceID() const
     119    {
     120        return m_id;
     121    }
     122
     123    const RealtimeMediaSourceSettings& settings() const
     124    {
     125        return m_settings;
     126    }
     127
    113128    const RealtimeMediaSourceCapabilities& capabilities() final
    114129    {
    … …  
    166181        ASSERT(type() == Type::Video);
    167182
    168         auto remoteSampleSize = remoteSample.size();
    169         setIntrinsicSize(remoteSampleSize);
    170 
    171         auto videoSampleSize = IntSize(m_settings.width(), m_settings.height());
    172         if (videoSampleSize.isZero())
    173             videoSampleSize = remoteSampleSize;
    174         else if (!videoSampleSize.height())
    175             videoSampleSize.setHeight(videoSampleSize.width() * (remoteSampleSize.height() / static_cast<double>(remoteSampleSize.width())));
    176         else if (!videoSampleSize.width())
    177             videoSampleSize.setWidth(videoSampleSize.height() * (remoteSampleSize.width() / static_cast<double>(remoteSampleSize.height())));
     183        setIntrinsicSize(remoteSample.size());
    178184
    179185        if (!m_imageTransferSession || m_imageTransferSession->pixelFormat() != remoteSample.videoFormat())
    … …  
    185191        }
    186192
    187         auto sampleRef = m_imageTransferSession->createMediaSample(remoteSample.surface(), remoteSample.time(), videoSampleSize);
     193        auto sampleRef = m_imageTransferSession->createMediaSample(remoteSample.surface(), remoteSample.time(), remoteSample.size());
    188194        if (!sampleRef) {
    189195            ASSERT_NOT_REACHED();
    … …  
    209215    }
    210216
     217    CaptureDevice::DeviceType deviceType() const final { return m_deviceType; }
     218
    211219private:
    212220    void startProducingData() final { m_manager.startProducingData(m_id); }
    213221    void stopProducingData() final { m_manager.stopProducingData(m_id); }
    214222    bool isCaptureSource() const final { return true; }
    215     CaptureDevice::DeviceType deviceType() const final { return m_deviceType; }
    216223
    217224    // RealtimeMediaSource
    … …  
    224231        m_manager.applyConstraints(m_id, constraints);
    225232        m_pendingApplyConstraintsCallbacks.append(WTFMove(completionHandler));
     233    }
     234
     235    void requestToEnd(RealtimeMediaSource::Observer&)
     236    {
     237        m_manager.requestToEnd(m_id);
     238    }
     239
     240    void stopBeingObserved()
     241    {
     242        m_manager.requestToEnd(m_id);
    226243    }
    227244
    … …  
    401418}
    402419
     420Ref<RealtimeMediaSource> UserMediaCaptureManager::cloneSource(Source& source)
     421{
     422    switch (source.type()) {
     423    case RealtimeMediaSource::Type::Video:
     424        return cloneVideoSource(source);
     425    case RealtimeMediaSource::Type::Audio:
     426        break;
     427    case RealtimeMediaSource::Type::None:
     428        ASSERT_NOT_REACHED();
     429    }
     430    return makeRef(source);
     431}
     432
     433Ref<RealtimeMediaSource> UserMediaCaptureManager::cloneVideoSource(Source& source)
     434{
     435    uint64_t id = nextSessionID();
     436    if (!m_process.send(Messages::UserMediaCaptureManagerProxy::Clone { source.sourceID(), id }, 0))
     437        return makeRef(source);
     438
     439    auto settings = source.settings();
     440    auto cloneSource = adoptRef(*new Source(String::number(id), source.type(), source.deviceType(), String { settings.label().string() }, source.deviceIDHashSalt(), id, *this));
     441    cloneSource->setSettings(WTFMove(settings));
     442    m_sources.add(id, cloneSource.copyRef());
     443    return cloneSource;
     444}
     445
     446void UserMediaCaptureManager::requestToEnd(uint64_t sourceID)
     447{
     448    m_process.send(Messages::UserMediaCaptureManagerProxy::RequestToEnd { sourceID }, 0);
     449}
     450
    403451#if PLATFORM(IOS_FAMILY)
    404452void UserMediaCaptureManager::AudioFactory::setAudioCapturePageState(bool interrupted, bool pageMuted)
    405453{
    406     if (auto* activeSource = static_cast<AudioCaptureFactory*>(this)->activeSource())
     454    if (auto* activeSource = this->activeSource())
    407455        activeSource->setInterrupted(interrupted, pageMuted);
    408456}
    … …  
    410458void UserMediaCaptureManager::VideoFactory::setVideoCapturePageState(bool interrupted, bool pageMuted)
    411459{
    412     if (auto* activeSource = static_cast<VideoCaptureFactory*>(this)->activeSource())
    413         activeSource->setInterrupted(interrupted, pageMuted);
    414 }
    415 #endif
    416 
    417 }
    418 
    419 #endif
     460    // In case of cloning, we might have more than a single source.
     461    for (auto& source : m_manager.m_sources.values()) {
     462        if (source->deviceType() == CaptureDevice::DeviceType::Camera)
     463            source->setInterrupted(interrupted, pageMuted);
     464    }
     465}
     466#endif
     467
     468}
     469
     470#endif

  • trunk/Source/WebKit/WebProcess/cocoa/UserMediaCaptureManager.h

    r253055 r253275  
    135135    class Source;
    136136    friend class Source;
     137
     138    void requestToEnd(uint64_t sourceID);
     139    Ref<WebCore::RealtimeMediaSource> cloneSource(Source&);
     140    Ref<WebCore::RealtimeMediaSource> cloneVideoSource(Source&);
     141
    137142    HashMap<uint64_t, RefPtr<Source>> m_sources;
    138143    WebProcess& m_process;
Note: See TracChangeset for help on using the changeset viewer.