⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 275871 in webkit


Ignore:
Timestamp:
Apr 13, 2021, 1:38:14 AM (6 years ago)
Author:
commit-queue@webkit.org
Message:

[GStreamer] CaptureDevice monitor used from UIProcess
​https://bugs.webkit.org/show_bug.cgi?id=222888

Patch by Philippe Normand <​pnormand@igalia.com> on 2021-04-13
Source/WebCore:

Reviewed by Youenn Fablet .

Move the device observer infrastructure from the WebKit UserMediaProcessManager to the
RealtimeMediaSourceCenter.

  • platform/mediastream/CaptureDevice.h:

(WebCore::haveDevicesChanged):

  • platform/mediastream/RealtimeMediaSourceCenter.cpp:

(WebCore::RealtimeMediaSourceCenter::RealtimeMediaSourceCenter):
(WebCore::RealtimeMediaSourceCenter::Observer::~Observer):
(WebCore::RealtimeMediaSourceCenter::addDevicesChangedObserver):
(WebCore::RealtimeMediaSourceCenter::removeDevicesChangedObserver):
(WebCore::RealtimeMediaSourceCenter::captureDevicesChanged):
(WebCore::RealtimeMediaSourceCenter::triggerDevicesChangedObservers):
(WebCore::RealtimeMediaSourceCenter::setDevicesChangedObserver): Deleted.

  • platform/mediastream/RealtimeMediaSourceCenter.h:

(WebCore::RealtimeMediaSourceCenter::Observer::devicesChanged):

Source/WebKit:

Reviewed by Youenn Fablet.

GStreamer ports aim to reduce usage of the GStreamer library in the UIProcess and thus
should not relay capture devicechange monitoring to the UIProcess. For the long term we plan
to migrate media processing to the GPUProcess but for now we have to maintain the media
pipelines in the WebProcess. Initializing GStreamer in multiple processes (including the
UIProcess) has a significant performance impact on embedded platforms, that's why we try to
avoid it as much as possible.

The device monitoring code from the UserMediaProcessManager was partly moved to the
RealtimeMediaSourceCenter so that it can be easily reused by GStreamer ports from the
UserMediaPermissionRequestManager.

  • UIProcess/UserMediaProcessManager.cpp:

(WebKit::UserMediaProcessManager::UserMediaProcessManager):
(WebKit::UserMediaProcessManager::updateCaptureDevices):
(WebKit::UserMediaProcessManager::devicesChanged):
(WebKit::UserMediaProcessManager::beginMonitoringCaptureDevices):

  • UIProcess/UserMediaProcessManager.h:
  • WebProcess/MediaStream/UserMediaPermissionRequestManager.cpp:

(WebKit::UserMediaPermissionRequestManager::UserMediaPermissionRequestManager):
(WebKit::UserMediaPermissionRequestManager::updateCaptureDevices):
(WebKit::UserMediaPermissionRequestManager::DeviceObserver::devicesChanged):
(WebKit::UserMediaPermissionRequestManager::addDeviceChangeObserver):

  • WebProcess/MediaStream/UserMediaPermissionRequestManager.h:

(WebKit::UserMediaPermissionRequestManager::DeviceObserver::DeviceObserver):

Location:
trunk/Source
Files:
9 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebCore/ChangeLog

    r275869 r275871  
     12021-04-13  Philippe Normand  <pnormand@igalia.com>
     2
     3        [GStreamer] CaptureDevice monitor used from UIProcess
     4        https://bugs.webkit.org/show_bug.cgi?id=222888
     5
     6        Reviewed by Youenn Fablet .
     7
     8        Move the device observer infrastructure from the WebKit UserMediaProcessManager to the
     9        RealtimeMediaSourceCenter.
     10
     11        * platform/mediastream/CaptureDevice.h:
     12        (WebCore::haveDevicesChanged):
     13        * platform/mediastream/RealtimeMediaSourceCenter.cpp:
     14        (WebCore::RealtimeMediaSourceCenter::RealtimeMediaSourceCenter):
     15        (WebCore::RealtimeMediaSourceCenter::Observer::~Observer):
     16        (WebCore::RealtimeMediaSourceCenter::addDevicesChangedObserver):
     17        (WebCore::RealtimeMediaSourceCenter::removeDevicesChangedObserver):
     18        (WebCore::RealtimeMediaSourceCenter::captureDevicesChanged):
     19        (WebCore::RealtimeMediaSourceCenter::triggerDevicesChangedObservers):
     20        (WebCore::RealtimeMediaSourceCenter::setDevicesChangedObserver): Deleted.
     21        * platform/mediastream/RealtimeMediaSourceCenter.h:
     22        (WebCore::RealtimeMediaSourceCenter::Observer::devicesChanged):
     23
    1242021-04-13  Antoine Quint  <graouts@webkit.org>
    225

  • trunk/Source/WebCore/platform/mediastream/CaptureDevice.h

    r274819 r275871  
    141141};
    142142
     143inline bool haveDevicesChanged(const Vector<CaptureDevice>& oldDevices, const Vector<CaptureDevice>& newDevices)
     144{
     145    if (oldDevices.size() != newDevices.size())
     146        return true;
     147
     148    for (auto& newDevice : newDevices) {
     149        if (newDevice.type() != CaptureDevice::DeviceType::Camera && newDevice.type() != CaptureDevice::DeviceType::Microphone)
     150            continue;
     151
     152        auto index = oldDevices.findMatching([&newDevice](auto& oldDevice) {
     153            return newDevice.persistentId() == oldDevice.persistentId() && newDevice.enabled() != oldDevice.enabled();
     154        });
     155
     156        if (index == notFound)
     157            return true;
     158    }
     159
     160    return false;
     161}
     162
    143163} // namespace WebCore
    144164

  • trunk/Source/WebCore/platform/mediastream/RealtimeMediaSourceCenter.cpp

    r271471 r275871  
    4343namespace WebCore {
    4444
     45static const Seconds deviceChangeDebounceTimerInterval { 200_ms };
     46
    4547RealtimeMediaSourceCenter& RealtimeMediaSourceCenter::singleton()
    4648{
    … …  
    5153
    5254RealtimeMediaSourceCenter::RealtimeMediaSourceCenter()
     55    : m_debounceTimer(RunLoop::main(), this, &RealtimeMediaSourceCenter::triggerDevicesChangedObservers)
    5356{
    5457    m_supportedConstraints.setSupportsEchoCancellation(true);
    … …  
    6366
    6467RealtimeMediaSourceCenter::~RealtimeMediaSourceCenter() = default;
     68
     69RealtimeMediaSourceCenter::Observer::~Observer() = default;
    6570
    6671void RealtimeMediaSourceCenter::createMediaStream(Ref<const Logger>&& logger, NewMediaStreamHandler&& completionHandler, String&& hashSalt, CaptureDevice&& audioDevice, CaptureDevice&& videoDevice, const MediaStreamRequest& request)
    … …  
    189194}
    190195
    191 void RealtimeMediaSourceCenter::setDevicesChangedObserver(std::function<void()>&& observer)
     196void RealtimeMediaSourceCenter::addDevicesChangedObserver(Observer& observer)
    192197{
    193198    ASSERT(isMainThread());
    194     ASSERT(!m_deviceChangedObserver);
    195     m_deviceChangedObserver = WTFMove(observer);
     199    m_observers.add(observer);
     200}
     201
     202void RealtimeMediaSourceCenter::removeDevicesChangedObserver(Observer& observer)
     203{
     204    ASSERT(isMainThread());
     205    m_observers.remove(observer);
    196206}
    197207
    … …  
    199209{
    200210    ASSERT(isMainThread());
    201     if (m_deviceChangedObserver)
    202         m_deviceChangedObserver();
     211
     212    // When a device with camera and microphone is attached or detached, the CaptureDevice notification for
     213    // the different devices won't arrive at the same time so delay a bit so we can coalesce the callbacks.
     214    if (!m_debounceTimer.isActive())
     215        m_debounceTimer.startOneShot(deviceChangeDebounceTimerInterval);
     216}
     217
     218void RealtimeMediaSourceCenter::triggerDevicesChangedObservers()
     219{
     220    auto protectedThis = makeRef(*this);
     221    m_observers.forEach([](auto& observer) {
     222        observer.devicesChanged();
     223    });
    203224}
    204225

  • trunk/Source/WebCore/platform/mediastream/RealtimeMediaSourceCenter.h

    r271471 r275871  
    4040#include "RealtimeMediaSourceFactory.h"
    4141#include "RealtimeMediaSourceSupportedConstraints.h"
     42#include "UserMediaClient.h"
    4243#include <wtf/Function.h>
    4344#include <wtf/RefPtr.h>
     45#include <wtf/RunLoop.h>
    4446#include <wtf/text/WTFString.h>
    4547
    … …  
    5355
    5456struct MediaConstraints;
    55    
    56 class RealtimeMediaSourceCenter {
     57
     58class WEBCORE_EXPORT RealtimeMediaSourceCenter : public ThreadSafeRefCounted<RealtimeMediaSourceCenter, WTF::DestructionThread::MainRunLoop> {
    5759public:
     60    class Observer : public CanMakeWeakPtr<Observer> {
     61    public:
     62        virtual ~Observer();
     63
     64        virtual void devicesChanged() = 0;
     65    };
     66
    5867    ~RealtimeMediaSourceCenter();
    5968
    … …  
    8594    WEBCORE_EXPORT String hashStringWithSalt(const String& id, const String& hashSalt);
    8695
    87     WEBCORE_EXPORT void setDevicesChangedObserver(std::function<void()>&&);
     96    WEBCORE_EXPORT void addDevicesChangedObserver(Observer&);
     97    WEBCORE_EXPORT void removeDevicesChangedObserver(Observer&);
    8898
    8999    void captureDevicesChanged();
    … …  
    109119    RealtimeMediaSourceSupportedConstraints m_supportedConstraints;
    110120
    111     WTF::Function<void()> m_deviceChangedObserver;
     121    RunLoop::Timer<RealtimeMediaSourceCenter> m_debounceTimer;
     122    void triggerDevicesChangedObservers();
     123
     124    WeakHashSet<Observer> m_observers;
    112125
    113126    AudioCaptureFactory* m_audioCaptureFactoryOverride { nullptr };

  • trunk/Source/WebKit/ChangeLog

    r275861 r275871  
     12021-04-13  Philippe Normand  <pnormand@igalia.com>
     2
     3        [GStreamer] CaptureDevice monitor used from UIProcess
     4        https://bugs.webkit.org/show_bug.cgi?id=222888
     5
     6        Reviewed by Youenn Fablet.
     7
     8        GStreamer ports aim to reduce usage of the GStreamer library in the UIProcess and thus
     9        should not relay capture devicechange monitoring to the UIProcess. For the long term we plan
     10        to migrate media processing to the GPUProcess but for now we have to maintain the media
     11        pipelines in the WebProcess. Initializing GStreamer in multiple processes (including the
     12        UIProcess) has a significant performance impact on embedded platforms, that's why we try to
     13        avoid it as much as possible.
     14
     15        The device monitoring code from the UserMediaProcessManager was partly moved to the
     16        RealtimeMediaSourceCenter so that it can be easily reused by GStreamer ports from the
     17        UserMediaPermissionRequestManager.
     18
     19        * UIProcess/UserMediaProcessManager.cpp:
     20        (WebKit::UserMediaProcessManager::UserMediaProcessManager):
     21        (WebKit::UserMediaProcessManager::updateCaptureDevices):
     22        (WebKit::UserMediaProcessManager::devicesChanged):
     23        (WebKit::UserMediaProcessManager::beginMonitoringCaptureDevices):
     24        * UIProcess/UserMediaProcessManager.h:
     25        * WebProcess/MediaStream/UserMediaPermissionRequestManager.cpp:
     26        (WebKit::UserMediaPermissionRequestManager::UserMediaPermissionRequestManager):
     27        (WebKit::UserMediaPermissionRequestManager::updateCaptureDevices):
     28        (WebKit::UserMediaPermissionRequestManager::DeviceObserver::devicesChanged):
     29        (WebKit::UserMediaPermissionRequestManager::addDeviceChangeObserver):
     30        * WebProcess/MediaStream/UserMediaPermissionRequestManager.h:
     31        (WebKit::UserMediaPermissionRequestManager::DeviceObserver::DeviceObserver):
     32
    1332021-04-12  Julian Gonzalez  <julian_a_gonzalez@apple.com>
    234

  • trunk/Source/WebKit/UIProcess/UserMediaProcessManager.cpp

    r274279 r275871  
    2828#include "WebProcessMessages.h"
    2929#include "WebProcessProxy.h"
    30 #include <WebCore/RealtimeMediaSourceCenter.h>
    3130#include <wtf/HashMap.h>
    3231#include <wtf/NeverDestroyed.h>
    … …  
    4241#endif
    4342
    44 static const Seconds deviceChangeDebounceTimerInterval { 200_ms };
    45 
    4643UserMediaProcessManager& UserMediaProcessManager::singleton()
    4744{
    … …  
    5148
    5249UserMediaProcessManager::UserMediaProcessManager()
    53     : m_debounceTimer(RunLoop::main(), this, &UserMediaProcessManager::captureDevicesChanged)
    5450{
    5551}
    … …  
    217213void UserMediaProcessManager::updateCaptureDevices(ShouldNotify shouldNotify)
    218214{
    219     WebCore::RealtimeMediaSourceCenter::singleton().getMediaStreamDevices([this, shouldNotify] (auto&& newDevices) mutable {
    220         auto oldDevices = WTFMove(m_captureDevices);
     215    WebCore::RealtimeMediaSourceCenter::singleton().getMediaStreamDevices([weakThis = makeWeakPtr(*this), this, shouldNotify](auto&& newDevices) mutable {
     216        if (!weakThis)
     217            return;
     218
     219        if (!haveDevicesChanged(m_captureDevices, newDevices))
     220            return;
     221
    221222        m_captureDevices = WTFMove(newDevices);
    222 
    223         if (shouldNotify == ShouldNotify::No)
    224             return;
    225 
    226         if (m_captureDevices.size() == oldDevices.size()) {
    227             bool haveChanges = false;
    228             for (auto &newDevice : m_captureDevices) {
    229                 if (newDevice.type() != WebCore::CaptureDevice::DeviceType::Camera && newDevice.type() != WebCore::CaptureDevice::DeviceType::Microphone)
    230                     continue;
    231 
    232                 auto index = oldDevices.findMatching([&newDevice] (auto& oldDevice) {
    233                     return newDevice.persistentId() == oldDevice.persistentId() && newDevice.enabled() != oldDevice.enabled();
    234                 });
    235 
    236                 if (index == notFound) {
    237                     haveChanges = true;
    238                     break;
    239                 }
    240             }
    241 
    242             if (!haveChanges)
    243                 return;
    244         }
    245 
    246         // When a device with camera and microphone is attached or detached, the CaptureDevice notification for
    247         // the different devices won't arrive at the same time so delay a bit so we can coalesce the callbacks.
    248         if (!m_debounceTimer.isActive())
    249             m_debounceTimer.startOneShot(deviceChangeDebounceTimerInterval);
    250     });
     223        if (shouldNotify == ShouldNotify::Yes)
     224            captureDevicesChanged();
     225    });
     226}
     227
     228void UserMediaProcessManager::devicesChanged()
     229{
     230    updateCaptureDevices(ShouldNotify::Yes);
    251231}
    252232
    … …  
    257237    std::call_once(onceFlag, [this] {
    258238        updateCaptureDevices(ShouldNotify::No);
    259 
    260         WebCore::RealtimeMediaSourceCenter::singleton().setDevicesChangedObserver([this]() {
    261             updateCaptureDevices(ShouldNotify::Yes);
    262         });
     239        WebCore::RealtimeMediaSourceCenter::singleton().addDevicesChangedObserver(*this);
    263240    });
    264241}

  • trunk/Source/WebKit/UIProcess/UserMediaProcessManager.h

    r271471 r275871  
    2424#include <WebCore/CaptureDevice.h>
    2525#include <WebCore/PageIdentifier.h>
    26 #include <wtf/RunLoop.h>
     26#include <WebCore/RealtimeMediaSourceCenter.h>
     27#include <WebCore/UserMediaClient.h>
     28#include <wtf/WeakPtr.h>
    2729
    2830namespace WebKit {
    … …  
    3032class WebProcessProxy;
    3133
    32 class UserMediaProcessManager {
     34class UserMediaProcessManager : public WebCore::RealtimeMediaSourceCenter::Observer {
    3335public:
    3436
    … …  
    4951
    5052private:
    51     enum class ShouldNotify { Yes, No };
     53    enum class ShouldNotify : bool { No, Yes };
    5254    void updateCaptureDevices(ShouldNotify);
    5355    void captureDevicesChanged();
    5456
     57    // RealtimeMediaSourceCenter::Observer
     58    void devicesChanged() final;
     59
    5560    Vector<WebCore::CaptureDevice> m_captureDevices;
    56     RunLoop::Timer<UserMediaProcessManager> m_debounceTimer;
    5761    bool m_captureEnabled { true };
    5862    bool m_denyNextRequest { false };

  • trunk/Source/WebKit/WebProcess/MediaStream/UserMediaPermissionRequestManager.cpp

    r269676 r275871  
    149149}
    150150
     151#if USE(GSTREAMER)
     152void UserMediaPermissionRequestManager::updateCaptureDevices(ShouldNotify shouldNotify)
     153{
     154    WebCore::RealtimeMediaSourceCenter::singleton().getMediaStreamDevices([weakThis = makeWeakPtr(*this), this, shouldNotify](auto&& newDevices) mutable {
     155        if (!weakThis)
     156            return;
     157
     158        if (!haveDevicesChanged(m_captureDevices, newDevices))
     159            return;
     160
     161        m_captureDevices = WTFMove(newDevices);
     162        if (shouldNotify == ShouldNotify::Yes)
     163            captureDevicesChanged();
     164    });
     165}
     166
     167void UserMediaPermissionRequestManager::devicesChanged()
     168{
     169    updateCaptureDevices(ShouldNotify::Yes);
     170}
     171#endif
     172
    151173UserMediaClient::DeviceChangeObserverToken UserMediaPermissionRequestManager::addDeviceChangeObserver(Function<void()>&& observer)
    152174{
    … …  
    156178    if (!m_monitoringDeviceChange) {
    157179        m_monitoringDeviceChange = true;
     180#if USE(GSTREAMER)
     181        updateCaptureDevices(ShouldNotify::No);
     182        WebCore::RealtimeMediaSourceCenter::singleton().addDevicesChangedObserver(*this);
     183#else
    158184        m_page.send(Messages::WebPageProxy::BeginMonitoringCaptureDevices());
     185#endif
    159186    }
    160187    return identifier;

  • trunk/Source/WebKit/WebProcess/MediaStream/UserMediaPermissionRequestManager.h

    r269676 r275871  
    2525#include <WebCore/MediaCanStartListener.h>
    2626#include <WebCore/MediaConstraints.h>
     27#include <WebCore/RealtimeMediaSourceCenter.h>
    2728#include <WebCore/UserMediaClient.h>
    2829#include <WebCore/UserMediaRequest.h>
    … …  
    3536class WebPage;
    3637
    37 class UserMediaPermissionRequestManager : private WebCore::MediaCanStartListener {
     38class UserMediaPermissionRequestManager : public WebCore::MediaCanStartListener
     39#if USE(GSTREAMER)
     40                                        , public WebCore::RealtimeMediaSourceCenter::Observer
     41#endif
     42{
    3843    WTF_MAKE_FAST_ALLOCATED;
    3944public:
     45    using WebCore::MediaCanStartListener::weakPtrFactory;
     46    using WeakValueType = WebCore::MediaCanStartListener::WeakValueType;
     47
    4048    explicit UserMediaPermissionRequestManager(WebPage&);
    4149    ~UserMediaPermissionRequestManager() = default;
    … …  
    5462
    5563private:
     64#if USE(GSTREAMER)
     65    // WebCore::RealtimeMediaSourceCenter::Observer
     66    void devicesChanged() final;
     67#endif
     68
    5669    void sendUserMediaRequest(WebCore::UserMediaRequest&);
    5770
    … …  
    6679    HashMap<WebCore::UserMediaClient::DeviceChangeObserverToken, Function<void()>> m_deviceChangeObserverMap;
    6780    bool m_monitoringDeviceChange { false };
     81
     82#if USE(GSTREAMER)
     83    enum class ShouldNotify : bool { No, Yes };
     84    void updateCaptureDevices(ShouldNotify);
     85
     86    Vector<WebCore::CaptureDevice> m_captureDevices;
     87#endif
    6888};
    6989
Note: See TracChangeset for help on using the changeset viewer.