⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 271401 in webkit


Ignore:
Timestamp:
Jan 12, 2021, 6:34:28 AM (6 years ago)
Author:
youenn@apple.com
Message:

Unmuting a track in case of end of interruption by another web process tab is not working well with capture muting icons
​https://bugs.webkit.org/show_bug.cgi?id=220058

Reviewed by Eric Carlson.

Source/WebCore:

Before the patch, we were muting all tracks in case of Audio Session interruption.
This works well in case another application triggers the interruption.

In case the interruption is done by another tab, this does not work well for two reasons:

  • The UIProcess is doing the work of muting the tracks when another tab is starting to capture
  • Unmuting in case of interruption might unmute several tabs if they are in the same process

As an example, if a tab is capturing, and another tab starts to capture, going back to first tab will automatically
restart the audio capture as the Audio Session will end its interruption. But it will not mute the camera track.

To fix that, the Audio Shared Unit is no longer directly muting or unmuting the audio tracks.
Instead, each page is notified of start/end of AudioSession interruption.
When an interruption starts, all capture tracks are muted.
When an interruption ends, all capture tracks are updated according the page muted state, which is controlled by UIProcess.
This also covers the case of several tabs capturing in the same process, since only one tab will not be set as muted by UIProcess.

In case of several capture MediaStreamTrack in the same document and they are not the active source, we pick the first one as there is
no way to know which one was the last unmuted one.
We should probably add some support to always remember the last living track of a tab in iOS.

Manually tested.

  • Modules/mediastream/MediaStreamTrack.cpp:

(WebCore::MediaStreamTrack::create):
(WebCore::MediaStreamTrack::updateCaptureAccordingToMutedState):
(WebCore::MediaStreamTrack::updateToPageMutedState):

  • Modules/mediastream/MediaStreamTrack.h:
  • dom/Document.cpp:

(WebCore::Document::visibilityStateChanged):
(WebCore::Document::pageMutedStateDidChange):

  • page/Page.cpp:

(WebCore::Page::beginAudioCaptureInterruption):
(WebCore::Page::endAudioCaptureInterruption):

  • page/Page.h:
  • platform/mediastream/mac/BaseAudioSharedUnit.cpp:

(WebCore::BaseAudioSharedUnit::resume):
(WebCore::BaseAudioSharedUnit::suspend):

Source/WebKit:

Add an observer of AudioSession interruptions that forwards the signal to all tabs of the process.

  • WebProcess/cocoa/WebProcessCocoa.mm:

(WebKit::WebProcess::platformInitializeWebProcess):
(WebKit::PageAudioSessionInterruptionObserver::PageAudioSessionInterruptionObserver):
(WebKit::PageAudioSessionInterruptionObserver::~PageAudioSessionInterruptionObserver):
(WebKit::PageAudioSessionInterruptionObserver::beginAudioSessionInterruption):
(WebKit::PageAudioSessionInterruptionObserver::endAudioSessionInterruption):
(WebKit::listenToAudioSessionInterruption):

Location:
trunk
Files:
12 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/fast/mediastream/media-stream-track-interrupted.html

    r268400 r271401  
    5151    function testTrack(track, title)
    5252    {
    53         promise_test((test) => {
    54             return new Promise((resolve, reject) => {
     53        promise_test(async (test) => {
     54            if (track.muted)
     55                await new Promise(resolve => track.onunmute = resolve);
     56
     57            await new Promise((resolve, reject) => {
    5558                let isVideo = track.kind == "video";
    5659                if (window.internals) {
    … …  
    6871                            waitForPageStateChange(10, pageState, resolve, reject)
    6972                        }
    70 
    7173                        if (window.internals) {
    72                             assert_true(pageMediaState.includes(isVideo ? 'HasMutedVideoCaptureDevice' : 'HasMutedAudioCaptureDevice'));
    73                             assert_false(pageMediaState.includes(isVideo ? 'HasMutedAudioCaptureDevice' : 'HasMutedVideoCaptureDevice'));
    74                             assert_true(pageMediaState.includes(isVideo ? 'HasActiveAudioCaptureDevice' : 'HasActiveVideoCaptureDevice'));
    75                             assert_false(pageMediaState.includes(isVideo ? 'HasActiveVideoCaptureDevice' : 'HasActiveAudioCaptureDevice'));
     74                            assert_true(pageMediaState.includes('HasMutedAudioCaptureDevice'), 'audio muted');
     75                            assert_true(pageMediaState.includes('HasMutedVideoCaptureDevice'), 'video muted');
    7676                            pageState = internals.pageMediaState();
    7777                            internals.setMediaStreamSourceInterrupted(track, false)
    … …  
    8686                setTimeout(() => reject("Muted state did not change in 1 second"), 1000);
    8787            });
     88            track.onmute = () => { };
     89            track.onunmute = () => { };
    8890        }, title);
    8991    }

  • trunk/Source/WebCore/ChangeLog

    r271399 r271401  
     12021-01-12  Youenn Fablet  <youenn@apple.com>
     2
     3        Unmuting a track in case of end of interruption by another web process tab is not working well with capture muting icons
     4        https://bugs.webkit.org/show_bug.cgi?id=220058
     5
     6        Reviewed by Eric Carlson.
     7
     8        Before the patch, we were muting all tracks in case of Audio Session interruption.
     9        This works well in case another application triggers the interruption.
     10
     11        In case the interruption is done by another tab, this does not work well for two reasons:
     12        - The UIProcess is doing the work of muting the tracks when another tab is starting to capture
     13        - Unmuting in case of interruption might unmute several tabs if they are in the same process
     14        As an example, if a tab is capturing, and another tab starts to capture, going back to first tab will automatically
     15        restart the audio capture as the Audio Session will end its interruption. But it will not mute the camera track.
     16
     17        To fix that, the Audio Shared Unit is no longer directly muting or unmuting the audio tracks.
     18        Instead, each page is notified of start/end of AudioSession interruption.
     19        When an interruption starts, all capture tracks are muted.
     20        When an interruption ends, all capture tracks are updated according the page muted state, which is controlled by UIProcess.
     21        This also covers the case of several tabs capturing in the same process, since only one tab will not be set as muted by UIProcess.
     22
     23        In case of several capture MediaStreamTrack in the same document and they are not the active source, we pick the first one as there is
     24        no way to know which one was the last unmuted one.
     25        We should probably add some support to always remember the last living track of a tab in iOS.
     26
     27        Manually tested.
     28
     29        * Modules/mediastream/MediaStreamTrack.cpp:
     30        (WebCore::MediaStreamTrack::create):
     31        (WebCore::MediaStreamTrack::updateCaptureAccordingToMutedState):
     32        (WebCore::MediaStreamTrack::updateToPageMutedState):
     33        * Modules/mediastream/MediaStreamTrack.h:
     34        * dom/Document.cpp:
     35        (WebCore::Document::visibilityStateChanged):
     36        (WebCore::Document::pageMutedStateDidChange):
     37        * page/Page.cpp:
     38        (WebCore::Page::beginAudioCaptureInterruption):
     39        (WebCore::Page::endAudioCaptureInterruption):
     40        * page/Page.h:
     41        * platform/mediastream/mac/BaseAudioSharedUnit.cpp:
     42        (WebCore::BaseAudioSharedUnit::resume):
     43        (WebCore::BaseAudioSharedUnit::suspend):
     44
    1452021-01-12  Philippe Normand  <pnormand@igalia.com>
    246

  • trunk/Source/WebCore/Modules/mediastream/MediaStreamTrack.cpp

    r271060 r271401  
    7272    track->suspendIfNeeded();
    7373
    74     if (track->isCaptureTrack())
    75         track->updateToPageMutedState();
     74    if (track->isCaptureTrack()) {
     75        if (auto* page = track->document()->page())
     76            track->updateToPageMutedState(page->mutedState());
     77    }
    7678
    7779    return track;
    … …  
    470472#endif
    471473
    472 void MediaStreamTrack::updateCaptureAccordingToMutedState(Document& document)
     474void MediaStreamTrack::updateCaptureAccordingToMutedState(Document& document, MediaProducer::MutedStateFlags mutedState)
    473475{
    474476#if PLATFORM(IOS_FAMILY)
    … …  
    477479        return;
    478480
     481    bool pageMuted = mutedState & MediaProducer::AudioAndVideoCaptureIsMuted;
     482
     483    auto updateTracksAccordingPageMutedState = [](const Document& document, CaptureDevice::DeviceType deviceType, bool pageMuted) {
     484        // We can only have one source at a time: we can mute all tracks but unmute only one of them.
     485        for (auto* captureTrack : allCaptureTracks()) {
     486            if (captureTrack->document() != &document || captureTrack->ended() || captureTrack->source().deviceType() != deviceType)
     487                continue;
     488            captureTrack->m_private->setMuted(pageMuted);
     489            // If unmuting, unmute the first source of the document we know.
     490            if (!pageMuted)
     491                break;
     492        }
     493    };
    479494    auto* activeAudioSource = RealtimeMediaSourceCenter::singleton().audioCaptureFactory().activeSource();
    480     if (activeAudioSource && isSourceCapturingForTrackInDocument(*activeAudioSource, document)) {
    481         bool pageMuted = page->mutedState() & MediaProducer::AudioAndVideoCaptureIsMuted;
     495    if (activeAudioSource && isSourceCapturingForTrackInDocument(*activeAudioSource, document))
    482496        activeAudioSource->setMuted(pageMuted || (document.hidden() && document.settings().interruptAudioOnPageVisibilityChangeEnabled()));
    483     }
     497    else
     498        updateTracksAccordingPageMutedState(document, CaptureDevice::DeviceType::Microphone, pageMuted);
    484499
    485500    auto* activeVideoSource = RealtimeMediaSourceCenter::singleton().videoCaptureFactory().activeSource();
    486     if (activeVideoSource && isSourceCapturingForTrackInDocument(*activeVideoSource, document)) {
    487         bool pageMuted = page->mutedState() & MediaProducer::AudioAndVideoCaptureIsMuted;
     501    if (activeVideoSource && isSourceCapturingForTrackInDocument(*activeVideoSource, document))
    488502        activeVideoSource->setMuted(pageMuted || document.hidden());
    489     }
     503    else
     504        updateTracksAccordingPageMutedState(document, CaptureDevice::DeviceType::Camera, pageMuted);
    490505#else
    491506    for (auto* captureTrack : allCaptureTracks()) {
    492507        if (captureTrack->document() == &document && !captureTrack->ended())
    493             captureTrack->updateToPageMutedState();
     508            captureTrack->updateToPageMutedState(mutedState);
    494509    }
    495510#endif
    496511}
    497512
    498 void MediaStreamTrack::updateToPageMutedState()
     513void MediaStreamTrack::updateToPageMutedState(MediaProducer::MutedStateFlags mutedState)
    499514{
    500515    ASSERT(isCaptureTrack());
    501     auto* page = document()->page();
    502     if (!page)
    503         return;
    504516
    505517    switch (source().deviceType()) {
    506518    case CaptureDevice::DeviceType::Microphone:
    507519    case CaptureDevice::DeviceType::Camera:
    508         m_private->setMuted(page->mutedState() & MediaProducer::AudioAndVideoCaptureIsMuted);
     520        m_private->setMuted(mutedState & MediaProducer::AudioAndVideoCaptureIsMuted);
    509521        break;
    510522    case CaptureDevice::DeviceType::Screen:
    511523    case CaptureDevice::DeviceType::Window:
    512         m_private->setMuted(page->mutedState() & MediaProducer::ScreenCaptureIsMuted);
     524        m_private->setMuted(mutedState & MediaProducer::ScreenCaptureIsMuted);
    513525        break;
    514526    case CaptureDevice::DeviceType::Speaker:

  • trunk/Source/WebCore/Modules/mediastream/MediaStreamTrack.h

    r271060 r271401  
    7373
    7474    static MediaProducer::MediaStateFlags captureState(Document&);
    75     static void updateCaptureAccordingToMutedState(Document&);
     75    static void updateCaptureAccordingToMutedState(Document&, MediaProducer::MutedStateFlags);
    7676
    7777    virtual bool isCanvas() const { return false; }
    … …  
    170170
    171171    void configureTrackRendering();
    172     void updateToPageMutedState();
     172    void updateToPageMutedState(MediaProducer::MutedStateFlags);
    173173
    174174    // ActiveDOMObject API.

  • trunk/Source/WebCore/dom/Document.cpp

    r271270 r271401  
    18111811
    18121812#if ENABLE(MEDIA_STREAM) && PLATFORM(IOS_FAMILY)
     1813    auto* page = this->page();
     1814    if (!page)
     1815        return;
     1816
    18131817    if (auto mediaSessionManager = PlatformMediaSessionManager::sharedManagerIfExists()) {
    18141818        if (!mediaSessionManager->isInterrupted())
    1815             MediaStreamTrack::updateCaptureAccordingToMutedState(*this);
     1819            MediaStreamTrack::updateCaptureAccordingToMutedState(*this, page->mutedState());
    18161820    }
    18171821#endif
    … …  
    42994303
    43004304#if ENABLE(MEDIA_STREAM)
    4301     MediaStreamTrack::updateCaptureAccordingToMutedState(*this);
     4305    MediaStreamTrack::updateCaptureAccordingToMutedState(*this, page()->mutedState());
    43024306#endif
    43034307}

  • trunk/Source/WebCore/page/Page.cpp

    r270480 r271401  
    8080#include "MediaCanStartListener.h"
    8181#include "MediaRecorderProvider.h"
     82#include "MediaStreamTrack.h"
    8283#include "Navigator.h"
    8384#include "PageConfiguration.h"
    … …  
    20752076}
    20762077
     2078void Page::beginAudioCaptureInterruption()
     2079{
     2080#if ENABLE(MEDIA_STREAM)
     2081    forEachDocument([mutedState = mutedState()](auto& document) {
     2082        MediaStreamTrack::updateCaptureAccordingToMutedState(document, mutedState | MediaProducer::AudioAndVideoCaptureIsMuted);
     2083    });
     2084#endif
     2085}
     2086
     2087void Page::endAudioCaptureInterruption()
     2088{
     2089#if ENABLE(MEDIA_STREAM)
     2090    forEachDocument([mutedState = mutedState()](auto& document) {
     2091        MediaStreamTrack::updateCaptureAccordingToMutedState(document, mutedState);
     2092    });
     2093#endif
     2094}
     2095
    20772096bool Page::mediaPlaybackExists()
    20782097{

  • trunk/Source/WebCore/page/Page.h

    r270480 r271401  
    832832    void setTextInteractionEnabled(bool value) { m_textInteractionEnabled = value; }
    833833
     834    WEBCORE_EXPORT void beginAudioCaptureInterruption();
     835    WEBCORE_EXPORT void endAudioCaptureInterruption();
     836
    834837private:
    835838    struct Navigation {

  • trunk/Source/WebCore/platform/mediastream/mac/BaseAudioSharedUnit.cpp

    r267838 r271401  
    184184        reconfigure();
    185185    }
    186 
    187     ASSERT(!m_producingCount);
    188 
    189     forEachClient([](auto& client) {
    190         client.setMuted(false);
    191     });
    192 
    193186    return 0;
    194187}
    … …  
    202195    m_suspended = true;
    203196    stopInternal();
    204 
    205     forEachClient([](auto& client) {
    206         client.setMuted(true);
    207     });
    208 
    209     ASSERT(!m_producingCount);
    210 
    211197    return 0;
    212198}

  • trunk/Source/WebCore/testing/Internals.cpp

    r270919 r271401  
    52745274void Internals::setMediaStreamSourceInterrupted(MediaStreamTrack& track, bool interrupted)
    52755275{
     5276    auto* document = contextDocument();
     5277    auto* page = document ? document->page() : nullptr;
     5278    if (!page)
     5279        return;
     5280
    52765281    track.source().setInterruptedForTesting(interrupted);
     5282    if (interrupted)
     5283        page->beginAudioCaptureInterruption();
     5284    else
     5285        page->endAudioCaptureInterruption();
    52775286}
    52785287

  • trunk/Source/WebKit/ChangeLog

    r271396 r271401  
     12021-01-12  Youenn Fablet  <youenn@apple.com>
     2
     3        Unmuting a track in case of end of interruption by another web process tab is not working well with capture muting icons
     4        https://bugs.webkit.org/show_bug.cgi?id=220058
     5
     6        Reviewed by Eric Carlson.
     7
     8        Add an observer of AudioSession interruptions that forwards the signal to all tabs of the process.
     9
     10        * WebProcess/cocoa/WebProcessCocoa.mm:
     11        (WebKit::WebProcess::platformInitializeWebProcess):
     12        (WebKit::PageAudioSessionInterruptionObserver::PageAudioSessionInterruptionObserver):
     13        (WebKit::PageAudioSessionInterruptionObserver::~PageAudioSessionInterruptionObserver):
     14        (WebKit::PageAudioSessionInterruptionObserver::beginAudioSessionInterruption):
     15        (WebKit::PageAudioSessionInterruptionObserver::endAudioSessionInterruption):
     16        (WebKit::listenToAudioSessionInterruption):
     17
    1182021-01-12  Philippe Normand  <pnormand@igalia.com>
    219

  • trunk/Source/WebKit/WebProcess/WebProcess.cpp

    r270932 r271401  
    18151815void WebProcess::revokeUserMediaDeviceSandboxExtensions(const Vector<String>& extensionIDs)
    18161816{
    1817     checkDocumentsCaptureStateConsistency(extensionIDs);
     1817    if (!MockRealtimeMediaSourceCenter::mockRealtimeMediaSourceCenterEnabled())
     1818        checkDocumentsCaptureStateConsistency(extensionIDs);
    18181819
    18191820    for (const auto& extensionID : extensionIDs) {

  • trunk/Source/WebKit/WebProcess/cocoa/WebProcessCocoa.mm

    r271294 r271401  
    5555#import <WebCore/AVAssetMIMETypeCache.h>
    5656#import <WebCore/AXObjectCache.h>
     57#import <WebCore/AudioSession.h>
    5758#import <WebCore/CPUMonitor.h>
    5859#import <WebCore/DisplayRefreshMonitorManager.h>
    … …  
    171172#endif
    172173
     174#if PLATFORM(IOS)
     175static void listenToAudioSessionInterruption();
     176#endif
     177
    173178void WebProcess::platformSetCacheModel(CacheModel)
    174179{
    … …  
    407412
    408413    WebCore::IOSurface::setMaximumSize(parameters.maximumIOSurfaceSize);
     414
     415#if PLATFORM(IOS)
     416    listenToAudioSessionInterruption();
     417#endif
    409418}
    410419
    … …  
    11781187}
    11791188
     1189#if PLATFORM(IOS)
     1190class PageAudioSessionInterruptionObserver : public AudioSession::InterruptionObserver {
     1191public:
     1192    PageAudioSessionInterruptionObserver();
     1193    ~PageAudioSessionInterruptionObserver();
     1194
     1195private:
     1196    void beginAudioSessionInterruption() final;
     1197    void endAudioSessionInterruption(WebCore::AudioSession::MayResume) final;
     1198};
     1199
     1200PageAudioSessionInterruptionObserver::PageAudioSessionInterruptionObserver()
     1201{
     1202    AudioSession::sharedSession().addInterruptionObserver(*this);
     1203}
     1204
     1205PageAudioSessionInterruptionObserver::~PageAudioSessionInterruptionObserver()
     1206{
     1207    AudioSession::sharedSession().removeInterruptionObserver(*this);
     1208}
     1209
     1210void PageAudioSessionInterruptionObserver::beginAudioSessionInterruption()
     1211{
     1212    Page::forEachPage([](auto& page) {
     1213        page.beginAudioCaptureInterruption();
     1214    });
     1215}
     1216
     1217void PageAudioSessionInterruptionObserver::endAudioSessionInterruption(AudioSession::MayResume mayResume)
     1218{
     1219    if (mayResume == AudioSession::MayResume::No)
     1220        return;
     1221    Page::forEachPage([](auto& page) {
     1222        page.endAudioCaptureInterruption();
     1223    });
     1224}
     1225
     1226void listenToAudioSessionInterruption()
     1227{
     1228    static NeverDestroyed<PageAudioSessionInterruptionObserver> observer;
     1229}
     1230#endif
     1231
    11801232} // namespace WebKit
    11811233
Note: See TracChangeset for help on using the changeset viewer.