⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 291759 in webkit


Ignore:
Timestamp:
Mar 23, 2022, 12:21:33 PM (5 years ago)
Author:
jer.noble@apple.com
Message:

[iOS] WebKit app is sometimes not "Now Playing" during initial playback
​https://bugs.webkit.org/show_bug.cgi?id=236993
<rdar://88827167>

Reviewed by Eric Carlson.

Source/WebCore:

Test: media/audio-session-category-at-most-recent-playback.html

Recently, we have added the concept of "preparingToPlay" to PlatformMediaSession to allow
the correct category to be set if updateSessionState() is called after playback is allowed
by the MediaSessionManager, but before playback is actually started by the media element.
However, this depends on updateSessionState() being called synchronously during playback.
We disabled this synchronous update in r269077 due to the large runtime cost when a large
number of media elements are created (but not used) at once.

Relax this asynchronous update in the case where the state is moving to "Playing", which
ensures that the correct AudioSessionCategory is set before playback starts, rather than
immediately afterward.

To support testing that the category was correctly set before playback started, add an
ivar to HTMLMediaElement that is set to the current AudioSessionCategory immediately before
the media element instructs the MediaPlayer to start playback. Expose this ivar to Internals.

Drive-by fixes: AudioSession::CategoryType cannot be forward declared, as it is a public
member of a class. Allow the enum to be forward declared by moving the declaration outside
the class, but allow current uses of the enum to continue by typedefing it inside the class
to the original enum name. Add an IDL enumeration matching the AudioSession one in Interals.idl
and convert the existing audioSessionCategory() call to use the new enumeration.

(Unforunately in the case where USE_AUDIO_SESSION is not set, the enumeration must be re-
declared. This can be removed and the entire implementation wrapped in a USE() check, once
the bindings generator is extended to allow "Conditional="-style attributes for USE checks.)

The added test is flakey due to a previous change which would keep the MediaSession category
in "MediaPlayback" for up to 2s after playback ends. To counteract this flakiness, reset the
state of the PlatformMediaSessionManager between tests.

  • html/HTMLMediaElement.cpp:

(WebCore::m_categoryAtMostRecentPlayback):
(WebCore::HTMLMediaElement::playPlayer):
(WebCore::m_logIdentifier): Deleted.

  • html/HTMLMediaElement.h:

(WebCore::HTMLMediaElement::categoryAtMostRecentPlayback const):

  • platform/audio/AudioSession.h:
  • platform/audio/PlatformMediaSessionManager.cpp:

(WebCore::PlatformMediaSessionManager::sessionStateChanged):

  • platform/audio/PlatformMediaSessionManager.h:

(WebCore::PlatformMediaSessionManager::resetSessionState):

  • platform/audio/cocoa/MediaSessionManagerCocoa.h:
  • platform/audio/cocoa/MediaSessionManagerCocoa.mm:

(WebCore::MediaSessionManagerCocoa::resetSessionState):

  • testing/Internals.cpp:

(WebCore::Internals::resetToConsistentState):

  • testing/Internals.cpp:

(WebCore::Internals::audioSessionCategory const):
(WebCore::Internals::categoryAtMostRecentPlayback const):

  • testing/Internals.h:
  • testing/Internals.idl:

LayoutTests:

Fix the audio-session-category test. Creating the oscilator actually does change the
AudioSession category to "Ambient", but (previously) only in the next run loop, so a
synchronous test that the AudioSession category is "None" succeeds. Now that starting
playback (as happens when creating an audio node) changes the AudioSession synchronously,
the test begins failing.

  • media/audio-session-category-at-most-recent-playback-expected.txt: Added.
  • media/audio-session-category-at-most-recent-playback.html: Added.
  • media/audio-session-category-expected.txt:
  • media/audio-session-category.html:
Location:
trunk
Files:
2 added
16 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r291753 r291759  
     12022-03-23  Jer Noble  <jer.noble@apple.com>
     2
     3        [iOS] WebKit app is sometimes not "Now Playing" during initial playback
     4        https://bugs.webkit.org/show_bug.cgi?id=236993
     5        <rdar://88827167>
     6
     7        Reviewed by Eric Carlson.
     8
     9        Fix the audio-session-category test. Creating the oscilator actually does change the
     10        AudioSession category to "Ambient", but (previously) only in the next run loop, so a
     11        synchronous test that the AudioSession category is "None" succeeds. Now that starting
     12        playback (as happens when creating an audio node) changes the AudioSession synchronously,
     13        the test begins failing.
     14
     15        * media/audio-session-category-at-most-recent-playback-expected.txt: Added.
     16        * media/audio-session-category-at-most-recent-playback.html: Added.
     17        * media/audio-session-category-expected.txt:
     18        * media/audio-session-category.html:
     19
    1202022-03-23  Matteo Flores  <matteo_flores@apple.com>
    221

  • trunk/LayoutTests/media/audio-session-category-expected.txt

    r288466 r291759  
    3535EXPECTED (internals.audioSessionCategory() == 'None') OK
    3636
    37 ** Check category after oscillator graph has been connected but not started.
     37** Check category after AudioContext has been created but not started.
    3838EXPECTED (internals.audioSessionCategory() == 'None') OK
    3939

  • trunk/LayoutTests/media/audio-session-category.html

    r288466 r291759  
    5959            await waitForCategory('None', 10, '<br>** Check category before creating AudioContext.');
    6060
    61             consoleWrite('<br>** Check category after oscillator graph has been connected but not started.');
     61            consoleWrite('<br>** Check category after AudioContext has been created but not started.');
    6262            let context = new AudioContext();
     63            testExpected('internals.audioSessionCategory()', 'None');
     64
    6365            let oscillator = null;
    6466            let gainNode = context.createGain();
    65 
    6667            oscillator = context.createOscillator();
    6768            oscillator.type = 'square';
    … …  
    7172            gainNode.connect(context.destination);
    7273            gainNode.gain.value = 0.1
    73             testExpected('internals.audioSessionCategory()', 'None');
    7474
    7575            consoleWrite('<br>** Check category after starting oscillator.');
     76            context.resume();
    7677            oscillator.start(0);
    7778            await sleepFor(500);

  • trunk/LayoutTests/platform/glib/TestExpectations

    r291747 r291759  
    18771877# Tests behavior specific to MediaSessionManagerCocoa
    18781878media/audio-session-category.html [ Skip ]
     1879media/audio-session-category-at-most-recent-playback.html [ Skip ]
    18791880
    18801881# This test assumes we cannot play RTSP, but we can.

  • trunk/LayoutTests/platform/mac-wk1/TestExpectations

    r291737 r291759  
    6666http/tests/navigation/page-cache-mediastream.html [ Skip ]
    6767media/audio-session-category.html [ Skip ]
     68media/audio-session-category-at-most-recent-playback.html [ Skip ]
    6869
    6970# Shared workers are only implemented for WebKit2.

  • trunk/Source/WebCore/ChangeLog

    r291754 r291759  
     12022-03-23  Jer Noble  <jer.noble@apple.com>
     2
     3        [iOS] WebKit app is sometimes not "Now Playing" during initial playback
     4        https://bugs.webkit.org/show_bug.cgi?id=236993
     5        <rdar://88827167>
     6
     7        Reviewed by Eric Carlson.
     8
     9        Test: media/audio-session-category-at-most-recent-playback.html
     10
     11        Recently, we have added the concept of "preparingToPlay" to PlatformMediaSession to allow
     12        the correct category to be set if updateSessionState() is called after playback is allowed
     13        by the MediaSessionManager, but before playback is actually started by the media element.
     14        However, this depends on updateSessionState() being called synchronously during playback.
     15        We disabled this synchronous update in r269077 due to the large runtime cost when a large
     16        number of media elements are created (but not used) at once.
     17
     18        Relax this asynchronous update in the case where the state is moving to "Playing", which
     19        ensures that the correct AudioSessionCategory is set before playback starts, rather than
     20        immediately afterward.
     21
     22        To support testing that the category was correctly set before playback started, add an
     23        ivar to HTMLMediaElement that is set to the current AudioSessionCategory immediately before
     24        the media element instructs the MediaPlayer to start playback. Expose this ivar to Internals.
     25
     26        Drive-by fixes: AudioSession::CategoryType cannot be forward declared, as it is a public
     27        member of a class. Allow the enum to be forward declared by moving the declaration outside
     28        the class, but allow current uses of the enum to continue by typedefing it inside the class
     29        to the original enum name. Add an IDL enumeration matching the AudioSession one in Interals.idl
     30        and convert the existing audioSessionCategory() call to use the new enumeration.
     31
     32        (Unforunately in the case where USE_AUDIO_SESSION is not set, the enumeration must be re-
     33        declared. This can be removed and the entire implementation wrapped in a USE() check, once
     34        the bindings generator is extended to allow "Conditional="-style attributes for USE checks.)
     35
     36        The added test is flakey due to a previous change which would keep the MediaSession category
     37        in "MediaPlayback" for up to 2s after playback ends. To counteract this flakiness, reset the
     38        state of the PlatformMediaSessionManager between tests.
     39
     40        * html/HTMLMediaElement.cpp:
     41        (WebCore::m_categoryAtMostRecentPlayback):
     42        (WebCore::HTMLMediaElement::playPlayer):
     43        (WebCore::m_logIdentifier): Deleted.
     44        * html/HTMLMediaElement.h:
     45        (WebCore::HTMLMediaElement::categoryAtMostRecentPlayback const):
     46        * platform/audio/AudioSession.h:
     47        * platform/audio/PlatformMediaSessionManager.cpp:
     48        (WebCore::PlatformMediaSessionManager::sessionStateChanged):
     49        * platform/audio/PlatformMediaSessionManager.h:
     50        (WebCore::PlatformMediaSessionManager::resetSessionState):
     51        * platform/audio/cocoa/MediaSessionManagerCocoa.h:
     52        * platform/audio/cocoa/MediaSessionManagerCocoa.mm:
     53        (WebCore::MediaSessionManagerCocoa::resetSessionState):
     54        * testing/Internals.cpp:
     55        (WebCore::Internals::resetToConsistentState):
     56        * testing/Internals.cpp:
     57        (WebCore::Internals::audioSessionCategory const):
     58        (WebCore::Internals::categoryAtMostRecentPlayback const):
     59        * testing/Internals.h:
     60        * testing/Internals.idl:
     61
    1622022-03-22  Brandon Stewart  <brandonstewart@apple.com>
    263

  • trunk/Source/WebCore/html/HTMLMediaElement.cpp

    r291296 r291759  
    131131#include <wtf/text/CString.h>
    132132
     133#if USE(AUDIO_SESSION)
     134#include "AudioSession.h"
     135#endif
     136
    133137#if ENABLE(WEB_AUDIO)
    134138#include "AudioSourceProvider.h"
    … …  
    459463    , m_logIdentifier(uniqueLogIdentifier())
    460464#endif
     465#if USE(AUDIO_SESSION)
     466    , m_categoryAtMostRecentPlayback(AudioSessionCategory::None)
     467#endif
    461468{
    462469    allMediaElements().add(this);
    … …  
    56955702        return;
    56965703
     5704#if USE(AUDIO_SESSION)
     5705    m_categoryAtMostRecentPlayback = AudioSession::sharedSession().category();
     5706#endif
     5707
    56975708#if ENABLE(MEDIA_SESSION) && ENABLE(MEDIA_SESSION_COORDINATOR)
    56985709    do {

  • trunk/Source/WebCore/html/HTMLMediaElement.h

    r291296 r291759  
    606606    WEBCORE_EXPORT bool mediaPlayerRenderingCanBeAccelerated() final;
    607607
     608#if USE(AUDIO_SESSION)
     609    WEBCORE_EXPORT AudioSessionCategory categoryAtMostRecentPlayback() const { return m_categoryAtMostRecentPlayback; }
     610#endif
     611
    608612protected:
    609613    HTMLMediaElement(const QualifiedName&, Document&, bool createdByParser);
    … …  
    12401244#endif
    12411245    String m_id;
     1246
     1247#if USE(AUDIO_SESSION)
     1248    AudioSessionCategory m_categoryAtMostRecentPlayback;
     1249#endif
    12421250};
    12431251

  • trunk/Source/WebCore/platform/audio/AudioSession.h

    r291268 r291759  
    4747};
    4848
     49enum class AudioSessionCategory : uint8_t {
     50    None,
     51    AmbientSound,
     52    SoloAmbientSound,
     53    MediaPlayback,
     54    RecordAudio,
     55    PlayAndRecord,
     56    AudioProcessing,
     57};
     58
    4959class AudioSessionRoutingArbitrationClient;
    5060
    … …  
    6474    virtual ~AudioSession();
    6575
    66     enum class CategoryType : uint8_t {
    67         None,
    68         AmbientSound,
    69         SoloAmbientSound,
    70         MediaPlayback,
    71         RecordAudio,
    72         PlayAndRecord,
    73         AudioProcessing,
    74     };
     76    using CategoryType = AudioSessionCategory;
    7577    virtual void setCategory(CategoryType, RouteSharingPolicy);
    7678    virtual CategoryType category() const;

  • trunk/Source/WebCore/platform/audio/PlatformMediaSessionManager.cpp

    r284093 r291759  
    291291}
    292292
    293 void PlatformMediaSessionManager::sessionStateChanged(PlatformMediaSession&)
    294 {
    295     scheduleUpdateSessionState();
     293void PlatformMediaSessionManager::sessionStateChanged(PlatformMediaSession& session)
     294{
     295    // Call updateSessionState() synchronously if the new state is Playing to ensure
     296    // the audio session is active and has the correct category before playback starts.
     297    if (session.state() == PlatformMediaSession::Playing)
     298        updateSessionState();
     299    else
     300        scheduleUpdateSessionState();
    296301}
    297302

  • trunk/Source/WebCore/platform/audio/PlatformMediaSessionManager.h

    r291390 r291759  
    169169
    170170    virtual void resetHaveEverRegisteredAsNowPlayingApplicationForTesting() { };
     171    virtual void resetSessionState() { };
    171172
    172173    bool isApplicationInBackground() const { return m_isApplicationInBackground; }

  • trunk/Source/WebCore/platform/audio/cocoa/MediaSessionManagerCocoa.h

    r290621 r291759  
    100100
    101101    void resetHaveEverRegisteredAsNowPlayingApplicationForTesting() final { m_haveEverRegisteredAsNowPlayingApplication = false; };
     102    void resetSessionState() final;
    102103
    103104private:

  • trunk/Source/WebCore/platform/audio/cocoa/MediaSessionManagerCocoa.mm

    r290621 r291759  
    205205}
    206206
     207void MediaSessionManagerCocoa::resetSessionState()
     208{
     209    ALWAYS_LOG(LOGIDENTIFIER);
     210    m_delayCategoryChangeTimer.stop();
     211    m_previousCategory = AudioSession::CategoryType::None;
     212    m_previousHadAudibleAudioOrVideoMediaType = false;
     213}
     214
    207215void MediaSessionManagerCocoa::beginInterruption(PlatformMediaSession::InterruptionType type)
    208216{

  • trunk/Source/WebCore/testing/Internals.cpp

    r291611 r291759  
    555555    PlatformMediaSessionManager::sharedManager().resetHaveEverRegisteredAsNowPlayingApplicationForTesting();
    556556    PlatformMediaSessionManager::sharedManager().resetRestrictions();
     557    PlatformMediaSessionManager::sharedManager().resetSessionState();
    557558    PlatformMediaSessionManager::sharedManager().setWillIgnoreSystemInterruptions(true);
    558559#endif
    … …  
    56595660}
    56605661
    5661 String Internals::audioSessionCategory() const
     5662auto Internals::audioSessionCategory() const -> AudioSessionCategory
    56625663{
    56635664#if USE(AUDIO_SESSION)
    5664     switch (AudioSession::sharedSession().category()) {
    5665     case AudioSession::CategoryType::AmbientSound:
    5666         return "AmbientSound"_s;
    5667     case AudioSession::CategoryType::SoloAmbientSound:
    5668         return "SoloAmbientSound"_s;
    5669     case AudioSession::CategoryType::MediaPlayback:
    5670         return "MediaPlayback"_s;
    5671     case AudioSession::CategoryType::RecordAudio:
    5672         return "RecordAudio"_s;
    5673     case AudioSession::CategoryType::PlayAndRecord:
    5674         return "PlayAndRecord"_s;
    5675     case AudioSession::CategoryType::AudioProcessing:
    5676         return "AudioProcessing"_s;
    5677     case AudioSession::CategoryType::None:
    5678         return "None"_s;
    5679     }
    5680 #endif
    5681     return emptyString();
    5682 }
     5665    return AudioSession::sharedSession().category();
     5666#else
     5667    return AudioSessionCategory::None;
     5668#endif
     5669}
     5670
     5671#if ENABLE(VIDEO)
     5672auto Internals::categoryAtMostRecentPlayback(HTMLMediaElement& element) const -> AudioSessionCategory
     5673{
     5674#if USE(AUDIO_SESSION)
     5675    return element.categoryAtMostRecentPlayback();
     5676#else
     5677    return AudioSessionCategory::None;
     5678#endif
     5679}
     5680#endif
    56835681
    56845682double Internals::preferredAudioBufferSize() const

  • trunk/Source/WebCore/testing/Internals.h

    r291611 r291759  
    4646#include "MediaElementSession.h"
    4747#include "MediaUniqueIdentifier.h"
     48#endif
     49
     50#if USE(AUDIO_SESSION)
     51#include "AudioSession.h"
    4852#endif
    4953
    … …  
    890894#endif
    891895
     896
     897#if USE(AUDIO_SESSION)
     898    using AudioSessionCategory = WebCore::AudioSessionCategory;
     899#else
     900    enum class AudioSessionCategory : uint8_t {
     901        None,
     902        AmbientSound,
     903        SoloAmbientSound,
     904        MediaPlayback,
     905        RecordAudio,
     906        PlayAndRecord,
     907        AudioProcessing,
     908    };
     909#endif
     910
    892911    bool supportsAudioSession() const;
    893     String audioSessionCategory() const;
     912    AudioSessionCategory audioSessionCategory() const;
     913#if ENABLE(VIDEO)
     914    AudioSessionCategory categoryAtMostRecentPlayback(HTMLMediaElement&) const;
     915#endif
    894916    double preferredAudioBufferSize() const;
    895917    double currentAudioBufferSize() const;

  • trunk/Source/WebCore/testing/Internals.idl

    r291517 r291759  
    104104    "GPUStatusFailure",
    105105    "Timeout"
     106};
     107
     108enum AudioSessionCategory {
     109    "None",
     110    "AmbientSound",
     111    "SoloAmbientSound",
     112    "MediaPlayback",
     113    "RecordAudio",
     114    "PlayAndRecord",
     115    "AudioProcessing"
    106116};
    107117
    … …  
    967977
    968978    readonly attribute boolean supportsAudioSession;
    969     DOMString audioSessionCategory();
     979    AudioSessionCategory audioSessionCategory();
     980    [Conditional=VIDEO] AudioSessionCategory categoryAtMostRecentPlayback(HTMLMediaElement element);
     981
    970982    double preferredAudioBufferSize();
    971983    double currentAudioBufferSize();
Note: See TracChangeset for help on using the changeset viewer.