Changeset 281880 in webkit


Ignore:
Timestamp:
Sep 1, 2021 2:49:04 PM (11 months ago)
Author:
eric.carlson@apple.com
Message:

[macOS] getDisplayMedia should capture at the constrained size if possible
​https://bugs.webkit.org/show_bug.cgi?id=229357
<rdar://problem/82191109>

Reviewed by Youenn Fablet.

Source/WebCore:

Create display stream with preferred width, height, and frame rate.

Test: fast/mediastream/get-display-media-capabilities.html

  • platform/mediastream/mac/CGDisplayStreamCaptureSource.cpp:

(WebCore::CGDisplayStreamCaptureSource::start): Remove frame rate parameter.
(WebCore::CGDisplayStreamCaptureSource::startDisplayStream): Ditto.
(WebCore::CGDisplayStreamCaptureSource::commitConfiguration): Pass source settings
instead of frame rate. Release and recreate the display stream if width, height,
or frame rate has changed.

  • platform/mediastream/mac/CGDisplayStreamCaptureSource.h:

(WebCore::CGDisplayStreamCaptureSource::width const):
(WebCore::CGDisplayStreamCaptureSource::height const):
(WebCore::CGDisplayStreamCaptureSource::frameRate const):

  • platform/mediastream/mac/CGDisplayStreamScreenCaptureSource.h:
  • platform/mediastream/mac/CGDisplayStreamScreenCaptureSource.mm:

(WebCore::CGDisplayStreamScreenCaptureSource::createDisplayStream): Remove
frame rate parameter. Create display stream with the configured width and height,
not the size of the screen.
(WebCore::CGDisplayStreamScreenCaptureSource::intrinsicSize const): Return screen
size so track capabilities are accurate.

  • platform/mediastream/mac/CGWindowCaptureSource.h:
  • platform/mediastream/mac/CGWindowCaptureSource.mm:

(WebCore::CGWindowCaptureSource::create): Remove frame rate parameter.
(WebCore::CGWindowCaptureSource::intrinsicSize const): Return window size.

  • platform/mediastream/mac/DisplayCaptureSourceMac.cpp:

(WebCore::DisplayCaptureSourceMac::capabilities): Use capturer intrinsic size for
width and height capabilities.

  • platform/mediastream/mac/DisplayCaptureSourceMac.h:
  • platform/mock/MockRealtimeMediaSourceCenter.cpp: Change the mock screen sizes to

make the first one is different than the old hard-coded sizes so we're able to detect
that it is used in a test.
(WebCore::MockDisplayCapturer::start): Remove frame rate parameter.
(WebCore::MockDisplayCapturer::intrinsicSize const):

LayoutTests:

  • fast/mediastream/get-display-media-capabilities-expected.txt: Added.
  • fast/mediastream/get-display-media-capabilities.html: Added.
Location:
trunk
Files:
2 added
13 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r281876 r281880  
     12021-09-01  Eric Carlson  <eric.carlson@apple.com>
     2
     3        [macOS] getDisplayMedia should capture at the constrained size if possible
     4        https://bugs.webkit.org/show_bug.cgi?id=229357
     5        <rdar://problem/82191109>
     6
     7        Reviewed by Youenn Fablet.
     8
     9        * fast/mediastream/get-display-media-capabilities-expected.txt: Added.
     10        * fast/mediastream/get-display-media-capabilities.html: Added.
     11
    1122021-09-01  Ayumi Kojima  <ayumi_kojima@apple.com>
    213

  • trunk/Source/WebCore/ChangeLog

    r281879 r281880  
     12021-09-01  Eric Carlson  <eric.carlson@apple.com>
     2
     3        [macOS] getDisplayMedia should capture at the constrained size if possible
     4        https://bugs.webkit.org/show_bug.cgi?id=229357
     5        <rdar://problem/82191109>
     6
     7        Reviewed by Youenn Fablet.
     8
     9        Create display stream with preferred width, height, and frame rate.
     10
     11        Test: fast/mediastream/get-display-media-capabilities.html
     12
     13        * platform/mediastream/mac/CGDisplayStreamCaptureSource.cpp:
     14        (WebCore::CGDisplayStreamCaptureSource::start): Remove frame rate parameter.
     15        (WebCore::CGDisplayStreamCaptureSource::startDisplayStream): Ditto.
     16        (WebCore::CGDisplayStreamCaptureSource::commitConfiguration): Pass source settings
     17        instead of frame rate. Release and recreate the display stream if width, height,
     18        or frame rate has changed.
     19        * platform/mediastream/mac/CGDisplayStreamCaptureSource.h:
     20        (WebCore::CGDisplayStreamCaptureSource::width const):
     21        (WebCore::CGDisplayStreamCaptureSource::height const):
     22        (WebCore::CGDisplayStreamCaptureSource::frameRate const):
     23
     24        * platform/mediastream/mac/CGDisplayStreamScreenCaptureSource.h:
     25        * platform/mediastream/mac/CGDisplayStreamScreenCaptureSource.mm:
     26        (WebCore::CGDisplayStreamScreenCaptureSource::createDisplayStream): Remove
     27        frame rate parameter. Create display stream with the configured width and height,
     28        not the size of the screen.
     29        (WebCore::CGDisplayStreamScreenCaptureSource::intrinsicSize const): Return screen
     30        size so track capabilities are accurate.
     31
     32        * platform/mediastream/mac/CGWindowCaptureSource.h:
     33        * platform/mediastream/mac/CGWindowCaptureSource.mm:
     34        (WebCore::CGWindowCaptureSource::create): Remove frame rate parameter.
     35        (WebCore::CGWindowCaptureSource::intrinsicSize const): Return window size.
     36
     37        * platform/mediastream/mac/DisplayCaptureSourceMac.cpp:
     38        (WebCore::DisplayCaptureSourceMac::capabilities): Use capturer intrinsic size for
     39        width and height capabilities.
     40
     41        * platform/mediastream/mac/DisplayCaptureSourceMac.h:
     42        * platform/mock/MockRealtimeMediaSourceCenter.cpp: Change the mock screen sizes to
     43        make the first one is different than the old hard-coded sizes so we're able to detect
     44        that it is used in a test.
     45        (WebCore::MockDisplayCapturer::start): Remove frame rate parameter.
     46        (WebCore::MockDisplayCapturer::intrinsicSize const):
     47
    1482021-09-01  Chris Dumez  <cdumez@apple.com>
    249

  • trunk/Source/WebCore/platform/mediastream/RealtimeMediaSource.cpp

    r279940 r281880  
    998998}
    999999
    1000 const IntSize RealtimeMediaSource::intrinsicSize() const
     1000IntSize RealtimeMediaSource::intrinsicSize() const
    10011001{
    10021002    return m_intrinsicSize;

  • trunk/Source/WebCore/platform/mediastream/RealtimeMediaSource.h

    r279940 r281880  
    150150    void setSize(const IntSize&);
    151151
    152     const IntSize intrinsicSize() const;
     152    IntSize intrinsicSize() const;
    153153    void setIntrinsicSize(const IntSize&, bool notifyObservers = true);
    154154

  • trunk/Source/WebCore/platform/mediastream/mac/CGDisplayStreamCaptureSource.cpp

    r281852 r281880  
    4949}
    5050
    51 bool CGDisplayStreamCaptureSource::start(float frameRate)
     51bool CGDisplayStreamCaptureSource::start()
    5252{
    5353    ALWAYS_LOG_IF(loggerPtr(), LOGIDENTIFIER);
    … …  
    5656        return true;
    5757
    58     return startDisplayStream(frameRate);
     58    return startDisplayStream();
    5959}
    6060
    … …  
    7777}
    7878
    79 bool CGDisplayStreamCaptureSource::startDisplayStream(float frameRate)
     79bool CGDisplayStreamCaptureSource::startDisplayStream()
    8080{
    8181    if (!checkDisplayStream())
    … …  
    8686
    8787    if (!m_displayStream) {
    88         m_displayStream = createDisplayStream(frameRate, frameAvailableHandler(), m_captureQueue.get());
     88        m_displayStream = createDisplayStream(frameAvailableHandler(), m_captureQueue.get());
    8989        if (!m_displayStream)
    9090            return false;
    … …  
    106106}
    107107
    108 void CGDisplayStreamCaptureSource::commitConfiguration(float frameRate)
     108void CGDisplayStreamCaptureSource::commitConfiguration(const RealtimeMediaSourceSettings& settings)
    109109{
    110     if (m_isRunning && !m_displayStream)
    111         startDisplayStream(frameRate);
     110    if (m_width == settings.width() && m_height == settings.height() && m_frameRate == settings.frameRate())
     111        return;
     112
     113    m_width = settings.width();
     114    m_height = settings.height();
     115    m_frameRate = settings.frameRate();
     116
     117    if (m_displayStream) {
     118        CGDisplayStreamStop(m_displayStream.get());
     119        m_displayStream = nullptr;
     120    }
     121
     122    if (m_isRunning)
     123        startDisplayStream();
    112124}
    113125

  • trunk/Source/WebCore/platform/mediastream/mac/CGDisplayStreamCaptureSource.h

    r281852 r281880  
    4848protected:
    4949    using FrameAvailableCallback = void (^)(CGDisplayStreamFrameStatus, uint64_t, IOSurfaceRef, CGDisplayStreamUpdateRef);
    50     virtual RetainPtr<CGDisplayStreamRef> createDisplayStream(float, FrameAvailableCallback, dispatch_queue_t) = 0;
     50    virtual RetainPtr<CGDisplayStreamRef> createDisplayStream(FrameAvailableCallback, dispatch_queue_t) = 0;
    5151    virtual bool checkDisplayStream() { return true; }
    5252
    … …  
    5454    void invalidateDisplayStream() { m_displayStream = nullptr; }
    5555
     56    uint32_t width() const { return m_width; }
     57    uint32_t height() const { return m_height; }
     58    float frameRate() const { return m_frameRate; }
     59
    5660private:
    5761    static void displayReconfigurationCallBack(CGDirectDisplayID, CGDisplayChangeSummaryFlags, void*);
    5862
    5963    // DisplayCaptureSourceMac::Capturer
    60     bool start(float frameRate) final;
     64    bool start() final;
    6165    void stop() final;
    6266    DisplayCaptureSourceMac::DisplayFrameType generateFrame() final;
    63     void commitConfiguration(float frameRate) final;
     67    void commitConfiguration(const RealtimeMediaSourceSettings&) final;
    6468
    6569    void displayWasReconfigured(CGDirectDisplayID, CGDisplayChangeSummaryFlags);
    66     bool startDisplayStream(float frameRate);
     70    bool startDisplayStream();
    6771    FrameAvailableCallback frameAvailableHandler();
    6872
    … …  
    106110    BlockPtr<void(CGDisplayStreamFrameStatus, uint64_t, IOSurfaceRef, CGDisplayStreamUpdateRef)> m_frameAvailableHandler;
    107111
     112    uint32_t m_width { 0 };
     113    uint32_t m_height { 0 };
     114    float m_frameRate { 0 };
     115
    108116    bool m_isRunning { false };
    109117    bool m_observingDisplayChanges { false };

  • trunk/Source/WebCore/platform/mediastream/mac/CGDisplayStreamScreenCaptureSource.h

    r281852 r281880  
    5555    CaptureDevice::DeviceType deviceType() const final { return CaptureDevice::DeviceType::Screen; }
    5656    RealtimeMediaSourceSettings::DisplaySurfaceType surfaceType() const final { return RealtimeMediaSourceSettings::DisplaySurfaceType::Monitor; }
     57    IntSize intrinsicSize() const final;
    5758#if !RELEASE_LOG_DISABLED
    5859    const char* logClassName() const final { return "CGDisplayStreamScreenCaptureSource"; }
    … …  
    6061
    6162    // CGDisplayStreamCaptureSource
    62     RetainPtr<CGDisplayStreamRef> createDisplayStream(float, FrameAvailableCallback, dispatch_queue_t) final;
     63    RetainPtr<CGDisplayStreamRef> createDisplayStream(FrameAvailableCallback, dispatch_queue_t) final;
    6364    bool checkDisplayStream() final;
    6465

  • trunk/Source/WebCore/platform/mediastream/mac/CGDisplayStreamScreenCaptureSource.mm

    r281852 r281880  
    112112}
    113113
    114 RetainPtr<CGDisplayStreamRef> CGDisplayStreamScreenCaptureSource::createDisplayStream(float frameRate, FrameAvailableCallback frameAvailableHandler, dispatch_queue_t queue)
     114RetainPtr<CGDisplayStreamRef> CGDisplayStreamScreenCaptureSource::createDisplayStream(FrameAvailableCallback frameAvailableHandler, dispatch_queue_t queue)
    115115{
    116116    static const int screenQueueMaximumLength = 6;
    … …  
    119119    ASSERT(m_displayID == updateDisplayID(m_displayID));
    120120
    121     ALWAYS_LOG_IF(loggerPtr(), LOGIDENTIFIER);
    122 
    123     auto displayMode = adoptCF(CGDisplayCopyDisplayMode(m_displayID));
    124     auto screenWidth = CGDisplayModeGetPixelsWide(displayMode.get());
    125     auto screenHeight = CGDisplayModeGetPixelsHigh(displayMode.get());
    126     if (!screenWidth || !screenHeight) {
    127         ERROR_LOG_IF(loggerPtr(), LOGIDENTIFIER, "unable to get screen width/height");
    128         return nullptr;
    129     }
     121    ALWAYS_LOG_IF(loggerPtr(), LOGIDENTIFIER, "frame rate ", frameRate(), ", size ", width(), "x", height());
    130122
    131123    NSDictionary* streamOptions = @{
    132         (__bridge NSString *)kCGDisplayStreamMinimumFrameTime : @(1 / frameRate),
     124        (__bridge NSString *)kCGDisplayStreamMinimumFrameTime : @(1 / frameRate()),
    133125        (__bridge NSString *)kCGDisplayStreamQueueDepth : @(screenQueueMaximumLength),
    134126        (__bridge NSString *)kCGDisplayStreamColorSpace : (__bridge id)sRGBColorSpaceRef(),
    … …  
    136128    };
    137129
    138     return adoptCF(CGDisplayStreamCreateWithDispatchQueue(m_displayID, screenWidth, screenHeight, preferedPixelBufferFormat(), (__bridge CFDictionaryRef)streamOptions, queue, frameAvailableHandler));
     130    return adoptCF(CGDisplayStreamCreateWithDispatchQueue(m_displayID, width(), height(), preferedPixelBufferFormat(), (__bridge CFDictionaryRef)streamOptions, queue, frameAvailableHandler));
     131}
     132
     133IntSize CGDisplayStreamScreenCaptureSource::intrinsicSize() const
     134{
     135    auto displayMode = adoptCF(CGDisplayCopyDisplayMode(m_displayID));
     136    auto screenWidth = CGDisplayModeGetPixelsWide(displayMode.get());
     137    auto screenHeight = CGDisplayModeGetPixelsHigh(displayMode.get());
     138
     139    return { Checked<int>(screenWidth), Checked<int>(screenHeight) };
    139140}
    140141

  • trunk/Source/WebCore/platform/mediastream/mac/CGWindowCaptureSource.h

    r281852 r281880  
    5151private:
    5252    // DisplayCaptureSourceMac::Capturer
    53     bool start(float) final { return true; }
     53    bool start() final { return true; }
    5454    void stop() final { }
    5555    DisplayCaptureSourceMac::DisplayFrameType generateFrame() final;
    5656    RealtimeMediaSourceSettings::DisplaySurfaceType surfaceType() const final { return RealtimeMediaSourceSettings::DisplaySurfaceType::Window; }
    5757    CaptureDevice::DeviceType deviceType() const final { return CaptureDevice::DeviceType::Window; }
    58     void commitConfiguration(float) final { }
     58    void commitConfiguration(const RealtimeMediaSourceSettings&) final { }
     59    IntSize intrinsicSize() const final;
    5960    const char* logClassName() const final { return "CGWindowCaptureSource"; }
    6061

  • trunk/Source/WebCore/platform/mediastream/mac/CGWindowCaptureSource.mm

    r281852 r281880  
    104104Expected<UniqueRef<DisplayCaptureSourceMac::Capturer>, String> CGWindowCaptureSource::create(const String& deviceID)
    105105{
    106     auto displayID = parseInteger<uint32_t>(deviceID);
    107     if (!displayID)
     106    auto windowID = parseInteger<uint32_t>(deviceID);
     107    if (!windowID)
    108108        return makeUnexpected("Invalid window device ID"_s);
    109109
    110     auto windowInfo = windowDescription(*displayID);
     110    auto windowInfo = windowDescription(*windowID);
    111111    if (!windowInfo)
    112112        return makeUnexpected("Invalid window ID"_s);
    113113
    114     return UniqueRef<DisplayCaptureSourceMac::Capturer>(makeUniqueRef<CGWindowCaptureSource>(*displayID));
     114    return UniqueRef<DisplayCaptureSourceMac::Capturer>(makeUniqueRef<CGWindowCaptureSource>(*windowID));
     115}
     116
     117IntSize CGWindowCaptureSource::intrinsicSize() const
     118{
     119    auto windowInfo = windowDescription(m_windowID);
     120    if (!windowInfo) {
     121        RELEASE_LOG(WebRTC, "Invalid window ID?");
     122        return { };
     123    }
     124
     125    auto boundsDictionary = checked_cf_cast<CFDictionaryRef>(CFDictionaryGetValue(windowInfo.get(), kCGWindowBounds));
     126    if (!boundsDictionary) {
     127        RELEASE_LOG(WebRTC, "Unable to get window bounds");
     128        return { };
     129    }
     130
     131    CGRect windowBounds;
     132    if (!CGRectMakeWithDictionaryRepresentation(boundsDictionary, &windowBounds)) {
     133        RELEASE_LOG(WebRTC, "Unable to decode window bounds");
     134        return { };
     135    }
     136
     137    return IntSize(windowBounds.size);
    115138}
    116139

  • trunk/Source/WebCore/platform/mediastream/mac/DisplayCaptureSourceMac.cpp

    r281852 r281880  
    104104        RealtimeMediaSourceCapabilities capabilities(settings().supportedConstraints());
    105105
    106         // FIXME: what should these be?
    107         capabilities.setWidth(CapabilityValueOrRange(1, 3840));
    108         capabilities.setHeight(CapabilityValueOrRange(1, 2160));
     106        auto intrinsicSize = m_capturer->intrinsicSize();
     107        capabilities.setWidth(CapabilityValueOrRange(1, intrinsicSize.width()));
     108        capabilities.setHeight(CapabilityValueOrRange(1, intrinsicSize.height()));
    109109        capabilities.setFrameRate(CapabilityValueOrRange(.01, 30.0));
    110110
    … …  
    146146        m_timer.startRepeating(1_s / frameRate());
    147147
    148     if (settings.containsAny({ RealtimeMediaSourceSettings::Flag::Width, RealtimeMediaSourceSettings::Flag::Height }))
    149         m_bufferAttributes = nullptr;
    150 
    151148    m_currentSettings = { };
    152149}
    … …  
    157154    m_timer.startRepeating(1_s / frameRate());
    158155
    159     if (!m_capturer->start(frameRate()))
     156    if (!m_capturer->start())
    160157        captureFailed();
    161158}

  • trunk/Source/WebCore/platform/mediastream/mac/DisplayCaptureSourceMac.h

    r281852 r281880  
    6060        virtual ~Capturer() = default;
    6161
    62         virtual bool start(float frameRate) = 0;
     62        virtual bool start() = 0;
    6363        virtual void stop() = 0;
    6464        virtual DisplayFrameType generateFrame() = 0;
    6565        virtual CaptureDevice::DeviceType deviceType() const = 0;
    6666        virtual RealtimeMediaSourceSettings::DisplaySurfaceType surfaceType() const = 0;
    67         virtual void commitConfiguration(float frameRate) = 0;
     67        virtual void commitConfiguration(const RealtimeMediaSourceSettings&) = 0;
     68        virtual IntSize intrinsicSize() const = 0;
    6869
    6970        virtual void setLogger(const Logger&, const void*);
    … …  
    9596    const RealtimeMediaSourceSettings& settings() final;
    9697    CaptureDevice::DeviceType deviceType() const { return m_capturer->deviceType(); }
    97     void commitConfiguration() final { return m_capturer->commitConfiguration(frameRate()); }
     98    void commitConfiguration() final { m_capturer->commitConfiguration(settings()); }
    9899
    99100    const char* logClassName() const final { return "DisplayCaptureSourceMac"; }
    … …  
    109110    Seconds m_elapsedTime { 0_s };
    110111
    111     RetainPtr<CFMutableDictionaryRef> m_bufferAttributes;
    112112    RunLoop::Timer<DisplayCaptureSourceMac> m_timer;
    113113

  • trunk/Source/WebCore/platform/mock/MockRealtimeMediaSourceCenter.cpp

    r281852 r281880  
    9292            } },
    9393
    94         MockMediaDevice { "SCREEN-1"_s, "Mock screen device 1"_s, MockDisplayProperties { CaptureDevice::DeviceType::Screen, Color::lightGray, { 3840, 2160 } } },
    95         MockMediaDevice { "SCREEN-2"_s, "Mock screen device 2"_s, MockDisplayProperties { CaptureDevice::DeviceType::Screen, Color::yellow, { 1920, 1080 } } },
     94        MockMediaDevice { "SCREEN-1"_s, "Mock screen device 1"_s, MockDisplayProperties { CaptureDevice::DeviceType::Screen, Color::lightGray, { 1920, 1080 } } },
     95        MockMediaDevice { "SCREEN-2"_s, "Mock screen device 2"_s, MockDisplayProperties { CaptureDevice::DeviceType::Screen, Color::yellow, { 3840, 2160 } } },
    9696
    9797        MockMediaDevice { "WINDOW-2"_s, "Mock window 1"_s, MockDisplayProperties { CaptureDevice::DeviceType::Screen, SRGBA<uint8_t> { 255, 241, 181 }, { 640, 480 } } },
    … …  
    121121
    122122private:
    123     bool start(float) final;
     123    bool start() final;
    124124    void stop() final  { m_source->stop(); }
    125125    DisplayCaptureSourceMac::DisplayFrameType generateFrame() final;
    126126    RealtimeMediaSourceSettings::DisplaySurfaceType surfaceType() const final { return RealtimeMediaSourceSettings::DisplaySurfaceType::Monitor; }
    127     void commitConfiguration(float) final { }
     127    void commitConfiguration(const RealtimeMediaSourceSettings&) final { }
    128128    CaptureDevice::DeviceType deviceType() const final { return CaptureDevice::DeviceType::Screen; }
     129    IntSize intrinsicSize() const final;
    129130#if !RELEASE_LOG_DISABLED
    130131    const char* logClassName() const final { return "MockDisplayCapturer"; }
    … …  
    139140}
    140141
    141 bool MockDisplayCapturer::start(float)
     142bool MockDisplayCapturer::start()
    142143{
    143144    m_source->start();
    … …  
    151152    return { };
    152153}
    153 #endif
     154
     155IntSize MockDisplayCapturer::intrinsicSize() const
     156{
     157    auto device = MockRealtimeMediaSourceCenter::mockDeviceWithPersistentID(m_source->persistentID());
     158    ASSERT(device);
     159    if (!device)
     160        return { };
     161
     162    ASSERT(device->isDisplay());
     163    if (!device->isDisplay())
     164        return { };
     165
     166    auto& properties = WTF::get<MockDisplayProperties>(device->properties);
     167    return properties.defaultSize;
     168}
     169#endif // PLATFORM(MAC)
    154170
    155171class MockRealtimeDisplaySourceFactory : public DisplayCaptureFactory {
Note: See TracChangeset for help on using the changeset viewer.