Changeset 280531 in webkit
- Timestamp:
- Aug 2, 2021, 8:39:22 AM (5 years ago)
- Location:
- trunk
- Files:
-
- 1 added
- 12 edited
-
LayoutTests/imported/w3c/ChangeLog (modified) (1 diff)
-
LayoutTests/imported/w3c/web-platform-tests/html/rendering/replaced-elements/attributes-for-embedded-content-and-images/video-aspect-ratio.html (modified) (2 diffs)
-
LayoutTests/imported/w3c/web-platform-tests/html/semantics/embedded-content/media-elements/video_loop_base.html (modified) (1 diff)
-
LayoutTests/imported/w3c/web-platform-tests/media/16x16-green.mp4 (added)
-
Source/WebCore/ChangeLog (modified) (1 diff)
-
Source/WebCore/html/HTMLMediaElement.cpp (modified) (2 diffs)
-
Source/WebCore/platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.cpp (modified) (5 diffs)
-
Source/WebCore/platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.h (modified) (3 diffs)
-
Source/WebCore/platform/graphics/avfoundation/cf/MediaPlayerPrivateAVFoundationCF.cpp (modified) (2 diffs)
-
Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.h (modified) (1 diff)
-
Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm (modified) (12 diffs)
-
Source/WebKit/ChangeLog (modified) (1 diff)
-
Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp (modified) (1 diff)
Legend:
- Unmodified
- Added
- Removed
-
trunk/LayoutTests/imported/w3c/ChangeLog
r280527 r280531 1 2021-08-02 Jer Noble <jer.noble@apple.com> 2 3 [Cocoa] Remove support for AVAssetImageGenerator 4 https://bugs.webkit.org/show_bug.cgi?id=228560 5 6 Reviewed by Eric Carlson. 7 8 The 2x2-green.mp4 file fails to decode as its natural size is too small for the system decoder to handle. Replace with 9 a media file of more reasonable size. 10 11 * web-platform-tests/html/rendering/replaced-elements/attributes-for-embedded-content-and-images/video-aspect-ratio.html: 12 * web-platform-tests/html/semantics/embedded-content/media-elements/video_loop_base.html: 13 * web-platform-tests/media/16x16-green.mp4: Added. 14 1 15 2021-08-02 Martin Robinson <mrobinson@webkit.org> 2 16 -
trunk/LayoutTests/imported/w3c/web-platform-tests/html/rendering/replaced-elements/attributes-for-embedded-content-and-images/video-aspect-ratio.html
r280511 r280531 27 27 t.step(function() { 28 28 var video = document.getElementById("contained"); 29 video.src = getVideoURI('/media/ 2x2-green');29 video.src = getVideoURI('/media/16x16-green'); 30 30 assert_ratio(video, 2.5); 31 31 }, "contain:size aspect ratio"); … … 38 38 video.setAttribute("width", "250"); 39 39 video.setAttribute("height", "100"); 40 video.src = getVideoURI('/media/ 2x2-green');40 video.src = getVideoURI('/media/16x16-green'); 41 41 document.body.appendChild(video); 42 42 // Videos default to a size of 300x150px and calculate their aspect ratio -
trunk/LayoutTests/imported/w3c/web-platform-tests/html/semantics/embedded-content/media-elements/video_loop_base.html
r280511 r280531 34 34 media.addEventListener("seeking", startTest, false); 35 35 media.loop = true; 36 media.src = getVideoURI("/media/ 2x2-green") + "?" + new Date() + Math.random();36 media.src = getVideoURI("/media/16x16-green") + "?" + new Date() + Math.random(); 37 37 media.play(); 38 38 </script> -
trunk/Source/WebCore/ChangeLog
r280530 r280531 1 2021-08-02 Jer Noble <jer.noble@apple.com> 2 3 [Cocoa] Remove support for AVAssetImageGenerator 4 https://bugs.webkit.org/show_bug.cgi?id=228560 5 6 Reviewed by Eric Carlson. 7 8 AVAssetImageGenerator is used when there is not yet an available image from 9 AVPlayerItemVideoOutput. However, AVAssetImageGenerator is a synchronous API call; if data 10 for the current time is not available, the generator will cause a request for that data to 11 be issued, and if the main thread is blocked in a synchronous call to the generator, that 12 request can never be completed, causing a deadlock. 13 14 Instead, always block moving to a readyState >= HAVE_CURRENT_DATA until either the 15 AVPlayerLayer or the AVPlayerItemVideoOutput report having an available frame. This 16 prevents clients from attempting to paint until a frame is available, and ensures that we 17 will always have an available frame for painting, removing the need for the synchronous 18 generator call. 19 20 To ensure the readyState is updated when AVPlayerItemVideoOutput has an available image, 21 register with the output at creation time for a notification as soon as an image is 22 available. And in the existing delegate object, send a message back to the 23 MediaPlayerPrivateAVFoundationObjC when that occurs. 24 25 No tests needed; this should reduce flakiness of existing tests and cause no regression in 26 behavior. 27 28 Note, changes from last attempt: 29 30 AVPlayerLayer will fail to move to the -readyForDisplay state if it's not attached to a 31 layer tree, so when we remove the HTMLMediaElement from the DOM, we must send MediaPlayer a 32 acceleratedRenderingStateChanged() notification. This will cause the 33 MediaPlayerPrivateAVFoundationObjC to throw away its AVPlayerLayer, and will not block 34 moving to HAVE_CURRENT_DATA waiting for the layer to become readyForDisplay. 35 36 When we throw away (or create) the AVPlayerLayer, we must conditionally call updateStates 37 () to recalculate the readyState; but because updateStates() may try to create or destroy 38 an AVPlayerLayer, we must do this on a task to avoid re-entrancy. Adopt these changes 39 inside MediaPlayerPrivateAVFoundationCF as well. 40 41 * html/HTMLMediaElement.cpp: 42 (WebCore::HTMLMediaElement::pauseAfterDetachedTask): 43 (WebCore::HTMLMediaElement::mediaPlayerRenderingCanBeAccelerated): 44 * platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.cpp: 45 (WebCore::MediaPlayerPrivateAVFoundation::setUpVideoRendering): 46 (WebCore::MediaPlayerPrivateAVFoundation::prepareForRendering): 47 (WebCore::MediaPlayerPrivateAVFoundation::renderingModeChanged): 48 (WebCore::MediaPlayerPrivateAVFoundation::scheduleUpdateStates): 49 (WebCore::MediaPlayerPrivateAVFoundation::updateStates): 50 * platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.h: 51 * platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.h: 52 * platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm: 53 (WebCore::MediaPlayerPrivateAVFoundationObjC::hasContextRenderer const): 54 (WebCore::MediaPlayerPrivateAVFoundationObjC::destroyContextVideoRenderer): 55 (WebCore::MediaPlayerPrivateAVFoundationObjC::createVideoLayer): 56 (WebCore::MediaPlayerPrivateAVFoundationObjC::destroyVideoLayer): 57 (WebCore::MediaPlayerPrivateAVFoundationObjC::paintCurrentFrameInContext): 58 (WebCore::MediaPlayerPrivateAVFoundationObjC::createVideoOutput): 59 (WebCore::MediaPlayerPrivateAVFoundationObjC::outputMediaDataWillChange): 60 (WebCore::MediaPlayerPrivateAVFoundationObjC::paintWithVideoOutput): 61 (-[WebCoreAVFPullDelegate setParent:]): 62 (-[WebCoreAVFPullDelegate outputMediaDataWillChange:]): 63 (WebCore::MediaPlayerPrivateAVFoundationObjC::createImageGenerator): Deleted. 64 (WebCore::MediaPlayerPrivateAVFoundationObjC::destroyImageGenerator): Deleted. 65 (WebCore::MediaPlayerPrivateAVFoundationObjC::paintWithImageGenerator): Deleted. 66 (WebCore::MediaPlayerPrivateAVFoundationObjC::createImageForTimeInRect): Deleted. 67 * platform/graphics/avfoundation/cf/MediaPlayerPrivateAVFoundationCF.cpp: 68 (WebCore::MediaPlayerPrivateAVFoundationCF::createVideoLayer): 69 (WebCore::MediaPlayerPrivateAVFoundationCF::destroyVideoLayer): 70 1 71 2021-08-02 Alan Bujtas <zalan@apple.com> 2 72 -
trunk/Source/WebCore/html/HTMLMediaElement.cpp
r280468 r280531 820 820 return; 821 821 822 m_player->acceleratedRenderingStateChanged(); 823 822 824 size_t extraMemoryCost = m_player->extraMemoryCost(); 823 825 if (extraMemoryCost > m_reportedExtraMemoryCost) { … … 5055 5057 return true; 5056 5058 5059 if (!m_inActiveDocument) 5060 return false; 5061 5057 5062 auto* renderer = this->renderer(); 5058 5063 return is<RenderVideo>(renderer) -
trunk/Source/WebCore/platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.cpp
r280511 r280531 126 126 preferredMode = MediaRenderingToContext; 127 127 128 if (currentMode == preferredMode && currentMode != MediaRenderingNone) 129 return; 130 131 if (currentMode != MediaRenderingNone) 132 tearDownVideoRendering(); 128 if (currentMode == preferredMode) 129 return; 130 131 ALWAYS_LOG(LOGIDENTIFIER, "preferredMode: ", preferredMode, ", currentMode: ", currentMode); 132 133 if (currentMode == MediaRenderingToLayer) 134 destroyVideoLayer(); 133 135 134 136 switch (preferredMode) { … … 142 144 break; 143 145 } 144 145 // If using a movie layer, inform the client so the compositing tree is updated.146 if (currentMode == MediaRenderingToLayer || preferredMode == MediaRenderingToLayer)147 m_player->renderingModeChanged();148 146 } 149 147 … … 444 442 445 443 setUpVideoRendering(); 446 447 if (currentRenderingMode() == MediaRenderingToLayer || preferredRenderingMode() == MediaRenderingToLayer)448 m_player->renderingModeChanged();449 444 } 450 445 … … 470 465 m_resolvedURL = WTFMove(resolvedURL); 471 466 m_resolvedOrigin = SecurityOrigin::create(m_resolvedURL); 467 } 468 469 void MediaPlayerPrivateAVFoundation::renderingModeChanged() 470 { 471 if (m_delayingReadyState && m_cachedHasVideo && hasAvailableVideoFrame()) 472 scheduleUpdateStates(); 473 m_player->renderingModeChanged(); 474 } 475 476 void MediaPlayerPrivateAVFoundation::scheduleUpdateStates() 477 { 478 queueTaskOnEventLoop([weakThis = makeWeakPtr(*this)] { 479 if (weakThis) 480 weakThis->updateStates(); 481 }); 472 482 } 473 483 … … 544 554 } 545 555 556 // Do not advance to HaveCurrentData unless there is a decoded frame available for display 557 if (newReadyState >= MediaPlayer::ReadyState::HaveCurrentData 558 && m_readyState < MediaPlayer::ReadyState::HaveCurrentData 559 && m_cachedHasVideo && !hasAvailableVideoFrame()) { 560 newReadyState = MediaPlayer::ReadyState::HaveMetadata; 561 m_delayingReadyState = true; 562 ALWAYS_LOG(LOGIDENTIFIER, "!hasAvailableVideoFrame(), lowering readyState to ", newReadyState); 563 } else 564 m_delayingReadyState = false; 565 546 566 if (isReadyForVideoSetup() && currentRenderingMode() != preferredRenderingMode()) 547 567 setUpVideoRendering(); 548 568 549 569 if (!m_haveReportedFirstVideoFrame && m_cachedHasVideo && hasAvailableVideoFrame()) { 550 if ( m_readyState < MediaPlayer::ReadyState::HaveCurrentData)570 if (newReadyState < MediaPlayer::ReadyState::HaveCurrentData) 551 571 newReadyState = MediaPlayer::ReadyState::HaveCurrentData; 552 572 m_haveReportedFirstVideoFrame = true; -
trunk/Source/WebCore/platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.h
r278746 r280531 272 272 273 273 protected: 274 void scheduleUpdateStates(); 274 275 void updateStates(); 275 276 … … 320 321 void setResolvedURL(URL&&); 321 322 const URL& resolvedURL() const { return m_resolvedURL; } 323 324 void renderingModeChanged(); 322 325 323 326 private: … … 369 372 bool m_shouldMaintainAspectRatio; 370 373 bool m_seeking; 374 bool m_delayingReadyState { false }; 371 375 }; 372 376 -
trunk/Source/WebCore/platform/graphics/avfoundation/cf/MediaPlayerPrivateAVFoundationCF.cpp
r277958 r280531 461 461 if (m_avfWrapper) 462 462 m_avfWrapper->createAVCFVideoLayer(); 463 464 renderingModeChanged(); 463 465 } 464 466 … … 469 471 if (m_avfWrapper) 470 472 m_avfWrapper->destroyVideoLayer(); 473 474 renderingModeChanged(); 471 475 } 472 476 -
trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.h
r280511 r280531 118 118 119 119 MediaTime currentMediaTime() const final; 120 121 void outputMediaDataWillChange(); 120 122 121 123 private: -
trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm
r280511 r280531 68 68 #import "WebCoreNSURLExtras.h" 69 69 #import "WebCoreNSURLSession.h" 70 #import <AVFoundation/AVAssetImageGenerator.h>71 70 #import <AVFoundation/AVAssetTrack.h> 72 71 #import <AVFoundation/AVMediaSelectionGroup.h> … … 197 196 BinarySemaphore m_semaphore; 198 197 } 198 - (void)setParent:(WebCore::MediaPlayerPrivateAVFoundationObjC&)parent; 199 199 - (void)outputMediaDataWillChange:(AVPlayerItemOutput *)sender; 200 200 - (void)outputSequenceWasFlushed:(AVPlayerItemOutput *)output; … … 562 562 bool MediaPlayerPrivateAVFoundationObjC::hasContextRenderer() const 563 563 { 564 return m_videoOutput || m_imageGenerator;564 return m_videoOutput; 565 565 } 566 566 … … 570 570 } 571 571 572 void MediaPlayerPrivateAVFoundationObjC::createImageGenerator()573 {574 INFO_LOG(LOGIDENTIFIER);575 576 if (!m_avAsset || m_imageGenerator)577 return;578 579 m_imageGenerator = [PAL::getAVAssetImageGeneratorClass() assetImageGeneratorWithAsset:m_avAsset.get()];580 581 [m_imageGenerator.get() setApertureMode:AVAssetImageGeneratorApertureModeCleanAperture];582 [m_imageGenerator.get() setAppliesPreferredTrackTransform:YES];583 [m_imageGenerator.get() setRequestedTimeToleranceBefore:PAL::kCMTimeZero];584 [m_imageGenerator.get() setRequestedTimeToleranceAfter:PAL::kCMTimeZero];585 }586 587 572 void MediaPlayerPrivateAVFoundationObjC::destroyContextVideoRenderer() 588 573 { 589 574 destroyVideoOutput(); 590 destroyImageGenerator();591 }592 593 void MediaPlayerPrivateAVFoundationObjC::destroyImageGenerator()594 {595 if (!m_imageGenerator)596 return;597 598 INFO_LOG(LOGIDENTIFIER);599 600 m_imageGenerator = 0;601 575 } 602 576 … … 620 594 createVideoOutput(); 621 595 622 player()->renderingModeChanged();596 renderingModeChanged(); 623 597 }); 624 598 } … … 659 633 660 634 m_videoLayer = nil; 635 636 renderingModeChanged(); 661 637 } 662 638 … … 1814 1790 if (videoOutputHasAvailableFrame() || (m_videoOutput && m_lastPixelBuffer)) 1815 1791 paintWithVideoOutput(context, rect); 1816 else1817 paintWithImageGenerator(context, rect);1818 1792 1819 1793 END_BLOCK_OBJC_EXCEPTIONS … … 1837 1811 1838 1812 paintCurrentFrameInContext(context, rect); 1839 }1840 1841 void MediaPlayerPrivateAVFoundationObjC::paintWithImageGenerator(GraphicsContext& context, const FloatRect& rect)1842 {1843 INFO_LOG(LOGIDENTIFIER);1844 1845 RetainPtr<CGImageRef> image = createImageForTimeInRect(currentTime(), rect);1846 if (image) {1847 GraphicsContextStateSaver stateSaver(context);1848 context.translate(rect.x(), rect.y() + rect.height());1849 context.scale(FloatSize(1.0f, -1.0f));1850 context.setImageInterpolationQuality(InterpolationQuality::Low);1851 IntRect paintRect(IntPoint(0, 0), IntSize(rect.width(), rect.height()));1852 CGContextDrawImage(context.platformContext(), CGRectMake(0, 0, paintRect.width(), paintRect.height()), image.get());1853 }1854 }1855 1856 RetainPtr<CGImageRef> MediaPlayerPrivateAVFoundationObjC::createImageForTimeInRect(float time, const FloatRect& rect)1857 {1858 if (!m_imageGenerator)1859 createImageGenerator();1860 ASSERT(m_imageGenerator);1861 1862 MonotonicTime start = MonotonicTime::now();1863 1864 [m_imageGenerator.get() setMaximumSize:CGSize(rect.size())];1865 RetainPtr<CGImageRef> rawImage = adoptCF([m_imageGenerator.get() copyCGImageAtTime:PAL::CMTimeMakeWithSeconds(time, 600) actualTime:nil error:nil]);1866 RetainPtr<CGImageRef> image = adoptCF(CGImageCreateCopyWithColorSpace(rawImage.get(), sRGBColorSpaceRef()));1867 1868 INFO_LOG(LOGIDENTIFIER, "creating image took ", (MonotonicTime::now() - start).seconds());1869 1870 return image;1871 1813 } 1872 1814 … … 2441 2383 2442 2384 m_videoOutputDelegate = adoptNS([[WebCoreAVFPullDelegate alloc] init]); 2385 [m_videoOutputDelegate setParent:*this]; 2443 2386 [m_videoOutput setDelegate:m_videoOutputDelegate.get() queue:globalPullDelegateQueue()]; 2387 [m_videoOutput requestNotificationOfMediaDataChangeWithAdvanceInterval:0]; 2444 2388 2445 2389 [m_avPlayerItem.get() addOutput:m_videoOutput.get()]; 2390 } 2391 2392 void MediaPlayerPrivateAVFoundationObjC::outputMediaDataWillChange() 2393 { 2394 updateStates(); 2446 2395 } 2447 2396 … … 2540 2489 FloatRect imageRect { FloatPoint::zero(), m_lastImage->size() }; 2541 2490 context.drawNativeImage(*m_lastImage, imageRect.size(), outputRect, imageRect); 2542 2543 // If we have created an AVAssetImageGenerator in the past due to m_videoOutput not having an available2544 // video frame, destroy it now that it is no longer needed.2545 if (m_imageGenerator)2546 destroyImageGenerator();2547 2548 2491 } 2549 2492 … … 3900 3843 @end 3901 3844 3902 @implementation WebCoreAVFPullDelegate 3845 @implementation WebCoreAVFPullDelegate { 3846 WeakPtr<WebCore::MediaPlayerPrivateAVFoundationObjC> _parent; 3847 } 3903 3848 3904 3849 @synthesize semaphore = m_semaphore; 3850 3851 - (void)setParent:(WebCore::MediaPlayerPrivateAVFoundationObjC&)parent 3852 { 3853 _parent = makeWeakPtr(parent); 3854 } 3905 3855 3906 3856 - (void)outputMediaDataWillChange:(AVPlayerItemVideoOutput *)output … … 3908 3858 UNUSED_PARAM(output); 3909 3859 m_semaphore.signal(); 3860 RunLoop::main().dispatch([parent = _parent] { 3861 if (parent) 3862 parent->outputMediaDataWillChange(); 3863 }); 3910 3864 } 3911 3865 -
trunk/Source/WebKit/ChangeLog
r280527 r280531 1 2021-08-02 Jer Noble <jer.noble@apple.com> 2 3 [Cocoa] Remove support for AVAssetImageGenerator 4 https://bugs.webkit.org/show_bug.cgi?id=228560 5 <rdar://problem/81336280> 6 7 Reviewed by Eric Carlson. 8 9 Drive-by fix: Passing the wrong value down to the GPU process when the acceleratedRenderingStateChanged(). 10 We should be passing the results of MediaPlayer::renderingCanBeAccelerated() which comes down from HTMLMediaElement, 11 not MediaPlayer::supportsAcceleratedRendering(), which comes up from the GPU process. 12 13 * WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp: 14 (WebKit::MediaPlayerPrivateRemote::acceleratedRenderingStateChanged): 15 1 16 2021-08-02 Martin Robinson <mrobinson@igalia.com> 2 17 -
trunk/Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp
r279786 r280531 456 456 { 457 457 if (auto player = makeRefPtr(m_player.get())) 458 connection().send(Messages::RemoteMediaPlayerProxy::AcceleratedRenderingStateChanged(player-> supportsAcceleratedRendering()), m_id);458 connection().send(Messages::RemoteMediaPlayerProxy::AcceleratedRenderingStateChanged(player->renderingCanBeAccelerated()), m_id); 459 459 } 460 460
Note:
See TracChangeset
for help on using the changeset viewer.