Changeset 248095 in webkit


Ignore:
Timestamp:
Jul 31, 2019 9:24:57 PM (5 years ago)
Author:
aestes@apple.com
Message:

REGRESSION (r240942): first visually non-empty layout milestone is not reached in media documents until after the video finishes loading
​https://bugs.webkit.org/show_bug.cgi?id=200293
<rdar://problem/52937749>

Reviewed by Alex Christensen.

Source/WebCore:

r240942 changed FrameView::qualifiesAsVisuallyNonEmpty() to consider only documents in the
Interactive or Complete ready states as "finished parsing". Documents considered finished
parsing can qualify as visually non-empty even without exceeding the visual character or
pixel thresholds, but documents considered not finished must first exceed one of these
thresholds in order to qualify as visually non-empty.

HTMLDocuments are placed in the Interactive ready state by their HTMLDocumentParsers.
However, HTMLDocument subclasses like ImageDocument and MediaDocument use their own custom
parsers that never set the Interactive ready state on their documents; these documents go
from Loading directly to Complete.

In order for these HTMLDocument subclasses to be considered visually non-empty before they
finish loading they must render something that exceeds the visual character or pixel
thresholds. For image documents, rendering the image is usually enough to cross the
threshold, but for media documents the visual pixel threshold was never crossed because
videos did not contribute to the visually non-empty pixel count.

As a result, media documents are not considered visually non-empty until the main resource
finishes loading. On iOS this means that the layer tree remains frozen until this point,
even though the media might have started autoplaying with audio long before it finished
loading.

Fix this by teaching RenderVideo to contribute the video player's size to FrameView's
visually non-empty pixel count once the video player has loaded enough data to determine its
intrinsic size. Videos that render more than 1024 pixels will qualify a media document as
visually non-empty even when it is still loading its main resource.

Added a new API test.

  • rendering/RenderImage.cpp:

(WebCore::RenderImage::imageChanged):
(WebCore::RenderImage::incrementVisuallyNonEmptyPixelCountIfNeeded):

  • rendering/RenderImage.h:
  • rendering/RenderVideo.cpp:

(WebCore::RenderVideo::updateIntrinsicSize):

Tools:

  • TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
  • TestWebKitAPI/Tests/WebKitCocoa/FirstVisuallyNonEmptyMilestone.mm: Renamed from Tools/TestWebKitAPI/Tests/WebKit/FirstVisuallyNonEmptyMilestoneWithDeferredScript.mm.

(-[FirstPaintMessageHandler userContentController:didReceiveScriptMessage:]):
(-[RenderingProgressNavigationDelegate _webView:renderingProgressDidChange:]):
(-[RenderingProgressNavigationDelegate webView:didFinishNavigation:]):
(TEST):

Location:
trunk
Files:
6 edited
1 moved

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebCore/ChangeLog

    r248083 r248095  
     12019-07-31  Andy Estes  <aestes@apple.com>
     2
     3        REGRESSION (r240942): first visually non-empty layout milestone is not reached in media documents until after the video finishes loading
     4        https://bugs.webkit.org/show_bug.cgi?id=200293
     5        <rdar://problem/52937749>
     6
     7        Reviewed by Alex Christensen.
     8
     9        r240942 changed FrameView::qualifiesAsVisuallyNonEmpty() to consider only documents in the
     10        Interactive or Complete ready states as "finished parsing". Documents considered finished
     11        parsing can qualify as visually non-empty even without exceeding the visual character or
     12        pixel thresholds, but documents considered not finished must first exceed one of these
     13        thresholds in order to qualify as visually non-empty.
     14
     15        HTMLDocuments are placed in the Interactive ready state by their HTMLDocumentParsers.
     16        However, HTMLDocument subclasses like ImageDocument and MediaDocument use their own custom
     17        parsers that never set the Interactive ready state on their documents; these documents go
     18        from Loading directly to Complete.
     19
     20        In order for these HTMLDocument subclasses to be considered visually non-empty before they
     21        finish loading they must render something that exceeds the visual character or pixel
     22        thresholds. For image documents, rendering the image is usually enough to cross the
     23        threshold, but for media documents the visual pixel threshold was never crossed because
     24        videos did not contribute to the visually non-empty pixel count.
     25
     26        As a result, media documents are not considered visually non-empty until the main resource
     27        finishes loading. On iOS this means that the layer tree remains frozen until this point,
     28        even though the media might have started autoplaying with audio long before it finished
     29        loading.
     30
     31        Fix this by teaching RenderVideo to contribute the video player's size to FrameView's
     32        visually non-empty pixel count once the video player has loaded enough data to determine its
     33        intrinsic size. Videos that render more than 1024 pixels will qualify a media document as
     34        visually non-empty even when it is still loading its main resource.
     35
     36        Added a new API test.
     37
     38        * rendering/RenderImage.cpp:
     39        (WebCore::RenderImage::imageChanged):
     40        (WebCore::RenderImage::incrementVisuallyNonEmptyPixelCountIfNeeded):
     41        * rendering/RenderImage.h:
     42        * rendering/RenderVideo.cpp:
     43        (WebCore::RenderVideo::updateIntrinsicSize):
     44
    1452019-07-31  Saam Barati  <sbarati@apple.com>
    246

  • trunk/Source/WebCore/rendering/RenderImage.cpp

    r245778 r248095  
    270270    if (newImage != imageResource().imagePtr() || !newImage)
    271271        return;
    272    
    273     if (!m_didIncrementVisuallyNonEmptyPixelCount) {
    274         // At a zoom level of 1 the image is guaranteed to have an integer size.
    275         view().frameView().incrementVisuallyNonEmptyPixelCount(flooredIntSize(imageResource().imageSize(1.0f)));
    276         m_didIncrementVisuallyNonEmptyPixelCount = true;
    277     }
     272
     273    // At a zoom level of 1 the image is guaranteed to have an integer size.
     274    incrementVisuallyNonEmptyPixelCountIfNeeded(flooredIntSize(imageResource().imageSize(1.0f)));
    278275
    279276    ImageSizeChangeType imageSizeChange = ImageSizeChangeNone;
    … …  
    851848}
    852849
     850void RenderImage::incrementVisuallyNonEmptyPixelCountIfNeeded(const IntSize& size)
     851{
     852    if (m_didIncrementVisuallyNonEmptyPixelCount)
     853        return;
     854
     855    view().frameView().incrementVisuallyNonEmptyPixelCount(size);
     856    m_didIncrementVisuallyNonEmptyPixelCount = true;
     857}
     858
    853859} // namespace WebCore

  • trunk/Source/WebCore/rendering/RenderImage.h

    r238108 r248095  
    101101    }
    102102
     103    void incrementVisuallyNonEmptyPixelCountIfNeeded(const IntSize&);
     104
    103105private:
    104106    const char* renderName() const override { return "RenderImage"; }

  • trunk/Source/WebCore/rendering/RenderVideo.cpp

    r241120 r248095  
    102102        return false;
    103103
     104    // Treat the media player's natural size as visually non-empty.
     105    if (videoElement().readyState() >= HTMLMediaElementEnums::HAVE_METADATA)
     106        incrementVisuallyNonEmptyPixelCountIfNeeded(roundedIntSize(size));
     107
    104108    if (size == intrinsicSize())
    105109        return false;
    … …  
    110114    return true;
    111115}
    112    
     116
    113117LayoutSize RenderVideo::calculateIntrinsicSize()
    114118{

  • trunk/Tools/ChangeLog

    r248088 r248095  
     12019-07-31  Andy Estes  <aestes@apple.com>
     2
     3        REGRESSION (r240942): first visually non-empty layout milestone is not reached in media documents until after the video finishes loading
     4        https://bugs.webkit.org/show_bug.cgi?id=200293
     5        <rdar://problem/52937749>
     6
     7        Reviewed by Alex Christensen.
     8
     9        * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
     10        * TestWebKitAPI/Tests/WebKitCocoa/FirstVisuallyNonEmptyMilestone.mm: Renamed from Tools/TestWebKitAPI/Tests/WebKit/FirstVisuallyNonEmptyMilestoneWithDeferredScript.mm.
     11        (-[FirstPaintMessageHandler userContentController:didReceiveScriptMessage:]):
     12        (-[RenderingProgressNavigationDelegate _webView:renderingProgressDidChange:]):
     13        (-[RenderingProgressNavigationDelegate webView:didFinishNavigation:]):
     14        (TEST):
     15
    1162019-07-31  Aakash Jain  <aakash_jain@apple.com>
    217

  • trunk/Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj

    r248029 r248095  
    6666                118153442208B7AC00B2CCD2 /* deferred-script-load.html in Copy Resources */ = {isa = PBXBuildFile; fileRef = 118153432208B7AC00B2CCD2 /* deferred-script-load.html */; };
    6767                118153462208B7E500B2CCD2 /* deferred-script.js in Copy Resources */ = {isa = PBXBuildFile; fileRef = 118153452208B7E500B2CCD2 /* deferred-script.js */; };
    68                 118153482208BADF00B2CCD2 /* FirstVisuallyNonEmptyMilestoneWithDeferredScript.mm in Sources */ = {isa = PBXBuildFile; fileRef = 118153472208BADF00B2CCD2 /* FirstVisuallyNonEmptyMilestoneWithDeferredScript.mm */; };
     68                118153482208BADF00B2CCD2 /* FirstVisuallyNonEmptyMilestone.mm in Sources */ = {isa = PBXBuildFile; fileRef = 118153472208BADF00B2CCD2 /* FirstVisuallyNonEmptyMilestone.mm */; };
    6969                11B7FD28219F47110069B27F /* FirstMeaningfulPaintMilestone.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 11B7FD22219F46DD0069B27F /* FirstMeaningfulPaintMilestone.cpp */; };
    7070                11C2598D21FA6324004C9E23 /* async-script-load.html in Copy Resources */ = {isa = PBXBuildFile; fileRef = 11C2598C21FA618D004C9E23 /* async-script-load.html */; };
    … …  
    14441444                118153432208B7AC00B2CCD2 /* deferred-script-load.html */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.html; path = "deferred-script-load.html"; sourceTree = "<group>"; };
    14451445                118153452208B7E500B2CCD2 /* deferred-script.js */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.javascript; path = "deferred-script.js"; sourceTree = "<group>"; };
    1446                 118153472208BADF00B2CCD2 /* FirstVisuallyNonEmptyMilestoneWithDeferredScript.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = FirstVisuallyNonEmptyMilestoneWithDeferredScript.mm; sourceTree = "<group>"; };
     1446                118153472208BADF00B2CCD2 /* FirstVisuallyNonEmptyMilestone.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = FirstVisuallyNonEmptyMilestone.mm; sourceTree = "<group>"; };
    14471447                11B7FD21219F46DD0069B27F /* FirstMeaningfulPaintMilestone_Bundle.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = FirstMeaningfulPaintMilestone_Bundle.cpp; sourceTree = "<group>"; };
    14481448                11B7FD22219F46DD0069B27F /* FirstMeaningfulPaintMilestone.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = FirstMeaningfulPaintMilestone.cpp; sourceTree = "<group>"; };
    … …  
    27062706                                CDA29B2820FD2A9900F15CED /* ExitFullscreenOnEnterPiP.mm */,
    27072707                                2D8104CB1BEC13E70020DA46 /* FindInPage.mm */,
     2708                                118153472208BADF00B2CCD2 /* FirstVisuallyNonEmptyMilestone.mm */,
    27082709                                2D1FE0AF1AD465C1006CD9E6 /* FixedLayoutSize.mm */,
    27092710                                2E92B8F8216490EA005B64F0 /* FontAttributes.mm */,
    … …  
    33973398                                11B7FD22219F46DD0069B27F /* FirstMeaningfulPaintMilestone.cpp */,
    33983399                                11B7FD21219F46DD0069B27F /* FirstMeaningfulPaintMilestone_Bundle.cpp */,
    3399                                 118153472208BADF00B2CCD2 /* FirstVisuallyNonEmptyMilestoneWithDeferredScript.mm */,
    34003400                                1ADBEFAD130C689C00D61D19 /* ForceRepaint.cpp */,
    34013401                                376C8C041D6E197C007D2BB9 /* FrameHandle.cpp */,
    … …  
    43744374                                7C83E0401D0A63E300FEBCF3 /* FirstResponderScrollingPosition.mm in Sources */,
    43754375                                C9E6DD351EA97D0800DD78AA /* FirstResponderSuppression.mm in Sources */,
    4376                                 118153482208BADF00B2CCD2 /* FirstVisuallyNonEmptyMilestoneWithDeferredScript.mm in Sources */,
     4376                                118153482208BADF00B2CCD2 /* FirstVisuallyNonEmptyMilestone.mm in Sources */,
    43774377                                7C83E0BC1D0A650700FEBCF3 /* FixedLayoutSize.mm in Sources */,
    43784378                                7A909A7E1D877480007E10F8 /* FloatPoint.cpp in Sources */,

  • trunk/Tools/TestWebKitAPI/Tests/WebKitCocoa/FirstVisuallyNonEmptyMilestone.mm

    r248088 r248095  
    2323 * THE POSSIBILITY OF SUCH DAMAGE.
    2424 */
     25
    2526#import "config.h"
    2627
    2728#import "PlatformUtilities.h"
    2829#import "TestNavigationDelegate.h"
     30#import "TestWKWebView.h"
     31#import <WebKit/WKWebViewConfigurationPrivate.h>
    2932#import <wtf/RetainPtr.h>
     33
     34#if PLATFORM(IOS_FAMILY)
     35#include <MobileCoreServices/MobileCoreServices.h>
     36#endif
    3037
    3138static bool didFirstVisuallyNonEmptyLayout;
    … …  
    7279    EXPECT_TRUE(didFirstVisuallyNonEmptyLayout);
    7380}
     81
     82@interface NeverFinishLoadingSchemeHandler : NSObject <WKURLSchemeHandler>
     83@property (nonatomic, readonly, class) NSString *URLScheme;
     84@end
     85
     86@implementation NeverFinishLoadingSchemeHandler
     87
     88+ (NSString *)URLScheme
     89{
     90    return @"never-finish-loading";
     91}
     92
     93static NSString *contentTypeForFileExtension(NSString *fileExtension)
     94{
     95    auto identifier = adoptCF(UTTypeCreatePreferredIdentifierForTag(kUTTagClassFilenameExtension, (__bridge CFStringRef)fileExtension, nullptr));
     96    auto mimeType = adoptCF(UTTypeCopyPreferredTagWithClass(identifier.get(), kUTTagClassMIMEType));
     97    return (__bridge NSString *)mimeType.autorelease();
     98}
     99
     100- (void)webView:(WKWebView *)webView startURLSchemeTask:(id <WKURLSchemeTask>)task
     101{
     102    NSURL *requestURL = task.request.URL;
     103    NSString *fileName = requestURL.lastPathComponent;
     104    NSString *fileExtension = fileName.pathExtension;
     105    NSURL *bundleURL = [NSBundle.mainBundle URLForResource:fileName.stringByDeletingPathExtension withExtension:fileExtension subdirectory:@"TestWebKitAPI.resources"];
     106
     107    NSData *responseData = [NSData dataWithContentsOfURL:bundleURL];
     108    NSUInteger responseLength = responseData.length;
     109
     110    auto response = adoptNS([[NSURLResponse alloc] initWithURL:requestURL MIMEType:contentTypeForFileExtension(fileExtension) expectedContentLength:responseLength textEncodingName:nil]);
     111    [task didReceiveResponse:response.get()];
     112
     113    [task didReceiveData:[responseData subdataWithRange:NSMakeRange(0, responseLength - 1024)]];
     114}
     115
     116- (void)webView:(WKWebView *)webView stopURLSchemeTask:(id <WKURLSchemeTask>)task
     117{
     118}
     119
     120@end
     121
     122TEST(WebKit, FirstVisuallyNonEmptyMilestoneWithMediaDocument)
     123{
     124    auto configuration = adoptNS([[WKWebViewConfiguration alloc] init]);
     125#if PLATFORM(IOS_FAMILY)
     126    [configuration setAllowsInlineMediaPlayback:YES];
     127    [configuration _setInlineMediaPlaybackRequiresPlaysInlineAttribute:NO];
     128#endif
     129
     130    auto schemeHandler = adoptNS([[NeverFinishLoadingSchemeHandler alloc] init]);
     131    [configuration setURLSchemeHandler:schemeHandler.get() forURLScheme:NeverFinishLoadingSchemeHandler.URLScheme];
     132
     133    auto navigationDelegate = adoptNS([[RenderingProgressNavigationDelegate alloc] init]);
     134    auto webView = adoptNS([[TestWKWebView alloc] initWithFrame:CGRectMake(0, 0, 800, 600) configuration:configuration.get() addToWindow:YES]);
     135    [webView setNavigationDelegate:navigationDelegate.get()];
     136    [webView _setAllowsMediaDocumentInlinePlayback:YES];
     137
     138    [webView loadRequest:[NSURLRequest requestWithURL:[NSURL URLWithString:@"never-finish-loading:///large-video-with-audio.mp4"]]];
     139
     140    didFirstVisuallyNonEmptyLayout = false;
     141    TestWebKitAPI::Util::run(&didFirstVisuallyNonEmptyLayout);
     142}
Note: See TracChangeset for help on using the changeset viewer.