Changes between Version 10 and Version 11 of Performance Tests

Dec 25, 2014 12:06:35 PM (2 years ago)

Deleted the section about replay tests since they are no longer supported.


  • Performance Tests

    v10 v11  
    155 = Replay Performance Tests =
    157 Replay tests are highly "experimental" page loading tests. Historically, Apple and Google have used PLT and [ Page Cycler Tests] but they could not be part of WebKit or publicly distributed otherwise because they contain copyrighted materials. WebKit replay tests works around this problem by measuring page loading time of web pages on [ Internet Archive] using local caches provided by [ web-page-replay] as a Web proxy.
    159 A replay test consists of a single text file with an URL in it. For example, [] contains
    160 {{{
    162 }}}
    163 as of July 27th, 2012. {{{run-perf-tests}}} creates {{{}}} and {{{}}} when preparing the local cache, and creates {{{}}} as it runs the test.
    165 === How to Run Replay Tests ===
    166 Replay tests are currently supported on Mac port and Chromium port on Mac and Linux. To run tests, you must set the local proxy to localhost at port 8080 for HTTP and port 8443 for HTTPs. This will allow DumpRenderTree or WebKitTestRunner to talk to web-page-replay to cache pages locally instead of directly accessing Exclude {{{*}}} as web-page-replay needs to be downloaded from Google Code on the initial run.
    168  - On Mac, the proxy can be set at System Preferences > Network > Advanced > Proxies.
    169  - On Linux, the proxy can be set by {{{$http_proxy}}}, {{{$https_proxy}}}, {{{$no_proxy}}} (specifies hosts to be excluded) environmental variables.
    171 Once the proxy is setup, run {{{run-perf-tests --replay}}}. Since all replay tests are located in {{{PerformanceTests/Replay}}}, you can only run replay tests by {{{run-perf-tests --replay PerformanceTests/Replay}}}.  {{{run-perf-tests}}} will first prepares local caches using web-page-replay's record mode, and then makes 20 measurements of page load times using the play mode.
    173 Make sure that .wpr files created for each test contain actual contents. For example, if the .wpr file is less than 100KB, it's likely that the test runner is accessing the remote servers directly and not going through web-page-replay. You can also make sure that tests are running properly by comparing the contents of {{{}}} and {{{}}}. Unfortunately, this image comparison cannot be automated as the image contains copyrighted material (preventing to be checked into the SVN repository) and it changes as WebKit is updated.
    175 === How to Write a Replay Test ===
    177 To write a new replay test for an URL, go visit [ Internet Archive] and look for an archive of the URL. If there is no archive for the URL, then we cannot create a replay test for this page. Also, if the archive doesn't contain a significant amount of essential non-HTML contents such as images, css, and plugins, it may not be suitable as a replay test.
    179 Once you've obtained an URL, then create a .reply file in [ PerformanceTests/Replay] and run {{{run-perf-tests --replay <path to new .replay file>}}} (don't forget to setup the proxy).
    181 Look for any errors web-page-replay reports. For example, failures to inject script is a very common error and can be ignored in most cases. However, "pipe broken" errors and other python exceptions tend to be an indication of the content not being served properly via web-page-replay. If these errors occur, try other archives of the same URL.
    183 When the tests finish successfully without errors, look at the mean and the standard deviation of the test. If the standard deviation is higher than 4-6% of the mean, try other archives of the same URL. It's important to recognize that different archives of the same URL can yield significantly different variances as follows:
    185 {{{
    187 RESULT Replay: Russian: 2164.80790941 ms
    188 median= 2173.47407341 ms, stdev= 216.239083036 ms, min= 2067.66700745 ms, max= 2248.77309799 ms
    191 RESULT Replay: Russian: 3299.88499692 ms
    192 median= 3802.16002464 ms, stdev= 4244.51026529 ms, min= 1394.64211464 ms, max= 3824.78284836 ms
    195 RESULT Replay: Russian: 1667.41889401 ms
    196 median= 1667.26398468 ms, stdev= 67.7172770899 ms, min= 1643.22805405 ms, max= 1702.38494873 ms
    197 }}}
    199 If python exceptions or other serious errors persist, or the ratio of standard deviation to mean is consistently higher than 7-10%, don't add the URL as a replay test regardless of how important that website is because we can't make a use of performance tests that have 10% variance.
    202155= Profiling Performance Tests =