Changes between Version 10 and Version 11 of Performance Tests


Ignore:
Timestamp:
Dec 25, 2014 12:06:35 PM (3 years ago)
Author:
rniwa@webkit.org
Comment:

Deleted the section about replay tests since they are no longer supported.

Legend:

Unmodified
Added
Removed
Modified
  • Performance Tests

    v10 v11  
    153153
    154154
    155 = Replay Performance Tests =
    156 
    157 Replay tests are highly "experimental" page loading tests. Historically, Apple and Google have used PLT and [http://www.chromium.org/developers/testing/page-cyclers Page Cycler Tests] but they could not be part of WebKit or publicly distributed otherwise because they contain copyrighted materials. WebKit replay tests works around this problem by measuring page loading time of web pages on [http://www.archive.org Internet Archive] using local caches provided by [http://code.google.com/p/web-page-replay/ web-page-replay] as a Web proxy.
    158 
    159 A replay test consists of a single text file with an URL in it. For example, [http://trac.webkit.org/browser/trunk/PerformanceTests/Replay/English/digg.com.replay digg.com.replay] contains
    160 {{{
    161 http://web.archive.org/web/20100730073647/http://digg.com/
    162 }}}
    163 as of July 27th, 2012. {{{run-perf-tests}}} creates {{{digg.com.wpr}}} and {{{digg.com-expected.png}}} when preparing the local cache, and creates {{{digg.com-actual.png}}} as it runs the test.
    164 
    165 === How to Run Replay Tests ===
    166 Replay tests are currently supported on Mac port and Chromium port on Mac and Linux. To run tests, you must set the local proxy to localhost at port 8080 for HTTP and port 8443 for HTTPs. This will allow DumpRenderTree or WebKitTestRunner to talk to web-page-replay to cache pages locally instead of directly accessing archive.org. Exclude {{{*.googlecode.com}}} as web-page-replay needs to be downloaded from Google Code on the initial run.
    167 
    168  - On Mac, the proxy can be set at System Preferences > Network > Advanced > Proxies.
    169  - On Linux, the proxy can be set by {{{$http_proxy}}}, {{{$https_proxy}}}, {{{$no_proxy}}} (specifies hosts to be excluded) environmental variables.
    170 
    171 Once the proxy is setup, run {{{run-perf-tests --replay}}}. Since all replay tests are located in {{{PerformanceTests/Replay}}}, you can only run replay tests by {{{run-perf-tests --replay PerformanceTests/Replay}}}.  {{{run-perf-tests}}} will first prepares local caches using web-page-replay's record mode, and then makes 20 measurements of page load times using the play mode.
    172 
    173 Make sure that .wpr files created for each test contain actual contents. For example, if the .wpr file is less than 100KB, it's likely that the test runner is accessing the remote servers directly and not going through web-page-replay. You can also make sure that tests are running properly by comparing the contents of {{{digg.com-expected.png}}} and {{{digg.com-actual.png}}}. Unfortunately, this image comparison cannot be automated as the image contains copyrighted material (preventing to be checked into the SVN repository) and it changes as WebKit is updated.
    174 
    175 === How to Write a Replay Test ===
    176 
    177 To write a new replay test for an URL, go visit [http://www.archive.org/ Internet Archive] and look for an archive of the URL. If there is no archive for the URL, then we cannot create a replay test for this page. Also, if the archive doesn't contain a significant amount of essential non-HTML contents such as images, css, and plugins, it may not be suitable as a replay test.
    178 
    179 Once you've obtained an archive.org URL, then create a .reply file in [http://trac.webkit.org/browser/trunk/PerformanceTests/Replay/ PerformanceTests/Replay] and run {{{run-perf-tests --replay <path to new .replay file>}}} (don't forget to setup the proxy).
    180 
    181 Look for any errors web-page-replay reports. For example, failures to inject script is a very common error and can be ignored in most cases. However, "pipe broken" errors and other python exceptions tend to be an indication of the content not being served properly via web-page-replay. If these errors occur, try other archives of the same URL.
    182 
    183 When the tests finish successfully without errors, look at the mean and the standard deviation of the test. If the standard deviation is higher than 4-6% of the mean, try other archives of the same URL. It's important to recognize that different archives of the same URL can yield significantly different variances as follows:
    184 
    185 {{{
    186 http://web.archive.org/web/20110729050650/http://www.kp.ru/
    187 RESULT Replay: Russian: www.kp.ru.replay= 2164.80790941 ms
    188 median= 2173.47407341 ms, stdev= 216.239083036 ms, min= 2067.66700745 ms, max= 2248.77309799 ms
    189 
    190 http://web.archive.org/web/20110119021944/http://www.kp.ru/
    191 RESULT Replay: Russian: www.kp.ru.replay= 3299.88499692 ms
    192 median= 3802.16002464 ms, stdev= 4244.51026529 ms, min= 1394.64211464 ms, max= 3824.78284836 ms
    193 
    194 http://web.archive.org/web/20110318023959/http://kp.ru/
    195 RESULT Replay: Russian: www.kp.ru.replay= 1667.41889401 ms
    196 median= 1667.26398468 ms, stdev= 67.7172770899 ms, min= 1643.22805405 ms, max= 1702.38494873 ms
    197 }}}
    198 
    199 If python exceptions or other serious errors persist, or the ratio of standard deviation to mean is consistently higher than 7-10%, don't add the URL as a replay test regardless of how important that website is because we can't make a use of performance tests that have 10% variance.
    200 
    201 
    202155= Profiling Performance Tests =
    203156