| 155 | | = Replay Performance Tests = |
| 156 | | |
| 157 | | Replay tests are highly "experimental" page loading tests. Historically, Apple and Google have used PLT and [http://www.chromium.org/developers/testing/page-cyclers Page Cycler Tests] but they could not be part of WebKit or publicly distributed otherwise because they contain copyrighted materials. WebKit replay tests works around this problem by measuring page loading time of web pages on [http://www.archive.org Internet Archive] using local caches provided by [http://code.google.com/p/web-page-replay/ web-page-replay] as a Web proxy. |
| 158 | | |
| 159 | | A replay test consists of a single text file with an URL in it. For example, [http://trac.webkit.org/browser/trunk/PerformanceTests/Replay/English/digg.com.replay digg.com.replay] contains |
| 160 | | {{{ |
| 161 | | http://web.archive.org/web/20100730073647/http://digg.com/ |
| 162 | | }}} |
| 163 | | as of July 27th, 2012. {{{run-perf-tests}}} creates {{{digg.com.wpr}}} and {{{digg.com-expected.png}}} when preparing the local cache, and creates {{{digg.com-actual.png}}} as it runs the test. |
| 164 | | |
| 165 | | === How to Run Replay Tests === |
| 166 | | Replay tests are currently supported on Mac port and Chromium port on Mac and Linux. To run tests, you must set the local proxy to localhost at port 8080 for HTTP and port 8443 for HTTPs. This will allow DumpRenderTree or WebKitTestRunner to talk to web-page-replay to cache pages locally instead of directly accessing archive.org. Exclude {{{*.googlecode.com}}} as web-page-replay needs to be downloaded from Google Code on the initial run. |
| 167 | | |
| 168 | | - On Mac, the proxy can be set at System Preferences > Network > Advanced > Proxies. |
| 169 | | - On Linux, the proxy can be set by {{{$http_proxy}}}, {{{$https_proxy}}}, {{{$no_proxy}}} (specifies hosts to be excluded) environmental variables. |
| 170 | | |
| 171 | | Once the proxy is setup, run {{{run-perf-tests --replay}}}. Since all replay tests are located in {{{PerformanceTests/Replay}}}, you can only run replay tests by {{{run-perf-tests --replay PerformanceTests/Replay}}}. {{{run-perf-tests}}} will first prepares local caches using web-page-replay's record mode, and then makes 20 measurements of page load times using the play mode. |
| 172 | | |
| 173 | | Make sure that .wpr files created for each test contain actual contents. For example, if the .wpr file is less than 100KB, it's likely that the test runner is accessing the remote servers directly and not going through web-page-replay. You can also make sure that tests are running properly by comparing the contents of {{{digg.com-expected.png}}} and {{{digg.com-actual.png}}}. Unfortunately, this image comparison cannot be automated as the image contains copyrighted material (preventing to be checked into the SVN repository) and it changes as WebKit is updated. |
| 174 | | |
| 175 | | === How to Write a Replay Test === |
| 176 | | |
| 177 | | To write a new replay test for an URL, go visit [http://www.archive.org/ Internet Archive] and look for an archive of the URL. If there is no archive for the URL, then we cannot create a replay test for this page. Also, if the archive doesn't contain a significant amount of essential non-HTML contents such as images, css, and plugins, it may not be suitable as a replay test. |
| 178 | | |
| 179 | | Once you've obtained an archive.org URL, then create a .reply file in [http://trac.webkit.org/browser/trunk/PerformanceTests/Replay/ PerformanceTests/Replay] and run {{{run-perf-tests --replay <path to new .replay file>}}} (don't forget to setup the proxy). |
| 180 | | |
| 181 | | Look for any errors web-page-replay reports. For example, failures to inject script is a very common error and can be ignored in most cases. However, "pipe broken" errors and other python exceptions tend to be an indication of the content not being served properly via web-page-replay. If these errors occur, try other archives of the same URL. |
| 182 | | |
| 183 | | When the tests finish successfully without errors, look at the mean and the standard deviation of the test. If the standard deviation is higher than 4-6% of the mean, try other archives of the same URL. It's important to recognize that different archives of the same URL can yield significantly different variances as follows: |
| 184 | | |
| 185 | | {{{ |
| 186 | | http://web.archive.org/web/20110729050650/http://www.kp.ru/ |
| 187 | | RESULT Replay: Russian: www.kp.ru.replay= 2164.80790941 ms |
| 188 | | median= 2173.47407341 ms, stdev= 216.239083036 ms, min= 2067.66700745 ms, max= 2248.77309799 ms |
| 189 | | |
| 190 | | http://web.archive.org/web/20110119021944/http://www.kp.ru/ |
| 191 | | RESULT Replay: Russian: www.kp.ru.replay= 3299.88499692 ms |
| 192 | | median= 3802.16002464 ms, stdev= 4244.51026529 ms, min= 1394.64211464 ms, max= 3824.78284836 ms |
| 193 | | |
| 194 | | http://web.archive.org/web/20110318023959/http://kp.ru/ |
| 195 | | RESULT Replay: Russian: www.kp.ru.replay= 1667.41889401 ms |
| 196 | | median= 1667.26398468 ms, stdev= 67.7172770899 ms, min= 1643.22805405 ms, max= 1702.38494873 ms |
| 197 | | }}} |
| 198 | | |
| 199 | | If python exceptions or other serious errors persist, or the ratio of standard deviation to mean is consistently higher than 7-10%, don't add the URL as a replay test regardless of how important that website is because we can't make a use of performance tests that have 10% variance. |
| 200 | | |
| 201 | | |
