Changes between Version 4 and Version 5 of Performance Tests


Ignore:
Timestamp:
Dec 17, 2012 11:20:27 PM (11 years ago)
Author:
rniwa@webkit.org
Comment:

Revised some

Legend:

Unmodified
Added
Removed
Modified

  • Performance Tests

    v4 v5  
    33= What is a Performance Test =
    44
    5 A performance test measures the run-time performance and memory usage (coming!) of WebKit. Unlike [http://www.webkit.org/quality/testing.html regression tests] (a.k.a layout tests) or conformance tests such as of W3C's, it doesn't necessarily test the correctness of WebKit features. Since the run-time and memory used by each run of the same test may vary, we can't conclude whether a given performance test passed or failed by just looking at the single run. For this reason, performance tests yields "values" such as the time taken to run the test instead of simple PASS and FAIL.
     5A performance test measures the run-time performance and memory usage of WebKit. Unlike [http://www.webkit.org/quality/testing.html regression tests] (a.k.a layout tests) or conformance tests such as of W3C's, it doesn't necessarily test the correctness of WebKit features. Since the run-time and memory used by each run of the same test may vary, we can't conclude whether a given performance test passed or failed by just looking at a single run. For this reason, performance tests yields "values" such as the time taken to run the test instead of simple PASS and FAIL.
    66
    77= Performance Test Results  =
    88
    9 You can find the performance results on http://webkit-perf.appspot.com/
    10 
    11 The performance bots can be listed on the [http://build.webkit.org/waterfall?show=Apple%20Lion%20Release%20%28Perf%29&show=Apple%20MountainLion%20Release%20%28Perf%29&show=Chromium%20Linux%20Release%20%28Perf%29&show=Chromium%20Mac%20Release%20%28Perf%29&show=Chromium%20Win%20Release%20%28Perf%29&show=Qt%20Linux%2064-bit%20Release%20%28Perf%29&show=Qt%20Linux%2064-bit%20Release%20%28WebKit2%20Perf%29 buildbot page].
     9We have continuous performance test bots on [http://build.webkit.org/waterfall?show=Apple%20Lion%20Release%20%28Perf%29&show=Apple%20MountainLion%20Release%20%28Perf%29&show=Chromium%20Linux%20Release%20%28Perf%29&show=Chromium%20Mac%20Release%20%28Perf%29&show=Chromium%20Win%20Release%20%28Perf%29&show=Qt%20Linux%2064-bit%20Release%20%28Perf%29&show=Qt%20Linux%2064-bit%20Release%20%28WebKit2%20Perf%29 build.webkit.org]. You can see test results submitted by these bots on http://webkit-perf.appspot.com/
    1210
    1311{{{#!td style="padding: 1em;"
    … …  
    2624WebKit's performance tests can be ran by [http://trac.webkit.org/browser/trunk/Tools/Scripts/run-perf-tests run-perf-tests]. Specify a list of directories or performance tests to run a subset. e.g. {{{run-perf-tests PerformanceTests/DOM}}} or {{{run-perf-tests DOM}}} will only run tests in [http://trac.webkit.org/browser/trunk/PerformanceTests/DOM]. It will automatically build DumpRenderTree and WebKitTestRunner as needed just like {{{run-webkit-tests}}}.
    2725
    28 We also have performance bots on http://build.webkit.org/, which submits results to http://webkit-perf.appspot.com/
    29 
    3026== Reducing noise on your machine ==
    3127
    … …  
    3733sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist
    3834}}}
    39 (To re-enable, run {{{sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist}}}
     35(To re-enable, run the same command with {{{load}}} in place of {{{unload}}})
    4036
    4137== Aggregating and Comparing Results ==
    … …  
    4339If you're running a performance test locally in order to verify your patch doesn't regress or improves the performance of WebKit, you may find {{{--output-json-path}}} useful. Specify a file path such as {{{perf-test.json}}} and {{{run-perf-tests}}} will automatically store the results in the JSON file and creates {{{perf-test.html}}} that visualizes the test results. Execute {{{run-perf-tests}}} multiple times with the same output JSON path and it will automatically aggregate results in the JSON and the corresponding HTML document.
    4440
    45 Suppose we have two WebKit checkouts: one without the patch and another with the patch. By executing {{{run-perf-tests --output-json-path=/Users/WebKitten/perf-test.json}}} in both checkouts, we can easily compare the test results from two runs by opening {{{~/perf-test.html}}}.
     41Suppose we have two WebKit checkouts: one without a patch and another one with the patch applied. By executing {{{run-perf-tests --output-json-path=/Users/WebKitten/perf-test.json}}} in both checkouts, we can easily compare the test results from two runs by opening {{{~/perf-test.html}}}.
    4642
    4743You can also specify a build directory as follows along with the output JSON path:
    … …  
    4945run-perf-tests --no-build --build-directory /Users/WebKitten/MyCustomBuild/ --output-json-path=/Users/WebKitten/perf-test.json
    5046}}}
    51 This allows you compare results from different builds without having to locally build DumpRenderTree or WebKitTestRunner.
     47This allows you to compare results from different builds without having to locally build DumpRenderTree or WebKitTestRunner.
    5248
    5349=== Bisecting regressions ===
    … …  
    5551Suppose you're bisecting a regression for a performance regression on [http://trac.webkit.org/browser/trunk/PerformanceTests/Bindings/node-list-access.html Bindings/node-list-access.html] as seen [http://webkit-perf.appspot.com/graph.html#tests=%5B%5B2966378%2C2001%2C32196%5D%5D&sel=1343895254495.4062,1344094768078.8474,116.85483870967741,148.30645161290323&displayrange=30&datatype=running here]. Looking at the graph we see that the culprit lies [http://trac.webkit.org/log/?rev=124582&stop_rev=124567&verbose=on "between r124567 and r124582"].
    5652
    57 To bisect this regression, I prepare two WebKit checkouts one synced to r124567 and another synced to r124582, and run the following commands in each checkout:
     53To bisect this regression, I create two WebKit checkouts one synced to r124567 and another synced to r124582, and run the following commands in each checkout:
    5854{{{
    5955svn up PerformanceTests
    … …  
    6561This step automatically produces {{{/Users/WebKitten/Desktop/node-list-access.html}}} for me to compare the results, each results labeled r124567 and r124582 (you can use {{{--description}}} option to annotate the results further) and I can confirm whether the regression reproduces locally or not. Sometimes, regression doesn't produce on your local machine due to differences in environment such as compilers used, memory size, and CPU speed.
    6662
    67 Once I confirmed that the regression is reproducible on my machine, I start bisecting builds. Here, I sync the checkout initially synced to r124582 to a slightly older version, say, r124580 and generate results again as follows:
     63Once I confirmed that the regression is reproducible on my machine, I can start bisecting builds. Here, I sync the checkout initially synced to r124582 to a slightly older version, say, r124580 and generate results again as follows:
    6864{{{
    6965svn up -r 124580
    … …  
    7369Tools/Scripts/run-perf-tests --output-json-path=/Users/WebKitten/Desktop/node-list-access.json PerformanceTests/Bindings/node-list-access.html
    7470}}}
    75 I repeat this process until the results recovers to the level we had at r124567, at which I identified the culprit. I don't typically do a strict binary search on perf. regressions because that typically results in spending a lot of time on re-building WebKit.
     71I repeat this process until the results recovers to the level we had at r124567, at which I identified the culprit. I don't typically do a strict binary search on perf. regressions because that typically results to avoid rebuilding the entire WebKit all the time.
    7672
    7773= Writing a Performance Test Using runner.js =