Changes between Version 9 and Version 10 of Performance Tests


Ignore:
Timestamp:
Dec 25, 2014 12:04:19 PM (9 years ago)
Author:
rniwa@webkit.org
Comment:

Updated Writing a Performance Test Using runner.js.

Legend:

Unmodified
Added
Removed
Modified

  • Performance Tests

    v9 v10  
    7070= Writing a Performance Test Using runner.js =
    7171
    72 The easiest way to write a performance test is using [http://trac.webkit.org/browser/trunk/PerformanceTests/resources/runner.js runner.js], which provides {{{PerfTestRunner}}} with various utility functions. Again, the easiest way to use this class is to call {{{PerfTestRunner.measureRunsPerSecond}}} with a test function. For example, see [http://trac.webkit.org/browser/trunk/PerformanceTests/Parser/tiny-innerHTML.html Parser/tiny-innerHTML.html]:
    73 
     72The easiest way to write a performance test is using [http://trac.webkit.org/browser/trunk/PerformanceTests/resources/runner.js runner.js], which provides {{{PerfTestRunner}}} with various utility functions.  Once you wrote a test, put it inside [http://trac.webkit.org/browser/trunk/PerformanceTests PerformanceTests] directory to be ran by run-perf-tests and performance bots.
     73
     74== Measuring Runs Per Second ==
     75
     76Our preferred method of measurement is runs (function calls) per second.  With runner.js, we can measure this metric by calling {{{PerfTestRunner.measureRunsPerSecond}}} with a test function. {{{PerfTestRunner.measureRunsPerSecond}}} measures the time of times {{{run}}} function could be called in one second, and reports the statistics after repeating it 20 times (configurable via {{{run-perf-tests}}}). The statistics includes arithmetic mean, standard deviation, median, minimum, and maximum values.
     77
     78For example, see [http://trac.webkit.org/browser/trunk/PerformanceTests/Parser/tiny-innerHTML.html Parser/tiny-innerHTML.html]:
    7479{{{
    7580<!DOCTYPE html>
    … …  
    9095}}}
    9196
    92 {{{PerfTestRunner.measureRunsPerSecond}}} measures the time of times {{{run}}} function could be called in one second, and reports the statistics after repeating it 20 times. The statistics includes arithmetic mean, standard deviation, median, minimum, and maximum values.  You can optionally specify the number of iterations (defaults to 2) and the test description as follows:
    93 {{{
    94 PerfTestRunner.measureRunsPerSecond({
    95     runCount: 50,
    96     description: "Test with lots of iterations",
    97     run: function () {...}
     97== Measuring Time ==
     98
     99In some tests, however, we cannot call the {{{run}}} function for an arbitrary number of times as done in {{{measureRunsPerSecond}}}.  In those tests, we can use {{{PerfTestRunner.measureTime}}} to measure the time {{{run}}} took to execute. {{{measureTime}}} calls the specified function once in each iteration and runs 20 iterations by default.
     100
     101Note that the runtime of a function gets smaller relative to the granularity of time measurement we can make as the WebKit's performance (or of the machines that run performance tests) improves.
     102
     103== Measuring Asynchronous Results ==
     104
     105In some tests such as ones that measure fps, values are measured asynchronously.  In those tests, we can use {{{PerfTestRunner.prepareToMeasureValuesAsync}}} and {{{PerfTestRunner.measureValueAsync}}} to report measured value at an arbitrary time.  At the beginning of a test, call {{{PerfTestRunner.prepareToMeasureValuesAsync}}} with an object with {{{unit}}} property, which specifies the name of the unit (either one of "ms", "fps", or "runs/s").  Call {{{PerfTestRunner.measureValueAsync}}} as a newly measured value comes in.  Once enough values are measured (20 by default), {{{PerfTestRunner.measureValueAsync}}} will automatically stop the test; do not expect or manually track the number of iterations in your test as this must be configurable via {{{run-perf-tests}}}.
     106
     107For example, see [http://trac.webkit.org/browser/trunk/PerformanceTests/Interactive/SelectAll.html Interactive/SelectAll.html]:
     108{{{
     109<!DOCTYPE html>
     110<html>
     111<body>
     112<script src="../resources/runner.js"></script>
     113<script>
     114
     115PerfTestRunner.prepareToMeasureValuesAsync({
     116    unit: 'ms',
     117    done: function () {
     118        var iframe = document.querySelector('iframe');
     119        iframe.parentNode.removeChild(iframe);
     120    }
    98121});
    99 }}}
    100 
    101 There is also {{{PerfTestRunner.run}}}, which measures the run time of a function. The use of this method is '''discouraged''' because the time gets smaller relative to the granularity of time measurement we can make as the WebKit's performance (or of the machines that run performance tests) improves. Nonetheless, it has been used in some animation tests where we cannot dynamically adjust the number of function calls as done in {{{measureRunsPerSecond}}}.
    102 
    103 {{{run}}} calls the specified function 10 times in each iteration and runs 20 iterations by default.  You can optionally specify how many times the function is called in each iteration and how many iterations are executed. You can also specify a function to be called after all iterations.
    104 {{{
    105 PerfTestRunner.run(runFunction, callsPerIteration, numberOfIterations, doneFunction)
    106 }}}
    107 
    108 Once you wrote a test, put it inside [http://trac.webkit.org/browser/trunk/PerformanceTests PerformanceTests] directory to be ran by run-perf-tests and performance bots.
     122
     123function runTest() {
     124    var iframe = document.querySelector('iframe');
     125    iframe.contentWindow.getSelection().removeAllRanges();
     126    iframe.contentDocument.body.offsetTop;
     127
     128    setTimeout(function () {
     129        var startTime = PerfTestRunner.now();
     130        iframe.contentDocument.execCommand('SelectAll');
     131        iframe.contentDocument.body.offsetTop;
     132        setTimeout(function () {
     133            if (!PerfTestRunner.measureValueAsync(PerfTestRunner.now() - startTime))
     134                return;
     135            PerfTestRunner.gc();
     136            setTimeout(runTest, 0);
     137        }, 0);
     138    }, 0);
     139}
     140
     141</script>
     142<iframe src="../Parser/resources/html5.html" onload="runTest()" width="800" height="600">
     143</body>
     144</html>
     145}}}
     146
     147== Optional Arguments ==
     148
     149{{{measureRunsPerSecond}}}, {{{measureTime}}}, and {{{PerfTestRunner.prepareToMeasureValuesAsync}}} described above optionally support the following arguments (as properties in the object they take):
     150 * {{{setup}}} - In {{{measureRunsPerSecond}}} and {{{measureTime}}}, this function gets called before the start of each iteration.  With {{{measureRunsPerSecond}}}, this function gets called exactly once before {{{run}}} gets called many times in each iteration.  If there is some work to be done before {{{run}}} can be called each time, use {{{measureTime}}} instead.  {{{PerfTestRunner.prepareToMeasureValuesAsync}}} ignores this function as it can't know when the next iteration starts.
     151 * {{{done}}} - This function gets called once all iterations are finished.
     152 * {{{description}}} - The description of the test.  {{{run-perf-tests}}} will print out this text.
     153
    109154
    110155= Replay Performance Tests =