Changeset 116916 in webkit


Ignore:
Timestamp:
May 13, 2012 10:23:37 PM (12 years ago)
Author:
rniwa@webkit.org
Message:

performance tests should be able to measure runs/sec rather than time
​https://bugs.webkit.org/show_bug.cgi?id=86021

Reviewed by Ojan Vafai.

PerformanceTests:

Add PerfTestRunner.runPerSecond. It uses _runLoop but replaces _runner by _perSecondRunner
to compute runs/s of runFunction.

When _perSecondRunner is called for the first time, i.e. _completedRuns is 0 (notice this is -1
in regular run/_runner), it slowly increases the number of function calls to runFunction between
time measurements in order to discount the time used by new Date() calls themselves until the
total time spent reaches 100 milliseconds.

By default, runPerSecond runs the test for at least 750 milliseconds in each run, and executes
21 runs, yielding the total run time of roughly 18 seconds. This is significantly faster than
most of existing performance tests. Also see ​http://ejohn.org/blog/accuracy-of-javascript-time/.

Finally, refactored the existing methods of PerfTestRunner to allow "runs/s" unit and share code.

  • Layout/flexbox-column-nowrap.html:
  • Layout/flexbox-column-wrap.html:
  • Layout/flexbox-row-nowrap.html:
  • Layout/flexbox-row-wrap.html:
  • resources/runner.js:

(PerfTestRunner.computeStatistics): Takes unit.
(PerfTestRunner.logStatistics): Ditto.
(PerfTestRunner._runLoop):
(PerfTestRunner._runner):
(PerfTestRunner.runPerSecond): Added.
(PerfTestRunner._perSecondRunner): Added. Called by _runLoop.
(PerfTestRunner._perSecondRunnerIterator): Added.

Tools:

Allow " runs/s" or " ms" to appear after numerical values in tests.

  • Scripts/webkitpy/performance_tests/perftest.py:

(PerfTest):

LayoutTests:

Add tests for PerfTestRunner.runPerSecond.

  • fast/harness/perftests/runs-per-second-iterations-expected.txt: Added.
  • fast/harness/perftests/runs-per-second-iterations.html: Added.
  • fast/harness/perftests/runs-per-second-log-expected.txt: Added.
  • fast/harness/perftests/runs-per-second-log.html: Added.
Location:
trunk
Files:
4 added
9 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r116914 r116916  
     12012-05-13  Ryosuke Niwa  <rniwa@webkit.org>
     2
     3        performance tests should be able to measure runs/sec rather than time
     4        https://bugs.webkit.org/show_bug.cgi?id=86021
     5
     6        Reviewed by Ojan Vafai.
     7
     8        Add tests for PerfTestRunner.runPerSecond.
     9
     10        * fast/harness/perftests/runs-per-second-iterations-expected.txt: Added.
     11        * fast/harness/perftests/runs-per-second-iterations.html: Added.
     12        * fast/harness/perftests/runs-per-second-log-expected.txt: Added.
     13        * fast/harness/perftests/runs-per-second-log.html: Added.
     14
    1152012-05-13  Mike Lawther  <mikelawther@chromium.org>
    216

  • trunk/PerformanceTests/ChangeLog

    r116564 r116916  
     12012-05-13  Ryosuke Niwa  <rniwa@webkit.org>
     2
     3        performance tests should be able to measure runs/sec rather than time
     4        https://bugs.webkit.org/show_bug.cgi?id=86021
     5
     6        Reviewed by Ojan Vafai.
     7
     8        Add PerfTestRunner.runPerSecond. It uses _runLoop but replaces _runner by _perSecondRunner
     9        to compute runs/s of runFunction.
     10
     11        When _perSecondRunner is called for the first time, i.e. _completedRuns is 0 (notice this is -1
     12        in regular run/_runner), it slowly increases the number of function calls to runFunction between
     13        time measurements in order to discount the time used by new Date() calls themselves until the
     14        total time spent reaches 100 milliseconds.
     15
     16        By default, runPerSecond runs the test for at least 750 milliseconds in each run, and executes
     17        21 runs, yielding the total run time of roughly 18 seconds. This is significantly faster than
     18        most of existing performance tests. Also see http://ejohn.org/blog/accuracy-of-javascript-time/.
     19
     20        Finally, refactored the existing methods of PerfTestRunner to allow "runs/s" unit and share code.
     21
     22        * Layout/flexbox-column-nowrap.html:
     23        * Layout/flexbox-column-wrap.html:
     24        * Layout/flexbox-row-nowrap.html:
     25        * Layout/flexbox-row-wrap.html:
     26        * resources/runner.js:
     27        (PerfTestRunner.computeStatistics): Takes unit.
     28        (PerfTestRunner.logStatistics): Ditto.
     29        (PerfTestRunner._runLoop):
     30        (PerfTestRunner._runner):
     31        (PerfTestRunner.runPerSecond): Added.
     32        (PerfTestRunner._perSecondRunner): Added. Called by _runLoop.
     33        (PerfTestRunner._perSecondRunnerIterator): Added.
     34
    1352012-05-09  Tony Chang  <tony@chromium.org>
    236

  • trunk/PerformanceTests/Layout/flexbox-column-nowrap.html

    r116564 r116916  
    2929}
    3030
    31 PerfTestRunner.run(runTest, 180, 20, function() {
     31PerfTestRunner.runPerSecond({run: runTest, done: function() {
    3232    document.getElementById("flexbox").style.display = 'none';
    33 });
     33}});
    3434</script>
    3535</html>

  • trunk/PerformanceTests/Layout/flexbox-column-wrap.html

    r116564 r116916  
    2929}
    3030
    31 PerfTestRunner.run(runTest, 180, 20, function() {
     31PerfTestRunner.runPerSecond({run: runTest, done:function() {
    3232    document.getElementById("flexbox").style.display = 'none';
    33 });
     33}});
    3434</script>
    3535</html>

  • trunk/PerformanceTests/Layout/flexbox-row-nowrap.html

    r116564 r116916  
    2929}
    3030
    31 PerfTestRunner.run(runTest, 100, 20, function() {
     31PerfTestRunner.runPerSecond({run:runTest, done:function() {
    3232    document.getElementById("flexbox").style.display = 'none';
    33 });
     33}});
    3434</script>
    3535</html>

  • trunk/PerformanceTests/Layout/flexbox-row-wrap.html

    r116564 r116916  
    2929}
    3030
    31 PerfTestRunner.run(runTest, 100, 20, function() {
     31PerfTestRunner.runPerSecond({run: runTest, done:function() {
    3232    document.getElementById("flexbox").style.display = 'none';
    33 });
     33}});
    3434</script>
    3535</html>

  • trunk/PerformanceTests/resources/runner.js

    r115942 r116916  
    5050}
    5151
    52 PerfTestRunner.computeStatistics = function (times) {
     52PerfTestRunner.computeStatistics = function (times, unit) {
    5353    var data = times.slice();
    5454
    … …  
    7777    result.variance = squareSum / data.length;
    7878    result.stdev = Math.sqrt(result.variance);
    79     result.unit = "ms";
     79    result.unit = unit || "ms";
    8080
    8181    return result;
    … …  
    8484PerfTestRunner.logStatistics = function (times) {
    8585    this.log("");
    86     var statistics = this.computeStatistics(times);
     86    var statistics = this.computeStatistics(times, this.unit);
    8787    this.printStatistics(statistics);
    8888}
    … …  
    118118        window.setTimeout(function () { PerfTestRunner._runner(); }, 0);
    119119    } else {
    120         this.logStatistics(this._times);
     120        this.logStatistics(this._results);
    121121        this._doneFunction();
    122122        if (window.layoutTestController)
    … …  
    141141    var time = totalTime ? totalTime : Date.now() - start;
    142142
     143    this.ignoreWarmUpAndLog(time);
     144    this._runLoop();
     145}
     146
     147PerfTestRunner.ignoreWarmUpAndLog = function (result) {
    143148    this._completedRuns++;
     149
     150    var labeledResult = result + " " + this.unit;
    144151    if (this._completedRuns <= 0)
    145         this.log("Ignoring warm-up run (" + time + ")");
     152        this.log("Ignoring warm-up run (" + labeledResult + ")");
    146153    else {
    147         this._times.push(time);
    148         this.log(time);
    149     }
     154        this._results.push(result);
     155        this.log(labeledResult);
     156    }
     157}
     158
     159PerfTestRunner.initAndStartLoop = function() {
     160    this._completedRuns = -1;
     161    this.customRunFunction = null;
     162    this._results = [];
     163    this.log("Running " + this._runCount + " times");
    150164    this._runLoop();
    151165}
    … …  
    156170    this._runCount = runCount || 20;
    157171    this._doneFunction = doneFunction || function () {};
    158     this._completedRuns = -1;
    159     this.customRunFunction = null;
    160     this._times = [];
    161 
    162     this.log("Running " + this._runCount + " times");
     172    this.unit = 'ms';
     173    this.initAndStartLoop();
     174}
     175
     176PerfTestRunner.runPerSecond = function (test) {
     177    this._doneFunction = function () { if (test.done) test.done(); };
     178    this._runCount = test.runCount || 20;
     179    this._callsPerIteration = 1;
     180    this.unit = 'runs/s';
     181
     182    this._test = test;
     183    this._runner = this._perSecondRunner;
     184    this.initAndStartLoop();
     185}
     186
     187PerfTestRunner._perSecondRunner = function () {
     188    var timeToRun = this._test.timeToRun || 750;
     189    var totalTime = 0;
     190    var i = 0;
     191    var callsPerIteration = this._callsPerIteration;
     192
     193    if (this._test.setup)
     194        this._test.setup();
     195
     196    while (totalTime < timeToRun) {
     197        totalTime += this._perSecondRunnerIterator(callsPerIteration);
     198        i += callsPerIteration;
     199        if (this._completedRuns <= 0 && totalTime < 100)
     200            callsPerIteration = Math.max(10, 2 * callsPerIteration);
     201    }
     202    this._callsPerIteration = callsPerIteration;
     203
     204    this.ignoreWarmUpAndLog(i * 1000 / totalTime);
    163205    this._runLoop();
     206}
     207
     208PerfTestRunner._perSecondRunnerIterator = function (callsPerIteration) {
     209    var startTime = Date.now();
     210    for (var i = 0; i < callsPerIteration; i++)
     211        this._test.run();
     212    return Date.now() - startTime;
    164213}
    165214

  • trunk/Tools/ChangeLog

    r116859 r116916  
     12012-05-13  Ryosuke Niwa  <rniwa@webkit.org>
     2
     3        performance tests should be able to measure runs/sec rather than time
     4        https://bugs.webkit.org/show_bug.cgi?id=86021
     5
     6        Reviewed by Ojan Vafai.
     7
     8        Allow " runs/s" or " ms" to appear after numerical values in tests.
     9
     10        * Scripts/webkitpy/performance_tests/perftest.py:
     11        (PerfTest):
     12
    1132012-05-12  Tim Horton  <timothy_horton@apple.com>
    214

  • trunk/Tools/Scripts/webkitpy/performance_tests/perftest.py

    r115466 r116916  
    7575        re.compile(r'^Ignoring warm-up '),
    7676        re.compile(r'^Info:'),
    77         re.compile(r'^\d+(.\d+)?$'),
     77        re.compile(r'^\d+(.\d+)?(\s*(runs\/s|ms))?$'),
    7878        # Following are for handle existing test like Dromaeo
    7979        re.compile(re.escape("""main frame - has 1 onunload handler(s)""")),
Note: See TracChangeset for help on using the changeset viewer.