Changeset 144478 in webkit


Ignore:
Timestamp:
Mar 1, 2013 12:04:13 PM (11 years ago)
Author:
rniwa@webkit.org
Message:

Don't return statistics in PerfTest.run
https://bugs.webkit.org/show_bug.cgi?id=111145

Reviewed by Dirk Pranke.

Simply return a list of values obtained in each iteration from PerfTest.run.
Also simplify various methods as needed.

  • Scripts/webkitpy/performance_tests/perftest.py:

(PerfTestMetric.init): Moved the code to convert 'Time' metric to 'FrameRate' and 'Runs'
as needed from PerfTestsRunner._generate_results_dict.

(PerfTestMetric.legacy_chromium_bot_compatible_test_name): Always append the metric name now
that we don't have to worry about the backward compatibility here.

(PerfTestMetric.iteration_values): Added.
(PerfTestMetric.unit): Added.
(PerfTestMetric.metric_to_unit):
(PerfTestMetric.time_unit_to_metric): Extracted from PerfTestsRunner._generate_results_dict.

(PerfTest.run):
(PerfTest.log_statistics): Merged compute_statistics and self.output_statistics.

  • Scripts/webkitpy/performance_tests/perftest_unittest.py:

(TestPerfTestMetric.test_init_set_missing_unit):
(TestPerfTestMetric.test_init_set_time_metric): Added.
(TestPerfTestMetric.test_legacy_chromium_bot_compatible_test_name):
(TestPerfTestMetric.test_append):
(TestPerfTestMetric.test_compute_statistics): Removed. Integration tests check some of these
test cases and it's not critical to keep math right as they're not used in output JSON anymore.
(TestPerfTest._assert_results_are_correct):
(TestReplayPerfTest.test_run_with_driver_accumulates_results):
(TestReplayPerfTest.test_run_with_driver_accumulates_memory_results):

  • Scripts/webkitpy/performance_tests/perftestsrunner.py:

(PerfTestsRunner._generate_results_dict):

  • Scripts/webkitpy/performance_tests/perftestsrunner_integrationtest.py:

(EventTargetWrapperTestData): The metric name shows up at the end of the test name as expected.
(SomeParserTestData): Ditto.
(MemoryTestData): Ditto.

Location:
trunk/Tools
Files:
5 edited

Legend:

Unmodified
Added
Removed
  • trunk/Tools/ChangeLog

    r144441 r144478  
     12013-03-01  Ryosuke Niwa  <rniwa@webkit.org>
     2
     3        Don't return statistics in PerfTest.run
     4        https://bugs.webkit.org/show_bug.cgi?id=111145
     5
     6        Reviewed by Dirk Pranke.
     7
     8        Simply return a list of values obtained in each iteration from PerfTest.run.
     9        Also simplify various methods as needed.
     10
     11        * Scripts/webkitpy/performance_tests/perftest.py:
     12        (PerfTestMetric.__init__): Moved the code to convert 'Time' metric to 'FrameRate' and 'Runs'
     13        as needed from PerfTestsRunner._generate_results_dict.
     14
     15        (PerfTestMetric.legacy_chromium_bot_compatible_test_name): Always append the metric name now
     16        that we don't have to worry about the backward compatibility here.
     17
     18        (PerfTestMetric.iteration_values): Added.
     19        (PerfTestMetric.unit): Added.
     20        (PerfTestMetric.metric_to_unit):
     21        (PerfTestMetric.time_unit_to_metric): Extracted from PerfTestsRunner._generate_results_dict.
     22
     23        (PerfTest.run):
     24        (PerfTest.log_statistics): Merged compute_statistics and self.output_statistics.
     25
     26        * Scripts/webkitpy/performance_tests/perftest_unittest.py:
     27        (TestPerfTestMetric.test_init_set_missing_unit):
     28        (TestPerfTestMetric.test_init_set_time_metric): Added.
     29        (TestPerfTestMetric.test_legacy_chromium_bot_compatible_test_name):
     30        (TestPerfTestMetric.test_append):
     31        (TestPerfTestMetric.test_compute_statistics): Removed. Integration tests check some of these
     32        test cases and it's not critical to keep math right as they're not used in output JSON anymore.
     33        (TestPerfTest._assert_results_are_correct):
     34        (TestReplayPerfTest.test_run_with_driver_accumulates_results):
     35        (TestReplayPerfTest.test_run_with_driver_accumulates_memory_results):
     36
     37        * Scripts/webkitpy/performance_tests/perftestsrunner.py:
     38        (PerfTestsRunner._generate_results_dict):
     39
     40        * Scripts/webkitpy/performance_tests/perftestsrunner_integrationtest.py:
     41        (EventTargetWrapperTestData): The metric name shows up at the end of the test name as expected.
     42        (SomeParserTestData): Ditto.
     43        (MemoryTestData): Ditto.
     44
    1452013-03-01  Allan Sandfeld Jensen  <allan.jensen@digia.com>
    246
  • trunk/Tools/Scripts/webkitpy/performance_tests/perftest.py

    r144363 r144478  
    5555class PerfTestMetric(object):
    5656    def __init__(self, metric, unit=None, iterations=None):
    57         self._metric = metric
     57        # FIXME: Fix runner.js to report correct metric names
    5858        self._iterations = iterations or []
    5959        self._unit = unit or self.metric_to_unit(metric)
     60        self._metric = self.time_unit_to_metric(self._unit) if metric == 'Time' else metric
    6061
    6162    def metric(self):
     
    6869    def legacy_chromium_bot_compatible_test_name(self, test_name_with_extension):
    6970        test_name = re.sub(r'\.\w+$', '', test_name_with_extension)
    70         return test_name if self._metric == 'Time' else test_name + ':' + self._metric
     71        return test_name + ':' + self._metric
    7172
    7273    def append(self, value):
    7374        self._iterations.append(value)
    7475
    75     def to_dict(self):
    76         assert self.has_values()
    77         statistics = self.compute_statistics(self._iterations)
    78         statistics['unit'] = self._unit
    79         statistics['values'] = self._iterations
    80         return statistics
    81 
    82     @classmethod
    83     def metric_to_unit(cls, metric):
     76    def iteration_values(self):
     77        return self._iterations
     78
     79    def unit(self):
     80        return self._unit
     81
     82    @staticmethod
     83    def metric_to_unit(metric):
    8484        assert metric in ('Time', 'Malloc', 'JSHeap')
    8585        return 'ms' if metric == 'Time' else 'bytes'
    8686
    8787    @staticmethod
    88     def compute_statistics(values):
    89         sorted_values = sorted(values)
    90 
    91         # Compute the mean and variance using Knuth's online algorithm (has good numerical stability).
    92         squareSum = 0
    93         mean = 0
    94         for i, time in enumerate(sorted_values):
    95             delta = time - mean
    96             sweep = i + 1.0
    97             mean += delta / sweep
    98             squareSum += delta * (time - mean)
    99 
    100         middle = int(len(sorted_values) / 2)
    101         result = {'avg': sum(sorted_values) / len(values),
    102             'min': sorted_values[0],
    103             'max': sorted_values[-1],
    104             'median': sorted_values[middle] if len(sorted_values) % 2 else (sorted_values[middle - 1] + sorted_values[middle]) / 2,
    105             'stdev': math.sqrt(squareSum / (len(sorted_values) - 1)) if len(sorted_values) > 1 else 0}
    106         return result
     88    def time_unit_to_metric(unit):
     89        return {'fps': 'FrameRate', 'runs/s': 'Runs', 'ms': 'Time'}[unit]
    10790
    10891
     
    139122            return metrics
    140123
     124        should_log = not self._port.get_option('profile')
     125        if should_log and self._description:
     126            _log.info('DESCRIPTION: %s' % self._description)
     127
    141128        results = {}
    142129        for metric in metrics:
    143130            legacy_test_name = metric.legacy_chromium_bot_compatible_test_name(self.test_name())
    144             results[legacy_test_name] = metric.to_dict()
    145 
    146         if not self._port.get_option('profile'):
    147             if self._description:
    148                 _log.info('DESCRIPTION: %s' % self._description)
    149             for result_name in sorted(results.keys()):
    150                 self.output_statistics(result_name, results[result_name])
     131            results[legacy_test_name] = metric.iteration_values()
     132            if should_log:
     133                self.log_statistics(legacy_test_name.replace(':', ': ').replace('/', ': '),
     134                    metric.iteration_values(), metric.unit())
    151135
    152136        return results
     137
     138    @staticmethod
     139    def log_statistics(test_name, values, unit):
     140        sorted_values = sorted(values)
     141
     142        # Compute the mean and variance using Knuth's online algorithm (has good numerical stability).
     143        square_sum = 0
     144        mean = 0
     145        for i, time in enumerate(sorted_values):
     146            delta = time - mean
     147            sweep = i + 1.0
     148            mean += delta / sweep
     149            square_sum += delta * (time - mean)
     150
     151        middle = int(len(sorted_values) / 2)
     152        mean = sum(sorted_values) / len(values)
     153        median = sorted_values[middle] if len(sorted_values) % 2 else (sorted_values[middle - 1] + sorted_values[middle]) / 2
     154        stdev = math.sqrt(square_sum / (len(sorted_values) - 1)) if len(sorted_values) > 1 else 0
     155
     156        _log.info('RESULT %s= %s %s' % (test_name, mean, unit))
     157        _log.info('median= %s %s, stdev= %s %s, min= %s %s, max= %s %s' %
     158            (median, unit, stdev, unit, sorted_values[0], unit, sorted_values[-1], unit))
    153159
    154160    _description_regex = re.compile(r'^Description: (?P<description>.*)$', re.IGNORECASE)
     
    243249            output.text = '\n'.join([line for line in re.split('\n', output.text) if not self._should_ignore_line(self._lines_to_ignore_in_parser_result, line)])
    244250
    245     def output_statistics(self, test_name, results):
    246         unit = results['unit']
    247         _log.info('RESULT %s= %s %s' % (test_name.replace(':', ': ').replace('/', ': '), results['avg'], unit))
    248         _log.info(', '.join(['%s= %s %s' % (key, results[key], unit) for key in self._statistics_keys[1:5]]))
    249 
    250251
    251252class ChromiumStylePerfTest(PerfTest):
  • trunk/Tools/Scripts/webkitpy/performance_tests/perftest_unittest.py

    r144363 r144478  
    5151class TestPerfTestMetric(unittest.TestCase):
    5252    def test_init_set_missing_unit(self):
    53         self.assertEqual(PerfTestMetric('Time', iterations=[1, 2, 3, 4, 5]).to_dict()['unit'], 'ms')
    54         self.assertEqual(PerfTestMetric('Malloc', iterations=[1, 2, 3, 4, 5]).to_dict()['unit'], 'bytes')
    55         self.assertEqual(PerfTestMetric('JSHeap', iterations=[1, 2, 3, 4, 5]).to_dict()['unit'], 'bytes')
     53        self.assertEqual(PerfTestMetric('Time', iterations=[1, 2, 3, 4, 5]).unit(), 'ms')
     54        self.assertEqual(PerfTestMetric('Malloc', iterations=[1, 2, 3, 4, 5]).unit(), 'bytes')
     55        self.assertEqual(PerfTestMetric('JSHeap', iterations=[1, 2, 3, 4, 5]).unit(), 'bytes')
     56
     57    def test_init_set_time_metric(self):
     58        self.assertEqual(PerfTestMetric('Time', 'ms').metric(), 'Time')
     59        self.assertEqual(PerfTestMetric('Time', 'fps').metric(), 'FrameRate')
     60        self.assertEqual(PerfTestMetric('Time', 'runs/s').metric(), 'Runs')
    5661
    5762    def test_legacy_chromium_bot_compatible_test_name(self):
    58         self.assertEqual(PerfTestMetric('Time').legacy_chromium_bot_compatible_test_name('test'), 'test')
     63        self.assertEqual(PerfTestMetric('Time').legacy_chromium_bot_compatible_test_name('test'), 'test:Time')
    5964        self.assertEqual(PerfTestMetric('Malloc').legacy_chromium_bot_compatible_test_name('test'), 'test:Malloc')
    6065        self.assertEqual(PerfTestMetric('JSHeap').legacy_chromium_bot_compatible_test_name('test'), 'test:JSHeap')
     
    7479        self.assertTrue(metric.has_values())
    7580        self.assertFalse(metric2.has_values())
    76         self.assertEqual(metric.to_dict()['values'], [1])
     81        self.assertEqual(metric.iteration_values(), [1])
    7782        metric.append(2)
    78         self.assertEqual(metric.to_dict()['values'], [1, 2])
     83        self.assertEqual(metric.iteration_values(), [1, 2])
    7984
    8085        metric2.append(3)
    8186        self.assertTrue(metric2.has_values())
    82         self.assertEqual(metric.to_dict()['values'], [1, 2])
    83         self.assertEqual(metric2.to_dict()['values'], [3])
    84 
    85     def test_compute_statistics(self):
    86         def compute_statistics(values):
    87             statistics = PerfTestMetric.compute_statistics(map(lambda x: float(x), values))
    88             return json.loads(json.dumps(statistics))
    89 
    90         statistics = compute_statistics([10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 20, 19, 18, 17, 16, 15, 14, 13, 12, 11])
    91         self.assertItemsEqual(statistics.keys(), ['avg', 'max', 'median', 'min', 'stdev'])
    92         self.assertEqual(statistics['avg'], 10.5)
    93         self.assertEqual(statistics['min'], 1)
    94         self.assertEqual(statistics['max'], 20)
    95         self.assertEqual(statistics['median'], 10.5)
    96         self.assertEqual(compute_statistics([8, 9, 10, 11, 12])['avg'], 10)
    97         self.assertEqual(compute_statistics([8, 9, 10, 11, 12] * 4)['avg'], 10)
    98         self.assertEqual(compute_statistics([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19])['avg'], 10)
    99         self.assertEqual(compute_statistics([1, 5, 2, 8, 7])['median'], 5)
    100         self.assertEqual(compute_statistics([1, 6, 2, 8, 7, 2])['median'], 4)
    101         self.assertAlmostEqual(statistics['stdev'], math.sqrt(35))
    102         self.assertAlmostEqual(compute_statistics([1])['stdev'], 0)
    103         self.assertAlmostEqual(compute_statistics([1, 2, 3, 4, 5, 6])['stdev'], math.sqrt(3.5))
    104         self.assertAlmostEqual(compute_statistics([4, 2, 5, 8, 6])['stdev'], math.sqrt(5))
     87        self.assertEqual(metric.iteration_values(), [1, 2])
     88        self.assertEqual(metric2.iteration_values(), [3])
    10589
    10690
     
    11094        parsed_results = test._run_with_driver(None, None)
    11195        self.assertEqual(len(parsed_results), 1)
    112         some_test_results = parsed_results[0].to_dict()
    113         self.assertItemsEqual(some_test_results.keys(), ['avg', 'max', 'median', 'min', 'stdev', 'unit', 'values'])
    114         self.assertEqual(some_test_results['values'], [1080, 1120, 1095, 1101, 1104])
    115         self.assertEqual(some_test_results['min'], 1080)
    116         self.assertEqual(some_test_results['max'], 1120)
    117         self.assertEqual(some_test_results['avg'], 1100)
    118         self.assertEqual(some_test_results['median'], 1101)
    119         self.assertAlmostEqual(some_test_results['stdev'], 14.50862, places=5)
    120         self.assertEqual(some_test_results['unit'], 'ms')
     96        self.assertEqual(parsed_results[0].iteration_values(), [1080, 1120, 1095, 1101, 1104])
    12197
    12298    def test_parse_output(self):
     
    342318        self.assertEqual(len(metrics), 1)
    343319        self.assertEqual(metrics[0].metric(), 'Time')
    344         self.assertEqual(metrics[0].to_dict(), {'max': 20000, 'avg': 11000.0, 'median': 11000, 'stdev': 5627.314338711378, 'min': 2000, 'unit': 'ms',
    345             'values': [float(i * 1000) for i in range(2, 21)]})
     320        self.assertEqual(metrics[0].iteration_values(), [float(i * 1000) for i in range(2, 21)])
    346321
    347322    def test_run_with_driver_accumulates_memory_results(self):
     
    369344        self.assertEqual(len(metrics), 3)
    370345        self.assertEqual(metrics[0].metric(), 'Time')
    371         self.assertEqual(metrics[0].to_dict(), {'max': 20000, 'avg': 11000.0, 'median': 11000, 'stdev': 5627.314338711378, 'min': 2000, 'unit': 'ms',
    372             'values': [float(i * 1000) for i in range(2, 21)]})
     346        self.assertEqual(metrics[0].iteration_values(), [float(i * 1000) for i in range(2, 21)])
    373347        self.assertEqual(metrics[1].metric(), 'Malloc')
    374         self.assertEqual(metrics[1].to_dict(), {'max': 10, 'avg': 10.0, 'median': 10, 'min': 10, 'stdev': 0.0, 'unit': 'bytes',
    375             'values': [float(10)] * 19})
     348        self.assertEqual(metrics[1].iteration_values(), [float(10)] * 19)
    376349        self.assertEqual(metrics[2].metric(), 'JSHeap')
    377         self.assertEqual(metrics[2].to_dict(), {'max': 5, 'avg': 5.0, 'median': 5, 'min': 5, 'stdev': 0.0, 'unit': 'bytes',
    378             'values': [float(5)] * 19})
     350        self.assertEqual(metrics[2].iteration_values(), [float(5)] * 19)
    379351
    380352    def test_prepare_fails_when_wait_until_ready_fails(self):
  • trunk/Tools/Scripts/webkitpy/performance_tests/perftestsrunner.py

    r144421 r144478  
    256256
    257257        # FIXME: Make this function shorter once we've transitioned to use perf.webkit.org.
    258         for metric_full_name, result in self._results.iteritems():
    259             if not isinstance(result, dict):  # We can't reports results without indivisual measurements.
     258        for metric_full_name, iteration_values in self._results.iteritems():
     259            if not isinstance(iteration_values, list):  # We can't reports results without individual measurements.
    260260                continue
    261261
    262262            assert metric_full_name.count(':') <= 1
    263263            test_full_name, _, metric = metric_full_name.partition(':')
    264             if not metric:
    265                 metric = {'fps': 'FrameRate', 'runs/s': 'Runs', 'ms': 'Time'}[result['unit']]
    266264
    267265            tests = contents['tests']
     
    279277                    current_test.setdefault('metrics', {})
    280278                    assert metric not in current_test['metrics']
    281                     current_test['metrics'][metric] = {'current': result['values']}
     279                    current_test['metrics'][metric] = {'current': iteration_values}
    282280                else:
    283281                    current_test.setdefault('tests', {})
  • trunk/Tools/Scripts/webkitpy/performance_tests/perftestsrunner_integrationtest.py

    r144421 r144478  
    8787
    8888    output = """Running Bindings/event-target-wrapper.html (1 of 2)
    89 RESULT Bindings: event-target-wrapper= 1490.0 ms
     89RESULT Bindings: event-target-wrapper: Time= 1490.0 ms
    9090median= 1488.0 ms, stdev= 15.13935 ms, min= 1471.0 ms, max= 1510.0 ms
    9191Finished: 0.1 s
     
    111111
    112112    output = """Running Parser/some-parser.html (2 of 2)
    113 RESULT Parser: some-parser= 1100.0 ms
     113RESULT Parser: some-parser: Time= 1100.0 ms
    114114median= 1101.0 ms, stdev= 14.50861 ms, min= 1080.0 ms, max= 1120.0 ms
    115115Finished: 0.1 s
     
    149149    output = """Running 1 tests
    150150Running Parser/memory-test.html (1 of 1)
    151 RESULT Parser: memory-test= 1100.0 ms
     151RESULT Parser: memory-test: Time= 1100.0 ms
    152152median= 1101.0 ms, stdev= 14.50861 ms, min= 1080.0 ms, max= 1120.0 ms
    153153RESULT Parser: memory-test: JSHeap= 830000.0 bytes
Note: See TracChangeset for help on using the changeset viewer.