Changeset 144478 in webkit
- Timestamp:
- Mar 1, 2013 12:04:13 PM (11 years ago)
- Location:
- trunk/Tools
- Files:
-
- 5 edited
Legend:
- Unmodified
- Added
- Removed
-
trunk/Tools/ChangeLog
r144441 r144478 1 2013-03-01 Ryosuke Niwa <rniwa@webkit.org> 2 3 Don't return statistics in PerfTest.run 4 https://bugs.webkit.org/show_bug.cgi?id=111145 5 6 Reviewed by Dirk Pranke. 7 8 Simply return a list of values obtained in each iteration from PerfTest.run. 9 Also simplify various methods as needed. 10 11 * Scripts/webkitpy/performance_tests/perftest.py: 12 (PerfTestMetric.__init__): Moved the code to convert 'Time' metric to 'FrameRate' and 'Runs' 13 as needed from PerfTestsRunner._generate_results_dict. 14 15 (PerfTestMetric.legacy_chromium_bot_compatible_test_name): Always append the metric name now 16 that we don't have to worry about the backward compatibility here. 17 18 (PerfTestMetric.iteration_values): Added. 19 (PerfTestMetric.unit): Added. 20 (PerfTestMetric.metric_to_unit): 21 (PerfTestMetric.time_unit_to_metric): Extracted from PerfTestsRunner._generate_results_dict. 22 23 (PerfTest.run): 24 (PerfTest.log_statistics): Merged compute_statistics and self.output_statistics. 25 26 * Scripts/webkitpy/performance_tests/perftest_unittest.py: 27 (TestPerfTestMetric.test_init_set_missing_unit): 28 (TestPerfTestMetric.test_init_set_time_metric): Added. 29 (TestPerfTestMetric.test_legacy_chromium_bot_compatible_test_name): 30 (TestPerfTestMetric.test_append): 31 (TestPerfTestMetric.test_compute_statistics): Removed. Integration tests check some of these 32 test cases and it's not critical to keep math right as they're not used in output JSON anymore. 33 (TestPerfTest._assert_results_are_correct): 34 (TestReplayPerfTest.test_run_with_driver_accumulates_results): 35 (TestReplayPerfTest.test_run_with_driver_accumulates_memory_results): 36 37 * Scripts/webkitpy/performance_tests/perftestsrunner.py: 38 (PerfTestsRunner._generate_results_dict): 39 40 * Scripts/webkitpy/performance_tests/perftestsrunner_integrationtest.py: 41 (EventTargetWrapperTestData): The metric name shows up at the end of the test name as expected. 42 (SomeParserTestData): Ditto. 43 (MemoryTestData): Ditto. 44 1 45 2013-03-01 Allan Sandfeld Jensen <allan.jensen@digia.com> 2 46 -
trunk/Tools/Scripts/webkitpy/performance_tests/perftest.py
r144363 r144478 55 55 class PerfTestMetric(object): 56 56 def __init__(self, metric, unit=None, iterations=None): 57 self._metric = metric57 # FIXME: Fix runner.js to report correct metric names 58 58 self._iterations = iterations or [] 59 59 self._unit = unit or self.metric_to_unit(metric) 60 self._metric = self.time_unit_to_metric(self._unit) if metric == 'Time' else metric 60 61 61 62 def metric(self): … … 68 69 def legacy_chromium_bot_compatible_test_name(self, test_name_with_extension): 69 70 test_name = re.sub(r'\.\w+$', '', test_name_with_extension) 70 return test_name if self._metric == 'Time' else test_name+ ':' + self._metric71 return test_name + ':' + self._metric 71 72 72 73 def append(self, value): 73 74 self._iterations.append(value) 74 75 75 def to_dict(self): 76 assert self.has_values() 77 statistics = self.compute_statistics(self._iterations) 78 statistics['unit'] = self._unit 79 statistics['values'] = self._iterations 80 return statistics 81 82 @classmethod 83 def metric_to_unit(cls, metric): 76 def iteration_values(self): 77 return self._iterations 78 79 def unit(self): 80 return self._unit 81 82 @staticmethod 83 def metric_to_unit(metric): 84 84 assert metric in ('Time', 'Malloc', 'JSHeap') 85 85 return 'ms' if metric == 'Time' else 'bytes' 86 86 87 87 @staticmethod 88 def compute_statistics(values): 89 sorted_values = sorted(values) 90 91 # Compute the mean and variance using Knuth's online algorithm (has good numerical stability). 92 squareSum = 0 93 mean = 0 94 for i, time in enumerate(sorted_values): 95 delta = time - mean 96 sweep = i + 1.0 97 mean += delta / sweep 98 squareSum += delta * (time - mean) 99 100 middle = int(len(sorted_values) / 2) 101 result = {'avg': sum(sorted_values) / len(values), 102 'min': sorted_values[0], 103 'max': sorted_values[-1], 104 'median': sorted_values[middle] if len(sorted_values) % 2 else (sorted_values[middle - 1] + sorted_values[middle]) / 2, 105 'stdev': math.sqrt(squareSum / (len(sorted_values) - 1)) if len(sorted_values) > 1 else 0} 106 return result 88 def time_unit_to_metric(unit): 89 return {'fps': 'FrameRate', 'runs/s': 'Runs', 'ms': 'Time'}[unit] 107 90 108 91 … … 139 122 return metrics 140 123 124 should_log = not self._port.get_option('profile') 125 if should_log and self._description: 126 _log.info('DESCRIPTION: %s' % self._description) 127 141 128 results = {} 142 129 for metric in metrics: 143 130 legacy_test_name = metric.legacy_chromium_bot_compatible_test_name(self.test_name()) 144 results[legacy_test_name] = metric.to_dict() 145 146 if not self._port.get_option('profile'): 147 if self._description: 148 _log.info('DESCRIPTION: %s' % self._description) 149 for result_name in sorted(results.keys()): 150 self.output_statistics(result_name, results[result_name]) 131 results[legacy_test_name] = metric.iteration_values() 132 if should_log: 133 self.log_statistics(legacy_test_name.replace(':', ': ').replace('/', ': '), 134 metric.iteration_values(), metric.unit()) 151 135 152 136 return results 137 138 @staticmethod 139 def log_statistics(test_name, values, unit): 140 sorted_values = sorted(values) 141 142 # Compute the mean and variance using Knuth's online algorithm (has good numerical stability). 143 square_sum = 0 144 mean = 0 145 for i, time in enumerate(sorted_values): 146 delta = time - mean 147 sweep = i + 1.0 148 mean += delta / sweep 149 square_sum += delta * (time - mean) 150 151 middle = int(len(sorted_values) / 2) 152 mean = sum(sorted_values) / len(values) 153 median = sorted_values[middle] if len(sorted_values) % 2 else (sorted_values[middle - 1] + sorted_values[middle]) / 2 154 stdev = math.sqrt(square_sum / (len(sorted_values) - 1)) if len(sorted_values) > 1 else 0 155 156 _log.info('RESULT %s= %s %s' % (test_name, mean, unit)) 157 _log.info('median= %s %s, stdev= %s %s, min= %s %s, max= %s %s' % 158 (median, unit, stdev, unit, sorted_values[0], unit, sorted_values[-1], unit)) 153 159 154 160 _description_regex = re.compile(r'^Description: (?P<description>.*)$', re.IGNORECASE) … … 243 249 output.text = '\n'.join([line for line in re.split('\n', output.text) if not self._should_ignore_line(self._lines_to_ignore_in_parser_result, line)]) 244 250 245 def output_statistics(self, test_name, results):246 unit = results['unit']247 _log.info('RESULT %s= %s %s' % (test_name.replace(':', ': ').replace('/', ': '), results['avg'], unit))248 _log.info(', '.join(['%s= %s %s' % (key, results[key], unit) for key in self._statistics_keys[1:5]]))249 250 251 251 252 class ChromiumStylePerfTest(PerfTest): -
trunk/Tools/Scripts/webkitpy/performance_tests/perftest_unittest.py
r144363 r144478 51 51 class TestPerfTestMetric(unittest.TestCase): 52 52 def test_init_set_missing_unit(self): 53 self.assertEqual(PerfTestMetric('Time', iterations=[1, 2, 3, 4, 5]).to_dict()['unit'], 'ms') 54 self.assertEqual(PerfTestMetric('Malloc', iterations=[1, 2, 3, 4, 5]).to_dict()['unit'], 'bytes') 55 self.assertEqual(PerfTestMetric('JSHeap', iterations=[1, 2, 3, 4, 5]).to_dict()['unit'], 'bytes') 53 self.assertEqual(PerfTestMetric('Time', iterations=[1, 2, 3, 4, 5]).unit(), 'ms') 54 self.assertEqual(PerfTestMetric('Malloc', iterations=[1, 2, 3, 4, 5]).unit(), 'bytes') 55 self.assertEqual(PerfTestMetric('JSHeap', iterations=[1, 2, 3, 4, 5]).unit(), 'bytes') 56 57 def test_init_set_time_metric(self): 58 self.assertEqual(PerfTestMetric('Time', 'ms').metric(), 'Time') 59 self.assertEqual(PerfTestMetric('Time', 'fps').metric(), 'FrameRate') 60 self.assertEqual(PerfTestMetric('Time', 'runs/s').metric(), 'Runs') 56 61 57 62 def test_legacy_chromium_bot_compatible_test_name(self): 58 self.assertEqual(PerfTestMetric('Time').legacy_chromium_bot_compatible_test_name('test'), 'test ')63 self.assertEqual(PerfTestMetric('Time').legacy_chromium_bot_compatible_test_name('test'), 'test:Time') 59 64 self.assertEqual(PerfTestMetric('Malloc').legacy_chromium_bot_compatible_test_name('test'), 'test:Malloc') 60 65 self.assertEqual(PerfTestMetric('JSHeap').legacy_chromium_bot_compatible_test_name('test'), 'test:JSHeap') … … 74 79 self.assertTrue(metric.has_values()) 75 80 self.assertFalse(metric2.has_values()) 76 self.assertEqual(metric. to_dict()['values'], [1])81 self.assertEqual(metric.iteration_values(), [1]) 77 82 metric.append(2) 78 self.assertEqual(metric. to_dict()['values'], [1, 2])83 self.assertEqual(metric.iteration_values(), [1, 2]) 79 84 80 85 metric2.append(3) 81 86 self.assertTrue(metric2.has_values()) 82 self.assertEqual(metric.to_dict()['values'], [1, 2]) 83 self.assertEqual(metric2.to_dict()['values'], [3]) 84 85 def test_compute_statistics(self): 86 def compute_statistics(values): 87 statistics = PerfTestMetric.compute_statistics(map(lambda x: float(x), values)) 88 return json.loads(json.dumps(statistics)) 89 90 statistics = compute_statistics([10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 20, 19, 18, 17, 16, 15, 14, 13, 12, 11]) 91 self.assertItemsEqual(statistics.keys(), ['avg', 'max', 'median', 'min', 'stdev']) 92 self.assertEqual(statistics['avg'], 10.5) 93 self.assertEqual(statistics['min'], 1) 94 self.assertEqual(statistics['max'], 20) 95 self.assertEqual(statistics['median'], 10.5) 96 self.assertEqual(compute_statistics([8, 9, 10, 11, 12])['avg'], 10) 97 self.assertEqual(compute_statistics([8, 9, 10, 11, 12] * 4)['avg'], 10) 98 self.assertEqual(compute_statistics([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19])['avg'], 10) 99 self.assertEqual(compute_statistics([1, 5, 2, 8, 7])['median'], 5) 100 self.assertEqual(compute_statistics([1, 6, 2, 8, 7, 2])['median'], 4) 101 self.assertAlmostEqual(statistics['stdev'], math.sqrt(35)) 102 self.assertAlmostEqual(compute_statistics([1])['stdev'], 0) 103 self.assertAlmostEqual(compute_statistics([1, 2, 3, 4, 5, 6])['stdev'], math.sqrt(3.5)) 104 self.assertAlmostEqual(compute_statistics([4, 2, 5, 8, 6])['stdev'], math.sqrt(5)) 87 self.assertEqual(metric.iteration_values(), [1, 2]) 88 self.assertEqual(metric2.iteration_values(), [3]) 105 89 106 90 … … 110 94 parsed_results = test._run_with_driver(None, None) 111 95 self.assertEqual(len(parsed_results), 1) 112 some_test_results = parsed_results[0].to_dict() 113 self.assertItemsEqual(some_test_results.keys(), ['avg', 'max', 'median', 'min', 'stdev', 'unit', 'values']) 114 self.assertEqual(some_test_results['values'], [1080, 1120, 1095, 1101, 1104]) 115 self.assertEqual(some_test_results['min'], 1080) 116 self.assertEqual(some_test_results['max'], 1120) 117 self.assertEqual(some_test_results['avg'], 1100) 118 self.assertEqual(some_test_results['median'], 1101) 119 self.assertAlmostEqual(some_test_results['stdev'], 14.50862, places=5) 120 self.assertEqual(some_test_results['unit'], 'ms') 96 self.assertEqual(parsed_results[0].iteration_values(), [1080, 1120, 1095, 1101, 1104]) 121 97 122 98 def test_parse_output(self): … … 342 318 self.assertEqual(len(metrics), 1) 343 319 self.assertEqual(metrics[0].metric(), 'Time') 344 self.assertEqual(metrics[0].to_dict(), {'max': 20000, 'avg': 11000.0, 'median': 11000, 'stdev': 5627.314338711378, 'min': 2000, 'unit': 'ms', 345 'values': [float(i * 1000) for i in range(2, 21)]}) 320 self.assertEqual(metrics[0].iteration_values(), [float(i * 1000) for i in range(2, 21)]) 346 321 347 322 def test_run_with_driver_accumulates_memory_results(self): … … 369 344 self.assertEqual(len(metrics), 3) 370 345 self.assertEqual(metrics[0].metric(), 'Time') 371 self.assertEqual(metrics[0].to_dict(), {'max': 20000, 'avg': 11000.0, 'median': 11000, 'stdev': 5627.314338711378, 'min': 2000, 'unit': 'ms', 372 'values': [float(i * 1000) for i in range(2, 21)]}) 346 self.assertEqual(metrics[0].iteration_values(), [float(i * 1000) for i in range(2, 21)]) 373 347 self.assertEqual(metrics[1].metric(), 'Malloc') 374 self.assertEqual(metrics[1].to_dict(), {'max': 10, 'avg': 10.0, 'median': 10, 'min': 10, 'stdev': 0.0, 'unit': 'bytes', 375 'values': [float(10)] * 19}) 348 self.assertEqual(metrics[1].iteration_values(), [float(10)] * 19) 376 349 self.assertEqual(metrics[2].metric(), 'JSHeap') 377 self.assertEqual(metrics[2].to_dict(), {'max': 5, 'avg': 5.0, 'median': 5, 'min': 5, 'stdev': 0.0, 'unit': 'bytes', 378 'values': [float(5)] * 19}) 350 self.assertEqual(metrics[2].iteration_values(), [float(5)] * 19) 379 351 380 352 def test_prepare_fails_when_wait_until_ready_fails(self): -
trunk/Tools/Scripts/webkitpy/performance_tests/perftestsrunner.py
r144421 r144478 256 256 257 257 # FIXME: Make this function shorter once we've transitioned to use perf.webkit.org. 258 for metric_full_name, resultin self._results.iteritems():259 if not isinstance( result, dict): # We can't reports results without indivisual measurements.258 for metric_full_name, iteration_values in self._results.iteritems(): 259 if not isinstance(iteration_values, list): # We can't reports results without individual measurements. 260 260 continue 261 261 262 262 assert metric_full_name.count(':') <= 1 263 263 test_full_name, _, metric = metric_full_name.partition(':') 264 if not metric:265 metric = {'fps': 'FrameRate', 'runs/s': 'Runs', 'ms': 'Time'}[result['unit']]266 264 267 265 tests = contents['tests'] … … 279 277 current_test.setdefault('metrics', {}) 280 278 assert metric not in current_test['metrics'] 281 current_test['metrics'][metric] = {'current': result['values']}279 current_test['metrics'][metric] = {'current': iteration_values} 282 280 else: 283 281 current_test.setdefault('tests', {}) -
trunk/Tools/Scripts/webkitpy/performance_tests/perftestsrunner_integrationtest.py
r144421 r144478 87 87 88 88 output = """Running Bindings/event-target-wrapper.html (1 of 2) 89 RESULT Bindings: event-target-wrapper = 1490.0 ms89 RESULT Bindings: event-target-wrapper: Time= 1490.0 ms 90 90 median= 1488.0 ms, stdev= 15.13935 ms, min= 1471.0 ms, max= 1510.0 ms 91 91 Finished: 0.1 s … … 111 111 112 112 output = """Running Parser/some-parser.html (2 of 2) 113 RESULT Parser: some-parser = 1100.0 ms113 RESULT Parser: some-parser: Time= 1100.0 ms 114 114 median= 1101.0 ms, stdev= 14.50861 ms, min= 1080.0 ms, max= 1120.0 ms 115 115 Finished: 0.1 s … … 149 149 output = """Running 1 tests 150 150 Running Parser/memory-test.html (1 of 1) 151 RESULT Parser: memory-test = 1100.0 ms151 RESULT Parser: memory-test: Time= 1100.0 ms 152 152 median= 1101.0 ms, stdev= 14.50861 ms, min= 1080.0 ms, max= 1120.0 ms 153 153 RESULT Parser: memory-test: JSHeap= 830000.0 bytes
Note: See TracChangeset
for help on using the changeset viewer.