Changeset 125332 in webkit


Ignore:
Timestamp:
Aug 10, 2012 3:17:57 PM (12 years ago)
Author:
rniwa@webkit.org
Message:

run-perf-tests should upload memory statistics to perf-o-matic
​https://bugs.webkit.org/show_bug.cgi?id=93690

Reviewed by Dirk Pranke.

Upload JS Heap and FastMalloc results for a test X/Y as: X/Y:JSHeap and X/Y:FastMalloc.
Note "JS Heap" is converted to the CamelCase JSHeap.

Also did some refactoring in PerfTest.parse_output and PerfTestRunnerTest and updated
some helps in PerfTest.parse_output per arv's comments.

  • Scripts/webkitpy/performance_tests/perftest.py:

(PerfTest):
(PerfTest.parse_output): Include JSHeap and FastMalloc statistics in results as well as of Time.
Also refactored it to avoid hard-coding indices in _result_classes and moved complied regular
exressions out of the function to avoid re-compiling them every time the method runs.
(PerfTest.output_statistics): Convert ':' to ': ' as well as '/'.
(ChromiumStylePerfTest.parse_output): Removed an unused variable.

  • Scripts/webkitpy/performance_tests/perftestsrunner.py:

(PerfTestsRunner._parse_args): Uploaded some helps per arv's feedback.
(PerfTestsRunner._output_json_path): Extracted from _generate_and_show_results to be used in
PerfTestRunnerTest.create_runner.
(PerfTestsRunner._generate_and_show_results):

  • Scripts/webkitpy/performance_tests/perftestsrunner_unittest.py:

(create_runner): Extracted runner.load_output_json from various tests.
(test_run_memory_test): Added.
(PerfTestRunner): Extracted _event_target_wrapper_and_inspector_results from various tests.
(test_run_with_json_output):
(test_run_with_description):
(test_run_generates_json_by_default):
(test_run_generates_and_show_results_page):
(test_run_with_json_source):
(test_run_with_multiple_repositories):

Location:
trunk/Tools
Files:
4 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r125320 r125332  
     12012-08-10  Ryosuke Niwa  <rniwa@webkit.org>
     2
     3        run-perf-tests should upload memory statistics to perf-o-matic
     4        https://bugs.webkit.org/show_bug.cgi?id=93690
     5
     6        Reviewed by Dirk Pranke.
     7
     8        Upload JS Heap and FastMalloc results for a test X/Y as: X/Y:JSHeap and X/Y:FastMalloc.
     9        Note "JS Heap" is converted to the CamelCase JSHeap.
     10
     11        Also did some refactoring in PerfTest.parse_output and PerfTestRunnerTest and updated
     12        some helps in PerfTest.parse_output per arv's comments.
     13
     14        * Scripts/webkitpy/performance_tests/perftest.py:
     15        (PerfTest):
     16        (PerfTest.parse_output): Include JSHeap and FastMalloc statistics in results as well as of Time.
     17        Also refactored it to avoid hard-coding indices in _result_classes and moved complied regular
     18        exressions out of the function to avoid re-compiling them every time the method runs.
     19        (PerfTest.output_statistics): Convert ':' to ': ' as well as '/'.
     20        (ChromiumStylePerfTest.parse_output): Removed an unused variable.
     21        * Scripts/webkitpy/performance_tests/perftestsrunner.py:
     22        (PerfTestsRunner._parse_args): Uploaded some helps per arv's feedback.
     23        (PerfTestsRunner._output_json_path): Extracted from _generate_and_show_results to be used in
     24        PerfTestRunnerTest.create_runner.
     25        (PerfTestsRunner._generate_and_show_results):
     26        * Scripts/webkitpy/performance_tests/perftestsrunner_unittest.py:
     27        (create_runner): Extracted runner.load_output_json from various tests.
     28        (test_run_memory_test): Added.
     29        (PerfTestRunner): Extracted _event_target_wrapper_and_inspector_results from various tests.
     30        (test_run_with_json_output):
     31        (test_run_with_description):
     32        (test_run_generates_json_by_default):
     33        (test_run_generates_and_show_results_page):
     34        (test_run_with_json_source):
     35        (test_run_with_multiple_repositories):
     36
    1372012-08-10  Benjamin Poulain  <bpoulain@apple.com>
    238

  • trunk/Tools/Scripts/webkitpy/performance_tests/perftest.py

    r125188 r125332  
    112112        return False
    113113
     114    _description_regex = re.compile(r'^Description: (?P<description>.*)$', re.IGNORECASE)
     115    _result_classes = ['Time', 'JS Heap', 'FastMalloc']
     116    _result_class_regex = re.compile(r'^(?P<resultclass>' + r'|'.join(_result_classes) + '):')
    114117    _statistics_keys = ['avg', 'median', 'stdev', 'min', 'max', 'unit']
    115     _result_classes = ['Time:', 'JS Heap:', 'FastMalloc:']
     118    _score_regex = re.compile(r'^(?P<key>' + r'|'.join(_statistics_keys) + r')\s+(?P<value>[0-9\.]+)\s*(?P<unit>.*)')
    116119
    117120    def parse_output(self, output):
    118         got_a_result = False
    119121        test_failed = False
    120         results = dict([(name, dict()) for name in self._result_classes])
    121         score_regex = re.compile(r'^(?P<key>' + r'|'.join(self._statistics_keys) + r')\s+(?P<value>[0-9\.]+)\s*(?P<unit>.*)')
    122         description_regex = re.compile(r'^Description: (?P<description>.*)$', re.IGNORECASE)
     122        results = {}
     123        ordered_results_keys = []
     124        test_name = re.sub(r'\.\w+$', '', self._test_name)
    123125        description_string = ""
    124         result_class_regex = re.compile(r'^(?P<resultclass>' + r'|'.join(self._result_classes) + ')')
    125 
    126126        result_class = ""
    127127        for line in re.split('\n', output.text):
    128             description = description_regex.match(line)
     128            description = self._description_regex.match(line)
    129129            if description:
    130130                description_string = description.group('description')
    131131                continue
    132132
    133             result_class_match = result_class_regex.match(line)
     133            result_class_match = self._result_class_regex.match(line)
    134134            if result_class_match:
    135135                result_class = result_class_match.group('resultclass')
    136136                continue
    137137
    138             score = score_regex.match(line)
     138            score = self._score_regex.match(line)
    139139            if score:
    140140                key = score.group('key')
    141141                value = float(score.group('value'))
    142142                unit = score.group('unit')
    143                 results[result_class]['unit'] = unit
    144                 results[result_class][key] = value
     143                name = test_name
     144                if result_class != 'Time':
     145                    name += ':' + result_class.replace(' ', '')
     146                if name not in ordered_results_keys:
     147                    ordered_results_keys.append(name)
     148                results.setdefault(name, {})
     149                results[name]['unit'] = unit
     150                results[name][key] = value
    145151                continue
    146152
    … …  
    149155                _log.error(line)
    150156
    151         if test_failed or set(self._statistics_keys) != set(results[self._result_classes[0]].keys()):
     157        if test_failed or set(self._statistics_keys) != set(results[test_name].keys()):
    152158            return None
    153159
    154         test_name = re.sub(r'\.\w+$', '', self._test_name)
    155         self.output_statistics(test_name, results[self._result_classes[0]], description_string)
    156         if results[self._result_classes[1]] and results[self._result_classes[2]]:
    157             self.output_statistics(test_name + "/JSHeap", results[self._result_classes[1]])
    158             self.output_statistics(test_name + "/FastMalloc", results[self._result_classes[2]])
    159         return {test_name: results['Time:']}
     160        for result_name in ordered_results_keys:
     161            if result_name == test_name:
     162                self.output_statistics(result_name, results[result_name], description_string)
     163            else:
     164                self.output_statistics(result_name, results[result_name])
     165        return results
    160166
    161167    def output_statistics(self, test_name, results, description_string=None):
    … …  
    163169        if description_string:
    164170            _log.info('DESCRIPTION: %s' % description_string)
    165         _log.info('RESULT %s= %s %s' % (test_name.replace('/', ': '), results['avg'], unit))
     171        _log.info('RESULT %s= %s %s' % (test_name.replace(':', ': ').replace('/', ': '), results['avg'], unit))
    166172        _log.info(', '.join(['%s= %s %s' % (key, results[key], unit) for key in self._statistics_keys[1:5]]))
    167173
    … …  
    175181    def parse_output(self, output):
    176182        test_failed = False
    177         got_a_result = False
    178183        results = {}
    179184        for line in re.split('\n', output.text):

  • trunk/Tools/Scripts/webkitpy/performance_tests/perftestsrunner.py

    r124949 r125332  
    100100                help="Do no generate results JSON and results page."),
    101101            optparse.make_option("--output-json-path",
    102                 help="Filename of the JSON file that summaries the results."),
     102                help="Path to generate a JSON file at; may contain previous results if it already exists."),
    103103            optparse.make_option("--source-json-path",
    104                 help="Path to a JSON file to be merged into the JSON file when --output-json-path is present."),
     104                help="Only used on bots. Path to a JSON file to be merged into the JSON file when --output-json-path is present."),
    105105            optparse.make_option("--description",
    106106                help="Add a description to the output JSON file if one is generated"),
    … …  
    169169        return unexpected
    170170
     171    def _output_json_path(self):
     172        output_json_path = self._options.output_json_path
     173        if output_json_path:
     174            return output_json_path
     175        return self._host.filesystem.join(self._port.perf_results_directory(), self._DEFAULT_JSON_FILENAME)
     176
    171177    def _generate_and_show_results(self):
    172178        options = self._options
    173         output_json_path = options.output_json_path
    174         if not output_json_path:
    175             output_json_path = self._host.filesystem.join(self._port.perf_results_directory(), self._DEFAULT_JSON_FILENAME)
    176 
     179        output_json_path = self._output_json_path()
    177180        output = self._generate_results_dict(self._timestamp, options.description, options.platform, options.builder_name, options.build_number)
    178181

  • trunk/Tools/Scripts/webkitpy/performance_tests/perftestsrunner_unittest.py

    r125188 r125332  
    106106max 1120 ms
    107107"""
     108            elif driver_input.test_name.endswith('memory-test.html'):
     109                text = """Running 20 times
     110Ignoring warm-up run (1115)
     111
     112Time:
     113avg 1100 ms
     114median 1101 ms
     115stdev 11 ms
     116min 1080 ms
     117max 1120 ms
     118
     119JS Heap:
     120avg 832000 bytes
     121median 829000 bytes
     122stdev 15000 bytes
     123min 811000 bytes
     124max 848000 bytes
     125
     126FastMalloc:
     127avg 532000 bytes
     128median 529000 bytes
     129stdev 13000 bytes
     130min 511000 bytes
     131max 548000 bytes
     132"""
    108133            return DriverOutput(text, '', '', '', crash=crash, timeout=timeout)
    109134
    … …  
    123148        runner._host.filesystem.maybe_make_directory(runner._base_path, 'Bindings')
    124149        runner._host.filesystem.maybe_make_directory(runner._base_path, 'Parser')
     150
     151        filesystem = runner._host.filesystem
     152        runner.load_output_json = lambda: json.loads(filesystem.read_text_file(runner._output_json_path()))
    125153        return runner, test_port
    126154
    … …  
    228256        '', '']))
    229257
     258    def test_run_memory_test(self):
     259        runner, port = self.create_runner_and_setup_results_template()
     260        runner._timestamp = 123456789
     261        port.host.filesystem.write_text_file(runner._base_path + '/Parser/memory-test.html', 'some content')
     262
     263        output = OutputCapture()
     264        output.capture_output()
     265        try:
     266            unexpected_result_count = runner.run()
     267        finally:
     268            stdout, stderr, log = output.restore_output()
     269        self.assertEqual(unexpected_result_count, 0)
     270        self.assertEqual(log, '\n'.join([
     271            'Running 1 tests',
     272            'Running Parser/memory-test.html (1 of 1)',
     273            'RESULT Parser: memory-test= 1100.0 ms',
     274            'median= 1101.0 ms, stdev= 11.0 ms, min= 1080.0 ms, max= 1120.0 ms',
     275            'RESULT Parser: memory-test: JSHeap= 832000.0 bytes',
     276            'median= 829000.0 bytes, stdev= 15000.0 bytes, min= 811000.0 bytes, max= 848000.0 bytes',
     277            'RESULT Parser: memory-test: FastMalloc= 532000.0 bytes',
     278            'median= 529000.0 bytes, stdev= 13000.0 bytes, min= 511000.0 bytes, max= 548000.0 bytes',
     279            '', '']))
     280        results = runner.load_output_json()[0]['results']
     281        self.assertEqual(results['Parser/memory-test'], {'min': 1080.0, 'max': 1120.0, 'median': 1101.0, 'stdev': 11.0, 'avg': 1100.0, 'unit': 'ms'})
     282        self.assertEqual(results['Parser/memory-test:JSHeap'], {'min': 811000.0, 'max': 848000.0, 'median': 829000.0, 'stdev': 15000.0, 'avg': 832000.0, 'unit': 'bytes'})
     283        self.assertEqual(results['Parser/memory-test:FastMalloc'], {'min': 511000.0, 'max': 548000.0, 'median': 529000.0, 'stdev': 13000.0, 'avg': 532000.0, 'unit': 'bytes'})
     284
    230285    def _test_run_with_json_output(self, runner, filesystem, upload_suceeds=True, expected_exit_code=0):
    231286        filesystem.write_text_file(runner._base_path + '/inspector/pass.html', 'some content')
    … …  
    263318        return uploaded[0]
    264319
     320    _event_target_wrapper_and_inspector_results = {
     321        "Bindings/event-target-wrapper": {"max": 1510, "avg": 1489.05, "median": 1487, "min": 1471, "stdev": 14.46, "unit": "ms"},
     322        "inspector/pass.html:group_name:test_name": 42}
     323
    265324    def test_run_with_json_output(self):
    266325        runner, port = self.create_runner(args=['--output-json-path=/mock-checkout/output.json',
    267326            '--test-results-server=some.host'])
    268327        self._test_run_with_json_output(runner, port.host.filesystem)
    269         self.assertEqual(json.loads(port.host.filesystem.read_text_file('/mock-checkout/output.json')), {
    270             "timestamp": 123456789, "results":
    271             {"Bindings/event-target-wrapper": {"max": 1510, "avg": 1489.05, "median": 1487, "min": 1471, "stdev": 14.46, "unit": "ms"},
    272             "inspector/pass.html:group_name:test_name": 42},
     328        self.assertEqual(runner.load_output_json(), {
     329            "timestamp": 123456789, "results": self._event_target_wrapper_and_inspector_results,
    273330            "webkit-revision": "5678", "branch": "webkit-trunk"})
    274331
    … …  
    277334            '--test-results-server=some.host', '--description', 'some description'])
    278335        self._test_run_with_json_output(runner, port.host.filesystem)
    279         self.assertEqual(json.loads(port.host.filesystem.read_text_file('/mock-checkout/output.json')), {
    280             "timestamp": 123456789, "description": "some description", "results":
    281             {"Bindings/event-target-wrapper": {"max": 1510, "avg": 1489.05, "median": 1487, "min": 1471, "stdev": 14.46, "unit": "ms"},
    282             "inspector/pass.html:group_name:test_name": 42},
     336        self.assertEqual(runner.load_output_json(), {
     337            "timestamp": 123456789, "description": "some description",
     338            "results": self._event_target_wrapper_and_inspector_results,
    283339            "webkit-revision": "5678", "branch": "webkit-trunk"})
    284340
    … …  
    310366
    311367        self.assertEqual(json.loads(port.host.filesystem.read_text_file(output_json_path)), [{
    312             "timestamp": 123456789, "results":
    313             {"Bindings/event-target-wrapper": {"max": 1510, "avg": 1489.05, "median": 1487, "min": 1471, "stdev": 14.46, "unit": "ms"},
    314             "inspector/pass.html:group_name:test_name": 42},
     368            "timestamp": 123456789, "results": self._event_target_wrapper_and_inspector_results,
    315369            "webkit-revision": "5678", "branch": "webkit-trunk"}])
    316370
    … …  
    325379        self._test_run_with_json_output(runner, filesystem)
    326380
    327         expected_entry = {"timestamp": 123456789, "results": {"Bindings/event-target-wrapper":
    328             {"max": 1510, "avg": 1489.05, "median": 1487, "min": 1471, "stdev": 14.46, "unit": "ms"},
    329             "inspector/pass.html:group_name:test_name": 42}, "webkit-revision": "5678", "branch": "webkit-trunk"}
     381        expected_entry = {"timestamp": 123456789, "results": self._event_target_wrapper_and_inspector_results,
     382            "webkit-revision": "5678", "branch": "webkit-trunk"}
    330383
    331384        self.maxDiff = None
    … …  
    372425        port.host.filesystem.write_text_file('/mock-checkout/source.json', '{"key": "value"}')
    373426        self._test_run_with_json_output(runner, port.host.filesystem)
    374         self.assertEqual(json.loads(port.host.filesystem.files['/mock-checkout/output.json']), {
    375             "timestamp": 123456789, "results":
    376             {"Bindings/event-target-wrapper": {"max": 1510, "avg": 1489.05, "median": 1487, "min": 1471, "stdev": 14.46, "unit": "ms"},
    377             "inspector/pass.html:group_name:test_name": 42},
    378             "webkit-revision": "5678", "branch": "webkit-trunk",
    379             "key": "value"})
     427        self.assertEqual(runner.load_output_json(), {
     428            "timestamp": 123456789, "results": self._event_target_wrapper_and_inspector_results,
     429            "webkit-revision": "5678", "branch": "webkit-trunk", "key": "value"})
    380430
    381431    def test_run_with_bad_json_source(self):
    … …  
    393443        port.repository_paths = lambda: [('webkit', '/mock-checkout'), ('some', '/mock-checkout/some')]
    394444        self._test_run_with_json_output(runner, port.host.filesystem)
    395         self.assertEqual(json.loads(port.host.filesystem.files['/mock-checkout/output.json']), {
    396             "timestamp": 123456789, "results":
    397             {"Bindings/event-target-wrapper": {"max": 1510, "avg": 1489.05, "median": 1487, "min": 1471, "stdev": 14.46, "unit": "ms"},
    398             "inspector/pass.html:group_name:test_name": 42.0},
     445        self.assertEqual(runner.load_output_json(), {
     446            "timestamp": 123456789, "results": self._event_target_wrapper_and_inspector_results,
    399447            "webkit-revision": "5678", "some-revision": "5678", "branch": "webkit-trunk"})
    400448
Note: See TracChangeset for help on using the changeset viewer.