Changeset 114149 in webkit


Ignore:
Timestamp:
Apr 13, 2012 11:16:26 AM (12 years ago)
Author:
rniwa@webkit.org
Message:

Enable SVG page loading performance tests
​https://bugs.webkit.org/show_bug.cgi?id=83861

Reviewed by Adam Barth.

PerformanceTests:

Unskip PageLoad directory and skip tests that load too fast (less than 100ms) and are unstable,
and the ones that take too long to load (more than 20s).

  • Skipped:

Tools:

Add support for page loading tests. These are tests without any test running harnesses, and we use
DriverOutput's test_timeout to measure the loading time.

Extracted PerfTest.run and PerfTest.run_failed from PerfTestRunner._run_single_test.

  • Scripts/webkitpy/performance_tests/perftest.py:

(PerfTest.run):
(PerfTest):
(PerfTest.run_failed):
(PerfTest.parse_output):
(PerfTest.output_statistics):
(ChromiumStylePerfTest.parse_output):
(PageLoadingPerfTest): Added.
(PageLoadingPerfTest.init):
(PageLoadingPerfTest.run): Run the driver 20 times to get the statistics for the loading time.

  • Scripts/webkitpy/performance_tests/perftest_unittest.py: Added.

(MockPrinter):
(MockPrinter.init):
(MockPrinter.write):
(MainTest):
(MainTest.test_parse_output):
(MainTest.test_parse_output_with_failing_line):
(TestPageLoadingPerfTest):
(TestPageLoadingPerfTest.assertWritten):
(TestPageLoadingPerfTest.MockDriver):
(TestPageLoadingPerfTest.MockDriver.init):
(TestPageLoadingPerfTest.MockDriver.run_test):
(TestPageLoadingPerfTest.test_run):
(TestPageLoadingPerfTest.test_run_with_bad_output):

  • Scripts/webkitpy/performance_tests/perftestsrunner.py:

(PerfTestsRunner):
(PerfTestsRunner._collect_tests._is_test_file): Now recognizes svg tests.
(PerfTestsRunner._collect_tests):
(PerfTestsRunner._run_tests_set):
(PerfTestsRunner._run_single_test):

  • Scripts/webkitpy/performance_tests/perftestsrunner_unittest.py:

(test_collect_tests_with_page_load_svg):
(test_collect_tests_with_page_load_svg.add_file):

Location:
trunk
Files:
1 added
6 edited

Legend:

Unmodified
Added
Removed

  • trunk/PerformanceTests/ChangeLog

    r113202 r114149  
     12012-04-12  Ryosuke Niwa  <rniwa@webkit.org>
     2
     3        Enable SVG page loading performance tests
     4        https://bugs.webkit.org/show_bug.cgi?id=83861
     5
     6        Reviewed by Adam Barth.
     7
     8        Unskip PageLoad directory and skip tests that load too fast (less than 100ms) and are unstable,
     9        and the ones that take too long to load (more than 20s).
     10
     11        * Skipped:
     12
    1132012-04-04  Ilya Tikhonovsky  <loislo@chromium.org>
    214

  • trunk/PerformanceTests/Skipped

    r110801 r114149  
    22Mutation
    33
     4# Unstable SVG page loading tests
     5PageLoad/svg/files/33041-Samurai.svg
     6PageLoad/svg/files/Sierpinski_carpet_6.svg # Takes too long to load
     7PageLoad/svg/files/az-lizard_benji_park_01.svg
     8PageLoad/svg/files/france.svg
     9PageLoad/svg/files/francobollo_gnome_ezechi_02.svg
     10PageLoad/svg/files/gearflowers.svg
     11PageLoad/svg/files/hereGear4.svg
     12PageLoad/svg/files/mtsthelens.svg # Takes 22s on MacBookPro
     13
    414# Do not conform to WebKit or Chromium perf test styles
    5 PageLoad
    615SunSpider
    716XSSAuditor

  • trunk/Tools/ChangeLog

    r114143 r114149  
     12012-04-12  Ryosuke Niwa  <rniwa@webkit.org>
     2
     3        Enable SVG page loading performance tests
     4        https://bugs.webkit.org/show_bug.cgi?id=83861
     5
     6        Reviewed by Adam Barth.
     7
     8        Add support for page loading tests. These are tests without any test running harnesses, and we use
     9        DriverOutput's test_timeout to measure the loading time.
     10
     11        Extracted PerfTest.run and PerfTest.run_failed from PerfTestRunner._run_single_test.
     12
     13        * Scripts/webkitpy/performance_tests/perftest.py:
     14        (PerfTest.run):
     15        (PerfTest):
     16        (PerfTest.run_failed):
     17        (PerfTest.parse_output):
     18        (PerfTest.output_statistics):
     19        (ChromiumStylePerfTest.parse_output):
     20        (PageLoadingPerfTest): Added.
     21        (PageLoadingPerfTest.__init__):
     22        (PageLoadingPerfTest.run): Run the driver 20 times to get the statistics for the loading time.
     23        * Scripts/webkitpy/performance_tests/perftest_unittest.py: Added.
     24        (MockPrinter):
     25        (MockPrinter.__init__):
     26        (MockPrinter.write):
     27        (MainTest):
     28        (MainTest.test_parse_output):
     29        (MainTest.test_parse_output_with_failing_line):
     30        (TestPageLoadingPerfTest):
     31        (TestPageLoadingPerfTest.assertWritten):
     32        (TestPageLoadingPerfTest.MockDriver):
     33        (TestPageLoadingPerfTest.MockDriver.__init__):
     34        (TestPageLoadingPerfTest.MockDriver.run_test):
     35        (TestPageLoadingPerfTest.test_run):
     36        (TestPageLoadingPerfTest.test_run_with_bad_output):
     37        * Scripts/webkitpy/performance_tests/perftestsrunner.py:
     38        (PerfTestsRunner):
     39        (PerfTestsRunner._collect_tests._is_test_file): Now recognizes svg tests.
     40        (PerfTestsRunner._collect_tests):
     41        (PerfTestsRunner._run_tests_set):
     42        (PerfTestsRunner._run_single_test):
     43        * Scripts/webkitpy/performance_tests/perftestsrunner_unittest.py:
     44        (test_collect_tests_with_page_load_svg):
     45        (test_collect_tests_with_page_load_svg.add_file):
     46
    1472012-04-10  Brady Eidson  <beidson@apple.com>
    248

  • trunk/Tools/Scripts/webkitpy/performance_tests/perftest.py

    r114076 r114149  
    2929
    3030
     31import math
    3132import re
     33
     34from webkitpy.layout_tests.port.driver import DriverInput
    3235
    3336
    … …  
    4750        return self._path_or_url
    4851
     52    def run(self, driver, timeout_ms, printer, buildbot_output):
     53        output = driver.run_test(DriverInput(self.path_or_url(), timeout_ms, None, False))
     54        if self.run_failed(output, printer):
     55            return None
     56        return self.parse_output(output, printer, buildbot_output)
     57
     58    def run_failed(self, output, printer):
     59        if output.text == None or output.error:
     60            pass
     61        elif output.timeout:
     62            printer.write('timeout: %s' % self.test_name())
     63        elif output.crash:
     64            printer.write('crash: %s' % self.test_name())
     65        else:
     66            return False
     67
     68        if output.error:
     69            printer.write('error: %s\n%s' % (self.test_name(), output.error))
     70
     71        return True
     72
    4973    _lines_to_ignore_in_parser_result = [
    5074        re.compile(r'^Running \d+ times$'),
    … …  
    5680        re.compile(re.escape("""frame "<!--framePath //<!--frame0-->-->" - has 1 onunload handler(s)""")),
    5781        re.compile(re.escape("""frame "<!--framePath //<!--frame0-->/<!--frame0-->-->" - has 1 onunload handler(s)"""))]
     82
     83    _statistics_keys = ['avg', 'median', 'stdev', 'min', 'max']
    5884
    5985    def _should_ignore_line_in_parser_test_result(self, line):
    … …  
    6995        test_failed = False
    7096        results = {}
    71         keys = ['avg', 'median', 'stdev', 'min', 'max']
    72         score_regex = re.compile(r'^(?P<key>' + r'|'.join(keys) + r')\s+(?P<value>[0-9\.]+)\s*(?P<unit>.*)')
     97        score_regex = re.compile(r'^(?P<key>' + r'|'.join(self._statistics_keys) + r')\s+(?P<value>[0-9\.]+)\s*(?P<unit>.*)')
    7398        unit = "ms"
    7499
    … …  
    85110                printer.write("%s" % line)
    86111
    87         if test_failed or set(keys) != set(results.keys()):
     112        if test_failed or set(self._statistics_keys) != set(results.keys()):
    88113            return None
    89114
    … …  
    91116
    92117        test_name = re.sub(r'\.\w+$', '', self._test_name)
    93         buildbot_output.write('RESULT %s= %s %s\n' % (test_name.replace('/', ': '), results['avg'], unit))
    94         buildbot_output.write(', '.join(['%s= %s %s' % (key, results[key], unit) for key in keys[1:]]) + '\n')
     118        self.output_statistics(test_name, results, buildbot_output)
    95119
    96120        return {test_name: results}
     121
     122    def output_statistics(self, test_name, results, buildbot_output):
     123        unit = results['unit']
     124        buildbot_output.write('RESULT %s= %s %s\n' % (test_name.replace('/', ': '), results['avg'], unit))
     125        buildbot_output.write(', '.join(['%s= %s %s' % (key, results[key], unit) for key in self._statistics_keys[1:]]) + '\n')
    97126
    98127
    … …  
    117146                printer.write("%s" % line)
    118147        return results if results and not test_failed else None
     148
     149
     150class PageLoadingPerfTest(PerfTest):
     151    def __init__(self, test_name, dirname, path_or_url):
     152        super(PageLoadingPerfTest, self).__init__(test_name, dirname, path_or_url)
     153
     154    def run(self, driver, timeout_ms, printer, buildbot_output):
     155        test_times = []
     156
     157        for i in range(0, 2):
     158            output = driver.run_test(DriverInput(self.path_or_url(), timeout_ms, None, False))
     159            if self.run_failed(output, printer):
     160                return None
     161            if i == 0:
     162                continue
     163            test_times.append(output.test_time * 1000)
     164
     165        test_times = sorted(test_times)
     166
     167        # Compute the mean and variance using a numerically stable algorithm.
     168        squareSum = 0
     169        mean = 0
     170        valueSum = sum(test_times)
     171        for i, time in enumerate(test_times):
     172            delta = time - mean
     173            sweep = i + 1.0
     174            mean += delta / sweep
     175            squareSum += delta * delta * (i / sweep)
     176
     177        middle = int(len(test_times) / 2)
     178        results = {'avg': mean,
     179            'min': min(test_times),
     180            'max': max(test_times),
     181            'median': test_times[middle] if len(test_times) % 2 else (test_times[middle - 1] + test_times[middle]) / 2,
     182            'stdev': math.sqrt(squareSum),
     183            'unit': 'ms'}
     184        self.output_statistics(self.test_name(), results, buildbot_output)
     185        return {self.test_name(): results}

  • trunk/Tools/Scripts/webkitpy/performance_tests/perftestsrunner.py

    r114076 r114149  
    4040from webkitpy.common.host import Host
    4141from webkitpy.common.net.file_uploader import FileUploader
    42 from webkitpy.layout_tests.port.driver import DriverInput
    4342from webkitpy.layout_tests.views import printing
    4443from webkitpy.performance_tests.perftest import ChromiumStylePerfTest
     44from webkitpy.performance_tests.perftest import PageLoadingPerfTest
    4545from webkitpy.performance_tests.perftest import PerfTest
    4646
     47
    4748_log = logging.getLogger(__name__)
    4849
    4950
    5051class PerfTestsRunner(object):
    51     _test_directories_for_chromium_style_tests = ['inspector']
     52    _pattern_for_chromium_style_tests = re.compile('^inspector/')
     53    _pattern_for_page_loading_tests = re.compile('^PageLoad/')
    5254    _default_branch = 'webkit-trunk'
    5355    _EXIT_CODE_BAD_BUILD = -1
    … …  
    113115
    114116        def _is_test_file(filesystem, dirname, filename):
    115             return filename.endswith('.html')
     117            return filesystem.splitext(filename)[1] in ['.html', '.svg']
    116118
    117119        filesystem = self._host.filesystem
    … …  
    133135            test_name = relative_path.replace('\\', '/')
    134136            dirname = filesystem.dirname(path)
    135             if self._host.filesystem.dirname(relative_path) in self._test_directories_for_chromium_style_tests:
     137            if self._pattern_for_chromium_style_tests.match(relative_path):
    136138                tests.append(ChromiumStylePerfTest(test_name, dirname, path))
     139            elif self._pattern_for_page_loading_tests.match(relative_path):
     140                tests.append(PageLoadingPerfTest(test_name, dirname, path))
    137141            else:
    138142                tests.append(PerfTest(test_name, dirname, path))
    … …  
    256260        return unexpected
    257261
    258     _inspector_result_regex = re.compile(r'^RESULT\s+(?P<name>[^=]+)\s*=\s+(?P<value>\d+(\.\d+)?)\s*(?P<unit>\w+)$')
    259 
    260     def _process_chromium_style_test_result(self, test, output):
    261         test_failed = False
    262         got_a_result = False
    263         for line in re.split('\n', output.text):
    264             resultLine = self._inspector_result_regex.match(line)
    265             if resultLine:
    266                 # FIXME: Store the unit
    267                 self._results[resultLine.group('name').replace(' ', '')] = float(resultLine.group('value'))
    268                 self._buildbot_output.write("%s\n" % line)
    269                 got_a_result = True
    270             elif not len(line) == 0:
    271                 test_failed = True
    272                 self._printer.write("%s" % line)
    273         return test_failed or not got_a_result
    274 
    275262    def _run_single_test(self, test, driver):
    276263        start_time = time.time()
    277264
    278         output = driver.run_test(DriverInput(test.path_or_url(), self._options.time_out_ms, None, False))
    279         new_results = None
    280 
    281         if output.text == None:
    282             pass
    283         elif output.timeout:
    284             self._printer.write('timeout: %s' % test.test_name())
    285         elif output.crash:
    286             self._printer.write('crash: %s' % test.test_name())
    287         else:
    288             new_results = test.parse_output(output, self._printer, self._buildbot_output)
    289 
    290         if len(output.error):
    291             self._printer.write('error:\n%s' % output.error)
    292             new_results = None
    293 
     265        new_results = test.run(driver, self._options.time_out_ms, self._printer, self._buildbot_output)
    294266        if new_results:
    295267            self._results.update(new_results)

  • trunk/Tools/Scripts/webkitpy/performance_tests/perftestsrunner_unittest.py

    r114076 r114149  
    376376        self.assertEqual(self._collect_tests_and_sort_test_name(runner), ['inspector/test1.html', 'inspector/test2.html'])
    377377
     378    def test_collect_tests_with_page_load_svg(self):
     379        runner = self.create_runner()
     380
     381        def add_file(dirname, filename, content=True):
     382            dirname = runner._host.filesystem.join(runner._base_path, dirname) if dirname else runner._base_path
     383            runner._host.filesystem.maybe_make_directory(dirname)
     384            runner._host.filesystem.files[runner._host.filesystem.join(dirname, filename)] = content
     385
     386        add_file('PageLoad', 'some-svg-test.svg')
     387        tests = runner._collect_tests()
     388        self.assertEqual(len(tests), 1)
     389        self.assertEqual(tests[0].__class__.__name__, 'PageLoadingPerfTest')
     390
    378391    def test_parse_args(self):
    379392        runner = self.create_runner()
Note: See TracChangeset for help on using the changeset viewer.