Changeset 143833 in webkit


Ignore:
Timestamp:
Feb 22, 2013 8:28:47 PM (11 years ago)
Author:
rniwa@webkit.org
Message:

Upload results to perf.webkit.org in addition to the one specified by --test-results-server
​https://bugs.webkit.org/show_bug.cgi?id=108577

Reviewed by Dirk Pranke.

Upload results to perf.webkit.org using new JSON format as well as the host specified by
--test-results-server. The new format is needed to provide extra information perf.webkit.org
need such as the subversion commit time and test URLs. This is a temporarily measure until
we complete the transition and the old JSON format and the code to upload results to
webkit-perf.appspot.com can be deleted.

This patch adds scm.timestamp_of_latest_commit to obtain the timestamp of the latest commit present
in a svn checkout or a git clone. This information is embedded in JSON submitted to perf.webkit.org
so that the app can sort performance test results based on the timestamp of the last commit.

It also changes the repository names returned by port objects to be properly capitalized
human readable names such as WebKit instead of lowercased names such as webkit since these names
are displayed on perf.webkit.org for humans. Several users of this feature has been updated
to explicitly lowercase the names.

  • Scripts/webkitpy/common/checkout/scm/git.py:

(Git.timestamp_of_latest_commit): Added. Obtains the timestamp of the last commit. Unfortunately,
git's timestamp granularity is seconds so we're losing some information compared to using a regular
subversion client. To make matters worse, git doesn't have any option to show ISO-format timestamp in
UTC so we're going to manually fiddle with timezone.

  • Scripts/webkitpy/common/checkout/scm/scm.py:

(SCM.timestamp_of_latest_commit): Added.

  • Scripts/webkitpy/common/checkout/scm/scm_mock.py:

(MockSCM.timestamp_of_latest_commit): Added.

  • Scripts/webkitpy/common/checkout/scm/scm_unittest.py:

(test_timestamp_of_latest_commit): Added a test for Git.timestamp_of_latest_commit.

  • Scripts/webkitpy/common/checkout/scm/svn.py:

(SVN.timestamp_of_latest_commit): Added. With svn, all we need to do is to use --xml option and parse
the timestamp which is always in UTC.

  • Scripts/webkitpy/layout_tests/layout_package/json_results_generator.py:

(JSONResultsGeneratorBase._insert_generic_metadata): Lowercase the name. Note that the name
'chromium' needs to be substituted by 'chrome' for historical reasons.

  • Scripts/webkitpy/layout_tests/port/base.py:

(Port.repository_paths): Return WebKit instead of webkit as noted above.

  • Scripts/webkitpy/layout_tests/port/chromium.py:

(ChromiumPort.repository_paths): Return Chromium instead of chromium as noted above.

  • Scripts/webkitpy/performance_tests/perftestsrunner.py:

(PerfTestsRunner.init): Store the current time in UTC as well as in local time.
(PerfTestsRunner._collect_tests):

(PerfTestsRunner._generate_and_show_results): Retrieve both regular output and one for perf.webkit.org,
and upload them appropriately.

(PerfTestsRunner._generate_results_dict): Store WebKit and Chromium revisions at which tests were ran
in revisions_for_perf_webkit and construct an output for perf.webkit.org.

(PerfTestsRunner._datetime_in_ES5_compatible_iso_format): Added.

(PerfTestsRunner._merge_slave_config_json): Merge slave configuration files into both regular output
and one for perf.webkit.org. Here, we prefix each key with "builder" for perf.webkit.org.
e.g. "processor" would be renamed to "builderProcessor".

(PerfTestsRunner._generate_output_files):

(PerfTestsRunner._upload_json): Added a remote path as an argument since we upload JSONs to /api/report
on perf.webkit.org whereas we upload it to /api/test/report on webkit-perf.appspot.com. Also added the code
to parse response as JSON when possible since perf.webkit.org returns a JSON response as opposed to
webkit-perf.appspot.com which spits out a plaintext response.

  • Scripts/webkitpy/performance_tests/perftestsrunner_integrationtest.py:

(MainTest._test_run_with_json_output.mock_upload_json): Tolerate perf.webkit.org/api/report for now.
(MainTest._test_run_with_json_output): Store a UTC time as perftestrunner would do.
(MainTest.test_run_with_upload_json_should_generate_perf_webkit_json): Added.

  • Scripts/webkitpy/performance_tests/perftestsrunner_unittest.py:

(MainTest.test_upload_json): Moved from itegrationtest.py since it really is a unit test. Also added test
cases to parse JSON responses.
(MainTest.test_upload_json.MockFileUploader): Refactored.
(MainTest.test_upload_json.MockFileUploader.reset): Added.
(MainTest.test_upload_json.MockFileUploader.init):
(MainTest.test_upload_json.MockFileUploader.upload_single_text_file):

Location:
trunk/Tools
Files:
12 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r143824 r143833  
     12013-02-22  Ryosuke Niwa  <rniwa@webkit.org>
     2
     3        Upload results to perf.webkit.org in addition to the one specified by --test-results-server
     4        https://bugs.webkit.org/show_bug.cgi?id=108577
     5
     6        Reviewed by Dirk Pranke.
     7
     8        Upload results to perf.webkit.org using new JSON format as well as the host specified by
     9        --test-results-server. The new format is needed to provide extra information perf.webkit.org
     10        need such as the subversion commit time and test URLs. This is a temporarily measure until
     11        we complete the transition and the old JSON format and the code to upload results to
     12        webkit-perf.appspot.com can be deleted.
     13
     14        This patch adds scm.timestamp_of_latest_commit to obtain the timestamp of the latest commit present
     15        in a svn checkout or a git clone. This information is embedded in JSON submitted to perf.webkit.org
     16        so that the app can sort performance test results based on the timestamp of the last commit.
     17
     18        It also changes the repository names returned by port objects to be properly capitalized
     19        human readable names such as WebKit instead of lowercased names such as webkit since these names
     20        are displayed on perf.webkit.org for humans. Several users of this feature has been updated
     21        to explicitly lowercase the names.
     22
     23
     24        * Scripts/webkitpy/common/checkout/scm/git.py:
     25        (Git.timestamp_of_latest_commit): Added. Obtains the timestamp of the last commit. Unfortunately,
     26        git's timestamp granularity is seconds so we're losing some information compared to using a regular
     27        subversion client. To make matters worse, git doesn't have any option to show ISO-format timestamp in
     28        UTC so we're going to manually fiddle with timezone.
     29
     30        * Scripts/webkitpy/common/checkout/scm/scm.py:
     31        (SCM.timestamp_of_latest_commit): Added.
     32
     33        * Scripts/webkitpy/common/checkout/scm/scm_mock.py:
     34        (MockSCM.timestamp_of_latest_commit): Added.
     35
     36        * Scripts/webkitpy/common/checkout/scm/scm_unittest.py:
     37        (test_timestamp_of_latest_commit): Added a test for Git.timestamp_of_latest_commit.
     38
     39        * Scripts/webkitpy/common/checkout/scm/svn.py:
     40        (SVN.timestamp_of_latest_commit): Added. With svn, all we need to do is to use --xml option and parse
     41        the timestamp which is always in UTC.
     42
     43        * Scripts/webkitpy/layout_tests/layout_package/json_results_generator.py:
     44        (JSONResultsGeneratorBase._insert_generic_metadata): Lowercase the name. Note that the name
     45        'chromium' needs to be substituted by 'chrome' for historical reasons.
     46
     47        * Scripts/webkitpy/layout_tests/port/base.py:
     48        (Port.repository_paths): Return WebKit instead of webkit as noted above.
     49
     50        * Scripts/webkitpy/layout_tests/port/chromium.py:
     51        (ChromiumPort.repository_paths): Return Chromium instead of chromium as noted above.
     52
     53        * Scripts/webkitpy/performance_tests/perftestsrunner.py:
     54        (PerfTestsRunner.__init__): Store the current time in UTC as well as in local time.
     55        (PerfTestsRunner._collect_tests):
     56
     57        (PerfTestsRunner._generate_and_show_results): Retrieve both regular output and one for perf.webkit.org,
     58        and upload them appropriately.
     59
     60        (PerfTestsRunner._generate_results_dict): Store WebKit and Chromium revisions at which tests were ran
     61        in revisions_for_perf_webkit and construct an output for perf.webkit.org.
     62
     63        (PerfTestsRunner._datetime_in_ES5_compatible_iso_format): Added.
     64
     65        (PerfTestsRunner._merge_slave_config_json): Merge slave configuration files into both regular output
     66        and one for perf.webkit.org. Here, we prefix each key with "builder" for perf.webkit.org.
     67        e.g. "processor" would be renamed to "builderProcessor".
     68
     69        (PerfTestsRunner._generate_output_files):
     70
     71        (PerfTestsRunner._upload_json): Added a remote path as an argument since we upload JSONs to /api/report
     72        on perf.webkit.org whereas we upload it to /api/test/report on webkit-perf.appspot.com. Also added the code
     73        to parse response as JSON when possible since perf.webkit.org returns a JSON response as opposed to
     74        webkit-perf.appspot.com which spits out a plaintext response.
     75
     76        * Scripts/webkitpy/performance_tests/perftestsrunner_integrationtest.py:
     77        (MainTest._test_run_with_json_output.mock_upload_json): Tolerate perf.webkit.org/api/report for now.
     78        (MainTest._test_run_with_json_output): Store a UTC time as perftestrunner would do.
     79        (MainTest.test_run_with_upload_json_should_generate_perf_webkit_json): Added.
     80
     81        * Scripts/webkitpy/performance_tests/perftestsrunner_unittest.py:
     82        (MainTest.test_upload_json): Moved from itegrationtest.py since it really is a unit test. Also added test
     83        cases to parse JSON responses.
     84        (MainTest.test_upload_json.MockFileUploader): Refactored.
     85        (MainTest.test_upload_json.MockFileUploader.reset): Added.
     86        (MainTest.test_upload_json.MockFileUploader.__init__):
     87        (MainTest.test_upload_json.MockFileUploader.upload_single_text_file):
     88
    1892013-02-22  Roger Fong  <roger_fong@apple.com>
    290

  • trunk/Tools/Scripts/webkitpy/common/checkout/scm/git.py

    r142178 r143833  
    2828# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    2929
     30import datetime
    3031import logging
    3132import os
    … …  
    253254            return ""
    254255        return str(match.group('svn_revision'))
     256
     257    def timestamp_of_latest_commit(self, path):
     258        git_log = self._run_git(['log', '-1', '--date=iso', self.find_checkout_root(path)])
     259        match = re.search("^Date:\s*(\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2}) ([+-])(\d{2})(\d{2})$", git_log, re.MULTILINE)
     260        if not match:
     261            return ""
     262
     263        # Manually modify the timezone since Git doesn't have an option to show it in UTC.
     264        # Git also truncates milliseconds but we're going to ignore that for now.
     265        time_with_timezone = datetime.datetime(int(match.group(1)), int(match.group(2)), int(match.group(3)),
     266            int(match.group(4)), int(match.group(5)), int(match.group(6)), 0)
     267
     268        sign = 1 if match.group(7) == '+' else -1
     269        time_without_timezone = time_with_timezone - datetime.timedelta(hours=sign * int(match.group(8)), minutes=int(match.group(9)))
     270        return time_without_timezone.strftime('%Y-%m-%dT%H:%M:%SZ')
    255271
    256272    def prepend_svn_revision(self, diff):

  • trunk/Tools/Scripts/webkitpy/common/checkout/scm/scm.py

    r139712 r143833  
    170170        self._subclass_must_implement()
    171171
     172    def timestamp_of_latest_commit(self, path):
     173        self._subclass_must_implement()
     174
    172175    def create_patch(self, git_commit=None, changed_files=None):
    173176        self._subclass_must_implement()

  • trunk/Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py

    r139712 r143833  
    8585        return '5678'
    8686
     87    def timestamp_of_latest_commit(self, path):
     88        return '2013-02-01 08:48:05 +0000'
     89
    8790    def create_patch(self, git_commit, changed_files=None):
    8891        return "Patch1"

  • trunk/Tools/Scripts/webkitpy/common/checkout/scm/scm_unittest.py

    r140089 r143833  
    15781578    def test_push_local_commits_to_server_without_username_and_with_password(self):
    15791579        self.assertRaises(AuthenticationError, self.make_scm().push_local_commits_to_server, {'password': 'blah'})
     1580
     1581    def test_timestamp_of_latest_commit(self):
     1582        scm = self.make_scm()
     1583        scm.find_checkout_root = lambda path: ''
     1584        scm._run_git = lambda args: 'Date: 2013-02-08 08:05:49 +0000'
     1585        self.assertEqual(scm.timestamp_of_latest_commit('some-path'), '2013-02-08T08:05:49Z')
     1586
     1587        scm._run_git = lambda args: 'Date: 2013-02-08 01:02:03 +0130'
     1588        self.assertEqual(scm.timestamp_of_latest_commit('some-path'), '2013-02-07T23:32:03Z')
     1589
     1590        scm._run_git = lambda args: 'Date: 2013-02-08 01:55:21 -0800'
     1591        self.assertEqual(scm.timestamp_of_latest_commit('some-path'), '2013-02-08T09:55:21Z')

  • trunk/Tools/Scripts/webkitpy/common/checkout/scm/svn.py

    r140508 r143833  
    247247        return self.value_from_svn_info(path, 'Revision')
    248248
     249    def timestamp_of_latest_commit(self, path):
     250        # We use --xml to get timestamps like 2013-02-08T08:18:04.964409Z
     251        info_output = Executive().run_command([self.executable_name, 'info', '--xml'], cwd=path).rstrip()
     252        match = re.search(r"^<date>(?P<value>.+)</date>$", info_output, re.MULTILINE)
     253        return match.group('value')
     254
    249255    # FIXME: This method should be on Checkout.
    250256    def create_patch(self, git_commit=None, changed_files=None):

  • trunk/Tools/Scripts/webkitpy/layout_tests/layout_package/json_results_generator.py

    r130356 r143833  
    527527            # Note: for JSON file's backward-compatibility we use 'chrome' rather
    528528            # than 'chromium' here.
    529             if name == 'chromium':
    530                 name = 'chrome'
     529            lowercase_name = name.lower()
     530            if lowercase_name == 'chromium':
     531                lowercase_name = 'chrome'
    531532            self._insert_item_into_raw_list(results_for_builder,
    532533                self._get_svn_revision(path),
    533                 name + 'Revision')
     534                lowercase_name + 'Revision')
    534535
    535536        self._insert_item_into_raw_list(results_for_builder,

  • trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py

    r142941 r143833  
    10981098    def repository_paths(self):
    10991099        """Returns a list of (repository_name, repository_path) tuples of its depending code base.
    1100         By default it returns a list that only contains a ('webkit', <webkitRepossitoryPath>) tuple."""
    1101 
    1102         # We use LayoutTest directory here because webkit_base isn't a part webkit repository in Chromium port
     1100        By default it returns a list that only contains a ('WebKit', <webkitRepositoryPath>) tuple."""
     1101
     1102        # We use LayoutTest directory here because webkit_base isn't a part of WebKit repository in Chromium port
    11031103        # where turnk isn't checked out as a whole.
    1104         return [('webkit', self.layout_tests_dir())]
     1104        return [('WebKit', self.layout_tests_dir())]
    11051105
    11061106    _WDIFF_DEL = '##WDIFF_DEL##'

  • trunk/Tools/Scripts/webkitpy/layout_tests/port/chromium.py

    r142941 r143833  
    365365    def repository_paths(self):
    366366        repos = super(ChromiumPort, self).repository_paths()
    367         repos.append(('chromium', self.path_from_chromium_base('build')))
     367        repos.append(('Chromium', self.path_from_chromium_base('build')))
    368368        return repos
    369369

  • trunk/Tools/Scripts/webkitpy/performance_tests/perftestsrunner.py

    r138058 r143833  
    3434import optparse
    3535import time
     36import datetime
    3637
    3738from webkitpy.common import find_files
    … …  
    6869        self._results = {}
    6970        self._timestamp = time.time()
     71        self._utc_timestamp = datetime.datetime.utcnow()
    7072        self._needs_http = None
    7173        self._has_http_lock = False
    … …  
    127129
    128130    def _collect_tests(self):
    129         """Return the list of tests found."""
    130 
    131131        test_extensions = ['.html', '.svg']
    132132        if self._options.replay:
    … …  
    209209        options = self._options
    210210        output_json_path = self._output_json_path()
    211         output = self._generate_results_dict(self._timestamp, options.description, options.platform, options.builder_name, options.build_number)
     211        output, perf_webkit_output = self._generate_results_dict(self._timestamp, options.description, options.platform, options.builder_name, options.build_number)
    212212
    213213        if options.slave_config_json_path:
    214             output = self._merge_slave_config_json(options.slave_config_json_path, output)
     214            output, perf_webkit_output = self._merge_slave_config_json(options.slave_config_json_path, output, perf_webkit_output)
    215215            if not output:
    216216                return self.EXIT_CODE_BAD_SOURCE_JSON
    … …  
    219219        if not output:
    220220            return self.EXIT_CODE_BAD_MERGE
     221        perf_webkit_output = [perf_webkit_output]
    221222
    222223        results_page_path = self._host.filesystem.splitext(output_json_path)[0] + '.html'
    223         self._generate_output_files(output_json_path, results_page_path, output)
     224        perf_webkit_json_path = self._host.filesystem.splitext(output_json_path)[0] + '-perf-webkit.json' if options.test_results_server else None
     225        self._generate_output_files(output_json_path, perf_webkit_json_path, results_page_path, output, perf_webkit_output)
    224226
    225227        if options.test_results_server:
    226228            if not self._upload_json(options.test_results_server, output_json_path):
     229                return self.EXIT_CODE_FAILED_UPLOADING
     230
     231            # FIXME: Remove this code once we've made transition to use perf.webkit.org
     232            if not self._upload_json('perf.webkit.org', perf_webkit_json_path, "/api/report"):
    227233                return self.EXIT_CODE_FAILED_UPLOADING
    228234
    … …  
    234240        if description:
    235241            contents['description'] = description
     242
     243        revisions_for_perf_webkit = {}
    236244        for (name, path) in self._port.repository_paths():
    237245            scm = SCMDetector(self._host.filesystem, self._host.executive).detect_scm_system(path) or self._host.scm()
    238             contents[name + '-revision'] = scm.svn_revision(path)
     246            revision = scm.svn_revision(path)
     247            contents[name.lower() + '-revision'] = revision
     248            revisions_for_perf_webkit[name] = {'revision': str(revision), 'timestamp': scm.timestamp_of_latest_commit(path)}
    239249
    240250        # FIXME: Add --branch or auto-detect the branch we're in
    … …  
    244254                contents[key] = value
    245255
    246         return contents
    247 
    248     def _merge_slave_config_json(self, slave_config_json_path, output):
     256        contents_for_perf_webkit = {
     257            'builderName': builder_name,
     258            'buildNumber': str(build_number),
     259            'buildTime': self._datetime_in_ES5_compatible_iso_format(self._utc_timestamp),
     260            'platform': platform,
     261            'revisions': revisions_for_perf_webkit,
     262            'tests': {}}
     263
     264        # FIXME: Make this function shorter once we've transitioned to use perf.webkit.org.
     265        for metric_full_name, result in self._results.iteritems():
     266            if not isinstance(result, dict):  # We can't reports results without indivisual measurements.
     267                continue
     268
     269            assert metric_full_name.count(':') <= 1
     270            test_full_name, _, metric = metric_full_name.partition(':')
     271            if not metric:
     272                metric = {'fps': 'FrameRate', 'runs/s': 'Runs', 'ms': 'Time'}[result['unit']]
     273
     274            tests = contents_for_perf_webkit['tests']
     275            path = test_full_name.split('/')
     276            for i in range(0, len(path)):
     277                # FIXME: We shouldn't assume HTML extension.
     278                is_last_token = i + 1 == len(path)
     279                url = 'http://trac.webkit.org/browser/trunk/PerformanceTests/' + '/'.join(path[0:i + 1])
     280                if is_last_token:
     281                    url += '.html'
     282
     283                tests.setdefault(path[i], {'url': url})
     284                current_test = tests[path[i]]
     285                if is_last_token:
     286                    current_test.setdefault('metrics', {})
     287                    assert metric not in current_test['metrics']
     288                    current_test['metrics'][metric] = {'current': result['values']}
     289                else:
     290                    current_test.setdefault('tests', {})
     291                    tests = current_test['tests']
     292
     293        return contents, contents_for_perf_webkit
     294
     295    @staticmethod
     296    def _datetime_in_ES5_compatible_iso_format(datetime):
     297        return datetime.strftime('%Y-%m-%dT%H:%M:%S.%f')
     298
     299    def _merge_slave_config_json(self, slave_config_json_path, contents, contents_for_perf_webkit):
    249300        if not self._host.filesystem.isfile(slave_config_json_path):
    250301            _log.error("Missing slave configuration JSON file: %s" % slave_config_json_path)
    251             return None
     302            return None, None
    252303
    253304        try:
    254305            slave_config_json = self._host.filesystem.open_text_file_for_reading(slave_config_json_path)
    255306            slave_config = json.load(slave_config_json)
    256             return dict(slave_config.items() + output.items())
     307            contents = dict(slave_config.items() + contents.items())
     308            for key in slave_config:
     309                contents_for_perf_webkit['builder' + key.capitalize()] = slave_config[key]
     310            return contents, contents_for_perf_webkit
    257311        except Exception, error:
    258312            _log.error("Failed to merge slave configuration JSON file %s: %s" % (slave_config_json_path, error))
    259         return None
     313        return None, None
    260314
    261315    def _merge_outputs_if_needed(self, output_json_path, output):
    … …  
    269323        return None
    270324
    271     def _generate_output_files(self, output_json_path, results_page_path, output):
     325    def _generate_output_files(self, output_json_path, perf_webkit_json_path, results_page_path, output, perf_webkit_output):
    272326        filesystem = self._host.filesystem
    273327
    274328        json_output = json.dumps(output)
    275329        filesystem.write_text_file(output_json_path, json_output)
     330
     331        if perf_webkit_json_path:
     332            filesystem.write_text_file(perf_webkit_json_path, json.dumps(perf_webkit_output))
    276333
    277334        if results_page_path:
    … …  
    285342            filesystem.write_text_file(results_page_path, results_page)
    286343
    287     def _upload_json(self, test_results_server, json_path, file_uploader=FileUploader):
    288         uploader = file_uploader("https://%s/api/test/report" % test_results_server, 120)
     344    def _upload_json(self, test_results_server, json_path, host_path="/api/test/report", file_uploader=FileUploader):
     345        url = "https://%s%s" % (test_results_server, host_path)
     346        uploader = file_uploader(url, 120)
    289347        try:
    290348            response = uploader.upload_single_text_file(self._host.filesystem, 'application/json', json_path)
    291349        except Exception, error:
    292             _log.error("Failed to upload JSON file in 120s: %s" % error)
     350            _log.error("Failed to upload JSON file to %s in 120s: %s" % (url, error))
    293351            return False
    294352
    295353        response_body = [line.strip('\n') for line in response]
    296354        if response_body != ['OK']:
    297             _log.error("Uploaded JSON but got a bad response:")
    298             for line in response_body:
    299                 _log.error(line)
    300             return False
    301 
    302         _log.info("JSON file uploaded.")
     355            try:
     356                parsed_response = json.loads('\n'.join(response_body))
     357            except:
     358                _log.error("Uploaded JSON to %s but got a bad response:" % url)
     359                for line in response_body:
     360                    _log.error(line)
     361                return False
     362            if parsed_response.get('status') != 'OK':
     363                _log.error("Uploaded JSON to %s but got an error:" % url)
     364                _log.error(json.dumps(parsed_response, indent=4))
     365                return False
     366
     367        _log.info("JSON file uploaded to %s." % url)
    303368        return True
    304369

  • trunk/Tools/Scripts/webkitpy/performance_tests/perftestsrunner_integrationtest.py

    r140510 r143833  
    3030
    3131import StringIO
     32import datetime
    3233import json
    3334import re
    … …  
    322323        uploaded = [False]
    323324
    324         def mock_upload_json(hostname, json_path):
    325             self.assertEqual(hostname, 'some.host')
    326             self.assertEqual(json_path, '/mock-checkout/output.json')
     325        def mock_upload_json(hostname, json_path, host_path=None):
     326            # FIXME: Get rid of the hard-coded perf.webkit.org once we've completed the transition.
     327            self.assertIn(hostname, ['some.host', 'perf.webkit.org'])
     328            self.assertIn(json_path, ['/mock-checkout/output.json', '/mock-checkout/output-perf-webkit.json'])
     329            self.assertIn(host_path, [None, '/api/report'])
    327330            uploaded[0] = upload_suceeds
    328331            return upload_suceeds
    … …  
    330333        runner._upload_json = mock_upload_json
    331334        runner._timestamp = 123456789
     335        runner._utc_timestamp = datetime.datetime(2013, 2, 8, 15, 19, 37, 460000)
    332336        output_capture = OutputCapture()
    333337        output_capture.capture_output()
    … …  
    522526        self._test_run_with_json_output(runner, port.host.filesystem, upload_suceeds=False, expected_exit_code=PerfTestsRunner.EXIT_CODE_FAILED_UPLOADING)
    523527
    524     def test_upload_json(self):
    525         runner, port = self.create_runner()
    526         port.host.filesystem.files['/mock-checkout/some.json'] = 'some content'
    527 
    528         called = []
    529         upload_single_text_file_throws = False
    530         upload_single_text_file_return_value = StringIO.StringIO('OK')
    531 
    532         class MockFileUploader:
    533             def __init__(mock, url, timeout):
    534                 self.assertEqual(url, 'https://some.host/api/test/report')
    535                 self.assertTrue(isinstance(timeout, int) and timeout)
    536                 called.append('FileUploader')
    537 
    538             def upload_single_text_file(mock, filesystem, content_type, filename):
    539                 self.assertEqual(filesystem, port.host.filesystem)
    540                 self.assertEqual(content_type, 'application/json')
    541                 self.assertEqual(filename, 'some.json')
    542                 called.append('upload_single_text_file')
    543                 if upload_single_text_file_throws:
    544                     raise "Some exception"
    545                 return upload_single_text_file_return_value
    546 
    547         runner._upload_json('some.host', 'some.json', MockFileUploader)
    548         self.assertEqual(called, ['FileUploader', 'upload_single_text_file'])
    549 
    550         output = OutputCapture()
    551         output.capture_output()
    552         upload_single_text_file_return_value = StringIO.StringIO('Some error')
    553         runner._upload_json('some.host', 'some.json', MockFileUploader)
    554         _, _, logs = output.restore_output()
    555         self.assertEqual(logs, 'Uploaded JSON but got a bad response:\nSome error\n')
    556 
    557         # Throwing an exception upload_single_text_file shouldn't blow up _upload_json
    558         called = []
    559         upload_single_text_file_throws = True
    560         runner._upload_json('some.host', 'some.json', MockFileUploader)
    561         self.assertEqual(called, ['FileUploader', 'upload_single_text_file'])
     528    def test_run_with_upload_json_should_generate_perf_webkit_json(self):
     529        runner, port = self.create_runner_and_setup_results_template(args=['--output-json-path=/mock-checkout/output.json',
     530            '--test-results-server', 'some.host', '--platform', 'platform1', '--builder-name', 'builder1', '--build-number', '123',
     531            '--slave-config-json-path=/mock-checkout/slave-config.json'])
     532        port.host.filesystem.write_text_file('/mock-checkout/slave-config.json', '{"key": "value1"}')
     533
     534        self._test_run_with_json_output(runner, port.host.filesystem, upload_suceeds=True)
     535        generated_json = json.loads(port.host.filesystem.files['/mock-checkout/output-perf-webkit.json'])
     536        self.assertTrue(isinstance(generated_json, list))
     537        self.assertEqual(len(generated_json), 1)
     538
     539        output = generated_json[0]
     540        self.maxDiff = None
     541        self.assertEqual(output['platform'], 'platform1')
     542        self.assertEqual(output['buildNumber'], '123')
     543        self.assertEqual(output['buildTime'], '2013-02-08T15:19:37.460000')
     544        self.assertEqual(output['builderName'], 'builder1')
     545        self.assertEqual(output['builderKey'], 'value1')
     546        self.assertEqual(output['revisions'], {'WebKit': {'revision': '5678', 'timestamp': '2013-02-01 08:48:05 +0000'}})
     547        self.assertEqual(output['tests'].keys(), ['Bindings'])
     548        self.assertEqual(sorted(output['tests']['Bindings'].keys()), ['tests', 'url'])
     549        self.assertEqual(output['tests']['Bindings']['url'], 'http://trac.webkit.org/browser/trunk/PerformanceTests/Bindings')
     550        self.assertEqual(output['tests']['Bindings']['tests'].keys(), ['event-target-wrapper'])
     551        self.assertEqual(output['tests']['Bindings']['tests']['event-target-wrapper'], {
     552            'url': 'http://trac.webkit.org/browser/trunk/PerformanceTests/Bindings/event-target-wrapper.html',
     553            'metrics': {'Time': {'current': [1486.0, 1471.0, 1510.0, 1505.0, 1478.0, 1490.0]}}})

  • trunk/Tools/Scripts/webkitpy/performance_tests/perftestsrunner_unittest.py

    r140510 r143833  
    3535
    3636from webkitpy.common.host_mock import MockHost
     37from webkitpy.common.system.outputcapture import OutputCapture
    3738from webkitpy.layout_tests.port.test import TestPort
    3839from webkitpy.performance_tests.perftestsrunner import PerfTestsRunner
    … …  
    148149        self.assertEqual(options.slave_config_json_path, 'a/source.json')
    149150        self.assertEqual(options.test_results_server, 'somehost')
     151
     152    def test_upload_json(self):
     153        runner, port = self.create_runner()
     154        port.host.filesystem.files['/mock-checkout/some.json'] = 'some content'
     155
     156        class MockFileUploader:
     157            called = []
     158            upload_single_text_file_throws = False
     159            upload_single_text_file_return_value = None
     160
     161            @classmethod
     162            def reset(cls):
     163                cls.called = []
     164                cls.upload_single_text_file_throws = False
     165                cls.upload_single_text_file_return_value = None
     166
     167            def __init__(mock, url, timeout):
     168                self.assertEqual(url, 'https://some.host/some/path')
     169                self.assertTrue(isinstance(timeout, int) and timeout)
     170                mock.called.append('FileUploader')
     171
     172            def upload_single_text_file(mock, filesystem, content_type, filename):
     173                self.assertEqual(filesystem, port.host.filesystem)
     174                self.assertEqual(content_type, 'application/json')
     175                self.assertEqual(filename, 'some.json')
     176                mock.called.append('upload_single_text_file')
     177                if mock.upload_single_text_file_throws:
     178                    raise Exception
     179                return mock.upload_single_text_file_return_value
     180
     181        MockFileUploader.upload_single_text_file_return_value = StringIO.StringIO('OK')
     182        self.assertTrue(runner._upload_json('some.host', 'some.json', '/some/path', MockFileUploader))
     183        self.assertEqual(MockFileUploader.called, ['FileUploader', 'upload_single_text_file'])
     184
     185        MockFileUploader.reset()
     186        MockFileUploader.upload_single_text_file_return_value = StringIO.StringIO('Some error')
     187        output = OutputCapture()
     188        output.capture_output()
     189        self.assertFalse(runner._upload_json('some.host', 'some.json', '/some/path', MockFileUploader))
     190        _, _, logs = output.restore_output()
     191        self.assertEqual(logs, 'Uploaded JSON to https://some.host/some/path but got a bad response:\nSome error\n')
     192
     193        # Throwing an exception upload_single_text_file shouldn't blow up _upload_json
     194        MockFileUploader.reset()
     195        MockFileUploader.upload_single_text_file_throws = True
     196        self.assertFalse(runner._upload_json('some.host', 'some.json', '/some/path', MockFileUploader))
     197        self.assertEqual(MockFileUploader.called, ['FileUploader', 'upload_single_text_file'])
     198
     199        MockFileUploader.reset()
     200        MockFileUploader.upload_single_text_file_return_value = StringIO.StringIO('{"status": "OK"}')
     201        self.assertTrue(runner._upload_json('some.host', 'some.json', '/some/path', MockFileUploader))
     202        self.assertEqual(MockFileUploader.called, ['FileUploader', 'upload_single_text_file'])
     203
     204        MockFileUploader.reset()
     205        MockFileUploader.upload_single_text_file_return_value = StringIO.StringIO('{"status": "SomethingHasFailed", "failureStored": false}')
     206        output = OutputCapture()
     207        output.capture_output()
     208        self.assertFalse(runner._upload_json('some.host', 'some.json', '/some/path', MockFileUploader))
     209        _, _, logs = output.restore_output()
     210        serialized_json = json.dumps({'status': 'SomethingHasFailed', 'failureStored': False}, indent=4)
     211        self.assertEqual(logs, 'Uploaded JSON to https://some.host/some/path but got an error:\n%s\n' % serialized_json)
Note: See TracChangeset for help on using the changeset viewer.