Changeset 243030 in webkit


Ignore:
Timestamp:
Mar 15, 2019 7:48:57 PM (5 years ago)
Author:
Jonathan Bedard
Message:

webkitpy: Upload test results
​https://bugs.webkit.org/show_bug.cgi?id=195755
<rdar://problem/48896182>

Reviewed by Aakash Jain.

Establish a new format for uploading results that is not tied to layout tests, apply
that format to webkitpy tests.

  • Scripts/webkitpy/common/checkout/scm/git.py:

(Git.native_branch): Return what branch the current checkout is on.

  • Scripts/webkitpy/common/checkout/scm/scm_mock.py:
  • Scripts/webkitpy/common/checkout/scm/scm_unittest.py:
  • Scripts/webkitpy/common/checkout/scm/svn.py:

(SVN.native_branch): Ditto.

  • Scripts/webkitpy/common/system/platforminfo.py:

(PlatformInfo.build_version): Return a build version for Mac.

  • Scripts/webkitpy/common/system/platforminfo_mock.py:

(MockPlatformInfo.init):
(MockPlatformInfo.build_version):

  • Scripts/webkitpy/results: Added.
  • Scripts/webkitpy/results/init.py: Added.
  • Scripts/webkitpy/results/options.py: Added.

(upload_options): OptParse list for upload options.

  • Scripts/webkitpy/results/upload.py: Added.

(Upload): Class which enforces the upload format expected by the results server.
(Upload.Expectations):
(Upload.create_configuration):
(Upload.create_commit):
(Upload.create_details):
(Upload.create_run_stats):
(Upload.create_test_result):
(Upload.init):
(Upload.Encoder): Encode Upload object as json.
(Upload.upload): Upload results to the results server, returning 'True' if the upload is successful.

  • Scripts/webkitpy/results/upload_unittest.py: Added.
  • Scripts/webkitpy/test/main.py:

(Tester._parse_args): Add upload arguments.
(Tester._run_tests): Allow results to be uploaded.

  • Scripts/webkitpy/test/runner.py:

(Runner.init): Record which tests were run, rather than just counting them.
(Runner.handle):

  • Scripts/webkitpy/test/runner_unittest.py:

(RunnerTest.test_run):

  • Scripts/webkitpy/thirdparty/init.py:

(AutoinstallImportHook.find_module): Add requests auto-install.
(AutoinstallImportHook._install_requests):

  • Scripts/webkitpy/tool/commands/queues_unittest.py:

(PatchProcessingQueueTest.test_upload_results_archive_for_patch): Update os name for testing.

Location:
trunk/Tools
Files:
5 added
12 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r243021 r243030  
     12019-03-15  Jonathan Bedard  <jbedard@apple.com>
     2
     3        webkitpy: Upload test results
     4        https://bugs.webkit.org/show_bug.cgi?id=195755
     5        <rdar://problem/48896182>
     6
     7        Reviewed by Aakash Jain.
     8
     9        Establish a new format for uploading results that is not tied to layout tests, apply
     10        that format to webkitpy tests.
     11
     12        * Scripts/webkitpy/common/checkout/scm/git.py:
     13        (Git.native_branch): Return what branch the current checkout is on.
     14        * Scripts/webkitpy/common/checkout/scm/scm_mock.py:
     15        * Scripts/webkitpy/common/checkout/scm/scm_unittest.py:
     16        * Scripts/webkitpy/common/checkout/scm/svn.py:
     17        (SVN.native_branch): Ditto.
     18        * Scripts/webkitpy/common/system/platforminfo.py:
     19        (PlatformInfo.build_version): Return a build version for Mac.
     20        * Scripts/webkitpy/common/system/platforminfo_mock.py:
     21        (MockPlatformInfo.__init__):
     22        (MockPlatformInfo.build_version):
     23        * Scripts/webkitpy/results: Added.
     24        * Scripts/webkitpy/results/__init__.py: Added.
     25        * Scripts/webkitpy/results/options.py: Added.
     26        (upload_options): OptParse list for upload options.
     27        * Scripts/webkitpy/results/upload.py: Added.
     28        (Upload): Class which enforces the upload format expected by the results server.
     29        (Upload.Expectations):
     30        (Upload.create_configuration):
     31        (Upload.create_commit):
     32        (Upload.create_details):
     33        (Upload.create_run_stats):
     34        (Upload.create_test_result):
     35        (Upload.__init__):
     36        (Upload.Encoder): Encode Upload object as json.
     37        (Upload.upload): Upload results to the results server, returning 'True' if the upload is successful.
     38        * Scripts/webkitpy/results/upload_unittest.py: Added.
     39        * Scripts/webkitpy/test/main.py:
     40        (Tester._parse_args): Add upload arguments.
     41        (Tester._run_tests): Allow results to be uploaded.
     42        * Scripts/webkitpy/test/runner.py:
     43        (Runner.__init__): Record which tests were run, rather than just counting them.
     44        (Runner.handle):
     45        * Scripts/webkitpy/test/runner_unittest.py:
     46        (RunnerTest.test_run):
     47        * Scripts/webkitpy/thirdparty/__init__.py:
     48        (AutoinstallImportHook.find_module): Add requests auto-install.
     49        (AutoinstallImportHook._install_requests):
     50        * Scripts/webkitpy/tool/commands/queues_unittest.py:
     51        (PatchProcessingQueueTest.test_upload_results_archive_for_patch): Update os name for testing.
     52
    1532019-03-15  Wenson Hsieh  <wenson_hsieh@apple.com>
    254

  • trunk/Tools/Scripts/webkitpy/common/checkout/scm/git.py

    r234096 r243030  
    291291        return self._run_git(['-C', self.find_checkout_root(path), 'log', '-1', '--pretty=format:%H'])
    292292
     293    def native_branch(self, path):
     294        result = self._run_git(['-C', self.find_checkout_root(path), 'rev-parse', '--abbrev-ref', 'HEAD']).rstrip()
     295
     296        # For git-svn
     297        if result.startswith('heads'):
     298            return result[6:]
     299        return result
     300
    293301    def svn_url(self):
    294302        git_command = ['svn', 'info']

  • trunk/Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py

    r213562 r243030  
    8888        return self.svn_revision(path)
    8989
     90    def native_branch(self, path):
     91        return 'trunk'
     92
    9093    def timestamp_of_revision(self, path, revision):
    9194        return '2013-02-01 08:48:05 +0000'

  • trunk/Tools/Scripts/webkitpy/common/checkout/scm/scm_unittest.py

    r234096 r243030  
    910910        self.assertEqual(self.scm.native_revision('.'), '5')
    911911
     912    def test_native_branch(self):
     913        self.assertEqual(self.scm.native_branch('.'), 'trunk')
     914
    912915    def test_propset_propget(self):
    913916        filepath = os.path.join(self.svn_checkout_path, "test_file")
    … …  
    11131116        self.assertEqual(scm.native_revision(scm.checkout_root), run_command(command).strip())
    11141117
     1118    def test_native_branch(self):
     1119        scm = self.tracking_scm
     1120        self.assertEqual('master', scm.native_branch(scm.checkout_root))
     1121
    11151122    def test_rename_files(self):
    11161123        scm = self.tracking_scm
    … …  
    16281635        self.assertEqual(self.scm.native_revision(self.git_checkout_path), run_command(command).strip())
    16291636
     1637    def test_native_branch(self):
     1638        self.assertEqual('trunk', self.scm.native_branch(self.git_checkout_path))
     1639
    16301640    def test_to_object_name(self):
    16311641        relpath = 'test_file_commit1'

  • trunk/Tools/Scripts/webkitpy/common/checkout/scm/svn.py

    r233474 r243030  
    8484        self._bogus_dir = None
    8585        if patch_directories == []:
    86             raise Exception(message='Empty list of patch directories passed to SCM.__init__')
     86            raise Exception('Empty list of patch directories passed to SCM.__init__')
    8787        elif patch_directories == None:
    8888            self._patch_directories = [self._filesystem.relpath(cwd, self.checkout_root)]
    … …  
    279279        return self.svn_revision(path)
    280280
     281    def native_branch(self, path):
     282        relative_url = self.value_from_svn_info(path, 'Relative URL')[2:]
     283        if relative_url.startswith('trunk'):
     284            return 'trunk'
     285        elif relative_url.startswith('branch'):
     286            return relative_url.split('/')[1]
     287        raise Exception('{} is not a branch'.format(relative_url.split('/')[0]))
     288
    281289    def timestamp_of_revision(self, path, revision):
    282290        # We use --xml to get timestamps like 2013-02-08T08:18:04.964409Z

  • trunk/Tools/Scripts/webkitpy/common/system/platforminfo.py

    r238903 r243030  
    147147            return sys.maxint
    148148
     149    def build_version(self):
     150        if self.is_mac():
     151            return self._executive.run_command(['/usr/bin/sw_vers', '-buildVersion'], return_stderr=False, ignore_errors=True).rstrip()
     152        return None
     153
    149154    def xcode_sdk_version(self, sdk_name):
    150155        if self.is_mac():

  • trunk/Tools/Scripts/webkitpy/common/system/platforminfo_mock.py

    r229099 r243030  
    3333
    3434class MockPlatformInfo(object):
    35     def __init__(self, os_name='mac', os_version=Version.from_name('Snow Leopard')):
     35    def __init__(self, os_name='mac', os_version=Version.from_name('High Sierra')):
    3636        assert isinstance(os_version, Version)
    3737        self.os_name = os_name
    … …  
    7979        return 80
    8080
     81    def build_version(self):
     82        if self.is_mac():
     83            return '17A405'
     84        return None
     85
    8186    def xcode_sdk_version(self, sdk_name):
    8287        return Version(8, 1)

  • trunk/Tools/Scripts/webkitpy/test/main.py

    r236667 r243030  
    11# Copyright (C) 2012 Google, Inc.
    22# Copyright (C) 2010 Chris Jerdonek (cjerdonek@webkit.org)
    3 # Copyright (C) 2018 Apple Inc. All rights reserved.
     3# Copyright (C) 2018-2019 Apple Inc. All rights reserved.
    44#
    55# Redistribution and use in source and binary forms, with or without
    … …  
    3939
    4040from webkitpy.common.system.logutils import configure_logging
    41 from webkitpy.common.system.executive import Executive, ScriptError
     41from webkitpy.common.system.executive import ScriptError
    4242from webkitpy.common.system.filesystem import FileSystem
    43 from webkitpy.common.system.systemhost import SystemHost
     43from webkitpy.common.host import Host
    4444from webkitpy.port.config import Config
    4545from webkitpy.test.finder import Finder
    4646from webkitpy.test.printer import Printer
    4747from webkitpy.test.runner import Runner, unit_test_name
     48from webkitpy.results.upload import Upload
     49from webkitpy.results.options import upload_options
    4850
    4951_log = logging.getLogger(__name__)
    5052
    51 _host = SystemHost()
     53_host = Host()
    5254_webkit_root = None
    5355
    … …  
    131133            help='Set the configuration to Release')
    132134        parser.add_option_group(configuration_group)
     135
     136        upload_group = optparse.OptionGroup(parser, 'Upload Options')
     137        upload_group.add_options(upload_options())
     138        parser.add_option_group(upload_group)
    133139
    134140        parser.add_option('-a', '--all', action='store_true', default=False,
    … …  
    182188        autoinstall_everything()
    183189
     190        start_time = time.time()
     191        config = Config(_host.executive, self.finder.filesystem)
     192        configuration_to_use = self._options.configuration or config.default_configuration()
    184193        if will_run_lldb_webkit_tests:
    185194            self.printer.write_update('Building lldbWebKitTester ...')
    186195            build_lldbwebkittester = self.finder.filesystem.join(_webkit_root, 'Tools', 'Scripts', 'build-lldbwebkittester')
    187             config = Config(_host.executive, self.finder.filesystem)
    188             configuration_to_use = self._options.configuration or config.default_configuration()
    189196            try:
    190197                _host.executive.run_and_throw_if_fail([build_lldbwebkittester, config.flag_for_configuration(configuration_to_use)], quiet=(not bool(self._options.verbose)))
    … …  
    219226        test_runner.run(parallel_tests, self._options.child_processes)
    220227        test_runner.run(serial_tests, 1)
     228        end_time = time.time()
    221229
    222230        self.printer.print_result(time.time() - start)
    … …  
    233241            cov.stop()
    234242            cov.save()
     243
     244        failed_uploads = 0
     245        if self._options.report_urls:
     246            self.printer.meter.writeln('\n')
     247            self.printer.write_update('Preparing upload data ...')
     248
     249            # Empty test results indicate a PASS.
     250            results = {test: {} for test in test_runner.tests_run}
     251            for test, errors in test_runner.errors:
     252                results[test] = Upload.create_test_result(actual=Upload.Expectations.ERROR, log='/n'.join(errors))
     253            for test, failures in test_runner.failures:
     254                results[test] = Upload.create_test_result(actual=Upload.Expectations.FAIL, log='/n'.join(failures))
     255
     256            _host.initialize_scm()
     257            upload = Upload(
     258                suite='webkitpy-tests',
     259                configuration=Upload.create_configuration(
     260                    platform=_host.platform.os_name,
     261                    version=str(_host.platform.os_version),
     262                    version_name=_host.platform.os_version_name(),
     263                    style=configuration_to_use,
     264                    sdk=_host.platform.build_version(),
     265                ),
     266                details=Upload.create_details(options=self._options),
     267                commits=[Upload.create_commit(
     268                    repository_id='webkit',
     269                    id=_host.scm().native_revision(_webkit_root),
     270                    branch=_host.scm().native_branch(_webkit_root),
     271                )],
     272                run_stats=Upload.create_run_stats(
     273                    start_time=start_time,
     274                    end_time=end_time,
     275                    tests_skipped=len(test_runner.tests_run) - len(parallel_tests) - len(serial_tests),
     276                ),
     277                results=results,
     278            )
     279            for url in self._options.report_urls:
     280                self.printer.write_update('Uploading to {} ...'.format(url))
     281                failed_uploads = failed_uploads if upload.upload(url, log_line_func=self.printer.meter.writeln) else (failed_uploads + 1)
     282            self.printer.meter.writeln('Uploads completed!')
     283
     284        if self._options.coverage:
    235285            cov.report(show_missing=False)
    236286
    237         return not self.printer.num_errors and not self.printer.num_failures
     287        return not self.printer.num_errors and not self.printer.num_failures and not failed_uploads
    238288
    239289    def _check_imports(self, names):

  • trunk/Tools/Scripts/webkitpy/test/runner.py

    r222510 r243030  
    4141        self.printer = printer
    4242        self.loader = loader
    43         self.tests_run = 0
     43        self.tests_run = []
    4444        self.errors = []
    4545        self.failures = []
    … …  
    6161            return
    6262
    63         self.tests_run += 1
     63        self.tests_run.append(test_name)
    6464        if failures:
    6565            self.failures.append((test_name, failures))

  • trunk/Tools/Scripts/webkitpy/test/runner_unittest.py

    r174136 r243030  
    9393        runner = Runner(Printer(stream, options), loader)
    9494        runner.run(['Foo.test1', 'Foo.test2', 'Foo.test3'], 1)
    95         self.assertEqual(runner.tests_run, 3)
     95        self.assertEqual(len(runner.tests_run), 3)
    9696        self.assertEqual(len(runner.failures), 1)
    9797        self.assertEqual(len(runner.errors), 1)

  • trunk/Tools/Scripts/webkitpy/thirdparty/__init__.py

    r241759 r243030  
    11# Copyright (C) 2010 Chris Jerdonek (cjerdonek@webkit.org)
     2# Copyright (C) 2019 Apple Inc. All rights reserved.
    23#
    34# Redistribution and use in source and binary forms, with or without
    … …  
    115116        elif '.pytest' in fullname:
    116117            self._install_pytest()
     118        elif '.requests' in fullname:
     119            self._install_requests()
    117120
    118121    def _install_mechanize(self):
    … …  
    162165                              "pytest-3.6.2/src/pytest.py")
    163166
     167    def _install_requests(self):
     168        self._ensure_autoinstalled_dir_is_in_sys_path()
     169        self._install("https://files.pythonhosted.org/packages/06/b8/d1ea38513c22e8c906275d135818fee16ad8495985956a9b7e2bb21942a1/certifi-2019.3.9.tar.gz",
     170                      "certifi-2019.3.9/certifi")
     171        self._install("https://files.pythonhosted.org/packages/fc/bb/a5768c230f9ddb03acc9ef3f0d4a3cf93462473795d18e9535498c8f929d/chardet-3.0.4.tar.gz",
     172                      "chardet-3.0.4/chardet")
     173        self._install("https://files.pythonhosted.org/packages/fc/bb/a5768c230f9ddb03acc9ef3f0d4a3cf93462473795d18e9535498c8f929d/chardet-3.0.4.tar.gz",
     174                      "chardet-3.0.4/chardet")
     175        self._install("https://files.pythonhosted.org/packages/ad/13/eb56951b6f7950cadb579ca166e448ba77f9d24efc03edd7e55fa57d04b7/idna-2.8.tar.gz",
     176                      "idna-2.8/idna")
     177        self._install("https://files.pythonhosted.org/packages/b1/53/37d82ab391393565f2f831b8eedbffd57db5a718216f82f1a8b4d381a1c1/urllib3-1.24.1.tar.gz",
     178                      "urllib3-1.24.1/src/urllib3")
     179        self._install("https://files.pythonhosted.org/packages/52/2c/514e4ac25da2b08ca5a464c50463682126385c4272c18193876e91f4bc38/requests-2.21.0.tar.gz",
     180                      "requests-2.21.0/requests")
     181
    164182    def _install_pylint(self):
    165183        self._ensure_autoinstalled_dir_is_in_sys_path()

  • trunk/Tools/Scripts/webkitpy/tool/commands/queues_unittest.py

    r234856 r243030  
    184184        queue._options.port = None
    185185        patch = queue._tool.bugs.fetch_attachment(10001)
    186         expected_logs = """MOCK add_attachment_to_bug: bug_id=50000, description=Archive of layout-test-results from bot for mac-snowleopard filename=layout-test-results.zip mimetype=None
     186        expected_logs = """MOCK add_attachment_to_bug: bug_id=50000, description=Archive of layout-test-results from bot for mac-highsierra filename=layout-test-results.zip mimetype=None
    187187-- Begin comment --
    188188The attached test failures were seen while running run-webkit-tests on the mock-queue.
    189 Port: mac-snowleopard  Platform: MockPlatform 1.0
     189Port: mac-highsierra  Platform: MockPlatform 1.0
    190190-- End comment --
    191191"""
Note: See TracChangeset for help on using the changeset viewer.