⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 243748 in webkit


Ignore:
Timestamp:
Apr 2, 2019, 10:51:06 AM (7 years ago)
Author:
Kocsen Chung
Message:

Cherry-pick r243732. rdar://problem/49356714

run-api-tests: Upload test results
​https://bugs.webkit.org/show_bug.cgi?id=196323
<rdar://problem/49356714>

Reviewed by Lucas Forschler.

  • Scripts/webkitpy/api_tests/manager.py: (Manager): (Manager.run): Upload results to a results database.
  • Scripts/webkitpy/api_tests/run_api_tests.py: (parse_args): Add upload arguments.
  • Scripts/webkitpy/port/base.py: (Port): (Port.configuration_for_upload): Creates a configuration dictionary for uploading results. (Port.commits_for_upload): Create a list of commits from the WebKit repository tests are run from along with commits from any other associated repositories.
  • Scripts/webkitpy/port/device.py: (Device): (Device.build_version): Access build_versoin of underlying platform device.
  • Scripts/webkitpy/port/ios_simulator_unittest.py: (IOSSimulatorTest): (IOSSimulatorTest.test_configuration_for_upload):
  • Scripts/webkitpy/port/device_port.py: (DevicePort): (DevicePort.configuration_for_upload): Devices are unique because their configuration is not the same as the machine uploading results.
  • Scripts/webkitpy/port/mac.py: (MacPort): (MacPort.configuration_for_upload): Define SDK in upload configuration for Mac.
  • Scripts/webkitpy/port/mac_unittest.py: (MacTest): (MacTest.test_configuration_for_upload):
  • Scripts/webkitpy/xcode/simulated_device.py: (SimulatedDeviceManager._create_device_with_runtime): (SimulatedDevice.init): Create simulated device with a build_version.
  • Scripts/webkitpy/xcode/simulated_device_unittest.py: (test_existing_simulator):

git-svn-id: ​https://svn.webkit.org/repository/webkit/trunk@243732 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Location:
branches/safari-607-branch/Tools
Files:
11 edited

Legend:

Unmodified
Added
Removed

  • branches/safari-607-branch/Tools/ChangeLog

    r243095 r243748  
     12019-04-02  Kocsen Chung  <kocsen_chung@apple.com>
     2
     3        Cherry-pick r243732. rdar://problem/49356714
     4
     5    run-api-tests: Upload test results
     6    https://bugs.webkit.org/show_bug.cgi?id=196323
     7    <rdar://problem/49356714>
     8   
     9    Reviewed by Lucas Forschler.
     10   
     11    * Scripts/webkitpy/api_tests/manager.py:
     12    (Manager):
     13    (Manager.run): Upload results to a results database.
     14    * Scripts/webkitpy/api_tests/run_api_tests.py:
     15    (parse_args): Add upload arguments.
     16    * Scripts/webkitpy/port/base.py:
     17    (Port):
     18    (Port.configuration_for_upload): Creates a configuration dictionary for uploading results.
     19    (Port.commits_for_upload): Create a list of commits from the WebKit repository tests are run from along
     20    with commits from any other associated repositories.
     21    * Scripts/webkitpy/port/device.py:
     22    (Device):
     23    (Device.build_version): Access build_versoin of underlying platform device.
     24    * Scripts/webkitpy/port/ios_simulator_unittest.py:
     25    (IOSSimulatorTest):
     26    (IOSSimulatorTest.test_configuration_for_upload):
     27    * Scripts/webkitpy/port/device_port.py:
     28    (DevicePort):
     29    (DevicePort.configuration_for_upload): Devices are unique because their configuration is not
     30    the same as the machine uploading results.
     31    * Scripts/webkitpy/port/mac.py:
     32    (MacPort):
     33    (MacPort.configuration_for_upload): Define SDK in upload configuration for Mac.
     34    * Scripts/webkitpy/port/mac_unittest.py:
     35    (MacTest):
     36    (MacTest.test_configuration_for_upload):
     37    * Scripts/webkitpy/xcode/simulated_device.py:
     38    (SimulatedDeviceManager._create_device_with_runtime):
     39    (SimulatedDevice.__init__): Create simulated device with a build_version.
     40    * Scripts/webkitpy/xcode/simulated_device_unittest.py:
     41    (test_existing_simulator):
     42   
     43    git-svn-id: https://svn.webkit.org/repository/webkit/trunk@243732 268f45cc-cd09-0410-ab3c-d52691b4dbfc
     44
     45    2019-04-01  Jonathan Bedard  <jbedard@apple.com>
     46
     47            run-api-tests: Upload test results
     48            https://bugs.webkit.org/show_bug.cgi?id=196323
     49            <rdar://problem/49356714>
     50
     51            Reviewed by Lucas Forschler.
     52
     53            * Scripts/webkitpy/api_tests/manager.py:
     54            (Manager):
     55            (Manager.run): Upload results to a results database.
     56            * Scripts/webkitpy/api_tests/run_api_tests.py:
     57            (parse_args): Add upload arguments.
     58            * Scripts/webkitpy/port/base.py:
     59            (Port):
     60            (Port.configuration_for_upload): Creates a configuration dictionary for uploading results.
     61            (Port.commits_for_upload): Create a list of commits from the WebKit repository tests are run from along
     62            with commits from any other associated repositories.
     63            * Scripts/webkitpy/port/device.py:
     64            (Device):
     65            (Device.build_version): Access build_versoin of underlying platform device.
     66            * Scripts/webkitpy/port/ios_simulator_unittest.py:
     67            (IOSSimulatorTest):
     68            (IOSSimulatorTest.test_configuration_for_upload):
     69            * Scripts/webkitpy/port/device_port.py:
     70            (DevicePort):
     71            (DevicePort.configuration_for_upload): Devices are unique because their configuration is not
     72            the same as the machine uploading results.
     73            * Scripts/webkitpy/port/mac.py:
     74            (MacPort):
     75            (MacPort.configuration_for_upload): Define SDK in upload configuration for Mac.
     76            * Scripts/webkitpy/port/mac_unittest.py:
     77            (MacTest):
     78            (MacTest.test_configuration_for_upload):
     79            * Scripts/webkitpy/xcode/simulated_device.py:
     80            (SimulatedDeviceManager._create_device_with_runtime):
     81            (SimulatedDevice.__init__): Create simulated device with a build_version.
     82            * Scripts/webkitpy/xcode/simulated_device_unittest.py:
     83            (test_existing_simulator):
     84
    1852019-03-18  Kocsen Chung  <kocsen_chung@apple.com>
    286

  • branches/safari-607-branch/Tools/Scripts/webkitpy/api_tests/manager.py

    r238763 r243748  
    1 # Copyright (C) 2018 Apple Inc. All rights reserved.
     1# Copyright (C) 2018-2019 Apple Inc. All rights reserved.
    22#
    33# Redistribution and use in source and binary forms, with or without
    … …  
    2323import json
    2424import logging
    25 import os
     25import time
    2626
    2727from webkitpy.api_tests.runner import Runner
    2828from webkitpy.common.system.executive import ScriptError
     29from webkitpy.results.upload import Upload
     30
    2931from webkitpy.xcode.simulated_device import DeviceRequest, SimulatedDeviceManager
    3032
    … …  
    4042    FAILED_COLLECT_TESTS = 2
    4143    FAILED_TESTS = 3
     44    FAILED_UPLOAD = 4
    4245
    4346    def __init__(self, port, options, stream):
    … …  
    155158                raise RuntimeError('Cannot write to {}'.format(json_output))
    156159
     160        start_time = time.time()
     161
    157162        self._stream.write_update('Checking build ...')
    158163        if not self._port.check_api_test_build(self._binaries_for_arguments(args)):
    … …  
    186191            self._stream.writeln('')
    187192
     193        end_time = time.time()
     194
    188195        successful = runner.result_map_by_status(runner.STATUS_PASSED)
    189196        disabled = len(runner.result_map_by_status(runner.STATUS_DISABLED))
    … …  
    198205
    199206        self._stream.writeln('-' * 30)
     207        result = Manager.SUCCESS
    200208        if len(successful) + disabled == len(test_names):
    201209            self._stream.writeln('All tests successfully passed!')
    202210            if json_output:
    203211                self.host.filesystem.write_text_file(json_output, json.dumps(result_dictionary, indent=4))
    204             return Manager.SUCCESS
    205 
    206         self._stream.writeln('Test suite failed')
    207         self._stream.writeln('')
    208 
    209         skipped = []
    210         for test in test_names:
    211             if test not in runner.results:
    212                 skipped.append(test)
    213                 result_dictionary['Skipped'].append({'name': test, 'output': None})
    214         if skipped:
    215             self._stream.writeln('Skipped {} tests'.format(len(skipped)))
     212        else:
     213            self._stream.writeln('Test suite failed')
    216214            self._stream.writeln('')
    217             if self._options.verbose:
    218                 for test in skipped:
    219                     self._stream.writeln('    {}'.format(test))
    220 
    221         self._print_tests_result_with_status(runner.STATUS_FAILED, runner)
    222         self._print_tests_result_with_status(runner.STATUS_CRASHED, runner)
    223         self._print_tests_result_with_status(runner.STATUS_TIMEOUT, runner)
    224 
    225         for test, result in runner.results.iteritems():
    226             status_to_string = {
    227                 runner.STATUS_FAILED: 'Failed',
    228                 runner.STATUS_CRASHED: 'Crashed',
    229                 runner.STATUS_TIMEOUT: 'Timedout',
    230             }.get(result[0])
    231             if not status_to_string:
    232                 continue
    233             result_dictionary[status_to_string].append({'name': test, 'output': result[1]})
    234 
    235         if json_output:
    236             self.host.filesystem.write_text_file(json_output, json.dumps(result_dictionary, indent=4))
    237 
    238         return Manager.FAILED_TESTS
     215
     216            skipped = []
     217            for test in test_names:
     218                if test not in runner.results:
     219                    skipped.append(test)
     220                    result_dictionary['Skipped'].append({'name': test, 'output': None})
     221            if skipped:
     222                self._stream.writeln('Skipped {} tests'.format(len(skipped)))
     223                self._stream.writeln('')
     224                if self._options.verbose:
     225                    for test in skipped:
     226                        self._stream.writeln('    {}'.format(test))
     227
     228            self._print_tests_result_with_status(runner.STATUS_FAILED, runner)
     229            self._print_tests_result_with_status(runner.STATUS_CRASHED, runner)
     230            self._print_tests_result_with_status(runner.STATUS_TIMEOUT, runner)
     231
     232            for test, result in runner.results.iteritems():
     233                status_to_string = {
     234                    runner.STATUS_FAILED: 'Failed',
     235                    runner.STATUS_CRASHED: 'Crashed',
     236                    runner.STATUS_TIMEOUT: 'Timedout',
     237                }.get(result[0])
     238                if not status_to_string:
     239                    continue
     240                result_dictionary[status_to_string].append({'name': test, 'output': result[1]})
     241
     242            if json_output:
     243                self.host.filesystem.write_text_file(json_output, json.dumps(result_dictionary, indent=4))
     244
     245            result = Manager.FAILED_TESTS
     246
     247        if self._options.report_urls:
     248            self._stream.writeln('\n')
     249            self._stream.write_update('Preparing upload data ...')
     250
     251            status_to_test_result = {
     252                runner.STATUS_PASSED: None,
     253                runner.STATUS_FAILED: Upload.Expectations.FAIL,
     254                runner.STATUS_CRASHED: Upload.Expectations.CRASH,
     255                runner.STATUS_TIMEOUT: Upload.Expectations.TIMEOUT,
     256            }
     257            upload = Upload(
     258                suite='api-tests',
     259                configuration=self._port.configuration_for_upload(self._port.target_host(0)),
     260                details=Upload.create_details(options=self._options),
     261                commits=self._port.commits_for_upload(),
     262                run_stats=Upload.create_run_stats(
     263                    start_time=start_time,
     264                    end_time=end_time,
     265                    tests_skipped=len(result_dictionary['Skipped']),
     266                ),
     267                results={test: Upload.create_test_result(actual=status_to_test_result[result[0]])
     268                         for test, result in runner.results.iteritems() if result[0] in status_to_test_result},
     269            )
     270            for url in self._options.report_urls:
     271                self._stream.write_update('Uploading to {} ...'.format(url))
     272                if not upload.upload(url, log_line_func=self._stream.writeln):
     273                    result = Manager.FAILED_UPLOAD
     274            self._stream.writeln('Uploads completed!')
     275
     276        return result

  • branches/safari-607-branch/Tools/Scripts/webkitpy/api_tests/run_api_tests.py

    r235252 r243748  
    3131from webkitpy.layout_tests.views.metered_stream import MeteredStream
    3232from webkitpy.port import configuration_options, platform_options, base, win
     33from webkitpy.results.options import upload_options
    3334
    3435EXCEPTIONAL_EXIT_STATUS = -1
    … …  
    135136                             help='Run all tests, even DISABLED tests'),
    136137    ]))
     138    option_group_definitions.append(('Upload Options', upload_options()))
    137139
    138140    option_parser = optparse.OptionParser(

  • branches/safari-607-branch/Tools/Scripts/webkitpy/port/base.py

    r239566 r243748  
    4646from webkitpy.common import find_files
    4747from webkitpy.common import read_checksum_from_png
     48from webkitpy.common.checkout.scm.detection import SCMDetector
    4849from webkitpy.common.memoized import memoized
    4950from webkitpy.common.prettypatch import PrettyPatch
    … …  
    6263from webkitpy.layout_tests.servers import web_platform_test_server
    6364from webkitpy.layout_tests.servers import websocket_server
     65from webkitpy.results.upload import Upload
    6466
    6567_log = logging.getLogger(__name__)
    … …  
    16291631        # such as closing file descriptors that were implicitly cloned to the worker.
    16301632        pass
     1633
     1634    def configuration_for_upload(self, host=None):
     1635        configuration = self.test_configuration()
     1636        host = self.host or host
     1637
     1638        return Upload.create_configuration(
     1639            platform=host.platform.os_name,
     1640            version=str(host.platform.os_version),
     1641            version_name=host.platform.os_version_name(INTERNAL_TABLE) or host.platform.os_version_name(),
     1642            architecture=configuration.architecture,
     1643            style='guard-malloc' if self.get_option('guard_malloc') else configuration.build_type,
     1644            sdk=host.platform.build_version(),
     1645        )
     1646
     1647    @memoized
     1648    def commits_for_upload(self):
     1649        self.host.initialize_scm()
     1650
     1651        if port_config.apple_additions() and getattr(port_config.apple_additions(), 'repos', False):
     1652            repos = port_config.apple_additions().repos()
     1653        else:
     1654            repos = {}
     1655
     1656        up = os.path.dirname
     1657        repos['webkit'] = up(up(up(up(up(os.path.abspath(__file__))))))
     1658
     1659        commits = []
     1660        for repo_id, path in repos.iteritems():
     1661            scm = SCMDetector(self._filesystem, self._executive).detect_scm_system(path)
     1662            commits.append(Upload.create_commit(
     1663                repository_id=repo_id,
     1664                id=scm.native_revision(path),
     1665                branch=scm.native_branch(path),
     1666            ))
     1667        return commits

  • branches/safari-607-branch/Tools/Scripts/webkitpy/port/device.py

    r222510 r243748  
    9898        return self.platform_device.udid
    9999
     100    @property
     101    def device_type(self):
     102        return self.platform_device.device_type
     103
     104    @property
     105    def build_version(self):
     106        return self.platform_device.build_version
     107
    100108    def __nonzero__(self):
    101109        return self.platform_device is not None

  • branches/safari-607-branch/Tools/Scripts/webkitpy/port/device_port.py

    r239344 r243748  
    1 # Copyright (C) 2018 Apple Inc. All rights reserved.
     1# Copyright (C) 2018-2019 Apple Inc. All rights reserved.
    22#
    33# Redistribution and use in source and binary forms, with or without
    … …  
    2424import traceback
    2525
     26from webkitpy.common.version_name_map import VersionNameMap, PUBLIC_TABLE, INTERNAL_TABLE
    2627from webkitpy.layout_tests.models.test_configuration import TestConfiguration
    2728from webkitpy.port.darwin import DarwinPort
    2829from webkitpy.port.simulator_process import SimulatorProcess
     30from webkitpy.results.upload import Upload
    2931from webkitpy.xcode.device_type import DeviceType
    3032from webkitpy.xcode.simulated_device import DeviceRequest, SimulatedDeviceManager
    … …  
    223225    def device_version(self):
    224226        raise NotImplementedError
     227
     228    def configuration_for_upload(self, host=None):
     229        configuration = self.test_configuration()
     230
     231        device_type = host.device_type if host else self.DEVICE_TYPE
     232        model = device_type.hardware_family
     233        if model and device_type.hardware_type:
     234            model += ' {}'.format(device_type.hardware_type)
     235
     236        version = self.device_version()
     237        version_name = None
     238        for table in [INTERNAL_TABLE, PUBLIC_TABLE]:
     239            version_name = VersionNameMap.map(self.host.platform).to_name(version, platform=device_type.software_variant.lower(), table=table)
     240            if version_name:
     241                break
     242
     243        return Upload.create_configuration(
     244            platform=device_type.software_variant.lower(),
     245            is_simulator=self.DEVICE_MANAGER == SimulatedDeviceManager,
     246            version=str(version),
     247            version_name=version_name,
     248            architecture=configuration.architecture,
     249            style='guard-malloc' if self.get_option('guard_malloc') else configuration.build_type,
     250            model=model,
     251            sdk=host.build_version if host else None,
     252        )

  • branches/safari-607-branch/Tools/Scripts/webkitpy/port/ios_simulator_unittest.py

    r238944 r243748  
    1 # Copyright (C) 2014-2016 Apple Inc. All rights reserved.
     1# Copyright (C) 2014-2019 Apple Inc. All rights reserved.
    22#
    33# Redistribution and use in source and binary forms, with or without
    … …  
    121121        port = self.make_port()
    122122        self.assertEqual(port.max_child_processes(DeviceType.from_string('Apple Watch')), 0)
     123
     124    def test_configuration_for_upload(self):
     125        port = self.make_port()
     126        self.assertEqual(
     127            dict(
     128                platform='ios',
     129                is_simulator=True,
     130                architecture='x86_64',
     131                version='11',
     132                version_name='iOS 11',
     133                style='release',
     134            ),
     135            port.configuration_for_upload(),
     136        )

  • branches/safari-607-branch/Tools/Scripts/webkitpy/port/mac.py

    r238903 r243748  
    11# Copyright (C) 2011 Google Inc. All rights reserved.
    2 # Copyright (C) 2012, 2013, 2016 Apple Inc. All rights reserved.
     2# Copyright (C) 2012-2019 Apple Inc. All rights reserved.
    33#
    44# Redistribution and use in source and binary forms, with or without
    … …  
    281281        worthless_patterns.append((re.compile('.*<<< FFR_Common >>>.*\n'), ''))
    282282        return worthless_patterns
     283
     284    def configuration_for_upload(self, host=None):
     285        host = host or self.host
     286        configuration = super(MacPort, self).configuration_for_upload(host=host)
     287
     288        output = host.executive.run_command(['/usr/sbin/sysctl', 'hw.model']).rstrip()
     289        match = re.match(r'hw.model: (?P<model>.*)', output)
     290        if match:
     291            configuration['model'] = match.group('model')
     292
     293        return configuration

  • branches/safari-607-branch/Tools/Scripts/webkitpy/port/mac_unittest.py

    r232483 r243748  
    11# Copyright (C) 2010 Google Inc. All rights reserved.
    2 # Copyright (C) 2014-2016 Apple Inc. All rights reserved.
     2# Copyright (C) 2014-2019 Apple Inc. All rights reserved.
    33#
    44# Redistribution and use in source and binary forms, with or without
    … …  
    225225        port = self.make_port(options=MockOptions(webkit_test_runner=False), port_name='mac-wk2')
    226226        self.assertEqual(port.driver_name(), 'WebKitTestRunner')
     227
     228    def test_configuration_for_upload(self):
     229        port = self.make_port()
     230        self.assertEqual(
     231            dict(
     232                platform='mac',
     233                is_simulator=False,
     234                architecture='x86_64',
     235                version='10.7',
     236                version_name='Lion',
     237                sdk='17A405',
     238                style='release',
     239            ),
     240            port.configuration_for_upload(),
     241        )

  • branches/safari-607-branch/Tools/Scripts/webkitpy/xcode/simulated_device.py

    r239941 r243748  
    114114            host=host,
    115115            device_type=device_type,
     116            build_version=runtime.build_version,
    116117        ))
    117118        SimulatedDeviceManager.AVAILABLE_DEVICES.append(result)
    … …  
    489490    ]
    490491
    491     def __init__(self, name, udid, host, device_type):
     492    def __init__(self, name, udid, host, device_type, build_version):
    492493        assert device_type.software_version
    493494
    … …  
    495496        self.udid = udid
    496497        self.device_type = device_type
     498        self.build_version = build_version
    497499        self._state = SimulatedDevice.DeviceState.SHUTTING_DOWN
    498500        self._last_updated_state = time.time()

  • branches/safari-607-branch/Tools/Scripts/webkitpy/xcode/simulated_device_unittest.py

    r238940 r243748  
    606606        self.assertEquals(1, len(SimulatedDeviceManager.INITIALIZED_DEVICES))
    607607        self.assertEquals('34FB476C-6FA0-43C8-8945-1BD7A4EBF0DE', SimulatedDeviceManager.INITIALIZED_DEVICES[0].udid)
     608        self.assertEquals('15A8401', SimulatedDeviceManager.INITIALIZED_DEVICES[0].build_version)
    608609        self.assertEquals(SimulatedDevice.DeviceState.BOOTED, SimulatedDeviceManager.INITIALIZED_DEVICES[0].platform_device.state())
    609610
Note: See TracChangeset for help on using the changeset viewer.