⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 243732 in webkit


Ignore:
Timestamp:
Apr 1, 2019, 7:57:05 PM (7 years ago)
Author:
Jonathan Bedard
Message:

run-api-tests: Upload test results
​https://bugs.webkit.org/show_bug.cgi?id=196323
<rdar://problem/49356714>

Reviewed by Lucas Forschler.

  • Scripts/webkitpy/api_tests/manager.py:

(Manager):
(Manager.run): Upload results to a results database.

  • Scripts/webkitpy/api_tests/run_api_tests.py:

(parse_args): Add upload arguments.

  • Scripts/webkitpy/port/base.py:

(Port):
(Port.configuration_for_upload): Creates a configuration dictionary for uploading results.
(Port.commits_for_upload): Create a list of commits from the WebKit repository tests are run from along
with commits from any other associated repositories.

  • Scripts/webkitpy/port/device.py:

(Device):
(Device.build_version): Access build_versoin of underlying platform device.

  • Scripts/webkitpy/port/ios_simulator_unittest.py:

(IOSSimulatorTest):
(IOSSimulatorTest.test_configuration_for_upload):

  • Scripts/webkitpy/port/device_port.py:

(DevicePort):
(DevicePort.configuration_for_upload): Devices are unique because their configuration is not
the same as the machine uploading results.

  • Scripts/webkitpy/port/mac.py:

(MacPort):
(MacPort.configuration_for_upload): Define SDK in upload configuration for Mac.

  • Scripts/webkitpy/port/mac_unittest.py:

(MacTest):
(MacTest.test_configuration_for_upload):

  • Scripts/webkitpy/xcode/simulated_device.py:

(SimulatedDeviceManager._create_device_with_runtime):
(SimulatedDevice.init): Create simulated device with a build_version.

  • Scripts/webkitpy/xcode/simulated_device_unittest.py:

(test_existing_simulator):

Location:
trunk/Tools
Files:
11 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r243728 r243732  
     12019-04-01  Jonathan Bedard  <jbedard@apple.com>
     2
     3        run-api-tests: Upload test results
     4        https://bugs.webkit.org/show_bug.cgi?id=196323
     5        <rdar://problem/49356714>
     6
     7        Reviewed by Lucas Forschler.
     8
     9        * Scripts/webkitpy/api_tests/manager.py:
     10        (Manager):
     11        (Manager.run): Upload results to a results database.
     12        * Scripts/webkitpy/api_tests/run_api_tests.py:
     13        (parse_args): Add upload arguments.
     14        * Scripts/webkitpy/port/base.py:
     15        (Port):
     16        (Port.configuration_for_upload): Creates a configuration dictionary for uploading results.
     17        (Port.commits_for_upload): Create a list of commits from the WebKit repository tests are run from along
     18        with commits from any other associated repositories.
     19        * Scripts/webkitpy/port/device.py:
     20        (Device):
     21        (Device.build_version): Access build_versoin of underlying platform device.
     22        * Scripts/webkitpy/port/ios_simulator_unittest.py:
     23        (IOSSimulatorTest):
     24        (IOSSimulatorTest.test_configuration_for_upload):
     25        * Scripts/webkitpy/port/device_port.py:
     26        (DevicePort):
     27        (DevicePort.configuration_for_upload): Devices are unique because their configuration is not
     28        the same as the machine uploading results.
     29        * Scripts/webkitpy/port/mac.py:
     30        (MacPort):
     31        (MacPort.configuration_for_upload): Define SDK in upload configuration for Mac.
     32        * Scripts/webkitpy/port/mac_unittest.py:
     33        (MacTest):
     34        (MacTest.test_configuration_for_upload):
     35        * Scripts/webkitpy/xcode/simulated_device.py:
     36        (SimulatedDeviceManager._create_device_with_runtime):
     37        (SimulatedDevice.__init__): Create simulated device with a build_version.
     38        * Scripts/webkitpy/xcode/simulated_device_unittest.py:
     39        (test_existing_simulator):
     40
    1412019-04-01  Aakash Jain  <aakash_jain@apple.com>
    242

  • trunk/Tools/Scripts/webkitpy/api_tests/manager.py

    r240151 r243732  
    1 # Copyright (C) 2018 Apple Inc. All rights reserved.
     1# Copyright (C) 2018-2019 Apple Inc. All rights reserved.
    22#
    33# Redistribution and use in source and binary forms, with or without
    … …  
    2323import json
    2424import logging
    25 import os
     25import time
    2626
    2727from webkitpy.api_tests.runner import Runner
    2828from webkitpy.common.system.executive import ScriptError
     29from webkitpy.results.upload import Upload
     30
    2931from webkitpy.xcode.simulated_device import DeviceRequest, SimulatedDeviceManager
    3032
    … …  
    4042    FAILED_COLLECT_TESTS = 2
    4143    FAILED_TESTS = 3
     44    FAILED_UPLOAD = 4
    4245
    4346    def __init__(self, port, options, stream):
    … …  
    155158                raise RuntimeError('Cannot write to {}'.format(json_output))
    156159
     160        start_time = time.time()
     161
    157162        self._stream.write_update('Checking build ...')
    158163        if not self._port.check_api_test_build(self._binaries_for_arguments(args)):
    … …  
    186191            self._stream.writeln('')
    187192
     193        end_time = time.time()
     194
    188195        successful = runner.result_map_by_status(runner.STATUS_PASSED)
    189196        disabled = len(runner.result_map_by_status(runner.STATUS_DISABLED))
    … …  
    198205
    199206        self._stream.writeln('-' * 30)
     207        result = Manager.SUCCESS
    200208        if len(successful) + disabled == len(test_names):
    201209            self._stream.writeln('All tests successfully passed!')
    202210            if json_output:
    203211                self.host.filesystem.write_text_file(json_output, json.dumps(result_dictionary, indent=4))
    204             return Manager.SUCCESS
    205 
    206         self._stream.writeln('Test suite failed')
    207         self._stream.writeln('')
    208 
    209         skipped = []
    210         for test in test_names:
    211             if test not in runner.results:
    212                 skipped.append(test)
    213                 result_dictionary['Skipped'].append({'name': test, 'output': None})
    214         if skipped:
    215             self._stream.writeln('Skipped {} tests'.format(len(skipped)))
     212        else:
     213            self._stream.writeln('Test suite failed')
    216214            self._stream.writeln('')
    217             if self._options.verbose:
    218                 for test in skipped:
    219                     self._stream.writeln('    {}'.format(test))
    220 
    221         self._print_tests_result_with_status(runner.STATUS_FAILED, runner)
    222         self._print_tests_result_with_status(runner.STATUS_CRASHED, runner)
    223         self._print_tests_result_with_status(runner.STATUS_TIMEOUT, runner)
    224 
    225         for test, result in runner.results.iteritems():
    226             status_to_string = {
    227                 runner.STATUS_FAILED: 'Failed',
    228                 runner.STATUS_CRASHED: 'Crashed',
    229                 runner.STATUS_TIMEOUT: 'Timedout',
    230             }.get(result[0])
    231             if not status_to_string:
    232                 continue
    233             result_dictionary[status_to_string].append({'name': test, 'output': result[1]})
    234 
    235         if json_output:
    236             self.host.filesystem.write_text_file(json_output, json.dumps(result_dictionary, indent=4))
    237 
    238         return Manager.FAILED_TESTS
     215
     216            skipped = []
     217            for test in test_names:
     218                if test not in runner.results:
     219                    skipped.append(test)
     220                    result_dictionary['Skipped'].append({'name': test, 'output': None})
     221            if skipped:
     222                self._stream.writeln('Skipped {} tests'.format(len(skipped)))
     223                self._stream.writeln('')
     224                if self._options.verbose:
     225                    for test in skipped:
     226                        self._stream.writeln('    {}'.format(test))
     227
     228            self._print_tests_result_with_status(runner.STATUS_FAILED, runner)
     229            self._print_tests_result_with_status(runner.STATUS_CRASHED, runner)
     230            self._print_tests_result_with_status(runner.STATUS_TIMEOUT, runner)
     231
     232            for test, result in runner.results.iteritems():
     233                status_to_string = {
     234                    runner.STATUS_FAILED: 'Failed',
     235                    runner.STATUS_CRASHED: 'Crashed',
     236                    runner.STATUS_TIMEOUT: 'Timedout',
     237                }.get(result[0])
     238                if not status_to_string:
     239                    continue
     240                result_dictionary[status_to_string].append({'name': test, 'output': result[1]})
     241
     242            if json_output:
     243                self.host.filesystem.write_text_file(json_output, json.dumps(result_dictionary, indent=4))
     244
     245            result = Manager.FAILED_TESTS
     246
     247        if self._options.report_urls:
     248            self._stream.writeln('\n')
     249            self._stream.write_update('Preparing upload data ...')
     250
     251            status_to_test_result = {
     252                runner.STATUS_PASSED: None,
     253                runner.STATUS_FAILED: Upload.Expectations.FAIL,
     254                runner.STATUS_CRASHED: Upload.Expectations.CRASH,
     255                runner.STATUS_TIMEOUT: Upload.Expectations.TIMEOUT,
     256            }
     257            upload = Upload(
     258                suite='api-tests',
     259                configuration=self._port.configuration_for_upload(self._port.target_host(0)),
     260                details=Upload.create_details(options=self._options),
     261                commits=self._port.commits_for_upload(),
     262                run_stats=Upload.create_run_stats(
     263                    start_time=start_time,
     264                    end_time=end_time,
     265                    tests_skipped=len(result_dictionary['Skipped']),
     266                ),
     267                results={test: Upload.create_test_result(actual=status_to_test_result[result[0]])
     268                         for test, result in runner.results.iteritems() if result[0] in status_to_test_result},
     269            )
     270            for url in self._options.report_urls:
     271                self._stream.write_update('Uploading to {} ...'.format(url))
     272                if not upload.upload(url, log_line_func=self._stream.writeln):
     273                    result = Manager.FAILED_UPLOAD
     274            self._stream.writeln('Uploads completed!')
     275
     276        return result

  • trunk/Tools/Scripts/webkitpy/api_tests/run_api_tests.py

    r235252 r243732  
    3131from webkitpy.layout_tests.views.metered_stream import MeteredStream
    3232from webkitpy.port import configuration_options, platform_options, base, win
     33from webkitpy.results.options import upload_options
    3334
    3435EXCEPTIONAL_EXIT_STATUS = -1
    … …  
    135136                             help='Run all tests, even DISABLED tests'),
    136137    ]))
     138    option_group_definitions.append(('Upload Options', upload_options()))
    137139
    138140    option_parser = optparse.OptionParser(

  • trunk/Tools/Scripts/webkitpy/port/base.py

    r243559 r243732  
    4646from webkitpy.common import find_files
    4747from webkitpy.common import read_checksum_from_png
     48from webkitpy.common.checkout.scm.detection import SCMDetector
    4849from webkitpy.common.memoized import memoized
    4950from webkitpy.common.prettypatch import PrettyPatch
    … …  
    6263from webkitpy.layout_tests.servers import web_platform_test_server
    6364from webkitpy.layout_tests.servers import websocket_server
     65from webkitpy.results.upload import Upload
    6466
    6567_log = logging.getLogger(__name__)
    … …  
    16351637        # such as closing file descriptors that were implicitly cloned to the worker.
    16361638        pass
     1639
     1640    def configuration_for_upload(self, host=None):
     1641        configuration = self.test_configuration()
     1642        host = self.host or host
     1643
     1644        return Upload.create_configuration(
     1645            platform=host.platform.os_name,
     1646            version=str(host.platform.os_version),
     1647            version_name=host.platform.os_version_name(INTERNAL_TABLE) or host.platform.os_version_name(),
     1648            architecture=configuration.architecture,
     1649            style='guard-malloc' if self.get_option('guard_malloc') else configuration.build_type,
     1650            sdk=host.platform.build_version(),
     1651        )
     1652
     1653    @memoized
     1654    def commits_for_upload(self):
     1655        self.host.initialize_scm()
     1656
     1657        if port_config.apple_additions() and getattr(port_config.apple_additions(), 'repos', False):
     1658            repos = port_config.apple_additions().repos()
     1659        else:
     1660            repos = {}
     1661
     1662        up = os.path.dirname
     1663        repos['webkit'] = up(up(up(up(up(os.path.abspath(__file__))))))
     1664
     1665        commits = []
     1666        for repo_id, path in repos.iteritems():
     1667            scm = SCMDetector(self._filesystem, self._executive).detect_scm_system(path)
     1668            commits.append(Upload.create_commit(
     1669                repository_id=repo_id,
     1670                id=scm.native_revision(path),
     1671                branch=scm.native_branch(path),
     1672            ))
     1673        return commits

  • trunk/Tools/Scripts/webkitpy/port/device.py

    r239945 r243732  
    102102        return self.platform_device.device_type
    103103
     104    @property
     105    def build_version(self):
     106        return self.platform_device.build_version
     107
    104108    def __nonzero__(self):
    105109        return self.platform_device is not None

  • trunk/Tools/Scripts/webkitpy/port/device_port.py

    r241147 r243732  
    1 # Copyright (C) 2018 Apple Inc. All rights reserved.
     1# Copyright (C) 2018-2019 Apple Inc. All rights reserved.
    22#
    33# Redistribution and use in source and binary forms, with or without
    … …  
    2424import traceback
    2525
     26from webkitpy.common.version_name_map import VersionNameMap, PUBLIC_TABLE, INTERNAL_TABLE
    2627from webkitpy.layout_tests.models.test_configuration import TestConfiguration
    2728from webkitpy.port.darwin import DarwinPort
    2829from webkitpy.port.simulator_process import SimulatorProcess
     30from webkitpy.results.upload import Upload
    2931from webkitpy.xcode.device_type import DeviceType
    3032from webkitpy.xcode.simulated_device import DeviceRequest, SimulatedDeviceManager
    … …  
    239241    def device_version(self):
    240242        raise NotImplementedError
     243
     244    def configuration_for_upload(self, host=None):
     245        configuration = self.test_configuration()
     246
     247        device_type = host.device_type if host else self.DEVICE_TYPE
     248        model = device_type.hardware_family
     249        if model and device_type.hardware_type:
     250            model += ' {}'.format(device_type.hardware_type)
     251
     252        version = self.device_version()
     253        version_name = None
     254        for table in [INTERNAL_TABLE, PUBLIC_TABLE]:
     255            version_name = VersionNameMap.map(self.host.platform).to_name(version, platform=device_type.software_variant.lower(), table=table)
     256            if version_name:
     257                break
     258
     259        return Upload.create_configuration(
     260            platform=device_type.software_variant.lower(),
     261            is_simulator=self.DEVICE_MANAGER == SimulatedDeviceManager,
     262            version=str(version),
     263            version_name=version_name,
     264            architecture=configuration.architecture,
     265            style='guard-malloc' if self.get_option('guard_malloc') else configuration.build_type,
     266            model=model,
     267            sdk=host.build_version if host else None,
     268        )

  • trunk/Tools/Scripts/webkitpy/port/ios_simulator_unittest.py

    r239875 r243732  
    1 # Copyright (C) 2014-2016 Apple Inc. All rights reserved.
     1# Copyright (C) 2014-2019 Apple Inc. All rights reserved.
    22#
    33# Redistribution and use in source and binary forms, with or without
    … …  
    162162        port = self.make_port()
    163163        self.assertEqual(port.max_child_processes(DeviceType.from_string('Apple Watch')), 0)
     164
     165    def test_configuration_for_upload(self):
     166        port = self.make_port()
     167        self.assertEqual(
     168            dict(
     169                platform='ios',
     170                is_simulator=True,
     171                architecture='x86_64',
     172                version='11',
     173                version_name='iOS 11',
     174                style='release',
     175            ),
     176            port.configuration_for_upload(),
     177        )

  • trunk/Tools/Scripts/webkitpy/port/mac.py

    r239989 r243732  
    11# Copyright (C) 2011 Google Inc. All rights reserved.
    2 # Copyright (C) 2012, 2013, 2016 Apple Inc. All rights reserved.
     2# Copyright (C) 2012-2019 Apple Inc. All rights reserved.
    33#
    44# Redistribution and use in source and binary forms, with or without
    … …  
    280280        worthless_patterns.append((re.compile('.*<<< FFR_Common >>>.*\n'), ''))
    281281        return worthless_patterns
     282
     283    def configuration_for_upload(self, host=None):
     284        host = host or self.host
     285        configuration = super(MacPort, self).configuration_for_upload(host=host)
     286
     287        output = host.executive.run_command(['/usr/sbin/sysctl', 'hw.model']).rstrip()
     288        match = re.match(r'hw.model: (?P<model>.*)', output)
     289        if match:
     290            configuration['model'] = match.group('model')
     291
     292        return configuration

  • trunk/Tools/Scripts/webkitpy/port/mac_unittest.py

    r232483 r243732  
    11# Copyright (C) 2010 Google Inc. All rights reserved.
    2 # Copyright (C) 2014-2016 Apple Inc. All rights reserved.
     2# Copyright (C) 2014-2019 Apple Inc. All rights reserved.
    33#
    44# Redistribution and use in source and binary forms, with or without
    … …  
    225225        port = self.make_port(options=MockOptions(webkit_test_runner=False), port_name='mac-wk2')
    226226        self.assertEqual(port.driver_name(), 'WebKitTestRunner')
     227
     228    def test_configuration_for_upload(self):
     229        port = self.make_port()
     230        self.assertEqual(
     231            dict(
     232                platform='mac',
     233                is_simulator=False,
     234                architecture='x86_64',
     235                version='10.7',
     236                version_name='Lion',
     237                sdk='17A405',
     238                style='release',
     239            ),
     240            port.configuration_for_upload(),
     241        )

  • trunk/Tools/Scripts/webkitpy/xcode/simulated_device.py

    r240022 r243732  
    114114            host=host,
    115115            device_type=device_type,
     116            build_version=runtime.build_version,
    116117        ))
    117118        SimulatedDeviceManager.AVAILABLE_DEVICES.append(result)
    … …  
    489490    ]
    490491
    491     def __init__(self, name, udid, host, device_type):
     492    def __init__(self, name, udid, host, device_type, build_version):
    492493        assert device_type.software_version
    493494
    … …  
    495496        self.udid = udid
    496497        self.device_type = device_type
     498        self.build_version = build_version
    497499        self._state = SimulatedDevice.DeviceState.SHUTTING_DOWN
    498500        self._last_updated_state = time.time()

  • trunk/Tools/Scripts/webkitpy/xcode/simulated_device_unittest.py

    r239945 r243732  
    606606        self.assertEquals(1, len(SimulatedDeviceManager.INITIALIZED_DEVICES))
    607607        self.assertEquals('34FB476C-6FA0-43C8-8945-1BD7A4EBF0DE', SimulatedDeviceManager.INITIALIZED_DEVICES[0].udid)
     608        self.assertEquals('15A8401', SimulatedDeviceManager.INITIALIZED_DEVICES[0].build_version)
    608609        self.assertEquals(SimulatedDevice.DeviceState.BOOTED, SimulatedDeviceManager.INITIALIZED_DEVICES[0].platform_device.state())
    609610
Note: See TracChangeset for help on using the changeset viewer.