Changeset 226715 in webkit
- Timestamp:
- Jan 10, 2018 8:05:18 AM (6 years ago)
- Location:
- trunk/Tools
- Files:
-
- 8 edited
Legend:
- Unmodified
- Added
- Removed
-
trunk/Tools/ChangeLog
r226711 r226715 1 2018-01-10 Jonathan Bedard <jbedard@apple.com> 2 3 webkitpy: Refactor simulator code (Part 2) 4 https://bugs.webkit.org/show_bug.cgi?id=180555 5 <rdar://problem/36131381> 6 7 Reviewed by Aakash Jain. 8 9 The patch leverages the SimulatedDeviceManager when running layout tests. 10 This patch is primarily code removal, since much of the IOSSimulatorPort 11 was dedicated to managing simulators. The removed code is either now owned by the 12 SimulatedDeviceManager (and committed in Part 1) or supported a technique to 13 boot multiple simulators which was made obsolete in Xcode 9. 14 15 * Scripts/webkitpy/layout_tests/controllers/manager.py: 16 (Manager._custom_device_for_test.in): Device names now have spaces in them, 17 strip those spaces. 18 * Scripts/webkitpy/layout_tests/run_webkit_tests.py: 19 (parse_args): Remove '--runtime' flag, this is achieved with the --version flag. 20 * Scripts/webkitpy/port/ios.py: 21 (IOSPort.__init__): The _current_device idiom is no longer required since the 22 SimulatedDeviceManager keeps track of any previously booted devices. 23 (IOSPort.target_host): Even when only a single device is used, it will be accessed 24 through the array of managed devices. 25 (IOSPort.using_multiple_devices): Deleted. 26 * Scripts/webkitpy/port/ios_device.py: 27 (IOSDevicePort.using_multiple_devices): Deleted. 28 * Scripts/webkitpy/port/ios_simulator.py: 29 (IOSSimulatorPort): Remove constants required to manage simulators, change device 30 class strings so they can be parsed by the DeviceType class. 31 (IOSSimulatorPort.__init__): Determine the number of processes to use by checking 32 the number of simulators currently booted, the number of child processes specified 33 and the maximum number of simulators supported by this system. 34 (IOSSimulatorPort._device_for_worker_number_map):Return the array of initialized 35 devices owned by the SimulatedDeviceManager. 36 (IOSSimulatorPort.ios_version): Remove support for the --runtime option. 37 (IOSSimulatorPort.default_child_processes): Use the SimulatedDeviceManager to check 38 the maximum number of supported simulators on this system. 39 (IOSSimulatorPort._create_devices): Construct a list of device requests with a 40 request for each child process and send this list to the SimulatedDeviceManager to 41 initialize the devices. 42 (IOSSimulatorPort.clean_up_test_run): 43 (IOSSimulatorPort.check_sys_deps): Check that there are simulators running the 44 specified version of iOS. 45 (IOSSimulatorPort.reset_preferences): 46 (IOSSimulatorPort.simulator_runtime): Deleted. 47 (IOSSimulatorPort.simulator_device_type): Deleted. 48 (IOSSimulatorPort._teardown_managed_simulators): Deleted. 49 (IOSSimulatorPort.use_multiple_simulator_apps): Deleted. 50 (IOSSimulatorPort._create_simulators): Deleted. 51 (IOSSimulatorPort._quit_ios_simulator): Deleted. 52 (IOSSimulatorPort._using_dedicated_simulators): Deleted. 53 (IOSSimulatorPort.using_multiple_devices): Deleted. 54 (IOSSimulatorPort._create_device): Deleted. 55 (IOSSimulatorPort.get_simulator_path): Deleted. 56 (IOSSimulatorPort._createSimulatorApps): Deleted. 57 (IOSSimulatorPort._createSimulatorApp): Deleted. 58 * Scripts/webkitpy/tool/commands/rebaseline_unittest.py: 59 (TestRebaseline.test_rebaseline): Filter out commands run when determining the 60 maximum number of simulators run on this system. 61 (TestRebaselineExpectations.test_rebaseline_expectations): Ditto. 62 * Scripts/webkitpy/xcode/new_simulated_device.py: 63 (SimulatedDeviceManager._does_fulfill_request): Fixed log statement. 64 1 65 2018-01-10 Carlos Garcia Campos <cgarcia@igalia.com> 2 66 -
trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py
r225733 r226715 105 105 def _custom_device_for_test(self, test): 106 106 for device_class in self._port.CUSTOM_DEVICE_CLASSES: 107 directory_suffix = device_class + self._port.TEST_PATH_SEPARATOR107 directory_suffix = device_class.lower().replace(' ', '') + self._port.TEST_PATH_SEPARATOR 108 108 if directory_suffix in test: 109 109 return device_class -
trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py
r225733 r226715 297 297 help='Skip install step for device and simulator testing'), 298 298 optparse.make_option('--version', help='Specify the version of iOS to be used. By default, this will adopt the runtime for iOS Simulator.'), 299 optparse.make_option('--runtime', help='iOS Simulator runtime identifier (default: latest runtime)'),300 299 optparse.make_option('--device-type', help='iOS Simulator device type identifier (default: i386 -> iPhone 5, x86_64 -> iPhone 5s)'), 301 300 optparse.make_option('--dedicated-simulators', action="store_true", default=False, -
trunk/Tools/Scripts/webkitpy/port/ios.py
r225856 r226715 45 45 self._test_runner_process_constructor = SimulatorProcess 46 46 self._printing_cmd_line = False 47 self._current_device = None48 47 49 48 def _device_for_worker_number_map(self): … … 80 79 return int(self.get_option('child_processes')) 81 80 82 def using_multiple_devices(self):83 return False84 85 81 def _testing_device(self, number): 86 82 device = self._device_for_worker_number_map()[number] … … 93 89 if self._printing_cmd_line or worker_number is None: 94 90 return self.host 95 # When using simulated devices, this means webkitpy is managing the devices. 96 if self.using_multiple_devices(): 97 return self._testing_device(worker_number) 98 return self._current_device 91 return self._testing_device(worker_number) 99 92 100 93 @memoized -
trunk/Tools/Scripts/webkitpy/port/ios_device.py
r225856 r226715 47 47 return apple_additions().ios_device_default_child_processes(self) 48 48 return 1 49 50 def using_multiple_devices(self):51 return True52 49 53 50 def _device_for_worker_number_map(self): -
trunk/Tools/Scripts/webkitpy/port/ios_simulator.py
r225856 r226715 21 21 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 22 22 23 import atexit24 23 import logging 25 import os26 import re27 import shutil28 import subprocess29 import time30 24 31 25 from webkitpy.common.memoized import memoized 32 from webkitpy.common.system.executive import ScriptError33 26 from webkitpy.common.version import Version 34 from webkitpy.port.device import Device35 27 from webkitpy.port.ios import IOSPort 36 from webkitpy.xcode.simulator import Simulator, Runtime, DeviceType 28 from webkitpy.xcode.device_type import DeviceType 29 from webkitpy.xcode.new_simulated_device import DeviceRequest, SimulatedDeviceManager 37 30 38 31 … … 47 40 DEFAULT_ARCHITECTURE = 'x86_64' 48 41 49 DEFAULT_DEVICE_CLASS = 'i phone'50 CUSTOM_DEVICE_CLASSES = ['i pad', 'iphone7']42 DEFAULT_DEVICE_CLASS = 'iPhone 5s' 43 CUSTOM_DEVICE_CLASSES = ['iPad', 'iPhone 7'] 51 44 SDK = 'iphonesimulator' 52 53 SIMULATOR_BUNDLE_ID = 'com.apple.iphonesimulator'54 SIMULATOR_DIRECTORY = "/tmp/WebKitTestingSimulators/"55 LSREGISTER_PATH = "/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Versions/Current/Support/lsregister"56 PROCESS_COUNT_ESTIMATE_PER_SIMULATOR_INSTANCE = 12557 58 DEVICE_CLASS_MAP = {59 'x86_64': {60 'iphone': 'iPhone 5s',61 'iphone7': 'iPhone 7',62 'ipad': 'iPad Air',63 },64 'x86': {65 'iphone': 'iPhone 5',66 'ipad': 'iPad Retina',67 },68 }69 70 #FIXME: Ports are recreated in each process. This is a problem for IOSSimulatorPort, it means devices are not71 # persistent and devices hold a listening socket expected to be persistent across processes.72 _DEVICE_MAP = {}73 _CURRENT_DEVICE = None74 45 75 46 def __init__(self, host, port_name, **kwargs): … … 80 51 _log.debug('IOSSimulatorPort _device_class is %s', self._device_class) 81 52 82 if not IOSSimulatorPort._CURRENT_DEVICE: 83 try: 84 IOSSimulatorPort._CURRENT_DEVICE = Device(Simulator(host).current_device()) 85 except ScriptError: 86 # Failure to find a current device should not result in an exception being thrown 87 IOSSimulatorPort._CURRENT_DEVICE = Device(None) 88 self._current_device = IOSSimulatorPort._CURRENT_DEVICE 89 if not self._current_device: 90 self.set_option('dedicated_simulators', True) 91 if not self.get_option('dedicated_simulators'): 92 if self.get_option('child_processes') > 1: 93 _log.warn('Cannot have more than one child process when using a running simulator. Setting child_processes to 1.') 94 self.set_option('child_processes', 1) 53 if self.get_option('child_processes', self.default_child_processes()) > SimulatedDeviceManager.max_supported_simulators(self.host): 54 _log.warn('The specified number of Simulated devices to be used is greater than the number supported by this machine.') 95 55 96 56 def _device_for_worker_number_map(self): 97 return IOSSimulatorPort._DEVICE_MAP 98 99 @property 100 @memoized 101 def simulator_runtime(self): 102 runtime_identifier = self.get_option('runtime') 103 if runtime_identifier: 104 runtime = Runtime.from_identifier(runtime_identifier) 105 elif self.get_option('version'): 106 runtime = Runtime.from_version(Version.from_string(self.get_option('version'))) 107 else: 108 runtime = Runtime.from_version(self.host.platform.xcode_sdk_version('iphonesimulator')) 109 return runtime 57 return SimulatedDeviceManager.INITIALIZED_DEVICES 110 58 111 59 @staticmethod … … 117 65 @memoized 118 66 def ios_version(self): 119 runtime_identifier = self.get_option('runtime')120 67 if self.get_option('version'): 121 68 return Version.from_string(self.get_option('version')) 122 if runtime_identifier:123 return Runtime.from_identifier(runtime_identifier).version124 69 return IOSSimulatorPort._version_from_name(self._name) if IOSSimulatorPort._version_from_name(self._name) else self.host.platform.xcode_sdk_version('iphonesimulator') 125 126 def simulator_device_type(self):127 device_type_identifier = self.get_option('device_type')128 if device_type_identifier:129 _log.debug('simulator_device_type for device identifier %s', device_type_identifier)130 device_type = DeviceType.from_identifier(device_type_identifier)131 else:132 _log.debug('simulator_device_type for device %s', self._device_class)133 device_name = self.DEVICE_CLASS_MAP[self.architecture()][self._device_class]134 if not device_name:135 raise Exception('Failed to find device for architecture {} and device class {}'.format(self.architecture()), self._device_class)136 device_type = DeviceType.from_name(device_name)137 return device_type138 70 139 71 @memoized 140 72 def default_child_processes(self): 141 """Return the number of Simulators instances to use for this port.""" 142 best_child_process_count_for_cpu = self._executive.cpu_count() / 2 143 system_process_count_limit = int(subprocess.check_output(["ulimit", "-u"]).strip()) 144 current_process_count = len(subprocess.check_output(["ps", "aux"]).strip().split('\n')) 145 _log.debug('Process limit: %d, current #processes: %d' % (system_process_count_limit, current_process_count)) 146 maximum_simulator_count_on_this_system = (system_process_count_limit - current_process_count) // self.PROCESS_COUNT_ESTIMATE_PER_SIMULATOR_INSTANCE 147 # FIXME: We should also take into account the available RAM. 73 def booted_ios_devices_filter(device): 74 if not device.platform_device.is_booted_or_booting(): 75 return False 76 return device.platform_device.device_type in DeviceType(software_variant='iOS', 77 software_version=self.ios_version()) 148 78 149 if (maximum_simulator_count_on_this_system < best_child_process_count_for_cpu): 150 _log.warn("This machine could support %s simulators, but is only configured for %s." 151 % (best_child_process_count_for_cpu, maximum_simulator_count_on_this_system)) 152 _log.warn('Please see <https://trac.webkit.org/wiki/IncreasingKernelLimits>.') 153 154 if maximum_simulator_count_on_this_system == 0: 155 maximum_simulator_count_on_this_system = 1 156 157 return min(maximum_simulator_count_on_this_system, best_child_process_count_for_cpu) 79 if self.get_option('dedicated_simulators', False): 80 num_booted_sims = len(SimulatedDeviceManager.device_by_filter(booted_ios_devices_filter, host=self.host)) 81 if num_booted_sims: 82 return num_booted_sims 83 return SimulatedDeviceManager.max_supported_simulators(self.host) 158 84 159 85 def _build_driver_flags(self): … … 165 91 self._device_class = device_class if device_class else self.DEFAULT_DEVICE_CLASS 166 92 167 # This function may be called more than once.168 def _teardown_managed_simulators(self):169 if not self._using_dedicated_simulators():170 return171 self._quit_ios_simulator()172 173 for i in xrange(len(Simulator.managed_devices)):174 simulator_path = self.get_simulator_path(i)175 device_udid = Simulator.managed_devices[i].udid176 Simulator.remove_device(i)177 178 if not os.path.exists(simulator_path):179 continue180 try:181 self._executive.run_command([IOSSimulatorPort.LSREGISTER_PATH, "-u", simulator_path])182 183 _log.debug('rmtree %s', simulator_path)184 self._filesystem.rmtree(simulator_path)185 186 logs_path = self._filesystem.join(self._filesystem.expanduser("~"), "Library/Logs/CoreSimulator/", device_udid)187 _log.debug('rmtree %s', logs_path)188 self._filesystem.rmtree(logs_path)189 190 saved_state_path = self._filesystem.join(self._filesystem.expanduser("~"), "Library/Saved Application State/", IOSSimulatorPort.SIMULATOR_BUNDLE_ID + str(i) + ".savedState")191 _log.debug('rmtree %s', saved_state_path)192 self._filesystem.rmtree(saved_state_path)193 except:194 _log.warning('Unable to remove Simulator' + str(i))195 196 def use_multiple_simulator_apps(self):197 return int(self.host.platform.xcode_version().major) < 9198 199 def _create_simulators(self):200 if (self.default_child_processes() < self.child_processes()):201 _log.warn('You have specified very high value({0}) for --child-processes'.format(self.child_processes()))202 _log.warn('maximum child-processes which can be supported on this system are: {0}'.format(self.default_child_processes()))203 _log.warn('This is very likely to fail.')204 205 if self._using_dedicated_simulators():206 atexit.register(lambda: self._teardown_managed_simulators())207 208 if self.use_multiple_simulator_apps():209 self._createSimulatorApps()210 211 for i in xrange(self.child_processes()):212 self._create_device(i)213 214 for i in xrange(self.child_processes()):215 device_udid = Simulator.managed_devices[i].udid216 Simulator.wait_until_device_is_in_state(device_udid, Simulator.DeviceState.SHUTDOWN)217 Simulator.reset_device(device_udid)218 else:219 assert(self._current_device)220 if self._current_device.platform_device.name != self.simulator_device_type().name:221 _log.warn("Expected simulator of type '" + self.simulator_device_type().name + "' but found simulator of type '" + self._current_device.platform_device.name + "'")222 _log.warn('The next block of tests may fail due to device mis-match')223 224 93 def _create_devices(self, device_class): 225 mac_os_version = self.host.platform.os_version226 227 94 self._set_device_class(device_class) 95 device_type = DeviceType.from_string(self._device_class, self.ios_version()) 228 96 229 97 _log.debug('') 230 _log.debug(' setup_test_run for %s', self._device_class)98 _log.debug('creating devices for {}'.format(device_type)) 231 99 232 self._create_simulators() 233 234 if not self._using_dedicated_simulators(): 235 return 236 237 for i in xrange(self.child_processes()): 238 device_udid = Simulator.managed_devices[i].udid 239 _log.debug('testing device %s has udid %s', i, device_udid) 240 241 # FIXME: <rdar://problem/20916140> Switch to using CoreSimulator.framework for launching and quitting iOS Simulator 242 if self.use_multiple_simulator_apps(): 243 self._executive.run_command([ 244 'open', '-g', '-b', self.SIMULATOR_BUNDLE_ID + str(i), 245 '--args', '-CurrentDeviceUDID', device_udid]) 246 else: 247 self._executive.run_command(['xcrun', 'simctl', 'boot', device_udid]) 248 249 if mac_os_version < Version.from_name('Sierra'): 250 time.sleep(2.5) 251 252 if not self.use_multiple_simulator_apps(): 253 self._executive.run_command(['open', '-g', '-b', self.SIMULATOR_BUNDLE_ID], return_exit_code=True) 254 255 _log.info('Waiting for all iOS Simulators to finish booting.') 256 for i in xrange(self.child_processes()): 257 Simulator.wait_until_device_is_booted(Simulator.managed_devices[i].udid) 258 _log.info('All simulators have booted.') 259 260 IOSSimulatorPort._DEVICE_MAP = {} 261 for i in xrange(self.child_processes()): 262 IOSSimulatorPort._DEVICE_MAP[i] = Device(Simulator.managed_devices[i]) 263 264 def _quit_ios_simulator(self): 265 if not self._using_dedicated_simulators(): 266 return 267 _log.debug("_quit_ios_simulator killing all Simulator processes") 268 # FIXME: We should kill only the Simulators we started. 269 subprocess.call(["killall", "-9", "-m", "Simulator"]) 100 request = DeviceRequest( 101 device_type, 102 use_booted_simulator=not self.get_option('dedicated_simulators', False), 103 use_existing_simulator=False, 104 allow_incomplete_match=True, 105 ) 106 SimulatedDeviceManager.initialize_devices([request] * self.child_processes(), self.host) 270 107 271 108 def clean_up_test_run(self): … … 273 110 _log.debug("clean_up_test_run") 274 111 275 if not self._using_dedicated_simulators(): 276 return 277 278 self._teardown_managed_simulators() 279 IOSSimulatorPort._DEVICE_MAP = {} 112 SimulatedDeviceManager.tear_down(self.host) 280 113 281 114 def setup_environ_for_server(self, server_name=None): … … 298 131 299 132 def check_sys_deps(self, needs_http): 300 if not self.simulator_runtime.available: 301 _log.error('The iOS Simulator runtime with identifier "{0}" cannot be used because it is unavailable.'.format(self.simulator_runtime.identifier)) 302 return False 303 return super(IOSSimulatorPort, self).check_sys_deps(needs_http) 304 305 SUBPROCESS_CRASH_REGEX = re.compile('#CRASHED - (?P<subprocess_name>\S+) \(pid (?P<subprocess_pid>\d+)\)') 306 307 def _using_dedicated_simulators(self): 308 return self.get_option('dedicated_simulators') 309 310 def using_multiple_devices(self): 311 return self._using_dedicated_simulators() 312 313 def _create_device(self, number): 314 return Simulator.create_device(number, self.simulator_device_type(), self.simulator_runtime) 315 316 def get_simulator_path(self, suffix=""): 317 return os.path.join(self.SIMULATOR_DIRECTORY, "Simulator" + str(suffix) + ".app") 133 target_device_type = DeviceType(software_variant='iOS', software_version=self.ios_version()) 134 for device in SimulatedDeviceManager.available_devices(self.host): 135 if device.platform_device.device_type in target_device_type: 136 return super(IOSSimulatorPort, self).check_sys_deps(needs_http) 137 _log.error('No Simulated device matching "{}" defined in Xcode iOS SDK'.format(str(target_device_type))) 138 return False 318 139 319 140 def reset_preferences(self): 320 141 _log.debug("reset_preferences") 321 self._quit_ios_simulator() 322 # Maybe this should delete all devices that we've created? 142 SimulatedDeviceManager.tear_down(self.host) 323 143 324 144 def nm_command(self): … … 335 155 def stderr_patterns_to_strip(self): 336 156 return [] 337 338 def _createSimulatorApps(self):339 for i in xrange(self.child_processes()):340 self._createSimulatorApp(i)341 342 def _createSimulatorApp(self, suffix):343 destination = self.get_simulator_path(suffix)344 _log.info("Creating app:" + destination)345 if os.path.exists(destination):346 shutil.rmtree(destination, ignore_errors=True)347 simulator_app_path = self.developer_dir + "/Applications/Simulator.app"348 shutil.copytree(simulator_app_path, destination)349 350 # Update app's package-name inside plist and re-code-sign it351 plist_path = destination + "/Contents/Info.plist"352 command = "Set CFBundleIdentifier com.apple.iphonesimulator" + str(suffix)353 subprocess.check_output(["/usr/libexec/PlistBuddy", "-c", command, plist_path])354 subprocess.check_output(["install_name_tool", "-add_rpath", self.developer_dir + "/Library/PrivateFrameworks/", destination + "/Contents/MacOS/Simulator"])355 subprocess.check_output(["install_name_tool", "-add_rpath", self.developer_dir + "/../Frameworks/", destination + "/Contents/MacOS/Simulator"])356 subprocess.check_output(["codesign", "-fs", "-", destination])357 subprocess.check_output([self.LSREGISTER_PATH, "-f", destination]) -
trunk/Tools/Scripts/webkitpy/tool/commands/rebaseline_unittest.py
r219507 r226715 296 296 builders._exact_matches = old_exact_matches 297 297 298 calls = filter(lambda x: x[0] != 'perl', self.tool.executive.calls)298 calls = filter(lambda x: x[0] not in ['perl', '/usr/bin/xcrun', '/usr/bin/ulimit'], self.tool.executive.calls) 299 299 self.assertEqual(calls, 300 300 [[['echo', 'rebaseline-test-internal', '--suffixes', 'txt,png', '--builder', 'MOCK builder', '--test', 'mock/path/to/test.html', '--verbose']]]) … … 318 318 319 319 # FIXME: change this to use the test- ports. 320 calls = self.tool.executive.calls320 calls = filter(lambda x: x[0] not in ['perl', '/usr/bin/xcrun', '/usr/bin/ulimit'], self.tool.executive.calls) 321 321 self.assertEqual(len(calls), 1) 322 322 self.assertEqual(len(calls[0]), 22) -
trunk/Tools/Scripts/webkitpy/xcode/new_simulated_device.py
r226263 r226715 293 293 if request.device_type.software_variant == device.platform_device.device_type.software_variant: 294 294 _log.warn("The request for '{}' incomplete-matched {}".format(request.device_type, device)) 295 _log.warn("This may cause unexpected behavior in code that expected the device type {}".format(request ))295 _log.warn("This may cause unexpected behavior in code that expected the device type {}".format(request.device_type)) 296 296 return request 297 297 return None
Note: See TracChangeset
for help on using the changeset viewer.