Changeset 252443 in webkit

Timestamp:

Nov 13, 2019, 6:08:56 PM (7 years ago)

Author:

Jonathan Bedard

Message:

Python 3: Add support in webkitpy.layout_tests.controllers
​https://bugs.webkit.org/show_bug.cgi?id=204180

Reviewed by Stephanie Lewis.

• Scripts/test-webkitpy-python3: Add webkitpy.layout_tests.controllers.
• Scripts/webkitpy/common/message_pool.py:

(_MessagePool.init): Use Python 3 queue syntax.
(_MessagePool._can_pickle): Use Python 3 pickle syntax.
(_MessagePool._loop): Use Python 3 queue syntax.
(_Worker.run): Use Python 3 queue syntax.
(_Worker._raise): Python 2 and Python 3 have different semantics for raising an exception
With a stack trace. However, 'raise' will raise the exception we are in the process of capturing,
Which is exactly what we want in this case.

• Scripts/webkitpy/common/read_checksum_from_png.py:

(read_checksum): Standardize checksum as a string.

• Scripts/webkitpy/common/system/filesystem.py:

(FileSystem.write_binary_file): Binary files should be written with bytes, not strings.

• Scripts/webkitpy/common/system/filesystem_mock.py:

(MockFileSystem.write_binary_file): Binary files should be written with bytes, not strings.

• Scripts/webkitpy/layout_tests/controllers/layout_test_finder_unittest.py: assertItemsEqual is not

Defined in Python 3.

• Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py:

(Worker._do_post_tests_work): Use compatible iteritems.
(Sharder._shard_by_directory): Ditto.

• Scripts/webkitpy/layout_tests/controllers/manager.py:

(Manager.run): Use compatible itervalues.
(Manager._look_for_new_crash_logs): Use Python 3 item iteration.
(Manager._results_to_upload_json_trie): Use compatible itervalues.
(Manager._stats_trie): Use compatible iteritems.

• Scripts/webkitpy/port/base.py:

(Port.expected_text): Be explicit about decoding text expectations.

• Scripts/webkitpy/port/mock_drt.py:

(MockDRT.write_test_output):

Location:

trunk/Tools

Files:

• ChangeLog (modified) (1 diff)
• Scripts/test-webkitpy-python3 (modified) (1 diff)
• Scripts/webkitpy/common/message_pool.py (modified) (7 diffs)
• Scripts/webkitpy/common/read_checksum_from_png.py (modified) (2 diffs)
• Scripts/webkitpy/common/system/filesystem.py (modified) (2 diffs)
• Scripts/webkitpy/common/system/filesystem_mock.py (modified) (1 diff)
• Scripts/webkitpy/layout_tests/controllers/layout_test_finder_unittest.py (modified) (5 diffs)
• Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py (modified) (3 diffs)
• Scripts/webkitpy/layout_tests/controllers/manager.py (modified) (7 diffs)
• Scripts/webkitpy/port/base.py (modified) (1 diff)
• Scripts/webkitpy/port/mock_drt.py (modified) (2 diffs)

trunk/Tools/ChangeLog

@@ +1 @@
+2019-11-13  Jonathan Bedard  <jbedard@apple.com>
+
+        Python 3: Add support in webkitpy.layout_tests.controllers
+        https://bugs.webkit.org/show_bug.cgi?id=204180
+
+        Reviewed by Stephanie Lewis.
+
+        * Scripts/test-webkitpy-python3: Add webkitpy.layout_tests.controllers.
+        * Scripts/webkitpy/common/message_pool.py:
+        (_MessagePool.__init__): Use Python 3 queue syntax.
+        (_MessagePool._can_pickle): Use Python 3 pickle syntax.
+        (_MessagePool._loop): Use Python 3 queue syntax.
+        (_Worker.run): Use Python 3 queue syntax.
+        (_Worker._raise): Python 2 and Python 3 have different semantics for raising an exception
+        With a stack trace. However, 'raise' will raise the exception we are in the process of capturing,
+        Which is exactly what we want in this case.
+        * Scripts/webkitpy/common/read_checksum_from_png.py:
+        (read_checksum): Standardize checksum as a string.
+        * Scripts/webkitpy/common/system/filesystem.py:
+        (FileSystem.write_binary_file): Binary files should be written with bytes, not strings.
+        * Scripts/webkitpy/common/system/filesystem_mock.py:
+        (MockFileSystem.write_binary_file): Binary files should be written with bytes, not strings.
+        * Scripts/webkitpy/layout_tests/controllers/layout_test_finder_unittest.py: assertItemsEqual is not
+        Defined in Python 3.
+        * Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py:
+        (Worker._do_post_tests_work): Use compatible iteritems.
+        (Sharder._shard_by_directory): Ditto.
+        * Scripts/webkitpy/layout_tests/controllers/manager.py:
+        (Manager.run): Use compatible itervalues.
+        (Manager._look_for_new_crash_logs): Use Python 3 item iteration.
+        (Manager._results_to_upload_json_trie): Use compatible itervalues.
+        (Manager._stats_trie): Use compatible iteritems.
+        * Scripts/webkitpy/port/base.py:
+        (Port.expected_text): Be explicit about decoding text expectations.
+        * Scripts/webkitpy/port/mock_drt.py:
+        (MockDRT.write_test_output):
+
 2019-11-13  Per Arne Vollan  <pvollan@apple.com>
 

trunk/Tools/Scripts/test-webkitpy-python3

@@ -36 +36 @@
 PYTHON3_COMPATIBLE_DIRECTORIES = [
   'webkitpy.common',
+  'webkitpy.layout_tests.controllers',
   'webkitpy.layout_tests.models',
   'webkitpy.port',

trunk/Tools/Scripts/webkitpy/common/message_pool.py

@@ -41 +41 @@
 """
 
-import cPickle
 import logging
 import multiprocessing
 import os
-import Queue
 import signal
 import sys
@@ -51 +49 @@
 import traceback
 
+if sys.version_info > (3, 0):
+    import pickle
+    import queue
+else:
+    import cPickle as pickle
+    import Queue as queue
 
 from webkitpy.common.host import Host
@@ -77 +81 @@
         self._timeout = timeout
         if self._running_inline:
-            self._messages_to_worker = Queue.Queue()
-            self._messages_to_manager = Queue.Queue()
+            self._messages_to_worker = queue.Queue()
+            self._messages_to_manager = queue.Queue()
         else:
             self._messages_to_worker = multiprocessing.Queue()
@@ -176 +180 @@
     def _can_pickle(self, host):
         try:
-            cPickle.dumps(host)
+            pickle.dumps(host)
             return True
         except TypeError:
@@ -194 +198 @@
                 assert method, 'bad message %s' % repr(message)
                 method(message.src, *message.args)
-        except Queue.Empty:
+        except queue.Empty:
             pass
 
@@ -273 +277 @@
 
             _log.debug("%s exiting" % self.name)
-        except Queue.Empty:
+        except queue.Empty:
             assert False, '%s: ran out of messages in worker queue.' % self.name
         except KeyboardInterrupt as e:
@@ -303 +307 @@
         exception_type, exception_value, exception_traceback = exc_info
         if self._running_inline:
-            raise exception_type, exception_value, exception_traceback
+            raise
 
         if exception_type == KeyboardInterrupt:

trunk/Tools/Scripts/webkitpy/common/read_checksum_from_png.py

@@ -28 +28 @@
 
 
+from webkitpy.common.unicode_compatibility import encode_if_necessary, decode_for
+
 def read_checksum(filehandle):
     # We expect the comment to be at the beginning of the file.
-    data = filehandle.read(2048)
-    comment_key = 'tEXtchecksum\x00'
+    data = encode_if_necessary(filehandle.read(2048))
+    comment_key = b'tEXtchecksum\x00'
     comment_pos = data.find(comment_key)
     if comment_pos == -1:
@@ -37 +39 @@
 
     checksum_pos = comment_pos + len(comment_key)
-    return data[checksum_pos:checksum_pos + 32]
+    return decode_for(data[checksum_pos:checksum_pos + 32], str)

trunk/Tools/Scripts/webkitpy/common/system/filesystem.py

@@ -40 +40 @@
 import time
 
-from webkitpy.common.unicode_compatibility import decode_if_necessary
+from webkitpy.common.unicode_compatibility import decode_if_necessary, encode_for
 
 
@@ -228 +228 @@
     def write_binary_file(self, path, contents):
         with open(path, 'wb') as f:
-            f.write(contents)
+            f.write(encode_for(contents, bytes))
 
     def open_text_file_for_reading(self, path, errors='strict'):

trunk/Tools/Scripts/webkitpy/common/system/filesystem_mock.py

@@ -330 +330 @@
         # FIXME: should this assert if dirname(path) doesn't exist?
         self.maybe_make_directory(self.dirname(path))
-        self.files[path] = contents
-        self.written_files[path] = contents
+        self.files[path] = unicode_compatibility.encode_for(contents, bytes)
+        self.written_files[path] = unicode_compatibility.encode_for(contents, bytes)
 
     def open_text_file_for_reading(self, path, errors='strict'):

trunk/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_finder_unittest.py

@@ -56 +56 @@
         paths = ['LayoutTests/test.html', 'LayoutTests/test', 'test2.html', 'Source/test1.html']
         fs, touched_tests = self.touched_files(paths)
-        self.assertItemsEqual(touched_tests, ['test.html'])
+        self.assertEqual(touched_tests, ['test.html'])
 
     def test_expected_touched_test(self):
@@ -63 +63 @@
         fs.write_text_file('/test.checkout/LayoutTests/test.html', 'This is a test')
         fs, touched_tests = self.touched_files(paths, fs)
-        self.assertItemsEqual(touched_tests, ['test.html'])
+        self.assertEqual(touched_tests, ['test.html'])
 
     def test_platform_expected_touched_test(self):
@@ -70 +70 @@
         fs.write_text_file('/test.checkout/LayoutTests/test.html', 'This is a test')
         fs, touched_tests = self.touched_files(paths, fs)
-        self.assertItemsEqual(touched_tests, ['test.html'])
+        self.assertEqual(touched_tests, ['test.html'])
 
     def test_platform_duplicate_touched_test(self):
@@ -77 +77 @@
         fs.write_text_file('/test.checkout/LayoutTests/test2.html', 'This is a test')
         fs, touched_tests = self.touched_files(paths, fs)
-        self.assertItemsEqual(touched_tests, ['test1.html', 'test2.html'])
+        self.assertEqual(sorted(touched_tests), sorted(['test1.html', 'test2.html']))
 
     def test_touched_but_skipped_test(self):
@@ -93 +93 @@
 
         touched_tests = LayoutTestFinder(port, optparse.Values({'skipped': 'always', 'skip_failing_tests': False, 'http': True})).find_touched_tests(paths)
-        self.assertItemsEqual(touched_tests, ['test0.html', 'test2.html'])
+        self.assertEqual(sorted(touched_tests), sorted(['test0.html', 'test2.html']))

trunk/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py

@@ -34 +34 @@
 
 from webkitpy.common import message_pool
+from webkitpy.common.iteration_compatibility import iteritems
 from webkitpy.layout_tests.controllers import single_test_runner
 from webkitpy.layout_tests.models.test_run_results import TestRunResults
@@ -332 +333 @@
         post_test_output = driver.do_post_tests_work()
         if post_test_output:
-            for test_name, doc_list in post_test_output.world_leaks_dict.iteritems():
+            for test_name, doc_list in iteritems(post_test_output.world_leaks_dict):
                 additional_results.append(test_results.TestResult(test_name, [test_failures.FailureDocumentLeak(doc_list)]))
         return additional_results
@@ -537 +538 @@
             tests_by_dir[directory].append(test_input)
 
-        for directory, test_inputs in tests_by_dir.iteritems():
+        for directory, test_inputs in iteritems(tests_by_dir):
             shard = TestShard(directory, test_inputs)
             shards.append(shard)

trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py

@@ -45 +45 @@
 from webkitpy.common.checkout.scm.detection import SCMDetector
 from webkitpy.common.net.file_uploader import FileUploader
+from webkitpy.common.iteration_compatibility import iteritems, itervalues
 from webkitpy.layout_tests.controllers.layout_test_finder import LayoutTestFinder
 from webkitpy.layout_tests.controllers.layout_test_runner import LayoutTestRunner
@@ -219 +220 @@
 
         # Check to make sure we're not skipping every test.
-        if not sum([len(tests) for tests in tests_to_run_by_device.itervalues()]):
+        if not sum([len(tests) for tests in itervalues(tests_to_run_by_device)]):
             _log.critical('No tests to run.')
             return test_run_results.RunDetails(exit_code=-1)
 
-        needs_http = any((self._is_http_test(test) and not self._needs_web_platform_test(test)) for tests in tests_to_run_by_device.itervalues() for test in tests)
-        needs_web_platform_test_server = any(self._needs_web_platform_test(test) for tests in tests_to_run_by_device.itervalues() for test in tests)
-        needs_websockets = any(self._is_websocket_test(test) for tests in tests_to_run_by_device.itervalues() for test in tests)
+        needs_http = any((self._is_http_test(test) and not self._needs_web_platform_test(test)) for tests in itervalues(tests_to_run_by_device) for test in tests)
+        needs_web_platform_test_server = any(self._needs_web_platform_test(test) for tests in itervalues(tests_to_run_by_device) for test in tests)
+        needs_websockets = any(self._is_websocket_test(test) for tests in itervalues(tests_to_run_by_device) for test in tests)
         self._runner = LayoutTestRunner(self._options, self._port, self._printer, self._results_directory, self._test_is_slow,
                                         needs_http=needs_http, needs_web_platform_test_server=needs_web_platform_test_server, needs_websockets=needs_websockets)
@@ -428 +429 @@
         """
         crashed_processes = []
-        for test, result in run_results.unexpected_results_by_name.iteritems():
+        for test, result in run_results.unexpected_results_by_name.items():
             if (result.type != test_expectations.CRASH):
                 continue
@@ -438 +439 @@
         sample_files = self._port.look_for_new_samples(crashed_processes, start_time)
         if sample_files:
-            for test, sample_file in sample_files.iteritems():
+            for test, sample_file in sample_files.items():
                 writer = TestResultWriter(self._port._filesystem, self._port, self._port.results_directory(), test)
                 writer.copy_sample_file(sample_file)
@@ -444 +445 @@
         crash_logs = self._port.look_for_new_crash_logs(crashed_processes, start_time)
         if crash_logs:
-            for test, crash_log in crash_logs.iteritems():
+            for test, crash_log in crash_logs.items():
                 writer = TestResultWriter(self._port._filesystem, self._port, self._port.results_directory(), test)
                 writer.write_crash_log(crash_log)
@@ -492 +493 @@
 
         results_trie = {}
-        for result in results.results_by_name.itervalues():
+        for result in itervalues(results.results_by_name):
             if result.type == test_expectations.SKIP:
                 continue
@@ -646 +647 @@
                 stats[result.test_name] = {'results': (_worker_number(result.worker_name), result.test_number, result.pid, int(result.test_run_time * 1000), int(result.total_run_time * 1000))}
         stats_trie = {}
-        for name, value in stats.iteritems():
+        for name, value in iteritems(stats):
             json_results_generator.add_path_to_trie(name, value, stats_trie)
         return stats_trie

trunk/Tools/Scripts/webkitpy/port/base.py

@@ -501 +501 @@
             if not self._filesystem.exists(baseline_path):
                 return None
-        text = self._filesystem.read_binary_file(baseline_path)
+        text = decode_for(self._filesystem.read_binary_file(baseline_path), str)
         return text.replace("\r\n", "\n")
 

trunk/Tools/Scripts/webkitpy/port/mock_drt.py

@@ -49 +49 @@
     sys.path.append(script_dir)
 
+from webkitpy.common.unicode_compatibility import decode_for
 from webkitpy.common.system.systemhost import SystemHost
 from webkitpy.port.driver import DriverInput, DriverOutput, DriverProxy
@@ -231 +232 @@
                 self._stdout.write('Content-Type: image/png\n')
                 self._stdout.write('Content-Length: %s\n' % len(output.image))
-                self._stdout.write(output.image)
+                self._stdout.write(decode_for(output.image, str))
         self._stdout.write('#EOF\n')
         self._stdout.flush()