Changeset 232746 in webkit

Timestamp:

Jun 12, 2018 12:55:42 AM (6 years ago)

Author:

zandobersek@gmail.com

Message:

Add run-web-platform-tests script
​https://bugs.webkit.org/show_bug.cgi?id=183356

Reviewed by Carlos Alberto Lopez Perez.

.:

Store port-specific test expectations and test manifest files for the
web-platform-tests test runs under the top-level WebPlatformTests
directory.

TestExpectations.json file lists all the failing or disabled tests
or subtests that are to be expected in the test run. This information is
parsed in the run-web-platform-tests script and used to build
test-specific metadata files (ending with .ini) inside a temporary
directory that is then used during the test run. JSON format is used to
follow the WebDriver test suite in how it manages expectations, and to
avoid having to manually manage .ini files for every deviant test case.

TestManifest.ini file is used to fine-tune which tests are to be enabled
by default. For the GTK+ port, we currently disable all tests by default
but then specifically enable tests under the 2dcontext and WebCryptoAPI
directories. This will allow for gradual enabling of further tests.

• WebPlatformTests/gtk/TestExpectations.json: Added.
• WebPlatformTests/gtk/TestManifest.ini: Added.

Tools:

Add the run-web-platform-tests script that allows running WebKit builds
against the external web-platform-tests project.

The script sets up all the port-specific information, including the
display driver. If no web-platform-tests repository location is
specified, the TestDownloader utility is used to clone the repository,
and path to that clone is leveraged instead.

Port-specific TestManifest.ini and TestExpectations.json file paths are
then constructed, bailing if any of the files is missing. The JSON file
is then used to construct the metadata file tree, creating an .ini file
for each test that is disabled, has a certain expectation, or has
subtests that themselves have certain expectations. This metadata tree
is built under the build directory, and is purged and reconstructed for
each separate run.

Lastly, the WPT python module is imported from the web-platform-tests
source location, and wpt.main() is invoked along with all the necessary
arguments that set up a web-platform-tests suite run using the WebKit
product.

• Scripts/run-web-platform-tests: Added.
• Scripts/webkitpy/common/config/ports_mock.py:

(MockPort.run_bindings_tests_command):
(MockPort):
(MockPort.wpt_metadata_directory):
(MockPort.wpt_manifest_file):

• Scripts/webkitpy/port/base.py:

(Port.wpt_metadata_directory):
(Port):
(Port.wpt_manifest_file):

• Scripts/webkitpy/w3c/common.py:

(is_file_exportable):
(WPTPaths):
(WPTPaths.checkout_directory):
(WPTPaths.wpt_checkout_path):

• Scripts/webkitpy/w3c/test_downloader.py:

(TestDownloader.clone_tests):
(TestDownloader):
(TestDownloader.download_tests):

• Scripts/webkitpy/w3c/test_exporter.py:

(WebPlatformTestExporter.init):

• Scripts/webkitpy/w3c/wpt_runner.py: Added.

(main):
(parse_args):
(create_webdriver):
(spawn_wpt):
(WPTRunner):
(WPTRunner.init):
(WPTRunner._prepare_wpt_checkout):
(WPTRunner._generate_metadata_directory):
(WPTRunner.run):

• Scripts/webkitpy/w3c/wpt_runner_unittest.py: Added.

(WPTRunnerTest):
(WPTRunnerTest.MockTestDownloader):
(WPTRunnerTest.MockTestDownloader.default_options):
(WPTRunnerTest.MockTestDownloader.init):
(WPTRunnerTest.MockTestDownloader.clone_tests):
(WPTRunnerTest.MockWebDriver):
(WPTRunnerTest.MockWebDriver.create):
(WPTRunnerTest.MockWebDriver.binary_path):
(WPTRunnerTest.MockWebDriver.browser_path):
(WPTRunnerTest.MockWebDriver.browser_args):
(WPTRunnerTest.MockSpawnWPT):
(WPTRunnerTest.MockSpawnWPT.init):
(WPTRunnerTest.MockSpawnWPT.call):
(WPTRunnerTest.TestInstance):
(WPTRunnerTest.TestInstance.init):
(WPTRunnerTest.TestInstance.prepare_mock_files_for_run):
(WPTRunnerTest.test_prepare_wpt_checkout):
(WPTRunnerTest.test_prepare_wpt_checkout_specified_path):
(WPTRunnerTest.test_generate_metadata_directory):
(WPTRunnerTest.test_run):
(WPTRunnerTest.test_run_with_specified_options):
(WPTRunnerTest.test_run_with_args):

Location:

trunk

Files:

• ChangeLog (modified) (1 diff)
• Tools/ChangeLog (modified) (1 diff)
• Tools/Scripts/run-web-platform-tests (added)
• Tools/Scripts/webkitpy/common/config/ports_mock.py (modified) (1 diff)
• Tools/Scripts/webkitpy/port/base.py (modified) (1 diff)
• Tools/Scripts/webkitpy/w3c/common.py (modified) (1 diff)
• Tools/Scripts/webkitpy/w3c/test_downloader.py (modified) (1 diff)
• Tools/Scripts/webkitpy/w3c/test_exporter.py (modified) (2 diffs)
• Tools/Scripts/webkitpy/w3c/wpt_runner.py (added)
• Tools/Scripts/webkitpy/w3c/wpt_runner_unittest.py (added)
• WebPlatformTests (added)
• WebPlatformTests/gtk (added)
• WebPlatformTests/gtk/TestExpectations.json (added)
• WebPlatformTests/gtk/TestManifest.ini (added)

trunk/ChangeLog

@@ +1 @@
+2018-06-12  Zan Dobersek  <zdobersek@igalia.com>
+
+        Add run-web-platform-tests script
+        https://bugs.webkit.org/show_bug.cgi?id=183356
+
+        Reviewed by Carlos Alberto Lopez Perez.
+
+        Store port-specific test expectations and test manifest files for the
+        web-platform-tests test runs under the top-level WebPlatformTests
+        directory.
+
+        TestExpectations.json file lists all the failing or disabled tests
+        or subtests that are to be expected in the test run. This information is
+        parsed in the run-web-platform-tests script and used to build
+        test-specific metadata files (ending with .ini) inside a temporary
+        directory that is then used during the test run. JSON format is used to
+        follow the WebDriver test suite in how it manages expectations, and to
+        avoid having to manually manage .ini files for every deviant test case.
+
+        TestManifest.ini file is used to fine-tune which tests are to be enabled
+        by default. For the GTK+ port, we currently disable all tests by default
+        but then specifically enable tests under the 2dcontext and WebCryptoAPI
+        directories. This will allow for gradual enabling of further tests.
+
+        * WebPlatformTests/gtk/TestExpectations.json: Added.
+        * WebPlatformTests/gtk/TestManifest.ini: Added.
+
 2018-06-11  Carlos Garcia Campos  <cgarcia@igalia.com>
 

trunk/Tools/ChangeLog

@@ +1 @@
+2018-06-12  Zan Dobersek  <zdobersek@igalia.com>
+
+        Add run-web-platform-tests script
+        https://bugs.webkit.org/show_bug.cgi?id=183356
+
+        Reviewed by Carlos Alberto Lopez Perez.
+
+        Add the run-web-platform-tests script that allows running WebKit builds
+        against the external web-platform-tests project.
+
+        The script sets up all the port-specific information, including the
+        display driver. If no web-platform-tests repository location is
+        specified, the TestDownloader utility is used to clone the repository,
+        and path to that clone is leveraged instead.
+
+        Port-specific TestManifest.ini and TestExpectations.json file paths are
+        then constructed, bailing if any of the files is missing. The JSON file
+        is then used to construct the metadata file tree, creating an .ini file
+        for each test that is disabled, has a certain expectation, or has
+        subtests that themselves have certain expectations. This metadata tree
+        is built under the build directory, and is purged and reconstructed for
+        each separate run.
+
+        Lastly, the WPT python module is imported from the web-platform-tests
+        source location, and wpt.main() is invoked along with all the necessary
+        arguments that set up a web-platform-tests suite run using the WebKit
+        product.
+
+        * Scripts/run-web-platform-tests: Added.
+        * Scripts/webkitpy/common/config/ports_mock.py:
+        (MockPort.run_bindings_tests_command):
+        (MockPort):
+        (MockPort.wpt_metadata_directory):
+        (MockPort.wpt_manifest_file):
+        * Scripts/webkitpy/port/base.py:
+        (Port.wpt_metadata_directory):
+        (Port):
+        (Port.wpt_manifest_file):
+        * Scripts/webkitpy/w3c/common.py:
+        (is_file_exportable):
+        (WPTPaths):
+        (WPTPaths.checkout_directory):
+        (WPTPaths.wpt_checkout_path):
+        * Scripts/webkitpy/w3c/test_downloader.py:
+        (TestDownloader.clone_tests):
+        (TestDownloader):
+        (TestDownloader.download_tests):
+        * Scripts/webkitpy/w3c/test_exporter.py:
+        (WebPlatformTestExporter.__init__):
+        * Scripts/webkitpy/w3c/wpt_runner.py: Added.
+        (main):
+        (parse_args):
+        (create_webdriver):
+        (spawn_wpt):
+        (WPTRunner):
+        (WPTRunner.__init__):
+        (WPTRunner._prepare_wpt_checkout):
+        (WPTRunner._generate_metadata_directory):
+        (WPTRunner.run):
+        * Scripts/webkitpy/w3c/wpt_runner_unittest.py: Added.
+        (WPTRunnerTest):
+        (WPTRunnerTest.MockTestDownloader):
+        (WPTRunnerTest.MockTestDownloader.default_options):
+        (WPTRunnerTest.MockTestDownloader.__init__):
+        (WPTRunnerTest.MockTestDownloader.clone_tests):
+        (WPTRunnerTest.MockWebDriver):
+        (WPTRunnerTest.MockWebDriver.create):
+        (WPTRunnerTest.MockWebDriver.binary_path):
+        (WPTRunnerTest.MockWebDriver.browser_path):
+        (WPTRunnerTest.MockWebDriver.browser_args):
+        (WPTRunnerTest.MockSpawnWPT):
+        (WPTRunnerTest.MockSpawnWPT.__init__):
+        (WPTRunnerTest.MockSpawnWPT.__call__):
+        (WPTRunnerTest.TestInstance):
+        (WPTRunnerTest.TestInstance.__init__):
+        (WPTRunnerTest.TestInstance.prepare_mock_files_for_run):
+        (WPTRunnerTest.test_prepare_wpt_checkout):
+        (WPTRunnerTest.test_prepare_wpt_checkout_specified_path):
+        (WPTRunnerTest.test_generate_metadata_directory):
+        (WPTRunnerTest.test_run):
+        (WPTRunnerTest.test_run_with_specified_options):
+        (WPTRunnerTest.test_run_with_args):
+
 2018-06-11  Mark Lam  <mark.lam@apple.com>
 

trunk/Tools/Scripts/webkitpy/common/config/ports_mock.py

@@ -59 +59 @@
     def run_bindings_tests_command(self):
         return ['mock-run-bindings-tests']
+
+    def wpt_metadata_directory(self):
+        return '/mock-path/mock-wpt-tests-metadata'
+
+    def wpt_manifest_file(self):
+        return '/mock-path/mock-wpt-manifest.json'

trunk/Tools/Scripts/webkitpy/port/base.py

@@ -872 +872 @@
         return self._build_path('layout-test-results')
 
+    def wpt_metadata_directory(self):
+        return self._build_path('web-platform-tests-metadata')
+
+    def wpt_manifest_file(self):
+        return self._build_path('web-platform-tests-manifest.json')
+
     def setup_test_run(self, device_class=None):
         """Perform port-specific work at the beginning of a test run."""

trunk/Tools/Scripts/webkitpy/w3c/common.py

@@ -99 +99 @@
     basename = path[path.rfind('/') + 1:]
     return not is_basename_skipped(basename)
+
+
+class WPTPaths:
+    CHECKOUT_DIRECTORY = ["WebKitBuild", "w3c-tests"]
+    WPT_CHECKOUT_PATH = CHECKOUT_DIRECTORY + ["web-platform-tests"]
+
+    @staticmethod
+    def checkout_directory(finder):
+        return finder.path_from_webkit_base(*WPTPaths.CHECKOUT_DIRECTORY)
+
+    @staticmethod
+    def wpt_checkout_path(finder):
+        return finder.path_from_webkit_base(*WPTPaths.WPT_CHECKOUT_PATH)

trunk/Tools/Scripts/webkitpy/w3c/test_downloader.py

@@ -220 +220 @@
         self._filesystem.write_text_file(self._filesystem.join(destination_directory, test_repository['name'], '.gitignore'), '\n'.join(rules))
 
-    def download_tests(self, destination_directory, test_paths=[], use_tip_of_tree=False):
+    def clone_tests(self, use_tip_of_tree=False):
         for test_repository in self.test_repositories:
             self.checkout_test_repository(test_repository['revision'] if not use_tip_of_tree else 'origin/master', test_repository['url'], self._filesystem.join(self.repository_directory, test_repository['name']))
+
+    def download_tests(self, destination_directory, test_paths=[], use_tip_of_tree=False):
+        self.clone_tests(use_tip_of_tree)
         self.copy_tests(destination_directory, test_paths)

trunk/Tools/Scripts/webkitpy/w3c/test_exporter.py

@@ -38 +38 @@
 from webkitpy.w3c.wpt_github import WPTGitHub
 from webkitpy.w3c.wpt_linter import WPTLinter
-from webkitpy.w3c.common import WPT_GH_ORG, WPT_GH_REPO_NAME, WPT_GH_URL
+from webkitpy.w3c.common import WPT_GH_ORG, WPT_GH_REPO_NAME, WPT_GH_URL, WPTPaths
 from webkitpy.common.memoized import memoized
 
@@ -70 +70 @@
         if not self._options.repository_directory:
             webkit_finder = WebKitFinder(self._filesystem)
-            self._options.repository_directory = webkit_finder.path_from_webkit_base('WebKitBuild', 'w3c-tests', 'web-platform-tests')
+            self._options.repository_directory = WPTPaths.wpt_checkout_path(webkit_finder)
 
         self._linter = WPTLinterClass(self._options.repository_directory, host.filesystem)