wiki:WebKitW3CTesting

Scope

Easy import (and re-import) of available test suites on the web, in particular W3C CSS and WPT test-suites.

Importing new tests

  1. Update LayoutTests/imported/w3c/resources/ImportExpectations to activate the import of selected tests
  2. Run "Tools/Scripts/import-w3c-tests" as is, or "Tools/Scripts/import-w3c-tests -t web-platform-tests/XXselected-testsXX" to have a faster import
  3. Run rwt to generate the new expectations on LayoutTests/imported/w3c/web-platform-tests/XXselected-testsXX
  4. Update if needed the list of imported files that should be marked as [ Skip ] in LayoutTests/TestExpectations
    • Tools/Scripts/import-w3c-tests will display in the console this to-be-skipped list of files

The import-w3c-tests is doing the following things:

  1. Clone the test repositories (as defined in LayoutTests/imported/w3c/resources/TestRepositories)
  2. Checkout a specified revision
    • It is important to keep the same revision for consistency
  3. Import tests
    • Only reference and JS tests should be imported at the moment

Manual edits of tests are highly discouraged

  • It might be hard to keep those edits when rebasing the tests
  • If required, document these clearly in the imported tests themselves
  • Upstream those changes to the original test repository if possible

Rebasing tests

The process consists in updating the revision and reimporting all tests. It is not yet automated. A preliminary patch is available at https://bugs.webkit.org/show_bug.cgi?id=134767

  1. Check whether WebKit testharness.js is up-to-date with the latest one used in the W3C repository
    • If not update WebKit testharness.js accordingly
  2. Update the repository revision in LayoutTests/imported/w3c/resources/TestRepositories
  3. Delete all html|xhtml|xhtm|js|svg files in the corresponding LayoutTests/imported/w3C sub-folder
  4. Run Tools/Scripts/import-w3c-tests
  5. Check for any manual edit that should be kept
  6. Regenerate expected files using rwt
  7. Remove all dangling expected.txt files and empty folders
    • Some bots have difficulties with removed/added empty files.

Laundry list

  • Handle WPT test repository recursive submodules
    • Automated generation of LayoutTests/imported/w3c/resources/web-platform-tests-modules.json
  • Generate WPT MANIFEST.json and use it to know whether a file is a test or a resource file
  • Check whether a WPT resource file is already skipped in LayoutTests/TestExpectations
  • Finalise rebasing test script
  • Update import script to handle non-W3C test repositories
    • Blink tests
    • WHATWG streams test suite
  • Move already imported test suites to LayoutTests/imported/w3c/web-platform-tests
    • LayoutTests/imported/w3c/canvas, LayoutTests/imported/w3c/html-templates, LayoutTests/imported/w3c/indexeddb
    • LayoutTests/imported/blink/imported/csswg-test/css-writing-modes-3, LayoutTests/imported/blink/imported/web-platform-tests
Last modified 4 years ago Last modified on Oct 23, 2015 1:36:55 AM