Changes between Version 2 and Version 3 of WebKitW3CTesting


Ignore:
Timestamp:
Oct 23, 2015 1:36:55 AM (7 years ago)
Author:
youenn.fablet@crf.canon.fr
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • WebKitW3CTesting

    v2 v3  
    11= Scope =
    22
    3 Make as easy and as automated as possible import of W3C test suites (CSS and WPT).
    4 * See https://github.com/canonresearchfrance/webkit/wiki/Running-W3C-Tests for some more information, but somehow outdated
     3Easy import (and re-import) of available test suites on the web, in particular W3C CSS and WPT test-suites.
    54
    6 = Tools =
     5= Importing new tests =
    76
    8 * A tool to automatically check conformance of WebKit against W3C test suites
    9  * See https://bugs.webkit.org/show_bug.cgi?id=134766
    10 * A tool to help import or reimport W3C tests
    11  * See https://bugs.webkit.org/show_bug.cgi?id=134767
     71. Update LayoutTests/imported/w3c/resources/ImportExpectations to activate the import of selected tests
     82. Run "Tools/Scripts/import-w3c-tests" as is, or "Tools/Scripts/import-w3c-tests -t web-platform-tests/XXselected-testsXX" to have a faster import
     93. Run rwt to generate the new expectations on LayoutTests/imported/w3c/web-platform-tests/XXselected-testsXX
     104. Update if needed the list of imported files that should be marked as [ Skip ] in LayoutTests/TestExpectations
     11 * Tools/Scripts/import-w3c-tests will display in the console this to-be-skipped list of files
    1212
    13 = Documentation =
     13The import-w3c-tests is doing the following things:
     141. Clone the test repositories (as defined in LayoutTests/imported/w3c/resources/TestRepositories)
     152. Checkout a specified revision
     16 * It is important to keep the same revision for consistency
     173. Import tests
     18 * Only reference and JS tests should be imported at the moment
    1419
    15 To be written once tools ship in WebKit trunk
     20Manual edits of tests are highly discouraged
     21 * It might be hard to keep those edits when rebasing the tests
     22 * If required, document these clearly in the imported tests themselves
     23 * Upstream those changes to the original test repository if possible
     24
     25= Rebasing tests =
     26
     27The process consists in updating the revision and reimporting all tests.
     28It is not yet automated. A preliminary patch is available at https://bugs.webkit.org/show_bug.cgi?id=134767
     29
     301. Check whether WebKit testharness.js is up-to-date with the latest one used in the W3C repository
     31 * If not update WebKit testharness.js accordingly
     322. Update the repository revision in LayoutTests/imported/w3c/resources/TestRepositories
     333. Delete all html|xhtml|xhtm|js|svg files in the corresponding LayoutTests/imported/w3C sub-folder
     344. Run Tools/Scripts/import-w3c-tests
     355. Check for any manual edit that should be kept
     366. Regenerate expected files using rwt
     377. Remove all dangling expected.txt files and empty folders
     38 * Some bots have difficulties with removed/added empty files.
     39
     40= Laundry list =
     41
     42* Handle WPT test repository recursive submodules
     43 * Automated generation of LayoutTests/imported/w3c/resources/web-platform-tests-modules.json
     44* Generate WPT MANIFEST.json and use it to know whether a file is a test or a resource file
     45* Check whether a WPT resource file is already skipped in LayoutTests/TestExpectations
     46* Finalise rebasing test script
     47* Update import script to handle non-W3C test repositories
     48 * Blink tests
     49 * WHATWG streams test suite
     50* Move already imported test suites to LayoutTests/imported/w3c/web-platform-tests
     51 * LayoutTests/imported/w3c/canvas, LayoutTests/imported/w3c/html-templates, LayoutTests/imported/w3c/indexeddb
     52 * LayoutTests/imported/blink/imported/csswg-test/css-writing-modes-3, LayoutTests/imported/blink/imported/web-platform-tests