| | 1 | = Importing Third Party Tests = |
| | 2 | |
| | 3 | == Proposed Process == |
| | 4 | |
| | 5 | 1. General Import Process |
| | 6 | a. Import entire test suite(s) from W3C repository: pixel tests, ref tests, manual tests, JS tests |
| | 7 | a. Run suite locally in WebKit directory |
| | 8 | i. RefTests |
| | 9 | * Pass - good, submit it |
| | 10 | * Fail - create a testName-expected-failure.txt file |
| | 11 | * This file will be used as the new test expectations, resulting in a pass for this test going forward |
| | 12 | * Potential regressions will be identified by changes to this output |
| | 13 | * Over time, individuals can clean up these tests by creating valid (passing) reference files and then removing the .txt file |
| | 14 | i. DRT / Pixel tests |
| | 15 | * Expectations: create render tree dumps for each test and use that output as the new test expectation |
| | 16 | * Potential regressions will be identified by changes to this output |
| | 17 | * Proposal (open to discussion) - stop the production of .PNGs for these imported tests |
| | 18 | * PROS |
| | 19 | * Avoid the increase to the overall size of the repository caused by all the new PNGs |
| | 20 | * Regressions will likely be caught by the render tree dumps |
| | 21 | * Avoid maintenance of all associated PNGs |
| | 22 | * CONS |
| | 23 | * Regressions may be missed without the use of .PNGs |
| | 24 | * May make test results harder to interpret |
| | 25 | i. JavaScript tests |
| | 26 | * Pass - good, submit it (along with new expected.txt file - W3C does not use an expected.txt file for JS tests) |
| | 27 | * Fail - Add to test_expectations file to avoid failures |
| | 28 | * Over time, individual can clean up failing JS tests |
| | 29 | i. Manual tests |
| | 30 | * Submit in their current form. |
| | 31 | * Over time, convert to ref tests to be submitted back to W3C |
| | 32 | |
| | 33 | == Questions Discussed At Contributors Meeting == |
| | 34 | |
| | 35 | 1. How should W3C tests that fail in WebKit be handled? |
| | 36 | a. Failures should be checked in. Details in General Import Process above. |
| | 37 | 1. Should a set frequency be used for importing tests? |
| | 38 | a. No, frequency is up to the people who want to do this task. |
| | 39 | 1. Can the approval process for previously reviewed W3C tests be streamlined? |
| | 40 | a. No, the process should be proscribed |
| | 41 | a. The intent is for the reviewer to confirm that the following type of actions were performed correctly: correct suites were imported, tests were run, updates made to test files, new output files generated, test_expectations updated, full test run after patch is clear of errors, etc. |
| | 42 | 1. Should other tests (from Mozilla, Microsoft, etc.) continue to be imported? |
| | 43 | a. Still open to discussion. |
| | 44 | a. One proposal: Yes, but only from W3C. We should encourage vendors to contribute their tests to W3C, we will then only import from W3C |
| | 45 | * Under this scenario - would we wait until tests are approved (could be a long process) or import submitted tests as well? |
| | 46 | * If we import submitted tests, would they go into a special directory to indicate that they are not yet approved? |
| | 47 | * Import script would need to handle the case where a test goes from submitted to approved |
| | 48 | 1. Should W3C pixel tests be imported? |
| | 49 | a. Yes. We should import entire test suites no matter what type of tests they contain |
| | 50 | 1. How should we identify imported test suites? |
| | 51 | a. Use a new directory structure |
| | 52 | a. Start putting imported tests into this structure, but ultimately move all existing tests into new hierarchy (i.e. there are some existing directories that could be collected under a new, top-level CSS directory - e.g. flexbox) |
| | 53 | a. Structure will be based on topic (feature) at the top level, and will then identify imported tests by their source. For example: |
| | 54 | {{{ |
| | 55 | - WebKit/LayoutTests/CSS - top level for all CSS related tests |
| | 56 | -- WebKit/LayoutTests/CSS/imported-w3c/ - top level for all imported CSS related tests |
| | 57 | |
| | 58 | Sub-directories under imported-w3c will match existing sub directories from the imported test suite |
| | 59 | }}} |
| | 60 | 1. Should we create a single, centralized test_expectations file that covers all common failures? |
| | 61 | a. Individual ports would still have their own test_expectations file for port-specific failures. This would facilitate excluding failing imported tests (e.g. JavaScript tests) |
| | 62 | 1. Import script should handle following cases |
| | 63 | a. Test added |
| | 64 | a. Test removed |
| | 65 | a. Test content changed |
| | 66 | a. Test that was not ref could have references added |
| | 67 | a. Ref test removed |
| | 68 | a. Test moved from submitted to approved folder (if submitted tests will be accepted) |
