NOTE: THIS PAGE IS UNDER DEVELOPMENT. THE CONTENTS OF THIS PAGE HAVE NOT YET BEEN FINALIZED

Importing Third Party Tests

Proposed Process

1. General Import Process
   1. Import entire test suite(s) from W3C repository: pixel tests, ref tests, manual tests, JS tests
   2. Run suite locally in WebKit directory
      1. Ref Tests
         • Pass - good, submit it
         • Fail - create a testName-expected-failure.txt file
           • This file will be used as the new test expectations, resulting in a pass for this test going forward
           • Potential regressions will be identified by changes to this output
           • Over time, individuals can clean up these tests by creating valid (passing) reference files and then removing the .txt file
      2. DRT / Pixel tests
         • Expectations: create render tree dumps for each test and use that output as the new test expectation
           • Potential regressions will be identified by changes to this output
         • Proposal (open to discussion) - stop the production of .PNGs for these imported tests
           • PROS
             • Avoid the increase to the overall size of the repository caused by all the new PNGs
             • Regressions will likely be caught by the render tree dumps
             • Avoid maintenance of all associated PNGs
           • CONS
             • Regressions may be missed without the use of .PNGs
             • May make test results harder to interpret
      3. JavaScript tests
         • Pass - good, submit it (along with new expected.txt file - W3C does not use an expected.txt file for JS tests)
         • Fail - Add to test_expectations file to avoid failures
           • Over time, individual can clean up failing JS tests
      4. Manual tests
         • Submit in their current form.
           • Over time, convert to ref tests to be submitted back to W3C

Questions Discussed At Contributors Meeting

1. How should W3C tests that fail in WebKit be handled?
   1. Failures should be checked in. Details in General Import Process above.
2. Should a set frequency be used for importing tests?
   1. No, frequency is up to the people who want to do this task.
3. Can the approval process for previously reviewed W3C tests be streamlined?
   1. No, the process should be proscribed
   2. The intent is for the reviewer to confirm that the following type of actions were performed correctly: correct suites were imported, tests were run, updates made to test files, new output files generated, test_expectations updated, full test run after patch is clear of errors, etc.
4. Should other tests (from Mozilla, Microsoft, etc.) continue to be imported?
   1. Still open to discussion.
   2. One proposal: Yes, but only from W3C. We should encourage vendors to contribute their tests to W3C, we will then only import from W3C
      • Under this scenario - would we wait until tests are approved (could be a long process) or import submitted tests as well?
      • If we import submitted tests, would they go into a special directory to indicate that they are not yet approved?
      • Import script would need to handle the case where a test goes from submitted to approved
5. Should W3C pixel tests be imported?
   1. Yes. We should import entire test suites no matter what type of tests they contain
6. How should we identify imported test suites?
   1. Use a new directory structure
   2. Start putting imported tests into this structure, but ultimately move all existing tests into new hierarchy (i.e. there are some existing directories that could be collected under a new, top-level CSS directory - e.g. flexbox)
   3. Structure will be based on topic (feature) at the top level, and will then identify imported tests by their source. For example:

        - WebKit/LayoutTests/CSS - top level for all CSS related tests 
         -- WebKit/LayoutTests/CSS/imported-w3c/ - top level for all imported CSS related tests
      
         Sub-directories under imported-w3c will match existing sub directories from the imported test suite
      

7. Should we create a single, centralized test_expectations file that covers all common failures?
   1. Individual ports would still have their own test_expectations file for port-specific failures. This would facilitate excluding failing imported tests (e.g. JavaScript tests)
8. Should scripts be used to help automate the import process?
   1. Yes. The import script should handle the following cases:
      • Test added
      • Test removed
      • Test content changed
      • Test that was not ref could have references added
      • Ref test removed
      • Test moved from submitted to approved folder (if submitted tests will be accepted)