Version 9 (modified by 12 years ago) ( diff ) | ,
---|
NOTE: THIS PAGE IS UNDER DEVELOPMENT. THE CONTENTS OF THIS PAGE HAVE NOT YET BEEN FINALIZED
Importing Third Party Tests
Proposed Process
- General Import Process
- Clone entire test suite(s) from W3C repository: pixel tests, ref tests, manual tests, JS tests to a local directory
- Run suite locally in WebKit directory
- Ref Tests
- Pass - good, submit it
- Fail - create a testName-expected-failure.txt file
- This file will be used as the new test expectations, resulting in a pass for this test going forward
- Potential regressions will be identified by changes to this output
- Over time, individuals can clean up these tests by creating valid (passing) reference files and then removing the .txt file
- DRT / Pixel tests
- Each port can follow its own convention regarding pixel tests
- Expectations (in general): Use whatever.PNG is produced by the test as the new expectation
- Potential regressions will be identified by changes to this output
- JavaScript tests
- Pass - good, submit it (along with new expected.txt file - W3C does not use an expected.txt file for JS tests)
- Fail - generate expected.txt file from test results, use it as new test expectations
- Manual tests
- Include test files without making any changes
- Over time, convert to ref tests to be submitted back to W3C
- Include test files without making any changes
- Ref Tests
Questions Discussed At Contributors Meeting
- How should W3C tests that fail in WebKit be handled?
- Failures should be checked in with new expectations such that they pass.
- New expectation files should have a naming convention that indicates if they are perceived to be correct, incorrect, or unknown
- Should a set frequency be used for importing tests?
- No, frequency is up to the people who want to do this task.
- Can the approval process for previously reviewed W3C tests be streamlined?
- No, the approval process should not be dictated - it is ultimately up to the reviewer to decide
- The intent is for the reviewer to confirm that the following type of actions were performed correctly: correct suites were imported, tests were run, updates made to test files, new output files generated, test_expectations updated, full test run after patch is clear of errors, etc.
- Should other tests (from Mozilla, Microsoft, etc.) continue to be imported?
- Still open to discussion.
- Proposal 1: Yes, but only from W3C. We should encourage vendors to contribute their tests to W3C, we will then only import from W3C
- Under this scenario - would we wait until tests are approved (could be a long process) or import submitted tests as well?
- We could import unapproved tests and put them under a directory structure that makes it clear that they have not yet been approved, and then update in the future
- If we import submitted tests, would they go into a special directory to indicate that they are not yet approved?
- Import script would need to handle the case where a test goes from submitted to approved
- Under this scenario - would we wait until tests are approved (could be a long process) or import submitted tests as well?
- Should W3C pixel tests be imported?
- Open for discussion
- Proposal 1: Yes. We should import entire test suites no matter what type of tests they contain
- Mitigate test duplication by:
- Upstreaming as many layout tests as possible to W3C
- Delete duplicate layout tests as W3C tests are imported
- Mitigate test duplication by:
- Proposal 2: Identify overlap and only import test suites that do not duplicate what already exists (or remove existing tests and import suite)
- How could we automate the identification of overlapping tests? May not be possible
- How should we identify imported test suites?
- Use a new directory structure
- Proposal 1: Keep baselines in a new directory separate from the tests and separate from the existing platform-specific directories. New directory would be used as a fallback after platform specific directories
- Structure will be based on topic (feature) at the top level, and will then identify imported tests by their source. For example:
- WebKit/LayoutTests/CSS - top level for all CSS related tests -- WebKit/LayoutTests/CSS/imported-w3c/ - top level for all imported CSS related tests Sub-directories under imported-w3c will match existing sub directories from the imported test suite
- Use a new directory structure
- Should we create a single, centralized test_expectations file that covers all common failures?
- Open to discussion - verdict still out on how many different test_expectations files to have
- Proposal 1: One file per implementation (qt/gtk/efl/chromium/apple, etc.) but that multiple platform/os-specific variants should share the same file
- Individual ports would still have their own test_expectations file for port-specific failures. This would facilitate excluding failing imported tests (e.g. JavaScript tests)
- Open to discussion - verdict still out on how many different test_expectations files to have
- Should scripts be used to help automate the import process?
- Yes. The import script should handle the following cases:
- Test added
- Test removed
- Test content changed
- Test that was not ref could have references added
- Ref test removed
- Test moved from submitted to approved folder (if submitted tests will be accepted)
- Yes. The import script should handle the following cases:
Note:
See TracWiki
for help on using the wiki.