wiki:ImportingThirdPartyTests
Last modified 2 years ago Last modified on 05/23/12 14:48:49

NOTE: THIS PAGE IS UNDER DEVELOPMENT. THE CONTENTS OF THIS PAGE HAVE NOT YET BEEN FINALIZED

Importing Third Party Tests

Proposed Process

  1. General Import Process
    1. Clone entire test suite(s) from W3C repository: pixel tests, ref tests, manual tests, JS tests to a local directory
    2. Run suite locally in WebKit directory
      1. Ref Tests
        • Pass - good, submit it
        • Fail - create a testName-expected-failure.txt file
          • This file will be used as the new test expectations, resulting in a pass for this test going forward
          • Potential regressions will be identified by changes to this output
          • Over time, individuals can clean up these tests by creating valid (passing) reference files and then removing the .txt file
      2. DRT / Pixel tests
        • Each port can follow its own convention regarding pixel tests
        • Expectations (in general): Use whatever.PNG is produced by the test as the new expectation
          • Potential regressions will be identified by changes to this output
      3. JavaScript tests
        • Pass - good, submit it (along with new expected.txt file - W3C does not use an expected.txt file for JS tests)
        • Fail - generate expected.txt file from test results, use it as new test expectations
      4. Manual tests
        • Include test files without making any changes
          • Over time, convert to ref tests to be submitted back to W3C

Questions Discussed At Contributors Meeting

  1. How should W3C tests that fail in WebKit be handled?
    1. Failures should be checked in with new expectations such that they pass.
    2. New expectation files should have a naming convention that indicates if they are perceived to be correct, incorrect, or unknown
  2. Should a set frequency be used for importing tests?
    1. No, frequency is up to the people who want to do this task.
  3. Can the approval process for previously reviewed W3C tests be streamlined?
    1. No, the approval process should not be dictated - it is ultimately up to the reviewer to decide
    2. The intent is for the reviewer to confirm that the following type of actions were performed correctly: correct suites were imported, tests were run, updates made to test files, new output files generated, test_expectations updated, full test run after patch is clear of errors, etc.
  4. Should other tests (from Mozilla, Microsoft, etc.) continue to be imported?
    1. Still open to discussion.
    2. Proposal 1: Yes, but only from W3C. We should encourage vendors to contribute their tests to W3C, we will then only import from W3C
      • Under this scenario - would we wait until tests are approved (could be a long process) or import submitted tests as well?
        • We could import unapproved tests and put them under a directory structure that makes it clear that they have not yet been approved, and then update in the future
      • If we import submitted tests, would they go into a special directory to indicate that they are not yet approved?
      • Import script would need to handle the case where a test goes from submitted to approved
  5. Should W3C pixel tests be imported?
    1. Open for discussion
    2. Proposal 1: Yes. We should import entire test suites no matter what type of tests they contain
      • Mitigate test duplication by:
        • Upstreaming as many layout tests as possible to W3C
        • Delete duplicate layout tests as W3C tests are imported
    3. Proposal 2: Identify overlap and only import test suites that do not duplicate what already exists (or remove existing tests and import suite)
      • How could we automate the identification of overlapping tests? May not be possible
  6. How should we identify imported test suites?
    1. Use a new directory structure
      • Proposal 1: Keep baselines in a new directory separate from the tests and separate from the existing platform-specific directories. New directory would be used as a fallback after platform specific directories
    2. Structure will be based on topic (feature) at the top level, and will then identify imported tests by their source. For example:
        - WebKit/LayoutTests/CSS - top level for all CSS related tests 
         -- WebKit/LayoutTests/CSS/imported-w3c/ - top level for all imported CSS related tests
      
         Sub-directories under imported-w3c will match existing sub directories from the imported test suite
      
  7. Should we create a single, centralized test_expectations file that covers all common failures?
    1. Open to discussion - verdict still out on how many different test_expectations files to have
      • Proposal 1: One file per implementation (qt/gtk/efl/chromium/apple, etc.) but that multiple platform/os-specific variants should share the same file
    2. Individual ports would still have their own test_expectations file for port-specific failures. This would facilitate excluding failing imported tests (e.g. JavaScript tests)
  8. Should scripts be used to help automate the import process?
    1. Yes. The import script should handle the following cases:
      • Test added
      • Test removed
      • Test content changed
      • Test that was not ref could have references added
      • Ref test removed
      • Test moved from submitted to approved folder (if submitted tests will be accepted)