Changes between Version 6 and Version 7 of ImportingThirdPartyTests


Ignore:
Timestamp:
May 23, 2012 2:44:29 PM (10 years ago)
Author:
jacobg@adobe.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • ImportingThirdPartyTests

    v6 v7  
    3535  a. No, frequency is up to the people who want to do this task. 
    3636 1. Can the approval process for previously reviewed W3C tests be streamlined?
    37   a. No, the process should be proscribed
     37  a. No, the approval process should not be dictated - it is ultimately up to the reviewer to decide
    3838  a. The intent is for the reviewer to confirm that the following type of actions were performed correctly: correct suites were imported, tests were run, updates made to test files, new output files generated, test_expectations updated, full test run after patch is clear of errors, etc.
    3939 1. Should other tests (from Mozilla, Microsoft, etc.) continue to be imported?
    4040  a. Still open to discussion. 
    41   a. One proposal: Yes, but only from W3C.  We should encourage vendors to contribute their tests to W3C, we will then only import from W3C
    42    * Under this scenario - would we wait until tests are approved (could be a long process) or import submitted tests as well? 
     41  a. Proposal 1: Yes, but only from W3C.  We should encourage vendors to contribute their tests to W3C, we will then only import from W3C
     42   * Under this scenario - would we wait until tests are approved (could be a long process) or import submitted tests as well?
     43      * We could import unapproved tests and put them under a directory structure that makes it clear that they have not yet been approved, and then update in the future
    4344   * If we import submitted tests, would they go into a special directory to indicate that they are not yet approved? 
    4445   * Import script would need to handle the case where a test goes from submitted to approved
    4546 1. Should W3C pixel tests be imported?
    46   a. Yes.  We should import entire test suites no matter what type of tests they contain
     47  a. Open for discussion
     48  a. Proposal 1: Yes.  We should import entire test suites no matter what type of tests they contain
     49  a. Proposal 2: Identify overlap and only import test suites that do not duplicate what already exists (or remove existing tests and import suite)
     50     * How could we automate the identification of overlapping tests?
    4751 1. How should we identify imported test suites?
    4852  a. Use a new directory structure
    49   a. Start putting imported tests into this structure, but ultimately move all existing tests into new hierarchy (i.e. there are some existing directories that could be collected under a new, top-level CSS directory - e.g. flexbox)
     53     * Keep baselines in a new directory separate from the tests and separate from the existing platform-specific directories.  Would be used as a fallback after platform specific directories
    5054  a. Structure will be based on topic (feature) at the top level, and will then identify imported tests by their source.  For example:
    5155{{{
     
    5660}}}
    5761 1. Should we create a single, centralized test_expectations file that covers all common failures? 
     62  a. Open to discussion - verdict still out on how many different test_expectations files to have
     63    * Proposal 1: One file per implementation (qt/gtk/efl/chromium/apple, etc.) but that multiple platform/os-specific variants should share the same file
    5864  a. Individual ports would still have their own test_expectations file for port-specific failures.  This would facilitate excluding failing imported tests (e.g. JavaScript tests)
    5965 1. Should scripts be used to help automate the import process?