Changes between Version 5 and Version 6 of ImportingThirdPartyTests

May 23, 2012 2:25:47 PM (11 years ago)



  • ImportingThirdPartyTests

    v5 v6  
    99 1. General Import Process
    10   a. Import entire test suite(s) from W3C repository: pixel tests, ref tests, manual tests, JS tests
     10  a. Clone entire test suite(s) from W3C repository: pixel tests, ref tests, manual tests, JS tests to a local directory
    1111  a. Run suite locally in WebKit directory
    1212   i. Ref Tests
    1717     * Over time, individuals can clean up these tests by creating valid (passing) reference files and then removing the .txt file
    1818   i. DRT / Pixel tests
    19     * Expectations: create render tree dumps for each test and use that output as the new test expectation
     19    * Each port can follow its own convention regarding pixel tests
     20    * Expectations (in general): Use whatever.PNG is produced by the test as the new expectation
    2021     * Potential regressions will be identified by changes to this output
    21     * Proposal (open to discussion) - stop the production of .PNGs for these imported tests
    22      * PROS
    23       * Avoid the increase to the overall size of the repository caused by all the new PNGs
    24       * Regressions will likely be caught by the render tree dumps
    25       * Avoid maintenance of all associated PNGs
    26      * CONS
    27       * Regressions may be missed without the use of .PNGs
    28       * May make test results harder to interpret
    2922   i. JavaScript tests
    3023    * Pass - good, submit it (along with new expected.txt file - W3C does not use an expected.txt file for JS tests)
    31     * Fail - Add to test_expectations file to avoid failures
    32      * Over time, individual can clean up failing JS tests
     24    * Fail - generate expected.txt file from test results, use it as new test expectations
    3325   i. Manual tests
    34     * Submit in their current form
     26    * Include test files without making any changes
    3527     * Over time, convert to ref tests to be submitted back to W3C
    3931 1. How should W3C tests that fail in WebKit be handled?
    40   a. Failures should be checked in.  Details in General Import Process above.
     32  a. Failures should be checked in with new expectations such that they pass. 
     33  a. New expectation files should have a naming convention that indicates if they are perceived to be correct, incorrect, or unknown
    4134 1. Should a set frequency be used for importing tests?
    4235  a. No, frequency is up to the people who want to do this task.