Changes between Version 5 and Version 6 of ImportingThirdPartyTests

Timestamp:

May 23, 2012 2:25:47 PM (12 years ago)

Author:

jacobg@adobe.com

Comment:

--

ImportingThirdPartyTests

@@ -8 +8 @@
 
  1. General Import Process
-  a. Import entire test suite(s) from W3C repository: pixel tests, ref tests, manual tests, JS tests
+  a. Clone entire test suite(s) from W3C repository: pixel tests, ref tests, manual tests, JS tests to a local directory
   a. Run suite locally in WebKit directory
    i. Ref Tests
@@ -17 +17 @@
      * Over time, individuals can clean up these tests by creating valid (passing) reference files and then removing the .txt file
    i. DRT / Pixel tests
-    * Expectations: create render tree dumps for each test and use that output as the new test expectation 
+    * Each port can follow its own convention regarding pixel tests
+    * Expectations (in general): Use whatever.PNG is produced by the test as the new expectation 
      * Potential regressions will be identified by changes to this output
-    * Proposal (open to discussion) - stop the production of .PNGs for these imported tests
-     * PROS
-      * Avoid the increase to the overall size of the repository caused by all the new PNGs
-      * Regressions will likely be caught by the render tree dumps 
-      * Avoid maintenance of all associated PNGs
-     * CONS
-      * Regressions may be missed without the use of .PNGs
-      * May make test results harder to interpret
    i. JavaScript tests
     * Pass - good, submit it (along with new expected.txt file - W3C does not use an expected.txt file for JS tests)
-    * Fail - Add to test_expectations file to avoid failures
-     * Over time, individual can clean up failing JS tests
+    * Fail - generate expected.txt file from test results, use it as new test expectations
    i. Manual tests
-    * Submit in their current form 
+    * Include test files without making any changes 
      * Over time, convert to ref tests to be submitted back to W3C
 
@@ -38 +30 @@
 
  1. How should W3C tests that fail in WebKit be handled?
-  a. Failures should be checked in.  Details in General Import Process above.
+  a. Failures should be checked in with new expectations such that they pass.  
+  a. New expectation files should have a naming convention that indicates if they are perceived to be correct, incorrect, or unknown
  1. Should a set frequency be used for importing tests?
   a. No, frequency is up to the people who want to do this task.