Changes between Initial Version and Version 1 of CreatingLayoutTests

Aug 7, 2009 1:19:43 PM (12 years ago)

Initial Version


  • CreatingLayoutTests

    v1 v1  
     1Note:  This page is a work in progress.
     3= Creating a Layout Test =
     5== Before creating a test ==
     7Layout Tests should be added any time new functionality is added to the Rendering system, or any time a bug is found that is not covered by the existing tests.
     9Before Creating a new test, always check the platform/Skipped tests to make sure it doesn't already exist.  This can be done easily by running {{{run-webkit-tests --skipped=only}}} after compiling with the updated code.
     11Any change that breaks currently working tests {{{run-webkit-tests}}} should be considered carefully before proceeding.
     13For more information see the documentation on [ running tests], [ writing new tests], and [wiki:"Writing Layout Tests for DumpRenderTree"]
     16== Test with verifiable text results ==
     18=== Guidelines ===
     20 * Keep the test as simple as possible and try to only test what you mean to test.
     22=== Create the Test ===
     24 * Create the test as a simple HTML page that showcases the bug / new behavior.
     25 * Test should be placed in the appropriate subdirectory under LayoutTests/  use exsiting tests as a guideline or ask on #webkit.
     27=== Generate Results ===
     29 * Automatically generate results for your by running {{{run-webkit-tests --new-test-results}}}.
     30 * Verify the created results match the desired output.
     31 * If your test is platform independent (results are the same for all platforms)
     33 * If your test applies to Mac and Windows or other platform
     34   * Add the primary results file to the main directory beside the test (all results should be testname-expected.txt)
     35 * If your test has platform dependent results
     36   * Add the results set for the platform you are working on in the corresponding directory the test under the platform/type/ directory.
     38=== Verify Build ===
     40Regardless of whether you are making a platform independent change or dependent change, it is your responsibility to ensure that the change does not negatively affect any other ports / platforms.
     42Once your change is landed, there are some steps required to verify the change.  Note: These steps apply for all changes.  If you are making a platform dependent change, should should expect that additional results will be required for each platform but the actions are the same.
     44 * Arrange to have the patch landed
     45 * Observe the builds with your change at
     46    * Any failures will be clearly marked at the top, builds may fail independent of your changes, scrolling down can verify the state of the previously build.
     47    * For each platform the following steps should be followed.
     48       * View the results and verify failure was the new test.
     49       * View the test actual output file for the current behavior.
     50          * If the output is correct
     51            * Add this text to the appropriate platform's directory.  (ie, platform/qt/fast/forms/testname-expected.txt)
     52          * If this output is incorrect (or is unclear)
     53            * If you can fix it yourself (ie, compile and test under that platform) do so, otherwise,
     54            * Add the new test to the platforms Skipped file (found at platform/type/Skipped)
     55            * File a bug that details the expected results so that someone else can fix it for that platform and add the results.
     56    * Once all tests / Skipped files have been updated, arrange to have the patch landed with these changes.
     57    * Observe the buildbot again to make sure all tests pass / are skipped and builds are happy.  If not, repeat.
     61== Pixel Tests ==