Note: This page is a work in progress.

Creating a Layout Test

Before creating a test

Layout Tests should be added any time new functionality is added to the Rendering system, or any time a bug is found that is not covered by the existing tests.

Before Creating a new test, always check the platform/Skipped tests to make sure it doesn't already exist. This can be done easily by running run-webkit-tests --skipped=only after compiling with the updated code.

Any change that breaks currently working tests run-webkit-tests should be considered carefully before proceeding.

For more information see the documentation on ​running tests, ​writing new tests, and Writing Layout Tests for DumpRenderTree

Design the Test

Guidelines

• Keep the test as simple as possible and try to only test what you mean to test.
• See Writing Layout Tests for DumpRenderTree for additional details on creating a good test.

Test with verifiable text results

• Create the test as a simple HTML page that showcases the bug / new behavior.
• Test should be placed in the appropriate subdirectory under LayoutTests/ use exsiting tests as a guideline or ask on #webkit.

Test with pixel based results

TODO: Fill me in.

Verify the behavior in the browser

• Use run-launcher to verify the test provides the desired behavior.

Generate Results for the Test

• Automatically generate results for your by running run-webkit-tests --new-test-results.
• Verify the created results match the desired output.
• If your test is platform independent (results are the same for all platforms)

OR

• If your test applies to Mac and Windows or other platform
  • Add the primary results file to the main directory beside the test (all results should be testname-expected.txt)
• If your test has platform dependent results
  • Add the results set for the platform you are working on in the corresponding directory the test under the platform/type/ directory.

Verify BuildBot Status / Update other platform results

Regardless of whether you are making a platform independent change or dependent change, it is your responsibility to ensure that the change does not negatively affect any other ports / platforms. This includes updating platform specific results or contacting a port maintainer to do this for you (for contacts, see: ​http://trac.webkit.org/wiki/WebKit%20Team).

Once your change is landed, there are some steps required to verify the change. Note: These steps apply for all changes. If you are making a platform dependent change, you should expect that additional results will be required for each platform but the actions are the same.

• Arrange to have the patch landed
• Observe the builds with your change at ​http://build.webkit.org/waterfall
  • Any failures will be clearly marked at the top, builds may fail independent of your changes, scrolling down can verify the state of the previously build.
  • For each platform the following steps should be followed.
    • View the results and verify failure was the new test.
    • View the test actual output file for the current behavior.
      • If the output is correct
        • Add this text to the appropriate platform's directory. (ie, platform/qt/fast/forms/testname-expected.txt)
      • If this output is incorrect (or is unclear)
        • If you can fix it yourself (ie, compile and test under that platform) do so, otherwise,
        • Add the new test to the platforms Skipped file (found at platform/type/Skipped)
        • File a bug that details the expected results so that someone else can fix it for that platform and add the results.
  • Once all tests / Skipped files have been updated, arrange to have the patch landed with these changes.
  • Observe the buildbot again to make sure all tests pass / are skipped and builds are happy. If not, repeat.