wiki:CreatingLayoutTests

Version 3 (modified by mike.fenton@torchmobile.com, 15 years ago) (diff)

Add link to writing a good test case.

Note: This page is a work in progress.

Creating a Layout Test

Before creating a test

Layout Tests should be added any time new functionality is added to the Rendering system, or any time a bug is found that is not covered by the existing tests.

Before Creating a new test, always check the platform/Skipped tests to make sure it doesn't already exist. This can be done easily by running run-webkit-tests --skipped=only after compiling with the updated code.

Any change that breaks currently working tests run-webkit-tests should be considered carefully before proceeding.

For more information see the documentation on running tests, writing new tests, and Writing Layout Tests for DumpRenderTree

Design the Test

Guidelines

Test with verifiable text results

  • Create the test as a simple HTML page that showcases the bug / new behavior.
  • Test should be placed in the appropriate subdirectory under LayoutTests/ use exsiting tests as a guideline or ask on #webkit.

Test with pixel based results

TODO: Fill me in.

Verify the behavior in the browser

  • Use run-launcher to verify the test provides the desired behavior.

Generate Results for the Test

  • Automatically generate results for your by running run-webkit-tests --new-test-results.
  • Verify the created results match the desired output.
  • If your test is platform independent (results are the same for all platforms)

OR

  • If your test applies to Mac and Windows or other platform
    • Add the primary results file to the main directory beside the test (all results should be testname-expected.txt)
  • If your test has platform dependent results
    • Add the results set for the platform you are working on in the corresponding directory the test under the platform/type/ directory.

Verify BuildBot Status / Update other platform results

Regardless of whether you are making a platform independent change or dependent change, it is your responsibility to ensure that the change does not negatively affect any other ports / platforms.

Once your change is landed, there are some steps required to verify the change. Note: These steps apply for all changes. If you are making a platform dependent change, should should expect that additional results will be required for each platform but the actions are the same.

  • Arrange to have the patch landed
  • Observe the builds with your change at http://build.webkit.org/waterfall
    • Any failures will be clearly marked at the top, builds may fail independent of your changes, scrolling down can verify the state of the previously build.
    • For each platform the following steps should be followed.
      • View the results and verify failure was the new test.
      • View the test actual output file for the current behavior.
        • If the output is correct
          • Add this text to the appropriate platform's directory. (ie, platform/qt/fast/forms/testname-expected.txt)
        • If this output is incorrect (or is unclear)
          • If you can fix it yourself (ie, compile and test under that platform) do so, otherwise,
          • Add the new test to the platforms Skipped file (found at platform/type/Skipped)
          • File a bug that details the expected results so that someone else can fix it for that platform and add the results.
    • Once all tests / Skipped files have been updated, arrange to have the patch landed with these changes.
    • Observe the buildbot again to make sure all tests pass / are skipped and builds are happy. If not, repeat.