wiki:CreatingLayoutTests

Version 2 (modified by mike.fenton@torchmobile.com, 15 years ago) (diff)

Re-Organization and update.

Note: This page is a work in progress.

Creating a Layout Test

Before creating a test

Layout Tests should be added any time new functionality is added to the Rendering system, or any time a bug is found that is not covered by the existing tests.

Before Creating a new test, always check the platform/Skipped tests to make sure it doesn't already exist. This can be done easily by running run-webkit-tests --skipped=only after compiling with the updated code.

Any change that breaks currently working tests run-webkit-tests should be considered carefully before proceeding.

For more information see the documentation on running tests, writing new tests, and Writing Layout Tests for DumpRenderTree

Design the Test

Guidelines

  • Keep the test as simple as possible and try to only test what you mean to test.

Test with verifiable text results

  • Create the test as a simple HTML page that showcases the bug / new behavior.
  • Test should be placed in the appropriate subdirectory under LayoutTests/ use exsiting tests as a guideline or ask on #webkit.

Test with pixel based results

TODO: Fill me in.

Verify the behavior in the browser

  • Use run-launcher to verify the test provides the desired behavior.

Generate Results for the Test

  • Automatically generate results for your by running run-webkit-tests --new-test-results.
  • Verify the created results match the desired output.
  • If your test is platform independent (results are the same for all platforms)

OR

  • If your test applies to Mac and Windows or other platform
    • Add the primary results file to the main directory beside the test (all results should be testname-expected.txt)
  • If your test has platform dependent results
    • Add the results set for the platform you are working on in the corresponding directory the test under the platform/type/ directory.

Verify BuildBot Status / Update other platform results

Regardless of whether you are making a platform independent change or dependent change, it is your responsibility to ensure that the change does not negatively affect any other ports / platforms.

Once your change is landed, there are some steps required to verify the change. Note: These steps apply for all changes. If you are making a platform dependent change, should should expect that additional results will be required for each platform but the actions are the same.

  • Arrange to have the patch landed
  • Observe the builds with your change at http://build.webkit.org/waterfall
    • Any failures will be clearly marked at the top, builds may fail independent of your changes, scrolling down can verify the state of the previously build.
    • For each platform the following steps should be followed.
      • View the results and verify failure was the new test.
      • View the test actual output file for the current behavior.
        • If the output is correct
          • Add this text to the appropriate platform's directory. (ie, platform/qt/fast/forms/testname-expected.txt)
        • If this output is incorrect (or is unclear)
          • If you can fix it yourself (ie, compile and test under that platform) do so, otherwise,
          • Add the new test to the platforms Skipped file (found at platform/type/Skipped)
          • File a bug that details the expected results so that someone else can fix it for that platform and add the results.
    • Once all tests / Skipped files have been updated, arrange to have the patch landed with these changes.
    • Observe the buildbot again to make sure all tests pass / are skipped and builds are happy. If not, repeat.