wiki:RunWebKitTestsDesign

Planned Design For run-webkit-tests

This is in bugzilla as http://bugs.webkit.org/show_bug.cgi?id=14740

Directory Structure

  • LayoutTests/
    • cross-platform tests and results
    • platform/
      • mac/
        • Mac-specific tests, results, and Skipped file
      • mac-leopard/
        • Leopard-specific tests, results, and Skipped file
      • qt/
        • Qt-specific tests, results, and Skipped file
      • win/
        • Windows-specific tests, results, and Skipped file

The platform directories are named based on a hyphenated-prefix system. For example, mac-leopard is a subset of mac.

Configuration

Each platform configuration should define one variable, $platformTestDirectory, that specifies the most specific directory under LayoutTests/platform/ that applies to that platform. For instance, on Mac OS X Leopard, mac-leopard should be specified.

This name is broken into all possible hyphenated prefixes. For example, if a platform specifies a-b-c as its $platformTestDirectory, the directories in its platform hierarchy become (in order of decreasing specificity):

  • a-b-c
  • a-b
  • a

Finding Tests

run-webkit-tests should search recursively through all the subdirectories of LayoutTests/ to find test files, just as it does currently. However, it should not search beneath platform/. It should then search through each of the directories in the platform hierarchy to find additional platform-specific tests to run.

Finding Skipped Files

Each platform directory can contain a file named Skipped that contains a list of tests/directories to be excluded from testing. run-webkit-tests should concatenate all the Skipped files found in the directories in the platform hierarchy before processing their contents.

Finding Expected Results

When searching for the expected results for a specific test, run-webkit-tests should search through each directory in the platform hierarchy in order of decreasing specificity, then search in LayoutTests/ if no platform-specific result is found. The first expected result found is the one used, regardless of whether it results in a success or failure for the test.

Generating New Results

New platform-specific tests should have their results generated right next to the test. For example, a new test called LayoutTests/platform/mac-leopard/fast/new-leopard-test.html should have its results placed in LayoutTests/platform/mac-leopard/fast/.

New cross-platform tests (those not in a subdirectory of platform/) should have their results generated in one of two places. For text-only tests, the results should be put next to the test itself. For render tree tests, the results should be put in the least-specific directory in the platform hierarchy (i.e., one of the top-level subdirectories of platform/). So, a new render tree test called LayoutTests/fast/new-test.html should have its results placed in LayoutTests/platform/<name>/fast/, where <name> is the least-specific platform name. (Note that this will mean that new render tree results will only be generated for one platform at a time.) If the user decides that the results are either more or less specific than the default location implies, the user should move the results manually.

Other Enhancements

  • When generating new results, run-webkit-tests should tell you what directory they were placed in.
  • When generating new render tree results and not running in --pixel-tests mode, run-webkit-tests should tell you that you need to generate pixel results as well.
  • There should be a way to mark tests in the Skipped file as being hangs so that they can be omitted when running with --skipped=only.
  • run-webkit-tests should be able to detect when DRT crashes and restart it, rather than reporting that the next ~500 tests "fail".
  • Consider renaming the "skipped" file to something like SkippedTests so that things could be added/changed between platforms without cluttering the run-webkit-tests script. Possible uses could include a SkippedLeaks for the leaks tool/valgrind/whatever.
Last modified 10 years ago Last modified on Jul 27, 2007 11:08:07 AM