= Planned Design For run-webkit-tests = == Directory Structure == * LayoutTests/ * ''cross-platform tests and results'' * platform/ * mac/ * ''Mac-specific tests, results, and Skipped file'' * mac-leopard/ * ''Leopard-specific tests, results, and Skipped file'' * qt/ * ''Qt-specific tests, results, and Skipped file'' * win/ * ''Windows-specific tests, results, and Skipped file'' The platform directories are named based on a hyphenated-prefix system. For example, `mac-leopard` is a subset of `mac`. == Configuration == Each platform configuration should define one variable, `$platformTestDirectory`, that specifies the most specific directory under `LayoutTests/platform/` that applies to that platform. For instance, on Mac OS X Leopard, `mac-leopard` should be specified. This name is broken into all possible hyphenated prefixes. For example, if a platform specifies `a-b-c` as its `$platformTestDirectory`, the directories in its ''platform hierarchy'' become (in order of decreasing specificity): * `a-b-c` * `a-b` * `a` == Finding Tests == `run-webkit-tests` should search recursively through all the subdirectories of `LayoutTests/` to find test files, just as it does currently. However, it should '''not''' search beneath `platform/`. It should then search through each of the directories in the platform hierarchy to find additional platform-specific tests to run. == Finding Expected Results == When searching for the expected results for a specific test, `run-webkit-tests` should search through each directory in the platform hierarchy in order of decreasing specificity, then search in `LayoutTests/` if no platform-specific result is found. The first expected result found is the one used, regardless of whether it results in a success or failure for the test. == Generating New Results == New platform-specific tests should have their results generated right next to the test. For example, a new test called `LayoutTests/platform/mac/leopard/fast/new-leopard-test.html` should have its results placed in `LayoutTests/platform/mac/leopard/fast/`. New cross-platform tests (those not in a subdirectory of `platform/`) should have their results generated in one of two places. For text-only tests, the results should be put next to the test itself. For render tree tests, the results should be put in the least-specific directory in the platform hierarchy (i.e., one of the top-level subdirectories of `platform/`). So, a new render tree test called `LayoutTests/fast/new-test.html` should have its results placed in `LayoutTests/platform//fast/`, where `` is the least-specific platform name. (Note that this will mean that new render tree results will only be generated for one platform at a time.) If the user decides that the results are either more or less specific than the default location implies, the user should move the results manually. == Other Enhancements == * When generating new results, `run-webkit-tests` should tell you what directory they were placed in. * When generating new render tree results and not running in `--pixel-tests` mode, `run-webkit-tests` should tell you that you need to generate pixel results as well. * There should be a way to mark tests in the `Skipped` file as being hangs so that they can be omitted when running with `--skipped=only`. * `run-webkit-tests` should be able to detect when DRT crashes and restart it, rather than reporting that the next ~500 tests "fail". * Consider renaming the "skipped" file to something like SkippedTests so that things could be added/changed between platforms without cluttering the run-webkit-tests script. Possible uses could include a SkippedLeaks for the leaks tool/valgrind/whatever.