Hacking on new-run-webkit-tests (an overview of the code)
See Also
The code for new-run-webkit-tests lives in Tools/Scripts/webkitpy/layout_tests (although it has dependencies on a few other webkitpy packages).
There are four major parts to the code:
- The port/ package, which encapsulates all of the logic that is either port-specific (to apple mac, gtk, etc.) or abstracts over stuff that *might* need to be customized, even if no one does it today.
- The models/ package, which captures the major data structures used during the layout tests
- The controllers/ package, which captures the basic control flow of a test run
- The layout_package/ and views/ packages, which are responsible for updating the logs, the output a console while running the test, and formatting the results for the dashboards.
Data Models
The TestExpectations object contains an in-memory representation of all of the entries in the various TestExpectations files
The test_run_results.RunDetails object is a plain data object containing:
- the results of the initial full run of tests (as a TestRunResults object)
- the results of any retried tests (another TestRunResults object)
- a summary of the combined results from both runs (as a plain python dictionary)
- the port-specific exit status corresponding to the summarized results
Each TestRunResults object is a list of TestResults (one for each test run) plus a bunch of aggregated statistics
Each TestResult contains a list of computed failure types (TestFailures, like text mismatch, image mismatch, crash) for each test as well as some statistics, but does not contain the actual output (i.e., the DriverOutput) from each test; that data is discarded except when failures occur, in which case the DriverOutput is written to the filesystem.
Control flow
The main run_webkit_tests.py parses the command line otpions and performs some setup (determining which Port object to use, and setting up a Printer to control output and logging), and then creates a Manager object to actually find and run the tests. Once the tests are run, we print the results on stdout as necessary and upload them to the dashboards if so desired.
The Manager creates a LayoutTestFinder to handle the complex logic of figuring out which tests to run, and a LayoutTestRunner to actually run the tests. The manager is also responsible for handling the retries and computing the RunDetails.
The LayoutTestRunner implements the multiprocess manager/worker-queue model used to run tests in parallel. All of the tests are turned into TestInput objects and written to a shared queue. The N child processes read from the queue, and run each test using a SingleTestRunner instance. the SingleTestRunner is responsible for actually executing the test (by talking to a Driver object obtained from a port, which talks to DumpRenderTree/WebKitTestRunner/content_shell) and examining the output. The results of examining the output are turned into a TestResults object. When there are failures, they are written to disk using a TestResultWriter.
