= new-run-webkit-tests = [[PageOutline]] {{{new-run-webkit-tests}}} (also known as NRWT) is a replacement for the original {{{run-webkit-tests}}} script that was written in Perl. The new script is written in Python and leverages much of the extensive Python infrastructure we have built up over the intervening years. It has a much more extensive unit test infrastructure and will hopefully be easier to maintain than the old script. (Time will tell.) In addition, NRWT has the ability to run the layout tests in parallel using multiple DumpRenderTrees (or WebkitTestRunners), much stronger features for reporting test failures and managing expected failures, and integration with various dashboards. This page describes how to run NRWT. == See also == * [wiki:HackingOnNewRunWebKitTests] for information on modifying or hacking on NRWT. * [wiki:TestExpectations] for information of managing the expected results of test runs. * [wiki:LayoutTestsSearchPath] for information on how a given port determines which results to use for a test. == Introduction == NRWT is designed to be run both directly by developers working in a terminal window and by the buildbots. It supports a large number of command line options to customize the behavior and the output; only a few of the most commonly used options are mentioned here. The complete list of options is given by {{{new-run-webkit-tests --help}}} and {{{new-run-webkit-tests --help-printing}}}. == Basic Usage == NRWT uses the concept of a "platform" to determine which WebKit port / implementation you are attempting to test. Where possible, it picks a default based on the operating system you are running on and the default configuration you have set (if any). However, sometimes you have to give it hints. By default, when run interactively in a terminal or console window, NRWT attempts to be fairly terse. It will display one line of output on screen, and update that in place with events as the test run occurs. If unexpected things happen (e.g., tests fails that aren't expected to fail), they are printed out as they happen, and then summarized again at the end. If everything passes, it will print a message saying so. Also, it returns the number of unexpected test failures in the exit code for the process. ==== Examples ==== Starting up initially {{{ % new-run-webkit-tests Collecting tests ... }}} (this may take a few seconds, during which time other message may be displayed, like "checking build", "parsing expectations", etc. will be displayed. Eventually the tests will start running, and you'll get something resembling a progress meter: {{{ % new-run-webkit-tests Testing (3%): 863 ran as expected, 0 didn't, 24080 left }}} and when the test completes, you'll get output something like: {{{ % new-run-webkit-tests animations/animation-direction-normal.html -> unexpected text diff mismatch compositing/reflections/nested-reflection-animated.html -> unexpected text diff mismatch Retrying 2 unexpected failure(s) ... animations/animation-direction-normal.html -> unexpected text diff mismatch compositing/reflections/nested-reflection-animated.html -> unexpected text diff mismatch 24894 tests ran as expected, 2 didn't: Regressions: Unexpected text diff mismatch : (10) animations/animation-direction-normal.html = TEXT compositing/reflections/nested-reflection-animated.html = TEXT % echo $? 2 % }}} In addition, by default, a web browser will be opened to an HTML page displaying the tests that failed, and links to the output from the tests. You can notice a few things about this test output: 1. Tests that fail are automatically retried, to see if the failure was "flaky" and only happens some of the time. A flaky failure (one that fails then passing during the retry is reported on screen, but does not count as an actual failure. 2. The retries are all done serially in a single thread, to avoid any load-related or ordering-related issues that might be contributing to the flakiness. 3. The test that failed and the type of test failure that happened are displayed in such a way as to be directly copied into the [wiki:TestExpectations test_expectations.txt] file. == Commonly used Command Line Flags == NRWT supports a ''lot'' of flags. Here are some of the most popular: || || --platform || the platform to run (e.g., "chromium-mac", "win-wk2", etc. || || || --debug || use the Debug configuration || || || --release || use the Release configuration || || -n || --dry-run || dry-run ... do everything but actually run the tests || || -p || --pixel-tests || run with pixel-tests enabled || || || --no-new-test-results || don't create any new test results for new tests (NRWT will create them by default || || || --no-show-results || don't display the summary HTML file in a web browser at the end of the run || == More Advanced Scenarios == === Figuring out the command line passed to DumpRenderTree === The command line used for DumpRenderTree is printed out if you pass the 'config' option to --print (along with a lot of other useful info): {{{ % new-run-webkit-tests --print config,default -n --platform chromium-mac Using port 'chromium-mac-snowleopard' Test configuration: Placing test results in /Volumes/Src/src/dev/src/webkit/Release/layout-test-results Using Release build Pixel tests enabled Regular timeout: 6000, slow test timeout: 30000 Command line: /Volumes/Src/src/dev/src/xcodebuild/Release/DumpRenderTree.app/Contents/MacOS/DumpRenderTree --test-shell --pixel-tests=/Volumes/Src/src/dev/src/webkit/Release/layout-test-results/png_result0.png Worker model: processes Running 16 DumpRenderTrees in parallel over 345 shards (1 locked) 8561 tests ran as expected (16450 didn't run). % }}} === A recommended set of command-line flags === dpranke@chromium.org has spent a lot of time hacking on NRWT. The command line argument he most frequently uses are: {{{ % new-run-webkit-tests --clobber-results --no-new-test-results --no-show-results --no-retry-failures --print config,default }}}