| Version 5 (modified by , 13 years ago) ( diff ) |
|---|
Managing Test Expectations with new-run-webkit-tests and old-run-webkit-tests
Contents
- How we manage tests that fail
- Suppressing failures using NRWT: the test_expectations.txt file
- Suppressing failures using ORWT: Skipped files and checked-in files.
See also
- NewRunWebKitTests
- Triaging test failures
How We manage tests that fail
Suppressing failures using NRWT: the test_expectations.txt file
As a bit of background for those of you not very familiar with the syntax of this file (I will put this on the wiki shortly, as well) ...
The syntax of the file is roughly:
<modifier> <modifier>* ":" <test-name> "=" <expected result>+
the expected result can be one of PASS, FAIL, TEXT, IMAGE, CRASH, TIMEOUT, IMAGE+TEXT, AUDIO.
the modifiers are a bit more complicated ... they include bug identifiers, configuration parameters, and "misc". Bug identifiers are the things we've been talking about.
"configuration parameters" describe which variations of your port the test expectation should apply to, e.g., "VISTA DEBUG" or "SNOWLEOPARD GPU"
"misc" includes "SLOW", "SKIP", "REBASLINE", "NOW", and "WONTFIX".
"SLOW" changes the default timeout for the test to be longer.
"SKIP" tells NRWT to skip the test altogether.
"WONTFIX" is a modifier that is mostly used for reporting to indicate that you have no plans to fix this expectation. E.g., if you never wanted to support webarchives, you might have a line that said "BUGXXX SKIP WONTFIX : webarchive = PASS FAIL".
"NOW" is not actually used anywhere. I added it at some point for some reporting but it can surely be deleted. There also used to be a "DEFER" but that was removed a while back. These concepts could be used to help track which fixes where expected to be handled in a given release, but that sort of thing is better done through a bug tracking system like bugzilla.
"REBASELINE" is used to tell one of our scripts (rebaseline-chromium-webkit-tests) which tests to pull new baselines for. This doesn't really belong in this file at all, but it was a convenient way to be able to tag multiple tests at once at the time.
