Changes between Version 4 and Version 5 of TestExpectations

07/07/11 15:05:29 (3 years ago)



  • TestExpectations

    v4 v5  
    1616== Suppressing failures using NRWT: the test_expectations.txt file == 
     18As a bit of background for those of you not very familiar with the syntax of this file (I will put this on the wiki shortly, as well) ... 
     20The syntax of the file is roughly: 
     22<modifier> <modifier>* ":" <test-name> "=" <expected result>+ 
     24the expected result can be one of PASS, FAIL, TEXT, IMAGE, CRASH, TIMEOUT, IMAGE+TEXT, AUDIO. 
     26the modifiers are a bit more complicated ... they include bug identifiers, configuration parameters, and "misc". Bug identifiers are the things we've been talking about. 
     28"configuration parameters" describe which variations of your port the test expectation should apply to, e.g., "VISTA DEBUG" or "SNOWLEOPARD GPU" 
     30"misc" includes "SLOW", "SKIP", "REBASLINE", "NOW", and "WONTFIX".  
     32"SLOW" changes the default timeout for the test to be longer. 
     34"SKIP" tells NRWT to skip the test altogether.  
     36"WONTFIX" is a modifier that is mostly used for reporting to indicate that you have no plans to fix this expectation. E.g., if you never wanted to support webarchives, you might have a line that said "BUGXXX SKIP WONTFIX : webarchive = PASS FAIL". 
     38"NOW" is not actually used anywhere. I added it at some point for some reporting but it can surely be deleted. There also used to be a "DEFER" but that was removed a while back. These concepts could be used to help track which fixes where expected to be handled in a given release, but that sort of thing is better done through a bug tracking system like bugzilla. 
     40"REBASELINE" is used to tell one of our scripts (rebaseline-chromium-webkit-tests) which tests to pull new baselines for. This doesn't really belong in this file at all, but it was a convenient way to be able to tag multiple tests at once at the time. 
    1843== Suppressing failures using ORWT: Skipped files and checked-in failures ==