Changes between Version 17 and Version 18 of TestExpectations


Ignore:
Timestamp:
Sep 21, 2012 3:47:01 PM (12 years ago)
Author:
dpranke@chromium.org
Comment:

delete description of the old syntax

Legend:

Unmodified
Added
Removed
Modified

  • TestExpectations

    v17 v18  
    3333The test expectations files are found in platform-specific directories under LayoutTests. Ports may use one or more files which are used in order, with later files overriding earlier ones.
    3434
    35 === (New) Syntax ===
     35=== Syntax ===
    3636
    3737The syntax of the file is roughly one expectation per line. An expectation can apply to either a directory of tests, or a specific tests. Lines prefixed with "# " are treated as comments,
    … …  
    4848* Modifiers can be one or more of `Mac`, `SnowLeopard`, `Lion`, `MountainLion`, `Win`, `XP`, `Vista`, `Win7`, `Win7SP0`, `Linux`, `Lucid`, `x86_64`, `x86`, `Release`, `Debug`. Not all modifiers make sense on all ports or in all lines.
    4949* Expectations can be one or more of `Crash`,  `Failure`, `ImageOnlyFailure`, `Pass`, `Rebaseline`, `Slow`, `Skip`, `Timeout`, `WontFix`
    50   * `WontFix` implies `Skip` and also indicates that we don't have any plans to make the test pass
    51   * `WontFix` and `Skip` must be the only expectation and cannot be specified alongside Crash or anything else; since the tests will be skipped, the other expectations will likely become stale.
    52   * `Slow` means that we expect the test to run slowly and will use a longer, port-specific timeout. A given line cannot have both `Slow` and `Timeout`
    53   * `Rebaseline` is an old expectation used in conjunction with `webkit-patch rebaseline-expectations` and is not allowed to be checked in
    54   * If no expectations are specified we default to Skip for compatibility with the Skipped files
    5550
    5651For example:
    … …  
    7570=== Semantics ===
    7671
    77 When parsing the file, we use two rules to figure out if an expectation line applies to the current run:
     72  * `WontFix` implies `Skip` and also indicates that we don't have any plans to make the test pass
     73  * `WontFix` and `Skip` must be the only expectation and cannot be specified alongside Crash or anything else; since the tests will be skipped, the other expectations will likely become stale.
     74  * `Slow` means that we expect the test to run slowly and will use a longer, port-specific timeout. A given line cannot have both `Slow` and `Timeout`
     75  * `Rebaseline` is an old expectation used in conjunction with `webkit-patch rebaseline-expectations` and is not allowed to be checked in
     76  * If no expectations are specified we default to Skip for compatibility with the Skipped files
     77
     78Also, when parsing the file, we use two rules to figure out if an expectation line applies to the current run:
    7879
    7980 1. If the configuration parameters don't match the configuration of the current run, the expectation is ignored.
    … …  
    9495 * {{{fast/html/submit.html}}} to pass (since the configuration parameters don't apply.
    9596
    96 Again, *duplicate expectations are not allowed* within a single file and will generate warnings. Ports may use multiple TestExpectations files, and entries in a later file override entries in an earlier file.
     97Again, '''duplicate expectations are not allowed within a single file''' and will generate warnings. Ports may use multiple TestExpectations files, and entries in a later file override entries in an earlier file. The list of files used by a port is determined by the port's implementation of `expectation_files` in `Tools/Scripts/webkitpy/layout_tests/port/{mac,win,qt,gtk,etc.}.py`
    9798
    9899You can verify that any changes you've made to an expectations file are correct by running:
    … …  
    115116  * If a test fails intermittently, use multiple expectations.
    116117
    117 
    118 === Old Syntax ===
    119 
    120 The syntax of the file is roughly one expectation per line. An expectation can apply to either a directory of tests, or a specific tests. Lines prefixed with "// " are treated as comments,
    121 and blank lines are allowed as well.
    122 
    123 The syntax of a line is roughly:
    124 
    125 <modifier> <modifier>* ":" <test-name> "=" <expected result>+
    126 
    127 For example:
    128 
    129 {{{
    130 BUGWK12345 WIN DEBUG : fast/html/keygen.html = CRASH
    131 }}}
    132 
    133 which indicates that the "fast/html/keygen.html" test file is expected to crash when run in the Debug configuration on Windows, and the tracking bug for this crash is bug #12345 in the webkit bug repository. Note that the test will still be run, so that we can notice if it doesn't actually crash.
    134 
    135 ==== Expected results (Old Syntax) ====
    136 
    137 The expected result can be one of PASS, FAIL, TEXT, IMAGE, CRASH, TIMEOUT, IMAGE+TEXT, AUDIO. These should be fairly self explanatory. Note that IMAGE+TEXT means that we expect *both* the text output and the image output to be different. In other words, this is an AND, not an OR.
    138 
    139 The "FAIL" modifier, on the other hand, is an OR, and is equivalent to saying that any one of TEXT, IMAGE, AUDIO, or IMAGE+TEXT might happen, but not TIMEOUT or CRASH.
    140 
    141 Multiple expected results are allowed, for tests that are flaky and may produce different results in different runs.
    142 
    143 ==== Modifiers (Old Syntax) ====
    144 
    145 The set of allowed modifiers are a bit more complicated ... they include bug identifiers, configuration parameters, and miscellaneous options.
    146 
    147 Bug identifiers allow you to identify the tracking bugs. They must start with a "BUG" prefix. There are three supported identifiers:
    148 
    149    1. BUGWK12345 indicates a bug in the WebKit bug database, and is equivalent to https://bugs.webkit.org/show_bug.cgi?id=12345
    150    2. BUGCR12345 indicates a bug in the Chromium bug database, and is equivalent to https://bugs.chromium.org/12345 .
    151    3. BUGV8_12345 indicates a bug in the V8 bug database, http://code.google.com/p/v8/issues .
    152    3. BUGDPRANKE is a "placeholder" that indicates that no bug has been filed yet, but you should bug that individual about the status.
    153 
    154 Configuration parameters describe which variations of your port the test expectation should apply to. Typically each TestExpectations file is used for multiple variations of a test run, because each variation usually has a lot of failures in common, and it's easier to manage all of the failures in one place. The exact set of configuration parameters will vary from port to port. Here are the options supported for Chromium:
    155 
    156  * LEOPARD SNOWLEOPARD XP VISTA WIN7 LUCID : these indicate particular versions of particular operating systems. Multiple options may appear on a single line, but duplicates are not allowed
    157    * MAC WIN LINUX: These are "macros" that expand out to all of the relevant versions of each operationg system. It is a syntax error to specify both one of the macros and one of the expanded options, e.g. "MAC LEOPARD"
    158  * DEBUG RELEASE: the different build types
    159  * GPU CPU: Whether we are testing the "GPU-accelerated" code paths, or the regular software-only (CPU) code paths
    160  * X86 X86_64: Whether we are testing the 32-bit or 64-bit versions of the code.
    161 
    162 Different ports may not support all of these options, since they may not be relevant.
    163 
    164 Note that not all parameters need to be listed, and if no parameters in a particular category are listed, the parser assumes that any combination applies.
    165 
    166 To figure out if a configuration matches, we take the logical OR of the modifiers in each category, and the AND of modifiers in different categories, so a line containing "LEOPARD VISTA DEBUG GPU" means that it will apply to tests run on either Mac Leopard or Windows Vista, in Debug mode, using the GPU acceleration code paths.
    167 
    168 There are also "miscellaneous" modifiers:
    169 
    170  * SLOW : This indicates that the test is expected to be slow.  Normally, a RELEASE mode test is allowed six seconds before being declared a TIMEOUT, and a DEBUG mode test double that (so, 12 seconds).  SLOW tests get five times the normal timeout for a test, so 30 and 60 seconds, respectively.  Note that SLOW tests cannot be expected to TIMEOUT.
    171  * SKIP: This indicates that the test will never pass and there's no point in running it. This is equivalent to listing the test in the Skipped files (see below)
    172  * WONTFIX: This is a modifier that does not affect anything in the test run itself, but can be used for reporting; it indicates that we don't ever expect the test to pass.
    173  * NOW: This modifier has never been used and should probably be removed.
    174  * REBASELINE: This modifier is used by "webkit-patch rebaseline-expectations", and is not allowed to exist in a checked-in version of the file.
    175 
    176 
    177118== Suppressing failures using ORWT: Skipped files and checked-in failures ==
    178119