Changes between Version 13 and Version 14 of TestExpectations


Ignore:
Timestamp:
Sep 12, 2012 3:50:45 PM (8 years ago)
Author:
dpranke@chromium.org
Comment:

add docs for the new syntax

Legend:

Unmodified
Added
Removed
Modified
  • TestExpectations

    v13 v14  
    3333The test expectations file is found in a platform-specific directory under LayoutTests. I will use the Chromium version as an example.
    3434
    35 === Syntax ===
     35=== (New) Syntax (Not quite yet landed) ===
     36
     37The syntax of the file is roughly one expectation per line. An expectation can apply to either a directory of tests, or a specific tests. Lines prefixed with "# " are treated as comments,
     38and blank lines are allowed as well.
     39
     40The syntax of a line is roughly:
     41
     42          [bugs] [ "[" modifiers "]" ] test_name [ "[" expectations "]" ]       
     43                           
     44* Tokens are separated by whitespace                                                   
     45* Lines are expected to have one or more bug identifiers, and the linter will complain about lines missing them. Bug identifiers are of the form "webkit.org/b/12345", "crbug.com/12345", or "bug(username)"
     46* If no modifiers are specified, the test applies to all of the configurations applicable to that file
     47* Modifiers can be one or more of `Mac`, `SnowLeopard`, `Lion`, `MountainLion`, `Win`, `XP`, `Vista`, `Win7`, `Win7SP0`, `Linux`, `Lucid`, `x86_64`, `x86`, `Release`, `Debug`. Not all modifiers make sense on all ports or in all lines.
     48* Expectations can be one or more of `Crash`,  `Failure`, `ImageOnlyFailure`, `Pass`, `Rebaseline`, `Slow`, `Skip`, `Timeout`, `WontFix`
     49  * `WontFix` implies `Skip` and also indicates that we don't have any plans to make the test pass
     50  * `Slow` means that we expect the test to run slowly and will use a longer, port-specific timeout. A given line cannot have both `Slow` and `Timeout`
     51  * `Rebaseline` is an old expectation used in conjunction with `webkit-patch rebaseline-expectations` and is not allowed to be checked in
     52  * If no expectations are specified we default to Skip for compatibility with the Skipped files
     53
     54For example:
     55
     56{{{
     57webkit.org/b/12345 [ Win Debug ] fast/html/keygen.html [ Crash ]
     58}}}
     59
     60which indicates that the "fast/html/keygen.html" test file is expected to crash when run in the Debug configuration on Windows, and the tracking bug for this crash is bug #12345 in the webkit bug repository. Note that the test will still be run, so that we can notice if it doesn't actually crash.
     61
     62The following will all produce tests that are skipped:
     63
     64{{{
     65fast/html/keygen.html
     66fast/html/keygen.html [ Skip ]
     67fast/html/keygen.html [ WontFix ]
     68fast/html/keygen.html [ WontFix Crash ] # Indicates that we expect the test to crash; we assume tests will Pass otherwise, in order to report failures as "unexpected"
     69}}}
     70
     71
     72=== Old Syntax ===
    3673
    3774The syntax of the file is roughly one expectation per line. An expectation can apply to either a directory of tests, or a specific tests. Lines prefixed with "// " are treated as comments,
     
    5087which indicates that the "fast/html/keygen.html" test file is expected to crash when run in the Debug configuration on Windows, and the tracking bug for this crash is bug #12345 in the webkit bug repository. Note that the test will still be run, so that we can notice if it doesn't actually crash.
    5188
    52 ==== Expected results ====
     89==== Expected results (Old Syntax) ====
    5390
    5491The expected result can be one of PASS, FAIL, TEXT, IMAGE, CRASH, TIMEOUT, IMAGE+TEXT, AUDIO. These should be fairly self explanatory. Note that IMAGE+TEXT means that we expect *both* the text output and the image output to be different. In other words, this is an AND, not an OR.
     
    5895Multiple expected results are allowed, for tests that are flaky and may produce different results in different runs.
    5996
    60 ==== Modifiers ====
     97==== Modifiers (Old Syntax) ====
    6198
    6299The set of allowed modifiers are a bit more complicated ... they include bug identifiers, configuration parameters, and miscellaneous options.