Changes between Version 18 and Version 19 of TestExpectations


Ignore:
Timestamp:
Sep 24, 2012 3:05:57 PM (10 years ago)
Author:
dpranke@chromium.org
Comment:

add comments about flaky tests

Legend:

Unmodified
Added
Removed
Modified
  • TestExpectations

    v18 v19  
    4747* If no modifiers are specified, the test applies to all of the configurations applicable to that file
    4848* Modifiers can be one or more of `Mac`, `SnowLeopard`, `Lion`, `MountainLion`, `Win`, `XP`, `Vista`, `Win7`, `Win7SP0`, `Linux`, `Lucid`, `x86_64`, `x86`, `Release`, `Debug`. Not all modifiers make sense on all ports or in all lines.
    49 * Expectations can be one or more of `Crash`,  `Failure`, `ImageOnlyFailure`, `Pass`, `Rebaseline`, `Slow`, `Skip`, `Timeout`, `WontFix`
     49* Expectations can be one or more of `Crash`,  `Failure`, `ImageOnlyFailure`, `Pass`, `Rebaseline`, `Slow`, `Skip`, `Timeout`, `WontFix`. If mulitple expectations are listed, the test is considered "flaky" and any of those results will be considered as expected.
    5050
    5151For example:
     
    8787webkit.org/b/12345 [ SnowLeopard ] fast/html/keygen.html [ Pass ]
    8888webkit.org/b/12345 [ Vista ] fast/forms/submit.html [ ImageOnlyFailure ]
     89webkit.org/b/12345 fast/html/section-element.html [ Failure Crash ]
    8990}}}
    9091
     
    9495 * {{{fast/html/keygen.html}}} to pass (since the exact match on the test name)
    9596 * {{{fast/html/submit.html}}} to pass (since the configuration parameters don't apply.
     97 * {{{fast/html/section-element.html}}} to either crash or produce a text (or image and text) failure, but not to do anything, i.e., it is not expected to time out or pass.
    9698
    9799Again, '''duplicate expectations are not allowed within a single file''' and will generate warnings. Ports may use multiple TestExpectations files, and entries in a later file override entries in an earlier file. The list of files used by a port is determined by the port's implementation of `expectation_files` in `Tools/Scripts/webkitpy/layout_tests/port/{mac,win,qt,gtk,etc.}.py`