Changes between Version 18 and Version 19 of TestExpectations

Timestamp:

Sep 24, 2012 3:05:57 PM (12 years ago)

Author:

dpranke@chromium.org

Comment:

add comments about flaky tests

TestExpectations

@@ -47 +47 @@
 * If no modifiers are specified, the test applies to all of the configurations applicable to that file
 * Modifiers can be one or more of `Mac`, `SnowLeopard`, `Lion`, `MountainLion`, `Win`, `XP`, `Vista`, `Win7`, `Win7SP0`, `Linux`, `Lucid`, `x86_64`, `x86`, `Release`, `Debug`. Not all modifiers make sense on all ports or in all lines.
-* Expectations can be one or more of `Crash`,  `Failure`, `ImageOnlyFailure`, `Pass`, `Rebaseline`, `Slow`, `Skip`, `Timeout`, `WontFix`
+* Expectations can be one or more of `Crash`,  `Failure`, `ImageOnlyFailure`, `Pass`, `Rebaseline`, `Slow`, `Skip`, `Timeout`, `WontFix`. If mulitple expectations are listed, the test is considered "flaky" and any of those results will be considered as expected.
 
 For example:
@@ -87 +87 @@
 webkit.org/b/12345 [ SnowLeopard ] fast/html/keygen.html [ Pass ]
 webkit.org/b/12345 [ Vista ] fast/forms/submit.html [ ImageOnlyFailure ]
+webkit.org/b/12345 fast/html/section-element.html [ Failure Crash ]
 }}}
 
@@ -94 +95 @@
  * {{{fast/html/keygen.html}}} to pass (since the exact match on the test name)
  * {{{fast/html/submit.html}}} to pass (since the configuration parameters don't apply.
+ * {{{fast/html/section-element.html}}} to either crash or produce a text (or image and text) failure, but not to do anything, i.e., it is not expected to time out or pass.
 
 Again, '''duplicate expectations are not allowed within a single file''' and will generate warnings. Ports may use multiple TestExpectations files, and entries in a later file override entries in an earlier file. The list of files used by a port is determined by the port's implementation of `expectation_files` in `Tools/Scripts/webkitpy/layout_tests/port/{mac,win,qt,gtk,etc.}.py`