Changes between Version 13 and Version 14 of TestExpectations

Timestamp:

Sep 12, 2012 3:50:45 PM (12 years ago)

Author:

dpranke@chromium.org

Comment:

add docs for the new syntax

TestExpectations

@@ -33 +33 @@
 The test expectations file is found in a platform-specific directory under LayoutTests. I will use the Chromium version as an example.
 
-=== Syntax ===
+=== (New) Syntax (Not quite yet landed) ===
+
+The syntax of the file is roughly one expectation per line. An expectation can apply to either a directory of tests, or a specific tests. Lines prefixed with "# " are treated as comments,
+and blank lines are allowed as well.
+
+The syntax of a line is roughly:
+
+          [bugs] [ "[" modifiers "]" ] test_name [ "[" expectations "]" ]       
+                            
+* Tokens are separated by whitespace                                                    
+* Lines are expected to have one or more bug identifiers, and the linter will complain about lines missing them. Bug identifiers are of the form "webkit.org/b/12345", "crbug.com/12345", or "bug(username)"
+* If no modifiers are specified, the test applies to all of the configurations applicable to that file
+* Modifiers can be one or more of `Mac`, `SnowLeopard`, `Lion`, `MountainLion`, `Win`, `XP`, `Vista`, `Win7`, `Win7SP0`, `Linux`, `Lucid`, `x86_64`, `x86`, `Release`, `Debug`. Not all modifiers make sense on all ports or in all lines.
+* Expectations can be one or more of `Crash`,  `Failure`, `ImageOnlyFailure`, `Pass`, `Rebaseline`, `Slow`, `Skip`, `Timeout`, `WontFix`
+  * `WontFix` implies `Skip` and also indicates that we don't have any plans to make the test pass
+  * `Slow` means that we expect the test to run slowly and will use a longer, port-specific timeout. A given line cannot have both `Slow` and `Timeout`
+  * `Rebaseline` is an old expectation used in conjunction with `webkit-patch rebaseline-expectations` and is not allowed to be checked in
+  * If no expectations are specified we default to Skip for compatibility with the Skipped files 
+
+For example:
+
+{{{
+webkit.org/b/12345 [ Win Debug ] fast/html/keygen.html [ Crash ]
+}}}
+
+which indicates that the "fast/html/keygen.html" test file is expected to crash when run in the Debug configuration on Windows, and the tracking bug for this crash is bug #12345 in the webkit bug repository. Note that the test will still be run, so that we can notice if it doesn't actually crash.
+
+The following will all produce tests that are skipped:
+
+{{{
+fast/html/keygen.html
+fast/html/keygen.html [ Skip ]
+fast/html/keygen.html [ WontFix ]
+fast/html/keygen.html [ WontFix Crash ] # Indicates that we expect the test to crash; we assume tests will Pass otherwise, in order to report failures as "unexpected"
+}}}
+
+
+=== Old Syntax ===
 
 The syntax of the file is roughly one expectation per line. An expectation can apply to either a directory of tests, or a specific tests. Lines prefixed with "// " are treated as comments,
@@ -50 +87 @@
 which indicates that the "fast/html/keygen.html" test file is expected to crash when run in the Debug configuration on Windows, and the tracking bug for this crash is bug #12345 in the webkit bug repository. Note that the test will still be run, so that we can notice if it doesn't actually crash.
 
-==== Expected results ====
+==== Expected results (Old Syntax) ====
 
 The expected result can be one of PASS, FAIL, TEXT, IMAGE, CRASH, TIMEOUT, IMAGE+TEXT, AUDIO. These should be fairly self explanatory. Note that IMAGE+TEXT means that we expect *both* the text output and the image output to be different. In other words, this is an AND, not an OR.
@@ -58 +95 @@
 Multiple expected results are allowed, for tests that are flaky and may produce different results in different runs.
 
-==== Modifiers ====
+==== Modifiers (Old Syntax) ====
 
 The set of allowed modifiers are a bit more complicated ... they include bug identifiers, configuration parameters, and miscellaneous options.