Changes between Version 14 and Version 15 of TestExpectations


Ignore:
Timestamp:
Sep 12, 2012 4:11:55 PM (8 years ago)
Author:
dpranke@chromium.org
Comment:

more editing, cleanup for the new syntax.

Legend:

Unmodified
Added
Removed
Modified
  • TestExpectations

    v14 v15  
    1 = Managing Test Expectations with {{{new-run-webkit-tests}}} and {{{old-run-webkit-tests}}} =
     1= Managing Test Expectations =
    22
    33[[PageOutline]]
     
    2929Lastly, we also support the concept of "reference tests", which check that two pages are rendered identically (pixel-by-pixel). As long as the two tests' output match, the tests pass. For more on reference tests, see [wiki:RefTests].
    3030
    31 == Suppressing failures using NRWT: the TestExpectations file ==
     31== Suppressing failures using the TestExpectations file ==
    3232
    33 The test expectations file is found in a platform-specific directory under LayoutTests. I will use the Chromium version as an example.
     33The test expectations files are found in platform-specific directories under LayoutTests. Ports may use one or more files which are used in order, with later files overriding earlier ones.
    3434
    3535=== (New) Syntax (Not quite yet landed) ===
     
    4848* Expectations can be one or more of `Crash`,  `Failure`, `ImageOnlyFailure`, `Pass`, `Rebaseline`, `Slow`, `Skip`, `Timeout`, `WontFix`
    4949  * `WontFix` implies `Skip` and also indicates that we don't have any plans to make the test pass
     50  * `WontFix` and `Skip` must be the only expectation and cannot be specified alongside Crash or anything else; since the tests will be skipped, the other expectations will likely become stale.
    5051  * `Slow` means that we expect the test to run slowly and will use a longer, port-specific timeout. A given line cannot have both `Slow` and `Timeout`
    5152  * `Rebaseline` is an old expectation used in conjunction with `webkit-patch rebaseline-expectations` and is not allowed to be checked in
     
    6869fast/html/keygen.html [ WontFix Crash ] # Indicates that we expect the test to crash; we assume tests will Pass otherwise, in order to report failures as "unexpected"
    6970}}}
     71
     72=== Semantics ===
     73
     74When parsing the file, we use two rules to figure out if an expectation line applies to the current run:
     75
     76 1. If the configuration parameters don't match the configuration of the current run, the expectation is ignored.
     77 2. Expectations that match more of a test name are used before expectations that match less of a test name.
     78
     79For example, if you had the following lines in your file, and you were running a debug build on Mac SnowLeopard:
     80
     81{{{
     82webkit.org/b/12345 [ SnowLeopard ] fast/html [ Failure ]
     83webkit.org/b/12345 [ SnowLeopard ] fast/html/keygen.html [ Pass ]
     84webkit.org/b/12345 [ Vista ] fast/forms/submit.html [ ImageOnlyFailure ]
     85}}}
     86
     87You'd expect:
     88
     89 * {{{fast/html/article-element.html}}} to fail with a text diff (since it is in the fast/html directory)
     90 * {{{fast/html/keygen.html}}} to pass (since the exact match on the test name)
     91 * {{{fast/html/submit.html}}} to pass (since the configuration parameters don't apply.
     92
     93Again, *duplicate expectations are not allowed* within a single file and will generate warnings. Ports may use multiple TestExpectations files, and entries in a later file override entries in an earlier file.
     94
     95You can verify that any changes you've made to an expectations file are correct by running:
     96
     97{{{
     98% new-run-webkit-tests --lint-test-files
     99}}}
     100
     101which will cycle through all of the possible combinations of configurations looking for problems.
     102
     103== Rules of Thumb for Suppressing Failures ==
     104
     105Here are some rules-of-thumb that you could apply when adding new expectations to the file:
     106
     107  * Only use WontFix when you know for sure we will never, ever implement the capability, tested by the test
     108  * Use Skip when the test:
     109     * throws JavaScript exception and makes text-only test manifest as pixel-test. This usually manifests in "Missing test expectations" failure.
     110     * disrupts running of the other tests. Although this is not typical, it may still be possible. Please make sure to give Pri-1 to the associated bug.
     111  * Try to specify platforms and configs as accurately as possible. If a test passes on all but on platform, it should only have that platform listed
     112  * If a test fails intermittently, use multiple expectations.
    70113
    71114
     
    128171 * REBASELINE: This modifier is used by "webkit-patch rebaseline-expectations", and is not allowed to exist in a checked-in version of the file.
    129172
    130 === Semantics ===
    131 
    132 When parsing the file, we use two rules to figure out if an expectation line applies to the current run:
    133 
    134  1. If the configuration parameters don't match the configuration of the current run, the expectation is ignored.
    135  2. Expectations that match more of a test name are used before expectations that match less of a test name.
    136 
    137 For example, if you had the following lines in your file, and you were running a debug build on Mac SnowLeopard:
    138 
    139 {{{
    140 BUGWK12345 SNOWLEOPARD : fast/html = TEXT
    141 BUGWK12345 SNOWLEOPARD : fast/html/keygen.html = PASS
    142 BUGWK12345 VISTA : fast/forms/submit.html = IMAGE
    143 }}}
    144 
    145 You'd expect:
    146 
    147  * {{{fast/html/article-element.html}}} to fail with a text diff (since it is in the fast/html directory)
    148  * {{{fast/html/keygen.html}}} to pass (since the exact match on the test name)
    149  * {{{fast/html/submit.html}}} to pass (since the configuration parameters don't apply.
    150 
    151 Again, *duplicate expectations are not allowed*.
    152 
    153 You can verify that any changes you've made to an expectations file are correct by running:
    154 
    155 {{{
    156 % new-run-webkit-tests --lint-test-files
    157 }}}
    158 
    159 which will cycle through all of the possible combinations of configurations looking for errors and conflicts. It's not instantaneous, but shouldn't take more than a minute or two.
    160 
    161 == Rules of Thumb for Suppressing Failures ==
    162 
    163 Here are some rules-of-thumb that you could apply when adding new expectations to the file:
    164 
    165   * Only use WONTFIX when you know for sure we will never, ever implement the capability, tested by the test
    166   * Use SKIP when the test:
    167      * throws JavaScript exception and makes text-only test manifest as pixel-test. This usually manifests in "Missing test expectations" failure.
    168      * disrupts running of the other tests. Although this is not typical, it may still be possible. Please make sure to give Pri-1 to the associated bug.
    169   * Try to specify platforms and configs as accurately as possible. If a test passes on all but on platform, it should only have that platform listed
    170   * If a test fails intermittently, use multiple expectations.
    171173
    172174== Suppressing failures using ORWT: Skipped files and checked-in failures ==