Changes between Version 6 and Version 7 of TestExpectations


Ignore:
Timestamp:
Jul 7, 2011 3:58:06 PM (9 years ago)
Author:
dpranke@chromium.org
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • TestExpectations

    v6 v7  
    1414== How we manage tests that fail ==
    1515
    16 The primary function of the LayoutTests is as a *regression test suite*. This means that, while we care about whether a page is being rendered correctly, we care more about whether the page is being rendered the way we expect it to. In other words, we look more for changes in behavior than we do for correctness.
     16The primary function of the LayoutTests is as a ''regression test suite''. This means that, while we care about whether a page is being rendered correctly, we care more about whether the page is being rendered the way we expect it to. In other words, we look more for changes in behavior than we do for correctness.
    1717
    1818All layout tests have "expected results", which may be one of several forms. The test may produce a text file containing javascript log messages, or a text rendering of the Render Tree. It may also produce a screen capture of the rendered page (if you are running with {{{--pixel-tests}}} enabled) as PNG files as well. For WebAudio tests, we can produce WAV files instead of either text or PNG files. For any of these types of tests, there are files checked into the LayoutTests directory named "-expected.{txt,png,wav}". In many (most?) cases, the output is expected to be generic and match on any webkit port.
     
    7272   3. BUGDPRANKE is a "placeholder" that indicates that no bug has been filed yet, but you should bug that individual about the status.
    7373
    74 
    7574Configuration parameters describe which variations of your port the test expectation should apply to. Typically each test_expectations.txt file is used for multiple variations of a test run, because each variation usually has a lot of failures in common, and it's easier to manage all of the failures in one place. The exact set of configuration parameters will vary from port to port. Here are the options supported for Chromium:
    7675
     
    118117Again, *duplicate expectations are not allowed*.
    119118
     119You can verify that any changes you've made to an expectations file are correct by running:
     120
     121{{{
     122% new-run-webkit-tests --lint-test-files
     123}}}
     124
     125which will cycle through all of the possible combinations of configurations looking for errors and conflicts. It's not instantaneous, but shouldn't take more than a minute or two.
     126
    120127== Suppressing failures using ORWT: Skipped files and checked-in failures ==
    121128