Changes between Version 1 and Version 2 of Writing Reftests


Ignore:
Timestamp:
Mar 8, 2011, 11:18:41 PM (15 years ago)
Author:
hayato@chromium.org
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Writing Reftests

    v1 v2  
    33Reftests is not supported yet in WebKit. Please see https://bugs.webkit.org/show_bug.cgi?id=36065 for the current status.
    44
    5 == Current proposal ==
    65
    7  * Reftests use a HTML file ('-expected.html') to represent the expected rendering result instead of statically prepared expected result files, such as '-expected.txt', '-expected.checksum' and '-expected.png'. Let's see the following sample reftest.
     6== Reftests in WebKit ==
    87
    9 hello.html
     8Reftests use a HTML file ('-expected.html') to represent the expected rendering result instead of pre-generated expected files, such as '-expected.txt', '-expected.checksum' and '-expected.png'.
    109
     10Let's see the following sample reftest.
     11  * hello.html
    1112{{{
    1213<html><head><title>reftest0001</title>
    … …  
    1516}}}
    1617
    17 
    18 hello-expected.html
    19 
     18  * hello-expected.html
    2019{{{
    2120<html><head><title>reftest0001</title>
    … …  
    2423}}}
    2524
    26 Todo:
     25A new-run-webkit-tests recognizes this is a reftest and run the test as if this is a normal layout test.
     26You can think that a '-expected.html' file is used to produce '-expected.checksum' and '-expected.png' on the fly.
     27If there is a difference between the rendering results of two html files, the test fails.
    2728
     29
     30Screenshot: Todo
     31
     32
     33There is yet another type of reftests, 'a mismatch reftest'.
     34
     35Let's see the following sample 'mismatch' reftest.
     36  * foo.html
     37{{{
     38<html><head><title>reftest0001</title>
     39<body><strong>Hello!</strong></body>
     40</html>
     41}}}
     42
     43  * foo-expected-mismatch.html
     44{{{
     45<html><head><title>reftest0001</title>
     46<body>Hello!</body>
     47</html>
     48}}}
     49
     50In this case, the test fails if both produces the same results. The test passes if there is any differences between them.
     51
     52
     53Screenshot: Todo
     54
     55
     56== Naming convention ==
     57
     58  * (normal) reftests: {testname}.html and {testname}-expected.html
     59  * 'mismatch' reftests: {testname}.html and {testname}-expected-mismatch.html
     60
     61== FAQ ==
     62
     63  * Q. Can we use a platform-specific expected html file?
     64  * A. Yes. Reftests try to obey the current layout test mechanism as possible as we can.
     65
     66  * Q. What happens if we 'rebase' for reftests?
     67  * A. A 'Rebasing' doesn't make sense in reftests. A rebase line tool simply skips reftests.
     68
     69  * Q. If we have hello.html, hello-expected.html and hello-expected-mismatch.html filles, what happens? Does that run two reftests?
     70  * A. No. That should be considerd a wrong configuration. Please don't use one test file for a (normal) reftest and a 'mismatch' reftest. Please choose the different test name for each test.
     71
     72  * Q. Can we use '-expected.html' and '-expected.{checksum,png}' at the same time for one test file?
     73  * A. No.
     74
     75  * Q. What kind of failures reftests produce?
     76  * A. If an image comparison failure happens in reftests, that is marked as a 'IMAGE' failure.
     77    If either of two html files crashes, it is considered as 'CRASH'. So does 'TIMEOUT'.
     78
     79
     80
     81
     82== References ==
     83
     84  * https://developer.mozilla.org/en/Creating_reftest-based_unit_tests
     85  * http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/README.txt