Changes between Version 25 and Version 26 of Writing Layout Tests for DumpRenderTree


Ignore:
Timestamp:
Oct 27, 2011 4:49:11 PM (12 years ago)
Author:
jchaffraix@webkit.org
Comment:

How to write good test case (edit 1), mostly text cases and 'do you need a pixel test'

Legend:

Unmodified
Added
Removed
Modified

  • Writing Layout Tests for DumpRenderTree

    v25 v26  
    3535'''Note:''' You get more points if your test works in ANY browsers.
    3636
    37 === fast ===
     37=== Fast ===
    3838
    3939WebKit has several thousands layout tests that are run in parallel. If your layout test take a lot time, it will slow down everyone's testing. Also DumpRenderTree has a default timer of several seconds before it considers that a test timed out. '''This is true even if you are using waitUntilDone / notifyDone.'''
    … …  
    4747=== Understandable & clear ===
    4848
    49 ''FIXME: Should have some tips here about how to make a test which is easy to know when it succeeds, things like a 100x100 green block, just printing SUCCESS or FAILURE, and using `dumpAsText()`.''
    50 
     49A good reading on how to write awesome test cases is the [http://www.w3.org/Style/CSS/Test/guidelines.html CSS2.1 Test Case Authoring Guidelines]. Especially read the ''section: 4. Writing ideal tests''
     50
     51Now in the context of WebKit, test should at least indicate:
     52 * which bug it belongs to. That means the bug number or the bugzilla URL (better as you can paste it in a browser) but also the bug title to see what is tested.
     53 * what is the condition for success or failure.
     54
     55Bad test:
     56
     57{{{
     58<style>
     59    .block { color: red; }
     60</style>
     61<p class="block"></p>
     62}}}
     63
     64This is bad because it misses all the WebKit information but also because it uses '''red''' when it means passing. Red should be reserved for failures.
     65
     66'''Note:''' This test is badly formatted and if this is not something your test needs, it is better to avoid it.
     67
     68Better test:
     69
     70{{{
     71<!DOCTYPE html>
     72<html>
     73<head>
     74<style>
     75    .block { color: green; }
     76</style>
     77</head>
     78<body>
     79<p> Bug <a href="http://webkit.org/b/XXXX">XXXX</a>: WebKit does not apply class-selector</p>
     80<p> For this test to pass, you should see a green rectangle below. </p>
     81<p class="block"></p>
     82</body>
     83</html>
     84}}}
     85
     86This test is a huge improvement but we can actually go one step further! (see the next section about writing pixel tests on that)
    5187
    5288One last comment (FIXME: move it to a better section):
    … …  
    5490 * Tests should not access the Internet. Avoid `http:` URLs in `src` and `href` attributes, in CSS properties and in XMLHttpRequest. Testing WebKit's network layer should be done using the HTTP test facility, to be described below.
    5591
    56 
    5792== Pixel test tips ==
     93
     94=== Do you really need a pixel test? ===
     95
     96That should be your first question. Pixel tests are a burden on every port as any small chance in the engine could lead to your test needing a new pixel result. That does not mean that pixel tests are bad, they are invaluable for catching visual regressions but there are way to avoid dumping the pixel while checking the behavior you need!
     97
     98Let's take the example from the previous section and make it better.
     99
     100Even better test:
     101
     102{{{
     103<!DOCTYPE html>
     104<html>
     105<head>
     106<style>
     107    .block { color: green; }
     108</style>
     109<script>
     110if (window.layoutTestController)
     111   layoutTestController.dumpAsText();
     112
     113function log(message)
     114{
     115    var console = document.getElementById('console');
     116    console.appendChild(document.createTextNode(message);
     117    console.appendChild(document.createElement('br'));
     118}
     119
     120function checkBlockColor()
     121{
     122    var block = document.getElementsByClassName('block')[0];
     123    var color = document.defaultView.getComputedStyle(block, null).getPropertyValue('color');
     124    if (color === 'rgb(0, 255, 0)')
     125        log('PASSED');
     126    else
     127        log('FAILED: color was not green but ' + color);
     128}
     129window.addEventListener('load', checkBlockColor, true);
     130</script>
     131</head>
     132<body>
     133<p> Bug <a href="http://webkit.org/b/XXXX">XXXX</a>: WebKit does not apply class-selector</p>
     134<p> For this test to pass, you should see a green rectangle below and / or you should see PASS below. </p>
     135<p id="console"></p>
     136<p class="block"></p>
     137</html>
     138}}}
     139
     140As you can see, we did not need the pixels to get the information. The idea is that in this case, we were testing CSS and not the painting part of WebKit so querying the property directly would give us the result we need without having to dump the pixels.
     141
     142=== How to write portable pixel tests ===
    58143
    59144 * Do not use fonts other than those bundled with Mac OS X (usually there's no need to specify font families in a test). The only exception to this is the ''Ahem'' font.