Changes between Version 28 and Version 29 of Writing Layout Tests for DumpRenderTree


Ignore:
Timestamp:
Oct 27, 2011 5:31:07 PM (12 years ago)
Author:
jchaffraix@webkit.org
Comment:

how to write good test case (edit 4), clean up grammar errors and gardening

Legend:

Unmodified
Added
Removed
Modified
  • Writing Layout Tests for DumpRenderTree

    v28 v29  
    3737=== Fast ===
    3838
    39 WebKit has several thousands layout tests that are run in parallel. If your layout test take a lot time, it will slow down everyone's testing. Also DumpRenderTree has a default timer of several seconds before it considers that a test timed out. '''This is true even if you are using waitUntilDone / notifyDone.'''
     39WebKit has several thousands layout tests that are run in parallel. If your layout test takes a lot time, it will slow down everyone's testing. Also DumpRenderTree has a default timer of several seconds before it considers that a test timed out. '''This is true even if you are using waitUntilDone / notifyDone.'''
    4040
    4141Here are some advices to avoid having a slow test case:
     
    4747=== Understandable & clear ===
    4848
    49 A good reading on how to write awesome test cases is the [http://www.w3.org/Style/CSS/Test/guidelines.html CSS2.1 Test Case Authoring Guidelines]. Especially read the ''section: 4. Writing ideal tests''
    50 
    51 Now in the context of WebKit, test should at least indicate:
    52  * which bug it belongs to. That means the bug number or the bugzilla URL (better as you can paste it in a browser) but also the bug title to see what is tested.
     49A good reading on how to write awesome test cases is the [http://www.w3.org/Style/CSS/Test/guidelines.html CSS2.1 Test Case Authoring Guidelines]. Especially read the section: ''4. Writing ideal tests''
     50
     51Now in the context of WebKit, a test should at least indicate:
     52 * which bug it belongs to. That means the bug number or the bugzilla URL (the latter is better as you can paste it in a browser) but also the bug title to see what is tested.
    5353 * what is the condition for success or failure.
     54
     55The rule to understand what you need to include is to position yourself as someone who has to investigate why your test is failing. He need to kickly understand if the new baseline is a progression, a regression or just a small change unrelated to what is tested.
    5456
    5557Bad test:
     
    8486}}}
    8587
    86 This test is a huge improvement but we can actually go one step further! (see the next section about writing pixel tests on that)
     88This test is a huge improvement over the previous one but we can actually go one step further! (see the next section about writing pixel tests on that)
    8789
    8890One last comment (FIXME: move it to a better section):
     
    9496=== Do you really need a pixel test? ===
    9597
    96 That should be your first question. Pixel tests are a burden on every port as any small chance in the engine could lead to your test needing a new pixel result. That does not mean that pixel tests are bad, they are invaluable for catching visual regressions but there are way to avoid dumping the pixel while checking the behavior you need!
     98That should be your first question. Pixel tests are a burden on every ports as any small change in the engine could lead to your test needing a new pixel result. That does not mean that pixel tests are bad, they are invaluable for catching visual regressions but there may be a way to avoid dumping the pixel while checking the behavior you need!
    9799
    98100Let's take the example from the previous section and make it better.
     
    148150==== Fonts ====
    149151
    150 Fonts are very dependent on the platform and the source of a lot of differences in our tests. That's because not all platform share the same fonts and in this case, the fallback mechanism is pretty much defined per machine as it depends on the OS and the available fonts!
    151 
    152 The golden rule is that '''if you can avoid text, you should'''.
    153 
    154 This seems against the previous section about writing understandable test cases and it is. However there are a couple of ways to put the information you need while keeping the portability of the no-text mantra:
    155 
    156   * put your information as comments into the HTML
    157   * put your text outside of the dumped area: by default DumpRenderTree dumps a viewport of 600 * 800 pixels (FIXME: not 100% sure about those numbers). This means that anything outside this range would not be dumped in DRT but will appear in a browser opening the test.
     152Fonts are very dependent on the platform and the source of a lot of differences in our tests' output. That's because not all platform share the same fonts and the fallback mechanism is pretty much defined per machine as it depends on the OS and the available fonts!
     153
     154The golden rule for writing pixel tests is that '''if you can avoid text, you should'''.
     155
     156This seems against the previous section about writing understandable test cases and it is. However there are ways to put the information you need while keeping the portability of the no-text mantra:
     157
     158  * put your information as comments in the HTML code (preferably somewhere prominent like the beginning)
     159  * put your text outside of the dumped area: by default DumpRenderTree dumps a viewport of 600 * 800 pixels (FIXME: not 100% sure about the exact size). This means that anything outside this range would not be dumped in DRT but will appear in a browser opening the test if the resolution is sufficient.
    158160  * add a 0 opacity to your text.
    159161
    160 '''Important note:''' For the 2 last points, the text will still be on the text render tree dump! You can however disable the text dump if you don't need it, see below.
    161 
    162 If you test '''really needs to use some fonts''', make sure you use the ''Ahem'' font as it is a very portable font. Make sure to specify the font-height too not to depend on the platform (FIXME: not sure if this is really needed but better safe than sorry).
     162'''Important note:''' For the 2 last points, the text will still be in the text render tree dump! You can however disable the text dump if you don't need it to achieve maximum portability, see below.
     163
     164If you test '''really needs to use some fonts''', make sure you use the ''Ahem'' font as it is a very portable font. Make sure to specify the font-height too not to depend on the platform for that (FIXME: not sure if the font-size is really needed but better safe than sorry).
    163165
    164166==== Native controls ====
     
    166168Native controls cover anything that is painted by the OS. In WebKit, this means buttons, scroll bars, media players, ...
    167169
    168 You can use the previous tips for text here to hide your button if you don't need it.
     170You can use the previous tips for text here to hide your button if you don't need to show it.
    169171
    170172For scroll bars, you just have to use the following rule in your CSS to disable them:
     
    183185}}}
    184186
    185 This is '''not''' a typo. It is an extremely confusing syntax but this disables the text dump (you dump an empty text file).
     187This is '''not''' a typo. It is an extremely confusing syntax but this disables the text dump (you basically dump an empty text file).
    186188
    187189=== How to disable the pixel dumps but keep the render tree dump ===