Changes between Version 28 and Version 29 of Writing Layout Tests for DumpRenderTree

Timestamp:

Oct 27, 2011, 5:31:07 PM (13 years ago)

Author:

jchaffraix@webkit.org

Comment:

how to write good test case (edit 4), clean up grammar errors and gardening

Writing Layout Tests for DumpRenderTree

@@ -37 +37 @@
 === Fast ===
 
-WebKit has several thousands layout tests that are run in parallel. If your layout test take a lot time, it will slow down everyone's testing. Also DumpRenderTree has a default timer of several seconds before it considers that a test timed out. '''This is true even if you are using waitUntilDone / notifyDone.'''
+WebKit has several thousands layout tests that are run in parallel. If your layout test takes a lot time, it will slow down everyone's testing. Also DumpRenderTree has a default timer of several seconds before it considers that a test timed out. '''This is true even if you are using waitUntilDone / notifyDone.'''
 
 Here are some advices to avoid having a slow test case:
@@ -47 +47 @@
 === Understandable & clear ===
 
-A good reading on how to write awesome test cases is the [http://www.w3.org/Style/CSS/Test/guidelines.html CSS2.1 Test Case Authoring Guidelines]. Especially read the ''section: 4. Writing ideal tests''
-
-Now in the context of WebKit, test should at least indicate:
- * which bug it belongs to. That means the bug number or the bugzilla URL (better as you can paste it in a browser) but also the bug title to see what is tested.
+A good reading on how to write awesome test cases is the [http://www.w3.org/Style/CSS/Test/guidelines.html CSS2.1 Test Case Authoring Guidelines]. Especially read the section: ''4. Writing ideal tests''
+
+Now in the context of WebKit, a test should at least indicate:
+ * which bug it belongs to. That means the bug number or the bugzilla URL (the latter is better as you can paste it in a browser) but also the bug title to see what is tested.
  * what is the condition for success or failure.
+
+The rule to understand what you need to include is to position yourself as someone who has to investigate why your test is failing. He need to kickly understand if the new baseline is a progression, a regression or just a small change unrelated to what is tested.
 
 Bad test:
@@ -84 +86 @@
 }}}
 
-This test is a huge improvement but we can actually go one step further! (see the next section about writing pixel tests on that)
+This test is a huge improvement over the previous one but we can actually go one step further! (see the next section about writing pixel tests on that)
 
 One last comment (FIXME: move it to a better section):
@@ -94 +96 @@
 === Do you really need a pixel test? ===
 
-That should be your first question. Pixel tests are a burden on every port as any small chance in the engine could lead to your test needing a new pixel result. That does not mean that pixel tests are bad, they are invaluable for catching visual regressions but there are way to avoid dumping the pixel while checking the behavior you need!
+That should be your first question. Pixel tests are a burden on every ports as any small change in the engine could lead to your test needing a new pixel result. That does not mean that pixel tests are bad, they are invaluable for catching visual regressions but there may be a way to avoid dumping the pixel while checking the behavior you need!
 
 Let's take the example from the previous section and make it better.
@@ -148 +150 @@
 ==== Fonts ====
 
-Fonts are very dependent on the platform and the source of a lot of differences in our tests. That's because not all platform share the same fonts and in this case, the fallback mechanism is pretty much defined per machine as it depends on the OS and the available fonts!
-
-The golden rule is that '''if you can avoid text, you should'''.
-
-This seems against the previous section about writing understandable test cases and it is. However there are a couple of ways to put the information you need while keeping the portability of the no-text mantra:
-
-  * put your information as comments into the HTML
-  * put your text outside of the dumped area: by default DumpRenderTree dumps a viewport of 600 * 800 pixels (FIXME: not 100% sure about those numbers). This means that anything outside this range would not be dumped in DRT but will appear in a browser opening the test.
+Fonts are very dependent on the platform and the source of a lot of differences in our tests' output. That's because not all platform share the same fonts and the fallback mechanism is pretty much defined per machine as it depends on the OS and the available fonts!
+
+The golden rule for writing pixel tests is that '''if you can avoid text, you should'''.
+
+This seems against the previous section about writing understandable test cases and it is. However there are ways to put the information you need while keeping the portability of the no-text mantra:
+
+  * put your information as comments in the HTML code (preferably somewhere prominent like the beginning)
+  * put your text outside of the dumped area: by default DumpRenderTree dumps a viewport of 600 * 800 pixels (FIXME: not 100% sure about the exact size). This means that anything outside this range would not be dumped in DRT but will appear in a browser opening the test if the resolution is sufficient.
   * add a 0 opacity to your text.
 
-'''Important note:''' For the 2 last points, the text will still be on the text render tree dump! You can however disable the text dump if you don't need it, see below.
-
-If you test '''really needs to use some fonts''', make sure you use the ''Ahem'' font as it is a very portable font. Make sure to specify the font-height too not to depend on the platform (FIXME: not sure if this is really needed but better safe than sorry).
+'''Important note:''' For the 2 last points, the text will still be in the text render tree dump! You can however disable the text dump if you don't need it to achieve maximum portability, see below.
+
+If you test '''really needs to use some fonts''', make sure you use the ''Ahem'' font as it is a very portable font. Make sure to specify the font-height too not to depend on the platform for that (FIXME: not sure if the font-size is really needed but better safe than sorry).
 
 ==== Native controls ====
@@ -166 +168 @@
 Native controls cover anything that is painted by the OS. In WebKit, this means buttons, scroll bars, media players, ...
 
-You can use the previous tips for text here to hide your button if you don't need it.
+You can use the previous tips for text here to hide your button if you don't need to show it.
 
 For scroll bars, you just have to use the following rule in your CSS to disable them:
@@ -183 +185 @@
 }}}
 
-This is '''not''' a typo. It is an extremely confusing syntax but this disables the text dump (you dump an empty text file).
+This is '''not''' a typo. It is an extremely confusing syntax but this disables the text dump (you basically dump an empty text file).
 
 === How to disable the pixel dumps but keep the render tree dump ===