Writing good test cases

General tips

A good test case should have the following properties:

• portable
• fast
• understandable & clear

Portable

A test should work in DumpRenderTree (of course!) and in a WebKit-base browser. This means that anything that is specific to DumpRenderTree should not make the test fail if it is not available.

Bad test:

<script>
layoutTestController.dumpAsText();
</script>

This will throw an exception in any WebKit-based browser.

Good test:

<script>
if (window.layoutTestController)
    layoutTestController.dumpAsText()
</script>

Note: You get more points if your test works in ANY browsers.

fast

WebKit has several thousands layout tests that are run in parallel. If your layout test take a lot time, it will slow down everyone's testing. Also DumpRenderTree has a default timer of several seconds before it considers that a test timed out. This is true even if you are using waitUntilDone / notifyDone.

Here are some advices to avoid having a slow test case:

• Do not make unnecessary use of window.setTimeout and similar functions which create a lower bound on the time it takes the test to complete. For separating events in time, you can use eventSender.leapForward; for waiting on sub-resources, you can use load events (e.g. iframe.onload = doNextStep;)

• Do not make unnecessary use of external resources: use inline JavaScript and style elements instead of links to external stylesheets. You can also use data: URLs sometimes for things like frames' src attribute. The exceptions are stylesheets and JavaScript libraries that are shared by multiple tests, and cases that test the loading of external resources. (There are various data: url generators on the net.)

Understandable & clear

FIXME: Should have some tips here about how to make a test which is easy to know when it succeeds, things like a 100x100 green block, just printing SUCCESS or FAILURE, and using dumpAsText().

One last comment (FIXME: move it to a better section):

• Tests should not access the Internet. Avoid http: URLs in src and href attributes, in CSS properties and in XMLHttpRequest. Testing WebKit's network layer should be done using the HTTP test facility, to be described below.

Pixel test tips

• Do not use fonts other than those bundled with Mac OS X (usually there's no need to specify font families in a test). The only exception to this is the Ahem font.

Writing JavaScript-based DOM-only Test Cases

When writing test cases that only test the DOM it is still preferred to use an .html file since it only requires one test file instead of two.

When writing these tests it is often useful to include ​LayoutTests/fast/js/resources/js-test-pre.js and ​LayoutTests/fast/js/resources/js-test-post.js. To get these to work you need to have an element with id description and another element with the id console.

To see what sort of special js functions are exposed to js-only tests, see ​LayoutTests/fast/js/resources/js-test-pre.js

DumpRenderTree JavaScript Environment

DumpRenderTree exposes a number of additional JavaScript objects in the testing environment.

These can be used to perform additional debugging-related tasks.

window.layoutTestController

dumpAsText()

Call this method to make your test output plain text instead of a render tree. This is useful if your test prints messages rather than testing fancy layout. For an example of how to print to a console in a test, check out ​LayoutTests/fast/dom/Element/attribute-uppercase.html.

waitUntilDone() and notifyDone()

By default, DumpRenderTree dumps each test file immediately after the document has loaded and the load event handlers have executed. If your test needs to do further processing after loading -- for example, waiting for a timer to fire -- call layoutTestController.waitUntilDone() to tell DumpRenderTree to delay its dump, and then call notifyDone when your results are ready.

overridePreference(key, value)

Changes a preference with name key (for example "WebKitEnableCaretBrowsing") with value (for example "1") for the duration of the test. The preference is reset when the test ends.

setCanOpenWindows()

If your layout test needs to open pop-up windows, call this method before it does.

clearBackForwardList()

Clears the back/forward list (i.e. history).

window.eventSender

mouseMoveTo(x, y)

Used to change the current mouse position.

leapForward(ms)

Jumps the current event time forward by a specified number of miliseconds.

mouseDown([buttonNumber [, modifiers]])

Sends a mouseDown event to the WebView at the current mouse position.

buttonNumber; 0:left button, 1:middle button, 2:right button.

modifiers; See keyDown().

mouseUp([buttonNumber [, modifiers]])

Sends a mouseUp event to the WebView at the current mouse position.

keyDown(character [, modifiers]])

Sends a keyDown event to the WebView.

modifiers: An array of strings. A string should be a modifier key name in the followings:

• "ctrlKey"
• "shiftKey"
• "altKey"
• "metaKey" (Command key in Mac)
• "addSelectionKey" (equivalent to metaKey in Mac, ctrlKey in Windows)
• "rangeSelectionKey" (equivalent to shiftKey in Mac and Windows)

enableDOMUIEventLogging

fireKeyboardEventsToElement

setDragMode

window.GCController

collect()

Performs JavaScript garbage collection.

collectOnAlternateThread(wait)

Performs JavaScript garbage collection on an alternate thread. The wait argument specifies whether script execution waits for garbage collection to finish.

window.textInputController

Needs to be filled in.

insertText

doCommand

setMarkedText

substringFromRange

attributedSubstringFromRange

firstRectForCharacterRange

characterIndexForPoint

makeAttributedString

window.appleScriptController

doJavaScript()

Needs to be filled in.

window.navigationController

The navigation controller is currently broken. ​http://bugs.webkit.org/show_bug.cgi?id=11042

evalAfterBackForwardNavigation(script [, destination])

To test a bug having to do with the loader or the back/forward cache, call this method to run a script after executing a back/forward navigation. The first argument is the script to run, and the second argument is the page to load during the navigation. The second argument is optional. It defaults to about:blank.

window.internals

createShadowContentElement(Document)

Creates a <content> element for use in shadow DOM. These elements can not be created from JavaScript, hence this constructor in the test harness.

elementRenderTreeAsText(Element)

Gets and returns the element’s renderer’s description of its tree as a String.

ensureShadowRoot(Element)

Given a host element, returns its shadow root node. If the element doesn’t have a shadow root one is created and attached to the element. If you want to just retrieve a shadow root without creating one, use shadowRoot. ensureShadowRoot only inspects DOM shadows; not SVG shadows.

isPreloaded(Document, String url)

Gets whether the specified document’s cached resource loader has the specified URL preloaded.

removeShadowRoot(Element)

Given a host element, removes its shadow root if it has one. removeShadowRoot only inspects DOM shadows; not SVG shadows.

shadowPseudoId(Element)

Gets the specified element’s CSS pseudo-id for styling when in shadow DOM.

shadowRoot(Element)

Given a host element gets its shadow root, if it has one; otherwise shadowRoot returns null.

Writing tests which require network access

run-webkit-tests (the script which runs DumpRenderTree) also launches a local Apache daemon (httpd) to allow real local-only network based testing (for incremental loads, XMLHttpRequest, etc.) ap needs to document how best to use this.