wiki:WebInspectorTests

Version 5 (modified by BJ Burg, 9 years ago) (diff)

More updates to Web Inspector test documentation.

This page describes how various parts of the Web Inspector are tested.


Types of Tests

There are several types of inspector tests:

  • Protocol Tests exercise the inspector backend independently of any particular frontend.
  • Frontend Tests exercise the models and controllers underlying the Web Inspector user interface.
  • Manual Tests exercise the user interface in ways that are difficult to automate or require infrastructure that doesn't exist yet.
  • Library Tests exercise subsystems such as pretty printing or the protocol generator in isolation from a running Web Inspector instance.

To date, the Web Inspector has no automated tests that exercise the user interface. In practice, the Inspector UI changes frequently, so such tests tend to be brittle, and have traditionally not been worth the trouble of maintaining.

How Tests Execute

Each test is an HTML file in a per-domain directory within LayoutTests/inspector/. Some tests may additionally include external files, which are included in special resources/ directories that are automatically excluded from the test search path. All tests must decide which test harness to use by including either protocol-test.js or inspector-test.js.

When the test page finishes loading, it calls the runTest() method provided, which signals the test harness to set up a test inspector instance that inspects the test page. Each test page defines a special test() method, which is automatically marshalled and injected into the Inspector instance's context. Most scripts execute in the inspector's JavaScript context, and occasionally evaluate some code in the test page's context to log test results and to trigger specific inspectable behaviors.

Protocol Tests

Protocol tests are appropriate for testing inspector features that require the use of a few commands and events between the backend and frontend, and do not require the inspected page to be reloaded. Protocol tests are fairly low-level, and exercise the Inspector backend independent of a particular frontend and its controllers and models. In other words, you cannot test Managers or other classes in the WebInspector namespace using a protocol test.

The protocol-test.js stub creates a dummy inspector frontend by using window.open() from the test page, and establishes bidirectional communication with the child inspector page using window.postMessage and a message event handler. The "inspector" page that is loaded into the iframe is located at Source/WebInspectorUI/Base/TestStub.html. The code that runs inside the Inspector frame (i.e., code within the test() method) has access to the protocol test harness, whose methods are prefixed with ProtocolTest. Protocol-specific methods for sending commands and awaiting on events are available in the InspectorProtocol namespace.

Frontend Tests

Frontend tests exercise the functionality of models and controllers specific to WebInspectorUI (the user interface included in WebKit trunk). They use a real, headless Web Inspector frontend that persists across navigations of the inspected (test) page.

The inspector-test.js stub creates a real (for WebKit2, separate process) inspector frontend. Instead of the normal Web Inspector base page (Main.html), it loads a smaller version (Test.html) which does not load Views and other code not used by tests. The code that runs inside the Inspector (i.e., code within the test() method) has access to the frontend test harness, whose methods are prefixed with InspectorTest. Like ordinary Web Inspector code, injected inspector code has full access to models and controllers in the WebInspector namespace. (However, as noted above, not all files are loaded in the test version of the Inspector. You may need to add additional files to Test.html when testing new code or adding inter-class dependencies.)

Manual Tests

A manual test requires manual interaction with a test page to exercise specific behaviors. The test page should describe the necessary interaction steps, and the expected output/behavior.

Library Tests

TODO


How to Write Tests

TODO

How to Debug Tests

In general, the strategies for debugging the Web Inspector and debugging WebCore/WebKit2 apply the same to debugging inspector tests. Sometimes, tests can be more difficult to debug because the test harness' marshalling code can be broken by incorrectly written tests or bugs in the test harness. The test stubs provide several flags that enable extra or more reliable logging for debug purposes. Flags can be set in the corresponding Test/TestStub.html file for all test runs, or at the top of a test() method to only affect one test.

For protocol tests:

// Debug logging is synchronous on the test page.
ProtocolTest.forceDebugLogging = false;

// Tee all TestHarness commands to stderr from within the Inspector.
ProtocolTest.dumpActivityToSystemConsole = false;

// Best used in combination with dumpActivityToSystemConsole.
ProtocolTest.dumpInspectorProtocolMessages = false;

For frontend tests:

// Debug logging is synchronous on the test page.
InspectorTest.forceDebugLogging = false;

// Tee all TestHarness commands to stderr from within the Inspector.
InspectorTest.dumpActivityToSystemConsole = false;

// Best used in combination with dumpActivityToSystemConsole.
    InspectorBackend.dumpInspectorProtocolMessages = false;

Attaching a Debugger to Tests with DumpRenderTree (WebKit1)

$ DYLD_FRAMEWORK_PATH=WebKitBuild/Debug lldb -- WebKitBuild/Debug/DumpRenderTree
(lldb) run LayoutTests/inspector/dom/focus.html

To run DumpRenderTree in "server mode", which the run-webkit-tests uses to run multiple tests without restarting the process, make a file called "tests-to-run.txt" with one test per line, and launch this way instead:

(lldb) process launch -i tests-to-run.txt "-"

Attaching a Debugger to Tests with WebKitTestRunner (WebKit2)

TODO