Changes between Version 5 and Version 6 of WebInspectorTests


Ignore:
Timestamp:
Aug 31, 2015 7:53:24 AM (7 years ago)
Author:
BJ Burg
Comment:

Add more details about testing Web Inspector using modern classes.

Legend:

Unmodified
Added
Removed
Modified
  • WebInspectorTests

    v5 v6  
    1616== How Tests Execute
    1717
    18  Each test is an HTML file in a per-domain directory within `LayoutTests/inspector/`. Some tests may additionally include external files, which are included in special `resources/` directories that are automatically excluded from the test search path. All tests must decide which test harness to use by including either `protocol-test.js` or `inspector-test.js`.
     18 Each test is an HTML file in a per-domain directory within `LayoutTests/inspector/` or `LayoutTests/http/tests/inspector/` (for tests that load files over HTTP). Some tests may additionally include external files, which are included in special `resources/` directories that are automatically excluded from the test search path. All tests must decide which test harness to use by including either `protocol-test.js` or `inspector-test.js`.
    1919
    2020 When the test page finishes loading, it calls the `runTest()` method provided, which signals the test harness to set up a test inspector instance that inspects the test page. Each test page defines a special `test()` method, which is automatically marshalled and injected into the Inspector instance's context. Most scripts execute in the inspector's JavaScript context, and occasionally evaluate some code in the test page's context to log test results and to trigger specific inspectable behaviors.
     
    3030Frontend tests exercise the functionality of models and controllers specific to WebInspectorUI (the user interface included in WebKit trunk). They use a real, headless Web Inspector frontend that persists across navigations of the inspected (test) page.
    3131
    32 The inspector-test.js stub creates a real (for WebKit2, separate process) inspector frontend. Instead of the normal Web Inspector base page (`Main.html`), it loads a smaller version (`Test.html`) which does not load Views and other code not used by tests. The code that runs inside the Inspector (i.e., code within the test() method) has access to the frontend test harness, whose methods are prefixed with `InspectorTest`. Like ordinary Web Inspector code, injected inspector code has full access to models and controllers in the `WebInspector` namespace. (However, as noted above, not all files are loaded in the test version of the Inspector. You may need to add additional files to `Test.html` when testing new code or adding inter-class dependencies.)
     32The `inspector-test.js` stub creates a real (for WebKit2, separate process) inspector frontend. Instead of the normal Web Inspector base page (`Main.html`), it loads a smaller version (`Test.html`) which does not load Views and other code not used by tests. The code that runs inside the Inspector (i.e., code within the test() method) has access to the frontend test harness, whose methods are prefixed with `InspectorTest`. Like ordinary Web Inspector code, injected inspector code has full access to models and controllers in the `WebInspector` namespace. (However, as noted above, not all files are loaded in the test version of the Inspector. You may need to add additional files to `Test.html` when testing new code or adding inter-class dependencies.)
    3333
    3434== Manual Tests
     
    3838== Library Tests
    3939
    40 TODO
     40 * Pretty printing tests: these cover behavior of our pretty-printing code, and should be converted into layout tests.
     41 * Protocol generator tests: these test inputs to the generator are designed to detect changes in the protocol generator's code generation routines. They do not contain any assertions. To run the tests, execute `Tools/Scripts/run-inspector-generator-tests`.
     42 * TODO: do we have other ad-hoc tests?
    4143
    4244-----
     
    4446== How to Write Tests
    4547
    46 TODO
     48The properties that we strive for when writing inspector tests are:
     49
     50 * '''consistent''': a test should be consistent between runs, and not sporadically fail or time out.
     51 * '''robust''': a test should be robust to underlying changes in data structures or other minor changes to the code being exercised. It should not require frequent adjustments.
     52 * '''high coverage''': to uncover bugs and unaddressed situations, a test should exercise as many normal and exceptional code paths as possible.
     53 * '''self-documenting''': a test should act as executable documentation for the expected and unexpected use cases or behaviors of the code being exercised.
     54
     55With these properties in mind, here are a few hints for writing good tests:
     56
     57 * Use good names in the test filename (`inspector/domain/description-of-test.html`), test suite name (`Domain.descriptionOfTest`), and in each test case's name (`TestSomethingInteresting`) and description (`"This test ensures something interesting"`).
     58 * Use `AsyncTestSuite` (documented below) to avoid common pitfalls involved in testing asynchronous code, such as when testing the result of a command sent to the inspector backend.
     59 * Use assertions to test invariants, pre-conditions, and post-conditions. Assertions should not need to be changed unless the code under test changes in significant ways.
     60 * Assertions should always document the expected condition in the message, usually using obligatory language such as "should be", "should contain", "should not", etc. For example, the following shows a good and bad assertion message:
     61
     62{{{
     63InspectorTest.expectThat(fontFamilyNames.length >= 5, "Has at least 5 fonts"); // BAD!
     64InspectorTest.expectThat(fontFamilyNames.length >= 5, "Family names list should contain at least 5 fonts"); // GOOD!
     65}}}
     66
     67* Use `expectThat` instead of `assert` whenever possible. The former will always add output to the test, prefixing the condition with `PASS:` or `FAIL:`; the latter only produces output when the condition evaluates to false. Why should the default be to always log? For someone trying to understand how a test works or debug a failing test, the extra output is very helpful in understanding control flow.
     68* Log runtime states sparingly, and only those that may help to diagnose a failing test. Logging runtime states can make tests more self-documenting at the expense of reducing robustness. For example, if a test logs source code locations, these could change (and cause the test to fail) if text is added to or removed from relevant file. It is better to assert actual output against known-good outputs. Don't dump runtime state that is machine-dependent.
     69
     70== Important Test Fixtures
     71
     72Common to both protocol tests and frontend tests are the `TestHarness` and `TestSuite` classes. TestHarness and its subclasses (bound to the globals ProtocolTest or InspectorTest) provide basic mechanisms for logging, asserting, and starting or stopping the test. Protocol and frontend tests each have their own subclass of `TestHarness` which contains methods specific to one environment.
     73
     74`TestSuite` and its subclasses `AsyncTestSuite` and `SyncTestSuite` help us to write robust, well-documented, and fast tests. All new tests should use these classes. Each test file consists of one (or more) test suite(s). A suite consists of multiple test cases which execute sequentially in the order that they are added. If a test case fails, later test cases are skipped to avoid spurious failures caused by dependencies between test cases. Test cases are added to the suite imperatively, and then executed using the `runTestCases()` or `runTestCasesAndFinish()` methods. This allows for programmatic construction of test suites that exercise code using many different inputs.
     75
     76A `SyncTestSuite` executes its test cases synchronously, one after another in a loop. It is usually used for unit tests that do not require communication with the backend. Each test case provides a test method which takes no arguments and returns `true` or `false` to indicate test success or failure, respectively.
     77
     78An `AsyncTestSuite` executes its test cases asynchronously, one after another, by chaining together promises created for each test. Each test case provides a test method which takes two callback arguments: `resolve` and `reject`. At runtime, each test method is turned into a Promise; like a Promise, the test signals success by calling `resolve()`, and signals failure by calling `reject()` or throwing an `Error` instance.
    4779
    4880== How to Debug Tests
     
    92124
    93125TODO
     126
     127= Example Test (uses inspector-test.js, AsyncTestSuite)
     128
     129{{{
     130<!DOCTYPE html>
     131<html>
     132<head>
     133<script src="../../http/tests/inspector/resources/inspector-test.js"></script>
     134<script>
     135function test()
     136{
     137    let addedStyleSheet;
     138    let mainFrame = WebInspector.frameResourceManager.mainFrame;
     139
     140    let suite = InspectorTest.createAsyncSuite("CSS.createStyleSheet");
     141
     142    suite.addTestCase({
     143        name: "CheckNoStyleSheets",
     144        description: "Ensure there are no stylesheets.",
     145        test: (resolve, reject) => {
     146            InspectorTest.expectThat(WebInspector.cssStyleManager.styleSheets.length === 0, "Should be no stylesheets.");
     147            resolve();
     148        }
     149    });
     150
     151    for (let i = 1; i <= 3; ++i) {
     152        suite.addTestCase({
     153            name: "CreateInspectorStyleSheetCall" + i,
     154            description: "Should create a new inspector stylesheet.",
     155            test: (resolve, reject) => {
     156                CSSAgent.createStyleSheet(mainFrame.id);
     157                WebInspector.cssStyleManager.singleFireEventListener(WebInspector.CSSStyleManager.Event.StyleSheetAdded, function(event) {
     158                    InspectorTest.expectThat(WebInspector.cssStyleManager.styleSheets.length === i, "Should increase the list of stylesheets.");
     159                    InspectorTest.expectThat(event.data.styleSheet.origin === WebInspector.CSSStyleSheet.Type.Inspector, "Added StyleSheet origin should be 'inspector'.");
     160                    InspectorTest.expectThat(event.data.styleSheet.isInspectorStyleSheet(), "StyleSheet.isInspectorStyleSheet() should be true.");
     161                    InspectorTest.expectThat(event.data.styleSheet.parentFrame === mainFrame, "Added StyleSheet frame should be the main frame.");
     162                    if (addedStyleSheet)
     163                        InspectorTest.expectThat(event.data.styleSheet !== addedStyleSheet, "Added StyleSheet should be different from the last added stylesheet.");
     164                    addedStyleSheet = event.data.styleSheet;
     165                    resolve();
     166                });
     167            }
     168        });
     169    }
     170
     171    WebInspector.cssStyleManager.singleFireEventListener(WebInspector.CSSStyleManager.Event.StyleSheetRemoved, function(event) {
     172        InspectorTest.assert(false, "Should not be removing any StyleSheets in this test.");
     173    });
     174
     175    suite.runTestCasesAndFinish();
     176}
     177</script>
     178</head>
     179<body onload="runTest()">
     180    <p>Test CSS.createStyleSheet.</p>
     181</body>
     182</html>
     183}}}