Changes between Version 2 and Version 3 of TriagingTestFailures


Ignore:
Timestamp:
Nov 2, 2010 9:25:18 AM (10 years ago)
Author:
Adam Roben
Comment:

Fleshed out a little more

Legend:

Unmodified
Added
Removed
Modified
  • TriagingTestFailures

    v2 v3  
    44
    55There are two main ways to do this:
     6
    67 * Browse build.webkit.org (you should probably start with this one)
    78   1. Find recent builds that have failed
     
    2223   1. Press Enter to continue. `webkit-patch` will look back through the recent builds for that builder until it has found when all current failures were introduced.
    2324
     25= Find out when each test started failing =
     26
     27You can either:
     28
     29 * Look back through old builds on build.webkit.org
     30 * Use the output from `webkit-patch failure-reason`
     31 * Use `svn log`/`git log` to find out when the test or its results were last changed
     32
     33= Try to figure out the reason each test is failing =
     34
     35(You probably won't be able to figure out exactly why every test is failing, but the more information you can get now, the better.)
     36
     37Look at the revision range where the failure was introduced. If you find that:
     38
     39 * The test and/or its expected output was modified
     40   * The test might need new results for the failing platform(s).
     41   * Are the test's results platform-specific (i.e., are they beneath `LayoutTests/platform/`)?
     42     * Yes: the failing platforms might just need new results checked in. You'll have to verify that the current output from those platforms is correct.
     43     * No: the failing platforms might have some missing functionality in WebKit or DumpRenderTree.
     44 * Related areas of WebKit were modified
     45   * Were the modifications platform-specific?
     46     * Yes: the failing platforms might need similar modifications made.
     47     * No: there might be some existing platform-specific code that is responsible for the different results.
     48
    2449= Group failures by "root cause" =
    2550
     
    3358 1. Include in your report:
    3459   * The name(s) of the failing test(s)
     60   * The root cause you determined, if any
    3561   * What platform(s) the failures occur on
    3662   * When the failures began, if known
     
    4571= Get the bots green again =
    4672
    47  * If the tests always fail with the same incorrect output, check in new results for the tests and include the bug URL in your ChangeLog
     73 * If you know what the root cause is, and know how to fix it (e.g., by making a change to WebKit or checking in new correct results), then fix it and close the bug!
     74 * If the tests always fail with the same incorrect output, check in new results for the tests and include the bug URL in your ChangeLog.
     75   * You should do this even if the test is "failing". By running the test against these "expected failure" results rather than skipping the test entirely, we can discover when new regressions are introduced.
    4876 * If the tests fail intermittently, or crash, or hang, add the tests to the appropriate Skipped files. Include a comment in the Skipped file with the bug URL and a brief description of how it fails (e.g., "Crashes", "Sometimes times out", etc.).