Changes between Version 1 and Version 2 of TriagingTestFailures


Ignore:
Timestamp:
Nov 2, 2010 9:11:57 AM (10 years ago)
Author:
Adam Roben
Comment:

Slightly more prose-y

Legend:

Unmodified
Added
Removed
Modified
  • TriagingTestFailures

    v1 v2  
    11This guide will help you triage test failures on build.webkit.org and get the bots back to being green.
    22
    3  1. Find out what is failing. There are two main ways to do this:
    4    * Browse build.webkit.org (you should probably start with this one)
    5      1. Find recent builds that have failed
    6        * Windows:
    7          1. Go to http://build.webkit.org/buildslaves.
    8          1. Click on the name of a slave you're interested in to see a summary of its recent builds.
    9        * Other platforms:
    10          1. Go to http://build.webkit.org/builders.
    11          1. Click on the name of a builder you're interested in to see a summary of its recent builds.
    12        * The '''Info''' column will tell you if any tests failed for that build.
    13        * To see the test output for a particular build, click on the link in the '''Build #''' column, then on '''view results''', then on '''results.html'''
    14    * Use `webkit-patch`
    15      1. Run this command:
     3= Find out what is failing =
     4
     5There are two main ways to do this:
     6 * Browse build.webkit.org (you should probably start with this one)
     7   1. Find recent builds that have failed
     8     * Windows:
     9       1. Go to http://build.webkit.org/buildslaves.
     10       1. Click on the name of a slave you're interested in to see a summary of its recent builds.
     11     * Other platforms:
     12       1. Go to http://build.webkit.org/builders.
     13       1. Click on the name of a builder you're interested in to see a summary of its recent builds.
     14     * The '''Info''' column will tell you if any tests failed for that build.
     15     * To see the test output for a particular build, click on the link in the '''Build #''' column, then on '''view results''', then on '''results.html'''
     16 * Use `webkit-patch`
     17   1. Run this command:
    1618{{{
    1719webkit-patch failure-reason
    1820}}}
    19      1. When prompted, specify which builder you're interested in.
    20      1. Press Enter to continue. `webkit-patch` will look back through the recent builds for that builder until it has found when all current failures were introduced.
    21  1. Group failures by "root causes"
    22    1. If a multiple tests are all failing for the same reason, they should be grouped together into a single root cause
    23  1. File one or more bugs for each root cause. (If a test fails on multiple platforms and those platforms will need separate fixes, you should file one bug for each failing platform.)
    24    1. Go to http://webkit.org/new-bug
    25    1. Include in your report:
    26      * The name(s) of the failing test(s)
    27      * What platform(s) the failures occur on
    28      * When the failures began, if known
    29      * A link to the failing output
    30        * If a single test had incorrect output, link to the pretty diff.
    31        * If multiple tests had incorrect output, or if the failure is a crash or hang, link to results.html.
    32    1. Apply keywords
    33      * `LayoutTestFailure`
    34      * `Regression`, if the failure is due to a regression in WebKit
    35      * `PlatformOnly`, if the test only fails on one platform
    36  1. Get the bots green again
    37    * If the tests always fail with the same incorrect output, check in new results for the tests and include the bug URL in your ChangeLog
    38    * If the tests fail intermittently, or crash, or hang, add the tests to the appropriate Skipped files. Include a comment in the Skipped file with the bug URL and a brief description of how it fails (e.g., "Crashes", "Sometimes times out", etc.).
     21   1. When prompted, specify which builder you're interested in.
     22   1. Press Enter to continue. `webkit-patch` will look back through the recent builds for that builder until it has found when all current failures were introduced.
     23
     24= Group failures by "root cause" =
     25
     26If a multiple tests are all failing for the same reason, they should be grouped together into a single root cause.
     27
     28= File one or more bugs for each root cause =
     29
     30If a test fails on multiple platforms and those platforms will need separate fixes, you should file one bug for each failing platform.
     31
     32 1. Go to http://webkit.org/new-bug
     33 1. Include in your report:
     34   * The name(s) of the failing test(s)
     35   * What platform(s) the failures occur on
     36   * When the failures began, if known
     37   * A link to the failing output
     38     * If a single test had incorrect output, link to the pretty diff.
     39     * If multiple tests had incorrect output, or if the failure is a crash or hang, link to results.html.
     40 1. Apply keywords
     41   * `LayoutTestFailure`
     42   * `Regression`, if the failure is due to a regression in WebKit
     43   * `PlatformOnly`, if the test only fails on one platform
     44
     45= Get the bots green again =
     46
     47 * If the tests always fail with the same incorrect output, check in new results for the tests and include the bug URL in your ChangeLog
     48 * If the tests fail intermittently, or crash, or hang, add the tests to the appropriate Skipped files. Include a comment in the Skipped file with the bug URL and a brief description of how it fails (e.g., "Crashes", "Sometimes times out", etc.).