| | 1 | This guide will help you triage test failures on build.webkit.org and get the bots back to being green. |
| | 2 | |
| | 3 | 1. Find out what is failing. There are two main ways to do this: |
| | 4 | * Browse build.webkit.org (you should probably start with this one) |
| | 5 | 1. Find recent builds that have failed |
| | 6 | * Windows: |
| | 7 | 1. Go to http://build.webkit.org/buildslaves. |
| | 8 | 1. Click on the name of a slave you're interested in to see a summary of its recent builds. |
| | 9 | * Other platforms: |
| | 10 | 1. Go to http://build.webkit.org/builders. |
| | 11 | 1. Click on the name of a builder you're interested in to see a summary of its recent builds. |
| | 12 | * The '''Info''' column will tell you if any tests failed for that build. |
| | 13 | * To see the test output for a particular build, click on the link in the '''Build #''' column, then on '''view results''', then on '''results.html''' |
| | 14 | * Use `webkit-patch` |
| | 15 | 1. Run this command: |
| | 16 | {{{ |
| | 17 | webkit-patch failure-reason |
| | 18 | }}} |
| | 19 | 1. When prompted, specify which builder you're interested in. |
| | 20 | 1. Press Enter to continue. `webkit-patch` will look back through the recent builds for that builder until it has found when all current failures were introduced. |
| | 21 | 1. Group failures by "root causes" |
| | 22 | 1. If a multiple tests are all failing for the same reason, they should be grouped together into a single root cause |
| | 23 | 1. File one or more bugs for each root cause. (If a test fails on multiple platforms and those platforms will need separate fixes, you should file one bug for each failing platform.) |
| | 24 | 1. Go to http://webkit.org/new-bug |
| | 25 | 1. Include in your report: |
| | 26 | * The name(s) of the failing test(s) |
| | 27 | * What platform(s) the failures occur on |
| | 28 | * When the failures began, if known |
| | 29 | * A link to the failing output |
| | 30 | * If a single test had incorrect output, link to the pretty diff. |
| | 31 | * If multiple tests had incorrect output, or if the failure is a crash or hang, link to results.html. |
| | 32 | 1. Apply keywords |
| | 33 | * `LayoutTestFailure` |
| | 34 | * `Regression`, if the failure is due to a regression in WebKit |
| | 35 | * `PlatformOnly`, if the test only fails on one platform |
| | 36 | 1. Get the bots green again |
| | 37 | * If the tests always fail with the same incorrect output, check in new results for the tests and include the bug URL in your ChangeLog |
| | 38 | * If the tests fail intermittently, or crash, or hang, add the tests to the appropriate Skipped files. Include a comment in the Skipped file with the bug URL and a brief description of how it fails (e.g., "Crashes", "Sometimes times out", etc.). |
