This guide will help you triage test failures on build.webkit.org and get the bots back to being green. = Find out what is failing = There are two main ways to do this: * Browse build.webkit.org (you should probably start with this one) 1. Find recent builds that have failed * Windows: 1. Go to http://build.webkit.org/buildslaves. 1. Click on the name of a slave you're interested in to see a summary of its recent builds. * Other platforms: 1. Go to http://build.webkit.org/builders. 1. Click on the name of a builder you're interested in to see a summary of its recent builds. * The '''Info''' column will tell you if any tests failed for that build. * To see the test output for a particular build, click on the link in the '''Build #''' column, then on '''view results''', then on '''results.html''' * Use `webkit-patch` 1. Run this command: {{{ webkit-patch failure-reason }}} 1. When prompted, specify which builder you're interested in. 1. Press Enter to continue. `webkit-patch` will look back through the recent builds for that builder until it has found when all current failures were introduced. = Group failures by "root cause" = If a multiple tests are all failing for the same reason, they should be grouped together into a single root cause. = File one or more bugs for each root cause = If a test fails on multiple platforms and those platforms will need separate fixes, you should file one bug for each failing platform. 1. Go to http://webkit.org/new-bug 1. Include in your report: * The name(s) of the failing test(s) * What platform(s) the failures occur on * When the failures began, if known * A link to the failing output * If a single test had incorrect output, link to the pretty diff. * If multiple tests had incorrect output, or if the failure is a crash or hang, link to results.html. 1. Apply keywords * `LayoutTestFailure` * `Regression`, if the failure is due to a regression in WebKit * `PlatformOnly`, if the test only fails on one platform = Get the bots green again = * If the tests always fail with the same incorrect output, check in new results for the tests and include the bug URL in your ChangeLog * If the tests fail intermittently, or crash, or hang, add the tests to the appropriate Skipped files. Include a comment in the Skipped file with the bug URL and a brief description of how it fails (e.g., "Crashes", "Sometimes times out", etc.).