Changeset 90571 in webkit

Timestamp:

Jul 7, 2011 10:18:09 AM (13 years ago)

Author:

Adam Roben

Message:

Completely ignore too-many-failures builds in TestFailures in most circumstances

Because a semi-arbitrary subset of tests are run in a too-many-failures build, we can't
really use them to perform regression analysis. The only time we want to pay attention to
too-many-failures builds is when we're trying to explain when the current bout of
too-many-failures started.

Fixes <​http://webkit.org/b/64106> TestFailures page sometimes claims a test started failing
in a build that didn't even run it (because it exited early due to too many failues)

Reviewed by David Kilzer.

• BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/LayoutTestHistoryAnalyzer.js:

(LayoutTestHistoryAnalyzer.prototype._incorporateBuildHistory): Removed old, broken
too-many-failures handling that would cause us to blame builds that didn't even run a given
test for breaking it. Instead, skip over all too-many-failures builds unless the most recent
build was itself a too-many-failures build.

Location:

trunk/Tools

Files:

• BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/LayoutTestHistoryAnalyzer.js (modified) (2 diffs)
• ChangeLog (modified) (1 diff)

trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/LayoutTestHistoryAnalyzer.js

@@ -119 +119 @@
         var self = this;
         self._loader.start(nextBuildName, function(tests, tooManyFailures) {
+            if (tooManyFailures) {
+                var firstBuildName = Object.keys(self._history)[0];
+                // If the first (i.e., current or most recent) build exited early due to too many
+                // failures, we want to process other too-many-failures builds normally to try to
+                // figure out when the too-many-failures started occurring. If the first/current
+                // build did not exit due to too many failures, then too-many-failures builds will
+                // only confuse our analysis (since they run a semi-arbitrary subset of tests), so
+                // we should just skip them entirely.
+                if (firstBuildName && !self._history[firstBuildName].tooManyFailures) {
+                    callback(true);
+                    return;
+                }
+            }
+
             ++self._testRunsSinceLastInterestingChange;
 
@@ -151 +165 @@
             }
 
-            if (tooManyFailures && previousHistoryItem) {
-                // Not all tests were run due to too many failures. Just assume that all the tests
-                // that failed in the last build would still have failed in this build had they been
-                // run.
-                for (var testName in previousHistoryItem.tests) {
-                    historyItem.tests[testName] = previousHistoryItem.tests[testName];
-                    delete previousHistoryItem.tests[testName];
-                }
-            }
-
             var previousUnexplainedFailuresCount = previousBuildName ? Object.keys(self._history[previousBuildName].tests).length : 0;
             var unexplainedFailuresCount = Object.keys(self._history[nextBuildName].tests).length;

trunk/Tools/ChangeLog

@@ +1 @@
+2011-07-07  Adam Roben  <aroben@apple.com>
+
+        Completely ignore too-many-failures builds in TestFailures in most circumstances
+
+        Because a semi-arbitrary subset of tests are run in a too-many-failures build, we can't
+        really use them to perform regression analysis. The only time we want to pay attention to
+        too-many-failures builds is when we're trying to explain when the current bout of
+        too-many-failures started.
+
+        Fixes <http://webkit.org/b/64106> TestFailures page sometimes claims a test started failing
+        in a build that didn't even run it (because it exited early due to too many failues)
+
+        Reviewed by David Kilzer.
+
+        * BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/LayoutTestHistoryAnalyzer.js:
+        (LayoutTestHistoryAnalyzer.prototype._incorporateBuildHistory): Removed old, broken
+        too-many-failures handling that would cause us to blame builds that didn't even run a given
+        test for breaking it. Instead, skip over all too-many-failures builds unless the most recent
+        build was itself a too-many-failures build.
+
 2011-07-07  Adam Roben  <aroben@apple.com>