Changeset 90054 in webkit


Ignore:
Timestamp:
Jun 29, 2011 3:24:48 PM (13 years ago)
Author:
Adam Roben
Message:

Teach TestFailures to detect possibly flaky tests and list them separately

Fixes <​http://webkit.org/b/61061> <rdar://problem/9452796> TestFailures page blames
arbitrary revisions for breaking flaky tests

Reviewed by Dan Bates.

  • BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/FlakyLayoutTestDetector.js: Added.

(FlakyLayoutTestDetector): This class identifies flaky tests when given the test results
from various builds (in reverse-chronological order).
(FlakyLayoutTestDetector.prototype.incorporateTestResults): Detects flaky tests. Tests move
monotonically through three states: LastSeenFailing, LastSeenPassing, and PossiblyFlaky.
(FlakyLayoutTestDetector.prototype.flakinessExamples): Finds examples of flakiness for the
given test. Essentially, finds all the transitions from passing to failing (or vice-versa)
and puts them in an array in reverse-chronological order.
(FlakyLayoutTestDetector.prototype.get possiblyFlakyTests): Returns all tests we've detected
to be possibly flaky.

  • BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/LayoutTestHistoryAnalyzer.js:

(LayoutTestHistoryAnalyzer): Initialize new members.
(LayoutTestHistoryAnalyzer.prototype.start): Now passes the callback an object with two
properties: history and possiblyFlaky. history holds the data this function used to pass to
the callback, while possiblyFlaky lists all tests that might be flaky and examples of their
flakiness. Updated documentation comment to match.
(LayoutTestHistoryAnalyzer.prototype._incorporateBuildHistory): Now uses a
FlakyLayoutTestDetector to identify possibly flaky tests. Any possibly flaky tests are
removed from the failure history, since when they started failing is no longer meaningful.
We tell our caller to keep calling until all current failures have been explained and we've
gone through 5 builds without any new flaky tests being identified.

  • BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/Utilities.js:

(sorted): New helper function to return a sorted copy of an array.
(Array.prototype.findLast): New helper function. Like findFirst, but finds the last item
that matches the predicate.

  • BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/ViewController.js:

(ViewController.prototype._displayBuilder): Updated for change in the object passed to us by
the analyzer. Now puts the list of possibly flaky tests after the failure history.
(ViewController.prototype._domForFailedTest): Moved some code from here...
(ViewController.prototype._domForFailureDiagnosis): ...to here.
(ViewController.prototype._domForPossiblyFlakyTests): New function, builds up a list of
possibly flaky tests and examples of their flakiness and returns it.

  • BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/index.html: Pull in

FlakyLayoutTestDetector.js.

Location:
trunk/Tools
Files:
1 added
5 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/LayoutTestHistoryAnalyzer.js

    r89926 r90054  
    2626function LayoutTestHistoryAnalyzer(builder) {
    2727    this._builder = builder;
     28    this._flakinessDetector = new FlakyLayoutTestDetector();
    2829    this._history = {};
    2930    this._loader = new LayoutTestResultsLoader(builder);
     31    this._testRunsSinceLastInterestingChange = 0;
    3032}
    3133
    3234LayoutTestHistoryAnalyzer.prototype = {
    3335    /*
    34      * Preiodically calls callback until all current failures have been explained. Callback is
     36     * Periodically calls callback until all current failures have been explained. Callback is
    3537     * passed an object like the following:
    3638     * {
    37      *     'r12347 (681)': {
    38      *         'tooManyFailures': false,
    39      *         'tests': {
    40      *             'css1/basic/class_as_selector2.html': 'fail',
     39     *     'history': {
     40     *         'r12347 (681)': {
     41     *             'tooManyFailures': false,
     42     *             'tests': {
     43     *                 'css1/basic/class_as_selector2.html': 'fail',
     44     *             },
     45     *         },
     46     *         'r12346 (680)': {
     47     *             'tooManyFailures': false,
     48     *             'tests': {},
     49     *         },
     50     *         'r12345 (679)': {
     51     *             'tooManyFailures': false,
     52     *             'tests': {
     53     *                 'css1/basic/class_as_selector.html': 'crash',
     54     *             },
    4155     *         },
    4256     *     },
    43      *     'r12346 (680)': {
    44      *         'tooManyFailures': false,
    45      *         'tests': {},
     57     *     'possiblyFlaky': {
     58     *         'fast/workers/worker-test.html': [
     59     *             { 'build': 'r12345 (679)', 'result': 'pass' },
     60     *             { 'build': 'r12344 (678)', 'result': 'fail' },
     61     *             { 'build': 'r12340 (676)', 'result': 'fail' },
     62     *             { 'build': 'r12338 (675)', 'result': 'pass' },
     63     *         ],
    4664     *     },
    47      *     'r12345 (679)': {
    48      *         'tooManyFailures': false,
    49      *         'tests': {
    50      *             'css1/basic/class_as_selector.html': 'crash',
    51      *         },
    52      *     },
    53      * },
    54      * Each build contains just the failures that a) are still occuring on the bots, and b) were new
     65     * }
     66     * Each build contains just the failures that a) are still occurring on the bots, and b) were new
    5567     * in that build.
    5668     */
    … …  
    6375                    if (nextIndex >= buildNames.length)
    6476                        callAgain = false;
    65                     callback(self._history, callAgain);
     77                    var data = {
     78                        history: self._history,
     79                        possiblyFlaky: {},
     80                    };
     81                    self._flakinessDetector.possiblyFlakyTests.forEach(function(testName) {
     82                        data.possiblyFlaky[testName] = self._flakinessDetector.flakinessExamples(testName);
     83                    });
     84                    callback(data, callAgain);
    6685                    if (!callAgain)
    6786                        return;
    … …  
    7998        var self = this;
    8099        self._loader.start(nextBuildName, function(tests, tooManyFailures) {
     100            ++self._testRunsSinceLastInterestingChange;
     101
    81102            self._history[nextBuildName] = {
    82103                tooManyFailures: tooManyFailures,
    83104                tests: {},
    84105            };
     106
     107            var newFlakyTests = self._flakinessDetector.incorporateTestResults(nextBuildName, tests, tooManyFailures);
     108            if (newFlakyTests.length) {
     109                self._testRunsSinceLastInterestingChange = 0;
     110                // Remove all possibly flaky tests from the failure history, since when they failed
     111                // is no longer meaningful.
     112                newFlakyTests.forEach(function(testName) {
     113                    for (var buildName in self._history)
     114                        delete self._history[buildName].tests[testName];
     115                });
     116            }
    85117
    86118            for (var testName in tests) {
    … …  
    93125            }
    94126
    95             callback(Object.keys(self._history[nextBuildName].tests).length);
     127            var previousUnexplainedFailuresCount = previousBuildName ? Object.keys(self._history[previousBuildName].tests).length : 0;
     128            var unexplainedFailuresCount = Object.keys(self._history[nextBuildName].tests).length;
     129
     130            if (previousUnexplainedFailuresCount && !unexplainedFailuresCount)
     131                self._testRunsSinceLastInterestingChange = 0;
     132
     133            const minimumRequiredTestRunsWithoutInterestingChanges = 5;
     134            callback(unexplainedFailuresCount || self._testRunsSinceLastInterestingChange < minimumRequiredTestRunsWithoutInterestingChanges);
    96135        },
    97136        function(tests) {

  • trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/Utilities.js

    r89926 r90054  
    104104}
    105105
     106function sorted(array) {
     107    var newArray = array.slice();
     108    newArray.sort();
     109    return newArray;
     110}
     111
    106112Array.prototype.findFirst = function(predicate) {
    107113    for (var i = 0; i < this.length; ++i) {
     114        if (predicate(this[i]))
     115            return this[i];
     116    }
     117    return null;
     118}
     119
     120Array.prototype.findLast = function(predicate) {
     121    for (var i = this.length - 1; i >= 0; --i) {
    108122        if (predicate(this[i]))
    109123            return this[i];

  • trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/ViewController.js

    r89926 r90054  
    4949    _displayBuilder: function(builder) {
    5050        var self = this;
    51         (new LayoutTestHistoryAnalyzer(builder)).start(function(history, stillFetchingData) {
     51        var lastDisplay = 0;
     52        (new LayoutTestHistoryAnalyzer(builder)).start(function(data, stillFetchingData) {
    5253            var list = document.createElement('ol');
    5354            list.id = 'failure-history';
    54             Object.keys(history).forEach(function(buildName, buildIndex, buildNameArray) {
    55                 var failingTestNames = Object.keys(history[buildName].tests);
     55            Object.keys(data.history).forEach(function(buildName, buildIndex, buildNameArray) {
     56                var failingTestNames = Object.keys(data.history[buildName].tests);
    5657                if (!failingTestNames.length)
    5758                    return;
    … …  
    6465
    6566                testList.className = 'test-list';
    66                 for (var testName in history[buildName].tests) {
     67                for (var testName in data.history[buildName].tests) {
    6768                    var testItem = document.createElement('li');
    68                     testItem.appendChild(self._domForFailedTest(builder, buildName, testName, history[buildName].tests[testName]));
     69                    testItem.appendChild(self._domForFailedTest(builder, buildName, testName, data.history[buildName].tests[testName]));
    6970                    testList.appendChild(testItem);
    7071                }
    7172
    72                 if (history[buildName].tooManyFailures) {
     73                if (data.history[buildName].tooManyFailures) {
    7374                    var p = document.createElement('p');
    7475                    p.className = 'info';
    … …  
    9394            document.body.appendChild(header);
    9495            document.body.appendChild(list);
     96            document.body.appendChild(self._domForPossiblyFlakyTests(builder, data.possiblyFlaky));
    9597
    9698            if (!stillFetchingData)
    … …  
    197199
    198200    _domForFailedTest: function(builder, buildName, testName, failureType) {
    199         var diagnosticInfo = builder.failureDiagnosisTextAndURL(buildName, testName, failureType);
    200 
    201201        var result = document.createDocumentFragment();
    202202        result.appendChild(document.createTextNode(testName));
    203203        result.appendChild(document.createTextNode(' ('));
     204        result.appendChild(this._domForFailureDiagnosis(builder, buildName, testName, failureType));
     205        result.appendChild(document.createTextNode(')'));
     206        return result;
     207    },
     208
     209    _domForFailureDiagnosis: function(builder, buildName, testName, failureType) {
     210        var diagnosticInfo = builder.failureDiagnosisTextAndURL(buildName, testName, failureType);
     211        if (!diagnosticInfo)
     212            return document.createTextNode(failureType);
    204213
    205214        var textNode = document.createTextNode(diagnosticInfo.text);
    206         if ('url' in diagnosticInfo) {
    207             var link = document.createElement('a');
    208             link.href = diagnosticInfo.url;
    209             link.appendChild(textNode);
    210             result.appendChild(link);
    211         } else
    212             result.appendChild(textNode);
    213 
    214         result.appendChild(document.createTextNode(')'));
    215 
    216         return result;
     215        if (!('url' in diagnosticInfo))
     216            return textNode;
     217
     218        var link = document.createElement('a');
     219        link.href = diagnosticInfo.url;
     220        link.appendChild(textNode);
     221        return link;
    217222    },
    218223
    … …  
    383388        return result;
    384389    },
     390
     391    _domForPossiblyFlakyTests: function(builder, possiblyFlakyTestData) {
     392        var result = document.createDocumentFragment();
     393        var flakyTests = Object.keys(possiblyFlakyTestData);
     394        if (!flakyTests.length)
     395            return result;
     396
     397        var flakyHeader = document.createElement('h2');
     398        result.appendChild(flakyHeader);
     399        flakyHeader.appendChild(document.createTextNode('Possibly Flaky Tests'));
     400
     401        var flakyList = document.createElement('ol');
     402        result.appendChild(flakyList);
     403
     404        var self = this;
     405        flakyList.appendChildren(sorted(flakyTests).map(function(testName) {
     406            var item = document.createElement('li');
     407            item.appendChild(document.createTextNode(testName));
     408            var historyList = document.createElement('ol');
     409            item.appendChild(historyList);
     410            historyList.appendChildren(possiblyFlakyTestData[testName].map(function(historyItem) {
     411                var item = document.createElement('li');
     412                item.appendChild(self._domForBuildName(builder, historyItem.build));
     413                item.appendChild(document.createTextNode(': '));
     414                item.appendChild(self._domForFailureDiagnosis(builder, historyItem.build, testName, historyItem.result));
     415                return item;
     416            }));
     417            return item;
     418        }));
     419
     420        return result;
     421    },
    385422};

  • trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/index.html

    r89838 r90054  
    3131    <script src="Buildbot.js"></script>
    3232    <script src="Builder.js"></script>
     33    <script src="FlakyLayoutTestDetector.js"></script>
    3334    <script src="LayoutTestHistoryAnalyzer.js"></script>
    3435    <script src="LayoutTestResultsLoader.js"></script>

  • trunk/Tools/ChangeLog

    r90052 r90054  
     12011-06-29  Adam Roben  <aroben@apple.com>
     2
     3        Teach TestFailures to detect possibly flaky tests and list them separately
     4
     5        Fixes <http://webkit.org/b/61061> <rdar://problem/9452796> TestFailures page blames
     6        arbitrary revisions for breaking flaky tests
     7
     8        Reviewed by Dan Bates.
     9
     10        * BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/FlakyLayoutTestDetector.js: Added.
     11        (FlakyLayoutTestDetector): This class identifies flaky tests when given the test results
     12        from various builds (in reverse-chronological order).
     13        (FlakyLayoutTestDetector.prototype.incorporateTestResults): Detects flaky tests. Tests move
     14        monotonically through three states: LastSeenFailing, LastSeenPassing, and PossiblyFlaky.
     15        (FlakyLayoutTestDetector.prototype.flakinessExamples): Finds examples of flakiness for the
     16        given test. Essentially, finds all the transitions from passing to failing (or vice-versa)
     17        and puts them in an array in reverse-chronological order.
     18        (FlakyLayoutTestDetector.prototype.get possiblyFlakyTests): Returns all tests we've detected
     19        to be possibly flaky.
     20
     21        * BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/LayoutTestHistoryAnalyzer.js:
     22        (LayoutTestHistoryAnalyzer): Initialize new members.
     23        (LayoutTestHistoryAnalyzer.prototype.start): Now passes the callback an object with two
     24        properties: history and possiblyFlaky. history holds the data this function used to pass to
     25        the callback, while possiblyFlaky lists all tests that might be flaky and examples of their
     26        flakiness. Updated documentation comment to match.
     27        (LayoutTestHistoryAnalyzer.prototype._incorporateBuildHistory): Now uses a
     28        FlakyLayoutTestDetector to identify possibly flaky tests. Any possibly flaky tests are
     29        removed from the failure history, since when they started failing is no longer meaningful.
     30        We tell our caller to keep calling until all current failures have been explained and we've
     31        gone through 5 builds without any new flaky tests being identified.
     32
     33        * BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/Utilities.js:
     34        (sorted): New helper function to return a sorted copy of an array.
     35        (Array.prototype.findLast): New helper function. Like findFirst, but finds the last item
     36        that matches the predicate.
     37
     38        * BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/ViewController.js:
     39        (ViewController.prototype._displayBuilder): Updated for change in the object passed to us by
     40        the analyzer. Now puts the list of possibly flaky tests after the failure history.
     41        (ViewController.prototype._domForFailedTest): Moved some code from here...
     42        (ViewController.prototype._domForFailureDiagnosis): ...to here.
     43        (ViewController.prototype._domForPossiblyFlakyTests): New function, builds up a list of
     44        possibly flaky tests and examples of their flakiness and returns it.
     45
     46        * BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/index.html: Pull in
     47        FlakyLayoutTestDetector.js.
     48
    1492011-06-29  Eric Seidel  <eric@webkit.org>
    250
Note: See TracChangeset for help on using the changeset viewer.