Changeset 252953 in webkit

Timestamp:

Nov 30, 2019 12:15:36 AM (4 years ago)

Author:

aakash_jain@apple.com

Message:

[EWS] Do not retry layout-tests build if the flaky test failures are also present in clean tree run
​https://bugs.webkit.org/show_bug.cgi?id=204704

Reviewed by Alexey Proskuryakov.

• BuildSlaveSupport/ews-build/steps.py:

(AnalyzeLayoutTestsResults.start):

• BuildSlaveSupport/ews-build/steps_unittest.py:

(TestAnalyzeLayoutTestsResults.test_flaky_and_inconsistent_failures_with_clean_tree_failures): Changed expected outcome to SUCCESS.
(TestAnalyzeLayoutTestsResults.test_flaky_and_consistent_failures_with_clean_tree_failures): Ditto.

Location:

trunk/Tools

Files:

• BuildSlaveSupport/ews-build/steps.py (modified) (1 diff)
• BuildSlaveSupport/ews-build/steps_unittest.py (modified) (2 diffs)
• ChangeLog (modified) (1 diff)

trunk/Tools/BuildSlaveSupport/ews-build/steps.py

@@ -1453 +1453 @@
                     return self.report_failure(failures_introduced_by_patch)
 
-            # At this point we know that at least one test flaked, but no consistent failures
-            # were introduced. This is a bit of a grey-zone.
+            new_failing_or_flaky_tests = first_results_failing_tests.union(second_results_failing_tests) - clean_tree_results_failing_tests
+            if not new_failing_or_flaky_tests:
+                return self.report_pre_existing_failures(clean_tree_results_failing_tests)
+
+            # At this point we know that at least one test flaked, and those flaky tests passed on clean tree,
+            # and no consistent failures were introduced. This is a bit of a grey-zone. So, we retry the build.
             return self.retry_build()
 

trunk/Tools/BuildSlaveSupport/ews-build/steps_unittest.py

@@ -1648 +1648 @@
         self.setProperty('second_run_failures', ['test3'])
         self.setProperty('clean_tree_run_failures', ['test1', 'test2', 'test3'])
-        self.expectOutcome(result=RETRY, state_string='Unable to confirm if test failures are introduced by patch, retrying build (retry)')
+        self.expectOutcome(result=SUCCESS, state_string='Passed layout tests')
         return self.runStep()
 
@@ -1656 +1656 @@
         self.setProperty('second_run_failures', ['test1'])
         self.setProperty('clean_tree_run_failures', ['test1', 'test2'])
-        self.expectOutcome(result=RETRY, state_string='Unable to confirm if test failures are introduced by patch, retrying build (retry)')
+        self.expectOutcome(result=SUCCESS, state_string='Passed layout tests')
         return self.runStep()
 

trunk/Tools/ChangeLog

@@ +1 @@
+2019-11-30  Aakash Jain  <aakash_jain@apple.com>
+
+        [EWS] Do not retry layout-tests build if the flaky test failures are also present in clean tree run
+        https://bugs.webkit.org/show_bug.cgi?id=204704
+
+        Reviewed by Alexey Proskuryakov.
+
+        * BuildSlaveSupport/ews-build/steps.py:
+        (AnalyzeLayoutTestsResults.start):
+        * BuildSlaveSupport/ews-build/steps_unittest.py:
+        (TestAnalyzeLayoutTestsResults.test_flaky_and_inconsistent_failures_with_clean_tree_failures): Changed expected outcome to SUCCESS.
+        (TestAnalyzeLayoutTestsResults.test_flaky_and_consistent_failures_with_clean_tree_failures): Ditto.
+
 2019-11-29  Zalan Bujtas  <zalan@apple.com>