Changeset 281660 in webkit

Timestamp:

Aug 26, 2021, 3:07:13 PM (5 years ago)

Author:

aakash_jain@apple.com

Message:

[EWS] compile-webkit-without-patch failed, but bot ran layout tests anyway and blamed the patch for a pre-existing failure
​https://bugs.webkit.org/show_bug.cgi?id=229533

Reviewed by Jonathan Bedard.

When the compile-webkit-without-patch step fails, instead of running further layout-tests, retry the build and email
bot watcher's. The retried build will hopefully be picked up by other bot eventually. This is similar to what we do
in case of other infrastructure issues like kill-old-processes step failure.

• CISupport/ews-build/steps.py:

(CompileWebKitWithoutPatch.init): Added retry_build_on_failure parameter.
(CompileWebKitWithoutPatch.evaluateCommand): If build failed unexpectedly, retry it and email bot watchers.
(CompileWebKitWithoutPatch.send_email_for_unexpected_build_failure):
(ReRunWebKitTests.evaluateCommand):
(ReRunAPITests.evaluateCommand):

Location:

trunk/Tools

Files:

• CISupport/ews-build/steps.py (modified) (4 diffs)
• ChangeLog (modified) (1 diff)

trunk/Tools/CISupport/ews-build/steps.py

@@ -1633 +1633 @@
     haltOnFailure = False
 
+    def __init__(self, retry_build_on_failure=False, **kwargs):
+        self.retry_build_on_failure = retry_build_on_failure
+        super(CompileWebKitWithoutPatch, self).__init__(**kwargs)
+
     def doStepIf(self, step):
         return self.getProperty('patchFailedToBuild') or self.getProperty('patchFailedTests')
@@ -1640 +1644 @@
 
     def evaluateCommand(self, cmd):
-        return shell.Compile.evaluateCommand(self, cmd)
+        rc = shell.Compile.evaluateCommand(self, cmd)
+        if rc == FAILURE and self.retry_build_on_failure:
+            message = 'Unable to build WebKit without patch, retrying build'
+            self.descriptionDone = message
+            self.send_email_for_unexpected_build_failure()
+            self.build.buildFinished([message], RETRY)
+        return rc
+
+    def send_email_for_unexpected_build_failure(self):
+        try:
+            builder_name = self.getProperty('buildername', '')
+            worker_name = self.getProperty('workername', '')
+            build_url = '{}#/builders/{}/builds/{}'.format(self.master.config.buildbotURL, self.build._builderid, self.build.number)
+            email_subject = '{} might be in bad state, unable to build WebKit'.format(worker_name)
+            email_text = '{} might be in bad state. It is unable to build WebKit.'.format(worker_name)
+            email_text += ' Same patch was built successfuly on builder queue previously.\n\nBuild: {}\n\nBuilder: {}'.format(build_url, builder_name)
+            reference = 'build-failure-{}'.format(worker_name)
+            send_email_to_bot_watchers(email_subject, email_text, builder_name, reference)
+        except Exception as e:
+            print('Error in sending email for unexpected build failure: {}'.format(e))
 
 
@@ -2389 +2412 @@
                                                 UnApplyPatchIfRequired(),
                                                 ValidatePatch(verifyBugClosed=False, addURLs=False),
-                                                CompileWebKitWithoutPatch(),
+                                                CompileWebKitWithoutPatch(retry_build_on_failure=True),
                                                 ValidatePatch(verifyBugClosed=False, addURLs=False),
                                                 KillOldProcesses(),
@@ -2857 +2880 @@
             elif platform == 'gtk':
                 steps_to_add.append(InstallGtkDependencies())
-            steps_to_add.append(CompileWebKitWithoutPatch())
+            steps_to_add.append(CompileWebKitWithoutPatch(retry_build_on_failure=True))
             steps_to_add.append(ValidatePatch(verifyBugClosed=False, addURLs=False))
             steps_to_add.append(KillOldProcesses())

trunk/Tools/ChangeLog

@@ +1 @@
+2021-08-26  Aakash Jain  <aakash_jain@apple.com>
+
+        [EWS] compile-webkit-without-patch failed, but bot ran layout tests anyway and blamed the patch for a pre-existing failure
+        https://bugs.webkit.org/show_bug.cgi?id=229533
+
+        Reviewed by Jonathan Bedard.
+
+        When the compile-webkit-without-patch step fails, instead of running further layout-tests, retry the build and email
+        bot watcher's. The retried build will hopefully be picked up by other bot eventually. This is similar to what we do
+        in case of other infrastructure issues like kill-old-processes step failure.
+        * CISupport/ews-build/steps.py:
+        (CompileWebKitWithoutPatch.__init__): Added retry_build_on_failure parameter.
+        (CompileWebKitWithoutPatch.evaluateCommand): If build failed unexpectedly, retry it and email bot watchers.
+        (CompileWebKitWithoutPatch.send_email_for_unexpected_build_failure):
+        (ReRunWebKitTests.evaluateCommand):
+        (ReRunAPITests.evaluateCommand):
+
 2021-08-26  Simon Fraser  <simon.fraser@apple.com>