Changeset 286128 in webkit

Timestamp:

Nov 23, 2021, 12:57:03 AM (5 years ago)

Author:

Angelos Oikonomopoulos

Message:

Fix buildbot command timeout issue for remote tests
​https://bugs.webkit.org/show_bug.cgi?id=233309

Reviewed by Adrian Perez de Castro.

Occasionally, buildbots that execute the JSC tests on remote devices
would stop producing any output and eventually get killed due to a timeout.

The issue is that run-jsc-stress-tests would start an ssh command
(overwhelmingly likely, the command to copy over the payload), then the remote
board would stop responding.

Introduce a timeout option to forEachRemote and bound the waiting time.
Could also do this for each command individually (and this would make it
straightforward to GC zombie processes) but seems way more fragile than
doing it at the thread level.

• Scripts/run-jsc-stress-tests:

Location:

trunk/Tools

Files:

• ChangeLog (modified) (1 diff)
• Scripts/run-jsc-stress-tests (modified) (5 diffs)

trunk/Tools/ChangeLog

@@ +1 @@
+2021-11-23  Angelos Oikonomopoulos  <angelos@igalia.com>
+
+        Fix buildbot command timeout issue for remote tests
+        https://bugs.webkit.org/show_bug.cgi?id=233309
+
+        Reviewed by Adrian Perez de Castro.
+
+        Occasionally, buildbots that execute the JSC tests on remote devices
+        would stop producing any output and eventually get killed due to a timeout.
+
+        The issue is that run-jsc-stress-tests would start an ssh command
+        (overwhelmingly likely, the command to copy over the payload), then the remote
+        board would stop responding.
+
+        Introduce a timeout option to forEachRemote and bound the waiting time.
+        Could also do this for each command individually (and this would make it
+        straightforward to GC zombie processes) but seems way more fragile than
+        doing it at the thread level.
+
+        * Scripts/run-jsc-stress-tests:
+
 2021-11-22  Wenson Hsieh  <wenson_hsieh@apple.com>
 

trunk/Tools/Scripts/run-jsc-stress-tests

@@ -82 +82 @@
                                   :iterationsCeiling => 10)
 
+REMOTE_TIMEOUT = 120
 begin
     require 'shellwords'
@@ -2602 +2603 @@
     }
 
+    etime = nil
+    if options.has_key?(:timeout)
+        etime = Time.now + options[:timeout]
+    end
     liveRemotes = []
     threads.each_index {
@@ -2607 +2612 @@
         thread = threads[index]
         begin
-            thread.join
+            if options.has_key?(:timeout)
+                if etime.nil?
+                    # If a timeout has been requested and etime is nil,
+                    # that means the timeout has expired and we shouldn't
+                    # wait at all.
+                    timeout = 0
+                else
+                    timeout = etime - Time.now
+                    if timeout < 0
+                        timeout = 0
+                    end
+                end
+                if thread.join(timeout).nil?
+                    if $verbosity > 0
+                        $stderr.puts("Timeout joining thread for remote #{remoteHosts[index]}")
+                    end
+                    # Timeout expired, so we can't block waiting for
+                    # any other threads. Either they're done or
+                    # they've also timed out.
+                    etime = nil
+                    raise CommandExecutionFailed
+                end
+            else
+                thread.join # No timeout requested, just block.
+            end
             liveRemotes << remoteHosts[index]
         rescue CommandExecutionFailed
@@ -2695 +2724 @@
 
 def unpackBundleGnuParallel(remoteHosts)
-    forEachRemote(remoteHosts, :dropOnFailure => true) {
+    forEachRemote(remoteHosts, :dropOnFailure => true, :timeout => REMOTE_TIMEOUT) {
         | _, remoteHost |
         mysys(["ssh", "-o", "NoHostAuthenticationForLocalhost=yes"] + 
@@ -2864 +2893 @@
         # iteration, we'll try to run test jobs on it, possibly using
         # an unrelated bundle from a previous run.
-        remoteHosts = forEachRemote(remoteHosts, {:dropOnFailure => true}) {
+        remoteHosts = forEachRemote(remoteHosts, {:dropOnFailure => true, :timeout => REMOTE_TIMEOUT}) {
             | _, remoteHost |
             getRemoteDirectoryIfNeeded(remoteHost)