Changeset 90650 in webkit

Timestamp:

Jul 8, 2011 12:05:17 PM (13 years ago)

Author:

Adam Roben

Message:

Teach buildbot to figure out how many webkitpy/webkitperl tests failed

Fixes <​http://webkit.org/b/64192> It's hard to tell how many test-webkitpy/test-webkitperl
tests failed when looking at build.webkit.org

Reviewed by Eric Seidel.

• BuildSlaveSupport/build.webkit.org-config/master.cfg:

(TestWithFailureCount): New class that represents a test build step which has an associated
failure count. Eventually we should move more of our test classes to deriving from this.
(TestWithFailureCount.countFailures): Method for subclasses to override to say how many
failures occurred.

(TestWithFailureCount.commandComplete):
(TestWithFailureCount.evaluateCommand):
(TestWithFailureCount.getText):
(TestWithFailureCount.getText2):
These were all based on RunGtkAPITests.

(RunPythonTests): Changed to inherit from TestWithFailureCount.
(RunPythonTests.countFailures): Parses the test-webkitpy output looking for the count of
failures.
(RunPerlTests): Changed to inherit from TestWithFailureCount.
(RunPerlTests.countFailures): Parses the test-webkitperl output looking for the count of
failures.

Location:

trunk/Tools

Files:

• BuildSlaveSupport/build.webkit.org-config/master.cfg (modified) (1 diff)
• ChangeLog (modified) (1 diff)

trunk/Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg

@@ -332 +332 @@
         return shell.Test.start(self)
 
-class RunPythonTests(shell.Test):
+
+class TestWithFailureCount(shell.Test):
+    failedTestsFormatString = "%d tests failed"
+
+    def countFailures(self, cmd):
+        return 0
+
+    def commandComplete(self, cmd):
+        shell.Test.commandComplete(self, cmd)
+        self.failedTestCount = self.countFailures(cmd)
+
+    def evaluateCommand(self, cmd):
+        if self.failedTestCount:
+            return FAILURE
+
+        if cmd.rc != 0:
+            return FAILURE
+
+        return SUCCESS
+
+    def getText(self, cmd, results):
+        return self.getText2(cmd, results)
+
+    def getText2(self, cmd, results):
+        if results != SUCCESS and self.failedTestCount:
+            return [self.failedTestsFormatString % self.failedTestCount]
+
+        return [self.name]
+
+
+class RunPythonTests(TestWithFailureCount):
     name = "webkitpy-test"
     description = ["python-tests running"]
     descriptionDone = ["python-tests"]
     command = ["python", "./Tools/Scripts/test-webkitpy"]
-
-
-class RunPerlTests(shell.Test):
+    failedTestsFormatString = "%d python tests failed"
+
+    def countFailures(self, cmd):
+        logText = cmd.logs['stdio'].getText()
+        # We're looking for the line that looks like this: FAILED (failures=2, errors=1)
+        regex = re.compile(r'^FAILED \((?P<counts>[^)]+)\)')
+        for line in logText.splitlines():
+            match = regex.match(line)
+            if not match:
+                continue
+            return sum(int(component.split('=')[1]) for component in match.group('counts').split(', '))
+        return 0
+
+
+class RunPerlTests(TestWithFailureCount):
     name = "webkitperl-test"
     description = ["perl-tests running"]
     descriptionDone = ["perl-tests"]
     command = ["perl", "./Tools/Scripts/test-webkitperl"]
+    failedTestsFormatString = "%d perl tests failed"
+
+    def countFailures(self, cmd):
+        logText = cmd.logs['stdio'].getText()
+        # We're looking for the line that looks like this: Failed 2/19 test programs. 5/363 subtests failed.
+        regex = re.compile(r'^Failed \d+/\d+ test programs\. (?P<count>\d+)/\d+ subtests failed\.')
+        for line in logText.splitlines():
+            match = regex.match(line)
+            if not match:
+                continue
+            return int(match.group('count'))
+        return 0
 
 

trunk/Tools/ChangeLog

@@ +1 @@
+2011-07-08  Adam Roben  <aroben@apple.com>
+
+        Teach buildbot to figure out how many webkitpy/webkitperl tests failed
+
+        Fixes <http://webkit.org/b/64192> It's hard to tell how many test-webkitpy/test-webkitperl
+        tests failed when looking at build.webkit.org
+
+        Reviewed by Eric Seidel.
+
+        * BuildSlaveSupport/build.webkit.org-config/master.cfg:
+        (TestWithFailureCount): New class that represents a test build step which has an associated
+        failure count. Eventually we should move more of our test classes to deriving from this.
+        (TestWithFailureCount.countFailures): Method for subclasses to override to say how many
+        failures occurred.
+
+        (TestWithFailureCount.commandComplete):
+        (TestWithFailureCount.evaluateCommand):
+        (TestWithFailureCount.getText):
+        (TestWithFailureCount.getText2):
+        These were all based on RunGtkAPITests.
+
+        (RunPythonTests): Changed to inherit from TestWithFailureCount.
+        (RunPythonTests.countFailures): Parses the test-webkitpy output looking for the count of
+        failures.
+        (RunPerlTests): Changed to inherit from TestWithFailureCount.
+        (RunPerlTests.countFailures): Parses the test-webkitperl output looking for the count of
+        failures.
+
 2011-07-08  Adam Roben  <aroben@apple.com>