Changeset 215599 in webkit

Timestamp:

Apr 20, 2017, 6:49:59 PM (9 years ago)

Author:

Joseph Pecoraro

Message:

Test262 bot does not go red with failures
​https://bugs.webkit.org/show_bug.cgi?id=171044

Reviewed by Aakash Jain.

The piped output of run-jsc-stress-tests is different from
its non-piped output. Switch to parsing the correct output.

• BuildSlaveSupport/build.webkit.org-config/master.cfg:

(RunTest262Tests.countFailures):
Just count the number of lines that start with "FAIL:".

• BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:

(RunTest262TestsTest.test_no_regressions_output):
(test_failure_output):
(test_failures_output):
Updated tests with new sample output produced partially from bots and
locally. In cases of failure the return code of run-jsc-stress-tests
is still 0 so update the expected return code in the tests.

Location:

trunk/Tools

Files:

• BuildSlaveSupport/build.webkit.org-config/master.cfg (modified) (1 diff)
• BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py (modified) (1 diff)
• ChangeLog (modified) (1 diff)

trunk/Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg

@@ -336 +336 @@
     def countFailures(self, cmd):
         logText = cmd.logs['stdio'].getText()
-        matches = re.findall(r'\(failed (\d+)\)', logText)
+        matches = re.findall(r'^FAIL:', logText, flags=re.MULTILINE)
         if matches:
-            return int(matches[-1])
+            return len(matches)
         return 0
 

trunk/Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py

@@ -154 +154 @@
     def test_no_regressions_output(self):
         self.assertResults(SUCCESS, ["test262-test"], 0, """Using the following jsc path: /WebKitBuild/Release/jsc
-168/168         """)
+
+Running test262.yaml/test262/test/annexB/built-ins/Date/prototype/getYear/length.js.default
+Running test262.yaml/test262/test/annexB/built-ins/Date/prototype/getYear/length.js.default-strict
+""")
 
     def test_failure_output(self):
-        self.assertResults(FAILURE, ["1 Test262 test failed"], 1, """Using the following jsc path: /WebKitBuild/Release/jsc
-test262.yaml/test262/test/built-ins/Array/from/iter-set-elem-prop-err.js.default: ERROR: Unexpected exit code: 0
-test262.yaml/test262/test/built-ins/Array/from/iter-set-elem-prop-err.js.default-strict: ERROR: Unexpected exit code: 0
-...
-43768/43768 (failed 1)         """)
+        self.assertResults(FAILURE, ["1 Test262 test failed"], 0, """Using the following jsc path: /WebKitBuild/Release/jsc
+
+Running test262.yaml/test262/test/annexB/built-ins/Date/prototype/getYear/length.js.default
+Running test262.yaml/test262/test/annexB/built-ins/Date/prototype/getYear/length.js.default-strict
+test262.yaml/test262/test/annexB/built-ins/Date/prototype/getYear/length.js.default-strict: ERROR: Unexpected exit code: 0
+FAIL: test262.yaml/test262/test/annexB/built-ins/Date/prototype/getYear/length.js.default-strict
+""")
 
     def test_failures_output(self):
-        self.assertResults(FAILURE, ["75 Test262 tests failed"], 75, """Using the following jsc path: /WebKitBuild/Release/jsc
-...
-43768/43768 (failed 75)         """)
+        self.assertResults(FAILURE, ["2 Test262 tests failed"], 0, """Using the following jsc path: /WebKitBuild/Release/jsc
+
+Running test262.yaml/test262/test/annexB/built-ins/Date/prototype/getYear/length.js.default
+test262.yaml/test262/test/annexB/built-ins/Date/prototype/getYear/length.js.default: ERROR: Unexpected exit code: 0
+FAIL: test262.yaml/test262/test/annexB/built-ins/Date/prototype/getYear/length.js.default
+Running test262.yaml/test262/test/annexB/built-ins/Date/prototype/getYear/length.js.default-strict
+test262.yaml/test262/test/annexB/built-ins/Date/prototype/getYear/length.js.default-strict: ERROR: Unexpected exit code: 0
+FAIL: test262.yaml/test262/test/annexB/built-ins/Date/prototype/getYear/length.js.default-strict
+""")
 
 

trunk/Tools/ChangeLog

@@ +1 @@
+2017-04-20  Joseph Pecoraro  <pecoraro@apple.com>
+
+        Test262 bot does not go red with failures
+        https://bugs.webkit.org/show_bug.cgi?id=171044
+
+        Reviewed by Aakash Jain.
+
+        The piped output of run-jsc-stress-tests is different from
+        its non-piped output. Switch to parsing the correct output.
+
+        * BuildSlaveSupport/build.webkit.org-config/master.cfg:
+        (RunTest262Tests.countFailures):
+        Just count the number of lines that start with "FAIL:".
+
+        * BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:
+        (RunTest262TestsTest.test_no_regressions_output):
+        (test_failure_output):
+        (test_failures_output):
+        Updated tests with new sample output produced partially from bots and
+        locally. In cases of failure the return code of run-jsc-stress-tests
+        is still 0 so update the expected return code in the tests.
+
 2017-04-20  Brady Eidson  <beidson@apple.com>