Changes between Version 5 and Version 6 of NewRunWebKitTests

Timestamp:

Jul 7, 2011, 3:11:44 PM (13 years ago)

Author:

dpranke@chromium.org

Comment:

--

NewRunWebKitTests

@@ -42 +42 @@
 }}}
 
-and when the test completes, you'll get output like:
+and when the test completes, you'll get output something like:
 
 {{{
 % new-run-webkit-tests
+  animations/animation-direction-normal.html -> unexpected text diff mismatch
+  compositing/reflections/nested-reflection-animated.html -> unexpected text diff mismatch
+                                                         
+Retrying 2 unexpected failure(s) ...
+
+  animations/animation-direction-normal.html -> unexpected text diff mismatch
+  compositing/reflections/nested-reflection-animated.html -> unexpected text diff mismatch
+                                                         
+24894 tests ran as expected, 2 didn't:
+
+Regressions: Unexpected text diff mismatch : (10)
+  animations/animation-direction-normal.html = TEXT
+  compositing/reflections/nested-reflection-animated.html = TEXT
+
+% echo $?
+2
+%
 }}}
 
+In addition, by default, a web browser will be opened to an HTML page displaying the tests that failed, and links to the output from the tests. You can notice a few things about this test output:
+
+ 1. Tests that fail are automatically retried, to see if the failure was "flaky" and only happens some of the time. A flaky failure (one that fails then passing during the retry is reported on screen, but does not count as an actual failure.
+ 2. The retries are all done serially in a single thread, to avoid any load-related or ordering-related issues that might be contributing to the flakiness.
+
 === More Advanced Scenarios ===