Changes between Version 1 and Version 2 of RegressionTestResearch


Ignore:
Timestamp:
Feb 13, 2012 4:00:38 AM (10 years ago)
Author:
Csaba Osztrogonác
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • RegressionTestResearch

    v1 v2  
    44Flaky tests are a nightmare for systematic analysis and enhancement of the regression tests. In many cases it was nearly impossible to make reasonable conclusions about the test results, only by tedious manual analysis. The goal of this project is to systematically assess how flaky the tests are. Can we safely isolate a set of test cases that are usualy flakes and that are never? If yes, we could create a more intelligent test sets that are consistent, hence more predictable.
    55Currently we are working on:
    6  * Randomize the order of test execution and observe the differences in the results
    7  * Execute all tests indivodually and see the differences to the batch execution
     6 * Randomize the order of test execution and observe the differences in the results (run-webkit-tests --randomize-order)
     7 * Execute all tests individually and see the differences to the batch execution (run-webkit-tests --run-singly)
    88
    99== Other research ideas: ==