Changes between Version 2 and Version 3 of RegressionTestResearch


Ignore:
Timestamp:
Apr 16, 2012, 4:57:51 AM (13 years ago)
Author:
beszedes@inf.u-szeged.hu
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • RegressionTestResearch

    v2 v3  
    1 = Isolation of flaky tests and further plans =
     1= Further research with regression tests =
    22
    3 == Flakies ==
     3== Isolation of flaky tests ==
     4
    45Flaky tests are a nightmare for systematic analysis and enhancement of the regression tests. In many cases it was nearly impossible to make reasonable conclusions about the test results, only by tedious manual analysis. The goal of this project is to systematically assess how flaky the tests are. Can we safely isolate a set of test cases that are usualy flakes and that are never? If yes, we could create a more intelligent test sets that are consistent, hence more predictable.
    56Currently we are working on:
    67 * Randomize the order of test execution and observe the differences in the results (run-webkit-tests --randomize-order)
    78 * Execute all tests individually and see the differences to the batch execution (run-webkit-tests --run-singly)
     9
     10== Invariant analysis of top-100 websites ==
     11
     12Layout tests could be extended with real online websites such as top-100 (top-300) by using invariant analysis which hides changing content (see [wiki:"Top100Invariant"]).
     13
    814
    915== Other research ideas: ==