Changes between Version 3 and Version 4 of SelectiveTestBuildBot


Ignore:
Timestamp:
Apr 18, 2012 12:14:55 PM (10 years ago)
Author:
beszedes@inf.u-szeged.hu
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • SelectiveTestBuildBot

    v3 v4  
    77on average less than 0.1% of the test cases were failing while on average 5 functions were changed per revision in this period. Moreover, the number of test cases affected by a specific change was usually very low. The coverage of all the tests was altogether about 70-75% of the total number of functions but if we perform the tests selectively we can still achieve the same coverage with respect to the changes but by executing only a fraction of all the tests.
    88
    9 In the same experiment, we got a high percentage of inclusiveness: over 95% of the failing test cases in the full test could be identified by using the function change coverage method, and at the same time a siginficant reduction in the number of test performed was achieved. Namely, we got this result by executing less than 1% of the test cases on average.
     9In the same experiment, we got a high percentage of inclusiveness: over 95% of the failing test cases in the full test could be identified by using the function change coverage method, and at the same time a significant reduction in the number of tests performed was achieved. Namely, we got this result by executing less than 1% of the test cases on average.
    1010
    1111== Usage ==
    1212There will be two types of use of the selective regression tests:
    1313 * As a special build bot (currently for the Qt port which started in November 2011). This provides a faster alternative to the build bot performing full test: it saves more than 77% of the total build time including the overhead required for the selection. Probably embedded platforms like ARM could benefit most of this method. The Qt build bot can be found here: http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20QuickTest
    14  * As part of the Chromium Early Warning System. This will allow developers and reviewers to check the patches in Bugzilla before landing in the repository. Due to the selection the time required to perform the tests will be much less. Detailed description is at [wiki:"SelectiveTestEWS"].
     14 * As part of the [http://queues.webkit.org/ official EWS ]. This will allow developers and reviewers to check the patches in Bugzilla before landing in the repository. Due to the selection the time required to perform the tests will be much less. Detailed description is at [wiki:"SelectiveTestEWS"].
    1515
    1616== Technicalities and how to get it ==
    17 The method for test selection is the following. We instrument the source code of the methods and functions to log entry and exit events during execution, and make such an instrumented build of the system (using the GCC instrumentation feature and our custom C++ instrumentation code). Then all tests are executed to produce the initial coverage information, which relates each test to a set of functions it executes. This information is stored in a relational database. Next, a set of changed functions is extracted from a revision under test (using an extended PrepareChangeLog script) and this set is used to query the database for a list of tests to execute.
     17The method for test selection is the following. We instrument the source code of the methods and functions to log entry and exit events during execution, and make such an instrumented build of the system (using the GCC instrumentation feature and our custom C++ instrumentation code). Then all tests are executed to produce the initial coverage information, which relates each test to a set of functions or methods it executes. This information is stored in a relational database. Next, a set of changed functions is extracted from a revision under test (using an extended PrepareChangeLog script) and this set is used to query the database for a list of tests to execute.
    1818
    19 The meta-bug describing the method can be found here: https://bugs.webkit.org/show_bug.cgi?id=78699
    20 
    21 The preliminary patch is (will be) here.
     19The meta-bug with patches describing the method can be found here: https://bugs.webkit.org/show_bug.cgi?id=78699
    2220
    2321== Some results ==