Selective Regression Tests in Early Warning System
Regression test selection based on procedure level code coverage of the changes will be applied to EWS. See the description of the selection method, meta-bug and patches at SelectiveTestBuildBot.
Before introducing it to the official EWS we plan to make some tests for its reliability and efficiency. The test system is available at: http://qt-ews.appspot.com/queue-status/qt-st-ews
The test system is modified so that it executes both the full and the selective test on the patch. First the test selection is performed based on the changes in the patch and using the coverage database, and if it returns a non-trivial selection (not 0 and not all the tests), then the full test is executed on the same patch. The system returns always with a failure so that we can examine the build logs to verify the correctness.
For example
Here will be some example, where the early warning system caught a bug with test selection:
- At revision 115655 the patch committed for bug 85217 failed on the fast/flexbox/flex-hang.html test with unexpected text diff mismatch. For this result the Early Warning System Selective Test had to run only 1036 test cases.
editing/deleting/5546763.html
editing/deleting/delete-block-merge-contents-005.html
editing/deleting/delete-block-merge-contents-006.html
editing/deleting/delete-block-merge-contents-008.html
editing/execCommand/format-block-multiple-paragraphs.html
editing/pasteboard/4076267-3.html
editing/pasteboard/copy-paste-bidi.html
editing/pasteboard/do-not-copy-unnecessary-styles.html
editing/pasteboard/merge-end-3.html
editing/pasteboard/paste-line-endings-009.html
editing/pasteboard/paste-table-002.html
with unexpected text diff mismatch. For this result Early Warning System Selective Test had to run only 361 test cases.
