⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 226944 in webkit


Ignore:
Timestamp:
Jan 15, 2018, 12:17:46 AM (9 years ago)
Author:
Carlos Garcia Campos
Message:

[GTK][WPE] Add support for unit test expectations
​https://bugs.webkit.org/show_bug.cgi?id=181589

Reviewed by Michael Catanzaro.

We currently have a way to skip tests by annotating them in the api test runner script. The main problem of this
approach is that we skip tests when they fail in the bots and we never notice if they stop failing, keeping the
tests skipped forever. This is indeed the case of several WebKit2 C API tests. Annotating skipped tests in the
script itself is not a good idea either.

This patch adds a generic TestExpectations class for simple tests based on tests with subtests, like our unit
tests, but also WebDriver tests. It parses a json file with the tests and subtests expectations and provides
convenient methods to query them.

  • Scripts/run-gtk-tests:

(GtkTestRunner): Remove all Skipped and Slow tests marked here.

  • Scripts/run-wpe-tests:

(WPETestRunner): Ditto.

  • Scripts/webkitpy/common/test_expectations.py: Added.

(TestExpectations):
(TestExpectations.init):
(TestExpectations._port_name_for_expected):
(TestExpectations._expected_value):
(TestExpectations.skipped_tests):
(TestExpectations.skipped_subtests):
(TestExpectations._expectation_value):
(TestExpectations.is_slow):
(TestExpectations.get_expectation):

  • Scripts/webkitpy/common/test_expectations_unittest.py: Added.

(MockTestExpectations):
(MockTestExpectations.init):
(MockTestExpectations.is_skip):
(ExpectationsTest):
(assert_exp):
(assert_not_exp):
(assert_bad_exp):
(assert_skip):
(test_basic):
(test_skip):
(test_flaky):
(test_build_type):

  • TestWebKitAPI/glib/TestExpectations.json: Added.
  • glib/api_test_runner.py:

(TestRunner): Remove SkippedTest implementation.
(TestRunner.init): Create a TestExpectations.
(TestRunner._test_cases_to_skip): Use TestExpectations to check skipped tests.
(TestRunner._should_run_test_program): Ditto.
(TestRunner._run_test_glib): Use TestExpectations to check if test suite is slow.
(TestRunner._run_test_glib.parse_line.set_test_result): Register also tests passing.
(TestRunner._run_google_test): Use TestExpectations to check if test cases is slow and register tests passing.
(TestRunner.run_tests): Check if actual result is the expected one and register also unexpected passes.
(TestRunner.run_tests.report): Helper to write report to stdout.

Location:
trunk/Tools
Files:
3 added
4 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r226940 r226944  
     12018-01-15  Carlos Garcia Campos  <cgarcia@igalia.com>
     2
     3        [GTK][WPE] Add support for unit test expectations
     4        https://bugs.webkit.org/show_bug.cgi?id=181589
     5
     6        Reviewed by Michael Catanzaro.
     7
     8        We currently have a way to skip tests by annotating them in the api test runner script. The main problem of this
     9        approach is that we skip tests when they fail in the bots and we never notice if they stop failing, keeping the
     10        tests skipped forever. This is indeed the case of several WebKit2 C API tests. Annotating skipped tests in the
     11        script itself is not a good idea either.
     12
     13        This patch adds a generic TestExpectations class for simple tests based on tests with subtests, like our unit
     14        tests, but also WebDriver tests. It parses a json file with the tests and subtests expectations and provides
     15        convenient methods to query them.
     16
     17        * Scripts/run-gtk-tests:
     18        (GtkTestRunner): Remove all Skipped and Slow tests marked here.
     19        * Scripts/run-wpe-tests:
     20        (WPETestRunner): Ditto.
     21        * Scripts/webkitpy/common/test_expectations.py: Added.
     22        (TestExpectations):
     23        (TestExpectations.__init__):
     24        (TestExpectations._port_name_for_expected):
     25        (TestExpectations._expected_value):
     26        (TestExpectations.skipped_tests):
     27        (TestExpectations.skipped_subtests):
     28        (TestExpectations._expectation_value):
     29        (TestExpectations.is_slow):
     30        (TestExpectations.get_expectation):
     31        * Scripts/webkitpy/common/test_expectations_unittest.py: Added.
     32        (MockTestExpectations):
     33        (MockTestExpectations.__init__):
     34        (MockTestExpectations.is_skip):
     35        (ExpectationsTest):
     36        (assert_exp):
     37        (assert_not_exp):
     38        (assert_bad_exp):
     39        (assert_skip):
     40        (test_basic):
     41        (test_skip):
     42        (test_flaky):
     43        (test_build_type):
     44        * TestWebKitAPI/glib/TestExpectations.json: Added.
     45        * glib/api_test_runner.py:
     46        (TestRunner): Remove SkippedTest implementation.
     47        (TestRunner.__init__): Create a TestExpectations.
     48        (TestRunner._test_cases_to_skip): Use TestExpectations to check skipped tests.
     49        (TestRunner._should_run_test_program): Ditto.
     50        (TestRunner._run_test_glib): Use TestExpectations to check if test suite is slow.
     51        (TestRunner._run_test_glib.parse_line.set_test_result): Register also tests passing.
     52        (TestRunner._run_google_test): Use TestExpectations to check if test cases is slow and register tests passing.
     53        (TestRunner.run_tests): Check if actual result is the expected one and register also unexpected passes.
     54        (TestRunner.run_tests.report): Helper to write report to stdout.
     55
    1562018-01-13  Mark Lam  <mark.lam@apple.com>
    257

  • trunk/Tools/Scripts/run-gtk-tests

    r226838 r226944  
    3030import common
    3131import jhbuildutils
    32 from api_test_runner import TestRunner, SkippedTest, add_options
     32from api_test_runner import TestRunner, add_options
    3333
    3434class GtkTestRunner(TestRunner):
    3535    TestRunner.TEST_DIRS = [ "WebKit2Gtk", "WebKit", "JavaScriptCore", "WTF", "WebCore" ]
    36 
    37     TestRunner.SKIPPED = [
    38         SkippedTest("WebKit2Gtk/TestUIClient", "/webkit2/WebKitWebView/mouse-target", "Test times out after r150890", 117689),
    39         SkippedTest("WebKit2Gtk/TestUIClient", "/webkit2/WebKitWebView/usermedia-permission-requests", "Test times out", 158257),
    40         SkippedTest("WebKit2Gtk/TestUIClient", "/webkit2/WebKitWebView/audio-usermedia-permission-request", "Test times out", 158257),
    41         SkippedTest("WebKit2Gtk/TestCookieManager", "/webkit2/WebKitCookieManager/persistent-storage", "Test is flaky", 134580),
    42         SkippedTest("WebKit2Gtk/TestPrinting", "/webkit2/WebKitPrintOperation/custom-widget", "Test is flaky", 168196),
    43         SkippedTest("WebKit2Gtk/TestWebViewEditor", "/webkit2/WebKitWebView/editable/editable", "Test hits an assertion in Debug builds", 151654, "Debug"),
    44         SkippedTest("WebKit2Gtk/TestWebExtensions", "/webkit2/WebKitWebView/install-missing-plugins-permission-request", "Test times out", 147822),
    45         SkippedTest("WebKit2Gtk/TestWebsiteData", "/webkit2/WebKitWebsiteData/databases", "Test fails to clear database data", 181251),
    46         SkippedTest("WebKit/TestWebKit", "WebKit.MouseMoveAfterCrash", "Test is flaky", 85066),
    47         SkippedTest("WebKit/TestWebKit", "WebKit.NewFirstVisuallyNonEmptyLayoutForImages", "Test is flaky", 85066),
    48         SkippedTest("WebKit/TestWebKit", "WebKit.NewFirstVisuallyNonEmptyLayoutFrames", "Test fails", 85037),
    49         SkippedTest("WebKit/TestWebKit", "WebKit.SpacebarScrolling", "Test fails", 84961),
    50         SkippedTest("WebKit/TestWebKit", "WebKit.WKConnection", "Tests fail and time out out", 84959),
    51         SkippedTest("WebKit/TestWebKit", "WebKit.ForceRepaint", "Test times out", 105532),
    52         SkippedTest("WebKit/TestWebKit", "WebKit.ReloadPageAfterCrash", "Test flakily times out", 110129),
    53         SkippedTest("WebKit/TestWebKit", "WebKit.DidAssociateFormControls", "Test times out", 120302),
    54         SkippedTest("WebKit/TestWebKit", "WebKit.InjectedBundleFrameHitTest", "Test times out", 120303),
    55         SkippedTest("WebKit/TestWebKit", "WebKit.TerminateTwice", "Test causes crash on the next test", 121970),
    56         SkippedTest("WebKit/TestWebKit", "WebKit.GeolocationTransitionToHighAccuracy", "Test causes crash on the next test", 125068),
    57         SkippedTest("WebKit/TestWebKit", "WebKit.GeolocationTransitionToLowAccuracy", "Test causes crash on the next test", 125068),
    58     ]
    59 
    60     TestRunner.SLOW = [
    61         "WTF_Lock.ContendedShortSection",
    62         "WTF_Lock.ContendedLongSection",
    63         "WTF_WordLock.ContendedShortSection",
    64         "WTF_WordLock.ContendedLongSection",
    65         "WebKit2Gtk/TestInspectorServer",
    66     ]
    6736
    6837    def __init__(self, options, tests=[]):

  • trunk/Tools/Scripts/run-wpe-tests

    r225047 r226944  
    2727sys.path.insert(0, os.path.join(top_level_directory, "Tools", "glib"))
    2828import jhbuildutils
    29 from api_test_runner import TestRunner, SkippedTest, add_options
     29from api_test_runner import TestRunner, add_options
    3030
    3131class WPETestRunner(TestRunner):
    3232    TestRunner.TEST_DIRS = [ "WPE", "WebKit", "JavaScriptCore", "WTF", "WebCore" ]
    33 
    34     TestRunner.SLOW = [
    35         "WTF_Lock.ContendedShortSection",
    36         "WTF_Lock.ContendedLongSection",
    37         "WTF_WordLock.ContendedShortSection",
    38         "WTF_WordLock.ContendedLongSection",
    39     ]
    4033
    4134    def __init__(self, options, tests=[]):

  • trunk/Tools/glib/api_test_runner.py

    r225047 r226944  
    2929import common
    3030from webkitpy.common.host import Host
    31 
    32 
    33 class SkippedTest:
    34     ENTIRE_SUITE = None
    35 
    36     def __init__(self, test, test_case, reason, bug, build_type=None):
    37         self.test = test
    38         self.test_case = test_case
    39         self.reason = reason
    40         self.bug = bug
    41         self.build_type = build_type
    42 
    43     def __str__(self):
    44         skipped_test_str = "%s" % self.test
    45 
    46         if not(self.skip_entire_suite()):
    47             skipped_test_str += " [%s]" % self.test_case
    48 
    49         skipped_test_str += ": %s (https://bugs.webkit.org/show_bug.cgi?id=%d)" % (self.reason, self.bug)
    50         return skipped_test_str
    51 
    52     def skip_entire_suite(self):
    53         return self.test_case == SkippedTest.ENTIRE_SUITE
    54 
    55     def skip_for_build_type(self, build_type):
    56         if self.build_type is None:
    57             return True
    58 
    59         return self.build_type == build_type
     31from webkitpy.common.test_expectations import TestExpectations
    6032
    6133
    … …  
    6638class TestRunner(object):
    6739    TEST_DIRS = []
    68     SKIPPED = []
    69     SLOW = []
    7040
    7141    def __init__(self, port, options, tests=[]):
    … …  
    7848
    7949        self._programs_path = common.binary_build_path()
     50        expectations_file = os.path.join(common.top_level_path(), "Tools", "TestWebKitAPI", "glib", "TestExpectations.json")
     51        self._expectations = TestExpectations(self._port.name(), expectations_file, self._build_type)
    8052        self._tests = self._get_tests(tests)
    81         self._skipped_tests = [skipped for skipped in TestRunner.SKIPPED if skipped.skip_for_build_type(self._build_type)]
    8253        self._disabled_tests = []
    8354
    … …  
    137108            return []
    138109
    139         test_cases = []
    140         for skipped in self._skipped_tests:
    141             if test_program.endswith(skipped.test) and not skipped.skip_entire_suite():
    142                 test_cases.append(skipped.test_case)
    143         return test_cases
     110        return self._expectations.skipped_subtests(os.path.basename(test_program))
    144111
    145112    def _should_run_test_program(self, test_program):
    … …  
    151118            return True
    152119
    153         for skipped in self._skipped_tests:
    154             if test_program.endswith(skipped.test) and skipped.skip_entire_suite():
    155                 return False
    156         return True
     120        return os.path.basename(test_program) not in self._expectations.skipped_tests()
    157121
    158122    def _kill_process(self, pid):
    … …  
    212176        timeout = self._options.timeout
    213177        test = os.path.join(os.path.basename(os.path.dirname(test_program)), os.path.basename(test_program))
    214         if test in TestRunner.SLOW:
     178        if self._expectations.is_slow(os.path.basename(test_program)):
    215179            timeout *= 5
    216180
    … …  
    233197                    else:
    234198                        test_context[test] = result
     199                else:
     200                    test_context[test] = 'PASS'
    235201                test_context["did-timeout"] = False
    236202                test_context["current_test"] = None
    … …  
    303269        command = [test_program, '--gtest_filter=%s' % (subtest)]
    304270        timeout = self._options.timeout
    305         if subtest in TestRunner.SLOW:
     271        if self._expectations.is_slow(os.path.basename(test_program), subtest):
    306272            timeout *= 5
    307273
    … …  
    329295            return {subtest: "FAIL"}
    330296
    331         return {}
     297        return {subtest: "PASS"}
    332298
    333299    def _run_google_test_suite(self, test_program):
    … …  
    368334        failed_tests = {}
    369335        timed_out_tests = {}
     336        passed_tests = {}
    370337        try:
    371338            for test in self._tests:
    372339                results = self._run_test(test)
    373340                for test_case, result in results.iteritems():
     341                    if result in self._expectations.get_expectation(os.path.basename(test), test_case):
     342                        continue
     343
    374344                    if result == "FAIL":
    375345                        failed_tests.setdefault(test, []).append(test_case)
    … …  
    378348                    elif result == "CRASH":
    379349                        crashed_tests.setdefault(test, []).append(test_case)
     350                    elif result == "PASS":
     351                        passed_tests.setdefault(test, []).append(test_case)
    380352        finally:
    381353            self._tear_down_testing_environment()
    382354
    383         if failed_tests:
    384             sys.stdout.write("\nUnexpected failures (%d)\n" % (sum(len(value) for value in failed_tests.itervalues())))
    385             for test in failed_tests:
    386                 sys.stdout.write("    %s\n" % (test.replace(self._test_programs_base_dir(), '', 1)))
    387                 for test_case in failed_tests[test]:
     355        def report(tests, title, base_dir):
     356            if not tests:
     357                return
     358            sys.stdout.write("\nUnexpected %s (%d)\n" % (title, sum(len(value) for value in tests.itervalues())))
     359            for test in tests:
     360                sys.stdout.write("    %s\n" % (test.replace(base_dir, '', 1)))
     361                for test_case in tests[test]:
    388362                    sys.stdout.write("        %s\n" % (test_case))
    389363            sys.stdout.flush()
    390364
    391         if crashed_tests:
    392             sys.stdout.write("\nUnexpected crashes (%d)\n" % (sum(len(value) for value in crashed_tests.itervalues())))
    393             for test in crashed_tests:
    394                 sys.stdout.write("    %s\n" % (test.replace(self._test_programs_base_dir(), '', 1)))
    395                 for test_case in crashed_tests[test]:
    396                     sys.stdout.write("        %s\n" % (test_case))
    397             sys.stdout.flush()
    398 
    399         if timed_out_tests:
    400             sys.stdout.write("\nUnexpected timeouts (%d)\n" % (sum(len(value) for value in timed_out_tests.itervalues())))
    401             for test in timed_out_tests:
    402                 sys.stdout.write("    %s\n" % (test.replace(self._test_programs_base_dir(), '', 1)))
    403                 for test_case in timed_out_tests[test]:
    404                     sys.stdout.write("        %s\n" % (test_case))
    405             sys.stdout.flush()
     365        report(failed_tests, "failures", self._test_programs_base_dir())
     366        report(crashed_tests, "crashes", self._test_programs_base_dir())
     367        report(timed_out_tests, "timeouts", self._test_programs_base_dir())
     368        report(passed_tests, "passes", self._test_programs_base_dir())
    406369
    407370        return len(failed_tests) + len(timed_out_tests)
Note: See TracChangeset for help on using the changeset viewer.