Changeset 235564 in webkit


Ignore:
Timestamp:
Aug 31, 2018 1:00:22 PM (6 years ago)
Author:
Simon Fraser
Message:

Make it possible to mark tests as leaks in TestExpectations
​https://bugs.webkit.org/show_bug.cgi?id=189088

Reviewed by Jon Lee.

Have webkitpy parse out "Leak" expectations in TestExpectations, and do the right
thing if the test run did not use --world-leaks. Add unit tests for leaks combined
with various other result combinations.

  • Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py:

(LayoutTestRunner._update_summary_with_result):
(LayoutTestRunner._annotate_results_with_additional_failures):

  • Scripts/webkitpy/layout_tests/models/test_expectations.py:

(TestExpectationParser):
(TestExpectations): The 'Leak' line was duplicated here, so remove a copy.
(TestExpectations.remove_leak_failures):
(TestExpectations.matches_an_expected_result):

  • Scripts/webkitpy/layout_tests/models/test_expectations_unittest.py:

(Base.get_basic_tests):

  • Scripts/webkitpy/port/test.py:

(TestList.add_reftest):
(unit_test_list):

Location:
trunk/Tools
Files:
7 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r235561 r235564  
     12018-08-31  Simon Fraser  <simon.fraser@apple.com>
     2
     3        Make it possible to mark tests as leaks in TestExpectations
     4        https://bugs.webkit.org/show_bug.cgi?id=189088
     5
     6        Reviewed by Jon Lee.
     7       
     8        Have webkitpy parse out "Leak" expectations in TestExpectations, and do the right
     9        thing if the test run did not use --world-leaks. Add unit tests for leaks combined
     10        with various other result combinations.
     11
     12        * Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py:
     13        (LayoutTestRunner._update_summary_with_result):
     14        (LayoutTestRunner._annotate_results_with_additional_failures):
     15        * Scripts/webkitpy/layout_tests/models/test_expectations.py:
     16        (TestExpectationParser):
     17        (TestExpectations): The 'Leak' line was duplicated here, so remove a copy.
     18        (TestExpectations.remove_leak_failures):
     19        (TestExpectations.matches_an_expected_result):
     20        * Scripts/webkitpy/layout_tests/models/test_expectations_unittest.py:
     21        (Base.get_basic_tests):
     22        * Scripts/webkitpy/port/test.py:
     23        (TestList.add_reftest):
     24        (unit_test_list):
     25
    1262018-08-31  Zalan Bujtas  <zalan@apple.com>
    227

  • trunk/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py

    r235467 r235564  
    183183            expected = True
    184184        else:
    185             expected = self._expectations.matches_an_expected_result(result.test_name, result.type, self._options.pixel_tests or result.reftest_type)
     185            expected = self._expectations.matches_an_expected_result(result.test_name, result.type, self._options.pixel_tests or result.reftest_type, self._options.world_leaks)
    186186            exp_str = self._expectations.model().get_expectations_string(result.test_name)
    187187            got_str = self._expectations.model().expectation_to_string(result.type)
    … …  
    199199            # and we don't want to modify the results of a test that didn't run. existing_result.test_number is only non-None for tests that ran.
    200200            if existing_result and existing_result.test_number is not None:
    201                 was_expected = self._expectations.matches_an_expected_result(new_result.test_name, existing_result.type, self._options.pixel_tests or existing_result.reftest_type)
    202                 now_expected = self._expectations.matches_an_expected_result(new_result.test_name, new_result.type, self._options.pixel_tests or new_result.reftest_type)
     201                was_expected = self._expectations.matches_an_expected_result(new_result.test_name, existing_result.type, self._options.pixel_tests or existing_result.reftest_type, self._options.world_leaks)
     202                now_expected = self._expectations.matches_an_expected_result(new_result.test_name, new_result.type, self._options.pixel_tests or new_result.reftest_type, self._options.world_leaks)
    203203                run_results.change_result_to_failure(existing_result, new_result, was_expected, now_expected)
    204204

  • trunk/Tools/Scripts/webkitpy/layout_tests/models/test_expectations.py

    r235467 r235564  
    242242    _expectation_tokens = {
    243243        'Crash': 'CRASH',
    244         'Leak': 'LEAK',
    245244        'Failure': 'FAIL',
    246245        'ImageOnlyFailure': 'IMAGE',
     246        'Leak': 'LEAK',
    247247        'Missing': 'MISSING',
    248         'Leak': 'LEAK',
    249248        'Pass': 'PASS',
    250249        'Rebaseline': 'REBASELINE',
    … …  
    405404        self.not_applicable_to_current_platform = False
    406405
     406    def __str__(self):
     407        return self.to_string(None)
     408
    407409    def is_invalid(self):
    408410        return self.warnings and self.warnings != [TestExpectationParser.MISSING_BUG_WARNING]
    … …  
    864866        if IMAGE in expected_results:
    865867            expected_results.remove(IMAGE)
    866             expected_results.add(PASS)
     868            expected_results.add(PASS) # FIXME: does it always become a pass?
     869        return expected_results
     870
     871    @staticmethod
     872    def remove_leak_failures(expected_results):
     873        """Returns a copy of the expected results for a test, except that we
     874        drop any leak failures and return the remaining expectations. For example,
     875        if we're not running with --world-leaks, then tests expected to fail as LEAK
     876        will PASS."""
     877        expected_results = expected_results.copy()
     878        if LEAK in expected_results:
     879            expected_results.remove(LEAK)
     880            if not expected_results:
     881                expected_results.add(PASS)
    867882        return expected_results
    868883
    … …  
    951966        return self._model.get_test_set(REBASELINE)
    952967
    953     def matches_an_expected_result(self, test, result, pixel_tests_are_enabled):
     968    def matches_an_expected_result(self, test, result, pixel_tests_are_enabled, world_leaks_are_enabled):
    954969        expected_results = self._model.get_expectations(test)
    955970        if not pixel_tests_are_enabled:
    956971            expected_results = self.remove_pixel_failures(expected_results)
     972
     973        if not world_leaks_are_enabled:
     974            expected_results = self.remove_leak_failures(expected_results)
     975
    957976        return self.result_was_expected(result,
    958977                                   expected_results,

  • trunk/Tools/Scripts/webkitpy/layout_tests/models/test_expectations_unittest.py

    r235467 r235564  
    5555                'failures/expected/crash.html',
    5656                'failures/expected/leak.html',
     57                'failures/expected/flaky-leak.html',
    5758                'failures/expected/missing_text.html',
    5859                'failures/expected/image.html',
     60                'failures/expected/reftest.html',
     61                'failures/expected/leaky-reftest.html',
    5962                'passes/text.html']
    6063
    … …  
    6366Bug(test) failures/expected/text.html [ Failure ]
    6467Bug(test) failures/expected/crash.html [ WontFix ]
    65 Bug(test) failures/expected/crash.html [ Leak ]
     68Bug(test) failures/expected/leak.html [ Leak ]
     69Bug(test) failures/expected/flaky-leak.html [ Failure Leak ]
    6670Bug(test) failures/expected/missing_image.html [ Rebaseline Missing ]
    6771Bug(test) failures/expected/image_checksum.html [ WontFix ]
    6872Bug(test) failures/expected/image.html [ WontFix Mac ]
     73Bug(test) failures/expected/reftest.html [ ImageOnlyFailure ]
     74Bug(test) failures/expected/leaky-reftest.html [ ImageOnlyFailure Leak ]
    6975"""
    7076
    … …  
    8086
    8187    def assert_exp(self, test, result):
    82         self.assertEqual(self._exp.model().get_expectations(test),
    83                           set([result]))
     88        self.assertEqual(self._exp.model().get_expectations(test), set([result]))
     89
     90    def assert_exp_set(self, test, result_set):
     91        self.assertEqual(self._exp.model().get_expectations(test), result_set)
    8492
    8593    def assert_bad_expectations(self, expectations, overrides=None):
    … …  
    92100        self.assert_exp('failures/expected/text.html', FAIL)
    93101        self.assert_exp('failures/expected/image_checksum.html', PASS)
    94         self.assert_exp('failures/expected/leak.html', PASS)
     102        self.assert_exp('failures/expected/reftest.html', IMAGE)
     103        self.assert_exp_set('failures/expected/leaky-reftest.html', set([IMAGE, LEAK]))
     104        self.assert_exp('failures/expected/leak.html', LEAK)
     105        self.assert_exp_set('failures/expected/flaky-leak.html', set([FAIL, LEAK]))
    95106        self.assert_exp('passes/text.html', PASS)
     107        # self.assert_exp_set('passes/flaky-leak.html', set([PASS, LEAK]))
    96108        self.assert_exp('failures/expected/image.html', PASS)
    97109
    … …  
    123135        self.assertEqual(TestExpectations.remove_pixel_failures(set([FAIL])), set([FAIL]))
    124136        self.assertEqual(TestExpectations.remove_pixel_failures(set([PASS, IMAGE, CRASH])), set([PASS, CRASH]))
     137
     138    def test_remove_leak_failures(self):
     139        self.assertEqual(TestExpectations.remove_leak_failures(set([FAIL])), set([FAIL]))
     140        self.assertEqual(TestExpectations.remove_leak_failures(set([PASS])), set([PASS]))
     141        self.assertEqual(TestExpectations.remove_leak_failures(set([LEAK])), set([PASS]))
     142        self.assertEqual(TestExpectations.remove_leak_failures(set([PASS, LEAK])), set([PASS]))
     143        self.assertEqual(TestExpectations.remove_leak_failures(set([FAIL, LEAK])), set([FAIL]))
     144        self.assertEqual(TestExpectations.remove_leak_failures(set([PASS, IMAGE, LEAK, CRASH])), set([PASS, IMAGE, CRASH]))
    125145
    126146    def test_suffixes_for_expectations(self):
    … …  
    153173        # Normal cases are handled by other tests.
    154174        self.parse_exp(self.get_basic_expectations())
    155         self.assertRaises(ValueError, self._exp.model().expectation_to_string,
    156                           -1)
     175        self.assertRaises(ValueError, self._exp.model().expectation_to_string, -1)
    157176
    158177    def test_get_test_set(self):
    … …  
    216235    def test_pixel_tests_flag(self):
    217236        def match(test, result, pixel_tests_enabled):
    218             return self._exp.matches_an_expected_result(
    219                 test, result, pixel_tests_enabled)
     237            return self._exp.matches_an_expected_result(test, result, pixel_tests_enabled, False)
    220238
    221239        self.parse_exp(self.get_basic_expectations())
    222         self.assertTrue(match('failures/expected/text.html', FAIL, True))
    223         self.assertTrue(match('failures/expected/text.html', FAIL, False))
    224         self.assertFalse(match('failures/expected/text.html', CRASH, True))
    225         self.assertFalse(match('failures/expected/text.html', CRASH, False))
    226         self.assertTrue(match('failures/expected/image_checksum.html', PASS,
    227                               True))
    228         self.assertTrue(match('failures/expected/image_checksum.html', PASS,
    229                               False))
    230         self.assertTrue(match('failures/expected/crash.html', PASS, False))
    231         self.assertTrue(match('passes/text.html', PASS, False))
     240        pixel_tests_enabled = True
     241        pixel_tests_disabled = False
     242        self.assertTrue(match('failures/expected/text.html', FAIL, pixel_tests_enabled))
     243        self.assertTrue(match('failures/expected/text.html', FAIL, pixel_tests_disabled))
     244        self.assertFalse(match('failures/expected/text.html', CRASH, pixel_tests_enabled))
     245        self.assertFalse(match('failures/expected/text.html', CRASH, pixel_tests_disabled))
     246        self.assertTrue(match('failures/expected/image_checksum.html', PASS, pixel_tests_enabled))
     247        self.assertTrue(match('failures/expected/image_checksum.html', PASS, pixel_tests_disabled))
     248        self.assertTrue(match('failures/expected/crash.html', PASS, pixel_tests_disabled))
     249        self.assertTrue(match('passes/text.html', PASS, pixel_tests_disabled))
     250
     251    def test_world_leaks_flag(self):
     252        def match(test, result, pixel_tests_enabled, world_leaks_enabled):
     253            return self._exp.matches_an_expected_result(test, result, pixel_tests_enabled, world_leaks_enabled)
     254
     255        pixel_tests_enabled = True
     256        pixel_tests_disabled = False
     257        world_leaks_enabled = True
     258        world_leaks_disabled = False
     259
     260        self.parse_exp(self.get_basic_expectations())
     261        self.assertTrue(match('failures/expected/leak.html', LEAK, pixel_tests_enabled, world_leaks_enabled))
     262        self.assertTrue(match('failures/expected/leak.html', PASS, pixel_tests_enabled, world_leaks_disabled))
     263        self.assertTrue(match('failures/expected/flaky-leak.html', FAIL, pixel_tests_enabled, world_leaks_disabled))
     264
     265        self.assertTrue(match('failures/expected/leaky-reftest.html', LEAK, pixel_tests_disabled, world_leaks_enabled))
     266        self.assertTrue(match('failures/expected/leaky-reftest.html', PASS, pixel_tests_disabled, world_leaks_disabled))
     267
     268        self.assertTrue(match('failures/expected/leaky-reftest.html', IMAGE, pixel_tests_enabled, world_leaks_enabled))
     269        self.assertTrue(match('failures/expected/leaky-reftest.html', LEAK, pixel_tests_enabled, world_leaks_enabled))
     270        self.assertTrue(match('failures/expected/leaky-reftest.html', IMAGE, pixel_tests_enabled, world_leaks_disabled))
     271
     272        self.assertFalse(match('failures/expected/text.html', PASS, pixel_tests_enabled, world_leaks_enabled))
     273        self.assertFalse(match('failures/expected/text.html', CRASH, pixel_tests_enabled, world_leaks_disabled))
     274        self.assertTrue(match('passes/text.html', PASS, pixel_tests_enabled, world_leaks_disabled))
    232275
    233276    def test_more_specific_override_resets_skip(self):
    … …  
    348391    def test_slow(self):
    349392        self.assert_tokenize_exp('foo.html [ Slow ]', modifiers=['SLOW'], expectations=['PASS'])
     393
     394    def test_leak(self):
     395        self.assert_tokenize_exp('foo.html [ Leak ]', modifiers=[], expectations=['LEAK'])
    350396
    351397    def test_wontfix(self):

  • trunk/Tools/Scripts/webkitpy/layout_tests/models/test_results.py

    r235467 r235564  
    6969        return self.test_name.__hash__()
    7070
     71    def __str__(self):
     72        result = self.test_name
     73        if len(self.failures):
     74            result += ' failures: ' + ','.join([failure.message() for failure in self.failures])
     75        if len(self.reftest_type):
     76            result += ' reftest_type: ' + ','.join(self.reftest_type)
     77        return result
     78
    7179    def convert_to_failure(self, failure_result):
    7280        if self.type is failure_result.type:

  • trunk/Tools/Scripts/webkitpy/layout_tests/models/test_run_results.py

    r235467 r235564  
    119119            if had_failures:
    120120                self.unexpected_failures -= 1
    121         else:
     121        elif existing_expected and not new_expected:
    122122            # test changed from expected to unexpected
    123123            self.expected -= 1

  • trunk/Tools/Scripts/webkitpy/port/test.py

    r235467 r235564  
    7979        self.tests[name] = test
    8080
    81     def add_reftest(self, name, reference_name, same_image):
     81    def add_reftest(self, name, reference_name, same_image, **kwargs):
    8282        self.add(name, actual_checksum='xxx', actual_image='XXX', is_reftest=True)
    8383        if same_image:
    … …  
    8686            self.add(reference_name, actual_checksum='yyy', actual_image='YYY', is_reftest=True)
    8787
     88        if kwargs:
     89            test = self.tests[name]
     90            for key, value in kwargs.items():
     91                test.__dict__[key] = value
     92
    8893    def keys(self):
    8994        return self.tests.keys()
    … …  
    98103# These numbers may need to be updated whenever we add or delete tests.
    99104#
    100 TOTAL_TESTS = 74
     105TOTAL_TESTS = 76
    101106TOTAL_SKIPS = 9
    102107TOTAL_RETRIES = 15
    … …  
    117122    tests.add('failures/expected/missing_text.html', expected_text=None)
    118123    tests.add('failures/expected/leak.html', leak=True)
     124    tests.add('failures/expected/flaky-leak.html', leak=True)
    119125    tests.add('failures/expected/image.html',
    120126              actual_image='image_fail-pngtEXtchecksum\x00checksum_fail',
    … …  
    216222    tests.add_reftest('passes/phpreftest.php', 'passes/phpreftest-expected-mismatch.svg', same_image=False)
    217223    tests.add_reftest('failures/expected/reftest.html', 'failures/expected/reftest-expected.html', same_image=False)
     224    tests.add_reftest('failures/expected/leaky-reftest.html', 'failures/expected/leaky-reftest-expected.html', same_image=False, leak=True)
    218225    tests.add_reftest('failures/expected/mismatch.html', 'failures/expected/mismatch-expected-mismatch.html', same_image=True)
    219226    tests.add_reftest('failures/unexpected/reftest.html', 'failures/unexpected/reftest-expected.html', same_image=False)
    … …  
    284291Bug(test) failures/expected/crash.html [ Crash ]
    285292Bug(test) failures/expected/leak.html [ Leak ]
     293Bug(test) failures/expected/flaky-leak.html [ Failure Leak ]
     294Bug(test) failures/expected/leaky-reftest.html [ ImageOnlyFailure Leak ]
    286295Bug(test) failures/expected/image.html [ ImageOnlyFailure ]
    287296Bug(test) failures/expected/audio.html [ Failure ]
    … …  
    600609
    601610        test_world_leaks_output = """TEST: file:///test.checkout/LayoutTests/failures/expected/leak.html
    602 ABANDONED DOCUMENT: file:///test.checkout/LayoutTests//failures/expected/leak.html
     611ABANDONED DOCUMENT: file:///test.checkout/LayoutTests/failures/expected/leak.html
    603612TEST: file:///test.checkout/LayoutTests/failures/unexpected/leak.html
    604 ABANDONED DOCUMENT: file:///test.checkout/LayoutTests//failures/expected/leak.html
     613ABANDONED DOCUMENT: file:///test.checkout/LayoutTests/failures/expected/flaky-leak.html
     614TEST: file:///test.checkout/LayoutTests/failures/unexpected/flaky-leak.html
     615ABANDONED DOCUMENT: file:///test.checkout/LayoutTests/failures/expected/leak.html
    605616TEST: file:///test.checkout/LayoutTests/failures/unexpected/leak.html
    606 ABANDONED DOCUMENT: file:///test.checkout/LayoutTests//failures/expected/leak-subframe.html"""
     617ABANDONED DOCUMENT: file:///test.checkout/LayoutTests/failures/expected/leak-subframe.html
     618TEST: file:///test.checkout/LayoutTests/failures/expected/leaky-reftest.html
     619ABANDONED DOCUMENT: file:///test.checkout/LayoutTests/failures/expected/leaky-reftest.html"""
    607620        return self._parse_world_leaks_output(test_world_leaks_output)
    608621
Note: See TracChangeset for help on using the changeset viewer.