Changeset 129184 in webkit


Ignore:
Timestamp:
Sep 20, 2012 6:33:38 PM (12 years ago)
Author:
dpranke@chromium.org
Message:

make Skip, WontFix be the only expectations on a line
​https://bugs.webkit.org/show_bug.cgi?id=97225

Reviewed by Ojan Vafai.

It is now incorrect in the new syntax to have a line like:

foo.html [ WontFix Crash ]

This will generate a lint warning and be treated as an invalid
line. Fixing this caused a whole bunch of unit tests to need updating
to no longer be marked as WontFix :). Also, this patch adjusts
the warnings so that missing Bug() identifiers will cause lint
warnings but will *not* cause the line to be treated as invalid.
Fixing these issues also revealed that test_hung_thread was no
longer testing the right logic, so I adjusted the timeouts in
test.py to make that test work again.

  • Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py:

(Worker._run_test_in_another_thread):

  • Scripts/webkitpy/layout_tests/controllers/manager_unittest.py:

(ResultSummaryTest.test_summarized_results_wontfix):

  • Scripts/webkitpy/layout_tests/models/test_expectations.py:

(TestExpectationParser._parse_modifiers):
(TestExpectationParser._tokenize_line_using_new_format):
(TestExpectationLine.is_invalid):

  • Scripts/webkitpy/layout_tests/models/test_expectations_unittest.py:

(BasicTests.test_basic):
(test_get_test_set):
(test_parse_warning):
(test_pixel_tests_flag):
(SemanticTests.test_missing_bugid):
(SemanticTests):
(SemanticTests.test_skip_and_wontfix):

  • Scripts/webkitpy/layout_tests/port/test.py:

(TestDriver.run_test):

  • Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py:

(MainTest.test_hung_thread):

  • Scripts/webkitpy/tool/commands/queries_unittest.py:

(PrintExpectationsTest.test_basic):
(PrintExpectationsTest.test_multiple):
(PrintExpectationsTest.test_full):
(PrintExpectationsTest.test_exclude):
(PrintExpectationsTest.test_csv):

Location:
trunk/Tools
Files:
8 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r129181 r129184  
     12012-09-20  Dirk Pranke  <dpranke@chromium.org>
     2
     3        make Skip, WontFix be the only expectations on a line
     4        https://bugs.webkit.org/show_bug.cgi?id=97225
     5
     6        Reviewed by Ojan Vafai.
     7
     8        It is now incorrect in the new syntax to have a line like:
     9
     10            foo.html [ WontFix Crash ]
     11
     12        This will generate a lint warning and be treated as an invalid
     13        line. Fixing this caused a whole bunch of unit tests to need updating
     14        to no longer be marked as WontFix :). Also, this patch adjusts
     15        the warnings so that missing Bug() identifiers will cause lint
     16        warnings but will *not* cause the line to be treated as invalid.
     17        Fixing these issues also revealed that test_hung_thread was no
     18        longer testing the right logic, so I adjusted the timeouts in
     19        test.py to make that test work again.
     20
     21        * Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py:
     22        (Worker._run_test_in_another_thread):
     23        * Scripts/webkitpy/layout_tests/controllers/manager_unittest.py:
     24        (ResultSummaryTest.test_summarized_results_wontfix):
     25        * Scripts/webkitpy/layout_tests/models/test_expectations.py:
     26        (TestExpectationParser._parse_modifiers):
     27        (TestExpectationParser._tokenize_line_using_new_format):
     28        (TestExpectationLine.is_invalid):
     29        * Scripts/webkitpy/layout_tests/models/test_expectations_unittest.py:
     30        (BasicTests.test_basic):
     31        (test_get_test_set):
     32        (test_parse_warning):
     33        (test_pixel_tests_flag):
     34        (SemanticTests.test_missing_bugid):
     35        (SemanticTests):
     36        (SemanticTests.test_skip_and_wontfix):
     37        * Scripts/webkitpy/layout_tests/port/test.py:
     38        (TestDriver.run_test):
     39        * Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py:
     40        (MainTest.test_hung_thread):
     41        * Scripts/webkitpy/tool/commands/queries_unittest.py:
     42        (PrintExpectationsTest.test_basic):
     43        (PrintExpectationsTest.test_multiple):
     44        (PrintExpectationsTest.test_full):
     45        (PrintExpectationsTest.test_exclude):
     46        (PrintExpectationsTest.test_csv):
     47
    1482012-09-20  Lucas Forschler  <lforschler@apple.com>
    249

  • trunk/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py

    r125812 r129184  
    421421        thread.join(thread_timeout_sec)
    422422        result = thread.result
     423        failures = []
    423424        if thread.isAlive():
    424425            # If join() returned with the thread still running, the
    … …  
    431432            # thread's results.
    432433            _log.error('Test thread hung: killing all DumpRenderTrees')
     434            failures = [test_failures.FailureTimeout()]
    433435
    434436        driver.stop()
    435437
    436438        if not result:
    437             result = test_results.TestResult(test_input.test_name, failures=[], test_run_time=0)
     439            result = test_results.TestResult(test_input.test_name, failures=failures, test_run_time=0)
    438440        return result
    439441

  • trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager_unittest.py

    r129047 r129184  
    193193        port._options.builder_name = 'dummy builder'
    194194        port._filesystem.write_text_file(port._filesystem.join(port.layout_tests_dir(), "failures/expected/wontfix.html"), "Dummy test contents")
    195         expected_results, unexpected_results = self.summarized_results(port, expected=False, passing=False, flaky=False, extra_tests=['failures/expected/wontfix.html'], extra_expectations='BUGX WONTFIX : failures/expected/wontfix.html = FAIL\n')
     195        expected_results, unexpected_results = self.summarized_results(port, expected=False, passing=False, flaky=False, extra_tests=['failures/expected/wontfix.html'], extra_expectations='Bug(x) failures/expected/wontfix.html [ WontFix ]\n')
    196196        self.assertTrue(expected_results['tests']['failures']['expected']['wontfix.html']['wontfix'])

  • trunk/Tools/Scripts/webkitpy/layout_tests/models/test_expectations.py

    r129148 r129184  
    7777    TIMEOUT_EXPECTATION = 'timeout'
    7878
     79    MISSING_BUG_WARNING = 'Test lacks BUG modifier.'
     80
    7981    def __init__(self, port, full_test_list, allow_rebaseline_modifier):
    8082        self._port = port
    … …  
    155157
    156158        if not expectation_line.parsed_bug_modifiers and not has_wontfix and not has_bugid:
    157             expectation_line.warnings.append('Test lacks BUG modifier.')
     159            expectation_line.warnings.append(self.MISSING_BUG_WARNING)
    158160
    159161        if self._allow_rebaseline_modifier and self.REBASELINE_MODIFIER in modifiers:
    … …  
    393395                warnings.append('Missing a "]"')
    394396
    395         if not expectations:
     397        if 'WONTFIX' in modifiers and 'SKIP' not in modifiers:
     398            modifiers.append('SKIP')
     399
     400        if 'SKIP' in modifiers and expectations:
     401            # FIXME: This is really a semantic warning and shouldn't be here. Remove when we drop the old syntax.
     402            warnings.append('A test marked Skip or WontFix must not have other expectations.')
     403        elif not expectations:
    396404            if 'SKIP' not in modifiers and 'REBASELINE' not in modifiers and 'SLOW' not in modifiers:
    397405                modifiers.append('SKIP')
    … …  
    432440
    433441    def is_invalid(self):
    434         return len(self.warnings) > 0
     442        return self.warnings and self.warnings != [TestExpectationParser.MISSING_BUG_WARNING]
    435443
    436444    def is_flaky(self):

  • trunk/Tools/Scripts/webkitpy/layout_tests/models/test_expectations_unittest.py

    r129052 r129184  
    6969        return """
    7070BUG_TEST : failures/expected/text.html = FAIL
    71 BUG_TEST WONTFIX SKIP : failures/expected/crash.html = CRASH
     71BUG_TEST WONTFIX : failures/expected/crash.html = PASS
    7272BUG_TEST REBASELINE : failures/expected/missing_image.html = MISSING
    73 BUG_TEST WONTFIX : failures/expected/image_checksum.html = IMAGE
    74 BUG_TEST WONTFIX MAC : failures/expected/image.html = IMAGE
     73BUG_TEST WONTFIX : failures/expected/image_checksum.html = PASS
     74BUG_TEST WONTFIX MAC : failures/expected/image.html = PASS
    7575"""
    7676
    … …  
    9595        self.parse_exp(self.get_basic_expectations())
    9696        self.assert_exp('failures/expected/text.html', FAIL)
    97         self.assert_exp('failures/expected/image_checksum.html', IMAGE)
     97        self.assert_exp('failures/expected/image_checksum.html', PASS)
    9898        self.assert_exp('passes/text.html', PASS)
    9999        self.assert_exp('failures/expected/image.html', PASS)
    … …  
    138138        # present in the expectations.
    139139        exp_str = """
    140 BUGX WONTFIX : failures/expected = IMAGE
     140BUGX WONTFIX : failures/expected = PASS
    141141"""
    142142        self.parse_exp(exp_str)
    … …  
    145145        self.assertRaises(KeyError, self._exp.get_expectations,
    146146                          unknown_test)
    147         self.assert_exp('failures/expected/crash.html', IMAGE)
     147        self.assert_exp('failures/expected/crash.html', PASS)
    148148
    149149    def test_get_modifiers(self):
    … …  
    171171            set([self.get_test('failures/expected/crash.html'),
    172172                 self.get_test('failures/expected/image_checksum.html')]))
    173         s = self._exp.get_test_set(WONTFIX, CRASH)
    174         self.assertEqual(s,
    175             set([self.get_test('failures/expected/crash.html')]))
    176         s = self._exp.get_test_set(WONTFIX, CRASH, include_skips=False)
    177         self.assertEqual(s, set([]))
    178173
    179174    def test_parse_warning(self):
    … …  
    192187            self.assertEqual(str(e), warnings)
    193188
    194         try:
    195             self.parse_exp('SKIP : failures/expected/text.html = FAIL', is_lint_mode=True)
    196             self.assertFalse(True, "ParseError wasn't raised")
    197         except ParseError, e:
    198             warnings = u'expectations:1 Test lacks BUG modifier. failures/expected/text.html'
    199             self.assertEqual(str(e), warnings)
    200 
    201189    def test_error_on_different_platform(self):
    202190        # parse_exp uses a Windows port. Assert errors on Mac show up in lint mode.
    … …  
    237225        self.assertFalse(match('failures/expected/text.html', CRASH, True))
    238226        self.assertFalse(match('failures/expected/text.html', CRASH, False))
    239         self.assertTrue(match('failures/expected/image_checksum.html', IMAGE,
     227        self.assertTrue(match('failures/expected/image_checksum.html', PASS,
    240228                              True))
    241229        self.assertTrue(match('failures/expected/image_checksum.html', PASS,
    242230                              False))
    243         self.assertTrue(match('failures/expected/crash.html', SKIP, False))
     231        self.assertTrue(match('failures/expected/crash.html', PASS, False))
    244232        self.assertTrue(match('passes/text.html', PASS, False))
    245233
    … …  
    471459        self.assertTrue(self._exp.has_warnings())
    472460
     461        self.parse_exp('failures/expected/text.html [ Failure ]')
     462        line = self._exp._model.get_expectation_line('failures/expected/text.html')
     463        self.assertFalse(line.is_invalid())
     464        self.assertEquals(line.warnings, ['Test lacks BUG modifier.'])
     465
     466    def test_skip_and_wontfix(self):
     467        # Skip and WontFix are not allowed to have other expectations as well, because those
     468        # expectations won't be exercised and may become stale .
     469        self.parse_exp('failures/expected/text.html [ Failure Skip ]')
     470        self.assertTrue(self._exp.has_warnings())
     471
     472        self.parse_exp('failures/expected/text.html [ Crash WontFix ]')
     473        self.assertTrue(self._exp.has_warnings())
     474
     475        # We can't warn against [ Pass WontFix ] until we get rid of the old syntax.
     476        self.parse_exp('failures/expected/text.html [ Pass WontFix ]')
     477        self.assertFalse(self._exp.has_warnings())
     478
    473479    def test_slow_and_timeout(self):
    474480        # A test cannot be SLOW and expected to TIMEOUT.
    … …  
    504510        exp_str = """
    505511BUGX : failures/expected/text.html = FAIL
    506 BUGX WONTFIX : failures/expected = IMAGE
     512BUGX WONTFIX : failures/expected = PASS
    507513"""
    508514        self.parse_exp(exp_str)
    509515        self.assert_exp('failures/expected/text.html', FAIL)
    510         self.assert_exp('failures/expected/crash.html', IMAGE)
     516        self.assert_exp('failures/expected/crash.html', PASS)
    511517
    512518        exp_str = """
    513 BUGX WONTFIX : failures/expected = IMAGE
     519BUGX WONTFIX : failures/expected = PASS
    514520BUGX : failures/expected/text.html = FAIL
    515521"""
    516522        self.parse_exp(exp_str)
    517523        self.assert_exp('failures/expected/text.html', FAIL)
    518         self.assert_exp('failures/expected/crash.html', IMAGE)
     524        self.assert_exp('failures/expected/crash.html', PASS)
    519525
    520526    def test_ambiguous(self):

  • trunk/Tools/Scripts/webkitpy/layout_tests/port/test.py

    r129047 r129184  
    259259    if not filesystem.exists(LAYOUT_TEST_DIR + '/platform/test/TestExpectations'):
    260260        filesystem.write_text_file(LAYOUT_TEST_DIR + '/platform/test/TestExpectations', """
    261 WONTFIX : failures/expected/crash.html = CRASH
    262 WONTFIX : failures/expected/image.html = IMAGE
    263 WONTFIX : failures/expected/audio.html = FAIL
    264 WONTFIX : failures/expected/image_checksum.html = IMAGE
    265 WONTFIX : failures/expected/mismatch.html = IMAGE
    266 WONTFIX : failures/expected/missing_check.html = MISSING PASS
    267 WONTFIX : failures/expected/missing_image.html = MISSING PASS
    268 WONTFIX : failures/expected/missing_audio.html = MISSING PASS
    269 WONTFIX : failures/expected/missing_text.html = MISSING PASS
    270 WONTFIX : failures/expected/newlines_leading.html = FAIL
    271 WONTFIX : failures/expected/newlines_trailing.html = FAIL
    272 WONTFIX : failures/expected/newlines_with_excess_CR.html = FAIL
    273 WONTFIX : failures/expected/reftest.html = IMAGE
    274 WONTFIX : failures/expected/text.html = FAIL
    275 WONTFIX : failures/expected/timeout.html = TIMEOUT
    276 WONTFIX SKIP : failures/expected/hang.html = TIMEOUT
    277 WONTFIX SKIP : failures/expected/keyboard.html = CRASH
    278 WONTFIX SKIP : failures/expected/exception.html = CRASH
    279 WONTFIX SKIP : passes/skipped/skip.html = PASS
     261Bug(test) failures/expected/crash.html [ Crash ]
     262Bug(test) failures/expected/image.html [ ImageOnlyFailure ]
     263Bug(test) failures/expected/audio.html [ Failure ]
     264Bug(test) failures/expected/image_checksum.html [ ImageOnlyFailure ]
     265Bug(test) failures/expected/mismatch.html [ ImageOnlyFailure ]
     266Bug(test) failures/expected/missing_check.html [ Missing Pass ]
     267Bug(test) failures/expected/missing_image.html [ Missing Pass ]
     268Bug(test) failures/expected/missing_audio.html [ Missing Pass ]
     269Bug(test) failures/expected/missing_text.html [ Missing Pass ]
     270Bug(test) failures/expected/newlines_leading.html [ Failure ]
     271Bug(test) failures/expected/newlines_trailing.html [ Failure ]
     272Bug(test) failures/expected/newlines_with_excess_CR.html [ Failure ]
     273Bug(test) failures/expected/reftest.html [ ImageOnlyFailure ]
     274Bug(test) failures/expected/text.html [ Failure ]
     275Bug(test) failures/expected/timeout.html [ Timeout ]
     276Bug(test) failures/expected/hang.html [ WontFix ]
     277Bug(test) failures/expected/keyboard.html [ WontFix ]
     278Bug(test) failures/expected/exception.html [ WontFix ]
     279Bug(test) passes/skipped/skip.html [ Skip ]
    280280""")
    281281
    … …  
    538538            raise ValueError('exception from ' + test_name)
    539539        if test.hang:
    540             time.sleep((float(test_input.timeout) * 4) / 1000.0)
     540            time.sleep((float(test_input.timeout) * 4) / 1000.0 + 1.0)  # The 1.0 comes from thread_padding_sec in layout_test_runnery.
    541541
    542542        audio = None

  • trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py

    r129148 r129184  
    348348                                          'failures/expected/hang.html'],
    349349                                          tests_included=True)
    350         self.assertEqual(res, 0)
     350        # Note that hang.html is marked as WontFix and all WontFix tests are
     351        # expected to Pass, so that actually running them generates an "unexpected" error.
     352        self.assertEqual(res, 1)
    351353        self.assertNotEmpty(out)
    352354        self.assertNotEmpty(err)

  • trunk/Tools/Scripts/webkitpy/tool/commands/queries_unittest.py

    r129052 r129184  
    179179        self.run_test(['failures/expected/text.html', 'failures/expected/image.html'],
    180180                      ('// For test-win-xp\n'
    181                        'failures/expected/image.html [ ImageOnlyFailure WontFix ]\n'
    182                        'failures/expected/text.html [ Failure WontFix ]\n'))
     181                       'failures/expected/image.html [ ImageOnlyFailure ]\n'
     182                       'failures/expected/text.html [ Failure ]\n'))
    183183
    184184    def test_multiple(self):
    185185        self.run_test(['failures/expected/text.html', 'failures/expected/image.html'],
    186186                      ('// For test-win-vista\n'
    187                        'failures/expected/image.html [ ImageOnlyFailure WontFix ]\n'
    188                        'failures/expected/text.html [ Failure WontFix ]\n'
     187                       'failures/expected/image.html [ ImageOnlyFailure ]\n'
     188                       'failures/expected/text.html [ Failure ]\n'
    189189                       '\n'
    190190                       '// For test-win-win7\n'
    191                        'failures/expected/image.html [ ImageOnlyFailure WontFix ]\n'
    192                        'failures/expected/text.html [ Failure WontFix ]\n'
     191                       'failures/expected/image.html [ ImageOnlyFailure ]\n'
     192                       'failures/expected/text.html [ Failure ]\n'
    193193                       '\n'
    194194                       '// For test-win-xp\n'
    195                        'failures/expected/image.html [ ImageOnlyFailure WontFix ]\n'
    196                        'failures/expected/text.html [ Failure WontFix ]\n'),
     195                       'failures/expected/image.html [ ImageOnlyFailure ]\n'
     196                       'failures/expected/text.html [ Failure ]\n'),
    197197                       platform='test-win-*')
    198198
    … …  
    200200        self.run_test(['failures/expected/text.html', 'failures/expected/image.html'],
    201201                      ('// For test-win-xp\n'
    202                        'failures/expected/image.html [ ImageOnlyFailure WontFix ]\n'
    203                        'failures/expected/text.html [ Failure WontFix ]\n'),
     202                       'Bug(test) failures/expected/image.html [ ImageOnlyFailure ]\n'
     203                       'Bug(test) failures/expected/text.html [ Failure ]\n'),
    204204                      full=True)
    205205
    … …  
    207207        self.run_test(['failures/expected/text.html', 'failures/expected/image.html'],
    208208                      ('// For test-win-xp\n'
    209                        'failures/expected/text.html [ Failure WontFix ]\n'),
     209                       'failures/expected/text.html [ Failure ]\n'),
    210210                      exclude_keyword=['image'])
    211211
    … …  
    218218    def test_csv(self):
    219219        self.run_test(['failures/expected/text.html', 'failures/expected/image.html'],
    220                       ('test-win-xp,failures/expected/image.html,WONTFIX,IMAGE\n'
    221                        'test-win-xp,failures/expected/text.html,WONTFIX,FAIL\n'),
     220                      ('test-win-xp,failures/expected/image.html,BUGTEST,IMAGE\n'
     221                       'test-win-xp,failures/expected/text.html,BUGTEST,FAIL\n'),
    222222                      csv=True)
    223223
Note: See TracChangeset for help on using the changeset viewer.