Changeset 129052 in webkit


Ignore:
Timestamp:
Sep 19, 2012 3:36:35 PM (12 years ago)
Author:
dpranke@chromium.org
Message:

Support new TestExpectations format alongside old one
​https://bugs.webkit.org/show_bug.cgi?id=96588

Reviewed by Ojan Vafai.

This patch adds support for actually reading in lines formatted
in the new style, and when we re-serialize/write out the file,
all lines will be written in the new style. Note that reading in
the old style is still supported, and no updates are being made
to the actual TestExpectations files as part of this change.

This change updates most but not all of the unit tests to use
the new syntax. I will update the rest when (or before where
possible) I drop support for the old format.

  • Scripts/webkitpy/layout_tests/models/test_expectations.py:

(TestExpectationParser):
(TestExpectationParser._tokenize_line):
(TestExpectationParser._tokenize_line_using_new_format):
(TestExpectationLine.to_string):
(TestExpectationLine._format_line):
(TestExpectations):

  • Scripts/webkitpy/layout_tests/models/test_expectations_unittest.py:

(test_parse_warning):
(SkippedTests.test_skipped_entry_dont_exist):
(NewExpectationSyntaxTests.test_warnings):
(RemoveConfigurationsTest.test_remove):
(test_remove_line):
(RebaseliningTest.test_remove):
(RebaseliningTest.test_no_get_rebaselining_failures):
(TestExpectationParserTests.test_tokenize_blank):
(TestExpectationParserTests.test_tokenize_extra_colon):
(TestExpectationParserTests.test_tokenize_missing_equal):
(TestExpectationParserTests.test_tokenize_extra_equal):
(TestExpectationSerializationTests.test_unparsed_to_string):
(TestExpectationSerializationTests.test_unparsed_list_to_string):
(TestExpectationSerializationTests.test_parsed_to_string):
(TestExpectationSerializationTests.test_format_line):
(TestExpectationSerializationTests.test_string_roundtrip):
(TestExpectationSerializationTests.test_list_roundtrip):
(TestExpectationSerializationTests.test_reconstitute_only_these):
(TestExpectationSerializationTests.test_string_whitespace_stripping):

  • Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py:

(LintTest.test_lint_test_fileserrors):

  • Scripts/webkitpy/tool/commands/queries_unittest.py:

(PrintExpectationsTest.test_basic):
(PrintExpectationsTest.test_multiple):
(PrintExpectationsTest.test_full):
(PrintExpectationsTest.test_exclude):

  • Scripts/webkitpy/tool/commands/rebaseline.py:

(RebaselineExpectations.execute):

  • Scripts/webkitpy/tool/commands/rebaseline_unittest.py:

(TestRebaseline.test_rebaseline_updates_expectations_file_noop):
(test_rebaseline_updates_expectations_file):
(test_rebaseline_does_not_include_overrides):
(test_rebaseline_expectations_noop):
(test_overrides_are_included_correctly):

Location:
trunk/Tools
Files:
7 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r129051 r129052  
     12012-09-19  Dirk Pranke  <dpranke@chromium.org>
     2
     3        Support new TestExpectations format alongside old one
     4        https://bugs.webkit.org/show_bug.cgi?id=96588
     5
     6        Reviewed by Ojan Vafai.
     7
     8        This patch adds support for actually reading in lines formatted
     9        in the new style, and when we re-serialize/write out the file,
     10        all lines will be written in the new style. Note that reading in
     11        the old style is still supported, and no updates are being made
     12        to the actual TestExpectations files as part of this change.
     13
     14        This change updates most but not all of the unit tests to use
     15        the new syntax. I will update the rest when (or before where
     16        possible) I drop support for the old format.
     17
     18        * Scripts/webkitpy/layout_tests/models/test_expectations.py:
     19        (TestExpectationParser):
     20        (TestExpectationParser._tokenize_line):
     21        (TestExpectationParser._tokenize_line_using_new_format):
     22        (TestExpectationLine.to_string):
     23        (TestExpectationLine._format_line):
     24        (TestExpectations):
     25        * Scripts/webkitpy/layout_tests/models/test_expectations_unittest.py:
     26        (test_parse_warning):
     27        (SkippedTests.test_skipped_entry_dont_exist):
     28        (NewExpectationSyntaxTests.test_warnings):
     29        (RemoveConfigurationsTest.test_remove):
     30        (test_remove_line):
     31        (RebaseliningTest.test_remove):
     32        (RebaseliningTest.test_no_get_rebaselining_failures):
     33        (TestExpectationParserTests.test_tokenize_blank):
     34        (TestExpectationParserTests.test_tokenize_extra_colon):
     35        (TestExpectationParserTests.test_tokenize_missing_equal):
     36        (TestExpectationParserTests.test_tokenize_extra_equal):
     37        (TestExpectationSerializationTests.test_unparsed_to_string):
     38        (TestExpectationSerializationTests.test_unparsed_list_to_string):
     39        (TestExpectationSerializationTests.test_parsed_to_string):
     40        (TestExpectationSerializationTests.test_format_line):
     41        (TestExpectationSerializationTests.test_string_roundtrip):
     42        (TestExpectationSerializationTests.test_list_roundtrip):
     43        (TestExpectationSerializationTests.test_reconstitute_only_these):
     44        (TestExpectationSerializationTests.test_string_whitespace_stripping):
     45        * Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py:
     46        (LintTest.test_lint_test_files__errors):
     47        * Scripts/webkitpy/tool/commands/queries_unittest.py:
     48        (PrintExpectationsTest.test_basic):
     49        (PrintExpectationsTest.test_multiple):
     50        (PrintExpectationsTest.test_full):
     51        (PrintExpectationsTest.test_exclude):
     52        * Scripts/webkitpy/tool/commands/rebaseline.py:
     53        (RebaselineExpectations.execute):
     54        * Scripts/webkitpy/tool/commands/rebaseline_unittest.py:
     55        (TestRebaseline.test_rebaseline_updates_expectations_file_noop):
     56        (test_rebaseline_updates_expectations_file):
     57        (test_rebaseline_does_not_include_overrides):
     58        (test_rebaseline_expectations_noop):
     59        (test_overrides_are_included_correctly):
     60
    1612012-09-19  Dirk Pranke  <dpranke@chromium.org>
    262

  • trunk/Tools/Scripts/webkitpy/layout_tests/models/test_expectations.py

    r129051 r129052  
    210210            expectation_line.matching_tests.append(expectation_line.path)
    211211
     212    # FIXME: Seems like these should be classmethods on TestExpectationLine instead of TestExpectationParser.
    212213    @classmethod
    213214    def _tokenize_line(cls, filename, expectation_string, line_number):
    214         # FIXME: Add in support for the new format as well.
    215         return cls._tokenize_line_using_old_format(filename, expectation_string, line_number)
     215        expectation_line = cls._tokenize_line_using_new_format(filename, expectation_string, line_number)
     216        if expectation_line.is_invalid():
     217            old_expectation_line = cls._tokenize_line_using_old_format(filename, expectation_string, line_number)
     218            if not old_expectation_line.is_invalid():
     219                return old_expectation_line
     220        return expectation_line
    216221
    217222    @classmethod
    … …  
    266271
    267272    _configuration_tokens = dict((token, token.upper()) for token in _configuration_tokens_list)
    268 
    269     # Note: we can't distinguish audio failures or image+text failures from text-only failures.
     273    _inverted_configuration_tokens = dict((value, name) for name, value in _configuration_tokens.iteritems())
     274
    270275    # FIXME: Update the original modifiers list and remove this once the old syntax is gone.
    271276    _expectation_tokens = {
    272         'WontFix': 'WONTFIX',
    273         'Pass': 'PASS',
     277        'Crash': 'CRASH',
    274278        'Failure': 'FAIL',
    275279        'ImageOnlyFailure': 'IMAGE',
    276         'Crash': 'CRASH',
     280        'Pass': 'PASS',
     281        'Rebaseline': 'REBASELINE',
     282        'Skip': 'SKIP',
     283        'Slow': 'SLOW',
    277284        'Timeout': 'TIMEOUT',
    278         'Slow': 'SLOW',
     285        'WontFix': 'WONTFIX',
    279286    }
     287
     288    _inverted_expectation_tokens = dict((value, name) for name, value in _expectation_tokens.iteritems())
    280289
    281290    @classmethod
    … …  
    291300        """
    292301        expectation_line = TestExpectationLine()
     302        expectation_line.original_string = expectation_string
    293303        expectation_line.filename = filename
    294304        expectation_line.line_number = line_number
    … …  
    445455
    446456        if self.name is None:
    447             return '' if self.comment is None else "//%s" % self.comment
     457            return '' if self.comment is None else "#%s" % self.comment
    448458
    449459        if test_configuration_converter and self.parsed_bug_modifiers:
    … …  
    451461            result = []
    452462            for specifiers in specifiers_list:
    453                 modifiers = self._serialize_parsed_modifiers(test_configuration_converter, specifiers)
    454                 expectations = self._serialize_parsed_expectations(parsed_expectation_to_string)
     463                # FIXME: this is silly that we join the modifiers and then immediately split them.
     464                modifiers = self._serialize_parsed_modifiers(test_configuration_converter, specifiers).split()
     465                expectations = self._serialize_parsed_expectations(parsed_expectation_to_string).split()
    455466                result.append(self._format_line(modifiers, self.name, expectations, self.comment))
    456467            return "\n".join(result) if result else None
    457468
    458         return self._format_line(" ".join(self.modifiers), self.name, " ".join(self.expectations), self.comment,
     469        return self._format_line(self.modifiers, self.name, self.expectations, self.comment,
    459470            include_modifiers, include_expectations, include_comment)
    460471
    … …  
    480491    @staticmethod
    481492    def _format_line(modifiers, name, expectations, comment, include_modifiers=True, include_expectations=True, include_comment=True):
     493        bugs = []
     494        new_modifiers = []
     495        new_expectations = []
     496        for modifier in modifiers:
     497            modifier = modifier.upper()
     498            if modifier.startswith('BUGWK'):
     499                bugs.append('webkit.org/b/' + modifier.replace('BUGWK', ''))
     500            elif modifier.startswith('BUGCR'):
     501                bugs.append('crbug.com/' + modifier.replace('BUGCR', ''))
     502            elif modifier.startswith('BUG'):
     503                # FIXME: we should preserve case once we can drop the old syntax.
     504                bugs.append('Bug(' + modifier[3:].lower() + ')')
     505            elif modifier in ('SLOW', 'SKIP', 'REBASELINE', 'WONTFIX'):
     506                new_expectations.append(TestExpectationParser._inverted_expectation_tokens.get(modifier))
     507            else:
     508                new_modifiers.append(TestExpectationParser._inverted_configuration_tokens.get(modifier, modifier))
     509
     510        for expectation in expectations:
     511            expectation = expectation.upper()
     512            new_expectations.append(TestExpectationParser._inverted_expectation_tokens.get(expectation, expectation))
     513
    482514        result = ''
    483         if include_modifiers:
    484             result += '%s : ' % modifiers.upper()
     515        if include_modifiers and (bugs or new_modifiers):
     516            if bugs:
     517                result += ' '.join(bugs) + ' '
     518            if new_modifiers:
     519                result += '[ %s ] ' % ' '.join(new_modifiers)
    485520        result += name
    486         if include_expectations:
    487             result += ' = %s' % expectations.upper()
     521        if include_expectations and new_expectations and set(new_expectations) != set(['Skip', 'Pass']):
     522            result += ' [ %s ]' % ' '.join(sorted(set(new_expectations)))
    488523        if include_comment and comment is not None:
    489             result += " //%s" % comment
     524            result += " #%s" % comment
    490525        return result
    491526
    … …  
    731766    directory and any subdirectory. The format is along the lines of:
    732767
    733       LayoutTests/fast/js/fixme.js = FAIL
    734       LayoutTests/fast/js/flaky.js = FAIL PASS
    735       LayoutTests/fast/js/crash.js = CRASH TIMEOUT FAIL PASS
     768      LayoutTests/fast/js/fixme.js [ Failure ]
     769      LayoutTests/fast/js/flaky.js [ Failure Pass ]
     770      LayoutTests/fast/js/crash.js [ Crash Failure Pass Timeout ]
    736771      ...
    737772
    738773    To add modifiers:
    739       SKIP : LayoutTests/fast/js/no-good.js = TIMEOUT PASS
    740       DEBUG : LayoutTests/fast/js/no-good.js = TIMEOUT PASS
    741       DEBUG SKIP : LayoutTests/fast/js/no-good.js = TIMEOUT PASS
    742       LINUX DEBUG SKIP : LayoutTests/fast/js/no-good.js = TIMEOUT PASS
    743       LINUX WIN : LayoutTests/fast/js/no-good.js = TIMEOUT PASS
    744 
    745     SKIP: Doesn't run the test.
    746     SLOW: The test takes a long time to run, but does not timeout indefinitely.
    747     WONTFIX: For tests that we never intend to pass on a given platform.
     774      LayoutTests/fast/js/no-good.js
     775      [ Debug ] LayoutTests/fast/js/no-good.js [ Pass Timeout ]
     776      [ Debug ] LayoutTests/fast/js/no-good.js [ Pass Skip Timeout ]
     777      [ Linux Debug ] LayoutTests/fast/js/no-good.js [ Pass Skip Timeout ]
     778      [ Linux Win ] LayoutTests/fast/js/no-good.js [ Pass Skip Timeout ]
     779
     780    Skip: Doesn't run the test.
     781    Slow: The test takes a long time to run, but does not timeout indefinitely.
     782    WontFix: For tests that we never intend to pass on a given platform (treated like Skip).
    748783
    749784    Notes:
    … …  
    754789    """
    755790
     791    # FIXME: Update to new syntax once the old format is no longer supported.
    756792    EXPECTATIONS = {'pass': PASS,
    757793                    'fail': FAIL,

  • trunk/Tools/Scripts/webkitpy/layout_tests/models/test_expectations_unittest.py

    r129051 r129052  
    4444
    4545
    46 class MockBugManager(object):
    47     def close_bug(self, bug_id, reference_bug_id=None):
    48         pass
    49 
    50     def create_bug(self):
    51         return "BUG_NEWLY_CREATED"
    52 
    53 
    5446class Base(unittest.TestCase):
    5547    # Note that all of these tests are written assuming the configuration
    … …  
    190182            filesystem.write_text_file(filesystem.join(self._port.layout_tests_dir(), 'disabled-test.html-disabled'), 'content')
    191183            self.get_test('disabled-test.html-disabled'),
    192             self.parse_exp("FOO : failures/expected/text.html = FAIL\n"
    193                 "SKIP : failures/expected/image.html\n"
    194                 "BUGRNIWA : non-existent-test.html = FAIL\n"
    195                 "BUGRNIWA : disabled-test.html-disabled = IMAGE", is_lint_mode=True)
     184            self.parse_exp("[ FOO ] failures/expected/text.html [ Failure ]\n"
     185                "Bug(rniwa) non-existent-test.html [ Failure ]\n"
     186                "Bug(rniwa) disabled-test.html-disabled [ ImageOnlyFailure ]", is_lint_mode=True)
    196187            self.assertFalse(True, "ParseError wasn't raised")
    197188        except ParseError, e:
    198189            warnings = ("expectations:1 Test lacks BUG modifier. failures/expected/text.html\n"
    199190                        "expectations:1 Unrecognized modifier 'foo' failures/expected/text.html\n"
    200                         "expectations:2 Missing expectations SKIP : failures/expected/image.html\n"
    201                         "expectations:3 Path does not exist. non-existent-test.html")
     191                        "expectations:2 Path does not exist. non-existent-test.html")
    202192            self.assertEqual(str(e), warnings)
    203193
    … …  
    314304        self.assertEqual('The following test foo/bar/baz.html from the Skipped list doesn\'t exist\n', logs)
    315305
     306
     307# FIXME: remove these tests when we stop supporting the old syntax.
    316308
    317309class ExpectationSyntaxTests(Base):
    … …  
    403395
    404396
     397class NewExpectationSyntaxTests(unittest.TestCase):
     398    def assert_exp(self, line, bugs=None, modifiers=None, expectations=None, warnings=None, comment=None, name='foo.html'):
     399        bugs = bugs or []
     400        modifiers = modifiers or []
     401        expectations = expectations or []
     402        warnings = warnings or []
     403        filename = 'TestExpectations'
     404        line_number = 1
     405        expectation_line = TestExpectationParser._tokenize_line_using_new_format(filename, line, line_number)
     406        self.assertEquals(expectation_line.warnings, warnings)
     407        self.assertEquals(expectation_line.name, name)
     408        self.assertEquals(expectation_line.filename, filename)
     409        self.assertEquals(expectation_line.line_number, line_number)
     410        if not warnings:
     411            self.assertEquals(expectation_line.modifiers, modifiers)
     412            self.assertEquals(expectation_line.expectations, expectations)
     413
     414    def test_bare_name(self):
     415        self.assert_exp('foo.html', modifiers=['SKIP'], expectations=['PASS'])
     416
     417    def test_bare_name_and_bugs(self):
     418        self.assert_exp('webkit.org/b/12345 foo.html', modifiers=['BUGWK12345', 'SKIP'], expectations=['PASS'])
     419        self.assert_exp('crbug.com/12345 foo.html', modifiers=['BUGCR12345', 'SKIP'], expectations=['PASS'])
     420        self.assert_exp('Bug(dpranke) foo.html', modifiers=['BUGDPRANKE', 'SKIP'], expectations=['PASS'])
     421        self.assert_exp('crbug.com/12345 crbug.com/34567 foo.html', modifiers=['BUGCR12345', 'BUGCR34567', 'SKIP'], expectations=['PASS'])
     422
     423    def test_comments(self):
     424        self.assert_exp("# comment", name=None, comment="# comment")
     425        self.assert_exp("foo.html # comment", comment="# comment", expectations=['PASS'], modifiers=['SKIP'])
     426
     427    def test_config_modifiers(self):
     428        self.assert_exp('[ Mac ] foo.html', modifiers=['MAC', 'SKIP'], expectations=['PASS'])
     429        self.assert_exp('[ Mac Vista ] foo.html', modifiers=['MAC', 'VISTA', 'SKIP'], expectations=['PASS'])
     430        self.assert_exp('[ Mac ] foo.html [ Failure ] ', modifiers=['MAC'], expectations=['FAIL'])
     431
     432    def test_unknown_config(self):
     433        self.assert_exp('[ Foo ] foo.html ', modifiers=['Foo', 'SKIP'], expectations=['PASS'])
     434
     435    def test_unknown_expectation(self):
     436        self.assert_exp('foo.html [ Audio ]', expectations=['Audio'])
     437
     438    def test_skip(self):
     439        self.assert_exp('foo.html [ Skip ]', modifiers=['SKIP'], expectations=['PASS'])
     440
     441    def test_slow(self):
     442        self.assert_exp('foo.html [ Slow ]', modifiers=['SLOW'], expectations=['PASS'])
     443
     444    def test_wontfix(self):
     445        self.assert_exp('foo.html [ WontFix ]', modifiers=['WONTFIX', 'SKIP'], expectations=['PASS'])
     446
     447    def test_blank_line(self):
     448        self.assert_exp('', name=None)
     449
     450    def test_warnings(self):
     451        self.assert_exp('[ Mac ]', warnings=['Did not find a test name.'], name=None)
     452        self.assert_exp('[ [', warnings=['unexpected "["'], name=None)
     453        self.assert_exp('crbug.com/12345 ]', warnings=['unexpected "]"'], name=None)
     454
     455        self.assert_exp('foo.html crbug.com/12345 ]', warnings=['"crbug.com/12345" is not at the start of the line.'])
     456
     457
    405458class SemanticTests(Base):
    406459    def test_bug_format(self):
    … …  
    490543
    491544        test_config = test_port.test_configuration()
    492         test_port.expectations_dict = lambda: {"expectations": """BUGX LINUX WIN RELEASE : failures/expected/foo.html = FAIL
    493 BUGY WIN MAC DEBUG : failures/expected/foo.html = CRASH
     545        test_port.expectations_dict = lambda: {"expectations": """Bug(x) [ Linux Win Release ] failures/expected/foo.html [ Failure ]
     546Bug(y) [ Win Mac Debug ] failures/expected/foo.html [ Crash ]
    494547"""}
    495548        expectations = TestExpectations(test_port, self.get_basic_tests())
    … …  
    497550        actual_expectations = expectations.remove_configuration_from_test('failures/expected/foo.html', test_config)
    498551
    499         self.assertEqual("""BUGX LINUX VISTA WIN7 RELEASE : failures/expected/foo.html = FAIL
    500 BUGY WIN MAC DEBUG : failures/expected/foo.html = CRASH
     552        self.assertEqual("""Bug(x) [ Linux Vista Win7 Release ] failures/expected/foo.html [ Failure ]
     553Bug(y) [ Win Mac Debug ] failures/expected/foo.html [ Crash ]
    501554""", actual_expectations)
    502555
    … …  
    508561
    509562        test_config = test_port.test_configuration()
    510         test_port.expectations_dict = lambda: {'expectations': """BUGX WIN RELEASE : failures/expected/foo.html = FAIL
    511 BUGY WIN DEBUG : failures/expected/foo.html = CRASH
     563        test_port.expectations_dict = lambda: {'expectations': """Bug(x) [ Win Release ] failures/expected/foo.html [ Failure ]
     564Bug(y) [ Win Debug ] failures/expected/foo.html [ Crash ]
    512565"""}
    513566        expectations = TestExpectations(test_port)
    … …  
    517570        actual_expectations = expectations.remove_configuration_from_test('failures/expected/foo.html', host.port_factory.get('test-win-win7', None).test_configuration())
    518571
    519         self.assertEqual("""BUGY WIN DEBUG : failures/expected/foo.html = CRASH
     572        self.assertEqual("""Bug(y) [ Win Debug ] failures/expected/foo.html [ Crash ]
    520573""", actual_expectations)
    521574
    … …  
    531584
    532585    def test_remove(self):
    533         self.assertRemove('BUGX REBASELINE : failures/expected/text.html = FAIL\n'
    534                           'BUGY : failures/expected/image.html = IMAGE\n'
    535                           'BUGZ REBASELINE : failures/expected/crash.html = CRASH\n',
    536                           'BUGXO : failures/expected/image.html = CRASH\n',
     586        self.assertRemove('Bug(x) failures/expected/text.html [ Failure Rebaseline ]\n'
     587                          'Bug(y) failures/expected/image.html [ ImageOnlyFailure Rebaseline ]\n'
     588                          'Bug(z) failures/expected/crash.html [ Crash ]\n',
     589                          'Bug(x0) failures/expected/image.html [ Crash ]\n',
    537590                          ['failures/expected/text.html'],
    538                           'BUGY : failures/expected/image.html = IMAGE\n'
    539                           'BUGZ REBASELINE : failures/expected/crash.html = CRASH\n',
    540                           'BUGXO : failures/expected/image.html = CRASH\n')
     591                          'Bug(y) failures/expected/image.html [ ImageOnlyFailure Rebaseline ]\n'
     592                          'Bug(z) failures/expected/crash.html [ Crash ]\n',
     593                          'Bug(x0) failures/expected/image.html [ Crash ]\n')
    541594
    542595
    … …  
    545598        self.assertEqual(len(self._exp.get_rebaselining_failures()), 0)
    546599
     600
     601# FIXME: remove most of these tests when we no longer support the old syntax.
    547602
    548603class TestExpectationParserTests(unittest.TestCase):
    … …  
    555610        self.assertEqual(len(expectation.warnings), 0)
    556611
    557     def test_tokenize_missing_colon(self):
    558         expectation = self._tokenize('Qux.')
    559         self.assertEqual(str(expectation.warnings), '["Missing a \':\'"]')
    560 
    561612    def test_tokenize_extra_colon(self):
    562613        expectation = self._tokenize('FOO : : bar')
    563         self.assertEqual(str(expectation.warnings), '["Extraneous \':\'"]')
     614        self.assertEqual(expectation.warnings, ['":" is not legal in the new TestExpectations syntax.'])
    564615
    565616    def test_tokenize_empty_comment(self):
    … …  
    575626    def test_tokenize_missing_equal(self):
    576627        expectation = self._tokenize('FOO : bar')
    577         self.assertEqual(str(expectation.warnings), "['Missing expectations\']")
     628        self.assertEqual(expectation.warnings, ['":" is not legal in the new TestExpectations syntax.'])
    578629
    579630    def test_tokenize_extra_equal(self):
    580631        expectation = self._tokenize('FOO : bar = BAZ = Qux.')
    581         self.assertEqual(str(expectation.warnings), '["Extraneous \'=\'"]')
     632        self.assertEqual(expectation.warnings, ['":" is not legal in the new TestExpectations syntax.'])
    582633
    583634    def test_tokenize_valid(self):
    … …  
    639690
    640691        self.assertEqual(expectation.to_string(self._converter), '')
    641         expectation.comment = 'Qux.'
    642         self.assertEqual(expectation.to_string(self._converter), '//Qux.')
     692        expectation.comment = ' Qux.'
     693        self.assertEqual(expectation.to_string(self._converter), '# Qux.')
    643694        expectation.name = 'bar'
    644         self.assertEqual(expectation.to_string(self._converter), ' : bar =  //Qux.')
     695        self.assertEqual(expectation.to_string(self._converter), 'bar # Qux.')
    645696        expectation.modifiers = ['foo']
    646         self.assertEqual(expectation.to_string(self._converter), 'FOO : bar =  //Qux.')
     697        # FIXME: case should be preserved here but we can't until we drop the old syntax.
     698        self.assertEqual(expectation.to_string(self._converter), '[ FOO ] bar # Qux.')
    647699        expectation.expectations = ['bAz']
    648         self.assertEqual(expectation.to_string(self._converter), 'FOO : bar = BAZ //Qux.')
     700        self.assertEqual(expectation.to_string(self._converter), '[ FOO ] bar [ BAZ ] # Qux.')
    649701        expectation.expectations = ['bAz1', 'baZ2']
    650         self.assertEqual(expectation.to_string(self._converter), 'FOO : bar = BAZ1 BAZ2 //Qux.')
     702        self.assertEqual(expectation.to_string(self._converter), '[ FOO ] bar [ BAZ1 BAZ2 ] # Qux.')
    651703        expectation.modifiers = ['foo1', 'foO2']
    652         self.assertEqual(expectation.to_string(self._converter), 'FOO1 FOO2 : bar = BAZ1 BAZ2 //Qux.')
     704        self.assertEqual(expectation.to_string(self._converter), '[ FOO1 FOO2 ] bar [ BAZ1 BAZ2 ] # Qux.')
    653705        expectation.warnings.append('Oh the horror.')
    654706        self.assertEqual(expectation.to_string(self._converter), '')
    … …  
    662714        expectation.modifiers = ['foo']
    663715        expectation.expectations = ['bAz1', 'baZ2']
    664         self.assertEqual(TestExpectations.list_to_string([expectation]), 'FOO : bar = BAZ1 BAZ2 //Qux.')
     716        # FIXME: case should be preserved here but we can't until we drop the old syntax.
     717        self.assertEqual(TestExpectations.list_to_string([expectation]), '[ FOO ] bar [ BAZ1 BAZ2 ] #Qux.')
    665718
    666719    def test_parsed_to_string(self):
    … …  
    671724        self.assertEqual(expectation_line.to_string(self._converter), None)
    672725        expectation_line.matching_configurations = set([TestConfiguration('xp', 'x86', 'release')])
    673         self.assertEqual(expectation_line.to_string(self._converter), 'BUGX XP RELEASE : test/name/for/realz.html = IMAGE')
     726        self.assertEqual(expectation_line.to_string(self._converter), 'Bug(x) [ XP Release ] test/name/for/realz.html [ ImageOnlyFailure ]')
    674727        expectation_line.matching_configurations = set([TestConfiguration('xp', 'x86', 'release'), TestConfiguration('xp', 'x86', 'debug')])
    675         self.assertEqual(expectation_line.to_string(self._converter), 'BUGX XP : test/name/for/realz.html = IMAGE')
     728        self.assertEqual(expectation_line.to_string(self._converter), 'Bug(x) [ XP ] test/name/for/realz.html [ ImageOnlyFailure ]')
    676729
    677730    def test_serialize_parsed_expectations(self):
    … …  
    704757
    705758    def test_format_line(self):
    706         self.assertEqual(TestExpectationLine._format_line('modifiers', 'name', 'expectations', 'comment'), 'MODIFIERS : name = EXPECTATIONS //comment')
    707         self.assertEqual(TestExpectationLine._format_line('modifiers', 'name', 'expectations', None), 'MODIFIERS : name = EXPECTATIONS')
     759        self.assertEqual(TestExpectationLine._format_line(['MODIFIERS'], 'name', ['EXPECTATIONS'], 'comment'), '[ MODIFIERS ] name [ EXPECTATIONS ] #comment')
     760        self.assertEqual(TestExpectationLine._format_line(['MODIFIERS'], 'name', ['EXPECTATIONS'], None), '[ MODIFIERS ] name [ EXPECTATIONS ]')
    708761
    709762    def test_string_roundtrip(self):
    710763        self.assert_round_trip('')
    711764        self.assert_round_trip('FOO')
    712         self.assert_round_trip(':')
    713         self.assert_round_trip('FOO :')
    714         self.assert_round_trip('FOO : bar')
    715         self.assert_round_trip('  FOO :')
    716         self.assert_round_trip('    FOO : bar')
    717         self.assert_round_trip('FOO : bar = BAZ')
    718         self.assert_round_trip('FOO : bar = BAZ //Qux.')
    719         self.assert_round_trip('FOO : bar = BAZ // Qux.')
    720         self.assert_round_trip('FOO : bar = BAZ // Qux.     ')
    721         self.assert_round_trip('FOO : bar = BAZ //        Qux.     ')
    722         self.assert_round_trip('FOO : : bar = BAZ')
    723         self.assert_round_trip('FOO : : bar = BAZ')
    724         self.assert_round_trip('FOO : : bar ==== BAZ')
     765        self.assert_round_trip('[')
     766        self.assert_round_trip('FOO [')
     767        self.assert_round_trip('FOO ] bar')
     768        self.assert_round_trip('  FOO [')
     769        self.assert_round_trip('    [ FOO ] ')
     770        self.assert_round_trip('[ FOO ] bar [ BAZ ]')
     771        self.assert_round_trip('[ FOO ] bar [ BAZ ] # Qux.')
     772        self.assert_round_trip('[ FOO ] bar [ BAZ ] # Qux.')
     773        self.assert_round_trip('[ FOO ] bar [ BAZ ] # Qux.     ')
     774        self.assert_round_trip('[ FOO ] bar [ BAZ ] #        Qux.     ')
     775        self.assert_round_trip('[ FOO ] ] ] bar BAZ')
     776        self.assert_round_trip('[ FOO ] ] ] bar [ BAZ ]')
     777        self.assert_round_trip('FOO ] ] bar ==== BAZ')
    725778        self.assert_round_trip('=')
    726         self.assert_round_trip('//')
    727         self.assert_round_trip('// ')
    728         self.assert_round_trip('// Foo')
    729         self.assert_round_trip('// Foo')
    730         self.assert_round_trip('// Foo :')
    731         self.assert_round_trip('// Foo : =')
     779        self.assert_round_trip('#')
     780        self.assert_round_trip('# ')
     781        self.assert_round_trip('# Foo')
     782        self.assert_round_trip('# Foo')
     783        self.assert_round_trip('# Foo :')
     784        self.assert_round_trip('# Foo : =')
    732785
    733786    def test_list_roundtrip(self):
    … …  
    736789        self.assert_list_round_trip('\n\n')
    737790        self.assert_list_round_trip('bar')
    738         self.assert_list_round_trip('bar\n//Qux.')
    739         self.assert_list_round_trip('bar\n//Qux.\n')
     791        self.assert_list_round_trip('bar\n# Qux.')
     792        self.assert_list_round_trip('bar\n# Qux.\n')
    740793
    741794    def test_reconstitute_only_these(self):
    … …  
    757810        add_line(set([TestConfiguration('xp', 'x86', 'release'), TestConfiguration('xp', 'x86', 'debug')]), False)
    758811        serialized = TestExpectations.list_to_string(lines, self._converter)
    759         self.assertEquals(serialized, "BUGX XP RELEASE : Yay = IMAGE\nBUGX XP : Yay = IMAGE")
     812        self.assertEquals(serialized, "Bug(x) [ XP Release ] Yay [ ImageOnlyFailure ]\nBug(x) [ XP ] Yay [ ImageOnlyFailure ]")
    760813        serialized = TestExpectations.list_to_string(lines, self._converter, reconstitute_only_these=reconstitute_only_these)
    761         self.assertEquals(serialized, "BUGX XP RELEASE : Yay = IMAGE\nNay")
     814        self.assertEquals(serialized, "Bug(x) [ XP Release ] Yay [ ImageOnlyFailure ]\nNay")
    762815
    763816    def test_string_whitespace_stripping(self):
    764817        self.assert_round_trip('\n', '')
    765         self.assert_round_trip('  FOO : bar = BAZ', 'FOO : bar = BAZ')
    766         self.assert_round_trip('FOO    : bar = BAZ', 'FOO : bar = BAZ')
    767         self.assert_round_trip('FOO : bar = BAZ       // Qux.', 'FOO : bar = BAZ // Qux.')
    768         self.assert_round_trip('FOO : bar =        BAZ // Qux.', 'FOO : bar = BAZ // Qux.')
    769         self.assert_round_trip('FOO :       bar =    BAZ // Qux.', 'FOO : bar = BAZ // Qux.')
    770         self.assert_round_trip('FOO :       bar     =    BAZ // Qux.', 'FOO : bar = BAZ // Qux.')
     818        self.assert_round_trip('  [ FOO ] bar [ BAZ ]', '[ FOO ] bar [ BAZ ]')
     819        self.assert_round_trip('[ FOO ]    bar [ BAZ ]', '[ FOO ] bar [ BAZ ]')
     820        self.assert_round_trip('[ FOO ] bar [ BAZ ]       # Qux.', '[ FOO ] bar [ BAZ ] # Qux.')
     821        self.assert_round_trip('[ FOO ] bar [        BAZ ]  # Qux.', '[ FOO ] bar [ BAZ ] # Qux.')
     822        self.assert_round_trip('[ FOO ]       bar [    BAZ ]  # Qux.', '[ FOO ] bar [ BAZ ] # Qux.')
     823        self.assert_round_trip('[ FOO ]       bar     [    BAZ ]  # Qux.', '[ FOO ] bar [ BAZ ] # Qux.')
    771824
    772825

  • trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py

    r129047 r129052  
    261261        host = MockHost()
    262262        port_obj = host.port_factory.get(options.platform, options=options)
    263         port_obj.expectations_dict = lambda: {'': '# syntax error'}
     263        port_obj.expectations_dict = lambda: {'': '-- syntax error'}
    264264        res, out, err = run_and_capture(port_obj, options, parsed_args)
    265265

  • trunk/Tools/Scripts/webkitpy/tool/commands/queries_unittest.py

    r129047 r129052  
    179179        self.run_test(['failures/expected/text.html', 'failures/expected/image.html'],
    180180                      ('// For test-win-xp\n'
    181                        'failures/expected/image.html = IMAGE\n'
    182                        'failures/expected/text.html = FAIL\n'))
     181                       'failures/expected/image.html [ ImageOnlyFailure WontFix ]\n'
     182                       'failures/expected/text.html [ Failure WontFix ]\n'))
    183183
    184184    def test_multiple(self):
    185185        self.run_test(['failures/expected/text.html', 'failures/expected/image.html'],
    186186                      ('// For test-win-vista\n'
    187                        'failures/expected/image.html = IMAGE\n'
    188                        'failures/expected/text.html = FAIL\n'
     187                       'failures/expected/image.html [ ImageOnlyFailure WontFix ]\n'
     188                       'failures/expected/text.html [ Failure WontFix ]\n'
    189189                       '\n'
    190190                       '// For test-win-win7\n'
    191                        'failures/expected/image.html = IMAGE\n'
    192                        'failures/expected/text.html = FAIL\n'
     191                       'failures/expected/image.html [ ImageOnlyFailure WontFix ]\n'
     192                       'failures/expected/text.html [ Failure WontFix ]\n'
    193193                       '\n'
    194194                       '// For test-win-xp\n'
    195                        'failures/expected/image.html = IMAGE\n'
    196                        'failures/expected/text.html = FAIL\n'),
     195                       'failures/expected/image.html [ ImageOnlyFailure WontFix ]\n'
     196                       'failures/expected/text.html [ Failure WontFix ]\n'),
    197197                       platform='test-win-*')
    198198
    … …  
    200200        self.run_test(['failures/expected/text.html', 'failures/expected/image.html'],
    201201                      ('// For test-win-xp\n'
    202                        'WONTFIX : failures/expected/image.html = IMAGE\n'
    203                        'WONTFIX : failures/expected/text.html = FAIL\n'),
     202                       'failures/expected/image.html [ ImageOnlyFailure WontFix ]\n'
     203                       'failures/expected/text.html [ Failure WontFix ]\n'),
    204204                      full=True)
    205205
    … …  
    207207        self.run_test(['failures/expected/text.html', 'failures/expected/image.html'],
    208208                      ('// For test-win-xp\n'
    209                        'failures/expected/text.html = FAIL\n'),
     209                       'failures/expected/text.html [ Failure WontFix ]\n'),
    210210                      exclude_keyword=['image'])
    211211

  • trunk/Tools/Scripts/webkitpy/tool/commands/rebaseline.py

    r127058 r129052  
    357357            self._add_tests_to_rebaseline_for_port(port_name)
    358358        if not self._test_list:
    359             _log.warning("Did not find any tests marked REBASELINE.")
     359            _log.warning("Did not find any tests marked Rebaseline.")
    360360            return
    361361

  • trunk/Tools/Scripts/webkitpy/tool/commands/rebaseline_unittest.py

    r129047 r129052  
    6868        for path in lion_port.expectations_files():
    6969            tool.filesystem.write_text_file(path, '')
    70         tool.filesystem.write_text_file(lion_port.path_to_test_expectations_file(), """BUGB MAC LINUX XP DEBUG : fast/dom/Window/window-postmessage-clone-really-deep-array.html = PASS
    71 BUGA DEBUG : fast/css/large-list-of-rules-crash.html = FAIL
     70        tool.filesystem.write_text_file(lion_port.path_to_test_expectations_file(), """Bug(B) [ Mac Linux XP Debug ] fast/dom/Window/window-postmessage-clone-really-deep-array.html [ Pass ]
     71Bug(A) [ Debug ] : fast/css/large-list-of-rules-crash.html [ Failure ]
    7272""")
    7373        tool.filesystem.write_text_file(os.path.join(lion_port.layout_tests_dir(), "fast/dom/Window/window-postmessage-clone-really-deep-array.html"), "Dummy test contents")
    … …  
    8282
    8383        new_expectations = tool.filesystem.read_text_file(lion_port.path_to_test_expectations_file())
    84         self.assertEqual(new_expectations, """BUGB MAC LINUX XP DEBUG : fast/dom/Window/window-postmessage-clone-really-deep-array.html = PASS
    85 BUGA DEBUG : fast/css/large-list-of-rules-crash.html = FAIL
     84        self.assertEqual(new_expectations, """Bug(B) [ Mac Linux XP Debug ] fast/dom/Window/window-postmessage-clone-really-deep-array.html [ Pass ]
     85Bug(A) [ Debug ] : fast/css/large-list-of-rules-crash.html [ Failure ]
    8686""")
    8787
    … …  
    9393        lion_port = tool.port_factory.get_from_builder_name("Webkit Mac10.7")
    9494        tool.filesystem.write_text_file(lion_port.path_from_chromium_base('skia', 'skia_test_expectations.txt'), '')
    95         tool.filesystem.write_text_file(lion_port.path_to_test_expectations_file(), "BUGX MAC : userscripts/another-test.html = IMAGE\nBUGZ LINUX : userscripts/another-test.html = IMAGE\n")
     95        tool.filesystem.write_text_file(lion_port.path_to_test_expectations_file(), "Bug(x) [ Mac ] userscripts/another-test.html [ ImageOnlyFailure ]\nbug(z) [ Linux ] userscripts/another-test.html [ ImageOnlyFailure ]\n")
    9696        tool.filesystem.write_text_file(os.path.join(lion_port.layout_tests_dir(), "userscripts/another-test.html"), "Dummy test contents")
    9797
    … …  
    103103
    104104        new_expectations = tool.filesystem.read_text_file(lion_port.path_to_test_expectations_file())
    105         self.assertEqual(new_expectations, "BUGX SNOWLEOPARD : userscripts/another-test.html = IMAGE\nBUGZ LINUX : userscripts/another-test.html = IMAGE\n")
     105        self.assertEqual(new_expectations, "Bug(x) [ SnowLeopard ] userscripts/another-test.html [ ImageOnlyFailure ]\nbug(z) [ Linux ] userscripts/another-test.html [ ImageOnlyFailure ]\n")
    106106
    107107    def test_rebaseline_does_not_include_overrides(self):
    … …  
    112112        lion_port = tool.port_factory.get_from_builder_name("Webkit Mac10.7")
    113113        tool.filesystem.write_text_file(lion_port.path_from_chromium_base('skia', 'skia_test_expectations.txt'), '')
    114         tool.filesystem.write_text_file(lion_port.path_to_test_expectations_file(), "BUGX MAC : userscripts/another-test.html = IMAGE\nBUGZ LINUX : userscripts/another-test.html = IMAGE\n")
    115         tool.filesystem.write_text_file(lion_port.path_from_chromium_base('skia', 'skia_test_expectations.txt'), "BUGY MAC : other-test.html = FAIL\n")
     114        tool.filesystem.write_text_file(lion_port.path_to_test_expectations_file(), "Bug(x) [ Mac ] userscripts/another-test.html [ ImageOnlyFailure ]\nBug(z) [ Linux ] userscripts/another-test.html [ ImageOnlyFailure ]\n")
     115        tool.filesystem.write_text_file(lion_port.path_from_chromium_base('skia', 'skia_test_expectations.txt'), "Bug(y) [ Mac ] other-test.html [ Failure ]\n")
    116116        tool.filesystem.write_text_file(os.path.join(lion_port.layout_tests_dir(), "userscripts/another-test.html"), "Dummy test contents")
    117117
    … …  
    123123
    124124        new_expectations = tool.filesystem.read_text_file(lion_port.path_to_test_expectations_file())
    125         self.assertEqual(new_expectations, "BUGX SNOWLEOPARD : userscripts/another-test.html = IMAGE\nBUGZ LINUX : userscripts/another-test.html = IMAGE\n")
     125        self.assertEqual(new_expectations, "Bug(x) [ SnowLeopard ] userscripts/another-test.html [ ImageOnlyFailure ]\nBug(z) [ Linux ] userscripts/another-test.html [ ImageOnlyFailure ]\n")
    126126
    127127    def test_rebaseline_test(self):
    … …  
    332332
    333333    def test_rebaseline_expectations_noop(self):
    334         self._assert_command(RebaselineExpectations(), expected_logs='Did not find any tests marked REBASELINE.\n')
     334        self._assert_command(RebaselineExpectations(), expected_logs='Did not find any tests marked Rebaseline.\n')
    335335
    336336    def test_overrides_are_included_correctly(self):
    … …  
    347347        port.expectations_dict = lambda: {
    348348            expectations_path: expectations_contents,
    349             'overrides': ('BUGX REBASELINE : userscripts/another-test.html = FAIL\n'
    350                           'BUGY : userscripts/test.html = CRASH\n')}
     349            'overrides': ('Bug(x) userscripts/another-test.html [ Failure Rebaseline ]\n'
     350                          'Bug(y) userscripts/test.html [ Crash ]\n')}
    351351
    352352        for path in port.expectations_files():
Note: See TracChangeset for help on using the changeset viewer.