Changeset 75576 in webkit


Ignore:
Timestamp:
Jan 11, 2011 5:15:46 PM (13 years ago)
Author:
dpranke@chromium.org
Message:

2011-01-11 Dirk Pranke <​dpranke@chromium.org>

Reviewed by Ojan Vafai.

The current modifier parsing code in test_expectations is
fragile and hard-coded, so it's not easy to understand the logic
or easily add new types of modifiers (like GPU vs. CPU testing
for graphics tests, or 32-bit vs. 64-bit differences).

This is the first of two patches that will add in more generic
support and then eliminate the GPU-specific test expectations
files for Chromium.

This patch adds two standalone objects for handling modifiers. The
rules for interpreting modifiers, precedence, and conflicts are
given in the docstring to the ModifierMatcher class, which
returns ModifierMatchResult objects.

This patch also adds routines to the Port interface and a
default set of values in the base object, in order to obtain the
values needed on a given test run. These values are then passed
to the expectation parser. This also allows us to clean up the
logic used to lint all of the different configurations in a
single test_expectations.txt file.

The next patch will merge in the separate GPU expectations file.

​https://bugs.webkit.org/show_bug.cgi?id=51222

  • Scripts/webkitpy/layout_tests/layout_package/printing_unittest.py:
  • Scripts/webkitpy/layout_tests/layout_package/test_expectations.py:
  • Scripts/webkitpy/layout_tests/layout_package/test_expectations_unittest.py:
  • Scripts/webkitpy/layout_tests/port/base.py:
  • Scripts/webkitpy/layout_tests/port/base_unittest.py:
  • Scripts/webkitpy/layout_tests/port/chromium.py:
  • Scripts/webkitpy/layout_tests/port/port_testcase.py:
  • Scripts/webkitpy/layout_tests/port/test.py:
  • Scripts/webkitpy/layout_tests/run_webkit_tests.py:
  • Scripts/webkitpy/layout_tests/run_webkit_tests_unittest.py:
Location:
trunk/Tools
Files:
13 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r75575 r75576  
     12011-01-11  Dirk Pranke  <dpranke@chromium.org>
     2
     3        Reviewed by Ojan Vafai.
     4
     5        The current modifier parsing code in test_expectations is
     6        fragile and hard-coded, so it's not easy to understand the logic
     7        or easily add new types of modifiers (like GPU vs. CPU testing
     8        for graphics tests, or 32-bit vs. 64-bit differences).
     9
     10        This is the first of two patches that will add in more generic
     11        support and then eliminate the GPU-specific test expectations
     12        files for Chromium.
     13
     14        This patch adds two standalone objects for handling modifiers. The
     15        rules for interpreting modifiers, precedence, and conflicts are
     16        given in the docstring to the ModifierMatcher class, which
     17        returns ModifierMatchResult objects.
     18       
     19        This patch also adds routines to the Port interface and a
     20        default set of values in the base object, in order to obtain the
     21        values needed on a given test run. These values are then passed
     22        to the expectation parser. This also allows us to clean up the
     23        logic used to lint all of the different configurations in a
     24        single test_expectations.txt file.
     25
     26        The next patch will merge in the separate GPU expectations file.
     27
     28        https://bugs.webkit.org/show_bug.cgi?id=51222
     29
     30        * Scripts/webkitpy/layout_tests/layout_package/printing_unittest.py:
     31        * Scripts/webkitpy/layout_tests/layout_package/test_expectations.py:
     32        * Scripts/webkitpy/layout_tests/layout_package/test_expectations_unittest.py:
     33        * Scripts/webkitpy/layout_tests/port/base.py:
     34        * Scripts/webkitpy/layout_tests/port/base_unittest.py:
     35        * Scripts/webkitpy/layout_tests/port/chromium.py:
     36        * Scripts/webkitpy/layout_tests/port/port_testcase.py:
     37        * Scripts/webkitpy/layout_tests/port/test.py:
     38        * Scripts/webkitpy/layout_tests/run_webkit_tests.py:
     39        * Scripts/webkitpy/layout_tests/run_webkit_tests_unittest.py:
     40
    1412011-01-11  Maciej Stachowiak  <mjs@apple.com>
    242

  • trunk/Tools/Scripts/webkitpy/layout_tests/layout_package/printing_unittest.py

    r75114 r75576  
    3232import os
    3333import optparse
    34 import pdb
    3534import sys
    3635import unittest
    … …  
    147146        expectations = test_expectations.TestExpectations(
    148147            self._port, test_paths, expectations_str,
    149             self._port.test_platform_name(), is_debug_mode=False,
     148            self._port.test_configuration(),
    150149            is_lint_mode=False)
    151150
    … …  
    366365        tests = ['passes/text.html', 'failures/expected/timeout.html',
    367366                 'failures/expected/crash.html']
    368         expectations = 'failures/expected/timeout.html = TIMEOUT'
     367        expectations = 'BUGX : failures/expected/timeout.html = TIMEOUT'
    369368
    370369        # first, test that it is disabled properly
    … …  
    572571
    573572        expectations = """
    574 failures/expected/crash.html = CRASH
    575 failures/expected/timeout.html = TIMEOUT
     573BUGX : failures/expected/crash.html = CRASH
     574BUGX : failures/expected/timeout.html = TIMEOUT
    576575"""
    577576        err.reset()

  • trunk/Tools/Scripts/webkitpy/layout_tests/layout_package/test_expectations.py

    r74070 r75576  
    3232"""
    3333
     34import itertools
    3435import logging
    3536import os
    … …  
    8788    TEST_LIST = "test_expectations.txt"
    8889
    89     def __init__(self, port, tests, expectations, test_platform_name,
    90                  is_debug_mode, is_lint_mode, overrides=None):
     90    def __init__(self, port, tests, expectations, test_config,
     91                 is_lint_mode, overrides=None):
    9192        """Loads and parses the test expectations given in the string.
    9293        Args:
    … …  
    9495            test: list of all of the test files
    9596            expectations: test expectations as a string
    96             test_platform_name: name of the platform to match expectations
    97                 against. Note that this may be different than
    98                 port.test_platform_name() when is_lint_mode is True.
    99             is_debug_mode: whether to use the DEBUG or RELEASE modifiers
    100                 in the expectations
     97            test_config: specific values to check against when
     98                parsing the file (usually port.test_config(),
     99                but may be different when linting or doing other things).
    101100            is_lint_mode: If True, just parse the expectations string
    102101                looking for errors.
    … …  
    107106        """
    108107        self._expected_failures = TestExpectationsFile(port, expectations,
    109             tests, test_platform_name, is_debug_mode, is_lint_mode,
     108            tests, test_config, is_lint_mode,
    110109            overrides=overrides)
    111110
    … …  
    200199
    201200    def __repr__(self):
    202         return 'ParseError(fatal=%s, errors=%s)' % (fatal, errors)
     201        return 'ParseError(fatal=%s, errors=%s)' % (self.fatal, self.errors)
    203202
    204203
    … …  
    305304                    'flaky': FLAKY}
    306305
    307     def __init__(self, port, expectations, full_test_list, test_platform_name,
    308         is_debug_mode, is_lint_mode, overrides=None):
    309         """
    310         expectations: Contents of the expectations file
    311         full_test_list: The list of all tests to be run pending processing of
    312             the expections for those tests.
    313         test_platform_name: name of the platform to match expectations
    314             against. Note that this may be different than
    315             port.test_platform_name() when is_lint_mode is True.
    316         is_debug_mode: Whether we testing a test_shell built debug mode.
    317         is_lint_mode: Whether this is just linting test_expecatations.txt.
    318         overrides: test expectations that are allowed to override any
    319             entries in |expectations|. This is used by callers
    320             that need to manage two sets of expectations (e.g., upstream
    321             and downstream expectations).
    322         """
     306    def __init__(self, port, expectations, full_test_list,
     307                 test_config, is_lint_mode, overrides=None):
     308        # See argument documentation in TestExpectation(), above.
    323309
    324310        self._port = port
    325311        self._expectations = expectations
    326312        self._full_test_list = full_test_list
    327         self._test_platform_name = test_platform_name
    328         self._is_debug_mode = is_debug_mode
     313        self._test_config = test_config
    329314        self._is_lint_mode = is_lint_mode
    330315        self._overrides = overrides
    … …  
    334319        # Maps relative test paths as listed in the expectations file to a
    335320        # list of maps containing modifiers and expectations for each time
    336         # the test is listed in the expectations file.
     321        # the test is listed in the expectations file. We use this to
     322        # keep a representation of the entire list of expectations, even
     323        # invalid ones.
    337324        self._all_expectations = {}
    338325
    … …  
    347334        self._test_to_modifiers = {}
    348335
    349         # Maps a test to the base path that it was listed with in the list.
     336        # Maps a test to the base path that it was listed with in the list and
     337        # the number of matches that base path had.
    350338        self._test_list_paths = {}
    351339
    … …  
    374362    def _handle_any_read_errors(self):
    375363        if len(self._errors) or len(self._non_fatal_errors):
    376             if self._is_debug_mode:
    377                 build_type = 'DEBUG'
    378             else:
    379                 build_type = 'RELEASE'
    380             _log.error('')
    381             _log.error("FAILURES FOR PLATFORM: %s, BUILD_TYPE: %s" %
    382                        (self._test_platform_name.upper(), build_type))
     364            _log.error("FAILURES FOR %s" % str(self._test_config))
    383365
    384366            for error in self._errors:
    … …  
    396378        options = []
    397379        modifiers = []
     380        num_matches = 0
    398381        if self._full_test_list:
    399382            for test in self._full_test_list:
    400383                if not test in self._test_list_paths:
    401                     self._add_test(test, modifiers, expectations, options,
    402                         overrides_allowed=False)
     384                    self._add_test(test, modifiers, num_matches, expectations,
     385                                   options, overrides_allowed=False)
    403386
    404387    def _dict_of_sets(self, strings_to_constants):
    … …  
    539522        options = []
    540523        if line.find(":") is -1:
    541             test_and_expectation = line.split("=")
    542         else:
    543             parts = line.split(":")
    544             options = self._get_options_list(parts[0])
    545             test_and_expectation = parts[1].split('=')
    546 
     524            self._add_error(lineno, "Missing a ':'", line)
     525            return (None, None, None)
     526
     527        parts = line.split(':')
     528
     529        # FIXME: verify that there is exactly one colon in the line.
     530
     531        options = self._get_options_list(parts[0])
     532        test_and_expectation = parts[1].split('=')
    547533        test = test_and_expectation[0].strip()
    548534        if (len(test_and_expectation) is not 2):
    … …  
    590576        return REMOVE_TEST
    591577
    592     def _has_valid_modifiers_for_current_platform(self, options, lineno,
    593         test_and_expectations, modifiers):
    594         """Returns true if the current platform is in the options list or if
    595         no platforms are listed and if there are no fatal errors in the
    596         options list.
    597 
    598         Args:
    599           options: List of lowercase options.
    600           lineno: The line in the file where the test is listed.
    601           test_and_expectations: The path and expectations for the test.
    602           modifiers: The set to populate with modifiers.
    603         """
    604         has_any_platform = False
    605         has_bug_id = False
    606         for option in options:
    607             if option in self.MODIFIERS:
    608                 modifiers.add(option)
    609             elif option in self._port.test_platform_names():
    610                 has_any_platform = True
    611             elif re.match(r'bug\d', option) != None:
    612                 self._add_error(lineno, 'Bug must be either BUGCR, BUGWK, or BUGV8_ for test: %s' %
    613                                 option, test_and_expectations)
    614             elif option.startswith('bug'):
    615                 has_bug_id = True
    616             elif option not in self.BUILD_TYPES:
    617                 self._add_error(lineno, 'Invalid modifier for test: %s' %
    618                                 option, test_and_expectations)
    619 
    620         if has_any_platform and not self._match_platform(options):
    621             return False
    622 
    623         if not has_bug_id and 'wontfix' not in options:
    624             # TODO(ojan): Turn this into an AddError call once all the
    625             # tests have BUG identifiers.
    626             self._log_non_fatal_error(lineno, 'Test lacks BUG modifier.',
    627                 test_and_expectations)
    628 
    629         if 'release' in options or 'debug' in options:
    630             if self._is_debug_mode and 'debug' not in options:
    631                 return False
    632             if not self._is_debug_mode and 'release' not in options:
    633                 return False
    634 
    635         if self._is_lint_mode and 'rebaseline' in options:
    636             self._add_error(lineno,
    637                 'REBASELINE should only be used for running rebaseline.py. '
    638                 'Cannot be checked in.', test_and_expectations)
    639 
    640         return True
    641 
    642     def _match_platform(self, options):
    643         """Match the list of options against our specified platform. If any
    644         of the options prefix-match self._platform, return True. This handles
    645         the case where a test is marked WIN and the platform is WIN-VISTA.
    646 
    647         Args:
    648           options: list of options
    649         """
    650         for opt in options:
    651             if self._test_platform_name.startswith(opt):
    652                 return True
    653         return False
    654 
    655578    def _add_to_all_expectations(self, test, options, expectations):
    656579        # Make all paths unix-style so the dashboard doesn't need to.
    … …  
    665588        expectations for it."""
    666589        lineno = 0
     590        matcher = ModifierMatcher(self._test_config)
    667591        for line in expectations:
    668592            lineno += 1
    669 
    670             test_list_path, options, expectations = \
    671                 self.parse_expectations_line(line, lineno)
    672             if not expectations:
    673                 continue
    674 
    675             self._add_to_all_expectations(test_list_path,
    676                                           " ".join(options).upper(),
    677                                           " ".join(expectations).upper())
    678 
    679             modifiers = set()
    680             if options and not self._has_valid_modifiers_for_current_platform(
    681                 options, lineno, test_list_path, modifiers):
    682                 continue
    683 
    684             expectations = self._parse_expectations(expectations, lineno,
    685                 test_list_path)
    686 
    687             if 'slow' in options and TIMEOUT in expectations:
    688                 self._add_error(lineno,
    689                     'A test can not be both slow and timeout. If it times out '
    690                     'indefinitely, then it should be just timeout.',
    691                     test_list_path)
    692 
    693             full_path = os.path.join(self._port.layout_tests_dir(),
    694                                      test_list_path)
    695             full_path = os.path.normpath(full_path)
    696             # WebKit's way of skipping tests is to add a -disabled suffix.
    697             # So we should consider the path existing if the path or the
    698             # -disabled version exists.
    699             if (not self._port.path_exists(full_path)
    700                 and not self._port.path_exists(full_path + '-disabled')):
    701                 # Log a non fatal error here since you hit this case any
    702                 # time you update test_expectations.txt without syncing
    703                 # the LayoutTests directory
    704                 self._log_non_fatal_error(lineno, 'Path does not exist.',
    705                                        test_list_path)
    706                 continue
    707 
    708             if not self._full_test_list:
    709                 tests = [test_list_path]
    710             else:
    711                 tests = self._expand_tests(test_list_path)
    712 
    713             self._add_tests(tests, expectations, test_list_path, lineno,
    714                            modifiers, options, overrides_allowed)
     593            self._process_line(line, lineno, matcher, overrides_allowed)
     594
     595    def _process_line(self, line, lineno, matcher, overrides_allowed):
     596        test_list_path, options, expectations = \
     597            self.parse_expectations_line(line, lineno)
     598        if not expectations:
     599            return
     600
     601        self._add_to_all_expectations(test_list_path,
     602                                        " ".join(options).upper(),
     603                                        " ".join(expectations).upper())
     604
     605        num_matches = self._check_options(matcher, options, lineno,
     606                                          test_list_path)
     607        if num_matches == ModifierMatcher.NO_MATCH:
     608            return
     609
     610        expectations = self._parse_expectations(expectations, lineno,
     611            test_list_path)
     612
     613        self._check_options_against_expectations(options, expectations,
     614            lineno, test_list_path)
     615
     616        if self._check_path_does_not_exist(lineno, test_list_path):
     617            return
     618
     619        if not self._full_test_list:
     620            tests = [test_list_path]
     621        else:
     622            tests = self._expand_tests(test_list_path)
     623
     624        modifiers = [o for o in options if o in self.MODIFIERS]
     625        self._add_tests(tests, expectations, test_list_path, lineno,
     626                        modifiers, num_matches, options, overrides_allowed)
    715627
    716628    def _get_options_list(self, listString):
    … …  
    728640        return result
    729641
     642    def _check_options(self, matcher, options, lineno, test_list_path):
     643        match_result = self._check_syntax(matcher, options, lineno,
     644                                          test_list_path)
     645        self._check_semantics(options, lineno, test_list_path)
     646        return match_result.num_matches
     647
     648    def _check_syntax(self, matcher, options, lineno, test_list_path):
     649        match_result = matcher.match(options)
     650        for error in match_result.errors:
     651            self._add_error(lineno, error, test_list_path)
     652        for warning in match_result.warnings:
     653            self._log_non_fatal_error(lineno, warning, test_list_path)
     654        return match_result
     655
     656    def _check_semantics(self, options, lineno, test_list_path):
     657        has_wontfix = 'wontfix' in options
     658        has_bug = False
     659        for opt in options:
     660            if opt.startswith('bug'):
     661                has_bug = True
     662                if re.match('bug\d+', opt):
     663                    self._add_error(lineno,
     664                        'BUG\d+ is not allowed, must be one of '
     665                        'BUGCR\d+, BUGWK\d+, BUGV8_\d+, '
     666                        'or a non-numeric bug identifier.', test_list_path)
     667
     668        if not has_bug and not has_wontfix:
     669            self._log_non_fatal_error(lineno, 'Test lacks BUG modifier.',
     670            test_list_path)
     671
     672        if self._is_lint_mode and 'rebaseline' in options:
     673            self._add_error(lineno,
     674            'REBASELINE should only be used for running rebaseline.py. '
     675            'Cannot be checked in.', test_list_path)
     676
     677    def _check_options_against_expectations(self, options, expectations,
     678                                            lineno, test_list_path):
     679        if 'slow' in options and TIMEOUT in expectations:
     680            self._add_error(lineno,
     681                'A test can not be both slow and timeout. If it times out '
     682                'indefinitely, then it should be just timeout.',
     683                test_list_path)
     684
     685    def _check_path_does_not_exist(self, lineno, test_list_path):
     686        full_path = os.path.join(self._port.layout_tests_dir(),
     687                                    test_list_path)
     688        full_path = os.path.normpath(full_path)
     689        # WebKit's way of skipping tests is to add a -disabled suffix.
     690            # So we should consider the path existing if the path or the
     691        # -disabled version exists.
     692        if (not self._port.path_exists(full_path)
     693            and not self._port.path_exists(full_path + '-disabled')):
     694            # Log a non fatal error here since you hit this case any
     695            # time you update test_expectations.txt without syncing
     696            # the LayoutTests directory
     697            self._log_non_fatal_error(lineno, 'Path does not exist.',
     698                                    test_list_path)
     699            return True
     700        return False
     701
    730702    def _expand_tests(self, test_list_path):
    731703        """Convert the test specification to an absolute, normalized
    … …  
    753725
    754726    def _add_tests(self, tests, expectations, test_list_path, lineno,
    755                    modifiers, options, overrides_allowed):
     727                   modifiers, num_matches, options, overrides_allowed):
    756728        for test in tests:
    757             if self._already_seen_test(test, test_list_path, lineno,
    758                                       overrides_allowed):
     729            if self._already_seen_better_match(test, test_list_path,
     730                num_matches, lineno, overrides_allowed):
    759731                continue
    760732
    761733            self._clear_expectations_for_test(test, test_list_path)
    762             self._add_test(test, modifiers, expectations, options,
     734            self._test_list_paths[test] = (os.path.normpath(test_list_path),
     735                num_matches, lineno)
     736            self._add_test(test, modifiers, num_matches, expectations, options,
    763737                           overrides_allowed)
    764738
    765     def _add_test(self, test, modifiers, expectations, options,
     739    def _add_test(self, test, modifiers, num_matches, expectations, options,
    766740                  overrides_allowed):
    767741        """Sets the expected state for a given test.
    … …  
    774748          test: test to add
    775749          modifiers: sequence of modifier keywords ('wontfix', 'slow', etc.)
     750          num_matches: number of modifiers that matched the configuration
    776751          expectations: sequence of expectations (PASS, IMAGE, etc.)
    777752          options: sequence of keywords and bug identifiers.
    … …  
    812787        """
    813788        if test in self._test_list_paths:
    814             self._test_to_expectations.pop(test, '')
    815             self._remove_from_sets(test, self._expectation_to_tests)
    816             self._remove_from_sets(test, self._modifier_to_tests)
    817             self._remove_from_sets(test, self._timeline_to_tests)
    818             self._remove_from_sets(test, self._result_type_to_tests)
    819 
    820         self._test_list_paths[test] = os.path.normpath(test_list_path)
     789            base_path = self._test_list_paths[test][0]
     790            self._test_to_expectations.pop(base_path, '')
     791            self._remove_from_sets(base_path, self._expectation_to_tests)
     792            self._remove_from_sets(base_path, self._modifier_to_tests)
     793            self._remove_from_sets(base_path, self._timeline_to_tests)
     794            self._remove_from_sets(base_path, self._result_type_to_tests)
     795
    821796
    822797    def _remove_from_sets(self, test, dict):
    … …  
    830805                set_of_tests.remove(test)
    831806
    832     def _already_seen_test(self, test, test_list_path, lineno,
    833                            allow_overrides):
    834         """Returns true if we've already seen a more precise path for this test
    835         than the test_list_path.
     807    def _already_seen_better_match(self, test, test_list_path, num_matches,
     808                                   lineno, overrides_allowed):
     809        """Returns whether we've seen a better match already in the file.
     810
     811        Returns True if we've already seen a test_list_path that matches more of the test
     812            than this path does
    836813        """
     814        # FIXME: See comment below about matching test configs and num_matches.
     815
    837816        if not test in self._test_list_paths:
     817            # We've never seen this test before.
    838818            return False
    839819
    840         prev_base_path = self._test_list_paths[test]
    841         if (prev_base_path == os.path.normpath(test_list_path)):
    842             if (not allow_overrides or test in self._overridding_tests):
    843                 if allow_overrides:
    844                     expectation_source = "override"
    845                 else:
    846                     expectation_source = "expectation"
    847                 self._add_error(lineno, 'Duplicate %s.' % expectation_source,
    848                                    test)
    849                 return True
    850             else:
    851                 # We have seen this path, but that's okay because its
    852                 # in the overrides and the earlier path was in the
    853                 # expectations.
    854                 return False
    855 
    856         # Check if we've already seen a more precise path.
    857         return prev_base_path.startswith(os.path.normpath(test_list_path))
     820        prev_base_path, prev_num_matches, prev_lineno = self._test_list_paths[test]
     821        base_path = os.path.normpath(test_list_path)
     822
     823        if len(prev_base_path) > len(base_path):
     824            # The previous path matched more of the test.
     825            return True
     826
     827        if len(prev_base_path) < len(base_path):
     828            # This path matches more of the test.
     829            return False
     830
     831        if overrides_allowed and test not in self._overridding_tests:
     832            # We have seen this path, but that's okay because it is
     833            # in the overrides and the earlier path was in the
     834            # expectations (not the overrides).
     835            return False
     836
     837        # At this point we know we have seen a previous exact match on this
     838        # base path, so we need to check the two sets of modifiers.
     839
     840        if overrides_allowed:
     841            expectation_source = "override"
     842        else:
     843            expectation_source = "expectation"
     844
     845        # FIXME: This code was originally designed to allow lines that matched
     846        # more modifiers to override lines that matched fewer modifiers.
     847        # However, we currently view these as errors. If we decide to make
     848        # this policy permanent, we can probably simplify this code
     849        # and the ModifierMatcher code a fair amount.
     850        #
     851        # To use the "more modifiers wins" policy, change the "_add_error" lines for overrides
     852        # to _log_non_fatal_error() and change the commented-out "return False".
     853
     854        if prev_num_matches == num_matches:
     855            self._add_error(lineno,
     856                'Duplicate or ambiguous %s.' % expectation_source,
     857                test)
     858            return True
     859
     860        if prev_num_matches < num_matches:
     861            self._add_error(lineno,
     862                'More specific entry on line %d overrides line %d' %
     863                (lineno, prev_lineno), test_list_path)
     864            # FIXME: return False if we want more specific to win.
     865            return True
     866
     867        self._add_error(lineno,
     868            'More specific entry on line %d overrides line %d' %
     869            (prev_lineno, lineno), test_list_path)
     870        return True
    858871
    859872    def _add_error(self, lineno, msg, path):
    … …  
    867880        still errors, but not bad enough to warrant breaking test running."""
    868881        self._non_fatal_errors.append('Line:%s %s %s' % (lineno, msg, path))
     882
     883
     884class ModifierMatchResult(object):
     885    def __init__(self, options):
     886        self.num_matches = ModifierMatcher.NO_MATCH
     887        self.options = options
     888        self.errors = []
     889        self.warnings = []
     890        self.modifiers = []
     891        self._matched_regexes = set()
     892        self._matched_macros = set()
     893
     894
     895class ModifierMatcher(object):
     896
     897    """
     898    This class manages the interpretation of the "modifiers" for a given
     899    line in the expectations file. Modifiers are the tokens that appear to the
     900    left of the colon on a line. For example, "BUG1234", "DEBUG", and "WIN" are
     901    all modifiers. This class gets what the valid modifiers are, and which
     902    modifiers are allowed to exist together on a line, from the
     903    TestConfiguration object that is passed in to the call.
     904
     905    This class detects *intra*-line errors like unknown modifiers, but
     906    does not detect *inter*-line modifiers like duplicate expectations.
     907
     908    More importantly, this class is also used to determine if a given line
     909    matches the port in question. Matches are ranked according to the number
     910    of modifiers that match on a line. A line with no modifiers matches
     911    everything and has a score of zero. A line with one modifier matches only
     912    ports that have that modifier and gets a score of 1, and so one. Ports
     913    that don't match at all get a score of -1.
     914
     915    Given two lines in a file that apply to the same test, if both expectations
     916    match the current config, then the expectation is considered ambiguous,
     917    even if one expectation matches more of the config than the other. For
     918    example, in:
     919
     920    BUG1 RELEASE : foo.html = FAIL
     921    BUG1 WIN RELEASE : foo.html = PASS
     922    BUG2 WIN : bar.html = FAIL
     923    BUG2 DEBUG : bar.html = PASS
     924
     925    lines 1 and 2 would produce an error on a Win XP Release bot (the scores
     926    would be 1 and 2, respectively), and lines three and four would produce
     927    a duplicate expectation on a Win Debug bot since both the 'win' and the
     928    'debug' expectations would apply (both had scores of 1).
     929
     930    In addition to the definitions of all of the modifiers, the class
     931    supports "macros" that are expanded prior to interpretation, and "ignore
     932    regexes" that can be used to skip over modifiers like the BUG* modifiers.
     933    """
     934    MACROS = {
     935        'mac-snowleopard': ['mac', 'snowleopard'],
     936        'mac-leopard': ['mac', 'leopard'],
     937        'win-xp': ['win', 'xp'],
     938        'win-vista': ['win', 'vista'],
     939        'win-7': ['win', 'win7'],
     940    }
     941
     942    # We don't include the "none" modifier because it isn't actually legal.
     943    REGEXES_TO_IGNORE = (['bug\w+'] +
     944                         TestExpectationsFile.MODIFIERS.keys()[:-1])
     945    DUPLICATE_REGEXES_ALLOWED = ['bug\w+']
     946
     947    # Magic value returned when the options don't match.
     948    NO_MATCH = -1
     949
     950    # FIXME: The code currently doesn't detect combinations of modifiers
     951    # that are syntactically valid but semantically invalid, like
     952    # 'MAC XP'. See ModifierMatchTest.test_invalid_combinations() in the
     953    # _unittest.py file.
     954
     955    def __init__(self, test_config):
     956        """Initialize a ModifierMatcher argument with the TestConfiguration it
     957        should be matched against."""
     958        self.test_config = test_config
     959        self.allowed_configurations = test_config.all_test_configurations()
     960        self.macros = self.MACROS
     961
     962        self.regexes_to_ignore = {}
     963        for regex_str in self.REGEXES_TO_IGNORE:
     964            self.regexes_to_ignore[regex_str] = re.compile(regex_str)
     965
     966        # Keep a set of all of the legal modifiers for quick checking.
     967        self._all_modifiers = set()
     968
     969        # Keep a dict mapping values back to their categories.
     970        self._categories_for_modifiers = {}
     971        for config in self.allowed_configurations:
     972            for category, modifier in config.items():
     973                self._categories_for_modifiers[modifier] = category
     974                self._all_modifiers.add(modifier)
     975
     976    def match(self, options):
     977        """Checks a list of options against the config set in the constructor.
     978        Options may be either actual modifier strings, "macro" strings
     979        that get expanded to a list of modifiers, or strings that are allowed
     980        to be ignored. All of the options must be passed in in lower case.
     981
     982        Returns the number of matching categories, or NO_MATCH (-1) if it
     983        doesn't match or there were errors found. Matches are prioritized
     984        by the number of matching categories, because the more specific
     985        the options list, the more categories will match.
     986
     987        The results of the most recent match are available in the 'options',
     988        'modifiers', 'num_matches', 'errors', and 'warnings' properties.
     989        """
     990        result = ModifierMatchResult(options)
     991        self._parse(result)
     992        if result.errors:
     993            return result
     994        self._count_matches(result)
     995        return result
     996
     997    def _parse(self, result):
     998        # FIXME: Should we warn about lines having every value in a category?
     999        for option in result.options:
     1000            self._parse_one(option, result)
     1001
     1002    def _parse_one(self, option, result):
     1003        if option in self._all_modifiers:
     1004            self._add_modifier(option, result)
     1005        elif option in self.macros:
     1006            self._expand_macro(option, result)
     1007        elif not self._matches_any_regex(option, result):
     1008            result.errors.append("Unrecognized option '%s'" % option)
     1009
     1010    def _add_modifier(self, option, result):
     1011        if option in result.modifiers:
     1012            result.errors.append("More than one '%s'" % option)
     1013        else:
     1014            result.modifiers.append(option)
     1015
     1016    def _expand_macro(self, macro, result):
     1017        if macro in result._matched_macros:
     1018            result.errors.append("More than one '%s'" % macro)
     1019            return
     1020
     1021        mods = []
     1022        for modifier in self.macros[macro]:
     1023            if modifier in result.options:
     1024                result.errors.append("Can't specify both modifier '%s' and "
     1025                                     "macro '%s'" % (modifier, macro))
     1026            else:
     1027                mods.append(modifier)
     1028        result._matched_macros.add(macro)
     1029        result.modifiers.extend(mods)
     1030
     1031    def _matches_any_regex(self, option, result):
     1032        for regex_str, pattern in self.regexes_to_ignore.iteritems():
     1033            if pattern.match(option):
     1034                self._handle_regex_match(regex_str, result)
     1035                return True
     1036        return False
     1037
     1038    def _handle_regex_match(self, regex_str, result):
     1039        if (regex_str in result._matched_regexes and
     1040            regex_str not in self.DUPLICATE_REGEXES_ALLOWED):
     1041            result.errors.append("More than one option matching '%s'" %
     1042                                 regex_str)
     1043        else:
     1044            result._matched_regexes.add(regex_str)
     1045
     1046    def _count_matches(self, result):
     1047        """Returns the number of modifiers that match the test config."""
     1048        categorized_modifiers = self._group_by_category(result.modifiers)
     1049        result.num_matches = 0
     1050        for category, modifier in self.test_config.items():
     1051            if category in categorized_modifiers:
     1052                if modifier in categorized_modifiers[category]:
     1053                    result.num_matches += 1
     1054                else:
     1055                    result.num_matches = self.NO_MATCH
     1056                    return
     1057
     1058    def _group_by_category(self, modifiers):
     1059        # Returns a dict of category name -> list of modifiers.
     1060        modifiers_by_category = {}
     1061        for m in modifiers:
     1062            modifiers_by_category.setdefault(self._category(m), []).append(m)
     1063        return modifiers_by_category
     1064
     1065    def _category(self, modifier):
     1066        return self._categories_for_modifiers[modifier]

  • trunk/Tools/Scripts/webkitpy/layout_tests/layout_package/test_expectations_unittest.py

    r74036 r75576  
    3535
    3636from webkitpy.layout_tests import port
     37from webkitpy.layout_tests.port import base
    3738from webkitpy.layout_tests.layout_package.test_expectations import *
    3839
    … …  
    8182
    8283class Base(unittest.TestCase):
     84    # Note that all of these tests are written assuming the configuration
     85    # being tested is Mac Leopard, Release build.
     86
    8387    def __init__(self, testFunc, setUp=None, tearDown=None, description=None):
    8488        self._port = port.get('test', None)
    … …  
    106110"""
    107111
    108     def parse_exp(self, expectations, overrides=None, is_lint_mode=False,
    109                   is_debug_mode=False):
     112    def parse_exp(self, expectations, overrides=None, is_lint_mode=False):
     113        test_config = self._port.test_configuration()
    110114        self._exp = TestExpectations(self._port,
    111115             tests=self.get_basic_tests(),
    112116             expectations=expectations,
    113              test_platform_name=self._port.test_platform_name(),
    114              is_debug_mode=is_debug_mode,
     117             test_config=test_config,
    115118             is_lint_mode=is_lint_mode,
    116119             overrides=overrides)
    … …  
    121124
    122125
    123 class TestExpectationsTest(Base):
     126class BasicTests(Base):
    124127    def test_basic(self):
    125128        self.parse_exp(self.get_basic_expectations())
    … …  
    129132        self.assert_exp('failures/expected/image.html', PASS)
    130133
     134
     135class MiscTests(Base):
    131136    def test_multiple_results(self):
    132137        self.parse_exp('BUGX : failures/expected/text.html = TEXT CRASH')
    … …  
    134139            self.get_test('failures/expected/text.html')),
    135140            set([TEXT, CRASH]))
    136 
    137     def test_precedence(self):
    138         # This tests handling precedence of specific lines over directories
    139         # and tests expectations covering entire directories.
    140         exp_str = """
    141 BUGX : failures/expected/text.html = TEXT
    142 BUGX WONTFIX : failures/expected = IMAGE
    143 """
    144         self.parse_exp(exp_str)
    145         self.assert_exp('failures/expected/text.html', TEXT)
    146         self.assert_exp('failures/expected/crash.html', IMAGE)
    147141
    148142    def test_category_expectations(self):
    … …  
    159153                          unknown_test)
    160154        self.assert_exp('failures/expected/crash.html', IMAGE)
    161 
    162     def test_release_mode(self):
    163         self.parse_exp('BUGX DEBUG : failures/expected/text.html = TEXT',
    164                        is_debug_mode=True)
    165         self.assert_exp('failures/expected/text.html', TEXT)
    166         self.parse_exp('BUGX RELEASE : failures/expected/text.html = TEXT',
    167                        is_debug_mode=True)
    168         self.assert_exp('failures/expected/text.html', PASS)
    169         self.parse_exp('BUGX DEBUG : failures/expected/text.html = TEXT',
    170                        is_debug_mode=False)
    171         self.assert_exp('failures/expected/text.html', PASS)
    172         self.parse_exp('BUGX RELEASE : failures/expected/text.html = TEXT',
    173                        is_debug_mode=False)
    174         self.assert_exp('failures/expected/text.html', TEXT)
    175155
    176156    def test_get_options(self):
    … …  
    218198        except ParseError, e:
    219199            self.assertTrue(e.fatal)
    220             exp_errors = [u'Line:1 Invalid modifier for test: foo failures/expected/text.html',
     200            exp_errors = [u"Line:1 Unrecognized option 'foo' failures/expected/text.html",
    221201                          u"Line:2 Missing expectations. [' failures/expected/image.html']"]
    222202            self.assertEqual(str(e), '\n'.join(map(str, exp_errors)))
    … …  
    234214            self.assertEqual(e.errors, exp_errors)
    235215
    236     def test_syntax_missing_expectation(self):
    237         # This is missing the expectation.
    238         self.assertRaises(ParseError, self.parse_exp,
    239                           'BUG_TEST: failures/expected/text.html',
    240                           is_debug_mode=True)
    241 
    242     def test_syntax_invalid_option(self):
    243         self.assertRaises(ParseError, self.parse_exp,
    244                           'BUG_TEST FOO: failures/expected/text.html = PASS')
    245 
    246     def test_syntax_invalid_expectation(self):
    247         # This is missing the expectation.
    248         self.assertRaises(ParseError, self.parse_exp,
    249                           'BUG_TEST: failures/expected/text.html = FOO')
    250 
    251     def test_syntax_missing_bugid(self):
    252         # This should log a non-fatal error.
    253         self.parse_exp('SLOW : failures/expected/text.html = TEXT')
    254         self.assertEqual(
    255             len(self._exp._expected_failures.get_non_fatal_errors()), 1)
    256 
    257     def test_semantic_slow_and_timeout(self):
    258         # A test cannot be SLOW and expected to TIMEOUT.
    259         self.assertRaises(ParseError, self.parse_exp,
    260             'BUG_TEST SLOW : failures/expected/timeout.html = TIMEOUT')
    261 
    262     def test_semantic_rebaseline(self):
    263         # Can't lint a file w/ 'REBASELINE' in it.
    264         self.assertRaises(ParseError, self.parse_exp,
    265             'BUG_TEST REBASELINE : failures/expected/text.html = TEXT',
    266             is_lint_mode=True)
    267 
    268     def test_semantic_duplicates(self):
    269         self.assertRaises(ParseError, self.parse_exp, """
    270 BUG_TEST : failures/expected/text.html = TEXT
    271 BUG_TEST : failures/expected/text.html = IMAGE""")
    272 
    273         self.assertRaises(ParseError, self.parse_exp,
    274             self.get_basic_expectations(), """
    275 BUG_TEST : failures/expected/text.html = TEXT
    276 BUG_TEST : failures/expected/text.html = IMAGE""")
    277 
    278     def test_semantic_missing_file(self):
    279         # This should log a non-fatal error.
    280         self.parse_exp('BUG_TEST : missing_file.html = TEXT')
    281         self.assertEqual(
    282             len(self._exp._expected_failures.get_non_fatal_errors()), 1)
    283 
    284 
    285216    def test_overrides(self):
    286         self.parse_exp(self.get_basic_expectations(), """
    287 BUG_OVERRIDE : failures/expected/text.html = IMAGE""")
     217        self.parse_exp("BUG_EXP: failures/expected/text.html = TEXT",
     218                       "BUG_OVERRIDE : failures/expected/text.html = IMAGE")
    288219        self.assert_exp('failures/expected/text.html', IMAGE)
    289220
    290     def test_matches_an_expected_result(self):
    291 
     221    def test_overrides__duplicate(self):
     222        self.assertRaises(ParseError, self.parse_exp,
     223             "BUG_EXP: failures/expected/text.html = TEXT",
     224             """
     225BUG_OVERRIDE : failures/expected/text.html = IMAGE
     226BUG_OVERRIDE : failures/expected/text.html = CRASH
     227""")
     228
     229    def test_pixel_tests_flag(self):
    292230        def match(test, result, pixel_tests_enabled):
    293231            return self._exp.matches_an_expected_result(
    … …  
    307245
    308246
     247class ExpectationSyntaxTests(Base):
     248    def test_missing_expectation(self):
     249        # This is missing the expectation.
     250        self.assertRaises(ParseError, self.parse_exp,
     251                          'BUG_TEST: failures/expected/text.html')
     252
     253    def test_missing_colon(self):
     254        # This is missing the modifiers and the ':'
     255        self.assertRaises(ParseError, self.parse_exp,
     256                          'failures/expected/text.html = TEXT')
     257
     258    def disabled_test_too_many_colons(self):
     259        # FIXME: Enable this test and fix the underlying bug.
     260        self.assertRaises(ParseError, self.parse_exp,
     261                          'BUG_TEST: failures/expected/text.html = PASS :')
     262
     263    def test_too_many_equals_signs(self):
     264        self.assertRaises(ParseError, self.parse_exp,
     265                          'BUG_TEST: failures/expected/text.html = TEXT = IMAGE')
     266
     267    def test_unrecognized_expectation(self):
     268        self.assertRaises(ParseError, self.parse_exp,
     269                          'BUG_TEST: failures/expected/text.html = UNKNOWN')
     270
     271    def test_macro(self):
     272        exp_str = """
     273BUG_TEST MAC-LEOPARD : failures/expected/text.html = TEXT
     274"""
     275        self.parse_exp(exp_str)
     276        self.assert_exp('failures/expected/text.html', TEXT)
     277
     278
     279class SemanticTests(Base):
     280    def test_bug_format(self):
     281        self.assertRaises(ParseError, self.parse_exp, 'BUG1234 : failures/expected/text.html = TEXT')
     282
     283    def test_missing_bugid(self):
     284        # This should log a non-fatal error.
     285        self.parse_exp('SLOW : failures/expected/text.html = TEXT')
     286        self.assertEqual(
     287            len(self._exp._expected_failures.get_non_fatal_errors()), 1)
     288
     289    def test_slow_and_timeout(self):
     290        # A test cannot be SLOW and expected to TIMEOUT.
     291        self.assertRaises(ParseError, self.parse_exp,
     292            'BUG_TEST SLOW : failures/expected/timeout.html = TIMEOUT')
     293
     294    def test_rebaseline(self):
     295        # Can't lint a file w/ 'REBASELINE' in it.
     296        self.assertRaises(ParseError, self.parse_exp,
     297            'BUG_TEST REBASELINE : failures/expected/text.html = TEXT',
     298            is_lint_mode=True)
     299
     300    def test_duplicates(self):
     301        self.assertRaises(ParseError, self.parse_exp, """
     302BUG_EXP : failures/expected/text.html = TEXT
     303BUG_EXP : failures/expected/text.html = IMAGE""")
     304
     305        self.assertRaises(ParseError, self.parse_exp,
     306            self.get_basic_expectations(), overrides="""
     307BUG_OVERRIDE : failures/expected/text.html = TEXT
     308BUG_OVERRIDE : failures/expected/text.html = IMAGE""", )
     309
     310    def test_missing_file(self):
     311        # This should log a non-fatal error.
     312        self.parse_exp('BUG_TEST : missing_file.html = TEXT')
     313        self.assertEqual(
     314            len(self._exp._expected_failures.get_non_fatal_errors()), 1)
     315
     316
     317class PrecedenceTests(Base):
     318    def test_file_over_directory(self):
     319        # This tests handling precedence of specific lines over directories
     320        # and tests expectations covering entire directories.
     321        exp_str = """
     322BUGX : failures/expected/text.html = TEXT
     323BUGX WONTFIX : failures/expected = IMAGE
     324"""
     325        self.parse_exp(exp_str)
     326        self.assert_exp('failures/expected/text.html', TEXT)
     327        self.assert_exp('failures/expected/crash.html', IMAGE)
     328
     329        exp_str = """
     330BUGX WONTFIX : failures/expected = IMAGE
     331BUGX : failures/expected/text.html = TEXT
     332"""
     333        self.parse_exp(exp_str)
     334        self.assert_exp('failures/expected/text.html', TEXT)
     335        self.assert_exp('failures/expected/crash.html', IMAGE)
     336
     337    def test_ambiguous(self):
     338        self.assertRaises(ParseError, self.parse_exp, """
     339BUG_TEST RELEASE : passes/text.html = PASS
     340BUG_TEST MAC : passes/text.html = FAIL
     341""")
     342
     343    def test_more_modifiers(self):
     344        exp_str = """
     345BUG_TEST RELEASE : passes/text.html = PASS
     346BUG_TEST MAC RELEASE : passes/text.html = TEXT
     347"""
     348        self.assertRaises(ParseError, self.parse_exp, exp_str)
     349
     350    def test_order_in_file(self):
     351        exp_str = """
     352BUG_TEST MAC RELEASE : passes/text.html = TEXT
     353BUG_TEST RELEASE : passes/text.html = PASS
     354"""
     355        self.assertRaises(ParseError, self.parse_exp, exp_str)
     356
     357    def test_version_overrides(self):
     358        exp_str = """
     359BUG_TEST MAC : passes/text.html = PASS
     360BUG_TEST MAC LEOPARD : passes/text.html = TEXT
     361"""
     362        self.assertRaises(ParseError, self.parse_exp, exp_str)
     363
     364    def test_macro_overrides(self):
     365        exp_str = """
     366BUG_TEST MAC : passes/text.html = PASS
     367BUG_TEST MAC-LEOPARD : passes/text.html = TEXT
     368"""
     369        self.assertRaises(ParseError, self.parse_exp, exp_str)
     370
     371
    309372class RebaseliningTest(Base):
    310373    """Test rebaselining-specific functionality."""
    … …  
    347410
    348411
     412class ModifierTests(unittest.TestCase):
     413    def setUp(self):
     414        port_obj = port.get('test', None)
     415        self.config = port_obj.test_configuration()
     416        self.matcher = ModifierMatcher(self.config)
     417
     418    def match(self, modifiers, expected_num_matches=-1, values=None, num_errors=0):
     419        matcher = self.matcher
     420        if values:
     421            matcher = ModifierMatcher(self.FakeTestConfiguration(values))
     422        match_result = matcher.match(modifiers)
     423        self.assertEqual(len(match_result.warnings), 0)
     424        self.assertEqual(len(match_result.errors), num_errors)
     425        self.assertEqual(match_result.num_matches, expected_num_matches,
     426             'match(%s, %s) returned -> %d, expected %d' %
     427             (modifiers, str(self.config.values()),
     428              match_result.num_matches, expected_num_matches))
     429
     430    def test_bad_match_modifier(self):
     431        self.match(['foo'], num_errors=1)
     432
     433    def test_none(self):
     434        self.match([], 0)
     435
     436    def test_one(self):
     437        self.match(['leopard'], 1)
     438        self.match(['mac'], 1)
     439        self.match(['release'], 1)
     440        self.match(['cpu'], 1)
     441        self.match(['x86'], 1)
     442        self.match(['xp'], -1)
     443        self.match(['gpu'], -1)
     444        self.match(['debug'], -1)
     445
     446    def test_two(self):
     447        self.match(['leopard', 'release'], 2)
     448        self.match(['xp', 'release'], -1)
     449        self.match(['xp', 'release'], -1)
     450        self.match(['snowleopard', 'leopard'], 1)
     451
     452    def test_three(self):
     453        self.match(['snowleopard', 'leopard', 'release'], 2)
     454        self.match(['leopard', 'debug', 'x86'], -1)
     455        self.match(['leopard', 'release', 'x86'], 3)
     456        self.match(['leopard', 'cpu', 'release'], 3)
     457
     458    def test_four(self):
     459        self.match(['leopard', 'release', 'cpu', 'x86'], 4)
     460        self.match(['snowleopard', 'leopard', 'release', 'cpu'], 3)
     461        self.match(['snowleopard', 'leopard', 'debug', 'cpu'], -1)
     462
     463    def test_case_insensitivity(self):
     464        self.match(['Mac'], num_errors=1)
     465        self.match(['MAC'], num_errors=1)
     466        self.match(['mac'], 1)
     467
     468    def test_duplicates(self):
     469        self.match(['release', 'release'], num_errors=1)
     470        self.match(['mac-leopard', 'leopard'], num_errors=1)
     471        self.match(['mac-leopard', 'mac-leopard'], num_errors=1)
     472        self.match(['xp', 'release', 'xp', 'release'], num_errors=2)
     473        self.match(['rebaseline', 'rebaseline'], num_errors=1)
     474
     475    def test_unknown_option(self):
     476        self.match(['vms'], num_errors=1)
     477
     478    def test_duplicate_bugs(self):
     479        # BUG* regexes can appear multiple times.
     480        self.match(['bugfoo', 'bugbar'], 0)
     481
     482    def test_invalid_combinations(self):
     483        # FIXME: This should probably raise an error instead of NO_MATCH.
     484        self.match(['mac', 'xp'], num_errors=0)
     485
     486    def test_regexes_are_ignored(self):
     487        self.match(['bug123xy', 'rebaseline', 'wontfix', 'slow', 'skip'], 0)
     488
     489    def test_none_is_invalid(self):
     490        self.match(['none'], num_errors=1)
     491
     492
    349493if __name__ == '__main__':
    350494    unittest.main()

  • trunk/Tools/Scripts/webkitpy/layout_tests/layout_package/test_runner.py

    r75114 r75576  
    213213    def lint(self):
    214214        lint_failed = False
    215 
    216         # Creating the expecations for each platform/configuration pair does
    217         # all the test list parsing and ensures it's correct syntax (e.g. no
    218         # dupes).
    219         for platform_name in self._port.test_platform_names():
     215        for test_configuration in self._port.all_test_configurations():
    220216            try:
    221                 self.parse_expectations(platform_name, is_debug_mode=True)
     217                self.lint_expectations(test_configuration)
    222218            except test_expectations.ParseError:
    223219                lint_failed = True
    224             try:
    225                 self.parse_expectations(platform_name, is_debug_mode=False)
    226             except test_expectations.ParseError:
    227                 lint_failed = True
    228 
    229         self._printer.write("")
     220                self._printer.write("")
     221
    230222        if lint_failed:
    231223            _log.error("Lint failed.")
    … …  
    235227        return 0
    236228
    237     def parse_expectations(self, test_platform_name, is_debug_mode):
     229    def lint_expectations(self, config):
     230        port = self._port
     231        test_expectations.TestExpectations(
     232            port,
     233            None,
     234            port.test_expectations(),
     235            config,
     236            self._options.lint_test_files,
     237            port.test_expectations_overrides())
     238
     239    def parse_expectations(self):
    238240        """Parse the expectations from the test_list files and return a data
    239241        structure holding them. Throws an error if the test_list files have
    240242        invalid syntax."""
    241         if self._options.lint_test_files:
    242             test_files = None
    243         else:
    244             test_files = self._test_files
    245 
    246         expectations_str = self._port.test_expectations()
    247         overrides_str = self._port.test_expectations_overrides()
     243        port = self._port
    248244        self._expectations = test_expectations.TestExpectations(
    249             self._port, test_files, expectations_str, test_platform_name,
    250             is_debug_mode, self._options.lint_test_files,
    251             overrides=overrides_str)
    252         return self._expectations
     245            port,
     246            self._test_files,
     247            port.test_expectations(),
     248            port.test_configuration(),
     249            self._options.lint_test_files,
     250            port.test_expectations_overrides())
    253251
    254252    # FIXME: This method is way too long and needs to be broken into pieces.
    … …  
    358356            self._test_files = set(self._test_files_list)
    359357
    360             self._expectations = self.parse_expectations(
    361                 self._port.test_platform_name(),
    362                 self._options.configuration == 'Debug')
     358            self.parse_expectations()
    363359
    364360            self._test_files = set(files)

  • trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py

    r74396 r75576  
    122122        if self._options.configuration is None:
    123123            self._options.configuration = self.default_configuration()
     124        self._test_configuration = None
    124125
    125126    def default_child_processes(self):
    … …  
    574575        if self._http_lock:
    575576            self._http_lock.cleanup_http_lock()
     577
     578    #
     579    # TEST EXPECTATION-RELATED METHODS
     580    #
     581
     582    def test_configuration(self):
     583        """Returns the current TestConfiguration for the port."""
     584        if not self._test_configuration:
     585            self._test_configuration = TestConfiguration(self)
     586        return self._test_configuration
     587
     588    def all_test_configurations(self):
     589        return self.test_configuration().all_test_configurations()
    576590
    577591    def test_expectations(self):
    … …  
    861875    def stop(self):
    862876        raise NotImplementedError('Driver.stop')
     877
     878
     879class TestConfiguration(object):
     880    def __init__(self, port=None, os=None, version=None, architecture=None,
     881                 build_type=None, graphics_type=None):
     882
     883        # FIXME: We can get the O/S and version from test_platform_name()
     884        # and version() for now, but those should go away and be cleaned up
     885        # with more generic methods like operation_system() and os_version()
     886        # or something. Note that we need to strip the leading '-' off the
     887        # version string if it is present.
     888        if port:
     889            port_version = port.version()
     890        self.os = os or port.test_platform_name().replace(port_version, '')
     891        self.version = version or port_version[1:]
     892        self.architecture = architecture or 'x86'
     893        self.build_type = build_type or port._options.configuration.lower()
     894        self.graphics_type = graphics_type or 'cpu'
     895
     896    def items(self):
     897        return self.__dict__.items()
     898
     899    def keys(self):
     900        return self.__dict__.keys()
     901
     902    def __str__(self):
     903        return ("<%(version)s, %(build_type)s, %(graphics_type)s>" %
     904                self.__dict__)
     905
     906    def __repr__(self):
     907        return "TestConfig(os='%(os)s', version='%(version)s', architecture='%(architecture)s', build_type='%(build_type)s', graphics_type='%(graphics_type)s')" % self.__dict__
     908
     909    def values(self):
     910        """Returns the configuration values of this instance as a tuple."""
     911        return self.__dict__.values()
     912
     913    def all_test_configurations(self):
     914        """Returns a sequence of the TestConfigurations the port supports."""
     915        # By default, we assume we want to test every graphics type in
     916        # every configuration on every system.
     917        test_configurations = []
     918        for system in self.all_systems():
     919            for build_type in self.all_build_types():
     920                for graphics_type in self.all_graphics_types():
     921                    test_configurations.append(TestConfiguration(
     922                        os=system[0],
     923                        version=system[1],
     924                        architecture=system[2],
     925                        build_type=build_type,
     926                        graphics_type=graphics_type))
     927        return test_configurations
     928
     929    def all_systems(self):
     930        return (('mac', 'leopard', 'x86'),
     931                ('mac', 'snowleopard', 'x86'),
     932                ('win', 'xp', 'x86'),
     933                ('win', 'vista', 'x86'),
     934                ('win', 'win7', 'x86'),
     935                ('linux', 'hardy', 'x86'))
     936
     937    def all_build_types(self):
     938        return ('debug', 'release')
     939
     940    def all_graphics_types(self):
     941        return ('cpu', 'gpu')

  • trunk/Tools/Scripts/webkitpy/layout_tests/port/chromium.py

    r75036 r75576  
    275275
    276276        expectations = test_expectations.TestExpectations(
    277             self, all_test_files, expectations_str, test_platform_name,
    278             is_debug_mode, is_lint_mode=False, overrides=overrides_str)
     277            self, all_test_files, expectations_str, self.test_configuration(),
     278            is_lint_mode=False, overrides=overrides_str)
    279279        tests_dir = self.layout_tests_dir()
    280280        return [self.relative_test_filename(test)

  • trunk/Tools/Scripts/webkitpy/layout_tests/port/port_testcase.py

    r72708 r75576  
    9696        port.start_websocket_server()
    9797        port.stop_websocket_server()
     98
     99    def test_test_configuration(self):
     100        port = self.make_port()
     101        if not port:
     102            return
     103        self.assertTrue(port.test_configuration())
     104
     105    def test_all_test_configurations(self):
     106        port = self.make_port()
     107        if not port:
     108            return
     109        self.assertTrue(len(port.all_test_configurations()) > 0)

  • trunk/Tools/Scripts/webkitpy/layout_tests/port/test.py

    r74362 r75576  
    152152        return True
    153153
     154    def default_configuration(self):
     155        return 'Release'
     156
    154157    def diff_image(self, expected_contents, actual_contents,
    155158                   diff_filename=None):
    … …  
    307310
    308311    def version(self):
    309         return ''
     312        return '-leopard'
    310313
    311314

  • trunk/Tools/Scripts/webkitpy/layout_tests/rebaseline_chromium_webkit_tests.py

    r74511 r75576  
    184184                                               None,
    185185                                               expectations_str,
    186                                                self._platform,
    187                                                False,
     186                                               self._target_port.test_configuration(),
    188187                                               False)
    189188        self._scm = scm.default_scm()

  • trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py

    r75036 r75576  
    102102            raise
    103103
    104         printer.print_update("Parsing expectations ...")
    105104        if options.lint_test_files:
    106105            return runner.lint()
    107         runner.parse_expectations(port.test_platform_name(),
    108                                   options.configuration == 'Debug')
     106
     107        printer.print_update("Parsing expectations ...")
     108        runner.parse_expectations()
    109109
    110110        printer.print_update("Checking build ...")

  • trunk/Tools/Scripts/webkitpy/style/checkers/test_expectations.py

    r73748 r75576  
    7676                      "will fail the check." % self._file_path)
    7777            self._port_obj = port.get('test')
    78         self._port_to_check = self._port_obj.test_platform_name()
    7978        # Suppress error messages of test_expectations module since they will be
    8079        # reported later.
    … …  
    9291            expectations = test_expectations.TestExpectationsFile(
    9392                port=self._port_obj, expectations=expectations_str, full_test_list=tests,
    94                 test_platform_name=self._port_to_check, is_debug_mode=False,
     93                test_config=self._port_obj.test_configuration(),
    9594                is_lint_mode=True, overrides=overrides)
    9695        except test_expectations.ParseError, error:

  • trunk/Tools/Scripts/webkitpy/style/checkers/test_expectations_unittest.py

    r74082 r75576  
    8585    def test_valid_expectations(self):
    8686        self.assert_lines_lint(
    87             ["passes/text.html = PASS"],
    88             "")
    89         self.assert_lines_lint(
    90             ["passes/text.html = FAIL PASS"],
    91             "")
    92         self.assert_lines_lint(
    93             ["passes/text.html = CRASH TIMEOUT FAIL PASS"],
    94             "")
    95         self.assert_lines_lint(
    9687            ["BUGCR1234 MAC : passes/text.html = PASS FAIL"],
    9788            "")
    … …  
    121112        self.assert_lines_lint(
    122113            ["BUG1234 : passes/text.html = FAIL"],
    123             'Bug must be either BUGCR, BUGWK, or BUGV8_ for test: bug1234 passes/text.html  [test/expectations] [5]')
     114            "BUG\\d+ is not allowed, must be one of BUGCR\\d+, BUGWK\\d+, BUGV8_\\d+, or a non-numeric bug identifier. passes/text.html  [test/expectations] [5]")
    124115
    125116    def test_valid_modifiers(self):
    126117        self.assert_lines_lint(
    127118            ["INVALID-MODIFIER : passes/text.html = PASS"],
    128             "Invalid modifier for test: invalid-modifier "
     119            "Unrecognized option 'invalid-modifier' "
    129120            "passes/text.html  [test/expectations] [5]")
    130121        self.assert_lines_lint(
    … …  
    136127        self.assert_lines_lint(
    137128            ["missing expectations"],
    138             "Missing expectations. ['missing expectations']  [test/expectations] [5]")
     129            "Missing a ':' missing expectations  [test/expectations] [5]")
    139130        self.assert_lines_lint(
    140131            ["SLOW : passes/text.html = TIMEOUT"],
    … …  
    143134            "passes/text.html  [test/expectations] [5]")
    144135        self.assert_lines_lint(
    145             ["does/not/exist.html = FAIL"],
     136            ["BUGWK1 : does/not/exist.html = FAIL"],
    146137            "Path does not exist. does/not/exist.html  [test/expectations] [2]")
    147138
    148139    def test_parse_expectations(self):
    149140        self.assert_lines_lint(
    150             ["passes/text.html = PASS"],
     141            ["BUGWK1 : passes/text.html = PASS"],
    151142            "")
    152143        self.assert_lines_lint(
    153             ["passes/text.html = UNSUPPORTED"],
     144            ["BUGWK1 : passes/text.html = UNSUPPORTED"],
    154145            "Unsupported expectation: unsupported "
    155146            "passes/text.html  [test/expectations] [5]")
    156147        self.assert_lines_lint(
    157             ["passes/text.html = PASS UNSUPPORTED"],
     148            ["BUGWK1 : passes/text.html = PASS UNSUPPORTED"],
    158149            "Unsupported expectation: unsupported "
    159150            "passes/text.html  [test/expectations] [5]")
    … …  
    161152    def test_already_seen_test(self):
    162153        self.assert_lines_lint(
    163             ["passes/text.html = PASS",
    164              "passes/text.html = TIMEOUT"],
    165             "Duplicate expectation. %s  [test/expectations] [5]" % self._test_file)
     154            ["BUGWK1 : passes/text.html = PASS",
     155             "BUGWK1 : passes/text.html = TIMEOUT"],
     156            "Duplicate or ambiguous expectation. %s  [test/expectations] [5]" % self._test_file)
    166157
    167158    def test_tab(self):
    168159        self.assert_lines_lint(
    169             ["\tpasses/text.html = PASS"],
     160            ["\tBUGWK1 : passes/text.html = PASS"],
    170161            "Line contains tab character.  [whitespace/tab] [5]")
    171162
Note: See TracChangeset for help on using the changeset viewer.