Changeset 142941 in webkit


Ignore:
Timestamp:
Feb 14, 2013 4:56:23 PM (11 years ago)
Author:
commit-queue@webkit.org
Message:

new-run-webkit-tests needs a shared TestExpectations between all WebKit ports
https://bugs.webkit.org/show_bug.cgi?id=37565

Introduce generic TestExpectations file that applies as a fallback for all ports, the location of which
is LayoutTests/TestExpectations.

Patch by Glenn Adams <glenn@skynav.com> on 2013-02-14
Reviewed by Dirk Pranke.

Tools:

  • Scripts/webkitpy/layout_tests/lint_test_expectations_unittest.py:

(FakePort.path_to_generic_test_expectations_file):

  • Scripts/webkitpy/layout_tests/models/test_expectations.py:

(TestExpectations.init):

  • Scripts/webkitpy/layout_tests/port/base.py:

(Port.path_to_generic_test_expectations_file):
(Port):
(Port._port_specific_expectations_files):
(Port.expectations_files):

  • Scripts/webkitpy/layout_tests/port/chromium.py:

(ChromiumPort._port_specific_expectations_files):

  • Scripts/webkitpy/layout_tests/port/chromium_android.py:

(ChromiumAndroidPort._port_specific_expectations_files):

  • Scripts/webkitpy/layout_tests/port/chromium_port_testcase.py:

(ChromiumPortTestCase.test_expectations_files):

  • Scripts/webkitpy/layout_tests/port/efl.py:

(EflPort._port_specific_expectations_files):

  • Scripts/webkitpy/layout_tests/port/gtk.py:

(GtkPort._port_specific_expectations_files):

  • Scripts/webkitpy/layout_tests/port/gtk_unittest.py:

(GtkPortTest.test_expectations_files):

  • Scripts/webkitpy/layout_tests/port/mac.py:

(MacPort._port_specific_expectations_files):

  • Scripts/webkitpy/layout_tests/port/port_testcase.py:

(PortTestCase.test_expectations_ordering):
(test_expectations_files):

  • Scripts/webkitpy/layout_tests/port/qt.py:

(QtPort._port_specific_expectations_files):

  • Scripts/webkitpy/layout_tests/port/qt_unittest.py:

(QtPortTest.test_expectations_files):

  • Scripts/webkitpy/layout_tests/port/win_unittest.py:

(WinPortTest.test_expectations_files):

  • Scripts/webkitpy/tool/commands/queries_unittest.py:

(PrintExpectationsTest.test_paths):

  • Scripts/webkitpy/tool/commands/rebaseline.py:

(RebaselineTest._update_expectations_file):

LayoutTests:

Location:
trunk
Files:
1 added
18 edited

Legend:

Unmodified
Added
Removed
  • trunk/LayoutTests/ChangeLog

    r142940 r142941  
     12013-02-14  Glenn Adams  <glenn@skynav.com>
     2
     3        new-run-webkit-tests needs a shared TestExpectations between all WebKit ports
     4        https://bugs.webkit.org/show_bug.cgi?id=37565
     5
     6        Introduce generic TestExpectations file that applies as a fallback for all ports, the location of which
     7        is LayoutTests/TestExpectations.
     8
     9        Reviewed by Dirk Pranke.
     10
     11        * TestExpectations: Added.
     12
    1132013-02-14  Hajime Morrita  <morrita@google.com>
    214
  • trunk/Tools/ChangeLog

    r142929 r142941  
     12013-02-14  Glenn Adams  <glenn@skynav.com>
     2
     3        new-run-webkit-tests needs a shared TestExpectations between all WebKit ports
     4        https://bugs.webkit.org/show_bug.cgi?id=37565
     5
     6        Introduce generic TestExpectations file that applies as a fallback for all ports, the location of which
     7        is LayoutTests/TestExpectations.
     8
     9        Reviewed by Dirk Pranke.
     10
     11        * Scripts/webkitpy/layout_tests/lint_test_expectations_unittest.py:
     12        (FakePort.path_to_generic_test_expectations_file):
     13        * Scripts/webkitpy/layout_tests/models/test_expectations.py:
     14        (TestExpectations.__init__):
     15        * Scripts/webkitpy/layout_tests/port/base.py:
     16        (Port.path_to_generic_test_expectations_file):
     17        (Port):
     18        (Port._port_specific_expectations_files):
     19        (Port.expectations_files):
     20        * Scripts/webkitpy/layout_tests/port/chromium.py:
     21        (ChromiumPort._port_specific_expectations_files):
     22        * Scripts/webkitpy/layout_tests/port/chromium_android.py:
     23        (ChromiumAndroidPort._port_specific_expectations_files):
     24        * Scripts/webkitpy/layout_tests/port/chromium_port_testcase.py:
     25        (ChromiumPortTestCase.test_expectations_files):
     26        * Scripts/webkitpy/layout_tests/port/efl.py:
     27        (EflPort._port_specific_expectations_files):
     28        * Scripts/webkitpy/layout_tests/port/gtk.py:
     29        (GtkPort._port_specific_expectations_files):
     30        * Scripts/webkitpy/layout_tests/port/gtk_unittest.py:
     31        (GtkPortTest.test_expectations_files):
     32        * Scripts/webkitpy/layout_tests/port/mac.py:
     33        (MacPort._port_specific_expectations_files):
     34        * Scripts/webkitpy/layout_tests/port/port_testcase.py:
     35        (PortTestCase.test_expectations_ordering):
     36        (test_expectations_files):
     37        * Scripts/webkitpy/layout_tests/port/qt.py:
     38        (QtPort._port_specific_expectations_files):
     39        * Scripts/webkitpy/layout_tests/port/qt_unittest.py:
     40        (QtPortTest.test_expectations_files):
     41        * Scripts/webkitpy/layout_tests/port/win_unittest.py:
     42        (WinPortTest.test_expectations_files):
     43        * Scripts/webkitpy/tool/commands/queries_unittest.py:
     44        (PrintExpectationsTest.test_paths):
     45        * Scripts/webkitpy/tool/commands/rebaseline.py:
     46        (RebaselineTest._update_expectations_file):
     47
    1482013-02-14  Jochen Eisinger  <jochen@chromium.org>
    249
  • trunk/Tools/Scripts/webkitpy/layout_tests/lint_test_expectations_unittest.py

    r140510 r142941  
    6060        return val
    6161
     62    def path_to_generic_test_expectations_file(self):
     63        return ''
    6264
    6365class FakeFactory(object):
  • trunk/Tools/Scripts/webkitpy/layout_tests/models/test_expectations.py

    r139786 r142941  
    843843    # the expectations into separate routines so that linting and handling overrides
    844844    # can be controlled separately, and the constructor can be more of a no-op.
    845     def __init__(self, port, tests=None, include_overrides=True, expectations_to_lint=None):
     845    def __init__(self, port, tests=None, include_generic=True, include_overrides=True, expectations_to_lint=None):
    846846        self._full_test_list = tests
    847847        self._test_config = port.test_configuration()
     
    851851        self._port = port
    852852        self._skipped_tests_warnings = []
     853        self._expectations = []
    853854
    854855        expectations_dict = expectations_to_lint or port.expectations_dict()
    855         self._expectations = self._parser.parse(expectations_dict.keys()[0], expectations_dict.values()[0])
    856         self._add_expectations(self._expectations)
    857 
    858         if len(expectations_dict) > 1 and include_overrides:
    859             for name in expectations_dict.keys()[1:]:
    860                 expectations = self._parser.parse(name, expectations_dict[name])
     856
     857        expectations_dict_index = 0
     858        # Populate generic expectations (if enabled by include_generic).
     859        if port.path_to_generic_test_expectations_file() in expectations_dict:
     860            if include_generic:
     861                expectations = self._parser.parse(expectations_dict.keys()[expectations_dict_index], expectations_dict.values()[expectations_dict_index])
    861862                self._add_expectations(expectations)
    862863                self._expectations += expectations
     864            expectations_dict_index += 1
     865
     866        # Populate default port expectations (always enabled).
     867        if len(expectations_dict) > expectations_dict_index:
     868            expectations = self._parser.parse(expectations_dict.keys()[expectations_dict_index], expectations_dict.values()[expectations_dict_index])
     869            self._add_expectations(expectations)
     870            self._expectations += expectations
     871            expectations_dict_index += 1
     872
     873        # Populate override expectations (if enabled by include_overrides).
     874        while len(expectations_dict) > expectations_dict_index and include_overrides:
     875            expectations = self._parser.parse(expectations_dict.keys()[expectations_dict_index], expectations_dict.values()[expectations_dict_index])
     876            self._add_expectations(expectations)
     877            self._expectations += expectations
     878            expectations_dict_index += 1
    863879
    864880        # FIXME: move ignore_tests into port.skipped_layout_tests()
  • trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py

    r141672 r142941  
    807807
    808808    @memoized
     809    def path_to_generic_test_expectations_file(self):
     810        return self._filesystem.join(self.layout_tests_dir(), 'TestExpectations')
     811
     812    @memoized
    809813    def path_to_test_expectations_file(self):
    810814        """Update the test expectations to the passed-in string.
     
    10711075        return expectations
    10721076
    1073     def expectations_files(self):
     1077    def _port_specific_expectations_files(self):
    10741078        # Unlike baseline_search_path, we only want to search [WK2-PORT, PORT-VERSION, PORT] and any directories
    10751079        # included via --additional-platform-directory, not the full casade.
     
    10881092
    10891093        return [self._filesystem.join(self._webkit_baseline_path(d), 'TestExpectations') for d in search_paths]
     1094
     1095    def expectations_files(self):
     1096        return [self.path_to_generic_test_expectations_file()] + self._port_specific_expectations_files()
    10901097
    10911098    def repository_paths(self):
  • trunk/Tools/Scripts/webkitpy/layout_tests/port/chromium.py

    r139560 r142941  
    350350        return True
    351351
    352     def expectations_files(self):
     352    def _port_specific_expectations_files(self):
    353353        paths = [self.path_to_test_expectations_file()]
    354354        skia_expectations_path = self.path_from_chromium_base('skia', 'skia_test_expectations.txt')
  • trunk/Tools/Scripts/webkitpy/layout_tests/port/chromium_android.py

    r139263 r142941  
    235235        return True
    236236
    237     def expectations_files(self):
     237    def _port_specific_expectations_files(self):
    238238        # LayoutTests/platform/chromium-android/TestExpectations should contain only the rules to
    239239        # skip tests for the features not supported or not testable on Android.
    240240        # Other rules should be in LayoutTests/platform/chromium/TestExpectations.
    241241        android_expectations_file = self.path_from_webkit_base('LayoutTests', 'platform', 'chromium-android', 'TestExpectations')
    242         return super(ChromiumAndroidPort, self).expectations_files() + [android_expectations_file]
     242        return super(ChromiumAndroidPort, self)._port_specific_expectations_files() + [android_expectations_file]
    243243
    244244    def requires_http_server(self):
  • trunk/Tools/Scripts/webkitpy/layout_tests/port/chromium_port_testcase.py

    r140510 r142941  
    177177        port.port_name = 'chromium'
    178178
     179        generic_path = port.path_to_generic_test_expectations_file()
    179180        expectations_path = port.path_to_test_expectations_file()
    180181        chromium_overrides_path = port.path_from_chromium_base(
     
    186187
    187188        port._options.builder_name = 'DUMMY_BUILDER_NAME'
    188         self.assertEqual(port.expectations_files(), [expectations_path, skia_overrides_path, chromium_overrides_path])
     189        self.assertEqual(port.expectations_files(), [generic_path, expectations_path, skia_overrides_path, chromium_overrides_path])
    189190
    190191        port._options.builder_name = 'builder (deps)'
    191         self.assertEqual(port.expectations_files(), [expectations_path, skia_overrides_path, chromium_overrides_path])
     192        self.assertEqual(port.expectations_files(), [generic_path, expectations_path, skia_overrides_path, chromium_overrides_path])
    192193
    193194        # A builder which does NOT observe the Chromium test_expectations,
    194195        # but still observes the Skia test_expectations...
    195196        port._options.builder_name = 'builder'
    196         self.assertEqual(port.expectations_files(), [expectations_path, skia_overrides_path])
     197        self.assertEqual(port.expectations_files(), [generic_path, expectations_path, skia_overrides_path])
    197198
    198199    def test_expectations_ordering(self):
  • trunk/Tools/Scripts/webkitpy/layout_tests/port/efl.py

    r142859 r142941  
    119119        return map(self._webkit_baseline_path, self._search_paths())
    120120
    121     def expectations_files(self):
     121    def _port_specific_expectations_files(self):
    122122        # FIXME: We should be able to use the default algorithm here.
    123123        return list(reversed([self._filesystem.join(self._webkit_baseline_path(p), 'TestExpectations') for p in self._search_paths()]))
  • trunk/Tools/Scripts/webkitpy/layout_tests/port/gtk.py

    r139973 r142941  
    112112        return search_paths
    113113
    114     def expectations_files(self):
     114    def _port_specific_expectations_files(self):
    115115        return [self._filesystem.join(self._webkit_baseline_path(p), 'TestExpectations') for p in reversed(self._search_paths())]
    116116
  • trunk/Tools/Scripts/webkitpy/layout_tests/port/gtk_unittest.py

    r140510 r142941  
    5353    def test_expectations_files(self):
    5454        port = self.make_port()
    55         self.assertEqual(port.expectations_files(), ['/mock-checkout/LayoutTests/platform/gtk/TestExpectations', '/mock-checkout/LayoutTests/platform/gtk-wk1/TestExpectations'])
     55        self.assertEqual(port.expectations_files(), ['/mock-checkout/LayoutTests/TestExpectations', '/mock-checkout/LayoutTests/platform/gtk/TestExpectations', '/mock-checkout/LayoutTests/platform/gtk-wk1/TestExpectations'])
    5656
    5757        port = self.make_port(options=MockOptions(webkit_test_runner=True))
    58         self.assertEqual(port.expectations_files(), ['/mock-checkout/LayoutTests/platform/gtk/TestExpectations', '/mock-checkout/LayoutTests/platform/wk2/TestExpectations', '/mock-checkout/LayoutTests/platform/gtk-wk2/TestExpectations'])
     58        self.assertEqual(port.expectations_files(), ['/mock-checkout/LayoutTests/TestExpectations', '/mock-checkout/LayoutTests/platform/gtk/TestExpectations', '/mock-checkout/LayoutTests/platform/wk2/TestExpectations', '/mock-checkout/LayoutTests/platform/gtk-wk2/TestExpectations'])
    5959
    6060    def test_show_results_html_file(self):
  • trunk/Tools/Scripts/webkitpy/layout_tests/port/mac.py

    r139965 r142941  
    8383        return map(self._webkit_baseline_path, fallback_names)
    8484
    85     def expectations_files(self):
    86         return reversed([self._filesystem.join(self._webkit_baseline_path(d), 'TestExpectations') for d in self.baseline_search_path()])
     85    def _port_specific_expectations_files(self):
     86        return list(reversed([self._filesystem.join(self._webkit_baseline_path(p), 'TestExpectations') for p in self.baseline_search_path()]))
    8787
    8888    def setup_environ_for_server(self, server_name=None):
  • trunk/Tools/Scripts/webkitpy/layout_tests/port/port_testcase.py

    r141672 r142941  
    425425            port._filesystem.write_text_file(path, '')
    426426        ordered_dict = port.expectations_dict()
    427         self.assertEqual(port.path_to_test_expectations_file(), ordered_dict.keys()[0])
     427        self.assertEqual(port.path_to_generic_test_expectations_file(), ordered_dict.keys()[0])
     428        self.assertEqual(port.path_to_test_expectations_file(), ordered_dict.keys()[1])
    428429
    429430        options = MockOptions(additional_expectations=['/tmp/foo', '/tmp/bar'])
     
    508509            return [port.host.filesystem.basename(port.host.filesystem.dirname(f)) for f in port.expectations_files()]
    509510
    510         self.assertEqual(platform_dirs(port), ['testwebkitport'])
     511        self.assertEqual(platform_dirs(port), ['LayoutTests', 'testwebkitport'])
    511512
    512513        port = TestWebKitPort(port_name="testwebkitport-version")
    513         self.assertEqual(platform_dirs(port), ['testwebkitport', 'testwebkitport-version'])
     514        self.assertEqual(platform_dirs(port), ['LayoutTests', 'testwebkitport', 'testwebkitport-version'])
    514515
    515516        port = TestWebKitPort(port_name="testwebkitport-version-wk2")
    516         self.assertEqual(platform_dirs(port), ['testwebkitport', 'testwebkitport-version', 'wk2', 'testwebkitport-wk2'])
     517        self.assertEqual(platform_dirs(port), ['LayoutTests', 'testwebkitport', 'testwebkitport-version', 'wk2', 'testwebkitport-wk2'])
    517518
    518519        port = TestWebKitPort(port_name="testwebkitport-version",
    519520                              options=MockOptions(additional_platform_directory=["internal-testwebkitport"]))
    520         self.assertEqual(platform_dirs(port), ['testwebkitport', 'testwebkitport-version', 'internal-testwebkitport'])
     521        self.assertEqual(platform_dirs(port), ['LayoutTests', 'testwebkitport', 'testwebkitport-version', 'internal-testwebkitport'])
    521522
    522523    def test_root_option(self):
  • trunk/Tools/Scripts/webkitpy/layout_tests/port/qt.py

    r139194 r142941  
    148148        return map(self._webkit_baseline_path, self._search_paths())
    149149
    150     def expectations_files(self):
     150    def _port_specific_expectations_files(self):
    151151        paths = self._search_paths()
    152152        if self.get_option('webkit_test_runner'):
  • trunk/Tools/Scripts/webkitpy/layout_tests/port/qt_unittest.py

    r140510 r142941  
    9898            if expectations_case['use_webkit2']:
    9999                expectations_case['search_paths'].append("wk2")
     100            expectations_case['search_paths'].append('')
    100101            expectations_case['search_paths'].reverse()
    101             expectations_case['search_paths'] = map(lambda path: '/mock-checkout/LayoutTests/platform/%s/TestExpectations' % (path), expectations_case['search_paths'])
     102            expectations_case['search_paths'] = map(lambda path: '/mock-checkout/LayoutTests/TestExpectations' if not path else '/mock-checkout/LayoutTests/platform/%s/TestExpectations' % (path), expectations_case['search_paths'])
    102103            self._assert_expectations_files(**expectations_case)
    103104
  • trunk/Tools/Scripts/webkitpy/layout_tests/port/win_unittest.py

    r140510 r142941  
    106106
    107107    def test_expectations_files(self):
    108         self.assertEqual(len(self.make_port().expectations_files()), 2)
    109         self.assertEqual(len(self.make_port(options=MockOptions(webkit_test_runner=True, configuration='Release')).expectations_files()), 4)
     108        self.assertEqual(len(self.make_port().expectations_files()), 3)
     109        self.assertEqual(len(self.make_port(options=MockOptions(webkit_test_runner=True, configuration='Release')).expectations_files()), 5)
  • trunk/Tools/Scripts/webkitpy/tool/commands/queries_unittest.py

    r140510 r142941  
    224224    def test_paths(self):
    225225        self.run_test([],
    226                       ('LayoutTests/platform/test/TestExpectations\n'
     226                      ('LayoutTests/TestExpectations\n'
     227                       'LayoutTests/platform/test/TestExpectations\n'
    227228                       'LayoutTests/platform/test-win-xp/TestExpectations\n'),
    228229                      paths=True)
  • trunk/Tools/Scripts/webkitpy/tool/commands/rebaseline.py

    r140508 r142941  
    153153            lock = self._tool.make_file_lock(path + '.lock')
    154154            lock.acquire_lock()
    155             expectations = TestExpectations(port, include_overrides=False)
     155            expectations = TestExpectations(port, include_generic=False, include_overrides=False)
    156156            for test_configuration in port.all_test_configurations():
    157157                if test_configuration.version == port.test_configuration().version:
Note: See TracChangeset for help on using the changeset viewer.