Changeset 89834 in webkit


Ignore:
Timestamp:
Jun 27, 2011 10:43:55 AM (13 years ago)
Author:
eric@webkit.org
Message:

2011-06-27 Eric Seidel <eric@webkit.org>

Reviewed by Adam Barth.

new-run-webkit-tests needs a --webkit-test-runner option
https://bugs.webkit.org/show_bug.cgi?id=63439

NRWT doesn't actually know how to run with the WebKitTestRunner yet
but it does have a --webkit-test-runner option and will build WebKitTestRunner correctly.

There is a bunch of other little cleanup in this patch which I added as I
took a tour through all of our hard-coded DumpRenderTree strings.

  • Scripts/webkitpy/common/net/layouttestresults.py: Added FIXME.
  • Scripts/webkitpy/common/system/outputcapture.py: Fixed spacing to pass PEP8.
  • Scripts/webkitpy/layout_tests/layout_package/manager.py: Made the FIXME slightly stronger (bad bug).
  • Scripts/webkitpy/layout_tests/layout_package/single_test_runner.py: test_shell is dead.
  • Scripts/webkitpy/layout_tests/layout_package/test_failures.py: Unwrapped silly wrapping.
  • Scripts/webkitpy/layout_tests/port/base.py: Removed two dead methods.
  • Scripts/webkitpy/layout_tests/port/config.py:
    • build_dumprendertree had no business in this class, removed it.
    • Exposed _FLAGS_FROM_CONFIGURATIONS through flag_for_configuration()
    • Exposed _script_path as script_path() (this probably belongs elsewhere).
  • Scripts/webkitpy/layout_tests/port/config_unittest.py:
    • Moved these tests to webkit_unittests.
  • Scripts/webkitpy/layout_tests/port/webkit.py:
    • Now respects the --webkit-test-runner option.
    • setup_test_run is empty in base.py too, no need to override it.
  • Scripts/webkitpy/layout_tests/port/webkit_unittest.py:
    • Test the new hotness.
    • Bad, bad, bad! The old code was using a real Executive during unit-testing! I think this code is still hitting disk during the unit tests. :(
  • Scripts/webkitpy/layout_tests/run_webkit_tests.py:
    • Add the option.
  • Scripts/webkitpy/tool/bot/queueengine_unittest.py:
    • Remove optional args option.
  • Scripts/webkitpy/tool/commands/commandtest.py:
    • Remove optional args option.
  • Scripts/webkitpy/tool/commands/queues_unittest.py:
    • Remove optional args option.
Location:
trunk/Tools
Files:
15 edited

Legend:

Unmodified
Added
Removed
  • trunk/Tools/ChangeLog

    r89825 r89834  
     12011-06-27  Eric Seidel  <eric@webkit.org>
     2
     3        Reviewed by Adam Barth.
     4
     5        new-run-webkit-tests needs a --webkit-test-runner option
     6        https://bugs.webkit.org/show_bug.cgi?id=63439
     7
     8        NRWT doesn't actually know how to run with the WebKitTestRunner yet
     9        but it does have a --webkit-test-runner option and will build WebKitTestRunner correctly.
     10
     11        There is a bunch of other little cleanup in this patch which I added as I
     12        took a tour through all of our hard-coded DumpRenderTree strings.
     13
     14        * Scripts/webkitpy/common/net/layouttestresults.py: Added FIXME.
     15        * Scripts/webkitpy/common/system/outputcapture.py: Fixed spacing to pass PEP8.
     16        * Scripts/webkitpy/layout_tests/layout_package/manager.py: Made the FIXME slightly stronger (bad bug).
     17        * Scripts/webkitpy/layout_tests/layout_package/single_test_runner.py: test_shell is dead.
     18        * Scripts/webkitpy/layout_tests/layout_package/test_failures.py: Unwrapped silly wrapping.
     19        * Scripts/webkitpy/layout_tests/port/base.py: Removed two dead methods.
     20        * Scripts/webkitpy/layout_tests/port/config.py:
     21         - build_dumprendertree had no business in this class, removed it.
     22         - Exposed _FLAGS_FROM_CONFIGURATIONS through flag_for_configuration()
     23         - Exposed _script_path as script_path() (this probably belongs elsewhere).
     24        * Scripts/webkitpy/layout_tests/port/config_unittest.py:
     25         - Moved these tests to webkit_unittests.
     26        * Scripts/webkitpy/layout_tests/port/webkit.py:
     27         - Now respects the --webkit-test-runner option.
     28         - setup_test_run is empty in base.py too, no need to override it.
     29        * Scripts/webkitpy/layout_tests/port/webkit_unittest.py:
     30         - Test the new hotness.
     31         - Bad, bad, bad!  The old code was using a real Executive during unit-testing!
     32           I think this code is still hitting disk during the unit tests. :(
     33        * Scripts/webkitpy/layout_tests/run_webkit_tests.py:
     34         - Add the option.
     35        * Scripts/webkitpy/tool/bot/queueengine_unittest.py:
     36         - Remove optional args option.
     37        * Scripts/webkitpy/tool/commands/commandtest.py:
     38         - Remove optional args option.
     39        * Scripts/webkitpy/tool/commands/queues_unittest.py:
     40         - Remove optional args option.
     41
    1422011-06-27  Chang Shu  <cshu@webkit.org>
    243
  • trunk/Tools/Scripts/webkitpy/common/net/layouttestresults.py

    r84633 r89834  
    4949    fail_key = u'Tests where results did not match expected results:'
    5050    timeout_key = u'Tests that timed out:'
     51    # FIXME: This may need to be made aware of WebKitTestRunner results for WebKit2.
    5152    crash_key = u'Tests that caused the DumpRenderTree tool to crash:'
    5253    missing_key = u'Tests that had no expected results (probably new):'
  • trunk/Tools/Scripts/webkitpy/common/system/outputcapture.py

    r68673 r89834  
    3232import unittest
    3333from StringIO import StringIO
     34
    3435
    3536class OutputCapture(object):
  • trunk/Tools/Scripts/webkitpy/layout_tests/layout_package/manager.py

    r89642 r89834  
    653653            # to thrash on something and time out its first few tests. Until
    654654            # we can figure out what's going on, sleep a bit in between
    655             # workers.
     655            # workers. This needs a bug filed.
    656656            time.sleep(0.1)
    657657
  • trunk/Tools/Scripts/webkitpy/layout_tests/layout_package/single_test_runner.py

    r87847 r89834  
    144144
    145145    def _save_baselines(self, driver_output):
    146         # Although all test_shell/DumpRenderTree output should be utf-8,
     146        # Although all DumpRenderTree output should be utf-8,
    147147        # we do not ever decode it inside run-webkit-tests.  For some tests
    148148        # DumpRenderTree may not output utf-8 text (e.g. webarchives).
  • trunk/Tools/Scripts/webkitpy/layout_tests/layout_package/test_failures.py

    r85609 r89834  
    105105
    106106    def should_kill_dump_render_tree(self):
    107         """Returns True if we should kill DumpRenderTree before the next
    108         test."""
     107        """Returns True if we should kill DumpRenderTree before the next test."""
    109108        return False
    110109
    111110
    112111class FailureTimeout(TestFailure):
    113     """Test timed out.  We also want to restart DumpRenderTree if this
    114     happens."""
     112    """Test timed out.  We also want to restart DumpRenderTree if this happens."""
    115113    def __init__(self, is_reftest=False):
    116114        self.is_reftest = is_reftest
     
    131129    @staticmethod
    132130    def message():
     131        # FIXME: This is wrong for WebKit2 (which uses WebKitTestRunner).
    133132        return "DumpRenderTree crashed"
    134133
  • trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py

    r89808 r89834  
    590590        return self._config.path_from_webkit_base(*comps)
    591591
    592     def script_path(self, script_name):
    593         return self._config.script_path(script_name)
    594 
    595     def script_shell_command(self, script_name):
    596         return self._config.script_shell_command(script_name)
    597 
    598592    def path_to_test_expectations_file(self):
    599593        """Update the test expectations to the passed-in string.
  • trunk/Tools/Scripts/webkitpy/layout_tests/port/config.py

    r76184 r89834  
    7272        """Returns the path to the build directory for the configuration."""
    7373        if configuration:
    74             flags = ["--configuration",
    75                      self._FLAGS_FROM_CONFIGURATIONS[configuration]]
     74            flags = ["--configuration", self.flag_for_configuration(configuration)]
    7675        else:
    7776            configuration = ""
     
    7978
    8079        if not self._build_directories.get(configuration):
    81             args = ["perl", self._script_path("webkit-build-directory")] + flags
     80            args = ["perl", self.script_path("webkit-build-directory")] + flags
    8281            self._build_directories[configuration] = (
    8382                self._executive.run_command(args).rstrip())
     
    8584        return self._build_directories[configuration]
    8685
    87     def build_dumprendertree(self, configuration):
    88         """Builds DRT in the given configuration.
    89 
    90         Returns True if the  build was successful and up-to-date."""
    91         flag = self._FLAGS_FROM_CONFIGURATIONS[configuration]
    92         exit_code = self._executive.run_command([
    93             self._script_path("build-dumprendertree"), flag],
    94             return_exit_code=True)
    95         if exit_code != 0:
    96             _log.error("Failed to build DumpRenderTree")
    97             return False
    98         return True
     86    def flag_for_configuration(self, configuration):
     87        return self._FLAGS_FROM_CONFIGURATIONS[configuration]
    9988
    10089    def default_configuration(self):
     
    10897            self._default_configuration = 'Release'
    10998        if self._default_configuration not in self._FLAGS_FROM_CONFIGURATIONS:
    110             _log.warn("Configuration \"%s\" is not a recognized value.\n" %
    111                       self._default_configuration)
    112             _log.warn("Scripts may fail.  "
    113                       "See 'set-webkit-configuration --help'.")
     99            _log.warn("Configuration \"%s\" is not a recognized value.\n" % self._default_configuration)
     100            _log.warn("Scripts may fail.  See 'set-webkit-configuration --help'.")
    114101        return self._default_configuration
    115102
     
    117104        return self._filesystem.join(self.webkit_base_dir(), *comps)
    118105
     106    # FIXME: We should only have one implementation of this logic,
     107    # if scm.find_checkout_root() is broken for Chromium, we should fix (or at least wrap) it!
    119108    def webkit_base_dir(self):
    120109        """Returns the absolute path to the top of the WebKit tree.
     
    134123        return self._webkit_base_dir
    135124
    136     def _script_path(self, script_name):
    137         return self._filesystem.join(self.webkit_base_dir(), "Tools",
    138                                      "Scripts", script_name)
     125    def script_path(self, script_name):
     126        return self._filesystem.join(self.webkit_base_dir(), "Tools", "Scripts", script_name)
    139127
    140128    def _determine_configuration(self):
  • trunk/Tools/Scripts/webkitpy/layout_tests/port/config_unittest.py

    r79042 r89834  
    9292        self.assertRaises(KeyError, c.build_directory, 'Unknown')
    9393
    94     def test_build_dumprendertree__success(self):
    95         c = self.make_config(exit_code=0)
    96         self.assertTrue(c.build_dumprendertree("Debug"))
    97         self.assertTrue(c.build_dumprendertree("Release"))
    98         self.assertRaises(KeyError, c.build_dumprendertree, "Unknown")
    99 
    100     def test_build_dumprendertree__failure(self):
    101         c = self.make_config(exit_code=-1)
    102 
    103         # FIXME: Build failures should log errors. However, the message we
    104         # get depends on how we're being called; as a standalone test,
    105         # we'll get the "no handlers found" message. As part of
    106         # test-webkitpy, we get the actual message. Really, we need
    107         # outputcapture to install its own handler.
    108         oc = outputcapture.OutputCapture()
    109         oc.capture_output()
    110         self.assertFalse(c.build_dumprendertree('Debug'))
    111         oc.restore_output()
    112 
    113         oc.capture_output()
    114         self.assertFalse(c.build_dumprendertree('Release'))
    115         oc.restore_output()
    116 
    11794    def test_default_configuration__release(self):
    11895        self.assert_configuration('Release', 'Release')
  • trunk/Tools/Scripts/webkitpy/layout_tests/port/webkit.py

    r89808 r89834  
    4242import webbrowser
    4343
     44
    4445from webkitpy.common.net.buildbot import BuildBot
    4546from webkitpy.common.system import ospath
     47from webkitpy.common.system.executive import ScriptError
    4648from webkitpy.layout_tests.port import base, builders, server_process
    4749
     
    5860        # FIXME: disable pixel tests until they are run by default on the build machines.
    5961        self.set_option_default("pixel_tests", False)
     62
     63    def driver_name(self):
     64        if self.get_option('webkit_test_runner'):
     65            return "WebKitTestRunner"
     66        return "DumpRenderTree"
    6067
    6168    def baseline_search_path(self):
     
    7683        return result_set
    7784
     85    def _driver_build_script_name(self):
     86        if self.get_option('webkit_test_runner'):
     87            return "build-webkittestrunner"
     88        return "build-dumprendertree"
     89
    7890    def _build_driver(self):
    7991        configuration = self.get_option('configuration')
    80         return self._config.build_dumprendertree(configuration)
     92        try:
     93            # FIXME: We should probably have a run_script helper which automatically adds the configuration flags.
     94            self._executive.run_command([
     95                self._config.script_path(self._driver_build_script_name()),
     96                self._config.flag_for_configuration(configuration)])
     97        except ScriptError:
     98            _log.error("Failed to build %s" % self.driver_name())
     99            return False
     100        return True
    81101
    82102    def _check_driver(self):
    83103        driver_path = self._path_to_driver()
    84104        if not self._filesystem.exists(driver_path):
    85             _log.error("DumpRenderTree was not found at %s" % driver_path)
     105            _log.error("%s was not found at %s" % (self.driver_name(), driver_path))
    86106            return False
    87107        return True
     
    178198        return self._build_path('layout-test-results')
    179199
    180     def setup_test_run(self):
    181         # This port doesn't require any specific configuration.
    182         pass
    183 
    184200    def create_driver(self, worker_number):
    185201        return WebKitDriver(self, worker_number)
     
    428444            error_lines.pop()  # Remove the expected "#EOF"
    429445        error = "\n".join(error_lines)
    430         # FIXME: This seems like the wrong section of code to be doing
    431         # this reset in.
     446
     447        # FIXME: This seems like the wrong section of code to be doing this reset in.
    432448        self._server_process.error = ""
    433449        return base.DriverOutput(text, image, actual_image_hash, audio,
  • trunk/Tools/Scripts/webkitpy/layout_tests/port/webkit_unittest.py

    r88994 r89834  
    2929
    3030from webkitpy.common.system import filesystem_mock
     31from webkitpy.common.system.outputcapture import OutputCapture
    3132
    3233from webkitpy.layout_tests.port.webkit import WebKitPort
    3334from webkitpy.layout_tests.port import port_testcase
    3435
     36from webkitpy.tool.mocktool import MockExecutive
     37from webkitpy.tool.mocktool import MockOptions
     38
    3539
    3640class TestWebKitPort(WebKitPort):
    3741    def __init__(self, symbol_list=None, feature_list=None,
    38                  expectations_file=None, skips_file=None, **kwargs):
     42                 expectations_file=None, skips_file=None,
     43                 executive=None, **kwargs):
    3944        self.symbol_list = symbol_list
    4045        self.feature_list = feature_list
    4146        self.expectations_file = expectations_file
    4247        self.skips_file = skips_file
    43         WebKitPort.__init__(self, **kwargs)
     48        executive = executive or MockExecutive(should_log=False)
     49        WebKitPort.__init__(self, executive=executive, **kwargs)
    4450
    4551    def _runtime_feature_list(self):
     
    9298
    9399    def test_skipped_layout_tests(self):
    94         self.assertEqual(TestWebKitPort(None, None).skipped_layout_tests(),
    95                          set(["media"]))
     100        self.assertEqual(TestWebKitPort(None, None).skipped_layout_tests(), set(["media"]))
    96101
    97102    def test_test_expectations(self):
     
    110115BUG_SKIPPED SKIP : media = FAIL""")
    111116
     117    def test_build_driver(self):
     118        output = OutputCapture()
     119        port = TestWebKitPort()
     120        # Delay setting _executive to avoid logging during construction
     121        port._executive = MockExecutive(should_log=True)
     122        port._options = MockOptions(configuration="Release")  # This should not be necessary, but I think TestWebKitPort is actually reading from disk (and thus detects the current configuration).
     123        expected_stderr = "MOCK run_command: ['/Projects/WebKit/Tools/Scripts/build-dumprendertree', '--release']\n"
     124        self.assertTrue(output.assert_outputs(self, port._build_driver, expected_stderr=expected_stderr))
    112125
    113 if __name__ == '__main__':
    114     unittest.main()
     126        # Make sure when passed --webkit-test-runner web build the right tool.
     127        port._options = MockOptions(webkit_test_runner=True, configuration="Release")
     128        expected_stderr = "MOCK run_command: ['/Projects/WebKit/Tools/Scripts/build-webkittestrunner', '--release']\n"
     129        self.assertTrue(output.assert_outputs(self, port._build_driver, expected_stderr=expected_stderr))
     130
     131        # Make sure that failure to build returns False.
     132        port._executive = MockExecutive(should_log=True, should_throw=True)
     133        expected_stderr = "MOCK run_command: ['/Projects/WebKit/Tools/Scripts/build-webkittestrunner', '--release']\n"
     134        self.assertFalse(output.assert_outputs(self, port._build_driver, expected_stderr=expected_stderr))
  • trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py

    r89620 r89834  
    269269            help="Use the complex text code path for all text (Mac OS X and Windows only)"),
    270270        optparse.make_option("--threaded", action="store_true", default=False,
    271             help="Run a concurrent JavaScript thead with each test")
     271            help="Run a concurrent JavaScript thead with each test"),
     272        optparse.make_option("--webkit-test-runner", "-2", action="store_true",
     273            help="Use WebKitTestRunner rather than DumpRenderTree."),
    272274    ]
    273275
  • trunk/Tools/Scripts/webkitpy/tool/bot/queueengine_unittest.py

    r69805 r89834  
    161161            termination_message = "Delegate terminated queue."
    162162        expected_stderr = "\n%s\n" % termination_message
    163         OutputCapture().assert_outputs(self, engine.run, [], expected_stderr=expected_stderr)
     163        OutputCapture().assert_outputs(self, engine.run, expected_stderr=expected_stderr)
    164164
    165165    def _test_terminating_queue(self, exception, termination_message):
  • trunk/Tools/Scripts/webkitpy/tool/commands/commandtest.py

    r76926 r89834  
    3333
    3434class CommandsTest(unittest.TestCase):
    35     def assert_execute_outputs(self, command, args, expected_stdout="", expected_stderr="", expected_exception=None, options=MockOptions(), tool=MockTool()):
     35    def assert_execute_outputs(self, command, args=[], expected_stdout="", expected_stderr="", expected_exception=None, options=MockOptions(), tool=MockTool()):
    3636        options.blocks = None
    3737        options.cc = 'MOCK cc'
  • trunk/Tools/Scripts/webkitpy/tool/commands/queues_unittest.py

    r85802 r89834  
    167167        expected_stdout = "MOCK: fetch_attachment: 2 is not a known attachment id\n"  # A mock-only message to prevent us from making mistakes.
    168168        expected_stderr = "MOCK: release_work_item: None 2\n"
    169         patch_id = OutputCapture().assert_outputs(self, queue._next_patch, [], expected_stdout=expected_stdout, expected_stderr=expected_stderr)
     169        patch_id = OutputCapture().assert_outputs(self, queue._next_patch, expected_stdout=expected_stdout, expected_stderr=expected_stderr)
    170170        self.assertEquals(patch_id, None)  # 2 is an invalid patch id
    171171        self.assertEquals(queue._next_patch().id(), 197)
Note: See TracChangeset for help on using the changeset viewer.