Changeset 57463 in webkit


Ignore:
Timestamp:
Apr 11, 2010 7:16:43 PM (14 years ago)
Author:
eric@webkit.org
Message:

2010-04-11 Eric Seidel <eric@webkit.org>

Reviewed by Adam Barth.

Break new-run-webkit-tests options into groups for easier re-use and possible relocation
https://bugs.webkit.org/show_bug.cgi?id=37408

new-run-webkit-tests currently has one huge function for
dealing with all options-parsing.
This patch is a first attempt at trying to split that large
function down into smaller (hopefully more readable?) chunks
dealing with the different areas of options.
For example, it would make sense to move configuration
options off into some module which deals with the vagries of
WebKit's configuration system. It would also make sense to move
Chromium options off onto the Chromium port object (where they are used).
It may make sense to move results.json options over to the results.json code.
This change is a first iteration, and we will certainly need more
refinement to this code over time. Hopefully I didn't make things
harder to read here.

  • Scripts/webkitpy/layout_tests/driver_test.py:
  • Scripts/webkitpy/layout_tests/run_webkit_tests.py:
Location:
trunk/WebKitTools
Files:
3 edited

Legend:

Unmodified
Added
Removed
  • trunk/WebKitTools/ChangeLog

    r57462 r57463  
     12010-04-11  Eric Seidel  <eric@webkit.org>
     2
     3        Reviewed by Adam Barth.
     4
     5        Break new-run-webkit-tests options into groups for easier re-use and possible relocation
     6        https://bugs.webkit.org/show_bug.cgi?id=37408
     7
     8        new-run-webkit-tests currently has one huge function for
     9        dealing with all options-parsing.
     10        This patch is a first attempt at trying to split that large
     11        function down into smaller (hopefully more readable?) chunks
     12        dealing with the different areas of options.
     13        For example, it would make sense to move configuration
     14        options off into some module which deals with the vagries of
     15        WebKit's configuration system.  It would also make sense to move
     16        Chromium options off onto the Chromium port object (where they are used).
     17        It may make sense to move results.json options over to the results.json code.
     18        This change is a first iteration, and we will certainly need more
     19        refinement to this code over time.  Hopefully I didn't make things
     20        harder to read here.
     21
     22        * Scripts/webkitpy/layout_tests/driver_test.py:
     23        * Scripts/webkitpy/layout_tests/run_webkit_tests.py:
     24
    1252010-04-11  Adam Barth  <abarth@webkit.org>
    226
  • trunk/WebKitTools/Scripts/webkitpy/layout_tests/driver_test.py

    r57256 r57463  
    6262
    6363if __name__ == '__main__':
    64     optparser = optparse.OptionParser()
    65     optparser.add_option('-p', '--platform', action='store', default='mac',
    66                          help='Platform to test (e.g., "mac", "chromium-mac", etc.')
    67     optparser.add_option('--debug', action='store_const', const='Debug',
    68                          dest="configuration", help='Set the configuration to Debug')
    69     optparser.add_option('--release', action='store_const', const='Release',
    70                          dest="configuration", help='Set the configuration to Release')
    71     optparser.add_option('', '--timeout', action='store', default='2000',
    72                          help='test timeout in milliseconds (2000 by default)')
    73     optparser.add_option('', '--wrapper', action='store')
    74     optparser.add_option('', '--no-pixel-tests', action='store_true',
    75                          default=False,
    76                          help='disable pixel-to-pixel PNG comparisons')
     64    # FIXME: configuration_options belong in a shared location.
     65    configuration_options = [
     66        make_option('--debug', action='store_const', const='Debug', dest="configuration", help='Set the configuration to Debug'),
     67        make_option('--release', action='store_const', const='Release', dest="configuration", help='Set the configuration to Release'),
     68    ]
     69    misc_options = [
     70        make_option('-p', '--platform', action='store', default='mac', help='Platform to test (e.g., "mac", "chromium-mac", etc.'),
     71        make_option('--timeout', action='store', default='2000', help='test timeout in milliseconds (2000 by default)'),
     72        make_option('--wrapper', action='store'),
     73        make_option('--no-pixel-tests', action='store_true', default=False, help='disable pixel-to-pixel PNG comparisons'),
     74    ]
     75    option_list = configuration_options + misc_options
     76    optparser = optparse.OptionParser(option_list=option_list)
    7777    options, args = optparser.parse_args()
    7878    p = port.get(options.platform, options)
  • trunk/WebKitTools/Scripts/webkitpy/layout_tests/run_webkit_tests.py

    r57443 r57463  
    15261526
    15271527    Returns a tuple of options, args from optparse"""
    1528     option_parser = optparse.OptionParser()
    1529     option_parser.add_option("", "--pixel-tests'", action="store_true",
    1530                              dest="pixel_tests",
    1531                              help="enable pixel-to-pixel PNG comparisons")
    1532     option_parser.add_option("", "--no-pixel-tests", action="store_false",
    1533                              dest="pixel_tests",
    1534                              help="disable pixel-to-pixel PNG comparisons")
    1535     option_parser.add_option("", "--fuzzy-pixel-tests", action="store_true",
    1536                              default=False,
    1537                              help="Also use fuzzy matching to compare pixel "
    1538                                   "test outputs.")
    1539     option_parser.add_option("", "--results-directory",
    1540                              default="layout-test-results",
    1541                              help="Output results directory source dir,"
    1542                                   " relative to Debug or Release")
    1543     option_parser.add_option("", "--new-baseline", action="store_true",
    1544                              default=False,
    1545                              help="save all generated results as new baselines"
    1546                                   " into the platform directory, overwriting "
    1547                                   "whatever's already there.")
    1548     option_parser.add_option("", "--noshow-results", action="store_true",
    1549                              default=False, help="don't launch DumpRenderTree"
    1550                              " with results after the tests are done")
    1551     option_parser.add_option("", "--full-results-html", action="store_true",
    1552                              default=False, help="show all failures in "
    1553                              "results.html, rather than only regressions")
    1554     option_parser.add_option("", "--clobber-old-results", action="store_true",
    1555                              default=False, help="Clobbers test results from "
    1556                              "previous runs.")
    1557     option_parser.add_option("", "--lint-test-files", action="store_true",
    1558                              default=False, help="Makes sure the test files "
    1559                              "parse for all configurations. Does not run any "
    1560                              "tests.")
    1561     option_parser.add_option("", "--force", action="store_true",
    1562                              default=False,
    1563                              help="Run all tests, even those marked SKIP "
    1564                                   "in the test list")
    1565     option_parser.add_option("", "--num-dump_render_trees",
    1566                              help="Number of DumpRenderTrees to run in parallel.")
    1567     option_parser.add_option("", "--use-apache", action="store_true",
    1568                              default=False,
    1569                              help="Whether to use apache instead of lighttpd.")
    1570     option_parser.add_option("", "--time-out-ms", default=None,
    1571                              help="Set the timeout for each test")
    1572     option_parser.add_option("", "--run-singly", action="store_true",
    1573                              default=False,
    1574                              help="run a separate DumpRenderTree for each test")
    1575     option_parser.add_option("", "--num-slow-tests-to-log", default=50,
    1576                              help="Number of slow tests whose timings "
    1577                                   "to print.")
    1578     option_parser.add_option("", "--platform",
    1579                              help="Override the platform for expected results")
    1580     option_parser.add_option('--debug', action='store_const', const='Debug',
    1581                              dest="configuration",
    1582                              help='Set the configuration to Debug')
    1583     option_parser.add_option('--release', action='store_const', const='Release',
    1584                              dest="configuration",
    1585                              help='Set the configuration to Release')
    1586     option_parser.add_option("-t", "--target", dest="configuration",
    1587                              help="(DEPRICATED)")
    1588     option_parser.add_option("", "--log", action="store",
    1589                              default="detailed-progress,unexpected",
    1590                              help="log various types of data. The param should"
    1591                              " be a comma-separated list of values from: "
    1592                              "actual,config," + LOG_DETAILED_PROGRESS +
    1593                              ",expected,timing," + LOG_UNEXPECTED + " "
    1594                              "(defaults to " +
    1595                              "--log detailed-progress,unexpected)")
    1596     option_parser.add_option("-v", "--verbose", action="store_true",
    1597                              default=False, help="include debug-level logging")
    1598     option_parser.add_option("", "--sources", action="store_true",
    1599                              help="show expected result file path for each "
    1600                                   "test (implies --verbose)")
    1601     option_parser.add_option("", "--startup-dialog", action="store_true",
    1602                              default=False,
    1603                              help="create a dialog on DumpRenderTree startup")
    1604     option_parser.add_option("", "--gp-fault-error-box", action="store_true",
    1605                              default=False,
    1606                              help="enable Windows GP fault error box")
    1607     option_parser.add_option("", "--wrapper",
    1608                              help="wrapper command to insert before "
    1609                                   "invocations of DumpRenderTree; option is split "
    1610                                   "on whitespace before running. (Example: "
    1611                                   "--wrapper='valgrind --smc-check=all')")
    1612     option_parser.add_option("", "--test-list", action="append",
    1613                              help="read list of tests to run from file",
    1614                              metavar="FILE")
    1615     option_parser.add_option("", "--nocheck-sys-deps", action="store_true",
    1616                              default=False,
    1617                              help="Don't check the system dependencies "
    1618                                   "(themes)")
    1619     option_parser.add_option("", "--randomize-order", action="store_true",
    1620                              default=False,
    1621                              help=("Run tests in random order (useful for "
    1622                                    "tracking down corruption)"))
    1623     option_parser.add_option("", "--run-chunk",
    1624                              default=None,
    1625                              help=("Run a specified chunk (n:l), the "
    1626                                    "nth of len l, of the layout tests"))
    1627     option_parser.add_option("", "--run-part",
    1628                              default=None,
    1629                              help=("Run a specified part (n:m), the nth of m"
    1630                                    " parts, of the layout tests"))
    1631     option_parser.add_option("", "--batch-size",
    1632                              default=None,
    1633                              help=("Run a the tests in batches (n), after "
    1634                                    "every n tests, DumpRenderTree is "
    1635                                    "relaunched."))
    1636     option_parser.add_option("", "--builder-name",
    1637                              default="DUMMY_BUILDER_NAME",
    1638                              help=("The name of the builder shown on the "
    1639                                    "waterfall running this script e.g. "
    1640                                    "WebKit."))
    1641     option_parser.add_option("", "--build-name",
    1642                              default="DUMMY_BUILD_NAME",
    1643                              help=("The name of the builder used in its path, "
    1644                                    "e.g. webkit-rel."))
    1645     option_parser.add_option("", "--build-number",
    1646                              default="DUMMY_BUILD_NUMBER",
    1647                              help=("The build number of the builder running"
    1648                                    "this script."))
    1649     option_parser.add_option("", "--experimental-fully-parallel",
    1650                              action="store_true", default=False,
    1651                              help="run all tests in parallel")
    1652     option_parser.add_option("", "--chromium",
    1653                              action="store_true", default=False,
    1654                              help="use the Chromium port")
     1528
     1529    # FIXME: All of these options should be stored closer to the code which actually uses them.
     1530    # FIXME: configuration_options should move to WebKitPort and be shared across all scripts.
     1531    configuration_options = [
     1532        make_option("-t", "--target", dest="configuration", help="(DEPRECATED)"),
     1533        # FIXME: --help should display which configuration is default.
     1534        make_option('--debug', action='store_const', const='Debug', dest="configuration", help='Set the configuration to Debug'),
     1535        make_option('--release', action='store_const', const='Release', dest="configuration", help='Set the configuration to Release'),
     1536        # old-run-webkit-tests also accepts -c, --configuration CONFIGURATION.
     1537    ]
     1538
     1539    logging_options = [
     1540        make_option("--log", action="store", default="detailed-progress,unexpected",
     1541                    help="log various types of data. The param should be a comma-separated list of values from: "
     1542                         "actual,config," + LOG_DETAILED_PROGRESS + ",expected,timing," + LOG_UNEXPECTED + " "
     1543                          "(defaults to " + "--log detailed-progress,unexpected)"),
     1544        make_option("-v", "--verbose", action="store_true", default=False, help="include debug-level logging"),
     1545        make_option("--sources", action="store_true", help="show expected result file path for each test (implies --verbose)"),
     1546        # old-run-webkit-tests has a --slowest option which just prints the slowest 10.
     1547        make_option("--num-slow-tests-to-log", default=50, help="Number of slow tests whose timings to print."),
     1548    ]
     1549
     1550    # FIXME: These options should move onto the ChromiumPort.
     1551    chromium_options = [
     1552        make_option("--chromium", action="store_true", default=False, help="use the Chromium port"),
     1553        make_option("--startup-dialog", action="store_true", default=False, help="create a dialog on DumpRenderTree startup"),
     1554        make_option("--gp-fault-error-box", action="store_true", default=False, help="enable Windows GP fault error box"),
     1555        make_option("--nocheck-sys-deps", action="store_true", default=False, help="Don't check the system dependencies (themes)"),
     1556    ]
     1557
     1558    # Missing Mac options:
     1559    # FIXME: Need: -g, --guard for guard malloc support on Mac.
     1560    # FIXME: Need: -l --leaks    Enable leaks checking.
     1561    # FIXME: Need: --[no-]sample-on-timeout Run sample on timeout (default: run) (Mac OS X only)
     1562
     1563    results_options = [
     1564        # NEED for bots: --use-remote-links-to-tests     Link to test files within the SVN repository in the results.
     1565        make_option("-p", "--pixel-tests'", action="store_true", dest="pixel_tests", help="Enable pixel-to-pixel PNG comparisons"),
     1566        make_option("--no-pixel-tests", action="store_false", dest="pixel_tests", help="Disable pixel-to-pixel PNG comparisons"),
     1567        make_option("--fuzzy-pixel-tests", action="store_true", default=False, help="Also use fuzzy matching to compare pixel test outputs."),
     1568        # old-run-webkit-tests allows a specific tollerance: --tolerance t  Ignore image differences less than this percentage (default: 0.1)
     1569        make_option("--results-directory", default="layout-test-results", help="Output results directory source dir, relative to Debug or Release"),
     1570        make_option("--new-baseline", action="store_true", default=False, help="Save all generated results as new baselines into the platform directory, overwriting whatever's already there."),
     1571        # FIXME: --noshow should be --no-show to match the GNU get_opt pattern.
     1572        make_option("--noshow-results", action="store_true", default=False, help="Don't launch a browser with results after the tests are done"),
     1573        # old-run-webkit-tests: --[no-]launch-safari            Launch (or do not launch) Safari to display test results (default: launch)
     1574        make_option("--full-results-html", action="store_true", default=False, help="Show all failures in results.html, rather than only regressions"),
     1575        make_option("--clobber-old-results", action="store_true", default=False, help="Clobbers test results from previous runs."),
     1576        make_option("--platform", help="Override the platform for expected results"),
     1577        # old-run-webkit-tests also has HTTP toggle options:
     1578        # --[no-]http                     Run (or do not run) http tests (default: run)
     1579        # --[no-]wait-for-httpd           Wait for httpd if some other test session is using it already (same as WEBKIT_WAIT_FOR_HTTPD=1). (default: 0)
     1580    ]
     1581
     1582    test_options = [
     1583        # old-run-webkit-tests has --valgrind instead of wrapper.
     1584        make_option("--wrapper", help="wrapper command to insert before invocations of DumpRenderTree; option is split "
     1585                                      "on whitespace before running. (Example: --wrapper='valgrind --smc-check=all')"),
     1586        # old-run-webkit-tests: -i|--ignore-tests               Comma-separated list of directories or tests to ignore
     1587        make_option("--test-list", action="append", help="read list of tests to run from file", metavar="FILE"),
     1588        # old-run-webkit-tests uses --skipped==[default|ignore|only] instead of --force:
     1589        make_option("--force", action="store_true", default=False, help="Run all tests, even those marked SKIP in the test list"),
     1590        make_option("--use-apache", action="store_true", default=False, help="Whether to use apache instead of lighttpd."),
     1591        make_option("--time-out-ms", help="Set the timeout for each test"),
     1592        # old-run-webkit-tests calls --randomize-order --random:
     1593        make_option("--randomize-order", action="store_true", default=False, help=("Run tests in random order (useful for tracking down corruption)")),
     1594        make_option("--run-chunk", help=("Run a specified chunk (n:l), the nth of len l, of the layout tests")),
     1595        make_option("--run-part", help=("Run a specified part (n:m), the nth of m parts, of the layout tests")),
     1596        # old-run-webkit-tests calls --batch-size: --nthly n   Restart DumpRenderTree every n tests (default: 1000)
     1597        make_option("--batch-size", help=("Run a the tests in batches (n), after every n tests, DumpRenderTree is relaunched.")),
     1598        # old-run-webkit-tests calls --run-singly: -1|--singly  Isolate each test case run (implies --nthly 1 --verbose)
     1599        make_option("--run-singly", action="store_true", default=False, help="run a separate DumpRenderTree for each test"),
     1600        make_option("--child-processes", help="Number of DumpRenderTrees to run in parallel."), # FIXME: Display default.
     1601        make_option("--experimental-fully-parallel", action="store_true", default=False, help="run all tests in parallel"),
     1602        # FIXME: Need --exit-after-n-failures N       Exit after the first N failures instead of running all tests
     1603        # FIXME: consider: --iterations n                  Number of times to run the set of tests (e.g. ABCABCABC)
     1604    ]
     1605
     1606    misc_options = [
     1607        make_option("--lint-test-files", action="store_true", default=False, help="Makes sure the test files parse for all configurations. Does not run any tests."),
     1608    ]
     1609
     1610    # FIXME: Move these into json_results_generator.py
     1611    results_json_options = [
     1612        make_option("--builder-name", default="DUMMY_BUILDER_NAME", help=("The name of the builder shown on the waterfall running this script e.g. WebKit.")),
     1613        make_option("--build-name", default="DUMMY_BUILD_NAME", help=("The name of the builder used in its path, e.g. webkit-rel.")),
     1614        make_option("--build-number", default="DUMMY_BUILD_NUMBER", help=("The build number of the builder running this script.")),
     1615    ]
     1616
     1617    options_list = configuration_options + logging_options + chromium_options + results_options + test_options + misc_options + results_json_options
     1618    option_parser = optparse.OptionParser(option_list=option_list)
    16551619    return option_parser.parse_args(args)
    16561620
Note: See TracChangeset for help on using the changeset viewer.