Changeset 57796 in webkit
- Timestamp:
- Apr 18, 2010 1:34:17 PM (14 years ago)
- Location:
- trunk/WebKitTools
- Files:
-
- 3 edited
- 1 copied
Legend:
- Unmodified
- Added
- Removed
-
trunk/WebKitTools/ChangeLog
r57795 r57796 1 2010-04-14 Eric Seidel <eric@webkit.org> 2 3 Reviewed by Adam Barth. 4 5 Add a layer of indirection when calling run-webkit-tests to 6 allow testing new-run-webkit-tests on various platforms. 7 https://bugs.webkit.org/show_bug.cgi?id=37632 8 9 This will let us test and fix bugs in new-run-webkit-tests 10 without needing to restart the buildbot master between tests. 11 12 Currently this change leaves run-webkit-tests as-is, but once 13 its landed we will easily be able to turn on/off 14 new-run-webkit-tests for various ports/configurations. 15 16 I will send a note out to webkit-dev about how we will 17 be using this launcher script to test on the bots. 18 19 * Scripts/old-run-webkit-tests: Copied from WebKitTools/Scripts/run-webkit-tests. 20 * Scripts/run-webkit-tests: 21 - A new script which decides whether to run new- or old- 22 run-webkit-tests based on the platform. 23 * Scripts/webkitpy/layout_tests/run_webkit_tests.py: 24 - Add some dummy argument handling for arguments which 25 old-run-webkit-tests supports but new-run-webkit-tests 26 does not yet. 27 1 28 2010-04-18 Eric Seidel <eric@webkit.org> 2 29 -
trunk/WebKitTools/Scripts/run-webkit-tests
r57678 r57796 1 1 #!/usr/bin/perl 2 3 # Copyright (C) 2005, 2006, 2007, 2008, 2009 Apple Inc. All rights reserved. 4 # Copyright (C) 2006 Alexey Proskuryakov (ap@nypop.com) 5 # Copyright (C) 2007 Matt Lilek (pewtermoose@gmail.com) 6 # Copyright (C) 2007 Eric Seidel <eric@webkit.org> 7 # Copyright (C) 2009 Google Inc. All rights reserved. 8 # Copyright (C) 2009 Andras Becsi (becsi.andras@stud.u-szeged.hu), University of Szeged 2 # Copyright (C) 2010 Google Inc. All rights reserved. 9 3 # 10 4 # Redistribution and use in source and binary forms, with or without 11 # modification, are permitted provided that the following conditions 12 # aremet:5 # modification, are permitted provided that the following conditions are 6 # met: 13 7 # 14 # 1. Redistributions of source code must retain the above copyright 15 # notice, this list of conditions and the following disclaimer. 16 # 2. Redistributions in binary form must reproduce the above copyright 17 # notice, this list of conditions and the following disclaimer in the 18 # documentation and/or other materials provided with the distribution. 19 # 3. Neither the name of Apple Computer, Inc. ("Apple") nor the names of 20 # its contributors may be used to endorse or promote products derived 21 # from this software without specific prior written permission. 8 # * Redistributions of source code must retain the above copyright 9 # notice, this list of conditions and the following disclaimer. 10 # * Redistributions in binary form must reproduce the above 11 # copyright notice, this list of conditions and the following disclaimer 12 # in the documentation and/or other materials provided with the 13 # distribution. 14 # * Neither the name of Google Inc. nor the names of its 15 # contributors may be used to endorse or promote products derived from 16 # this software without specific prior written permission. 22 17 # 23 # THIS SOFTWARE IS PROVIDED BY APPLE AND ITS CONTRIBUTORS "AS IS" AND ANY 24 # EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 25 # WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 26 # DISCLAIMED. IN NO EVENT SHALL APPLE OR ITS CONTRIBUTORS BE LIABLE FOR ANY 27 # DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 28 # (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 29 # LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 30 # ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 31 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 32 # THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 18 # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 19 # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 20 # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 21 # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 22 # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 23 # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 24 # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 25 # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 26 # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 27 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 28 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 33 29 34 # Script to run the WebKit Open Source Project layout tests. 30 # This file is a temporary hack. 31 # It will be removed as soon as all platforms are are ready to move to 32 # new-run-webkit-tests and we can then update the buildbots to explicitly 33 # call old-run-webkit-tests for any platforms which will never support 34 # a Python run-webkit-tests. 35 35 36 # Run all the tests passed in on the command line. 37 # If no tests are passed, find all the .html, .shtml, .xml, .xhtml, .xhtmlmp, .pl, .php (and svg) files in the test directory. 38 39 # Run each text. 40 # Compare against the existing file xxx-expected.txt. 41 # If there is a mismatch, generate xxx-actual.txt and xxx-diffs.txt. 42 43 # At the end, report: 44 # the number of tests that got the expected results 45 # the number of tests that ran, but did not get the expected results 46 # the number of tests that failed to run 47 # the number of tests that were run but had no expected results to compare against 36 # This is intentionally written in Perl to guarantee support on 37 # the same set of platforms as old-run-webkit-tests currently supports. 38 # The buildbot master.cfg also currently passes run-webkit-tests to 39 # perl directly instead of executing it in a shell. 48 40 49 41 use strict; 50 42 use warnings; 51 43 52 use Cwd;53 use Data::Dumper;54 use Fcntl qw(F_GETFL F_SETFL O_NONBLOCK);55 use File::Basename;56 use File::Copy;57 use File::Find;58 use File::Path;59 use File::Spec;60 use File::Spec::Functions;61 use File::Temp;62 44 use FindBin; 63 use Getopt::Long; 64 use IPC::Open2; 65 use IPC::Open3; 66 use Time::HiRes qw(time usleep); 45 use lib $FindBin::Bin; 46 use webkitdirs; 67 47 68 use List::Util 'shuffle'; 69 70 use lib $FindBin::Bin; 71 use webkitperl::features; 72 use webkitperl::httpd; 73 use webkitdirs; 74 use VCSUtils; 75 use POSIX; 76 77 sub buildPlatformResultHierarchy(); 78 sub buildPlatformTestHierarchy(@); 79 sub closeCygpaths(); 80 sub closeDumpTool(); 81 sub closeWebSocketServer(); 82 sub configureAndOpenHTTPDIfNeeded(); 83 sub countAndPrintLeaks($$$); 84 sub countFinishedTest($$$$); 85 sub deleteExpectedAndActualResults($); 86 sub dumpToolDidCrash(); 87 sub epiloguesAndPrologues($$); 88 sub expectedDirectoryForTest($;$;$); 89 sub fileNameWithNumber($$); 90 sub htmlForResultsSection(\@$&); 91 sub isTextOnlyTest($); 92 sub launchWithEnv(\@\%); 93 sub resolveAndMakeTestResultsDirectory(); 94 sub numericcmp($$); 95 sub openDiffTool(); 96 sub openDumpTool(); 97 sub parseLeaksandPrintUniqueLeaks(); 98 sub openWebSocketServerIfNeeded(); 99 sub pathcmp($$); 100 sub printFailureMessageForTest($$); 101 sub processIgnoreTests($$); 102 sub readFromDumpToolWithTimer(**); 103 sub readSkippedFiles($); 104 sub recordActualResultsAndDiff($$); 105 sub sampleDumpTool(); 106 sub setFileHandleNonBlocking(*$); 107 sub slowestcmp($$); 108 sub splitpath($); 109 sub stopRunningTestsEarlyIfNeeded(); 110 sub stripExtension($); 111 sub stripMetrics($$); 112 sub testCrashedOrTimedOut($$$$$); 113 sub toURL($); 114 sub toWindowsPath($); 115 sub validateSkippedArg($$;$); 116 sub writeToFile($$); 117 118 # Argument handling 119 my $addPlatformExceptions = 0; 120 my $complexText = 0; 121 my $exitAfterNFailures = 0; 122 my $generateNewResults = isAppleMacWebKit() ? 1 : 0; 123 my $guardMalloc = ''; 124 # FIXME: Dynamic HTTP-port configuration in this file is wrong. The various 125 # apache config files in LayoutTests/http/config govern the port numbers. 126 # Dynamic configuration as-written will also cause random failures in 127 # an IPv6 environment. See https://bugs.webkit.org/show_bug.cgi?id=37104. 128 my $httpdPort = 8000; 129 my $httpdSSLPort = 8443; 130 my $ignoreMetrics = 0; 131 my $webSocketPort = 8880; 132 # wss is disabled until all platforms support pyOpenSSL. 133 # my $webSocketSecurePort = 9323; 134 my $ignoreTests = ''; 135 my $iterations = 1; 136 my $launchSafari = 1; 137 my $mergeDepth; 138 my $pixelTests = ''; 139 my $platform; 140 my $quiet = ''; 141 my $randomizeTests = 0; 142 my $repeatEach = 1; 143 my $report10Slowest = 0; 144 my $resetResults = 0; 145 my $reverseTests = 0; 146 my $root; 147 my $runSample = 1; 148 my $shouldCheckLeaks = 0; 149 my $showHelp = 0; 150 my $stripEditingCallbacks = isCygwin(); 151 my $testHTTP = 1; 152 my $testMedia = 1; 153 my $tmpDir = "/tmp"; 154 my $testResultsDirectory = File::Spec->catfile($tmpDir, "layout-test-results"); 155 my $testsPerDumpTool = 1000; 156 my $threaded = 0; 157 # DumpRenderTree has an internal timeout of 30 seconds, so this must be > 30. 158 my $timeoutSeconds = 35; 159 my $tolerance = 0; 160 my $treatSkipped = "default"; 161 my $useRemoteLinksToTests = 0; 162 my $useValgrind = 0; 163 my $verbose = 0; 164 my $shouldWaitForHTTPD = 0; 165 166 my @leaksFilenames; 167 168 if (isWindows() || isMsys()) { 169 print "This script has to be run under Cygwin to function correctly.\n"; 170 exit 1; 48 sub runningOnBuildBot() 49 { 50 # This is a hack to detect if we're running on the buildbot so we can 51 # pass --verbose to new-run-webkit-tests. This will be removed when we 52 # update the buildbot config to call new-run-webkit-tests explicitly. 53 my %isBuildBotUser = ("apple" => 1, "buildbot" => 1); 54 return $isBuildBotUser{$ENV{"USER"}}; 171 55 } 172 56 173 # Default to --no-http for wx for now. 174 $testHTTP = 0 if (isWx()); 175 176 my $expectedTag = "expected"; 177 my $actualTag = "actual"; 178 my $prettyDiffTag = "pretty-diff"; 179 my $diffsTag = "diffs"; 180 my $errorTag = "stderr"; 181 182 my @macPlatforms = ("mac-tiger", "mac-leopard", "mac-snowleopard", "mac"); 183 184 if (isAppleMacWebKit()) { 185 if (isTiger()) { 186 $platform = "mac-tiger"; 187 $tolerance = 1.0; 188 } elsif (isLeopard()) { 189 $platform = "mac-leopard"; 190 $tolerance = 0.1; 191 } elsif (isSnowLeopard()) { 192 $platform = "mac-snowleopard"; 193 $tolerance = 0.1; 194 } else { 195 $platform = "mac"; 196 } 197 } elsif (isQt()) { 198 if (isDarwin()) { 199 $platform = "qt-mac"; 200 } elsif (isLinux()) { 201 $platform = "qt-linux"; 202 } elsif (isWindows() || isCygwin()) { 203 $platform = "qt-win"; 204 } else { 205 $platform = "qt"; 206 } 207 } elsif (isGtk()) { 208 $platform = "gtk"; 209 if (!$ENV{"WEBKIT_TESTFONTS"}) { 210 print "The WEBKIT_TESTFONTS environment variable is not defined.\n"; 211 print "You must set it before running the tests.\n"; 212 print "Use git to grab the actual fonts from http://gitorious.org/qtwebkit/testfonts\n"; 213 exit 1; 214 } 215 } elsif (isWx()) { 216 $platform = "wx"; 217 } elsif (isCygwin()) { 218 $platform = "win"; 57 sub useNewRunWebKitTests() 58 { 59 # Change this check to control which platforms use 60 # new-run-webkit-tests by default. 61 return 0; 219 62 } 220 63 221 if (!defined($platform)) { 222 print "WARNING: Your platform is not recognized. Any platform-specific results will be generated in platform/undefined.\n"; 223 $platform = "undefined"; 224 } 64 my $harnessName = "old-run-webkit-tests"; 225 65 226 my $programName = basename($0); 227 my $launchSafariDefault = $launchSafari ? "launch" : "do not launch"; 228 my $httpDefault = $testHTTP ? "run" : "do not run"; 229 my $sampleDefault = $runSample ? "run" : "do not run"; 230 231 my $usage = <<EOF; 232 Usage: $programName [options] [testdir|testpath ...] 233 --add-platform-exceptions Put new results for non-platform-specific failing tests into the platform-specific results directory 234 --complex-text Use the complex text code path for all text (Mac OS X and Windows only) 235 -c|--configuration config Set DumpRenderTree build configuration 236 -g|--guard-malloc Enable malloc guard 237 --exit-after-n-failures N Exit after the first N failures instead of running all tests 238 -h|--help Show this help message 239 --[no-]http Run (or do not run) http tests (default: $httpDefault) 240 --[no-]wait-for-httpd Wait for httpd if some other test session is using it already (same as WEBKIT_WAIT_FOR_HTTPD=1). (default: $shouldWaitForHTTPD) 241 -i|--ignore-tests Comma-separated list of directories or tests to ignore 242 --iterations n Number of times to run the set of tests (e.g. ABCABCABC) 243 --[no-]launch-safari Launch (or do not launch) Safari to display test results (default: $launchSafariDefault) 244 -l|--leaks Enable leaks checking 245 --[no-]new-test-results Generate results for new tests 246 --nthly n Restart DumpRenderTree every n tests (default: $testsPerDumpTool) 247 -p|--pixel-tests Enable pixel tests 248 --tolerance t Ignore image differences less than this percentage (default: $tolerance) 249 --platform Override the detected platform to use for tests and results (default: $platform) 250 --port Web server port to use with http tests 251 -q|--quiet Less verbose output 252 --reset-results Reset ALL results (including pixel tests if --pixel-tests is set) 253 -o|--results-directory Output results directory (default: $testResultsDirectory) 254 --random Run the tests in a random order 255 --repeat-each n Number of times to run each test (e.g. AAABBBCCC) 256 --reverse Run the tests in reverse alphabetical order 257 --root Path to root tools build 258 --[no-]sample-on-timeout Run sample on timeout (default: $sampleDefault) (Mac OS X only) 259 -1|--singly Isolate each test case run (implies --nthly 1 --verbose) 260 --skipped=[default|ignore|only] Specifies how to treat the Skipped file 261 default: Tests/directories listed in the Skipped file are not tested 262 ignore: The Skipped file is ignored 263 only: Only those tests/directories listed in the Skipped file will be run 264 --slowest Report the 10 slowest tests 265 --ignore-metrics Ignore metrics in tests 266 --[no-]strip-editing-callbacks Remove editing callbacks from expected results 267 -t|--threaded Run a concurrent JavaScript thead with each test 268 --timeout t Sets the number of seconds before a test times out (default: $timeoutSeconds) 269 --valgrind Run DumpRenderTree inside valgrind (Qt/Linux only) 270 -v|--verbose More verbose output (overrides --quiet) 271 -m|--merge-leak-depth arg Merges leak callStacks and prints the number of unique leaks beneath a callstack depth of arg. Defaults to 5. 272 --use-remote-links-to-tests Link to test files within the SVN repository in the results. 273 EOF 274 275 setConfiguration(); 276 277 my $getOptionsResult = GetOptions( 278 'add-platform-exceptions' => \$addPlatformExceptions, 279 'complex-text' => \$complexText, 280 'exit-after-n-failures=i' => \$exitAfterNFailures, 281 'guard-malloc|g' => \$guardMalloc, 282 'help|h' => \$showHelp, 283 'http!' => \$testHTTP, 284 'wait-for-httpd!' => \$shouldWaitForHTTPD, 285 'ignore-metrics!' => \$ignoreMetrics, 286 'ignore-tests|i=s' => \$ignoreTests, 287 'iterations=i' => \$iterations, 288 'launch-safari!' => \$launchSafari, 289 'leaks|l' => \$shouldCheckLeaks, 290 'merge-leak-depth|m:5' => \$mergeDepth, 291 'new-test-results!' => \$generateNewResults, 292 'nthly=i' => \$testsPerDumpTool, 293 'pixel-tests|p' => \$pixelTests, 294 'platform=s' => \$platform, 295 'port=i' => \$httpdPort, 296 'quiet|q' => \$quiet, 297 'random' => \$randomizeTests, 298 'repeat-each=i' => \$repeatEach, 299 'reset-results' => \$resetResults, 300 'results-directory|o=s' => \$testResultsDirectory, 301 'reverse' => \$reverseTests, 302 'root=s' => \$root, 303 'sample-on-timeout!' => \$runSample, 304 'singly|1' => sub { $testsPerDumpTool = 1; }, 305 'skipped=s' => \&validateSkippedArg, 306 'slowest' => \$report10Slowest, 307 'strip-editing-callbacks!' => \$stripEditingCallbacks, 308 'threaded|t' => \$threaded, 309 'timeout=i' => \$timeoutSeconds, 310 'tolerance=f' => \$tolerance, 311 'use-remote-links-to-tests' => \$useRemoteLinksToTests, 312 'valgrind' => \$useValgrind, 313 'verbose|v' => \$verbose, 314 ); 315 316 if (!$getOptionsResult || $showHelp) { 317 print STDERR $usage; 318 exit 1; 319 } 320 321 my $ignoreSkipped = $treatSkipped eq "ignore"; 322 my $skippedOnly = $treatSkipped eq "only"; 323 324 my $configuration = configuration(); 325 326 # We need an environment variable to be able to enable the feature per-slave 327 $shouldWaitForHTTPD = $ENV{"WEBKIT_WAIT_FOR_HTTPD"} unless ($shouldWaitForHTTPD); 328 $verbose = 1 if $testsPerDumpTool == 1; 329 330 if ($shouldCheckLeaks && $testsPerDumpTool > 1000) { 331 print STDERR "\nWARNING: Running more than 1000 tests at a time with MallocStackLogging enabled may cause a crash.\n\n"; 332 } 333 334 # Stack logging does not play well with QuickTime on Tiger (rdar://problem/5537157) 335 $testMedia = 0 if $shouldCheckLeaks && isTiger(); 336 337 # Generating remote links causes a lot of unnecessary spew on GTK build bot 338 $useRemoteLinksToTests = 0 if isGtk(); 339 340 setConfigurationProductDir(Cwd::abs_path($root)) if (defined($root)); 341 my $productDir = productDir(); 342 $productDir .= "/bin" if isQt(); 343 $productDir .= "/Programs" if isGtk(); 344 345 chdirWebKit(); 346 347 if (!defined($root)) { 348 print STDERR "Running build-dumprendertree\n"; 349 350 local *DEVNULL; 351 my ($childIn, $childOut, $childErr); 352 if ($quiet) { 353 open(DEVNULL, ">", File::Spec->devnull()) or die "Failed to open /dev/null"; 354 $childOut = ">&DEVNULL"; 355 $childErr = ">&DEVNULL"; 356 } else { 357 # When not quiet, let the child use our stdout/stderr. 358 $childOut = ">&STDOUT"; 359 $childErr = ">&STDERR"; 360 } 361 362 my @args = argumentsForConfiguration(); 363 my $buildProcess = open3($childIn, $childOut, $childErr, "WebKitTools/Scripts/build-dumprendertree", @args) or die "Failed to run build-dumprendertree"; 364 close($childIn); 365 waitpid $buildProcess, 0; 366 my $buildResult = $?; 367 close($childOut); 368 close($childErr); 369 370 close DEVNULL if ($quiet); 371 372 if ($buildResult) { 373 print STDERR "Compiling DumpRenderTree failed!\n"; 374 exit exitStatus($buildResult); 66 if (useNewRunWebKitTests()) { 67 $harnessName = "new-run-webkit-tests"; 68 if (runningOnBuildBot()) { 69 push(@ARGV, "--verbose"); 70 # old-run-webkit-tests treats --results-directory as $CWD relative. 71 # new-run-webkit-tests treats --results-directory as build directory relative. 72 # Override the passed in --results-directory by appending a new one 73 # (later arguments override earlier ones in Python's optparse). 74 push(@ARGV, "--results-directory"); 75 # The buildbot always uses $SRCDIR/layout-test-results, hardcode it: 76 push(@ARGV, sourceDir() . "/layout-test-results"); 375 77 } 376 78 } 377 79 378 my $dumpToolName = "DumpRenderTree"; 379 $dumpToolName .= "_debug" if isCygwin() && configurationForVisualStudio() !~ /^Release|Debug_Internal$/; 380 my $dumpTool = "$productDir/$dumpToolName"; 381 die "can't find executable $dumpToolName (looked in $productDir)\n" unless -x $dumpTool; 382 383 my $imageDiffTool = "$productDir/ImageDiff"; 384 $imageDiffTool .= "_debug" if isCygwin() && configurationForVisualStudio() !~ /^Release|Debug_Internal$/; 385 die "can't find executable $imageDiffTool (looked in $productDir)\n" if $pixelTests && !-x $imageDiffTool; 386 387 checkFrameworks() unless isCygwin(); 388 389 if (isAppleMacWebKit()) { 390 push @INC, $productDir; 391 require DumpRenderTreeSupport; 392 } 393 394 my $layoutTestsName = "LayoutTests"; 395 my $testDirectory = File::Spec->rel2abs($layoutTestsName); 396 my $expectedDirectory = $testDirectory; 397 my $platformBaseDirectory = catdir($testDirectory, "platform"); 398 my $platformTestDirectory = catdir($platformBaseDirectory, $platform); 399 my @platformResultHierarchy = buildPlatformResultHierarchy(); 400 my @platformTestHierarchy = buildPlatformTestHierarchy(@platformResultHierarchy); 401 402 $expectedDirectory = $ENV{"WebKitExpectedTestResultsDirectory"} if $ENV{"WebKitExpectedTestResultsDirectory"}; 403 404 $testResultsDirectory = File::Spec->rel2abs($testResultsDirectory); 405 my $testResults = File::Spec->catfile($testResultsDirectory, "results.html"); 406 407 if (isAppleMacWebKit()) { 408 print STDERR "Compiling Java tests\n"; 409 my $javaTestsDirectory = catdir($testDirectory, "java"); 410 411 if (system("/usr/bin/make", "-C", "$javaTestsDirectory")) { 412 exit 1; 413 } 414 } 415 416 417 print "Running tests from $testDirectory\n"; 418 if ($pixelTests) { 419 print "Enabling pixel tests with a tolerance of $tolerance%\n"; 420 if (isDarwin()) { 421 print "WARNING: Temporarily changing the main display color profile:\n"; 422 print "\tThe colors on your screen will change for the duration of the testing.\n"; 423 print "\tThis allows the pixel tests to have consistent color values across all machines.\n"; 424 425 if (isPerianInstalled()) { 426 print "WARNING: Perian's QuickTime component is installed and this may affect pixel test results!\n"; 427 print "\tYou should avoid generating new pixel results in this environment.\n"; 428 print "\tSee https://bugs.webkit.org/show_bug.cgi?id=22615 for details.\n"; 429 } 430 } 431 } 432 433 system "ln", "-s", $testDirectory, "/tmp/LayoutTests" unless -x "/tmp/LayoutTests"; 434 435 my %ignoredFiles = ( "results.html" => 1 ); 436 my %ignoredDirectories = map { $_ => 1 } qw(platform); 437 my %ignoredLocalDirectories = map { $_ => 1 } qw(.svn _svn resources script-tests); 438 my %supportedFileExtensions = map { $_ => 1 } qw(html shtml xml xhtml xhtmlmp pl php); 439 440 if (!checkWebCoreFeatureSupport("MathML", 0)) { 441 $ignoredDirectories{'mathml'} = 1; 442 } 443 444 # FIXME: We should fix webkitperl/features.pm:hasFeature() to do the correct feature detection for Cygwin. 445 if (checkWebCoreFeatureSupport("SVG", 0)) { 446 $supportedFileExtensions{'svg'} = 1; 447 } elsif (isCygwin()) { 448 $supportedFileExtensions{'svg'} = 1; 449 } else { 450 $ignoredLocalDirectories{'svg'} = 1; 451 } 452 453 if (!$testHTTP) { 454 $ignoredDirectories{'http'} = 1; 455 $ignoredDirectories{'websocket'} = 1; 456 } 457 458 if (!$testMedia) { 459 $ignoredDirectories{'media'} = 1; 460 $ignoredDirectories{'http/tests/media'} = 1; 461 } 462 463 my $supportedFeaturesResult = ""; 464 465 if (isCygwin()) { 466 # Collect supported features list 467 setPathForRunningWebKitApp(\%ENV); 468 my $supportedFeaturesCommand = $dumpTool . " --print-supported-features 2>&1"; 469 $supportedFeaturesResult = `$supportedFeaturesCommand 2>&1`; 470 } 471 472 my $hasAcceleratedCompositing = 0; 473 my $has3DRendering = 0; 474 475 if (isCygwin()) { 476 $hasAcceleratedCompositing = $supportedFeaturesResult =~ /AcceleratedCompositing/; 477 $has3DRendering = $supportedFeaturesResult =~ /3DRendering/; 478 } else { 479 $hasAcceleratedCompositing = checkWebCoreFeatureSupport("Accelerated Compositing", 0); 480 $has3DRendering = checkWebCoreFeatureSupport("3D Rendering", 0); 481 } 482 483 if (!$hasAcceleratedCompositing) { 484 $ignoredDirectories{'compositing'} = 1; 485 } 486 487 if (!$has3DRendering) { 488 $ignoredDirectories{'animations/3d'} = 1; 489 $ignoredDirectories{'transforms/3d'} = 1; 490 } 491 492 if (!checkWebCoreFeatureSupport("3D Canvas", 0)) { 493 $ignoredDirectories{'fast/canvas/webgl'} = 1; 494 } 495 496 if (checkWebCoreFeatureSupport("WML", 0)) { 497 $supportedFileExtensions{'wml'} = 1; 498 } else { 499 $ignoredDirectories{'http/tests/wml'} = 1; 500 $ignoredDirectories{'fast/wml'} = 1; 501 $ignoredDirectories{'wml'} = 1; 502 } 503 504 if (!checkWebCoreFeatureSupport("XHTMLMP", 0)) { 505 $ignoredDirectories{'fast/xhtmlmp'} = 1; 506 } 507 508 processIgnoreTests($ignoreTests, "ignore-tests") if $ignoreTests; 509 if (!$ignoreSkipped) { 510 if (!$skippedOnly || @ARGV == 0) { 511 readSkippedFiles(""); 512 } else { 513 # Since readSkippedFiles() appends to @ARGV, we must use a foreach 514 # loop so that we only iterate over the original argument list. 515 foreach my $argnum (0 .. $#ARGV) { 516 readSkippedFiles(shift @ARGV); 517 } 518 } 519 } 520 521 my @tests = findTestsToRun(); 522 523 die "no tests to run\n" if !@tests; 524 525 my %counts; 526 my %tests; 527 my %imagesPresent; 528 my %imageDifferences; 529 my %durations; 530 my $count = 0; 531 my $leaksOutputFileNumber = 1; 532 my $totalLeaks = 0; 533 534 my @toolArgs = (); 535 push @toolArgs, "--pixel-tests" if $pixelTests; 536 push @toolArgs, "--threaded" if $threaded; 537 push @toolArgs, "--complex-text" if $complexText; 538 push @toolArgs, "-"; 539 540 my @diffToolArgs = (); 541 push @diffToolArgs, "--tolerance", $tolerance; 542 543 $| = 1; 544 545 my $dumpToolPID; 546 my $isDumpToolOpen = 0; 547 my $dumpToolCrashed = 0; 548 my $imageDiffToolPID; 549 my $isDiffToolOpen = 0; 550 551 my $atLineStart = 1; 552 my $lastDirectory = ""; 553 554 my $isHttpdOpen = 0; 555 my $isWebSocketServerOpen = 0; 556 my $webSocketServerPID = 0; 557 my $failedToStartWebSocketServer = 0; 558 # wss is disabled until all platforms support pyOpenSSL. 559 # my $webSocketSecureServerPID = 0; 560 561 sub catch_pipe { $dumpToolCrashed = 1; } 562 $SIG{"PIPE"} = "catch_pipe"; 563 564 print "Testing ", scalar @tests, " test cases"; 565 print " $iterations times" if ($iterations > 1); 566 print ", repeating each test $repeatEach times" if ($repeatEach > 1); 567 print ".\n"; 568 569 my $overallStartTime = time; 570 571 my %expectedResultPaths; 572 573 my @originalTests = @tests; 574 # Add individual test repetitions 575 if ($repeatEach > 1) { 576 @tests = (); 577 foreach my $test (@originalTests) { 578 for (my $i = 0; $i < $repeatEach; $i++) { 579 push(@tests, $test); 580 } 581 } 582 } 583 # Add test set repetitions 584 for (my $i = 1; $i < $iterations; $i++) { 585 push(@tests, @originalTests); 586 } 587 588 for my $test (@tests) { 589 my $newDumpTool = not $isDumpToolOpen; 590 openDumpTool(); 591 592 my $base = stripExtension($test); 593 my $expectedExtension = ".txt"; 594 595 my $dir = $base; 596 $dir =~ s|/[^/]+$||; 597 598 if ($newDumpTool || $dir ne $lastDirectory) { 599 foreach my $logue (epiloguesAndPrologues($newDumpTool ? "" : $lastDirectory, $dir)) { 600 if (isCygwin()) { 601 $logue = toWindowsPath($logue); 602 } else { 603 $logue = canonpath($logue); 604 } 605 if ($verbose) { 606 print "running epilogue or prologue $logue\n"; 607 } 608 print OUT "$logue\n"; 609 # Throw away output from DumpRenderTree. 610 # Once for the test output and once for pixel results (empty) 611 while (<IN>) { 612 last if /#EOF/; 613 } 614 while (<IN>) { 615 last if /#EOF/; 616 } 617 } 618 } 619 620 if ($verbose) { 621 print "running $test -> "; 622 $atLineStart = 0; 623 } elsif (!$quiet) { 624 if ($dir ne $lastDirectory) { 625 print "\n" unless $atLineStart; 626 print "$dir "; 627 } 628 print "."; 629 $atLineStart = 0; 630 } 631 632 $lastDirectory = $dir; 633 634 my $result; 635 636 my $startTime = time if $report10Slowest; 637 638 # Try to read expected hash file for pixel tests 639 my $suffixExpectedHash = ""; 640 if ($pixelTests && !$resetResults) { 641 my $expectedPixelDir = expectedDirectoryForTest($base, 0, "png"); 642 if (open EXPECTEDHASH, "$expectedPixelDir/$base-$expectedTag.checksum") { 643 my $expectedHash = <EXPECTEDHASH>; 644 chomp($expectedHash); 645 close EXPECTEDHASH; 646 647 # Format expected hash into a suffix string that is appended to the path / URL passed to DRT 648 $suffixExpectedHash = "'$expectedHash"; 649 } 650 } 651 652 if ($test =~ /^http\//) { 653 configureAndOpenHTTPDIfNeeded(); 654 if ($test !~ /^http\/tests\/local\// && $test !~ /^http\/tests\/ssl\// && $test !~ /^http\/tests\/wml\// && $test !~ /^http\/tests\/media\//) { 655 my $path = canonpath($test); 656 $path =~ s/^http\/tests\///; 657 print OUT "http://127.0.0.1:$httpdPort/$path$suffixExpectedHash\n"; 658 } elsif ($test =~ /^http\/tests\/ssl\//) { 659 my $path = canonpath($test); 660 $path =~ s/^http\/tests\///; 661 print OUT "https://127.0.0.1:$httpdSSLPort/$path$suffixExpectedHash\n"; 662 } else { 663 my $testPath = "$testDirectory/$test"; 664 if (isCygwin()) { 665 $testPath = toWindowsPath($testPath); 666 } else { 667 $testPath = canonpath($testPath); 668 } 669 print OUT "$testPath$suffixExpectedHash\n"; 670 } 671 } elsif ($test =~ /^websocket\//) { 672 if ($test =~ /^websocket\/tests\/local\//) { 673 my $testPath = "$testDirectory/$test"; 674 if (isCygwin()) { 675 $testPath = toWindowsPath($testPath); 676 } else { 677 $testPath = canonpath($testPath); 678 } 679 print OUT "$testPath\n"; 680 } else { 681 if (openWebSocketServerIfNeeded()) { 682 my $path = canonpath($test); 683 if ($test =~ /^websocket\/tests\/ssl\//) { 684 # wss is disabled until all platforms support pyOpenSSL. 685 print STDERR "Error: wss is disabled until all platforms support pyOpenSSL."; 686 # print OUT "https://127.0.0.1:$webSocketSecurePort/$path\n"; 687 } else { 688 print OUT "http://127.0.0.1:$webSocketPort/$path\n"; 689 } 690 } else { 691 # We failed to launch the WebSocket server. Display a useful error message rather than attempting 692 # to run tests that expect the server to be available. 693 my $errorMessagePath = "$testDirectory/websocket/resources/server-failed-to-start.html"; 694 $errorMessagePath = isCygwin() ? toWindowsPath($errorMessagePath) : canonpath($errorMessagePath); 695 print OUT "$errorMessagePath\n"; 696 } 697 } 698 } else { 699 my $testPath = "$testDirectory/$test"; 700 if (isCygwin()) { 701 $testPath = toWindowsPath($testPath); 702 } else { 703 $testPath = canonpath($testPath); 704 } 705 print OUT "$testPath$suffixExpectedHash\n" if defined $testPath; 706 } 707 708 # DumpRenderTree is expected to dump two "blocks" to stdout for each test. 709 # Each block is terminated by a #EOF on a line by itself. 710 # The first block is the output of the test (in text, RenderTree or other formats). 711 # The second block is for optional pixel data in PNG format, and may be empty if 712 # pixel tests are not being run, or the test does not dump pixels (e.g. text tests). 713 my $readResults = readFromDumpToolWithTimer(IN, ERROR); 714 715 my $actual = $readResults->{output}; 716 my $error = $readResults->{error}; 717 718 $expectedExtension = $readResults->{extension}; 719 my $expectedFileName = "$base-$expectedTag.$expectedExtension"; 720 721 my $isText = isTextOnlyTest($actual); 722 723 my $expectedDir = expectedDirectoryForTest($base, $isText, $expectedExtension); 724 $expectedResultPaths{$base} = "$expectedDir/$expectedFileName"; 725 726 unless ($readResults->{status} eq "success") { 727 my $crashed = $readResults->{status} eq "crashed"; 728 testCrashedOrTimedOut($test, $base, $crashed, $actual, $error); 729 countFinishedTest($test, $base, $crashed ? "crash" : "timedout", 0); 730 last if stopRunningTestsEarlyIfNeeded(); 731 next; 732 } 733 734 $durations{$test} = time - $startTime if $report10Slowest; 735 736 my $expected; 737 738 if (!$resetResults && open EXPECTED, "<", "$expectedDir/$expectedFileName") { 739 $expected = ""; 740 while (<EXPECTED>) { 741 next if $stripEditingCallbacks && $_ =~ /^EDITING DELEGATE:/; 742 $expected .= $_; 743 } 744 close EXPECTED; 745 } 746 747 if ($ignoreMetrics && !$isText && defined $expected) { 748 ($actual, $expected) = stripMetrics($actual, $expected); 749 } 750 751 if ($shouldCheckLeaks && $testsPerDumpTool == 1) { 752 print " $test -> "; 753 } 754 755 my $actualPNG = ""; 756 my $diffPNG = ""; 757 my $diffPercentage = 0; 758 my $diffResult = "passed"; 759 760 my $actualHash = ""; 761 my $expectedHash = ""; 762 my $actualPNGSize = 0; 763 764 while (<IN>) { 765 last if /#EOF/; 766 if (/ActualHash: ([a-f0-9]{32})/) { 767 $actualHash = $1; 768 } elsif (/ExpectedHash: ([a-f0-9]{32})/) { 769 $expectedHash = $1; 770 } elsif (/Content-Length: (\d+)\s*/) { 771 $actualPNGSize = $1; 772 read(IN, $actualPNG, $actualPNGSize); 773 } 774 } 775 776 if ($verbose && $pixelTests && !$resetResults && $actualPNGSize) { 777 if ($actualHash eq "" && $expectedHash eq "") { 778 printFailureMessageForTest($test, "WARNING: actual & expected pixel hashes are missing!"); 779 } elsif ($actualHash eq "") { 780 printFailureMessageForTest($test, "WARNING: actual pixel hash is missing!"); 781 } elsif ($expectedHash eq "") { 782 printFailureMessageForTest($test, "WARNING: expected pixel hash is missing!"); 783 } 784 } 785 786 if ($actualPNGSize > 0) { 787 my $expectedPixelDir = expectedDirectoryForTest($base, 0, "png"); 788 789 if (!$resetResults && ($expectedHash ne $actualHash || ($actualHash eq "" && $expectedHash eq ""))) { 790 if (-f "$expectedPixelDir/$base-$expectedTag.png") { 791 my $expectedPNGSize = -s "$expectedPixelDir/$base-$expectedTag.png"; 792 my $expectedPNG = ""; 793 open EXPECTEDPNG, "$expectedPixelDir/$base-$expectedTag.png"; 794 read(EXPECTEDPNG, $expectedPNG, $expectedPNGSize); 795 796 openDiffTool(); 797 print DIFFOUT "Content-Length: $actualPNGSize\n"; 798 print DIFFOUT $actualPNG; 799 800 print DIFFOUT "Content-Length: $expectedPNGSize\n"; 801 print DIFFOUT $expectedPNG; 802 803 while (<DIFFIN>) { 804 last if /^error/ || /^diff:/; 805 if (/Content-Length: (\d+)\s*/) { 806 read(DIFFIN, $diffPNG, $1); 807 } 808 } 809 810 if (/^diff: (.+)% (passed|failed)/) { 811 $diffPercentage = $1 + 0; 812 $imageDifferences{$base} = $diffPercentage; 813 $diffResult = $2; 814 } 815 816 if (!$diffPercentage) { 817 printFailureMessageForTest($test, "pixel hash failed (but pixel test still passes)"); 818 } 819 } elsif ($verbose) { 820 printFailureMessageForTest($test, "WARNING: expected image is missing!"); 821 } 822 } 823 824 if ($resetResults || !-f "$expectedPixelDir/$base-$expectedTag.png") { 825 mkpath catfile($expectedPixelDir, dirname($base)) if $testDirectory ne $expectedPixelDir; 826 writeToFile("$expectedPixelDir/$base-$expectedTag.png", $actualPNG); 827 } 828 829 if ($actualHash ne "" && ($resetResults || !-f "$expectedPixelDir/$base-$expectedTag.checksum")) { 830 writeToFile("$expectedPixelDir/$base-$expectedTag.checksum", $actualHash); 831 } 832 } 833 834 if (dumpToolDidCrash()) { 835 $result = "crash"; 836 testCrashedOrTimedOut($test, $base, 1, $actual, $error); 837 } elsif (!defined $expected) { 838 if ($verbose) { 839 print "new " . ($resetResults ? "result" : "test") ."\n"; 840 $atLineStart = 1; 841 } 842 $result = "new"; 843 844 if ($generateNewResults || $resetResults) { 845 mkpath catfile($expectedDir, dirname($base)) if $testDirectory ne $expectedDir; 846 writeToFile("$expectedDir/$expectedFileName", $actual); 847 } 848 deleteExpectedAndActualResults($base); 849 recordActualResultsAndDiff($base, $actual); 850 if (!$resetResults) { 851 # Always print the file name for new tests, as they will probably need some manual inspection. 852 # in verbose mode we already printed the test case, so no need to do it again. 853 unless ($verbose) { 854 print "\n" unless $atLineStart; 855 print "$test -> "; 856 } 857 my $resultsDir = catdir($expectedDir, dirname($base)); 858 if ($generateNewResults) { 859 print "new (results generated in $resultsDir)\n"; 860 } else { 861 print "new\n"; 862 } 863 $atLineStart = 1; 864 } 865 } elsif ($actual eq $expected && $diffResult eq "passed") { 866 if ($verbose) { 867 print "succeeded\n"; 868 $atLineStart = 1; 869 } 870 $result = "match"; 871 deleteExpectedAndActualResults($base); 872 } else { 873 $result = "mismatch"; 874 875 my $pixelTestFailed = $pixelTests && $diffPNG && $diffPNG ne ""; 876 my $testFailed = $actual ne $expected; 877 878 my $message = !$testFailed ? "pixel test failed" : "failed"; 879 880 if (($testFailed || $pixelTestFailed) && $addPlatformExceptions) { 881 my $testBase = catfile($testDirectory, $base); 882 my $expectedBase = catfile($expectedDir, $base); 883 my $testIsMaximallyPlatformSpecific = $testBase =~ m|^\Q$platformTestDirectory\E/|; 884 my $expectedResultIsMaximallyPlatformSpecific = $expectedBase =~ m|^\Q$platformTestDirectory\E/|; 885 if (!$testIsMaximallyPlatformSpecific && !$expectedResultIsMaximallyPlatformSpecific) { 886 mkpath catfile($platformTestDirectory, dirname($base)); 887 if ($testFailed) { 888 my $expectedFile = catfile($platformTestDirectory, "$expectedFileName"); 889 writeToFile("$expectedFile", $actual); 890 } 891 if ($pixelTestFailed) { 892 my $expectedFile = catfile($platformTestDirectory, "$base-$expectedTag.checksum"); 893 writeToFile("$expectedFile", $actualHash); 894 895 $expectedFile = catfile($platformTestDirectory, "$base-$expectedTag.png"); 896 writeToFile("$expectedFile", $actualPNG); 897 } 898 $message .= " (results generated in $platformTestDirectory)"; 899 } 900 } 901 902 printFailureMessageForTest($test, $message); 903 904 my $dir = "$testResultsDirectory/$base"; 905 $dir =~ s|/([^/]+)$|| or die "Failed to find test name from base\n"; 906 my $testName = $1; 907 mkpath $dir; 908 909 deleteExpectedAndActualResults($base); 910 recordActualResultsAndDiff($base, $actual); 911 912 if ($pixelTestFailed) { 913 $imagesPresent{$base} = 1; 914 915 writeToFile("$testResultsDirectory/$base-$actualTag.png", $actualPNG); 916 writeToFile("$testResultsDirectory/$base-$diffsTag.png", $diffPNG); 917 918 my $expectedPixelDir = expectedDirectoryForTest($base, 0, "png"); 919 copy("$expectedPixelDir/$base-$expectedTag.png", "$testResultsDirectory/$base-$expectedTag.png"); 920 921 open DIFFHTML, ">$testResultsDirectory/$base-$diffsTag.html" or die; 922 print DIFFHTML "<html>\n"; 923 print DIFFHTML "<head>\n"; 924 print DIFFHTML "<title>$base Image Compare</title>\n"; 925 print DIFFHTML "<script language=\"Javascript\" type=\"text/javascript\">\n"; 926 print DIFFHTML "var currentImage = 0;\n"; 927 print DIFFHTML "var imageNames = new Array(\"Actual\", \"Expected\");\n"; 928 print DIFFHTML "var imagePaths = new Array(\"$testName-$actualTag.png\", \"$testName-$expectedTag.png\");\n"; 929 if (-f "$testDirectory/$base-w3c.png") { 930 copy("$testDirectory/$base-w3c.png", "$testResultsDirectory/$base-w3c.png"); 931 print DIFFHTML "imageNames.push(\"W3C\");\n"; 932 print DIFFHTML "imagePaths.push(\"$testName-w3c.png\");\n"; 933 } 934 print DIFFHTML "function animateImage() {\n"; 935 print DIFFHTML " var image = document.getElementById(\"animatedImage\");\n"; 936 print DIFFHTML " var imageText = document.getElementById(\"imageText\");\n"; 937 print DIFFHTML " image.src = imagePaths[currentImage];\n"; 938 print DIFFHTML " imageText.innerHTML = imageNames[currentImage] + \" Image\";\n"; 939 print DIFFHTML " currentImage = (currentImage + 1) % imageNames.length;\n"; 940 print DIFFHTML " setTimeout('animateImage()',2000);\n"; 941 print DIFFHTML "}\n"; 942 print DIFFHTML "</script>\n"; 943 print DIFFHTML "</head>\n"; 944 print DIFFHTML "<body onLoad=\"animateImage();\">\n"; 945 print DIFFHTML "<table>\n"; 946 if ($diffPercentage) { 947 print DIFFHTML "<tr>\n"; 948 print DIFFHTML "<td>Difference between images: <a href=\"$testName-$diffsTag.png\">$diffPercentage%</a></td>\n"; 949 print DIFFHTML "</tr>\n"; 950 } 951 print DIFFHTML "<tr>\n"; 952 print DIFFHTML "<td><a href=\"" . toURL("$testDirectory/$test") . "\">test file</a></td>\n"; 953 print DIFFHTML "</tr>\n"; 954 print DIFFHTML "<tr>\n"; 955 print DIFFHTML "<td id=\"imageText\" style=\"text-weight: bold;\">Actual Image</td>\n"; 956 print DIFFHTML "</tr>\n"; 957 print DIFFHTML "<tr>\n"; 958 print DIFFHTML "<td><img src=\"$testName-$actualTag.png\" id=\"animatedImage\"></td>\n"; 959 print DIFFHTML "</tr>\n"; 960 print DIFFHTML "</table>\n"; 961 print DIFFHTML "</body>\n"; 962 print DIFFHTML "</html>\n"; 963 } 964 } 965 966 if ($error) { 967 my $dir = "$testResultsDirectory/$base"; 968 $dir =~ s|/([^/]+)$|| or die "Failed to find test name from base\n"; 969 mkpath $dir; 970 971 writeToFile("$testResultsDirectory/$base-$errorTag.txt", $error); 972 973 $counts{error}++; 974 push @{$tests{error}}, $test; 975 } 976 977 countFinishedTest($test, $base, $result, $isText); 978 last if stopRunningTestsEarlyIfNeeded(); 979 } 980 981 my $totalTestingTime = time - $overallStartTime; 982 my $waitTime = getWaitTime(); 983 if ($waitTime > 0.1) { 984 my $normalizedTestingTime = $totalTestingTime - $waitTime; 985 printf "\n%0.2fs HTTPD waiting time\n", $waitTime . ""; 986 printf "%0.2fs normalized testing time", $normalizedTestingTime . ""; 987 } 988 printf "\n%0.2fs total testing time\n", $totalTestingTime . ""; 989 990 !$isDumpToolOpen || die "Failed to close $dumpToolName.\n"; 991 992 $isHttpdOpen = !closeHTTPD(); 993 closeWebSocketServer(); 994 995 # Because multiple instances of this script are running concurrently we cannot 996 # safely delete this symlink. 997 # system "rm /tmp/LayoutTests"; 998 999 # FIXME: Do we really want to check the image-comparison tool for leaks every time? 1000 if ($isDiffToolOpen && $shouldCheckLeaks) { 1001 $totalLeaks += countAndPrintLeaks("ImageDiff", $imageDiffToolPID, "$testResultsDirectory/ImageDiff-leaks.txt"); 1002 } 1003 1004 if ($totalLeaks) { 1005 if ($mergeDepth) { 1006 parseLeaksandPrintUniqueLeaks(); 1007 } else { 1008 print "\nWARNING: $totalLeaks total leaks found!\n"; 1009 print "See above for individual leaks results.\n" if ($leaksOutputFileNumber > 2); 1010 } 1011 } 1012 1013 close IN; 1014 close OUT; 1015 close ERROR; 1016 1017 if ($report10Slowest) { 1018 print "\n\nThe 10 slowest tests:\n\n"; 1019 my $count = 0; 1020 for my $test (sort slowestcmp keys %durations) { 1021 printf "%0.2f secs: %s\n", $durations{$test}, $test; 1022 last if ++$count == 10; 1023 } 1024 } 1025 1026 print "\n"; 1027 1028 if ($skippedOnly && $counts{"match"}) { 1029 print "The following tests are in the Skipped file (" . File::Spec->abs2rel("$platformTestDirectory/Skipped", $testDirectory) . "), but succeeded:\n"; 1030 foreach my $test (@{$tests{"match"}}) { 1031 print " $test\n"; 1032 } 1033 } 1034 1035 if ($resetResults || ($counts{match} && $counts{match} == $count)) { 1036 print "all $count test cases succeeded\n"; 1037 unlink $testResults; 1038 exit; 1039 } 1040 1041 printResults(); 1042 1043 mkpath $testResultsDirectory; 1044 1045 open HTML, ">", $testResults or die "Failed to open $testResults. $!"; 1046 print HTML "<html>\n"; 1047 print HTML "<head>\n"; 1048 print HTML "<title>Layout Test Results</title>\n"; 1049 print HTML "</head>\n"; 1050 print HTML "<body>\n"; 1051 1052 if ($ignoreMetrics) { 1053 print HTML "<h4>Tested with metrics ignored.</h4>"; 1054 } 1055 1056 print HTML htmlForResultsSection(@{$tests{mismatch}}, "Tests where results did not match expected results", \&linksForMismatchTest); 1057 print HTML htmlForResultsSection(@{$tests{timedout}}, "Tests that timed out", \&linksForErrorTest); 1058 print HTML htmlForResultsSection(@{$tests{crash}}, "Tests that caused the DumpRenderTree tool to crash", \&linksForErrorTest); 1059 print HTML htmlForResultsSection(@{$tests{error}}, "Tests that had stderr output", \&linksForErrorTest); 1060 print HTML htmlForResultsSection(@{$tests{new}}, "Tests that had no expected results (probably new)", \&linksForNewTest); 1061 1062 print HTML "</body>\n"; 1063 print HTML "</html>\n"; 1064 close HTML; 1065 1066 my @configurationArgs = argumentsForConfiguration(); 1067 1068 if (isGtk()) { 1069 system "WebKitTools/Scripts/run-launcher", @configurationArgs, "file://".$testResults if $launchSafari; 1070 } elsif (isQt()) { 1071 unshift @configurationArgs, qw(-graphicssystem raster -style windows); 1072 if (isCygwin()) { 1073 $testResults = "/" . toWindowsPath($testResults); 1074 $testResults =~ s/\\/\//g; 1075 } 1076 system "WebKitTools/Scripts/run-launcher", @configurationArgs, "file://".$testResults if $launchSafari; 1077 } elsif (isCygwin()) { 1078 system "cygstart", $testResults if $launchSafari; 1079 } else { 1080 system "WebKitTools/Scripts/run-safari", @configurationArgs, "-NSOpen", $testResults if $launchSafari; 1081 } 1082 1083 closeCygpaths() if isCygwin(); 1084 1085 exit 1; 1086 1087 sub countAndPrintLeaks($$$) 1088 { 1089 my ($dumpToolName, $dumpToolPID, $leaksFilePath) = @_; 1090 1091 print "\n" unless $atLineStart; 1092 $atLineStart = 1; 1093 1094 # We are excluding the following reported leaks so they don't get in our way when looking for WebKit leaks: 1095 # This allows us ignore known leaks and only be alerted when new leaks occur. Some leaks are in the old 1096 # versions of the system frameworks that are being used by the leaks bots. Even though a leak has been 1097 # fixed, it will be listed here until the bot has been updated with the newer frameworks. 1098 1099 my @typesToExclude = ( 1100 ); 1101 1102 my @callStacksToExclude = ( 1103 "Flash_EnforceLocalSecurity" # leaks in Flash plug-in code, rdar://problem/4449747 1104 ); 1105 1106 if (isTiger()) { 1107 # Leak list for the version of Tiger used on the build bot. 1108 push @callStacksToExclude, ( 1109 "CFRunLoopRunSpecific \\| malloc_zone_malloc", "CFRunLoopRunSpecific \\| CFAllocatorAllocate ", # leak in CFRunLoopRunSpecific, rdar://problem/4670839 1110 "CGImageSourceGetPropertiesAtIndex", # leak in ImageIO, rdar://problem/4628809 1111 "FOGetCoveredUnicodeChars", # leak in ATS, rdar://problem/3943604 1112 "GetLineDirectionPreference", "InitUnicodeUtilities", # leaks tool falsely reporting leak in CFNotificationCenterAddObserver, rdar://problem/4964790 1113 "ICCFPrefWrapper::GetPrefDictionary", # leaks in Internet Config. code, rdar://problem/4449794 1114 "NSHTTPURLProtocol setResponseHeader:", # leak in multipart/mixed-replace handling in Foundation, no Radar, but fixed in Leopard 1115 "NSURLCache cachedResponseForRequest", # leak in CFURL cache, rdar://problem/4768430 1116 "PCFragPrepareClosureFromFile", # leak in Code Fragment Manager, rdar://problem/3426998 1117 "WebCore::Selection::toRange", # bug in 'leaks', rdar://problem/4967949 1118 "WebCore::SubresourceLoader::create", # bug in 'leaks', rdar://problem/4985806 1119 "_CFPreferencesDomainDeepCopyDictionary", # leak in CFPreferences, rdar://problem/4220786 1120 "_objc_msgForward", # leak in NSSpellChecker, rdar://problem/4965278 1121 "gldGetString", # leak in OpenGL, rdar://problem/5013699 1122 "_setDefaultUserInfoFromURL", # leak in NSHTTPAuthenticator, rdar://problem/5546453 1123 "SSLHandshake", # leak in SSL, rdar://problem/5546440 1124 "SecCertificateCreateFromData", # leak in SSL code, rdar://problem/4464397 1125 ); 1126 push @typesToExclude, ( 1127 "THRD", # bug in 'leaks', rdar://problem/3387783 1128 "DRHT", # ditto (endian little hate i) 1129 ); 1130 } 1131 1132 if (isLeopard()) { 1133 # Leak list for the version of Leopard used on the build bot. 1134 push @callStacksToExclude, ( 1135 "CFHTTPMessageAppendBytes", # leak in CFNetwork, rdar://problem/5435912 1136 "sendDidReceiveDataCallback", # leak in CFNetwork, rdar://problem/5441619 1137 "_CFHTTPReadStreamReadMark", # leak in CFNetwork, rdar://problem/5441468 1138 "httpProtocolStart", # leak in CFNetwork, rdar://problem/5468837 1139 "_CFURLConnectionSendCallbacks", # leak in CFNetwork, rdar://problem/5441600 1140 "DispatchQTMsg", # leak in QuickTime, PPC only, rdar://problem/5667132 1141 "QTMovieContentView createVisualContext", # leak in QuickTime, PPC only, rdar://problem/5667132 1142 "_CopyArchitecturesForJVMVersion", # leak in Java, rdar://problem/5910823 1143 ); 1144 } 1145 1146 if (isSnowLeopard()) { 1147 push @callStacksToExclude, ( 1148 "readMakerNoteProps", # <rdar://problem/7156432> leak in ImageIO 1149 "QTKitMovieControllerView completeUISetup", # <rdar://problem/7155156> leak in QTKit 1150 "getVMInitArgs", # <rdar://problem/7714444> leak in Java 1151 "Java_java_lang_System_initProperties", # <rdar://problem/7714465> leak in Java 1152 ); 1153 } 1154 1155 my $leaksTool = sourceDir() . "/WebKitTools/Scripts/run-leaks"; 1156 my $excludeString = "--exclude-callstack '" . (join "' --exclude-callstack '", @callStacksToExclude) . "'"; 1157 $excludeString .= " --exclude-type '" . (join "' --exclude-type '", @typesToExclude) . "'" if @typesToExclude; 1158 1159 print " ? checking for leaks in $dumpToolName\n"; 1160 my $leaksOutput = `$leaksTool $excludeString $dumpToolPID`; 1161 my ($count, $bytes) = $leaksOutput =~ /Process $dumpToolPID: (\d+) leaks? for (\d+) total/; 1162 my ($excluded) = $leaksOutput =~ /(\d+) leaks? excluded/; 1163 1164 my $adjustedCount = $count; 1165 $adjustedCount -= $excluded if $excluded; 1166 1167 if (!$adjustedCount) { 1168 print " - no leaks found\n"; 1169 unlink $leaksFilePath; 1170 return 0; 1171 } else { 1172 my $dir = $leaksFilePath; 1173 $dir =~ s|/[^/]+$|| or die; 1174 mkpath $dir; 1175 1176 if ($excluded) { 1177 print " + $adjustedCount leaks ($bytes bytes including $excluded excluded leaks) were found, details in $leaksFilePath\n"; 1178 } else { 1179 print " + $count leaks ($bytes bytes) were found, details in $leaksFilePath\n"; 1180 } 1181 1182 writeToFile($leaksFilePath, $leaksOutput); 1183 1184 push @leaksFilenames, $leaksFilePath; 1185 } 1186 1187 return $adjustedCount; 1188 } 1189 1190 sub writeToFile($$) 1191 { 1192 my ($filePath, $contents) = @_; 1193 open NEWFILE, ">", "$filePath" or die "Could not create $filePath. $!\n"; 1194 print NEWFILE $contents; 1195 close NEWFILE; 1196 } 1197 1198 # Break up a path into the directory (with slash) and base name. 1199 sub splitpath($) 1200 { 1201 my ($path) = @_; 1202 1203 my $pathSeparator = "/"; 1204 my $dirname = dirname($path) . $pathSeparator; 1205 $dirname = "" if $dirname eq "." . $pathSeparator; 1206 1207 return ($dirname, basename($path)); 1208 } 1209 1210 # Sort first by directory, then by file, so all paths in one directory are grouped 1211 # rather than being interspersed with items from subdirectories. 1212 # Use numericcmp to sort directory and filenames to make order logical. 1213 sub pathcmp($$) 1214 { 1215 my ($patha, $pathb) = @_; 1216 1217 my ($dira, $namea) = splitpath($patha); 1218 my ($dirb, $nameb) = splitpath($pathb); 1219 1220 return numericcmp($dira, $dirb) if $dira ne $dirb; 1221 return numericcmp($namea, $nameb); 1222 } 1223 1224 # Sort numeric parts of strings as numbers, other parts as strings. 1225 # Makes 1.33 come after 1.3, which is cool. 1226 sub numericcmp($$) 1227 { 1228 my ($aa, $bb) = @_; 1229 1230 my @a = split /(\d+)/, $aa; 1231 my @b = split /(\d+)/, $bb; 1232 1233 # Compare one chunk at a time. 1234 # Each chunk is either all numeric digits, or all not numeric digits. 1235 while (@a && @b) { 1236 my $a = shift @a; 1237 my $b = shift @b; 1238 1239 # Use numeric comparison if chunks are non-equal numbers. 1240 return $a <=> $b if $a =~ /^\d/ && $b =~ /^\d/ && $a != $b; 1241 1242 # Use string comparison if chunks are any other kind of non-equal string. 1243 return $a cmp $b if $a ne $b; 1244 } 1245 1246 # One of the two is now empty; compare lengths for result in this case. 1247 return @a <=> @b; 1248 } 1249 1250 # Sort slowest tests first. 1251 sub slowestcmp($$) 1252 { 1253 my ($testa, $testb) = @_; 1254 1255 my $dura = $durations{$testa}; 1256 my $durb = $durations{$testb}; 1257 return $durb <=> $dura if $dura != $durb; 1258 return pathcmp($testa, $testb); 1259 } 1260 1261 sub launchWithEnv(\@\%) 1262 { 1263 my ($args, $env) = @_; 1264 1265 # Dump the current environment as perl code and then put it in quotes so it is one parameter. 1266 my $environmentDumper = Data::Dumper->new([\%{$env}], [qw(*ENV)]); 1267 $environmentDumper->Indent(0); 1268 $environmentDumper->Purity(1); 1269 my $allEnvVars = $environmentDumper->Dump(); 1270 unshift @{$args}, "\"$allEnvVars\""; 1271 1272 my $execScript = File::Spec->catfile(sourceDir(), qw(WebKitTools Scripts execAppWithEnv)); 1273 unshift @{$args}, $execScript; 1274 return @{$args}; 1275 } 1276 1277 sub resolveAndMakeTestResultsDirectory() 1278 { 1279 my $absTestResultsDirectory = File::Spec->rel2abs(glob $testResultsDirectory); 1280 mkpath $absTestResultsDirectory; 1281 return $absTestResultsDirectory; 1282 } 1283 1284 sub openDiffTool() 1285 { 1286 return if $isDiffToolOpen; 1287 return if !$pixelTests; 1288 1289 my %CLEAN_ENV; 1290 $CLEAN_ENV{MallocStackLogging} = 1 if $shouldCheckLeaks; 1291 $imageDiffToolPID = open2(\*DIFFIN, \*DIFFOUT, $imageDiffTool, launchWithEnv(@diffToolArgs, %CLEAN_ENV)) or die "unable to open $imageDiffTool\n"; 1292 $isDiffToolOpen = 1; 1293 } 1294 1295 sub openDumpTool() 1296 { 1297 return if $isDumpToolOpen; 1298 1299 my %CLEAN_ENV; 1300 1301 # Generic environment variables 1302 if (defined $ENV{'WEBKIT_TESTFONTS'}) { 1303 $CLEAN_ENV{WEBKIT_TESTFONTS} = $ENV{'WEBKIT_TESTFONTS'}; 1304 } 1305 1306 # unique temporary directory for each DumpRendertree - needed for running more DumpRenderTree in parallel 1307 $CLEAN_ENV{DUMPRENDERTREE_TEMP} = File::Temp::tempdir('DumpRenderTree-XXXXXX', TMPDIR => 1, CLEANUP => 1); 1308 $CLEAN_ENV{XML_CATALOG_FILES} = ""; # work around missing /etc/catalog <rdar://problem/4292995> 1309 1310 # Platform spesifics 1311 if (isLinux()) { 1312 if (defined $ENV{'DISPLAY'}) { 1313 $CLEAN_ENV{DISPLAY} = $ENV{'DISPLAY'}; 1314 } else { 1315 $CLEAN_ENV{DISPLAY} = ":1"; 1316 } 1317 if (defined $ENV{'XAUTHORITY'}) { 1318 $CLEAN_ENV{XAUTHORITY} = $ENV{'XAUTHORITY'}; 1319 } 1320 1321 $CLEAN_ENV{HOME} = $ENV{'HOME'}; 1322 1323 if (defined $ENV{'LD_LIBRARY_PATH'}) { 1324 $CLEAN_ENV{LD_LIBRARY_PATH} = $ENV{'LD_LIBRARY_PATH'}; 1325 } 1326 if (defined $ENV{'DBUS_SESSION_BUS_ADDRESS'}) { 1327 $CLEAN_ENV{DBUS_SESSION_BUS_ADDRESS} = $ENV{'DBUS_SESSION_BUS_ADDRESS'}; 1328 } 1329 } elsif (isDarwin()) { 1330 if (defined $ENV{'DYLD_LIBRARY_PATH'}) { 1331 $CLEAN_ENV{DYLD_LIBRARY_PATH} = $ENV{'DYLD_LIBRARY_PATH'}; 1332 } 1333 1334 $CLEAN_ENV{DYLD_FRAMEWORK_PATH} = $productDir; 1335 $CLEAN_ENV{DYLD_INSERT_LIBRARIES} = "/usr/lib/libgmalloc.dylib" if $guardMalloc; 1336 } elsif (isCygwin()) { 1337 $CLEAN_ENV{HOMEDRIVE} = $ENV{'HOMEDRIVE'}; 1338 $CLEAN_ENV{HOMEPATH} = $ENV{'HOMEPATH'}; 1339 1340 setPathForRunningWebKitApp(\%CLEAN_ENV); 1341 } 1342 1343 # Port spesifics 1344 if (isQt()) { 1345 $CLEAN_ENV{QTWEBKIT_PLUGIN_PATH} = productDir() . "/lib/plugins"; 1346 } 1347 1348 my @args = ($dumpTool, @toolArgs); 1349 if (isAppleMacWebKit() and !isTiger()) { 1350 unshift @args, "arch", "-" . architecture(); 1351 } 1352 1353 if ($useValgrind) { 1354 unshift @args, "valgrind", "--suppressions=$platformBaseDirectory/qt/SuppressedValgrindErrors"; 1355 } 1356 1357 $CLEAN_ENV{MallocStackLogging} = 1 if $shouldCheckLeaks; 1358 1359 $dumpToolPID = open3(\*OUT, \*IN, \*ERROR, launchWithEnv(@args, %CLEAN_ENV)) or die "Failed to start tool: $dumpTool\n"; 1360 $isDumpToolOpen = 1; 1361 $dumpToolCrashed = 0; 1362 } 1363 1364 sub closeDumpTool() 1365 { 1366 return if !$isDumpToolOpen; 1367 1368 close IN; 1369 close OUT; 1370 waitpid $dumpToolPID, 0; 1371 1372 # check for WebCore counter leaks. 1373 if ($shouldCheckLeaks) { 1374 while (<ERROR>) { 1375 print; 1376 } 1377 } 1378 close ERROR; 1379 $isDumpToolOpen = 0; 1380 } 1381 1382 sub dumpToolDidCrash() 1383 { 1384 return 1 if $dumpToolCrashed; 1385 return 0 unless $isDumpToolOpen; 1386 my $pid = waitpid(-1, WNOHANG); 1387 return 1 if ($pid == $dumpToolPID); 1388 1389 # On Mac OS X, crashing may be significantly delayed by crash reporter. 1390 return 0 unless isAppleMacWebKit(); 1391 1392 return DumpRenderTreeSupport::processIsCrashing($dumpToolPID); 1393 } 1394 1395 sub configureAndOpenHTTPDIfNeeded() 1396 { 1397 return if $isHttpdOpen; 1398 my $absTestResultsDirectory = resolveAndMakeTestResultsDirectory(); 1399 my $listen = "127.0.0.1:$httpdPort"; 1400 my @args = ( 1401 "-c", "CustomLog \"$absTestResultsDirectory/access_log.txt\" common", 1402 "-c", "ErrorLog \"$absTestResultsDirectory/error_log.txt\"", 1403 "-C", "Listen $listen" 1404 ); 1405 1406 my @defaultArgs = getDefaultConfigForTestDirectory($testDirectory); 1407 @args = (@defaultArgs, @args); 1408 1409 waitForHTTPDLock() if $shouldWaitForHTTPD; 1410 $isHttpdOpen = openHTTPD(@args); 1411 } 1412 1413 sub openWebSocketServerIfNeeded() 1414 { 1415 return 1 if $isWebSocketServerOpen; 1416 return 0 if $failedToStartWebSocketServer; 1417 1418 my $webSocketServerPath = "/usr/bin/python"; 1419 my $webSocketPythonPath = "WebKitTools/Scripts/webkitpy/thirdparty/pywebsocket"; 1420 my $webSocketHandlerDir = "$testDirectory"; 1421 my $webSocketHandlerScanDir = "$testDirectory/websocket/tests"; 1422 my $webSocketHandlerMapFile = "$webSocketHandlerScanDir/handler_map.txt"; 1423 my $sslCertificate = "$testDirectory/http/conf/webkit-httpd.pem"; 1424 my $absTestResultsDirectory = resolveAndMakeTestResultsDirectory(); 1425 my $logFile = "$absTestResultsDirectory/pywebsocket_log.txt"; 1426 1427 my @args = ( 1428 "WebKitTools/Scripts/webkitpy/thirdparty/pywebsocket/mod_pywebsocket/standalone.py", 1429 "--server-host", "127.0.0.1", 1430 "--port", "$webSocketPort", 1431 "--document-root", "$webSocketHandlerDir", 1432 "--scan-dir", "$webSocketHandlerScanDir", 1433 "--websock-handlers-map-file", "$webSocketHandlerMapFile", 1434 "--cgi-paths", "/websocket/tests", 1435 "--log-file", "$logFile", 1436 "--strict", 1437 ); 1438 # wss is disabled until all platforms support pyOpenSSL. 1439 # my @argsSecure = ( 1440 # "WebKitTools/Scripts/webkitpy/thirdparty/pywebsocket/mod_pywebsocket/standalone.py", 1441 # "-p", "$webSocketSecurePort", 1442 # "-d", "$webSocketHandlerDir", 1443 # "-t", 1444 # "-k", "$sslCertificate", 1445 # "-c", "$sslCertificate", 1446 # ); 1447 1448 $ENV{"PYTHONPATH"} = $webSocketPythonPath; 1449 $webSocketServerPID = open3(\*WEBSOCKETSERVER_IN, \*WEBSOCKETSERVER_OUT, \*WEBSOCKETSERVER_ERR, $webSocketServerPath, @args); 1450 # wss is disabled until all platforms support pyOpenSSL. 1451 # $webSocketSecureServerPID = open3(\*WEBSOCKETSECURESERVER_IN, \*WEBSOCKETSECURESERVER_OUT, \*WEBSOCKETSECURESERVER_ERR, $webSocketServerPath, @argsSecure); 1452 # my @listen = ("http://127.0.0.1:$webSocketPort", "https://127.0.0.1:$webSocketSecurePort"); 1453 my @listen = ("http://127.0.0.1:$webSocketPort"); 1454 for (my $i = 0; $i < @listen; $i++) { 1455 my $retryCount = 10; 1456 while (system("/usr/bin/curl -k -q --silent --stderr - --output /dev/null $listen[$i]") && $retryCount) { 1457 sleep 1; 1458 --$retryCount; 1459 } 1460 unless ($retryCount) { 1461 print STDERR "Timed out waiting for WebSocketServer to start.\n"; 1462 $failedToStartWebSocketServer = 1; 1463 return 0; 1464 } 1465 } 1466 1467 $isWebSocketServerOpen = 1; 1468 return 1; 1469 } 1470 1471 sub closeWebSocketServer() 1472 { 1473 return if !$isWebSocketServerOpen; 1474 1475 close WEBSOCKETSERVER_IN; 1476 close WEBSOCKETSERVER_OUT; 1477 close WEBSOCKETSERVER_ERR; 1478 kill 15, $webSocketServerPID; 1479 1480 # wss is disabled until all platforms support pyOpenSSL. 1481 # close WEBSOCKETSECURESERVER_IN; 1482 # close WEBSOCKETSECURESERVER_OUT; 1483 # close WEBSOCKETSECURESERVER_ERR; 1484 # kill 15, $webSocketSecureServerPID; 1485 1486 $isWebSocketServerOpen = 0; 1487 } 1488 1489 sub fileNameWithNumber($$) 1490 { 1491 my ($base, $number) = @_; 1492 return "$base$number" if ($number > 1); 1493 return $base; 1494 } 1495 1496 sub processIgnoreTests($$) 1497 { 1498 my @ignoreList = split(/\s*,\s*/, shift); 1499 my $listName = shift; 1500 1501 my $disabledSuffix = "-disabled"; 1502 1503 my $addIgnoredDirectories = sub { 1504 return () if exists $ignoredLocalDirectories{basename($File::Find::dir)}; 1505 $ignoredDirectories{File::Spec->abs2rel($File::Find::dir, $testDirectory)} = 1; 1506 return @_; 1507 }; 1508 foreach my $item (@ignoreList) { 1509 my $path = catfile($testDirectory, $item); 1510 if (-d $path) { 1511 $ignoredDirectories{$item} = 1; 1512 find({ preprocess => $addIgnoredDirectories, wanted => sub {} }, $path); 1513 } 1514 elsif (-f $path) { 1515 $ignoredFiles{$item} = 1; 1516 } elsif (-f $path . $disabledSuffix) { 1517 # The test is disabled, so do nothing. 1518 } else { 1519 print "$listName list contained '$item', but no file of that name could be found\n"; 1520 } 1521 } 1522 } 1523 1524 sub stripExtension($) 1525 { 1526 my ($test) = @_; 1527 1528 $test =~ s/\.[a-zA-Z]+$//; 1529 return $test; 1530 } 1531 1532 sub isTextOnlyTest($) 1533 { 1534 my ($actual) = @_; 1535 my $isText; 1536 if ($actual =~ /^layer at/ms) { 1537 $isText = 0; 1538 } else { 1539 $isText = 1; 1540 } 1541 return $isText; 1542 } 1543 1544 sub expectedDirectoryForTest($;$;$) 1545 { 1546 my ($base, $isText, $expectedExtension) = @_; 1547 1548 my @directories = @platformResultHierarchy; 1549 push @directories, map { catdir($platformBaseDirectory, $_) } qw(mac-snowleopard mac) if isCygwin(); 1550 push @directories, $expectedDirectory; 1551 1552 # If we already have expected results, just return their location. 1553 foreach my $directory (@directories) { 1554 return $directory if (-f "$directory/$base-$expectedTag.$expectedExtension"); 1555 } 1556 1557 # For cross-platform tests, text-only results should go in the cross-platform directory, 1558 # while render tree dumps should go in the least-specific platform directory. 1559 return $isText ? $expectedDirectory : $platformResultHierarchy[$#platformResultHierarchy]; 1560 } 1561 1562 sub countFinishedTest($$$$) 1563 { 1564 my ($test, $base, $result, $isText) = @_; 1565 1566 if (($count + 1) % $testsPerDumpTool == 0 || $count == $#tests) { 1567 if ($shouldCheckLeaks) { 1568 my $fileName; 1569 if ($testsPerDumpTool == 1) { 1570 $fileName = "$testResultsDirectory/$base-leaks.txt"; 1571 } else { 1572 $fileName = "$testResultsDirectory/" . fileNameWithNumber($dumpToolName, $leaksOutputFileNumber) . "-leaks.txt"; 1573 } 1574 my $leakCount = countAndPrintLeaks($dumpToolName, $dumpToolPID, $fileName); 1575 $totalLeaks += $leakCount; 1576 $leaksOutputFileNumber++ if ($leakCount); 1577 } 1578 1579 closeDumpTool(); 1580 } 1581 1582 $count++; 1583 $counts{$result}++; 1584 push @{$tests{$result}}, $test; 1585 } 1586 1587 sub testCrashedOrTimedOut($$$$$) 1588 { 1589 my ($test, $base, $didCrash, $actual, $error) = @_; 1590 1591 printFailureMessageForTest($test, $didCrash ? "crashed" : "timed out"); 1592 1593 sampleDumpTool() unless $didCrash; 1594 1595 my $dir = "$testResultsDirectory/$base"; 1596 $dir =~ s|/([^/]+)$|| or die "Failed to find test name from base\n"; 1597 mkpath $dir; 1598 1599 deleteExpectedAndActualResults($base); 1600 1601 if (defined($error) && length($error)) { 1602 writeToFile("$testResultsDirectory/$base-$errorTag.txt", $error); 1603 } 1604 1605 recordActualResultsAndDiff($base, $actual); 1606 1607 kill 9, $dumpToolPID unless $didCrash; 1608 1609 closeDumpTool(); 1610 } 1611 1612 sub printFailureMessageForTest($$) 1613 { 1614 my ($test, $description) = @_; 1615 1616 unless ($verbose) { 1617 print "\n" unless $atLineStart; 1618 print "$test -> "; 1619 } 1620 print "$description\n"; 1621 $atLineStart = 1; 1622 } 1623 1624 my %cygpaths = (); 1625 1626 sub openCygpathIfNeeded($) 1627 { 1628 my ($options) = @_; 1629 1630 return unless isCygwin(); 1631 return $cygpaths{$options} if $cygpaths{$options} && $cygpaths{$options}->{"open"}; 1632 1633 local (*CYGPATHIN, *CYGPATHOUT); 1634 my $pid = open2(\*CYGPATHIN, \*CYGPATHOUT, "cygpath -f - $options"); 1635 my $cygpath = { 1636 "pid" => $pid, 1637 "in" => *CYGPATHIN, 1638 "out" => *CYGPATHOUT, 1639 "open" => 1 1640 }; 1641 1642 $cygpaths{$options} = $cygpath; 1643 1644 return $cygpath; 1645 } 1646 1647 sub closeCygpaths() 1648 { 1649 return unless isCygwin(); 1650 1651 foreach my $cygpath (values(%cygpaths)) { 1652 close $cygpath->{"in"}; 1653 close $cygpath->{"out"}; 1654 waitpid($cygpath->{"pid"}, 0); 1655 $cygpath->{"open"} = 0; 1656 1657 } 1658 } 1659 1660 sub convertPathUsingCygpath($$) 1661 { 1662 my ($path, $options) = @_; 1663 1664 # cygpath -f (at least in Cygwin 1.7) converts spaces into newlines. We remove spaces here and 1665 # add them back in after conversion to work around this. 1666 my $spaceSubstitute = "__NOTASPACE__"; 1667 $path =~ s/ /\Q$spaceSubstitute\E/g; 1668 1669 my $cygpath = openCygpathIfNeeded($options); 1670 local *inFH = $cygpath->{"in"}; 1671 local *outFH = $cygpath->{"out"}; 1672 print outFH $path . "\n"; 1673 my $convertedPath = <inFH>; 1674 chomp($convertedPath) if defined $convertedPath; 1675 1676 $convertedPath =~ s/\Q$spaceSubstitute\E/ /g; 1677 return $convertedPath; 1678 } 1679 1680 sub toWindowsPath($) 1681 { 1682 my ($path) = @_; 1683 return unless isCygwin(); 1684 1685 return convertPathUsingCygpath($path, "-w"); 1686 } 1687 1688 sub toURL($) 1689 { 1690 my ($path) = @_; 1691 1692 if ($useRemoteLinksToTests) { 1693 my $relativePath = File::Spec->abs2rel($path, $testDirectory); 1694 1695 # If the file is below the test directory then convert it into a link to the file in SVN 1696 if ($relativePath !~ /^\.\.\//) { 1697 my $revision = svnRevisionForDirectory($testDirectory); 1698 my $svnPath = pathRelativeToSVNRepositoryRootForPath($path); 1699 return "http://trac.webkit.org/export/$revision/$svnPath"; 1700 } 1701 } 1702 1703 return $path unless isCygwin(); 1704 1705 return "file:///" . convertPathUsingCygpath($path, "-m"); 1706 } 1707 1708 sub validateSkippedArg($$;$) 1709 { 1710 my ($option, $value, $value2) = @_; 1711 my %validSkippedValues = map { $_ => 1 } qw(default ignore only); 1712 $value = lc($value); 1713 die "Invalid argument '" . $value . "' for option $option" unless $validSkippedValues{$value}; 1714 $treatSkipped = $value; 1715 } 1716 1717 sub htmlForResultsSection(\@$&) 1718 { 1719 my ($tests, $description, $linkGetter) = @_; 1720 1721 my @html = (); 1722 return join("\n", @html) unless @{$tests}; 1723 1724 push @html, "<p>$description:</p>"; 1725 push @html, "<table>"; 1726 foreach my $test (@{$tests}) { 1727 push @html, "<tr>"; 1728 push @html, "<td><a href=\"" . toURL("$testDirectory/$test") . "\">$test</a></td>"; 1729 foreach my $link (@{&{$linkGetter}($test)}) { 1730 push @html, "<td><a href=\"$link->{href}\">$link->{text}</a></td>"; 1731 } 1732 push @html, "</tr>"; 1733 } 1734 push @html, "</table>"; 1735 1736 return join("\n", @html); 1737 } 1738 1739 sub linksForExpectedAndActualResults($) 1740 { 1741 my ($base) = @_; 1742 1743 my @links = (); 1744 1745 return \@links unless -s "$testResultsDirectory/$base-$diffsTag.txt"; 1746 1747 my $expectedResultPath = $expectedResultPaths{$base}; 1748 my ($expectedResultFileName, $expectedResultsDirectory, $expectedResultExtension) = fileparse($expectedResultPath, qr{\.[^.]+$}); 1749 1750 push @links, { href => "$base-$expectedTag$expectedResultExtension", text => "expected" }; 1751 push @links, { href => "$base-$actualTag$expectedResultExtension", text => "actual" }; 1752 push @links, { href => "$base-$diffsTag.txt", text => "diff" }; 1753 push @links, { href => "$base-$prettyDiffTag.html", text => "pretty diff" }; 1754 1755 return \@links; 1756 } 1757 1758 sub linksForMismatchTest 1759 { 1760 my ($test) = @_; 1761 1762 my @links = (); 1763 1764 my $base = stripExtension($test); 1765 1766 push @links, @{linksForExpectedAndActualResults($base)}; 1767 return \@links unless $pixelTests && $imagesPresent{$base}; 1768 1769 push @links, { href => "$base-$expectedTag.png", text => "expected image" }; 1770 push @links, { href => "$base-$diffsTag.html", text => "image diffs" }; 1771 push @links, { href => "$base-$diffsTag.png", text => "$imageDifferences{$base}%" }; 1772 1773 return \@links; 1774 } 1775 1776 sub linksForErrorTest 1777 { 1778 my ($test) = @_; 1779 1780 my @links = (); 1781 1782 my $base = stripExtension($test); 1783 1784 push @links, @{linksForExpectedAndActualResults($base)}; 1785 push @links, { href => "$base-$errorTag.txt", text => "stderr" }; 1786 1787 return \@links; 1788 } 1789 1790 sub linksForNewTest 1791 { 1792 my ($test) = @_; 1793 1794 my @links = (); 1795 1796 my $base = stripExtension($test); 1797 1798 my $expectedResultPath = $expectedResultPaths{$base}; 1799 my ($expectedResultFileName, $expectedResultsDirectory, $expectedResultExtension) = fileparse($expectedResultPath, qr{\.[^.]+$}); 1800 1801 push @links, { href => "$base-$actualTag$expectedResultExtension", text => "result" }; 1802 if ($pixelTests && $imagesPresent{$base}) { 1803 push @links, { href => "$base-$expectedTag.png", text => "image" }; 1804 } 1805 1806 return \@links; 1807 } 1808 1809 sub deleteExpectedAndActualResults($) 1810 { 1811 my ($base) = @_; 1812 1813 unlink "$testResultsDirectory/$base-$actualTag.txt"; 1814 unlink "$testResultsDirectory/$base-$diffsTag.txt"; 1815 unlink "$testResultsDirectory/$base-$errorTag.txt"; 1816 } 1817 1818 sub recordActualResultsAndDiff($$) 1819 { 1820 my ($base, $actualResults) = @_; 1821 1822 return unless defined($actualResults) && length($actualResults); 1823 1824 my $expectedResultPath = $expectedResultPaths{$base}; 1825 my ($expectedResultFileNameMinusExtension, $expectedResultDirectoryPath, $expectedResultExtension) = fileparse($expectedResultPath, qr{\.[^.]+$}); 1826 my $actualResultsPath = "$testResultsDirectory/$base-$actualTag$expectedResultExtension"; 1827 my $copiedExpectedResultsPath = "$testResultsDirectory/$base-$expectedTag$expectedResultExtension"; 1828 1829 mkpath(dirname($actualResultsPath)); 1830 writeToFile("$actualResultsPath", $actualResults); 1831 1832 if (-f $expectedResultPath) { 1833 copy("$expectedResultPath", "$copiedExpectedResultsPath"); 1834 } else { 1835 open EMPTY, ">$copiedExpectedResultsPath"; 1836 close EMPTY; 1837 } 1838 1839 my $diffOuputBasePath = "$testResultsDirectory/$base"; 1840 my $diffOutputPath = "$diffOuputBasePath-$diffsTag.txt"; 1841 system "diff -u \"$copiedExpectedResultsPath\" \"$actualResultsPath\" > \"$diffOutputPath\""; 1842 1843 my $prettyDiffOutputPath = "$diffOuputBasePath-$prettyDiffTag.html"; 1844 my $prettyPatchPath = "BugsSite/PrettyPatch/"; 1845 my $prettifyPath = "$prettyPatchPath/prettify.rb"; 1846 system "ruby -I \"$prettyPatchPath\" \"$prettifyPath\" \"$diffOutputPath\" > \"$prettyDiffOutputPath\""; 1847 } 1848 1849 sub buildPlatformResultHierarchy() 1850 { 1851 mkpath($platformTestDirectory) if ($platform eq "undefined" && !-d "$platformTestDirectory"); 1852 1853 my @platforms; 1854 if ($platform =~ /^mac-/) { 1855 my $i; 1856 for ($i = 0; $i < @macPlatforms; $i++) { 1857 last if $macPlatforms[$i] eq $platform; 1858 } 1859 for (; $i < @macPlatforms; $i++) { 1860 push @platforms, $macPlatforms[$i]; 1861 } 1862 } elsif ($platform =~ /^qt-/) { 1863 push @platforms, $platform; 1864 push @platforms, "qt"; 1865 } else { 1866 @platforms = $platform; 1867 } 1868 1869 my @hierarchy; 1870 for (my $i = 0; $i < @platforms; $i++) { 1871 my $scoped = catdir($platformBaseDirectory, $platforms[$i]); 1872 push(@hierarchy, $scoped) if (-d $scoped); 1873 } 1874 1875 return @hierarchy; 1876 } 1877 1878 sub buildPlatformTestHierarchy(@) 1879 { 1880 my (@platformHierarchy) = @_; 1881 return @platformHierarchy if (@platformHierarchy < 2); 1882 1883 return ($platformHierarchy[0], $platformHierarchy[$#platformHierarchy]); 1884 } 1885 1886 sub epiloguesAndPrologues($$) 1887 { 1888 my ($lastDirectory, $directory) = @_; 1889 my @lastComponents = split('/', $lastDirectory); 1890 my @components = split('/', $directory); 1891 1892 while (@lastComponents) { 1893 if (!defined($components[0]) || $lastComponents[0] ne $components[0]) { 1894 last; 1895 } 1896 shift @components; 1897 shift @lastComponents; 1898 } 1899 1900 my @result; 1901 my $leaving = $lastDirectory; 1902 foreach (@lastComponents) { 1903 my $epilogue = $leaving . "/resources/run-webkit-tests-epilogue.html"; 1904 foreach (@platformResultHierarchy) { 1905 push @result, catdir($_, $epilogue) if (stat(catdir($_, $epilogue))); 1906 } 1907 push @result, catdir($testDirectory, $epilogue) if (stat(catdir($testDirectory, $epilogue))); 1908 $leaving =~ s|(^\|/)[^/]+$||; 1909 } 1910 1911 my $entering = $leaving; 1912 foreach (@components) { 1913 $entering .= '/' . $_; 1914 my $prologue = $entering . "/resources/run-webkit-tests-prologue.html"; 1915 push @result, catdir($testDirectory, $prologue) if (stat(catdir($testDirectory, $prologue))); 1916 foreach (reverse @platformResultHierarchy) { 1917 push @result, catdir($_, $prologue) if (stat(catdir($_, $prologue))); 1918 } 1919 } 1920 return @result; 1921 } 1922 1923 sub parseLeaksandPrintUniqueLeaks() 1924 { 1925 return unless @leaksFilenames; 1926 1927 my $mergedFilenames = join " ", @leaksFilenames; 1928 my $parseMallocHistoryTool = sourceDir() . "/WebKitTools/Scripts/parse-malloc-history"; 1929 1930 open MERGED_LEAKS, "cat $mergedFilenames | $parseMallocHistoryTool --merge-depth $mergeDepth - |" ; 1931 my @leakLines = <MERGED_LEAKS>; 1932 close MERGED_LEAKS; 1933 1934 my $uniqueLeakCount = 0; 1935 my $totalBytes; 1936 foreach my $line (@leakLines) { 1937 ++$uniqueLeakCount if ($line =~ /^(\d*)\scalls/); 1938 $totalBytes = $1 if $line =~ /^total\:\s(.*)\s\(/; 1939 } 1940 1941 print "\nWARNING: $totalLeaks total leaks found for a total of $totalBytes!\n"; 1942 print "WARNING: $uniqueLeakCount unique leaks found!\n"; 1943 print "See above for individual leaks results.\n" if ($leaksOutputFileNumber > 2); 1944 1945 } 1946 1947 sub extensionForMimeType($) 1948 { 1949 my ($mimeType) = @_; 1950 1951 if ($mimeType eq "application/x-webarchive") { 1952 return "webarchive"; 1953 } elsif ($mimeType eq "application/pdf") { 1954 return "pdf"; 1955 } 1956 return "txt"; 1957 } 1958 1959 # Read up to the first #EOF (the content block of the test), or until detecting crashes or timeouts. 1960 sub readFromDumpToolWithTimer(**) 1961 { 1962 my ($fhIn, $fhError) = @_; 1963 1964 setFileHandleNonBlocking($fhIn, 1); 1965 setFileHandleNonBlocking($fhError, 1); 1966 1967 my $maximumSecondsWithoutOutput = $timeoutSeconds; 1968 $maximumSecondsWithoutOutput *= 10 if $guardMalloc; 1969 my $microsecondsToWaitBeforeReadingAgain = 1000; 1970 1971 my $timeOfLastSuccessfulRead = time; 1972 1973 my @output = (); 1974 my @error = (); 1975 my $status = "success"; 1976 my $mimeType = "text/plain"; 1977 # We don't have a very good way to know when the "headers" stop 1978 # and the content starts, so we use this as a hack: 1979 my $haveSeenContentType = 0; 1980 my $haveSeenEofIn = 0; 1981 my $haveSeenEofError = 0; 1982 1983 while (1) { 1984 if (time - $timeOfLastSuccessfulRead > $maximumSecondsWithoutOutput) { 1985 $status = dumpToolDidCrash() ? "crashed" : "timedOut"; 1986 last; 1987 } 1988 1989 # Once we've seen the EOF, we must not read anymore. 1990 my $lineIn = readline($fhIn) unless $haveSeenEofIn; 1991 my $lineError = readline($fhError) unless $haveSeenEofError; 1992 if (!defined($lineIn) && !defined($lineError)) { 1993 last if ($haveSeenEofIn && $haveSeenEofError); 1994 1995 if ($! != EAGAIN) { 1996 $status = "crashed"; 1997 last; 1998 } 1999 2000 # No data ready 2001 usleep($microsecondsToWaitBeforeReadingAgain); 2002 next; 2003 } 2004 2005 $timeOfLastSuccessfulRead = time; 2006 2007 if (defined($lineIn)) { 2008 if (!$haveSeenContentType && $lineIn =~ /^Content-Type: (\S+)$/) { 2009 $mimeType = $1; 2010 $haveSeenContentType = 1; 2011 } elsif ($lineIn =~ /#EOF/) { 2012 $haveSeenEofIn = 1; 2013 } else { 2014 push @output, $lineIn; 2015 } 2016 } 2017 if (defined($lineError)) { 2018 if ($lineError =~ /#EOF/) { 2019 $haveSeenEofError = 1; 2020 } else { 2021 push @error, $lineError; 2022 } 2023 } 2024 } 2025 2026 setFileHandleNonBlocking($fhIn, 0); 2027 setFileHandleNonBlocking($fhError, 0); 2028 return { 2029 output => join("", @output), 2030 error => join("", @error), 2031 status => $status, 2032 mimeType => $mimeType, 2033 extension => extensionForMimeType($mimeType) 2034 }; 2035 } 2036 2037 sub setFileHandleNonBlocking(*$) 2038 { 2039 my ($fh, $nonBlocking) = @_; 2040 2041 my $flags = fcntl($fh, F_GETFL, 0) or die "Couldn't get filehandle flags"; 2042 2043 if ($nonBlocking) { 2044 $flags |= O_NONBLOCK; 2045 } else { 2046 $flags &= ~O_NONBLOCK; 2047 } 2048 2049 fcntl($fh, F_SETFL, $flags) or die "Couldn't set filehandle flags"; 2050 2051 return 1; 2052 } 2053 2054 sub sampleDumpTool() 2055 { 2056 return unless isAppleMacWebKit(); 2057 return unless $runSample; 2058 2059 my $outputDirectory = "$ENV{HOME}/Library/Logs/DumpRenderTree"; 2060 -d $outputDirectory or mkdir $outputDirectory; 2061 2062 my $outputFile = "$outputDirectory/HangReport.txt"; 2063 system "/usr/bin/sample", $dumpToolPID, qw(10 10 -file), $outputFile; 2064 } 2065 2066 sub stripMetrics($$) 2067 { 2068 my ($actual, $expected) = @_; 2069 2070 foreach my $result ($actual, $expected) { 2071 $result =~ s/at \(-?[0-9]+,-?[0-9]+\) *//g; 2072 $result =~ s/size -?[0-9]+x-?[0-9]+ *//g; 2073 $result =~ s/text run width -?[0-9]+: //g; 2074 $result =~ s/text run width -?[0-9]+ [a-zA-Z ]+: //g; 2075 $result =~ s/RenderButton {BUTTON} .*/RenderButton {BUTTON}/g; 2076 $result =~ s/RenderImage {INPUT} .*/RenderImage {INPUT}/g; 2077 $result =~ s/RenderBlock {INPUT} .*/RenderBlock {INPUT}/g; 2078 $result =~ s/RenderTextControl {INPUT} .*/RenderTextControl {INPUT}/g; 2079 $result =~ s/\([0-9]+px/px/g; 2080 $result =~ s/ *" *\n +" */ /g; 2081 $result =~ s/" +$/"/g; 2082 2083 $result =~ s/- /-/g; 2084 $result =~ s/\n( *)"\s+/\n$1"/g; 2085 $result =~ s/\s+"\n/"\n/g; 2086 $result =~ s/scrollWidth [0-9]+/scrollWidth/g; 2087 $result =~ s/scrollHeight [0-9]+/scrollHeight/g; 2088 } 2089 2090 return ($actual, $expected); 2091 } 2092 2093 sub fileShouldBeIgnored 2094 { 2095 my ($filePath) = @_; 2096 foreach my $ignoredDir (keys %ignoredDirectories) { 2097 if ($filePath =~ m/^$ignoredDir/) { 2098 return 1; 2099 } 2100 } 2101 return 0; 2102 } 2103 2104 sub readSkippedFiles($) 2105 { 2106 my ($constraintPath) = @_; 2107 2108 foreach my $level (@platformTestHierarchy) { 2109 if (open SKIPPED, "<", "$level/Skipped") { 2110 if ($verbose) { 2111 my ($dir, $name) = splitpath($level); 2112 print "Skipped tests in $name:\n"; 2113 } 2114 2115 while (<SKIPPED>) { 2116 my $skipped = $_; 2117 chomp $skipped; 2118 $skipped =~ s/^[ \n\r]+//; 2119 $skipped =~ s/[ \n\r]+$//; 2120 if ($skipped && $skipped !~ /^#/) { 2121 if ($skippedOnly) { 2122 if (!fileShouldBeIgnored($skipped)) { 2123 if (!$constraintPath) { 2124 # Always add $skipped since no constraint path was specified on the command line. 2125 push(@ARGV, $skipped); 2126 } elsif ($skipped =~ /^($constraintPath)/) { 2127 # Add $skipped only if it matches the current path constraint, e.g., 2128 # "--skipped=only dir1" with "dir1/file1.html" on the skipped list. 2129 push(@ARGV, $skipped); 2130 } elsif ($constraintPath =~ /^($skipped)/) { 2131 # Add current path constraint if it is more specific than the skip list entry, 2132 # e.g., "--skipped=only dir1/dir2/dir3" with "dir1" on the skipped list. 2133 push(@ARGV, $constraintPath); 2134 } 2135 } elsif ($verbose) { 2136 print " $skipped\n"; 2137 } 2138 } else { 2139 if ($verbose) { 2140 print " $skipped\n"; 2141 } 2142 processIgnoreTests($skipped, "Skipped"); 2143 } 2144 } 2145 } 2146 close SKIPPED; 2147 } 2148 } 2149 } 2150 2151 my @testsFound; 2152 2153 sub directoryFilter 2154 { 2155 return () if exists $ignoredLocalDirectories{basename($File::Find::dir)}; 2156 return () if exists $ignoredDirectories{File::Spec->abs2rel($File::Find::dir, $testDirectory)}; 2157 return @_; 2158 } 2159 2160 sub fileFilter 2161 { 2162 my $filename = $_; 2163 if ($filename =~ /\.([^.]+)$/) { 2164 if (exists $supportedFileExtensions{$1}) { 2165 my $path = File::Spec->abs2rel(catfile($File::Find::dir, $filename), $testDirectory); 2166 push @testsFound, $path if !exists $ignoredFiles{$path}; 2167 } 2168 } 2169 } 2170 2171 sub findTestsToRun 2172 { 2173 my @testsToRun = (); 2174 2175 for my $test (@ARGV) { 2176 $test =~ s/^($layoutTestsName|$testDirectory)\///; 2177 my $fullPath = catfile($testDirectory, $test); 2178 if (file_name_is_absolute($test)) { 2179 print "can't run test $test outside $testDirectory\n"; 2180 } elsif (-f $fullPath) { 2181 my ($filename, $pathname, $fileExtension) = fileparse($test, qr{\.[^.]+$}); 2182 if (!exists $supportedFileExtensions{substr($fileExtension, 1)}) { 2183 print "test $test does not have a supported extension\n"; 2184 } elsif ($testHTTP || $pathname !~ /^http\//) { 2185 push @testsToRun, $test; 2186 } 2187 } elsif (-d $fullPath) { 2188 @testsFound = (); 2189 find({ preprocess => \&directoryFilter, wanted => \&fileFilter }, $fullPath); 2190 for my $level (@platformTestHierarchy) { 2191 my $platformPath = catfile($level, $test); 2192 find({ preprocess => \&directoryFilter, wanted => \&fileFilter }, $platformPath) if (-d $platformPath); 2193 } 2194 push @testsToRun, sort pathcmp @testsFound; 2195 @testsFound = (); 2196 } else { 2197 print "test $test not found\n"; 2198 } 2199 } 2200 2201 if (!scalar @ARGV) { 2202 @testsFound = (); 2203 find({ preprocess => \&directoryFilter, wanted => \&fileFilter }, $testDirectory); 2204 for my $level (@platformTestHierarchy) { 2205 find({ preprocess => \&directoryFilter, wanted => \&fileFilter }, $level); 2206 } 2207 push @testsToRun, sort pathcmp @testsFound; 2208 @testsFound = (); 2209 2210 # We need to minimize the time when Apache and WebSocketServer is locked by tests 2211 # so run them last if no explicit order was specified in the argument list. 2212 my @httpTests; 2213 my @websocketTests; 2214 my @otherTests; 2215 foreach my $test (@testsToRun) { 2216 if ($test =~ /^http\//) { 2217 push(@httpTests, $test); 2218 } elsif ($test =~ /^websocket\//) { 2219 push(@websocketTests, $test); 2220 } else { 2221 push(@otherTests, $test); 2222 } 2223 } 2224 @testsToRun = (@otherTests, @httpTests, @websocketTests); 2225 } 2226 2227 # Reverse the tests 2228 @testsToRun = reverse @testsToRun if $reverseTests; 2229 2230 # Shuffle the array 2231 @testsToRun = shuffle(@testsToRun) if $randomizeTests; 2232 2233 return @testsToRun; 2234 } 2235 2236 sub printResults 2237 { 2238 my %text = ( 2239 match => "succeeded", 2240 mismatch => "had incorrect layout", 2241 new => "were new", 2242 timedout => "timed out", 2243 crash => "crashed", 2244 error => "had stderr output" 2245 ); 2246 2247 for my $type ("match", "mismatch", "new", "timedout", "crash", "error") { 2248 my $typeCount = $counts{$type}; 2249 next unless $typeCount; 2250 my $typeText = $text{$type}; 2251 my $message; 2252 if ($typeCount == 1) { 2253 $typeText =~ s/were/was/; 2254 $message = sprintf "1 test case (%d%%) %s\n", 1 * 100 / $count, $typeText; 2255 } else { 2256 $message = sprintf "%d test cases (%d%%) %s\n", $typeCount, $typeCount * 100 / $count, $typeText; 2257 } 2258 $message =~ s-\(0%\)-(<1%)-; 2259 print $message; 2260 } 2261 } 2262 2263 sub stopRunningTestsEarlyIfNeeded() 2264 { 2265 # --reset-results does not check pass vs. fail, so exitAfterNFailures makes no sense with --reset-results. 2266 return 0 if !$exitAfterNFailures || $resetResults; 2267 2268 my $passCount = $counts{match} || 0; # $counts{match} will be undefined if we've not yet passed a test (e.g. the first test fails). 2269 my $failureCount = $count - $passCount; # "Failure" here includes new tests, timeouts, crashes, etc. 2270 return 0 if $failureCount < $exitAfterNFailures; 2271 2272 print "\nExiting early after $failureCount failures. $count tests run."; 2273 closeDumpTool(); 2274 return 1; 2275 } 80 exec($harnessName, @ARGV); -
trunk/WebKitTools/Scripts/webkitpy/layout_tests/run_webkit_tests.py
r57673 r57796 689 689 # Write the summary to disk (results.html) and display it if requested. 690 690 wrote_results = self._write_results_html_file(result_summary) 691 if not self._options.noshow_results and wrote_results:691 if self._options.show_results and wrote_results: 692 692 self._show_results_html_file() 693 693 … … 1534 1534 1535 1535 1536 def _compat_shim_callback(option, opt_str, value, parser): 1537 print "Ignoring unsupported option: %s" % opt_str 1538 1539 1540 def _compat_shim_option(option_name, nargs=0): 1541 return optparse.make_option(option_name, action="callback", callback=_compat_shim_callback, nargs=nargs, help="Ignored, for old-run-webkit-tests compat only.") 1542 1543 1536 1544 def parse_args(args=None): 1537 1545 """Provides a default set of command line args. … … 1587 1595 ] 1588 1596 1589 # Missing Mac options:1597 # Missing Mac-specific old-run-webkit-tests options: 1590 1598 # FIXME: Need: -g, --guard for guard malloc support on Mac. 1591 1599 # FIXME: Need: -l --leaks Enable leaks checking. 1592 # FIXME: Need: --[no-]sample-on-timeout Run sample on timeout 1593 # FIXME: Need: (default: run) (Mac OS X only) 1600 # FIXME: Need: --sample-on-timeout Run sample on timeout 1601 1602 old_run_webkit_tests_compat = [ 1603 # NRWT doesn't generate results by default anyway. 1604 _compat_shim_option("--no-new-test-results"), 1605 # NRWT doesn't sample on timeout yet anyway. 1606 _compat_shim_option("--no-sample-on-timeout"), 1607 # FIXME: NRWT needs to support remote links eventually. 1608 _compat_shim_option("--use-remote-links-to-tests"), 1609 # FIXME: NRWT doesn't need this option as much since failures are 1610 # designed to be cheap. We eventually plan to add this support. 1611 _compat_shim_option("--exit-after-n-failures", nargs=1), 1612 ] 1594 1613 1595 1614 results_options = [ 1596 1615 # NEED for bots: --use-remote-links-to-tests Link to test files 1597 1616 # within the SVN repository in the results. 1598 optparse.make_option("-p", "--pixel-tests '", action="store_true",1617 optparse.make_option("-p", "--pixel-tests", action="store_true", 1599 1618 dest="pixel_tests", help="Enable pixel-to-pixel PNG comparisons"), 1600 1619 optparse.make_option("--no-pixel-tests", action="store_false", … … 1613 1632 "into the platform directory, overwriting whatever's " 1614 1633 "already there."), 1615 # FIXME: --noshow should be --no-show to match the GNU get_opt pattern. 1616 optparse.make_option("--noshow-results", action="store_true", 1617 default=False, 1634 optparse.make_option("--no-show-results", action="store_false", 1635 default=True, dest="show_results", 1618 1636 help="Don't launch a browser with results after the tests " 1619 1637 "are done"), 1638 # FIXME: We should have a helper function to do this sort of 1639 # deprectated mapping and automatically log, etc. 1640 optparse.make_option("--noshow-results", action="store_false", 1641 dest="show_results", 1642 help="Deprecated, same as --no-show-results."), 1643 optparse.make_option("--no-launch-safari", action="store_false", 1644 dest="show_results", 1645 help="old-run-webkit-tests compat, same as --noshow-results."), 1620 1646 # old-run-webkit-tests: 1621 1647 # --[no-]launch-safari Launch (or do not launch) Safari to display … … 1707 1733 option_list = (configuration_options + logging_options + 1708 1734 chromium_options + results_options + test_options + 1709 misc_options + results_json_options) 1735 misc_options + results_json_options + 1736 old_run_webkit_tests_compat) 1710 1737 option_parser = optparse.OptionParser(option_list=option_list) 1711 1738 return option_parser.parse_args(args)
Note: See TracChangeset
for help on using the changeset viewer.