Changeset 155383 in webkit

Timestamp:

Sep 9, 2013, 2:36:54 PM (13 years ago)

Author:

fpizlo@apple.com

Message:

Stress tests should test the jsc profiler (-p)
​https://bugs.webkit.org/show_bug.cgi?id=121043

PerformanceTests/SunSpider:

Reviewed by Mark Hahnenberg.

Add a jsc-stress-test that tries to profile SunSpider.

• profiler-test.yaml: Added.

Tools:

Reviewed by Mark Hahnenberg.

Add a runProfiler command that all tests could use. This requires profiler-test-helper,
which first runs the JS test and then tries the output with display-profiler-output.
But if any of the things required for this to work aren't present, we just do a simpler
test that just uses "-p".

Because I didn't want to pollute SunSpider with "@ runProfiler", I added the ability
to create test collections using a yaml file that specifies the test path and the
command to run.

• Scripts/jsc-stress-test-helpers: Added.
• Scripts/jsc-stress-test-helpers/profiler-test-helper: Added.
• Scripts/run-javascriptcore-tests:
• Scripts/run-jsc-stress-tests:

Location:

trunk

Files:

• PerformanceTests/SunSpider/ChangeLog (modified) (1 diff)
• PerformanceTests/SunSpider/profiler-test.yaml (added)
• Tools/ChangeLog (modified) (1 diff)
• Tools/Scripts/jsc-stress-test-helpers (added)
• Tools/Scripts/jsc-stress-test-helpers/profiler-test-helper (added)
• Tools/Scripts/run-javascriptcore-tests (modified) (1 diff)
• Tools/Scripts/run-jsc-stress-tests (modified) (12 diffs)

trunk/PerformanceTests/SunSpider/ChangeLog

@@ +1 @@
+2013-09-09  Filip Pizlo  <fpizlo@apple.com>
+
+        Stress tests should test the jsc profiler (-p)
+        https://bugs.webkit.org/show_bug.cgi?id=121043
+
+        Reviewed by Mark Hahnenberg.
+        
+        Add a jsc-stress-test that tries to profile SunSpider.
+
+        * profiler-test.yaml: Added.
+
 2013-08-08  Cosmin Truta  <ctruta@blackberry.com>
 

trunk/Tools/ChangeLog

@@ +1 @@
+2013-09-09  Filip Pizlo  <fpizlo@apple.com>
+
+        Stress tests should test the jsc profiler (-p)
+        https://bugs.webkit.org/show_bug.cgi?id=121043
+
+        Reviewed by Mark Hahnenberg.
+        
+        Add a runProfiler command that all tests could use. This requires profiler-test-helper,
+        which first runs the JS test and then tries the output with display-profiler-output.
+        But if any of the things required for this to work aren't present, we just do a simpler
+        test that just uses "-p".
+        
+        Because I didn't want to pollute SunSpider with "//@ runProfiler", I added the ability
+        to create test collections using a yaml file that specifies the test path and the
+        command to run.
+
+        * Scripts/jsc-stress-test-helpers: Added.
+        * Scripts/jsc-stress-test-helpers/profiler-test-helper: Added.
+        * Scripts/run-javascriptcore-tests:
+        * Scripts/run-jsc-stress-tests:
+
 2013-09-09  Zan Dobersek  <zdobersek@igalia.com>
 

trunk/Tools/Scripts/run-javascriptcore-tests

@@ -244 +244 @@
         "PerformanceTests/SunSpider/tests/sunspider-1.0",
         "PerformanceTests/SunSpider/tests/v8-v6",
-        "LayoutTests/fast/js/regress/script-tests"
+        "LayoutTests/fast/js/regress/script-tests",
+        "PerformanceTests/SunSpider/profiler-test.yaml"
     );
     if ($enableFTL) {

trunk/Tools/Scripts/run-jsc-stress-tests

@@ -26 +26 @@
 require 'getoptlong'
 require 'pathname'
+require 'yaml'
+
+THIS_SCRIPT_PATH = Pathname.new(__FILE__).realpath
+SCRIPTS_PATH = THIS_SCRIPT_PATH.dirname
+raise unless SCRIPTS_PATH.basename.to_s == "Scripts"
+raise unless SCRIPTS_PATH.dirname.basename.to_s == "Tools"
+
+HELPERS_PATH = SCRIPTS_PATH + "jsc-stress-test-helpers"
 
 $haveShellwords = false
@@ -37 +45 @@
 end
 
-def mysys(*cmd)
+$canRunDisplayProfilerOutput = false
+
+begin
+    require 'json'
+    require 'highline'
+    $canRunDisplayProfilerOutput = true
+rescue => e
+    $stderr.puts "Warning: did not find json or highline; some features will be disabled."
+    $stderr.puts "Error: #{e.inspect}"
+end
+
+def printCommandArray(*cmd)
     begin
         commandArray = cmd.each{|value| Shellwords.shellescape(value.to_s)}.join(' ')
@@ -44 +63 @@
     end
     $stderr.puts ">> #{commandArray}"
+end
+
+def mysys(*cmd)
+    printCommandArray(*cmd)
     raise "Command failed: #{$?.inspect}" unless system(*cmd)
 end
@@ -54 +77 @@
 $outputDir = Pathname.new("results")
 $parallel = ($haveShellwords and numProcessors > 1)
+$verbosity = 0
 
 def usage
@@ -62 +86 @@
     puts "--[no-]parallel         Run in parallel, or not. Default is #{$parallel}."
     puts "--output-dir     (-o)   Path where to put results. Default is #{$outputDir}."
+    puts "--verbose        (-v)   Print more things while running."
     puts "--help           (-h)   Print this message."
     exit 1
@@ -71 +96 @@
                ['--parallel', GetoptLong::NO_ARGUMENT],
                ['--no-parallel', GetoptLong::NO_ARGUMENT],
-               ['--output-dir', '-o', GetoptLong::REQUIRED_ARGUMENT]).each {
+               ['--output-dir', '-o', GetoptLong::REQUIRED_ARGUMENT],
+               ['--verbose', '-v', GetoptLong::NO_ARGUMENT]).each {
     | opt, arg |
     case opt
@@ -86 +112 @@
     when '--no-parallel'
         $parallel = false
+    when '--verbose'
+        $verbosity += 1
     end
 }
@@ -114 +142 @@
             | outp |
             outp.puts "echo Running #{Shellwords.shellescape(@name)}"
-            outp.puts("(cd #{Shellwords.shellescape(@directory.to_s)} && " +
-                      @arguments.map{|v| Shellwords.shellescape(v)}.join(' ') +
-                      ") || #{failCommand}")
-        }
-    end
+            cmd = ("(cd #{Shellwords.shellescape(@directory.to_s)} && " +
+                   @arguments.map{|v| Shellwords.shellescape(v)}.join(' ') +
+                   ") || #{failCommand}")
+            if $verbosity >= 1
+                outp.puts "echo #{Shellwords.shellescape(cmd)}"
+            end
+            outp.puts cmd
+        }
+    end
+end
+
+$uniqueFilenameCounter = 0
+def uniqueFilename(extension)
+    payloadDir = $outputDir + "_payload"
+    Dir.mkdir payloadDir unless payloadDir.directory?
+    result = payloadDir.realpath + "temp-#{$uniqueFilenameCounter}#{extension}"
+    $uniqueFilenameCounter += 1
+    result
+end
+
+def addRunCommand(kind, command)
+    $runlist << Plan.new($benchmarkDirectory, command, "#{$collectionName}/#{$benchmark}.#{kind}")
 end
 
 def run(kind, *options)
-    $runlist << Plan.new($collection,
-                         [$jscPath.to_s] + options + [$benchmark],
-                         "#{$collectionName}/#{$benchmark}.#{kind}")
+    addRunCommand(kind, [$jscPath.to_s] + options + [$benchmark.to_s])
 end
 
@@ -149 +192 @@
 def runFTLEager
     run("ftl-eager", "--useExperimentalFTL=true", *EAGER_OPTIONS)
+end
+
+def runProfiler
+    profilerOutput = uniqueFilename(".json")
+    if $haveShellwords and $canRunDisplayProfilerOutput
+        addRunCommand("profiler", ["ruby", (HELPERS_PATH + "profiler-test-helper").to_s, (SCRIPTS_PATH + "display-profiler-output").to_s, profilerOutput.to_s, $jscPath.to_s, "-p", profilerOutput.to_s, $benchmark.to_s])
+    else
+        run("profiler-simple", "-p", profilerOutput.to_s)
+    end
 end
 
@@ -167 +219 @@
 end
 
-collectionNames = {}
-
 Dir.mkdir($outputDir) unless $outputDir.directory?
 begin
@@ -177 +227 @@
 $outputDir = $outputDir.realpath
 
-ARGV.each {
-    | collection |
-    $collection = Pathname.new(collection)
-    outerDir = $collection.dirname
-    name = $collection.basename
+def allJSFiles(path)
+    if path.file?
+        [path]
+    else
+        result = []
+        Dir.foreach(path) {
+            | filename |
+            next unless filename =~ /\.js$/
+            next unless (path + filename).file?
+            result << path + filename
+        }
+        result
+    end
+end
+
+# Returns [collectionPath, collectionName]
+def simplifyCollectionName(collectionNames, collectionPath)
+    outerDir = collectionPath.dirname
+    name = collectionPath.basename
     lastName = name
-    while lastName.to_s =~ /test/
-        lastName = outerDir.basename
-        name = lastName + name
-        outerDir = outerDir.dirname
-    end
-    $collectionName = name.to_s
+    if collectionPath.directory?
+        while lastName.to_s =~ /test/
+            lastName = outerDir.basename
+            name = lastName + name
+            outerDir = outerDir.dirname
+        end
+    end
+    collectionName = name.to_s
     toAdd = 1
-    while collectionNames[$collectionName]
-        $collectionName = File.basename(name.to_s) + "-#{toAdd}"
+    while collectionNames[collectionName]
+        collectionName = File.basename(name.to_s) + "-#{toAdd}"
         toAdd += 1
     end
-    collectionNames[$collectionName] = true
+    collectionNames[collectionName] = true
+    [collectionPath, collectionName]
+end
+
+def prepareCollection(name)
     dir = $outputDir
-    name.each_filename {
+    Pathname.new(name).each_filename {
         | filename |
         dir = dir + filename
         Dir.mkdir(dir) unless dir.directory?
     }
-
-    Dir.foreach($collection) {
-        | benchmark |
-        next unless benchmark =~ /\.js$/
-        next unless ($collection + benchmark).file?
+end
+
+collectionNames = {}
+
+ARGV.each {
+    | collection |
+    collection, collectionName = simplifyCollectionName(collectionNames, Pathname.new(collection))
+    
+    if collection.file?
+        subCollectionNames = {}
+        YAML::load(IO::read(collection)).each {
+            | entry |
+            path = collection.dirname + entry["path"]
+            
+            subCollection, subCollectionName = simplifyCollectionName(subCollectionNames, path)
+            
+            $collection = subCollection
+            $collectionName = (Pathname.new(collectionName) + subCollectionName).to_s
+            
+            prepareCollection($collectionName)
+            
+            allJSFiles(path).each {
+                | path |
+                
+                path = path.realpath
+                
+                $benchmark = path.basename
+                $benchmarkDirectory = path.dirname
+                
+                eval entry["cmd"]
+            }
+        }
+    else
+        prepareCollection(collectionName)
         
-        $benchmark = benchmark
-        
-        didRun = false
-        File.open($collection + benchmark) {
-            | inp |
-            inp.each_line {
-                | line |
-                next unless line =~ /^\/\/@/
-                eval $~.post_match
-                didRun = true
+        $collection = collection
+        $collectionName = collectionName
+        $benchmarkDirectory = $collection
+        allJSFiles($collection).each {
+            | path |
+            
+            $benchmark = path.basename
+            
+            didRun = false
+            File.open($collection + $benchmark) {
+                | inp |
+                inp.each_line {
+                    | line |
+                    next unless line =~ /^\/\/@/
+                    eval $~.post_match
+                    didRun = true
+                }
             }
-        }
-        
-        defaultRun unless didRun
-    }
+            
+            defaultRun unless didRun
+        }
+    end
 }
 
@@ -344 +450 @@
         
         Dir.chdir(plan.directory) {
+            if $verbosity >= 1
+                printCommandArray(*plan.arguments)
+            end
             if system(*plan.arguments)
                 puts "OK."