Changeset 96462 in webkit


Ignore:
Timestamp:
Oct 1, 2011 2:58:45 PM (13 years ago)
Author:
fpizlo@apple.com
Message:

Bencher script makes it difficult to do automated performance testing
​https://bugs.webkit.org/show_bug.cgi?id=69207

Reviewed by Sam Weinig.

This adds two new features:

The ability to disable automatic VM detection, which is flaky if any
profiling features are enabled in jsc.

The ability to compute, and report, a scaled result for all benchmark
suites. It is the geometric mean of three numbers: SunSpider's
arithmetic mean, V8's geometric mean, and Kraken's arithmetic mean.
It is also possible to turn off all other output from bencher and just
get this number with the --brief option.

  • Scripts/bencher:
Location:
trunk/Tools
Files:
2 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r96460 r96462  
     12011-10-01  Filip Pizlo  <fpizlo@apple.com>
     2
     3        Bencher script makes it difficult to do automated performance testing
     4        https://bugs.webkit.org/show_bug.cgi?id=69207
     5
     6        Reviewed by Sam Weinig.
     7       
     8        This adds two new features:
     9       
     10        The ability to disable automatic VM detection, which is flaky if any
     11        profiling features are enabled in jsc.
     12       
     13        The ability to compute, and report, a scaled result for all benchmark
     14        suites. It is the geometric mean of three numbers: SunSpider's
     15        arithmetic mean, V8's geometric mean, and Kraken's arithmetic mean.
     16        It is also possible to turn off all other output from bencher and just
     17        get this number with the --brief option.
     18
     19        * Scripts/bencher:
     20
    1212011-10-01  Sam Weinig  <sam@webkit.org>
    222

  • trunk/Tools/Scripts/bencher

    r94103 r96462  
    210210$timeMode=:auto
    211211$keepFiles=false
     212$forceVMKind=nil
     213$brief=false
    212214
    213215# Helpful functions and classes
    … …  
    251253  puts "                     are 'preciseTime', 'date', and 'auto'.  Default is"
    252254  puts "                     'auto', which automatically detects the best way."
     255  puts "--force-vm-kind      Turn off auto-detection of VM kind, and assume that it is"
     256  puts "                     the one specified.  Valid arguments are 'jsc' or"
     257  puts "                     'DumpRenderTree'."
    253258  puts "--v8-only            Only run V8."
    254259  puts "--sunspider-only     Only run SunSpider."
    … …  
    260265  puts "--keep-files         Keep temporary files.  Useful for debugging."
    261266  puts "--verbose or -v      Print more stuff."
     267  puts "--brief              Print only the final result for each VM."
    262268  puts "--help or -h         Display this message."
    263269  puts
    … …  
    692698    @nameKind = nameKind
    693699   
    694     Tempfile.open("bencher-vmtest") {
    695       | file |
    696       file.puts "print(\"here\");"
    697       file.flush
    698      
    699       result = nil
    700       @vmType = :jsc
    701       run(file.path) {
    702         | inp |
    703         result = inp.read
     700    if $forceVMKind
     701      @vmType = $forceVMKind
     702    else
     703      Tempfile.open("bencher-vmtest") {
     704        | file |
     705        file.puts "print(\"here\");"
     706        file.flush
     707       
     708        result = nil
     709        @vmType = :jsc
     710        run(file.path) {
     711          | inp |
     712          result = inp.read
     713          $stderr.puts "stdout: #{result}" if $verbosity>=2
     714        }
     715       
     716        if result.chomp == "here"
     717          $stderr.puts "#{@name} is definitely a jsc-style VM." if $verbosity>=1
     718          @vmType = :jsc
     719        else
     720          $stderr.puts "Assuming that #{@name} is a DumpRenderTree-style VM." if $verbosity>=1
     721          @vmType = :dumpRenderTree
     722        end
    704723      }
    705      
    706       if result.chomp == "here"
    707         $stderr.puts "#{@name} is definitely a jsc-style VM." if $verbosity>=1
    708         @vmType = :jsc
    709       else
    710         $stderr.puts "Assuming that #{@name} is a DumpRenderTree-style VM." if $verbosity>=1
    711         @vmType = :dumpRenderTree
    712       end
    713     }
     724    end
    714725  end
    715726 
    … …  
    929940
    930941class BenchmarkSuite
    931   def initialize(name, path)
     942  def initialize(name, path, preferredMean)
    932943    @name = name
    933944    @path = path
     945    @preferredMean = preferredMean
    934946    @benchmarks = []
    935947  end
    … …  
    967979      not yield benchmark
    968980    }
     981  end
     982 
     983  def preferredMean
     984    @preferredMean
     985  end
     986 
     987  def computeMean(stat)
     988    stat.send @preferredMean
    969989  end
    970990end
    … …  
    10941114
    10951115def statsToStr(stats)
    1096   lpad(numToStr(stats.mean),11)+"+-"+rpad(numToStr(stats.confInt),9)
     1116  if $inner*$outer == 1
     1117    string = numToStr(stats.mean)
     1118    raise unless string =~ /\./
     1119    left = $~.pre_match
     1120    right = $~.post_match
     1121    lpad(left,12)+"."+rpad(right,9)
     1122  else
     1123    lpad(numToStr(stats.mean),11)+"+-"+rpad(numToStr(stats.confInt),9)
     1124  end
    10971125end
    10981126 
    … …  
    11091137                 ['--exclude-kraken', GetoptLong::NO_ARGUMENT],
    11101138                 ['--benchmarks', GetoptLong::REQUIRED_ARGUMENT],
     1139                 ['--force-vm-kind', GetoptLong::REQUIRED_ARGUMENT],
    11111140                 ['--load-once', GetoptLong::NO_ARGUMENT],
    11121141                 ['--keep-files', GetoptLong::NO_ARGUMENT],
    11131142                 ['--verbose', '-v', GetoptLong::NO_ARGUMENT],
     1143                 ['--brief', GetoptLong::NO_ARGUMENT],
    11141144                 ['--help', '-h', GetoptLong::NO_ARGUMENT]).each {
    11151145    | opt, arg |
    … …  
    11301160      else
    11311161        quickFail("Expected either 'preciseTime', 'date', or 'auto' for --time-mode, but got '#{arg}'.",
     1162                  "Invalid argument for command-line option")
     1163      end
     1164    when '--force-vm-kind'
     1165      if arg.upcase == "JSC"
     1166        $forceVMKind = :jsc
     1167      elsif arg.upcase == "DUMPRENDERTREE"
     1168        $forceVMKind = :dumpRenderTree
     1169      elsif arg.upcase == "AUTO"
     1170        $forceVMKind = nil
     1171      else
     1172        quickFail("Expected either 'jsc' or 'DumpRenderTree' for --force-vm-kind, but got '#{arg}'.",
    11321173                  "Invalid argument for command-line option")
    11331174      end
    … …  
    11551196    when '--verbose'
    11561197      $verbosity += 1
     1198    when '--brief'
     1199      $brief = true
    11571200    when '--help'
    11581201      usage
    … …  
    11671210  end
    11681211 
    1169   SUNSPIDER = BenchmarkSuite.new("SunSpider", SUNSPIDER_PATH)
     1212  SUNSPIDER = BenchmarkSuite.new("SunSpider", SUNSPIDER_PATH, :arithmeticMean)
    11701213  ["3d-cube", "3d-morph", "3d-raytrace", "access-binary-trees",
    11711214   "access-fannkuch", "access-nbody", "access-nsieve",
    … …  
    11801223  }
    11811224
    1182   V8 = BenchmarkSuite.new("V8", V8_PATH)
     1225  V8 = BenchmarkSuite.new("V8", V8_PATH, :geometricMean)
    11831226  ["crypto", "deltablue", "earley-boyer", "raytrace",
    11841227   "regexp", "richards", "splay"].each {
    … …  
    11871230  }
    11881231
    1189   KRAKEN = BenchmarkSuite.new("Kraken", KRAKEN_PATH)
     1232  KRAKEN = BenchmarkSuite.new("Kraken", KRAKEN_PATH, :arithmeticMean)
    11901233  ["ai-astar", "audio-beat-detection", "audio-dft", "audio-fft",
    11911234   "audio-oscillator", "imaging-darkroom", "imaging-desaturate",
    … …  
    12861329    $suitesOnVMsForSuite[suite] = []
    12871330  }
     1331  $suitesOnVMsForVM = {}
     1332  $vms.each {
     1333    | vm |
     1334    $suitesOnVMsForVM[vm] = []
     1335  }
    12881336 
    12891337  $benchmarksOnVMs = []
    … …  
    13011349      $suitesOnVMs << suiteOnVM
    13021350      $suitesOnVMsForSuite[suite] << suiteOnVM
     1351      $suitesOnVMsForVM[vm] << suiteOnVM
    13031352      suite.benchmarks.each {
    13041353        | benchmark |
    … …  
    13271376 
    13281377  $benchpad = ($benchmarks +
    1329                ["<arithmetic>", "<geometric>", "<harmonic>"]).collect {
     1378               ["<arithmetic> *", "<geometric> *", "<harmonic> *"]).collect {
    13301379    | benchmark |
    13311380    benchmark.to_s.size
    … …  
    13361385    vm.to_s.size
    13371386  }.max + 1
     1387
     1388  unless $brief
     1389    3.times {
     1390      | idx |
     1391      $stderr.print "\rStarting in #{3-idx}..."
     1392      $stderr.flush
     1393      sleep 1
     1394    }
     1395    $stderr.print "\r                       \r"
     1396    $stderr.flush
     1397  end
    13381398 
    13391399  $plans.each_with_index {
    13401400    | plan, idx |
    1341     if $verbosity == 0
     1401    if $verbosity == 0 and not $brief
    13421402      text1 = lpad(idx.to_s,$plans.size.to_s.size)+"/"+$plans.size.to_s
    13431403      text2 = plan.suite.to_s+"/"+plan.benchmark.to_s+"/"+plan.vm.to_s
    … …  
    13491409  }
    13501410 
    1351   if $verbosity == 0
     1411  if $verbosity == 0 and not $brief
    13521412    $stderr.print "\r#{$plans.size}/#{$plans.size} #{' '*($suitepad+1+$benchpad+1+$vmpad)}"
    13531413    $stderr.puts "\r#{$plans.size}/#{$plans.size}"
    13541414  end
     1415 
     1416  # Compute the geomean of the preferred means of results on a SuiteOnVM
     1417  $overallResults = []
     1418  $vms.each {
     1419    | vm |
     1420    result = Stats.new
     1421    $outer.times {
     1422      | outerIndex |
     1423      $inner.times {
     1424        | innerIndex |
     1425        curResult = Stats.new
     1426        $suitesOnVMsForVM[vm].each {
     1427          | suiteOnVM |
     1428          # For a given iteration, suite, and VM, compute the suite's preferred mean
     1429          # over the data collected for all benchmarks in that suite. We'll have one
     1430          # sample per benchmark. For example on V8 this will be the geomean of 1
     1431          # sample for crypto, 1 sample for deltablue, and so on, and 1 sample for
     1432          # splay.
     1433          curResult.add(suiteOnVM.suite.computeMean(suiteOnVM.statsForIteration(outerIndex, innerIndex)))
     1434        }
     1435       
     1436        # curResult now holds 1 sample for each of the means computed in the above
     1437        # loop. Compute the geomean over this, and store it.
     1438        result.add(curResult.geometricMean)
     1439      }
     1440    }
     1441
     1442    # $overallResults will have a Stats for each VM. That Stats object will hold
     1443    # $inner*$outer geomeans, allowing us to compute the arithmetic mean and
     1444    # confidence interval of the geomeans of preferred means. Convoluted, but
     1445    # useful and probably sound.
     1446    $overallResults << result
     1447  }
    13551448 
    13561449  if $verbosity >= 2
    … …  
    13661459    }
    13671460  end
    1368  
     1461
    13691462  reportName =
    13701463    (if ($vms.collect {
    … …  
    13901483     end) +
    13911484    "_benchReport.txt"
    1392  
    1393   $stderr.puts "Generating benchmark report at #{reportName}"
     1485
     1486  unless $brief
     1487    $stderr.puts "Generating benchmark report at #{reportName}"
     1488  end
    13941489 
    13951490  outp = $stdout
    … …  
    15221617  end
    15231618 
    1524   def allSummaryStats(outp, accumulators)
    1525     summaryStats(outp, accumulators, "<arithmetic>") {
     1619  def meanName(currentMean, preferredMean)
     1620    result = "<#{currentMean}>"
     1621    if "#{currentMean}Mean" == preferredMean.to_s
     1622      result += " *"
     1623    end
     1624    result
     1625  end
     1626 
     1627  def allSummaryStats(outp, accumulators, preferredMean)
     1628    summaryStats(outp, accumulators, meanName("arithmetic", preferredMean)) {
    15261629      | stat |
    15271630      stat.arithmeticMean
    15281631    }
    15291632   
    1530     summaryStats(outp, accumulators, "<geometric>") {
     1633    summaryStats(outp, accumulators, meanName("geometric", preferredMean)) {
    15311634      | stat |
    15321635      stat.geometricMean
    15331636    }
    15341637   
    1535     summaryStats(outp, accumulators, "<harmonic>") {
     1638    summaryStats(outp, accumulators, meanName("harmonic", preferredMean)) {
    15361639      | stat |
    15371640      stat.harmonicMean
    … …  
    15661669    }
    15671670    outp.puts
    1568     allSummaryStats(outp, $suitesOnVMsForSuite[suite])
     1671    allSummaryStats(outp, $suitesOnVMsForSuite[suite], suite.preferredMean)
    15691672    outp.puts if $suites.size > 1
    15701673  }
    … …  
    15731676    printVMs(outp)
    15741677    outp.puts "All benchmarks:"
    1575     allSummaryStats(outp, $vms)
    1576   end
    1577  
    1578   if outp != $stdout
     1678    allSummaryStats(outp, $vms, nil)
     1679   
     1680    outp.puts
     1681    printVMs(outp)
     1682    outp.puts "Geomean of preferred means:"
     1683    outp.print "   "
     1684    outp.print rpad("<scaled-result>", $benchpad)
     1685    outp.print " "
     1686    $vms.size.times {
     1687      | index |
     1688      if index != 0
     1689        outp.print " "+$overallResults[index].compareTo($overallResults[index-1]).shortForm
     1690      end
     1691      outp.print statsToStr($overallResults[index])
     1692    }
     1693    if $overallResults.size>=2
     1694      outp.print("    "+$overallResults[-1].compareTo($overallResults[0]).to_s)
     1695    end
     1696    outp.puts
     1697    outp.puts
     1698  end
     1699 
     1700  if outp != $stdout and not $brief
    15791701    outp.close
    15801702    puts
    … …  
    15851707  end
    15861708 
     1709  if $brief
     1710    puts($overallResults.collect{|stats| stats.mean}.join("\t"))
     1711    puts($overallResults.collect{|stats| stats.confInt}.join("\t"))
     1712  end
     1713 
    15871714rescue => e
    15881715  fail(e)
Note: See TracChangeset for help on using the changeset viewer.