Changeset 185425 in webkit

Timestamp:

Jun 10, 2015, 11:44:50 AM (11 years ago)

Author:

fpizlo@apple.com

Message:

JetStream should have a more rational story for jitter-oriented latency tests
​https://bugs.webkit.org/show_bug.cgi?id=145762

Reviewed by Geoffrey Garen.

JetStream has some latency tests that are meant to measure jitter. Prior to this change, they
did this by computing the RMS. But the RMS is a pretty bad metric. The thing that it rewards
isn't really the thing that you'd want your browser to do. These RMS-based tests involve taking
the geomean of the RMS of some samples and the sample average. The lower the geomean, the better
(in the JetStream harness we then invert the scores so that higher is better, but let's ignore
that for this discussion and assume that lower is better). Here's an example of how this can go
bad. A browser that always computes a task in some horribly long time (say, 1000ms) but never
varies that time will perform better than a browser that usually computes the task super quickly
(say, 10ms) and sometimes just a little bit less quickly (say, 15ms). The former browser will
have an RMS of 0 and an average of 1000. The latter will have a RMS somewhere around 3.5 and an
average of 12.5 (assuming equal probability of 10ms and 15ms). The geomean of (0, 1000) is 0.
The geomean of (3.5, 12.5) is 6.6. Lower is better, so the former browser scores higher - even
though it's obviously never better to have a browser always complete a task in 1000ms when a
different browser can do it in 15ms in the worst case.

JetStream should not have this pathology. The right way of avoiding it is to replace RMS with
some other metric of how bad things get. A good metric is the average of the worst percentile.
The worst 1% or the worst 5% would be good things to average. This will catch cases where the VM
jittered due to JIT or GC, but it never have the pathology that we end up giving the better score
to a VM whose best case is worst than another VM's worst case.

For now, this change uses the highest samples above the 95% percentile. I'm not yet sure if that
is the best thing - it might include too many scores that are around the best-case performance -
but it's certainly better than RMS and it might be good enough to keep. But because of that
uncertainty, I'm setting the version to be "1.1-alpha1" to indicate that we aren't ready to
release this yet.

• JetStream/Octane2/base.js:

(.this.Setup.setup.setup):
(.this.TearDown.tearDown.tearDown):
(BenchmarkSuite.GeometricMeanTime):
(BenchmarkSuite.AverageAbovePercentile):
(BenchmarkSuite.GeometricMeanLatency):
(BenchmarkSuite.prototype.NotifyStep):
(BenchmarkSuite.prototype.RunSingleBenchmark):

• JetStream/Octane2/mandreel.js:

(setupMandreel):
(updateMandreelStats):
(startMandreelTimer):
(latencyMandreel):
(tearDownMandreel):
(RMSMandreel): Deleted.

• JetStream/Octane2/splay.js:

(GenerateKey):
(SplayUpdateStats):
(InsertNewNode):
(SplayTearDown):
(SplayRMS): Deleted.

• JetStream/create.rb:

Location:

trunk

Files:

• PerformanceTests/ChangeLog (modified) (1 diff)
• PerformanceTests/JetStream/Octane2/base.js (modified) (5 diffs)
• PerformanceTests/JetStream/Octane2/mandreel.js (modified) (5 diffs)
• PerformanceTests/JetStream/Octane2/splay.js (modified) (5 diffs)
• PerformanceTests/JetStream/Reference.js (modified) (1 diff)
• PerformanceTests/JetStream/create.rb (modified) (2 diffs)
• Source/JavaScriptCore/bytecode/CodeBlock.h (modified) (1 diff)
• Source/JavaScriptCore/bytecode/DFGExitProfile.cpp (modified) (1 diff)
• Source/JavaScriptCore/bytecode/DFGExitProfile.h (modified) (1 diff)
• Source/JavaScriptCore/dfg/DFGByteCodeParser.cpp (modified) (1 diff)
• Source/JavaScriptCore/runtime/Options.h (modified) (1 diff)
• Tools/Scripts/run-jsc-benchmarks (modified) (2 diffs)

trunk/PerformanceTests/ChangeLog

@@ +1 @@
+2015-06-08  Filip Pizlo  <fpizlo@apple.com>
+
+        JetStream should have a more rational story for jitter-oriented latency tests
+        https://bugs.webkit.org/show_bug.cgi?id=145762
+
+        Reviewed by Geoffrey Garen.
+        
+        JetStream has some latency tests that are meant to measure jitter.  Prior to this change, they
+        did this by computing the RMS.  But the RMS is a pretty bad metric.  The thing that it rewards
+        isn't really the thing that you'd want your browser to do.  These RMS-based tests involve taking
+        the geomean of the RMS of some samples and the sample average.  The lower the geomean, the better
+        (in the JetStream harness we then invert the scores so that higher is better, but let's ignore
+        that for this discussion and assume that lower is better).  Here's an example of how this can go
+        bad.  A browser that always computes a task in some horribly long time (say, 1000ms) but never
+        varies that time will perform better than a browser that usually computes the task super quickly
+        (say, 10ms) and sometimes just a little bit less quickly (say, 15ms).  The former browser will
+        have an RMS of 0 and an average of 1000.  The latter will have a RMS somewhere around 3.5 and an
+        average of 12.5 (assuming equal probability of 10ms and 15ms).  The geomean of (0, 1000) is 0.
+        The geomean of (3.5, 12.5) is 6.6.  Lower is better, so the former browser scores higher - even
+        though it's obviously never better to have a browser always complete a task in 1000ms when a
+        different browser can do it in 15ms in the worst case.
+
+        JetStream should not have this pathology.  The right way of avoiding it is to replace RMS with
+        some other metric of how bad things get.  A good metric is the average of the worst percentile.
+        The worst 1% or the worst 5% would be good things to average.  This will catch cases where the VM
+        jittered due to JIT or GC, but it never have the pathology that we end up giving the better score
+        to a VM whose best case is worst than another VM's worst case.
+        
+        For now, this change uses the highest samples above the 95% percentile. I'm not yet sure if that
+        is the best thing - it might include too many scores that are around the best-case performance -
+        but it's certainly better than RMS and it might be good enough to keep. But because of that
+        uncertainty, I'm setting the version to be "1.1-alpha1" to indicate that we aren't ready to
+        release this yet.
+
+        * JetStream/Octane2/base.js:
+        (.this.Setup.setup.setup):
+        (.this.TearDown.tearDown.tearDown):
+        (BenchmarkSuite.GeometricMeanTime):
+        (BenchmarkSuite.AverageAbovePercentile):
+        (BenchmarkSuite.GeometricMeanLatency):
+        (BenchmarkSuite.prototype.NotifyStep):
+        (BenchmarkSuite.prototype.RunSingleBenchmark):
+        * JetStream/Octane2/mandreel.js:
+        (setupMandreel):
+        (updateMandreelStats):
+        (startMandreelTimer):
+        (latencyMandreel):
+        (tearDownMandreel):
+        (RMSMandreel): Deleted.
+        * JetStream/Octane2/splay.js:
+        (GenerateKey):
+        (SplayUpdateStats):
+        (InsertNewNode):
+        (SplayTearDown):
+        (SplayRMS): Deleted.
+        * JetStream/create.rb:
+
 2015-06-03  Zalan Bujtas  <zalan@apple.com>
 

trunk/PerformanceTests/JetStream/Octane2/base.js

@@ -1 +1 @@
 // Copyright 2013 the V8 project authors. All rights reserved.
+// Copyright (C) 2015 Apple Inc. All rights reserved.
 // Redistribution and use in source and binary forms, with or without
 // modification, are permitted provided that the following conditions are
@@ -47 +48 @@
 // running the benchmark, but the running time of these functions will
 // not be accounted for in the benchmark score.
-function Benchmark(name, doWarmup, doDeterministic, run, setup, tearDown, rmsResult, minIterations) {
+function Benchmark(name, doWarmup, doDeterministic, run, setup, tearDown, latencyResult, minIterations) {
   this.name = name;
   this.doWarmup = doWarmup;
@@ -54 +55 @@
   this.Setup = setup ? setup : function() { };
   this.TearDown = tearDown ? tearDown : function() { };
-  this.rmsResult = rmsResult ? rmsResult : null; 
+  this.latencyResult = latencyResult ? latencyResult : null; 
   this.minIterations = minIterations ? minIterations : 32;
 }
@@ -190 +191 @@
 
 
-// Computes the geometric mean of a set of rms measurements.
+// Computes the average of the worst samples. For example, if percentile is 99, this will report the
+// average of the worst 1% of the samples.
+BenchmarkSuite.AverageAbovePercentile = function(numbers, percentile) {
+  // Don't change the original array.
+  numbers = numbers.slice();
+  
+  // Sort in ascending order.
+  numbers.sort(function(a, b) { return a - b; });
+  
+  // Now the elements we want are at the end. Keep removing them until the array size shrinks too much.
+  // Examples assuming percentile = 99:
+  //
+  // - numbers.length starts at 100: we will remove just the worst entry and then not remove anymore,
+  //   since then numbers.length / originalLength = 0.99.
+  //
+  // - numbers.length starts at 1000: we will remove the ten worst.
+  //
+  // - numbers.length starts at 10: we will remove just the worst.
+  var numbersWeWant = [];
+  var originalLength = numbers.length;
+  while (numbers.length / originalLength > percentile / 100)
+    numbersWeWant.push(numbers.pop());
+  
+  var sum = 0;
+  for (var i = 0; i < numbersWeWant.length; ++i)
+    sum += numbersWeWant[i];
+  
+  var result = sum / numbersWeWant.length;
+  
+  // Do a sanity check.
+  if (numbers.length && result < numbers[numbers.length - 1]) {
+    throw "Sanity check fail: the worst case result is " + result +
+      " but we didn't take into account " + numbers;
+  }
+  
+  return result;
+}
+
+
+// Computes the geometric mean of a set of latency measurements.
 BenchmarkSuite.GeometricMeanLatency = function(measurements) {
   var log = 0;
@@ -294 +334 @@
     if (data.runs < benchmark.minIterations) return data;
     var usec = (data.elapsed * 1000) / data.runs;
-    var rms = (benchmark.rmsResult != null) ? benchmark.rmsResult() : 0;
-    this.NotifyStep(new BenchmarkResult(benchmark, usec, rms));
+    var latencySamples = (benchmark.latencyResult != null) ? benchmark.latencyResult() : [0];
+    var percentile = 95;
+    var latency = BenchmarkSuite.AverageAbovePercentile(latencySamples, percentile) * 1000;
+    this.NotifyStep(new BenchmarkResult(benchmark, usec, latency));
     return null;
   }

trunk/PerformanceTests/JetStream/Octane2/mandreel.js

@@ -1 +1 @@
 // Portions copyright 2012 Google, Inc.
 // Copyright 2012 Onan Games. All rights reserved.
+// Copyright (C) 2015 Apple Inc. All rights reserved.
 // Redistribution and use in source and binary forms, with or without
 // modification, are permitted provided that the following conditions are
@@ -34 +35 @@
                                                            setupMandreel,
                                                            tearDownMandreel,
-                                                           RMSMandreel,
+                                                           latencyMandreel,
                                                            4)]);
 
-var mandreelSumSquaredPauses = 0;
-var mandreelSamples = 0;
+var mandreelPauseTimes = [];
 var mandreelSampleTimeStart = 0.0;
 
@@ -81 +81 @@
   var pause = time - mandreelSampleTimeStart;
   mandreelSampleTimeStart = time;
-  mandreelSumSquaredPauses += (pause * pause);
-  mandreelSamples++;
+  mandreelPauseTimes.push(pause);
 }
 
@@ -89 +88 @@
 }
 
-function RMSMandreel() {
-  return Math.round(Math.sqrt(mandreelSumSquaredPauses / mandreelSamples) * 100);
+function latencyMandreel() {
+  return mandreelPauseTimes;
 }
 
@@ -109 +108 @@
   heapDouble = null;
   mandreelAppUsePackAsyncTexture = null;
-  mandreelSumSquaredPauses = 0;
-  mandreelSamples = 0;
+  mandreelPauseTimes = [];
 }
 

trunk/PerformanceTests/JetStream/Octane2/splay.js

@@ -1 +1 @@
 // Copyright 2009 the V8 project authors. All rights reserved.
+// Copyright (C) 2015 Apple Inc. All rights reserved.
 // Redistribution and use in source and binary forms, with or without
 // modification, are permitted provided that the following conditions are
@@ -36 +37 @@
 var Splay = new BenchmarkSuite('Splay', [81491, 2739514], [
   new Benchmark("Splay", true, false, 
-    SplayRun, SplaySetup, SplayTearDown, SplayRMS)
+    SplayRun, SplaySetup, SplayTearDown, SplayLatency)
 ]);
 
@@ -69 +70 @@
 }
 
-var splaySamples = 0;
-var splaySumOfSquaredPauses = 0;
-
-function SplayRMS() {
-  return Math.round(Math.sqrt(splaySumOfSquaredPauses / splaySamples) * 10000);
+var splaySamples = [];
+
+function SplayLatency() {
+  return splaySamples;
 }
 
@@ -79 +79 @@
   var pause = time - splaySampleTimeStart;
   splaySampleTimeStart = time;
-  splaySamples++;
-  splaySumOfSquaredPauses += pause * pause;
+  splaySamples.push(pause);
 }
 
@@ -120 +119 @@
   splayTree = null;
 
-  splaySamples = 0;
-  splaySumOfSquaredPauses = 0;
+  splaySamples = [];
 
   // Verify that the splay tree has the right size.

trunk/PerformanceTests/JetStream/Reference.js

@@ -55 +55 @@
     "regexp-2010": 61.82352941176467,
     "splay": 0.9286376274328075,
-    "splay-latency": 32.399999999999984,
+    "splay-latency": 3.524855736311738,
     "navier-stokes": 9.653846153846146,
     "pdfjs": 88.4166666666666,
     "mandreel": 157.14285714285708,
-    "mandreel-latency": 0.5129999999999998,
+    "mandreel-latency": 1.4940079129942474,
     "gbemu": 135.9999999999998,
     "code-first-load": 2.3249465349251905,

trunk/PerformanceTests/JetStream/create.rb

@@ -1 +1 @@
 #!/usr/bin/env ruby
 
-# Copyright (C) 2014 Apple Inc. All rights reserved.
+# Copyright (C) 2014, 2015 Apple Inc. All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
@@ -27 +27 @@
 require "shellwords"
 
-VERSION = "1.0.1"
+VERSION = "1.1-alpha1"
 DIRECTORY_NAME = "JetStream-#{VERSION}"
 

trunk/Source/JavaScriptCore/bytecode/CodeBlock.h

@@ -538 +538 @@
         return hasExitSite(locker, site);
     }
+    
+    size_t numberOfExitSites() const
+    {
+        ConcurrentJITLocker locker(m_lock);
+        return m_exitProfile.size();
+    }
 
     DFG::ExitProfile& exitProfile() { return m_exitProfile; }

trunk/Source/JavaScriptCore/bytecode/DFGExitProfile.cpp

@@ -45 +45 @@
         return true;
     }
-    
-    // Don't add it if it's already there. This is O(n), but that's OK, because we
-    // know that the total number of places where code exits tends to not be large,
-    // and this code is only used when recompilation is triggered.
-    for (unsigned i = 0; i < m_frequentExitSites->size(); ++i) {
-        if (m_frequentExitSites->at(i) == site)
-            return false;
-    }
-    
+
+    // Always add even if it's a duplicate. The side-effect of doing this is it gives us a
+    // complete count of the number of times we've exited here. 
     m_frequentExitSites->append(site);
     return true;

trunk/Source/JavaScriptCore/bytecode/DFGExitProfile.h

@@ -180 +180 @@
     }
     
+    size_t size() const
+    {
+        if (!m_frequentExitSites)
+            return 0;
+        return m_frequentExitSites->size();
+    }
+    
 private:
     friend class QueryableExitProfile;

trunk/Source/JavaScriptCore/dfg/DFGByteCodeParser.cpp

@@ -1274 +1274 @@
     
     // It might be possible to inline.
-    return codeBlock->instructionCount();
+    return codeBlock->instructionCount() * pow(1 + codeBlock->numberOfExitSites(), Options::exitSitePowerForInlineCost());
 }
 

trunk/Source/JavaScriptCore/runtime/Options.h

@@ -205 +205 @@
     v(unsigned, maximumOptimizationCandidateInstructionCount, 100000, nullptr) \
     \
-    v(unsigned, maximumFunctionForCallInlineCandidateInstructionCount, 180, nullptr) \
-    v(unsigned, maximumFunctionForClosureCallInlineCandidateInstructionCount, 100, nullptr) \
-    v(unsigned, maximumFunctionForConstructInlineCandidateInstructionCount, 100, nullptr) \
+    v(unsigned, maximumFunctionForCallInlineCandidateInstructionCount, 250, nullptr) \
+    v(unsigned, maximumFunctionForClosureCallInlineCandidateInstructionCount, 180, nullptr) \
+    v(unsigned, maximumFunctionForConstructInlineCandidateInstructionCount, 180, nullptr) \
     \
     v(unsigned, maximumFTLCandidateInstructionCount, 20000, nullptr) \
+    \
+    v(double, exitSitePowerForInlineCost, 0.5, nullptr) \
     \
     /* Depth of inline stack, so 1 = no inlining, 2 = one level, etc. */ \

trunk/Tools/Scripts/run-jsc-benchmarks

@@ -2844 +2844 @@
   ARGV.each {
     | vm |
-    if vm =~ /([a-zA-Z0-9_ ]+):/
+    if vm =~ /([a-zA-Z0-9_ .]+):/
       name = $1
       nameKind = :given
@@ -2853 +2853 @@
     end
     envs = []
-    while vm =~ /([a-zA-Z0-9_]+)=([a-zA-Z0-9_:]+):/
+    while vm =~ /([a-zA-Z0-9_]+)=([a-zA-Z0-9_:.]+):/
       envs << [$1, $2]
       vm = $~.post_match