Changeset 248878 in webkit


Ignore:
Timestamp:
Aug 19, 2019, 5:21:29 PM (6 years ago)
Author:
ysuzuki@apple.com
Message:

[JSC] OSR entry to Wasm OMG
​https://bugs.webkit.org/show_bug.cgi?id=200362

Reviewed by Michael Saboff.

JSTests:

  • wasm/stress/osr-entry-basic.js: Added.

(instance.exports.loop):

  • wasm/stress/osr-entry-many-locals-f32.js: Added.
  • wasm/stress/osr-entry-many-locals-f64.js: Added.
  • wasm/stress/osr-entry-many-locals-i32.js: Added.
  • wasm/stress/osr-entry-many-locals-i64.js: Added.
  • wasm/stress/osr-entry-many-stacks-f32.js: Added.
  • wasm/stress/osr-entry-many-stacks-f64.js: Added.
  • wasm/stress/osr-entry-many-stacks-i32.js: Added.
  • wasm/stress/osr-entry-many-stacks-i64.js: Added.

Source/JavaScriptCore:

This patch implements Wasm OSR entry mechanism from BBQ tier to OMG tier.
We found that one of JetStream2 test heavily relies on OSR entry feature. gcc-loops-wasm consumes
most of time in BBQ tier since one of the function takes significantly long time. And since we did
not have OSR entry feature, we cannot use OMG function until that BBQ function finishes.

To implement Wasm OSR feature, we first capture all locals and stacks in the patchpoint to generate
the stackmap. Once the threshold is crossed, the patchpoint calls MacroAssembler::probe feature to
capture whole register context, and C++ runtime function reads stackmap and Probe::Context to perform
OSR entry. This patch intentionally makes OSR entry written in C++ runtime side as much as possible
to make it easily reusable for the other tiers. For example, we are planning to introduce Wasm interpreter,
and it can easily use this tier-up function. Because of this simplicity, this generic implementation can
cover both BBQ Air and BBQ B3 tier-up features. So, in the feature, it is possible that we revive BBQ B3,
and construct the wasm pipeline like, interpreter->BBQ B3->OMG B3.

To generate OMG code for OSR entry, we add a new mode OMGForOSREntry, which mimics the FTLForOSREntry.
In FTLForOSREntry, we cut unrelated blocks including the usual entry point in DFG tier and later convert
graph to SSA. This is possible because DFG is not SSA. On the other hand, B3 is SSA and we cannot take the
same thing without a hack.

This patch introduce a hack: making all wasm locals and stack values B3::Variable for OMGForOSREntry mode.
Then, we can cut blocks easily and we can generate the B3 graph without doing reachability analysis from the
OSR entry point. B3 will remove unreachable blocks later.

Tier-up function mimics DFG->FTL OSR entry heuristics and threshold as much as possible. And this patch adjusts
the tier-up count threshold to make it close to DFG->FTL ones. Wasm tier-up is now using ExecutionCounter, which
is inherited from Wasm::TierUpCount. Since wasm can execute concurrently, the tier-up counter can be racily updated.
But this is OK in practice. Even if we see some more tier-up function calls or tier-up function calls are delayed,
the critical part is guarded by a lock in tier-up function.

In iMac Pro, it shows ~4x runtime improvement for gcc-loops-wasm. On iOS device (iPhone XR), we saw ~2x improvement.

ToT:

HashSet-wasm:Score: 24.6pt stdev=4.6%

:Time:Geometric: 204ms stdev=4.4%
Runtime:Time: 689ms stdev=1.0%
Startup:Time: 60.3ms stdev=8.4%

gcc-loops-wasm:Score: 8.41pt stdev=6.7%

:Time:Geometric: 597ms stdev=6.5%
Runtime:Time: 8.509s stdev=0.7%
Startup:Time: 42ms stdev=12.4%

quicksort-wasm:Score: 347pt stdev=20.9%

:Time:Geometric: 15ms stdev=18.6%
Runtime:Time: 28.2ms stdev=7.9%
Startup:Time: 8.2ms stdev=35.0%

richards-wasm:Score: 77.6pt stdev=4.5%

:Time:Geometric: 64.6ms stdev=4.4%
Runtime:Time: 544ms stdev=3.3%
Startup:Time: 7.67ms stdev=6.7%

tsf-wasm:Score: 47.9pt stdev=4.5%

:Time:Geometric: 104ms stdev=4.8%
Runtime:Time: 259ms stdev=4.4%
Startup:Time: 42.2ms stdev=8.5%

Patched:

HashSet-wasm:Score: 24.1pt stdev=4.1%

:Time:Geometric: 208ms stdev=4.1%
Runtime:Time: 684ms stdev=1.1%
Startup:Time: 63.2ms stdev=8.1%

gcc-loops-wasm:Score: 15.7pt stdev=5.1%

:Time:Geometric: 319ms stdev=5.3%
Runtime:Time: 2.491s stdev=0.7%
Startup:Time: 41ms stdev=11.0%

quicksort-wasm:Score: 353pt stdev=13.7%

:Time:Geometric: 14ms stdev=12.7%
Runtime:Time: 26.2ms stdev=2.9%
Startup:Time: 8.0ms stdev=23.7%

richards-wasm:Score: 77.4pt stdev=5.3%

:Time:Geometric: 64.7ms stdev=5.3%
Runtime:Time: 536ms stdev=1.5%
Startup:Time: 7.83ms stdev=9.6%

tsf-wasm:Score: 47.3pt stdev=5.7%

:Time:Geometric: 106ms stdev=6.1%
Runtime:Time: 250ms stdev=3.5%
Startup:Time: 45ms stdev=13.8%

  • JavaScriptCore.xcodeproj/project.pbxproj:
  • Sources.txt:
  • assembler/MacroAssemblerARM64.h:

(JSC::MacroAssemblerARM64::branchAdd32):

  • b3/B3ValueRep.h:
  • bytecode/CodeBlock.h:
  • bytecode/ExecutionCounter.cpp:

(JSC::applyMemoryUsageHeuristics):
(JSC::ExecutionCounter<countingVariant>::setThreshold):

  • bytecode/ExecutionCounter.h:

(JSC::ExecutionCounter::clippedThreshold):

  • dfg/DFGJITCode.h:
  • dfg/DFGOperations.cpp:
  • jit/AssemblyHelpers.h:

(JSC::AssemblyHelpers::prologueStackPointerDelta):

  • runtime/Options.h:
  • wasm/WasmAirIRGenerator.cpp:

(JSC::Wasm::AirIRGenerator::createStack):
(JSC::Wasm::AirIRGenerator::emitPatchpoint):
(JSC::Wasm::AirIRGenerator::outerLoopIndex const):
(JSC::Wasm::AirIRGenerator::AirIRGenerator):
(JSC::Wasm::AirIRGenerator::emitEntryTierUpCheck):
(JSC::Wasm::AirIRGenerator::emitLoopTierUpCheck):
(JSC::Wasm::AirIRGenerator::addLoop):
(JSC::Wasm::AirIRGenerator::addElse):
(JSC::Wasm::AirIRGenerator::addBranch):
(JSC::Wasm::AirIRGenerator::addSwitch):
(JSC::Wasm::AirIRGenerator::endBlock):
(JSC::Wasm::AirIRGenerator::addEndToUnreachable):
(JSC::Wasm::AirIRGenerator::unifyValuesWithBlock):
(JSC::Wasm::AirIRGenerator::dump):
(JSC::Wasm::AirIRGenerator::emitTierUpCheck): Deleted.

  • wasm/WasmB3IRGenerator.cpp:

(JSC::Wasm::B3IRGenerator::Stack::Stack):
(JSC::Wasm::B3IRGenerator::Stack::append):
(JSC::Wasm::B3IRGenerator::Stack::takeLast):
(JSC::Wasm::B3IRGenerator::Stack::last):
(JSC::Wasm::B3IRGenerator::Stack::size const):
(JSC::Wasm::B3IRGenerator::Stack::isEmpty const):
(JSC::Wasm::B3IRGenerator::Stack::convertToExpressionList):
(JSC::Wasm::B3IRGenerator::Stack::at const):
(JSC::Wasm::B3IRGenerator::Stack::variableAt const):
(JSC::Wasm::B3IRGenerator::Stack::shrink):
(JSC::Wasm::B3IRGenerator::Stack::swap):
(JSC::Wasm::B3IRGenerator::Stack::dump const):
(JSC::Wasm::B3IRGenerator::createStack):
(JSC::Wasm::B3IRGenerator::outerLoopIndex const):
(JSC::Wasm::B3IRGenerator::B3IRGenerator):
(JSC::Wasm::B3IRGenerator::emitEntryTierUpCheck):
(JSC::Wasm::B3IRGenerator::emitLoopTierUpCheck):
(JSC::Wasm::B3IRGenerator::addLoop):
(JSC::Wasm::B3IRGenerator::addElse):
(JSC::Wasm::B3IRGenerator::addBranch):
(JSC::Wasm::B3IRGenerator::addSwitch):
(JSC::Wasm::B3IRGenerator::endBlock):
(JSC::Wasm::B3IRGenerator::addEndToUnreachable):
(JSC::Wasm::B3IRGenerator::unifyValuesWithBlock):
(JSC::Wasm::B3IRGenerator::dump):
(JSC::Wasm::parseAndCompile):
(JSC::Wasm::B3IRGenerator::emitTierUpCheck): Deleted.
(JSC::Wasm::dumpExpressionStack): Deleted.

  • wasm/WasmB3IRGenerator.h:
  • wasm/WasmBBQPlan.cpp:

(JSC::Wasm::BBQPlan::compileFunctions):

  • wasm/WasmBBQPlan.h:
  • wasm/WasmBBQPlanInlines.h:

(JSC::Wasm::BBQPlan::initializeCallees):

  • wasm/WasmCallee.h:
  • wasm/WasmCodeBlock.cpp:

(JSC::Wasm::CodeBlock::CodeBlock):

  • wasm/WasmCodeBlock.h:

(JSC::Wasm::CodeBlock::wasmBBQCalleeFromFunctionIndexSpace):
(JSC::Wasm::CodeBlock::entrypointLoadLocationFromFunctionIndexSpace):
(JSC::Wasm::CodeBlock::tierUpCount): Deleted.

  • wasm/WasmCompilationMode.cpp:

(JSC::Wasm::makeString):

  • wasm/WasmCompilationMode.h:
  • wasm/WasmContext.cpp: Copied from Source/JavaScriptCore/wasm/WasmCompilationMode.cpp.

(JSC::Wasm::Context::scratchBufferForSize):

  • wasm/WasmContext.h:
  • wasm/WasmContextInlines.h:

(JSC::Wasm::Context::tryLoadInstanceFromTLS):

  • wasm/WasmFunctionParser.h:

(JSC::Wasm::FunctionParser<Context>::FunctionParser):
(JSC::Wasm::FunctionParser<Context>::parseBody):
(JSC::Wasm::FunctionParser<Context>::parseExpression):

  • wasm/WasmOMGForOSREntryPlan.cpp: Copied from Source/JavaScriptCore/wasm/WasmOMGPlan.cpp.

(JSC::Wasm::OMGForOSREntryPlan::OMGForOSREntryPlan):
(JSC::Wasm::OMGForOSREntryPlan::work):

  • wasm/WasmOMGForOSREntryPlan.h: Copied from Source/JavaScriptCore/wasm/WasmOMGPlan.h.
  • wasm/WasmOMGPlan.cpp:

(JSC::Wasm::OMGPlan::work):
(JSC::Wasm::OMGPlan::runForIndex): Deleted.

  • wasm/WasmOMGPlan.h:
  • wasm/WasmOSREntryData.h: Copied from Source/JavaScriptCore/wasm/WasmContext.h.

(JSC::Wasm::OSREntryValue::OSREntryValue):
(JSC::Wasm::OSREntryValue::type const):
(JSC::Wasm::OSREntryData::OSREntryData):
(JSC::Wasm::OSREntryData::functionIndex const):
(JSC::Wasm::OSREntryData::loopIndex const):
(JSC::Wasm::OSREntryData::values):

  • wasm/WasmOperations.cpp: Added.

(JSC::Wasm::shouldTriggerOMGCompile):
(JSC::Wasm::triggerOMGReplacementCompile):
(JSC::Wasm::doOSREntry):
(JSC::Wasm::triggerOSREntryNow):
(JSC::Wasm::triggerTierUpNow):

  • wasm/WasmOperations.h: Copied from Source/JavaScriptCore/wasm/WasmCompilationMode.h.
  • wasm/WasmThunks.cpp:

(JSC::Wasm::triggerOMGEntryTierUpThunkGenerator):
(JSC::Wasm::triggerOMGTierUpThunkGenerator): Deleted.

  • wasm/WasmThunks.h:
  • wasm/WasmTierUpCount.cpp: Copied from Source/JavaScriptCore/wasm/WasmCompilationMode.cpp.

(JSC::Wasm::TierUpCount::TierUpCount):
(JSC::Wasm::TierUpCount::addOSREntryData):

  • wasm/WasmTierUpCount.h:

(JSC::Wasm::TierUpCount::loopIncrement):
(JSC::Wasm::TierUpCount::functionEntryIncrement):
(JSC::Wasm::TierUpCount::osrEntryTriggers):
(JSC::Wasm::TierUpCount::outerLoops):
(JSC::Wasm::TierUpCount::getLock):
(JSC::Wasm::TierUpCount::optimizeAfterWarmUp):
(JSC::Wasm::TierUpCount::checkIfOptimizationThresholdReached):
(JSC::Wasm::TierUpCount::dontOptimizeAnytimeSoon):
(JSC::Wasm::TierUpCount::optimizeNextInvocation):
(JSC::Wasm::TierUpCount::optimizeSoon):
(JSC::Wasm::TierUpCount::setOptimizationThresholdBasedOnCompilationResult):
(JSC::Wasm::TierUpCount::TierUpCount): Deleted.
(JSC::Wasm::TierUpCount::loopDecrement): Deleted.
(JSC::Wasm::TierUpCount::functionEntryDecrement): Deleted.
(JSC::Wasm::TierUpCount::shouldStartTierUp): Deleted.
(JSC::Wasm::TierUpCount::count): Deleted.

  • wasm/WasmValidate.cpp:

(JSC::Wasm::Validate::createStack):
(JSC::Wasm::Validate::addLoop):
(JSC::Wasm::Validate::addElse):
(JSC::Wasm::Validate::checkBranchTarget):
(JSC::Wasm::Validate::addBranch):
(JSC::Wasm::Validate::addSwitch):
(JSC::Wasm::Validate::endBlock):
(JSC::Wasm::Validate::unify):
(JSC::Wasm::dumpExpressionStack):
(JSC::Wasm::Validate::dump):

Tools:

  • Scripts/run-jsc-stress-tests:
Location:
trunk
Files:
10 added
35 edited
6 copied

Legend:

Unmodified
Added
Removed

  • trunk/JSTests/ChangeLog

    r248876 r248878  
     12019-08-19  Yusuke Suzuki  <ysuzuki@apple.com>
     2
     3        [JSC] OSR entry to Wasm OMG
     4        https://bugs.webkit.org/show_bug.cgi?id=200362
     5
     6        Reviewed by Michael Saboff.
     7
     8        * wasm/stress/osr-entry-basic.js: Added.
     9        (instance.exports.loop):
     10        * wasm/stress/osr-entry-many-locals-f32.js: Added.
     11        * wasm/stress/osr-entry-many-locals-f64.js: Added.
     12        * wasm/stress/osr-entry-many-locals-i32.js: Added.
     13        * wasm/stress/osr-entry-many-locals-i64.js: Added.
     14        * wasm/stress/osr-entry-many-stacks-f32.js: Added.
     15        * wasm/stress/osr-entry-many-stacks-f64.js: Added.
     16        * wasm/stress/osr-entry-many-stacks-i32.js: Added.
     17        * wasm/stress/osr-entry-many-stacks-i64.js: Added.
     18
    1192019-08-19  Alexey Shvayka  <shvaikalesh@gmail.com>
    220

  • trunk/Source/JavaScriptCore/ChangeLog

    r248876 r248878  
     12019-08-19  Yusuke Suzuki  <ysuzuki@apple.com>
     2
     3        [JSC] OSR entry to Wasm OMG
     4        https://bugs.webkit.org/show_bug.cgi?id=200362
     5
     6        Reviewed by Michael Saboff.
     7
     8        This patch implements Wasm OSR entry mechanism from BBQ tier to OMG tier.
     9        We found that one of JetStream2 test heavily relies on OSR entry feature. gcc-loops-wasm consumes
     10        most of time in BBQ tier since one of the function takes significantly long time. And since we did
     11        not have OSR entry feature, we cannot use OMG function until that BBQ function finishes.
     12
     13        To implement Wasm OSR feature, we first capture all locals and stacks in the patchpoint to generate
     14        the stackmap. Once the threshold is crossed, the patchpoint calls `MacroAssembler::probe` feature to
     15        capture whole register context, and C++ runtime function reads stackmap and Probe::Context to perform
     16        OSR entry. This patch intentionally makes OSR entry written in C++ runtime side as much as possible
     17        to make it easily reusable for the other tiers. For example, we are planning to introduce Wasm interpreter,
     18        and it can easily use this tier-up function. Because of this simplicity, this generic implementation can
     19        cover both BBQ Air and BBQ B3 tier-up features. So, in the feature, it is possible that we revive BBQ B3,
     20        and construct the wasm pipeline like, interpreter->BBQ B3->OMG B3.
     21
     22        To generate OMG code for OSR entry, we add a new mode OMGForOSREntry, which mimics the FTLForOSREntry.
     23        In FTLForOSREntry, we cut unrelated blocks including the usual entry point in DFG tier and later convert
     24        graph to SSA. This is possible because DFG is not SSA. On the other hand, B3 is SSA and we cannot take the
     25        same thing without a hack.
     26
     27        This patch introduce a hack: making all wasm locals and stack values B3::Variable for OMGForOSREntry mode.
     28        Then, we can cut blocks easily and we can generate the B3 graph without doing reachability analysis from the
     29        OSR entry point. B3 will remove unreachable blocks later.
     30
     31        Tier-up function mimics DFG->FTL OSR entry heuristics and threshold as much as possible. And this patch adjusts
     32        the tier-up count threshold to make it close to DFG->FTL ones. Wasm tier-up is now using ExecutionCounter, which
     33        is inherited from Wasm::TierUpCount. Since wasm can execute concurrently, the tier-up counter can be racily updated.
     34        But this is OK in practice. Even if we see some more tier-up function calls or tier-up function calls are delayed,
     35        the critical part is guarded by a lock in tier-up function.
     36
     37        In iMac Pro, it shows ~4x runtime improvement for gcc-loops-wasm. On iOS device (iPhone XR), we saw ~2x improvement.
     38
     39            ToT:
     40                HashSet-wasm:Score: 24.6pt stdev=4.6%
     41                            :Time:Geometric: 204ms stdev=4.4%
     42                            Runtime:Time: 689ms stdev=1.0%
     43                            Startup:Time: 60.3ms stdev=8.4%
     44                gcc-loops-wasm:Score: 8.41pt stdev=6.7%
     45                              :Time:Geometric: 597ms stdev=6.5%
     46                              Runtime:Time: 8.509s stdev=0.7%
     47                              Startup:Time: 42ms stdev=12.4%
     48                quicksort-wasm:Score: 347pt stdev=20.9%
     49                              :Time:Geometric: 15ms stdev=18.6%
     50                              Runtime:Time: 28.2ms stdev=7.9%
     51                              Startup:Time: 8.2ms stdev=35.0%
     52                richards-wasm:Score: 77.6pt stdev=4.5%
     53                             :Time:Geometric: 64.6ms stdev=4.4%
     54                             Runtime:Time: 544ms stdev=3.3%
     55                             Startup:Time: 7.67ms stdev=6.7%
     56                tsf-wasm:Score: 47.9pt stdev=4.5%
     57                        :Time:Geometric: 104ms stdev=4.8%
     58                        Runtime:Time: 259ms stdev=4.4%
     59                        Startup:Time: 42.2ms stdev=8.5%
     60
     61            Patched:
     62                HashSet-wasm:Score: 24.1pt stdev=4.1%
     63                            :Time:Geometric: 208ms stdev=4.1%
     64                            Runtime:Time: 684ms stdev=1.1%
     65                            Startup:Time: 63.2ms stdev=8.1%
     66                gcc-loops-wasm:Score: 15.7pt stdev=5.1%
     67                              :Time:Geometric: 319ms stdev=5.3%
     68                              Runtime:Time: 2.491s stdev=0.7%
     69                              Startup:Time: 41ms stdev=11.0%
     70                quicksort-wasm:Score: 353pt stdev=13.7%
     71                              :Time:Geometric: 14ms stdev=12.7%
     72                              Runtime:Time: 26.2ms stdev=2.9%
     73                              Startup:Time: 8.0ms stdev=23.7%
     74                richards-wasm:Score: 77.4pt stdev=5.3%
     75                             :Time:Geometric: 64.7ms stdev=5.3%
     76                             Runtime:Time: 536ms stdev=1.5%
     77                             Startup:Time: 7.83ms stdev=9.6%
     78                tsf-wasm:Score: 47.3pt stdev=5.7%
     79                        :Time:Geometric: 106ms stdev=6.1%
     80                        Runtime:Time: 250ms stdev=3.5%
     81                        Startup:Time: 45ms stdev=13.8%
     82
     83        * JavaScriptCore.xcodeproj/project.pbxproj:
     84        * Sources.txt:
     85        * assembler/MacroAssemblerARM64.h:
     86        (JSC::MacroAssemblerARM64::branchAdd32):
     87        * b3/B3ValueRep.h:
     88        * bytecode/CodeBlock.h:
     89        * bytecode/ExecutionCounter.cpp:
     90        (JSC::applyMemoryUsageHeuristics):
     91        (JSC::ExecutionCounter<countingVariant>::setThreshold):
     92        * bytecode/ExecutionCounter.h:
     93        (JSC::ExecutionCounter::clippedThreshold):
     94        * dfg/DFGJITCode.h:
     95        * dfg/DFGOperations.cpp:
     96        * jit/AssemblyHelpers.h:
     97        (JSC::AssemblyHelpers::prologueStackPointerDelta):
     98        * runtime/Options.h:
     99        * wasm/WasmAirIRGenerator.cpp:
     100        (JSC::Wasm::AirIRGenerator::createStack):
     101        (JSC::Wasm::AirIRGenerator::emitPatchpoint):
     102        (JSC::Wasm::AirIRGenerator::outerLoopIndex const):
     103        (JSC::Wasm::AirIRGenerator::AirIRGenerator):
     104        (JSC::Wasm::AirIRGenerator::emitEntryTierUpCheck):
     105        (JSC::Wasm::AirIRGenerator::emitLoopTierUpCheck):
     106        (JSC::Wasm::AirIRGenerator::addLoop):
     107        (JSC::Wasm::AirIRGenerator::addElse):
     108        (JSC::Wasm::AirIRGenerator::addBranch):
     109        (JSC::Wasm::AirIRGenerator::addSwitch):
     110        (JSC::Wasm::AirIRGenerator::endBlock):
     111        (JSC::Wasm::AirIRGenerator::addEndToUnreachable):
     112        (JSC::Wasm::AirIRGenerator::unifyValuesWithBlock):
     113        (JSC::Wasm::AirIRGenerator::dump):
     114        (JSC::Wasm::AirIRGenerator::emitTierUpCheck): Deleted.
     115        * wasm/WasmB3IRGenerator.cpp:
     116        (JSC::Wasm::B3IRGenerator::Stack::Stack):
     117        (JSC::Wasm::B3IRGenerator::Stack::append):
     118        (JSC::Wasm::B3IRGenerator::Stack::takeLast):
     119        (JSC::Wasm::B3IRGenerator::Stack::last):
     120        (JSC::Wasm::B3IRGenerator::Stack::size const):
     121        (JSC::Wasm::B3IRGenerator::Stack::isEmpty const):
     122        (JSC::Wasm::B3IRGenerator::Stack::convertToExpressionList):
     123        (JSC::Wasm::B3IRGenerator::Stack::at const):
     124        (JSC::Wasm::B3IRGenerator::Stack::variableAt const):
     125        (JSC::Wasm::B3IRGenerator::Stack::shrink):
     126        (JSC::Wasm::B3IRGenerator::Stack::swap):
     127        (JSC::Wasm::B3IRGenerator::Stack::dump const):
     128        (JSC::Wasm::B3IRGenerator::createStack):
     129        (JSC::Wasm::B3IRGenerator::outerLoopIndex const):
     130        (JSC::Wasm::B3IRGenerator::B3IRGenerator):
     131        (JSC::Wasm::B3IRGenerator::emitEntryTierUpCheck):
     132        (JSC::Wasm::B3IRGenerator::emitLoopTierUpCheck):
     133        (JSC::Wasm::B3IRGenerator::addLoop):
     134        (JSC::Wasm::B3IRGenerator::addElse):
     135        (JSC::Wasm::B3IRGenerator::addBranch):
     136        (JSC::Wasm::B3IRGenerator::addSwitch):
     137        (JSC::Wasm::B3IRGenerator::endBlock):
     138        (JSC::Wasm::B3IRGenerator::addEndToUnreachable):
     139        (JSC::Wasm::B3IRGenerator::unifyValuesWithBlock):
     140        (JSC::Wasm::B3IRGenerator::dump):
     141        (JSC::Wasm::parseAndCompile):
     142        (JSC::Wasm::B3IRGenerator::emitTierUpCheck): Deleted.
     143        (JSC::Wasm::dumpExpressionStack): Deleted.
     144        * wasm/WasmB3IRGenerator.h:
     145        * wasm/WasmBBQPlan.cpp:
     146        (JSC::Wasm::BBQPlan::compileFunctions):
     147        * wasm/WasmBBQPlan.h:
     148        * wasm/WasmBBQPlanInlines.h:
     149        (JSC::Wasm::BBQPlan::initializeCallees):
     150        * wasm/WasmCallee.h:
     151        * wasm/WasmCodeBlock.cpp:
     152        (JSC::Wasm::CodeBlock::CodeBlock):
     153        * wasm/WasmCodeBlock.h:
     154        (JSC::Wasm::CodeBlock::wasmBBQCalleeFromFunctionIndexSpace):
     155        (JSC::Wasm::CodeBlock::entrypointLoadLocationFromFunctionIndexSpace):
     156        (JSC::Wasm::CodeBlock::tierUpCount): Deleted.
     157        * wasm/WasmCompilationMode.cpp:
     158        (JSC::Wasm::makeString):
     159        * wasm/WasmCompilationMode.h:
     160        * wasm/WasmContext.cpp: Copied from Source/JavaScriptCore/wasm/WasmCompilationMode.cpp.
     161        (JSC::Wasm::Context::scratchBufferForSize):
     162        * wasm/WasmContext.h:
     163        * wasm/WasmContextInlines.h:
     164        (JSC::Wasm::Context::tryLoadInstanceFromTLS):
     165        * wasm/WasmFunctionParser.h:
     166        (JSC::Wasm::FunctionParser<Context>::FunctionParser):
     167        (JSC::Wasm::FunctionParser<Context>::parseBody):
     168        (JSC::Wasm::FunctionParser<Context>::parseExpression):
     169        * wasm/WasmOMGForOSREntryPlan.cpp: Copied from Source/JavaScriptCore/wasm/WasmOMGPlan.cpp.
     170        (JSC::Wasm::OMGForOSREntryPlan::OMGForOSREntryPlan):
     171        (JSC::Wasm::OMGForOSREntryPlan::work):
     172        * wasm/WasmOMGForOSREntryPlan.h: Copied from Source/JavaScriptCore/wasm/WasmOMGPlan.h.
     173        * wasm/WasmOMGPlan.cpp:
     174        (JSC::Wasm::OMGPlan::work):
     175        (JSC::Wasm::OMGPlan::runForIndex): Deleted.
     176        * wasm/WasmOMGPlan.h:
     177        * wasm/WasmOSREntryData.h: Copied from Source/JavaScriptCore/wasm/WasmContext.h.
     178        (JSC::Wasm::OSREntryValue::OSREntryValue):
     179        (JSC::Wasm::OSREntryValue::type const):
     180        (JSC::Wasm::OSREntryData::OSREntryData):
     181        (JSC::Wasm::OSREntryData::functionIndex const):
     182        (JSC::Wasm::OSREntryData::loopIndex const):
     183        (JSC::Wasm::OSREntryData::values):
     184        * wasm/WasmOperations.cpp: Added.
     185        (JSC::Wasm::shouldTriggerOMGCompile):
     186        (JSC::Wasm::triggerOMGReplacementCompile):
     187        (JSC::Wasm::doOSREntry):
     188        (JSC::Wasm::triggerOSREntryNow):
     189        (JSC::Wasm::triggerTierUpNow):
     190        * wasm/WasmOperations.h: Copied from Source/JavaScriptCore/wasm/WasmCompilationMode.h.
     191        * wasm/WasmThunks.cpp:
     192        (JSC::Wasm::triggerOMGEntryTierUpThunkGenerator):
     193        (JSC::Wasm::triggerOMGTierUpThunkGenerator): Deleted.
     194        * wasm/WasmThunks.h:
     195        * wasm/WasmTierUpCount.cpp: Copied from Source/JavaScriptCore/wasm/WasmCompilationMode.cpp.
     196        (JSC::Wasm::TierUpCount::TierUpCount):
     197        (JSC::Wasm::TierUpCount::addOSREntryData):
     198        * wasm/WasmTierUpCount.h:
     199        (JSC::Wasm::TierUpCount::loopIncrement):
     200        (JSC::Wasm::TierUpCount::functionEntryIncrement):
     201        (JSC::Wasm::TierUpCount::osrEntryTriggers):
     202        (JSC::Wasm::TierUpCount::outerLoops):
     203        (JSC::Wasm::TierUpCount::getLock):
     204        (JSC::Wasm::TierUpCount::optimizeAfterWarmUp):
     205        (JSC::Wasm::TierUpCount::checkIfOptimizationThresholdReached):
     206        (JSC::Wasm::TierUpCount::dontOptimizeAnytimeSoon):
     207        (JSC::Wasm::TierUpCount::optimizeNextInvocation):
     208        (JSC::Wasm::TierUpCount::optimizeSoon):
     209        (JSC::Wasm::TierUpCount::setOptimizationThresholdBasedOnCompilationResult):
     210        (JSC::Wasm::TierUpCount::TierUpCount): Deleted.
     211        (JSC::Wasm::TierUpCount::loopDecrement): Deleted.
     212        (JSC::Wasm::TierUpCount::functionEntryDecrement): Deleted.
     213        (JSC::Wasm::TierUpCount::shouldStartTierUp): Deleted.
     214        (JSC::Wasm::TierUpCount::count): Deleted.
     215        * wasm/WasmValidate.cpp:
     216        (JSC::Wasm::Validate::createStack):
     217        (JSC::Wasm::Validate::addLoop):
     218        (JSC::Wasm::Validate::addElse):
     219        (JSC::Wasm::Validate::checkBranchTarget):
     220        (JSC::Wasm::Validate::addBranch):
     221        (JSC::Wasm::Validate::addSwitch):
     222        (JSC::Wasm::Validate::endBlock):
     223        (JSC::Wasm::Validate::unify):
     224        (JSC::Wasm::dumpExpressionStack):
     225        (JSC::Wasm::Validate::dump):
     226
    12272019-08-19  Alexey Shvayka  <shvaikalesh@gmail.com>
    2228

  • trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj

    r248866 r248878  
    18111811                E35CA1561DBC3A5F00F83516 /* DOMJITAbstractHeap.h in Headers */ = {isa = PBXBuildFile; fileRef = E35CA1501DBC3A5600F83516 /* DOMJITAbstractHeap.h */; settings = {ATTRIBUTES = (Private, ); }; };
    18121812                E35E03601B7AB43E0073AD2A /* InspectorInstrumentationObject.h in Headers */ = {isa = PBXBuildFile; fileRef = E35E035E1B7AB43E0073AD2A /* InspectorInstrumentationObject.h */; settings = {ATTRIBUTES = (Private, ); }; };
     1813                E36B767022F8D61900D09818 /* WasmOMGForOSREntryPlan.h in Headers */ = {isa = PBXBuildFile; fileRef = E36B766F22F8D61100D09818 /* WasmOMGForOSREntryPlan.h */; };
    18131814                E36CC9472086314F0051FFD6 /* WasmCreationMode.h in Headers */ = {isa = PBXBuildFile; fileRef = E36CC9462086314F0051FFD6 /* WasmCreationMode.h */; settings = {ATTRIBUTES = (Private, ); }; };
    18141815                E3794E761B77EB97005543AE /* ModuleAnalyzer.h in Headers */ = {isa = PBXBuildFile; fileRef = E3794E741B77EB97005543AE /* ModuleAnalyzer.h */; settings = {ATTRIBUTES = (Private, ); }; };
    … …  
    18201821                E39BF39922A2288B00BD183E /* SymbolTableInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = E39BF39822A2288B00BD183E /* SymbolTableInlines.h */; };
    18211822                E39D45F51D39005600B3B377 /* InterpreterInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = E39D9D841D39000600667282 /* InterpreterInlines.h */; settings = {ATTRIBUTES = (Private, ); }; };
     1823                E39D8B2E23021E2600265852 /* WasmOperations.h in Headers */ = {isa = PBXBuildFile; fileRef = E39D8B2D23021E1E00265852 /* WasmOperations.h */; };
    18221824                E39DA4A71B7E8B7C0084F33A /* JSModuleRecord.h in Headers */ = {isa = PBXBuildFile; fileRef = E39DA4A51B7E8B7C0084F33A /* JSModuleRecord.h */; settings = {ATTRIBUTES = (Private, ); }; };
    18231825                E39EEAF322812450008474F4 /* ObjectToStringAdaptiveStructureWatchpoint.h in Headers */ = {isa = PBXBuildFile; fileRef = E39EEAF22281244C008474F4 /* ObjectToStringAdaptiveStructureWatchpoint.h */; };
    … …  
    18311833                E3BFD0BC1DAF808E0065DEA2 /* AccessCaseSnippetParams.h in Headers */ = {isa = PBXBuildFile; fileRef = E3BFD0BA1DAF807C0065DEA2 /* AccessCaseSnippetParams.h */; };
    18321834                E3C295DD1ED2CBDA00D3016F /* ObjectPropertyChangeAdaptiveWatchpoint.h in Headers */ = {isa = PBXBuildFile; fileRef = E3C295DC1ED2CBAA00D3016F /* ObjectPropertyChangeAdaptiveWatchpoint.h */; };
     1835                E3C694B323026877006FBE42 /* WasmOSREntryData.h in Headers */ = {isa = PBXBuildFile; fileRef = E3C694B123026873006FBE42 /* WasmOSREntryData.h */; };
    18331836                E3C79CAB1DB9A4DC00D1ECA4 /* DOMJITEffect.h in Headers */ = {isa = PBXBuildFile; fileRef = E3C79CAA1DB9A4D600D1ECA4 /* DOMJITEffect.h */; settings = {ATTRIBUTES = (Private, ); }; };
    18341837                E3D239C91B829C1C00BBEF67 /* JSModuleEnvironment.h in Headers */ = {isa = PBXBuildFile; fileRef = E3D239C71B829C1C00BBEF67 /* JSModuleEnvironment.h */; settings = {ATTRIBUTES = (Private, ); }; };
    … …  
    49034906                E35E035E1B7AB43E0073AD2A /* InspectorInstrumentationObject.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = InspectorInstrumentationObject.h; sourceTree = "<group>"; };
    49044907                E35E03611B7AB4850073AD2A /* InspectorInstrumentationObject.js */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.javascript; path = InspectorInstrumentationObject.js; sourceTree = "<group>"; };
     4908                E36B766E22F8D61100D09818 /* WasmOMGForOSREntryPlan.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = WasmOMGForOSREntryPlan.cpp; sourceTree = "<group>"; };
     4909                E36B766F22F8D61100D09818 /* WasmOMGForOSREntryPlan.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = WasmOMGForOSREntryPlan.h; sourceTree = "<group>"; };
    49054910                E36CC9462086314F0051FFD6 /* WasmCreationMode.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WasmCreationMode.h; sourceTree = "<group>"; };
    49064911                E3794E731B77EB97005543AE /* ModuleAnalyzer.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ModuleAnalyzer.cpp; sourceTree = "<group>"; };
    … …  
    49174922                E38D999B221B789F00D50474 /* JSNonDestructibleProxy.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = JSNonDestructibleProxy.cpp; sourceTree = "<group>"; };
    49184923                E39006202208BFC3001019CF /* SubspaceAccess.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = SubspaceAccess.h; sourceTree = "<group>"; };
     4924                E3915C062309682900CB2561 /* WasmContext.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = WasmContext.cpp; sourceTree = "<group>"; };
    49194925                E393ADD71FE702CC0022D681 /* WeakMapImplInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WeakMapImplInlines.h; sourceTree = "<group>"; };
    49204926                E3963CEC1B73F75000EB4CE5 /* NodesAnalyzeModule.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = NodesAnalyzeModule.cpp; sourceTree = "<group>"; };
    49214927                E39BF39822A2288B00BD183E /* SymbolTableInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = SymbolTableInlines.h; sourceTree = "<group>"; };
     4928                E39D8B2C23021E1E00265852 /* WasmOperations.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = WasmOperations.cpp; sourceTree = "<group>"; };
     4929                E39D8B2D23021E1E00265852 /* WasmOperations.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = WasmOperations.h; sourceTree = "<group>"; };
    49224930                E39D9D841D39000600667282 /* InterpreterInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = InterpreterInlines.h; sourceTree = "<group>"; };
    49234931                E39DA4A41B7E8B7C0084F33A /* JSModuleRecord.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = JSModuleRecord.cpp; sourceTree = "<group>"; };
    … …  
    49404948                E3BFD0BA1DAF807C0065DEA2 /* AccessCaseSnippetParams.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AccessCaseSnippetParams.h; sourceTree = "<group>"; };
    49414949                E3C295DC1ED2CBAA00D3016F /* ObjectPropertyChangeAdaptiveWatchpoint.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ObjectPropertyChangeAdaptiveWatchpoint.h; sourceTree = "<group>"; };
     4950                E3C694B123026873006FBE42 /* WasmOSREntryData.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = WasmOSREntryData.h; sourceTree = "<group>"; };
     4951                E3C694B223026874006FBE42 /* WasmTierUpCount.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = WasmTierUpCount.cpp; sourceTree = "<group>"; };
    49424952                E3C79CAA1DB9A4D600D1ECA4 /* DOMJITEffect.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = DOMJITEffect.h; sourceTree = "<group>"; };
    49434953                E3D239C61B829C1C00BBEF67 /* JSModuleEnvironment.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = JSModuleEnvironment.cpp; sourceTree = "<group>"; };
    … …  
    67096719                                E37CFB2D22F27C57009A7B38 /* WasmCompilationMode.cpp */,
    67106720                                E3BD2B7522F275020011765C /* WasmCompilationMode.h */,
     6721                                E3915C062309682900CB2561 /* WasmContext.cpp */,
    67116722                                AD412B321E7B2E8A008AF157 /* WasmContext.h */,
    67126723                                A27958D7FA1142B0AC9E364D /* WasmContextInlines.h */,
    … …  
    67426753                                ADD8FA441EB3077100DF542F /* WasmNameSectionParser.cpp */,
    67436754                                ADD8FA431EB3077100DF542F /* WasmNameSectionParser.h */,
     6755                                E36B766E22F8D61100D09818 /* WasmOMGForOSREntryPlan.cpp */,
     6756                                E36B766F22F8D61100D09818 /* WasmOMGForOSREntryPlan.h */,
    67446757                                5311BD481EA581E500525281 /* WasmOMGPlan.cpp */,
    67456758                                5311BD491EA581E500525281 /* WasmOMGPlan.h */,
    67466759                                53C6FEF01E8AFE0C00B18425 /* WasmOpcodeOrigin.cpp */,
    67476760                                53C6FEEE1E8ADFA900B18425 /* WasmOpcodeOrigin.h */,
     6761                                E39D8B2C23021E1E00265852 /* WasmOperations.cpp */,
     6762                                E39D8B2D23021E1E00265852 /* WasmOperations.h */,
     6763                                E3C694B123026873006FBE42 /* WasmOSREntryData.h */,
    67486764                                ADB6F67C1E15D7500082F384 /* WasmPageCount.cpp */,
    67496765                                79B759731DFA4C600052174C /* WasmPageCount.h */,
    … …  
    67636779                                5250D2CF1E8DA05A0029A932 /* WasmThunks.cpp */,
    67646780                                5250D2D01E8DA05A0029A932 /* WasmThunks.h */,
     6781                                E3C694B223026874006FBE42 /* WasmTierUpCount.cpp */,
    67656782                                53E9E0AE1EAEC45700FEE251 /* WasmTierUpCount.h */,
    67666783                                53FF7F9A1DBFD2B900A26CCC /* WasmValidate.cpp */,
    … …  
    1013310150                                AD7B4B2E1FA3E29800C9DF79 /* WasmNameSection.h in Headers */,
    1013410151                                ADD8FA461EB3079700DF542F /* WasmNameSectionParser.h in Headers */,
     10152                                E36B767022F8D61900D09818 /* WasmOMGForOSREntryPlan.h in Headers */,
    1013510153                                5311BD4B1EA581E500525281 /* WasmOMGPlan.h in Headers */,
    1013610154                                53C6FEEF1E8ADFA900B18425 /* WasmOpcodeOrigin.h in Headers */,
     10155                                E39D8B2E23021E2600265852 /* WasmOperations.h in Headers */,
    1013710156                                53B4BD121F68B32500D2BEA3 /* WasmOps.h in Headers */,
     10157                                E3C694B323026877006FBE42 /* WasmOSREntryData.h in Headers */,
    1013810158                                79B759761DFA4C600052174C /* WasmPageCount.h in Headers */,
    1013910159                                53F40E8D1D5901F20099A1B6 /* WasmParser.h in Headers */,

  • trunk/Source/JavaScriptCore/Sources.txt

    r248187 r248878  
    10011001wasm/WasmCodeBlock.cpp
    10021002wasm/WasmCompilationMode.cpp
     1003wasm/WasmContext.cpp
    10031004wasm/WasmEmbedder.h
    10041005wasm/WasmFaultSignalHandler.cpp
    … …  
    10151016wasm/WasmModuleParser.cpp
    10161017wasm/WasmNameSectionParser.cpp
     1018wasm/WasmOMGForOSREntryPlan.cpp
    10171019wasm/WasmOMGPlan.cpp
    10181020wasm/WasmOpcodeOrigin.cpp
     1021wasm/WasmOperations.cpp
    10191022wasm/WasmPageCount.cpp
    10201023wasm/WasmPlan.cpp
    … …  
    10251028wasm/WasmTable.h
    10261029wasm/WasmThunks.cpp
     1030wasm/WasmTierUpCount.cpp
    10271031wasm/WasmValidate.cpp
    10281032wasm/WasmWorklist.cpp

  • trunk/Source/JavaScriptCore/assembler/MacroAssemblerARM64.h

    r248686 r248878  
    29722972    }
    29732973
     2974    Jump branchAdd32(ResultCondition cond, TrustedImm32 imm, Address address)
     2975    {
     2976        load32(address, getCachedDataTempRegisterIDAndInvalidate());
     2977
     2978        if (isUInt12(imm.m_value))
     2979            m_assembler.add<32, S>(dataTempRegister, dataTempRegister, UInt12(imm.m_value));
     2980        else if (isUInt12(-imm.m_value))
     2981            m_assembler.sub<32, S>(dataTempRegister, dataTempRegister, UInt12(-imm.m_value));
     2982        else {
     2983            move(imm, getCachedMemoryTempRegisterIDAndInvalidate());
     2984            m_assembler.add<32, S>(dataTempRegister, dataTempRegister, memoryTempRegister);
     2985        }
     2986
     2987        store32(dataTempRegister, address);
     2988        return Jump(makeBranch(cond));
     2989    }
     2990
    29742991    Jump branchAdd64(ResultCondition cond, RegisterID op1, RegisterID op2, RegisterID dest)
    29752992    {

  • trunk/Source/JavaScriptCore/b3/B3ValueRep.h

    r248710 r248878  
    4848
    4949class ValueRep {
     50    WTF_MAKE_FAST_ALLOCATED;
    5051public:
    51     enum Kind {
     52    enum Kind : uint8_t {
    5253        // As an input representation, this means that B3 can pick any representation. As an output
    5354        // representation, this means that we don't know. This will only arise as an output
    … …  
    282283
    283284private:
    284     Kind m_kind;
    285285    union U {
    286286        Reg reg;
    … …  
    294294        }
    295295    } u;
     296    Kind m_kind;
    296297};
    297298

  • trunk/Source/JavaScriptCore/bytecode/CodeBlock.h

    r248846 r248878  
    10601060
    10611061#define CODEBLOCK_LOG_EVENT(codeBlock, summary, details) \
    1062     (codeBlock->vm()->logEvent(codeBlock, summary, [&] () { return toCString details; }))
     1062    do { \
     1063        if (codeBlock) \
     1064            (codeBlock->vm()->logEvent(codeBlock, summary, [&] () { return toCString details; })); \
     1065    } while (0)
    10631066
    10641067

  • trunk/Source/JavaScriptCore/bytecode/ExecutionCounter.cpp

    r225618 r248878  
    7676double applyMemoryUsageHeuristics(int32_t value, CodeBlock* codeBlock)
    7777{
     78    double multiplier = 1.0;
     79    if (codeBlock) {
    7880#if ENABLE(JIT)
    79     double multiplier =
    80         ExecutableAllocator::memoryPressureMultiplier(
    81             codeBlock->baselineAlternative()->predictedMachineCodeSize());
    82 #else
    83     // This code path will probably not be taken, but if it is, we fake it.
    84     double multiplier = 1.0;
    85     UNUSED_PARAM(codeBlock);
     81        multiplier =
     82            ExecutableAllocator::memoryPressureMultiplier(
     83                codeBlock->baselineAlternative()->predictedMachineCodeSize());
    8684#endif
     85    }
    8786    ASSERT(multiplier >= 1.0);
    8887    return multiplier * value;
    … …  
    162161    }
    163162
    164     threshold = clippedThreshold(codeBlock->globalObject(), threshold);
     163    threshold = clippedThreshold(codeBlock ? codeBlock->globalObject() : nullptr, threshold);
    165164   
    166165    m_counter = static_cast<int32_t>(-threshold);

  • trunk/Source/JavaScriptCore/bytecode/ExecutionCounter.h

    r248546 r248878  
    9090    {
    9191        int32_t maxThreshold;
    92         if (Options::randomizeExecutionCountsBetweenCheckpoints())
     92        if (Options::randomizeExecutionCountsBetweenCheckpoints() && globalObject)
    9393            maxThreshold = globalObject->weakRandomInteger() % maximumExecutionCountsBetweenCheckpoints();
    9494        else

  • trunk/Source/JavaScriptCore/dfg/DFGJITCode.h

    r245017 r248878  
    168168    HashMap<unsigned, TriggerReason> tierUpEntryTriggers;
    169169
    170     // Set of bytecode that were the target of a TierUp operation.
    171     HashSet<unsigned, WTF::IntHash<unsigned>, WTF::UnsignedWithZeroKeyHashTraits<unsigned>> tierUpEntrySeen;
    172 
    173170    WriteBarrier<CodeBlock> m_osrEntryBlock;
    174171    unsigned osrEntryRetry;

  • trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp

    r248798 r248878  
    35413541
    35423542    JITCode* jitCode = codeBlock->jitCode()->dfg();
    3543     jitCode->tierUpEntrySeen.add(bytecodeIndex);
    35443543
    35453544    if (Options::verboseOSR()) {

  • trunk/Source/JavaScriptCore/jit/AssemblyHelpers.h

    r247101 r248878  
    483483
    484484#if CPU(X86_64) || CPU(X86)
    485     static size_t prologueStackPointerDelta()
     485    static constexpr size_t prologueStackPointerDelta()
    486486    {
    487487        // Prologue only saves the framePointerRegister
    … …  
    523523
    524524#if CPU(ARM_THUMB2) || CPU(ARM64)
    525     static size_t prologueStackPointerDelta()
     525    static constexpr size_t prologueStackPointerDelta()
    526526    {
    527527        // Prologue saves the framePointerRegister and linkRegister
    … …  
    564564
    565565#if CPU(MIPS)
    566     static size_t prologueStackPointerDelta()
     566    static constexpr size_t prologueStackPointerDelta()
    567567    {
    568568        // Prologue saves the framePointerRegister and returnAddressRegister

  • trunk/Source/JavaScriptCore/runtime/Options.h

    r248829 r248878  
    482482    \
    483483    v(bool, useBBQTierUpChecks, true, Normal, "Enables tier up checks for our BBQ code.") \
    484     v(unsigned, webAssemblyOMGTierUpCount, 5000, Normal, "The countdown before we tier up a function to OMG.") \
    485     v(unsigned, webAssemblyLoopDecrement, 15, Normal, "The amount the tier up countdown is decremented on each loop backedge.") \
    486     v(unsigned, webAssemblyFunctionEntryDecrement, 1, Normal, "The amount the tier up countdown is decremented on each function entry.") \
     484    v(bool, useWebAssemblyOSR, true, Normal, nullptr) \
     485    v(int32, thresholdForOMGOptimizeAfterWarmUp, 50000, Normal, "The count before we tier up a function to OMG.") \
     486    v(int32, thresholdForOMGOptimizeSoon, 500, Normal, nullptr) \
     487    v(int32, omgTierUpCounterIncrementForLoop, 1, Normal, "The amount the tier up counter is incremented on each loop backedge.") \
     488    v(int32, omgTierUpCounterIncrementForEntry, 15, Normal, "The amount the tier up counter is incremented on each function entry.") \
    487489    /* FIXME: enable fast memories on iOS and pre-allocate them. https://bugs.webkit.org/show_bug.cgi?id=170774 */ \
    488490    v(bool, useWebAssemblyFastMemory, !isIOS(), Normal, "If true, we will try to use a 32-bit address space with a signal handler to bounds check wasm memory.") \

  • trunk/Source/JavaScriptCore/wasm/WasmAirIRGenerator.cpp

    r248846 r248878  
    5353#include "WasmMemory.h"
    5454#include "WasmOMGPlan.h"
     55#include "WasmOSREntryData.h"
    5556#include "WasmOpcodeOrigin.h"
     57#include "WasmOperations.h"
    5658#include "WasmSignatureInlines.h"
    5759#include "WasmThunks.h"
    … …  
    204206    using ControlType = ControlData;
    205207    using ExpressionList = Vector<ExpressionType, 1>;
     208    using Stack = ExpressionList;
    206209    using ResultList = ControlData::ResultList;
    207210    using ControlEntry = FunctionParser<AirIRGenerator>::ControlEntry;
    208211
    209212    static ExpressionType emptyExpression() { return { }; };
     213    Stack createStack() { return Stack(); }
    210214
    211215    using ErrorType = String;
    … …  
    268272    ControlData WARN_UNUSED_RETURN addTopLevel(Type signature);
    269273    ControlData WARN_UNUSED_RETURN addBlock(Type signature);
    270     ControlData WARN_UNUSED_RETURN addLoop(Type signature);
     274    ControlData WARN_UNUSED_RETURN addLoop(Type signature, const Stack&, uint32_t loopIndex);
    271275    PartialResult WARN_UNUSED_RETURN addIf(ExpressionType condition, Type signature, ControlData& result);
    272     PartialResult WARN_UNUSED_RETURN addElse(ControlData&, const ExpressionList&);
     276    PartialResult WARN_UNUSED_RETURN addElse(ControlData&, const Stack&);
    273277    PartialResult WARN_UNUSED_RETURN addElseToUnreachable(ControlData&);
    274278
    275279    PartialResult WARN_UNUSED_RETURN addReturn(const ControlData&, const ExpressionList& returnValues);
    276     PartialResult WARN_UNUSED_RETURN addBranch(ControlData&, ExpressionType condition, const ExpressionList& returnValues);
    277     PartialResult WARN_UNUSED_RETURN addSwitch(ExpressionType condition, const Vector<ControlData*>& targets, ControlData& defaultTargets, const ExpressionList& expressionStack);
    278     PartialResult WARN_UNUSED_RETURN endBlock(ControlEntry&, ExpressionList& expressionStack);
     280    PartialResult WARN_UNUSED_RETURN addBranch(ControlData&, ExpressionType condition, const Stack& returnValues);
     281    PartialResult WARN_UNUSED_RETURN addSwitch(ExpressionType condition, const Vector<ControlData*>& targets, ControlData& defaultTargets, const Stack& expressionStack);
     282    PartialResult WARN_UNUSED_RETURN endBlock(ControlEntry&, Stack& expressionStack);
    279283    PartialResult WARN_UNUSED_RETURN addEndToUnreachable(ControlEntry&);
    280284
    … …  
    289293    PartialResult addFloatingPointBinOp(Type, B3::Air::Opcode, ExpressionType lhs, ExpressionType rhs, ExpressionType& result);
    290294
    291     void dump(const Vector<ControlEntry>& controlStack, const ExpressionList* expressionStack);
     295    void dump(const Vector<ControlEntry>& controlStack, const Stack* expressionStack);
    292296    void setParser(FunctionParser<AirIRGenerator>* parser) { m_parser = parser; };
    293297
    … …  
    446450            patch->append(dummyValue, tmp.rep);
    447451            switch (tmp.rep.kind()) {
     452            case B3::ValueRep::ColdAny: // B3::Value propagates ColdAny information and later Air will allocate appropriate stack.
    448453            case B3::ValueRep::SomeRegister:
    449454                inst.args.append(tmp.tmp);
    … …  
    579584    void emitThrowException(CCallHelpers&, ExceptionType);
    580585
    581     void emitTierUpCheck(uint32_t decrementCount, B3::Origin);
     586    void emitEntryTierUpCheck(int32_t incrementCount, B3::Origin);
     587    void emitLoopTierUpCheck(int32_t incrementCount, const Stack&, uint32_t, uint32_t, B3::Origin);
    582588
    583589    void emitWriteBarrierForJSWrapper();
    … …  
    587593
    588594    void unify(const ExpressionType& dst, const ExpressionType& source);
    589     void unifyValuesWithBlock(const ExpressionList& resultStack, const ResultList& stack);
     595    void unifyValuesWithBlock(const Stack& resultStack, const ResultList& stack);
    590596
    591597    template <typename IntType>
    … …  
    606612
    607613    B3::Origin origin();
     614
     615    uint32_t outerLoopIndex() const
     616    {
     617        if (m_outerLoops.isEmpty())
     618            return UINT32_MAX;
     619        return m_outerLoops.last();
     620    }
    608621
    609622    FunctionParser<AirIRGenerator>* m_parser { nullptr };
    … …  
    611624    const MemoryMode m_mode { MemoryMode::BoundsChecking };
    612625    const unsigned m_functionIndex { UINT_MAX };
    613     const TierUpCount* m_tierUp { nullptr };
     626    TierUpCount* m_tierUp { nullptr };
    614627
    615628    B3::Procedure& m_proc;
    616629    Code& m_code;
     630    Vector<uint32_t> m_outerLoops;
    617631    BasicBlock* m_currentBlock { nullptr };
    618632    BasicBlock* m_rootBlock { nullptr };
    … …  
    828842    });
    829843
    830     emitTierUpCheck(TierUpCount::functionEntryDecrement(), B3::Origin());
     844    emitEntryTierUpCheck(TierUpCount::functionEntryIncrement(), B3::Origin());
    831845}
    832846
    … …  
    15791593}
    15801594
    1581 void AirIRGenerator::emitTierUpCheck(uint32_t decrementCount, B3::Origin origin)
     1595void AirIRGenerator::emitEntryTierUpCheck(int32_t incrementCount, B3::Origin origin)
    15821596{
    15831597    UNUSED_PARAM(origin);
    … …  
    15871601
    15881602    auto countdownPtr = g64();
    1589     auto oldCountdown = g64();
    1590     auto newCountdown = g64();
    1591 
    1592     append(Move, Arg::bigImm(reinterpret_cast<uint64_t>(m_tierUp)), countdownPtr);
    1593     append(Move32, Arg::addr(countdownPtr), oldCountdown);
    1594 
    1595     RELEASE_ASSERT(Arg::isValidImmForm(decrementCount));
    1596     append(Move32, oldCountdown, newCountdown);
    1597     append(Sub32, Arg::imm(decrementCount), newCountdown);
    1598     append(Move32, newCountdown, Arg::addr(countdownPtr));
     1603
     1604    append(Move, Arg::bigImm(reinterpret_cast<uint64_t>(&m_tierUp->m_counter)), countdownPtr);
    15991605
    16001606    auto* patch = addPatchpoint(B3::Void);
    … …  
    16031609    effects.writes = B3::HeapRange::top();
    16041610    patch->effects = effects;
     1611    patch->clobber(RegisterSet::macroScratchRegisters());
    16051612
    16061613    patch->setGenerator([=] (CCallHelpers& jit, const B3::StackmapGenerationParams& params) {
    1607         MacroAssembler::Jump tierUp = jit.branch32(MacroAssembler::Above, params[0].gpr(), params[1].gpr());
    1608         MacroAssembler::Label tierUpResume = jit.label();
     1614        AllowMacroScratchRegisterUsage allowScratch(jit);
     1615
     1616        CCallHelpers::Jump tierUp = jit.branchAdd32(CCallHelpers::PositiveOrZero, CCallHelpers::TrustedImm32(incrementCount), CCallHelpers::Address(params[0].gpr()));
     1617        CCallHelpers::Label tierUpResume = jit.label();
    16091618
    16101619        params.addLatePath([=] (CCallHelpers& jit) {
    … …  
    16231632
    16241633            jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
    1625                 MacroAssembler::repatchNearCall(linkBuffer.locationOfNearCall<NoPtrTag>(call), CodeLocationLabel<JITThunkPtrTag>(Thunks::singleton().stub(triggerOMGTierUpThunkGenerator).code()));
    1626 
     1634                MacroAssembler::repatchNearCall(linkBuffer.locationOfNearCall<NoPtrTag>(call), CodeLocationLabel<JITThunkPtrTag>(Thunks::singleton().stub(triggerOMGEntryTierUpThunkGenerator).code()));
    16271635            });
    16281636        });
    16291637    });
    16301638
    1631     emitPatchpoint(patch, Tmp(), newCountdown, oldCountdown);
    1632 }
    1633 
    1634 AirIRGenerator::ControlData AirIRGenerator::addLoop(Type signature)
     1639    emitPatchpoint(patch, Tmp(), countdownPtr);
     1640}
     1641
     1642void AirIRGenerator::emitLoopTierUpCheck(int32_t incrementCount, const Stack& expressionStack, uint32_t loopIndex, uint32_t outerLoopIndex, B3::Origin origin)
     1643{
     1644    UNUSED_PARAM(origin);
     1645
     1646    if (!m_tierUp)
     1647        return;
     1648
     1649    ASSERT(m_tierUp->osrEntryTriggers().size() == loopIndex);
     1650    m_tierUp->osrEntryTriggers().append(TierUpCount::TriggerReason::DontTrigger);
     1651    m_tierUp->outerLoops().append(outerLoopIndex);
     1652
     1653    auto countdownPtr = g64();
     1654
     1655    append(Move, Arg::bigImm(reinterpret_cast<uint64_t>(&m_tierUp->m_counter)), countdownPtr);
     1656
     1657    auto* patch = addPatchpoint(B3::Void);
     1658    B3::Effects effects = B3::Effects::none();
     1659    effects.reads = B3::HeapRange::top();
     1660    effects.writes = B3::HeapRange::top();
     1661    effects.exitsSideways = true;
     1662    patch->effects = effects;
     1663
     1664    patch->clobber(RegisterSet::macroScratchRegisters());
     1665    RegisterSet clobberLate;
     1666    clobberLate.add(GPRInfo::argumentGPR0);
     1667    patch->clobberLate(clobberLate);
     1668
     1669    Vector<ConstrainedTmp> patchArgs;
     1670    patchArgs.append(countdownPtr);
     1671
     1672    Vector<B3::Type> types;
     1673    for (auto& local : m_locals) {
     1674        patchArgs.append(ConstrainedTmp(local, B3::ValueRep::ColdAny));
     1675        types.append(toB3Type(local.type()));
     1676    }
     1677    for (auto& expression : expressionStack) {
     1678        patchArgs.append(ConstrainedTmp(expression, B3::ValueRep::ColdAny));
     1679        types.append(toB3Type(expression.type()));
     1680    }
     1681
     1682    TierUpCount::TriggerReason* forceEntryTrigger = &(m_tierUp->osrEntryTriggers().last());
     1683    static_assert(!static_cast<uint8_t>(TierUpCount::TriggerReason::DontTrigger), "the JIT code assumes non-zero means 'enter'");
     1684    static_assert(sizeof(TierUpCount::TriggerReason) == 1, "branchTest8 assumes this size");
     1685    patch->setGenerator([=] (CCallHelpers& jit, const B3::StackmapGenerationParams& params) {
     1686        AllowMacroScratchRegisterUsage allowScratch(jit);
     1687        CCallHelpers::Jump forceOSREntry = jit.branchTest8(CCallHelpers::NonZero, CCallHelpers::AbsoluteAddress(forceEntryTrigger));
     1688        CCallHelpers::Jump tierUp = jit.branchAdd32(CCallHelpers::PositiveOrZero, CCallHelpers::TrustedImm32(incrementCount), CCallHelpers::Address(params[0].gpr()));
     1689        MacroAssembler::Label tierUpResume = jit.label();
     1690
     1691        OSREntryData& osrEntryData = m_tierUp->addOSREntryData(m_functionIndex, loopIndex);
     1692        for (unsigned index = 0; index < types.size(); ++index)
     1693            osrEntryData.values().constructAndAppend(params[index + 1], types[index]);
     1694        OSREntryData* osrEntryDataPtr = &osrEntryData;
     1695
     1696        params.addLatePath([=] (CCallHelpers& jit) {
     1697            AllowMacroScratchRegisterUsage allowScratch(jit);
     1698            forceOSREntry.link(&jit);
     1699            tierUp.link(&jit);
     1700
     1701            jit.probe(triggerOSREntryNow, osrEntryDataPtr);
     1702            jit.branchTestPtr(CCallHelpers::Zero, GPRInfo::argumentGPR0).linkTo(tierUpResume, &jit);
     1703            jit.farJump(GPRInfo::argumentGPR1, WasmEntryPtrTag);
     1704        });
     1705    });
     1706
     1707    emitPatchpoint(patch, Tmp(), WTFMove(patchArgs));
     1708}
     1709
     1710AirIRGenerator::ControlData AirIRGenerator::addLoop(Type signature, const Stack& expressionStack, uint32_t loopIndex)
    16351711{
    16361712    BasicBlock* body = m_code.addBlock();
    … …  
    16401716    m_currentBlock->setSuccessors(body);
    16411717
     1718    uint32_t outerLoopIndex = this->outerLoopIndex();
     1719    m_outerLoops.append(loopIndex);
    16421720    m_currentBlock = body;
    1643     emitTierUpCheck(TierUpCount::loopDecrement(), origin());
     1721    emitLoopTierUpCheck(TierUpCount::loopIncrement(), expressionStack, loopIndex, outerLoopIndex, origin());
    16441722
    16451723    return ControlData(origin(), signature, tmpForType(signature), BlockType::Loop, continuation, body);
    … …  
    16711749}
    16721750
    1673 auto AirIRGenerator::addElse(ControlData& data, const ExpressionList& currentStack) -> PartialResult
     1751auto AirIRGenerator::addElse(ControlData& data, const Stack& currentStack) -> PartialResult
    16741752{
    16751753    unifyValuesWithBlock(currentStack, data.result);
    … …  
    17221800// NOTE: All branches in Wasm are on 32-bit ints
    17231801
    1724 auto AirIRGenerator::addBranch(ControlData& data, ExpressionType condition, const ExpressionList& returnValues) -> PartialResult
     1802auto AirIRGenerator::addBranch(ControlData& data, ExpressionType condition, const Stack& returnValues) -> PartialResult
    17251803{
    17261804    unifyValuesWithBlock(returnValues, data.resultForBranch());
    … …  
    17401818}
    17411819
    1742 auto AirIRGenerator::addSwitch(ExpressionType condition, const Vector<ControlData*>& targets, ControlData& defaultTarget, const ExpressionList& expressionStack) -> PartialResult
     1820auto AirIRGenerator::addSwitch(ExpressionType condition, const Vector<ControlData*>& targets, ControlData& defaultTarget, const Stack& expressionStack) -> PartialResult
    17431821{
    17441822    auto& successors = m_currentBlock->successors();
    … …  
    18031881}
    18041882
    1805 auto AirIRGenerator::endBlock(ControlEntry& entry, ExpressionList& expressionStack) -> PartialResult
     1883auto AirIRGenerator::endBlock(ControlEntry& entry, Stack& expressionStack) -> PartialResult
    18061884{
    18071885    ControlData& data = entry.controlData;
    … …  
    18241902        data.special->setSuccessors(m_currentBlock);
    18251903    }
     1904
     1905    if (data.type() == BlockType::Loop)
     1906        m_outerLoops.removeLast();
    18261907
    18271908    for (const auto& result : data.result)
    … …  
    21202201}
    21212202
    2122 void AirIRGenerator::unifyValuesWithBlock(const ExpressionList& resultStack, const ResultList& result)
     2203void AirIRGenerator::unifyValuesWithBlock(const Stack& resultStack, const ResultList& result)
    21232204{
    21242205    ASSERT(result.size() <= resultStack.size());
    … …  
    21282209}
    21292210
    2130 void AirIRGenerator::dump(const Vector<ControlEntry>&, const ExpressionList*)
     2211void AirIRGenerator::dump(const Vector<ControlEntry>&, const Stack*)
    21312212{
    21322213}

  • trunk/Source/JavaScriptCore/wasm/WasmB3IRGenerator.cpp

    r248846 r248878  
    6060#include "WasmMemory.h"
    6161#include "WasmOMGPlan.h"
     62#include "WasmOSREntryData.h"
    6263#include "WasmOpcodeOrigin.h"
     64#include "WasmOperations.h"
    6365#include "WasmSignatureInlines.h"
    6466#include "WasmThunks.h"
    … …  
    158160
    159161    typedef Value* ExpressionType;
    160     typedef ControlData ControlType;
    161162    typedef Vector<ExpressionType, 1> ExpressionList;
    162     typedef ControlData::ResultList ResultList;
    163     typedef FunctionParser<B3IRGenerator>::ControlEntry ControlEntry;
     163
     164    friend class Stack;
     165    class Stack {
     166    public:
     167        Stack(B3IRGenerator* generator)
     168            : m_generator(generator)
     169        {
     170        }
     171
     172        void append(ExpressionType expression)
     173        {
     174            if (m_generator->m_compilationMode == CompilationMode::OMGForOSREntryMode) {
     175                Variable* variable = m_generator->m_proc.addVariable(expression->type());
     176                m_generator->m_currentBlock->appendNew<VariableValue>(m_generator->m_proc, Set, m_generator->origin(), variable, expression);
     177                m_stack.append(variable);
     178                return;
     179            }
     180            m_data.append(expression);
     181        }
     182
     183        ExpressionType takeLast()
     184        {
     185            if (m_generator->m_compilationMode == CompilationMode::OMGForOSREntryMode)
     186                return m_generator->m_currentBlock->appendNew<VariableValue>(m_generator->m_proc, B3::Get, m_generator->origin(), m_stack.takeLast());
     187            return m_data.takeLast();
     188        }
     189
     190        ExpressionType last()
     191        {
     192            if (m_generator->m_compilationMode == CompilationMode::OMGForOSREntryMode)
     193                return m_generator->m_currentBlock->appendNew<VariableValue>(m_generator->m_proc, B3::Get, m_generator->origin(), m_stack.last());
     194            return m_data.last();
     195        }
     196
     197        unsigned size() const
     198        {
     199            if (m_generator->m_compilationMode == CompilationMode::OMGForOSREntryMode)
     200                return m_stack.size();
     201            return m_data.size();
     202        }
     203        bool isEmpty() const { return size() == 0; }
     204
     205        ExpressionList convertToExpressionList()
     206        {
     207            if (m_generator->m_compilationMode == CompilationMode::OMGForOSREntryMode) {
     208                ExpressionList results;
     209                for (unsigned i = 0; i < m_stack.size(); ++i)
     210                    results.append(at(i));
     211                return results;
     212            }
     213            return m_data;
     214        }
     215
     216        ExpressionType at(unsigned i) const
     217        {
     218            if (m_generator->m_compilationMode == CompilationMode::OMGForOSREntryMode)
     219                return m_generator->m_currentBlock->appendNew<VariableValue>(m_generator->m_proc, B3::Get, m_generator->origin(), m_stack.at(i));
     220            return m_data.at(i);
     221        }
     222
     223        Variable* variableAt(unsigned i) const
     224        {
     225            if (m_generator->m_compilationMode == CompilationMode::OMGForOSREntryMode)
     226                return m_stack.at(i);
     227            return nullptr;
     228        }
     229
     230        void shrink(unsigned i)
     231        {
     232            if (m_generator->m_compilationMode == CompilationMode::OMGForOSREntryMode) {
     233                m_stack.shrink(i);
     234                return;
     235            }
     236            m_data.shrink(i);
     237        }
     238
     239        void swap(Stack& stack)
     240        {
     241            std::swap(m_generator, stack.m_generator);
     242            m_data.swap(stack.m_data);
     243            m_stack.swap(stack.m_stack);
     244        }
     245
     246        void dump() const
     247        {
     248            CommaPrinter comma(", ", "");
     249            dataLog(comma, "ExpressionStack:");
     250            if (m_generator->m_compilationMode == CompilationMode::OMGForOSREntryMode) {
     251                for (const auto& variable : m_stack)
     252                    dataLog(comma, *variable);
     253                return;
     254            }
     255            for (const auto& expression : m_data)
     256                dataLog(comma, *expression);
     257        }
     258
     259    private:
     260        B3IRGenerator* m_generator { nullptr };
     261        ExpressionList m_data;
     262        Vector<Variable*> m_stack;
     263    };
     264    Stack createStack() { return Stack(this); }
     265
     266    using ControlType = ControlData;
     267    using ResultList = ControlData::ResultList;
     268    using ControlEntry = FunctionParser<B3IRGenerator>::ControlEntry;
    164269
    165270    static constexpr ExpressionType emptyExpression() { return nullptr; }
    … …  
    180285    } while (0)
    181286
    182     B3IRGenerator(const ModuleInformation&, Procedure&, InternalFunction*, Vector<UnlinkedWasmToWasmCall>&, MemoryMode, CompilationMode, unsigned functionIndex, TierUpCount*, ThrowWasmException);
     287    B3IRGenerator(const ModuleInformation&, Procedure&, InternalFunction*, Vector<UnlinkedWasmToWasmCall>&, unsigned& osrEntryScratchBufferSize, MemoryMode, CompilationMode, unsigned functionIndex, unsigned loopIndexForOSREntry, TierUpCount*, ThrowWasmException);
    183288
    184289    PartialResult WARN_UNUSED_RETURN addArguments(const Signature&);
    … …  
    220325    ControlData WARN_UNUSED_RETURN addTopLevel(Type signature);
    221326    ControlData WARN_UNUSED_RETURN addBlock(Type signature);
    222     ControlData WARN_UNUSED_RETURN addLoop(Type signature);
     327    ControlData WARN_UNUSED_RETURN addLoop(Type signature, const Stack&, uint32_t);
    223328    PartialResult WARN_UNUSED_RETURN addIf(ExpressionType condition, Type signature, ControlData& result);
    224     PartialResult WARN_UNUSED_RETURN addElse(ControlData&, const ExpressionList&);
     329    PartialResult WARN_UNUSED_RETURN addElse(ControlData&, const Stack&);
    225330    PartialResult WARN_UNUSED_RETURN addElseToUnreachable(ControlData&);
    226331
    227332    PartialResult WARN_UNUSED_RETURN addReturn(const ControlData&, const ExpressionList& returnValues);
    228     PartialResult WARN_UNUSED_RETURN addBranch(ControlData&, ExpressionType condition, const ExpressionList& returnValues);
    229     PartialResult WARN_UNUSED_RETURN addSwitch(ExpressionType condition, const Vector<ControlData*>& targets, ControlData& defaultTargets, const ExpressionList& expressionStack);
    230     PartialResult WARN_UNUSED_RETURN endBlock(ControlEntry&, ExpressionList& expressionStack);
     333    PartialResult WARN_UNUSED_RETURN addBranch(ControlData&, ExpressionType condition, const Stack& returnValues);
     334    PartialResult WARN_UNUSED_RETURN addSwitch(ExpressionType condition, const Vector<ControlData*>& targets, ControlData& defaultTargets, const Stack& expressionStack);
     335    PartialResult WARN_UNUSED_RETURN endBlock(ControlEntry&, Stack& expressionStack);
    231336    PartialResult WARN_UNUSED_RETURN addEndToUnreachable(ControlEntry&);
    232337
    … …  
    236341    PartialResult WARN_UNUSED_RETURN addUnreachable();
    237342
    238     void dump(const Vector<ControlEntry>& controlStack, const ExpressionList* expressionStack);
     343    void dump(const Vector<ControlEntry>& controlStack, const Stack* expressionStack);
    239344    void setParser(FunctionParser<B3IRGenerator>* parser) { m_parser = parser; };
    240345
    … …  
    247352    void emitExceptionCheck(CCallHelpers&, ExceptionType);
    248353
    249     void emitTierUpCheck(uint32_t decrementCount, Origin);
     354    void emitEntryTierUpCheck(int32_t incrementCount, B3::Origin);
     355    void emitLoopTierUpCheck(int32_t incrementCount, const Stack&, uint32_t, uint32_t, B3::Origin);
    250356
    251357    void emitWriteBarrierForJSWrapper();
    … …  
    256362
    257363    void unify(const ExpressionType phi, const ExpressionType source);
    258     void unifyValuesWithBlock(const ExpressionList& resultStack, const ResultList& stack);
     364    void unifyValuesWithBlock(const Stack& resultStack, const ResultList& stack);
    259365
    260366    void emitChecksForModOrDiv(B3::Opcode, ExpressionType left, ExpressionType right);
    … …  
    267373
    268374    Origin origin();
     375
     376    uint32_t outerLoopIndex() const
     377    {
     378        if (m_outerLoops.isEmpty())
     379            return UINT32_MAX;
     380        return m_outerLoops.last();
     381    }
    269382
    270383    FunctionParser<B3IRGenerator>* m_parser { nullptr };
    … …  
    273386    const CompilationMode m_compilationMode { CompilationMode::BBQMode };
    274387    const unsigned m_functionIndex { UINT_MAX };
    275     const TierUpCount* m_tierUp { nullptr };
     388    const unsigned m_loopIndexForOSREntry { UINT_MAX };
     389    TierUpCount* m_tierUp { nullptr };
    276390
    277391    Procedure& m_proc;
     392    BasicBlock* m_rootBlock { nullptr };
    278393    BasicBlock* m_currentBlock { nullptr };
     394    Vector<uint32_t> m_outerLoops;
    279395    Vector<Variable*> m_locals;
    280396    Vector<UnlinkedWasmToWasmCall>& m_unlinkedWasmToWasmCalls; // List each call site and the function index whose address it should be patched with.
     397    unsigned& m_osrEntryScratchBufferSize;
    281398    HashMap<ValueKey, Value*> m_constantPool;
    282399    InsertionSet m_constantInsertionValues;
    … …  
    338455}
    339456
    340 B3IRGenerator::B3IRGenerator(const ModuleInformation& info, Procedure& procedure, InternalFunction* compilation, Vector<UnlinkedWasmToWasmCall>& unlinkedWasmToWasmCalls, MemoryMode mode, CompilationMode compilationMode, unsigned functionIndex, TierUpCount* tierUp, ThrowWasmException throwWasmException)
     457B3IRGenerator::B3IRGenerator(const ModuleInformation& info, Procedure& procedure, InternalFunction* compilation, Vector<UnlinkedWasmToWasmCall>& unlinkedWasmToWasmCalls, unsigned& osrEntryScratchBufferSize, MemoryMode mode, CompilationMode compilationMode, unsigned functionIndex, unsigned loopIndexForOSREntry, TierUpCount* tierUp, ThrowWasmException throwWasmException)
    341458    : m_info(info)
    342459    , m_mode(mode)
    343460    , m_compilationMode(compilationMode)
    344461    , m_functionIndex(functionIndex)
     462    , m_loopIndexForOSREntry(loopIndexForOSREntry)
    345463    , m_tierUp(tierUp)
    346464    , m_proc(procedure)
    347465    , m_unlinkedWasmToWasmCalls(unlinkedWasmToWasmCalls)
     466    , m_osrEntryScratchBufferSize(osrEntryScratchBufferSize)
    348467    , m_constantInsertionValues(m_proc)
    349468    , m_numImportFunctions(info.importFunctionCount())
    350469{
    351     m_currentBlock = m_proc.addBlock();
     470    m_rootBlock = m_proc.addBlock();
     471    m_currentBlock = m_rootBlock;
    352472
    353473    // FIXME we don't really need to pin registers here if there's no memory. It makes wasm -> wasm thunks simpler for now. https://bugs.webkit.org/show_bug.cgi?id=166623
    … …  
    465585    }
    466586
    467     emitTierUpCheck(TierUpCount::functionEntryDecrement(), Origin());
     587    emitEntryTierUpCheck(TierUpCount::functionEntryIncrement(), Origin());
     588
     589    if (m_compilationMode == CompilationMode::OMGForOSREntryMode)
     590        m_currentBlock = m_proc.addBlock();
    468591}
    469592
    … …  
    10681191}
    10691192
    1070 void B3IRGenerator::emitTierUpCheck(uint32_t decrementCount, Origin origin)
     1193void B3IRGenerator::emitEntryTierUpCheck(int32_t incrementCount, Origin origin)
    10711194{
    10721195    if (!m_tierUp)
    … …  
    10741197
    10751198    ASSERT(m_tierUp);
    1076     Value* countDownLocation = constant(pointerType(), reinterpret_cast<uint64_t>(m_tierUp), origin);
    1077     Value* oldCountDown = m_currentBlock->appendNew<MemoryValue>(m_proc, Load, Int32, origin, countDownLocation);
    1078     Value* newCountDown = m_currentBlock->appendNew<Value>(m_proc, Sub, origin, oldCountDown, constant(Int32, decrementCount, origin));
    1079     m_currentBlock->appendNew<MemoryValue>(m_proc, Store, origin, newCountDown, countDownLocation);
     1199    Value* countDownLocation = constant(pointerType(), reinterpret_cast<uint64_t>(&m_tierUp->m_counter), origin);
    10801200
    10811201    PatchpointValue* patch = m_currentBlock->appendNew<PatchpointValue>(m_proc, B3::Void, origin);
    … …  
    10851205    effects.writes = B3::HeapRange::top();
    10861206    patch->effects = effects;
    1087 
    1088     patch->append(newCountDown, ValueRep::SomeRegister);
    1089     patch->append(oldCountDown, ValueRep::SomeRegister);
     1207    patch->clobber(RegisterSet::macroScratchRegisters());
     1208
     1209    patch->append(countDownLocation, ValueRep::SomeRegister);
    10901210    patch->setGenerator([=] (CCallHelpers& jit, const StackmapGenerationParams& params) {
    1091         MacroAssembler::Jump tierUp = jit.branch32(MacroAssembler::Above, params[0].gpr(), params[1].gpr());
    1092         MacroAssembler::Label tierUpResume = jit.label();
     1211        AllowMacroScratchRegisterUsage allowScratch(jit);
     1212        CCallHelpers::Jump tierUp = jit.branchAdd32(CCallHelpers::PositiveOrZero, CCallHelpers::TrustedImm32(incrementCount), CCallHelpers::Address(params[0].gpr()));
     1213        CCallHelpers::Label tierUpResume = jit.label();
    10931214
    10941215        params.addLatePath([=] (CCallHelpers& jit) {
    … …  
    11071228
    11081229            jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
    1109                 MacroAssembler::repatchNearCall(linkBuffer.locationOfNearCall<NoPtrTag>(call), CodeLocationLabel<JITThunkPtrTag>(Thunks::singleton().stub(triggerOMGTierUpThunkGenerator).code()));
    1110 
     1230                MacroAssembler::repatchNearCall(linkBuffer.locationOfNearCall<NoPtrTag>(call), CodeLocationLabel<JITThunkPtrTag>(Thunks::singleton().stub(triggerOMGEntryTierUpThunkGenerator).code()));
    11111231            });
    11121232        });
    … …  
    11141234}
    11151235
    1116 B3IRGenerator::ControlData B3IRGenerator::addLoop(Type signature)
     1236void B3IRGenerator::emitLoopTierUpCheck(int32_t incrementCount, const Stack& expressionStack, uint32_t loopIndex, uint32_t outerLoopIndex, B3::Origin origin)
     1237{
     1238    if (!m_tierUp)
     1239        return;
     1240
     1241    ASSERT(m_tierUp);
     1242
     1243    ASSERT(m_tierUp->osrEntryTriggers().size() == loopIndex);
     1244    m_tierUp->osrEntryTriggers().append(TierUpCount::TriggerReason::DontTrigger);
     1245    m_tierUp->outerLoops().append(outerLoopIndex);
     1246
     1247    Value* countDownLocation = constant(pointerType(), reinterpret_cast<uint64_t>(&m_tierUp->m_counter), origin);
     1248
     1249    Vector<ExpressionType> stackmap;
     1250    Vector<B3::Type> types;
     1251    for (auto& local : m_locals) {
     1252        ExpressionType result = m_currentBlock->appendNew<VariableValue>(m_proc, B3::Get, origin, local);
     1253        stackmap.append(result);
     1254        types.append(result->type());
     1255    }
     1256    for (unsigned i = 0; i < expressionStack.size(); ++i) {
     1257        ExpressionType result = expressionStack.at(i);
     1258        stackmap.append(result);
     1259        types.append(result->type());
     1260    }
     1261
     1262    PatchpointValue* patch = m_currentBlock->appendNew<PatchpointValue>(m_proc, B3::Void, origin);
     1263    Effects effects = Effects::none();
     1264    // FIXME: we should have a more precise heap range for the tier up count.
     1265    effects.reads = B3::HeapRange::top();
     1266    effects.writes = B3::HeapRange::top();
     1267    effects.exitsSideways = true;
     1268    patch->effects = effects;
     1269
     1270    patch->clobber(RegisterSet::macroScratchRegisters());
     1271    RegisterSet clobberLate;
     1272    clobberLate.add(GPRInfo::argumentGPR0);
     1273    patch->clobberLate(clobberLate);
     1274
     1275    patch->append(countDownLocation, ValueRep::SomeRegister);
     1276    patch->appendVectorWithRep(stackmap, ValueRep::ColdAny);
     1277
     1278    TierUpCount::TriggerReason* forceEntryTrigger = &(m_tierUp->osrEntryTriggers().last());
     1279    static_assert(!static_cast<uint8_t>(TierUpCount::TriggerReason::DontTrigger), "the JIT code assumes non-zero means 'enter'");
     1280    static_assert(sizeof(TierUpCount::TriggerReason) == 1, "branchTest8 assumes this size");
     1281    patch->setGenerator([=] (CCallHelpers& jit, const StackmapGenerationParams& params) {
     1282        AllowMacroScratchRegisterUsage allowScratch(jit);
     1283        CCallHelpers::Jump forceOSREntry = jit.branchTest8(CCallHelpers::NonZero, CCallHelpers::AbsoluteAddress(forceEntryTrigger));
     1284        CCallHelpers::Jump tierUp = jit.branchAdd32(CCallHelpers::PositiveOrZero, CCallHelpers::TrustedImm32(incrementCount), CCallHelpers::Address(params[0].gpr()));
     1285        MacroAssembler::Label tierUpResume = jit.label();
     1286
     1287        OSREntryData& osrEntryData = m_tierUp->addOSREntryData(m_functionIndex, loopIndex);
     1288        for (unsigned index = 0; index < types.size(); ++index)
     1289            osrEntryData.values().constructAndAppend(params[index + 1], types[index]);
     1290        OSREntryData* osrEntryDataPtr = &osrEntryData;
     1291
     1292        params.addLatePath([=] (CCallHelpers& jit) {
     1293            AllowMacroScratchRegisterUsage allowScratch(jit);
     1294            forceOSREntry.link(&jit);
     1295            tierUp.link(&jit);
     1296
     1297            jit.probe(triggerOSREntryNow, osrEntryDataPtr);
     1298            jit.branchTestPtr(CCallHelpers::Zero, GPRInfo::argumentGPR0).linkTo(tierUpResume, &jit);
     1299            jit.farJump(GPRInfo::argumentGPR1, WasmEntryPtrTag);
     1300        });
     1301    });
     1302}
     1303
     1304B3IRGenerator::ControlData B3IRGenerator::addLoop(Type signature, const Stack& stack, uint32_t loopIndex)
    11171305{
    11181306    BasicBlock* body = m_proc.addBlock();
    … …  
    11201308
    11211309    m_currentBlock->appendNewControlValue(m_proc, Jump, origin(), body);
    1122 
     1310    if (loopIndex == m_loopIndexForOSREntry) {
     1311        m_currentBlock = m_rootBlock;
     1312        m_osrEntryScratchBufferSize = m_locals.size() + stack.size();
     1313        Value* pointer = m_rootBlock->appendNew<ArgumentRegValue>(m_proc, Origin(), GPRInfo::argumentGPR0);
     1314
     1315        auto loadFromScratchBuffer = [&] (B3::Type type, unsigned index) {
     1316            size_t offset = sizeof(uint64_t) * index;
     1317            switch (type.kind()) {
     1318            case B3::Int32:
     1319                return m_currentBlock->appendNew<MemoryValue>(m_proc, Load, Int32, origin(), pointer, offset);
     1320            case B3::Int64:
     1321                return m_currentBlock->appendNew<MemoryValue>(m_proc, Load, B3::Int64, origin(), pointer, offset);
     1322            case B3::Float:
     1323                return m_currentBlock->appendNew<MemoryValue>(m_proc, Load, B3::Float, origin(), pointer, offset);
     1324            case B3::Double:
     1325                return m_currentBlock->appendNew<MemoryValue>(m_proc, Load, B3::Double, origin(), pointer, offset);
     1326            default:
     1327                RELEASE_ASSERT_NOT_REACHED();
     1328                break;
     1329            }
     1330        };
     1331
     1332        unsigned indexInBuffer = 0;
     1333        for (auto& local : m_locals)
     1334            m_currentBlock->appendNew<VariableValue>(m_proc, Set, Origin(), local, loadFromScratchBuffer(local->type(), indexInBuffer++));
     1335        for (unsigned i = 0; i < stack.size(); ++i) {
     1336            auto* variable = stack.variableAt(i);
     1337            m_currentBlock->appendNew<VariableValue>(m_proc, Set, Origin(), variable, loadFromScratchBuffer(variable->type(), indexInBuffer++));
     1338        }
     1339        m_currentBlock->appendNewControlValue(m_proc, Jump, origin(), body);
     1340        body->addPredecessor(m_currentBlock);
     1341    }
     1342
     1343    uint32_t outerLoopIndex = this->outerLoopIndex();
     1344    m_outerLoops.append(loopIndex);
    11231345    m_currentBlock = body;
    1124     emitTierUpCheck(TierUpCount::loopDecrement(), origin());
     1346    emitLoopTierUpCheck(TierUpCount::loopIncrement(), stack, loopIndex, outerLoopIndex, origin());
    11251347
    11261348    return ControlData(m_proc, origin(), signature, BlockType::Loop, continuation, body);
    … …  
    11551377}
    11561378
    1157 auto B3IRGenerator::addElse(ControlData& data, const ExpressionList& currentStack) -> PartialResult
     1379auto B3IRGenerator::addElse(ControlData& data, const Stack& currentStack) -> PartialResult
    11581380{
    11591381    unifyValuesWithBlock(currentStack, data.result);
    … …  
    11801402}
    11811403
    1182 auto B3IRGenerator::addBranch(ControlData& data, ExpressionType condition, const ExpressionList& returnValues) -> PartialResult
     1404auto B3IRGenerator::addBranch(ControlData& data, ExpressionType condition, const Stack& returnValues) -> PartialResult
    11831405{
    11841406    unifyValuesWithBlock(returnValues, data.resultForBranch());
    … …  
    12001422}
    12011423
    1202 auto B3IRGenerator::addSwitch(ExpressionType condition, const Vector<ControlData*>& targets, ControlData& defaultTarget, const ExpressionList& expressionStack) -> PartialResult
     1424auto B3IRGenerator::addSwitch(ExpressionType condition, const Vector<ControlData*>& targets, ControlData& defaultTarget, const Stack& expressionStack) -> PartialResult
    12031425{
    12041426    for (size_t i = 0; i < targets.size(); ++i)
    … …  
    12141436}
    12151437
    1216 auto B3IRGenerator::endBlock(ControlEntry& entry, ExpressionList& expressionStack) -> PartialResult
     1438auto B3IRGenerator::endBlock(ControlEntry& entry, Stack& expressionStack) -> PartialResult
    12171439{
    12181440    ControlData& data = entry.controlData;
    … …  
    12221444    data.continuation->addPredecessor(m_currentBlock);
    12231445
     1446    if (data.type() == BlockType::Loop)
     1447        m_outerLoops.removeLast();
     1448
    12241449    return addEndToUnreachable(entry);
    12251450}
    … …  
    12431468    // TopLevel does not have any code after this so we need to make sure we emit a return here.
    12441469    if (data.type() == BlockType::TopLevel)
    1245         return addReturn(entry.controlData, entry.enclosedExpressionStack);
     1470        return addReturn(entry.controlData, entry.enclosedExpressionStack.convertToExpressionList());
    12461471
    12471472    return { };
    … …  
    15071732}
    15081733
    1509 void B3IRGenerator::unifyValuesWithBlock(const ExpressionList& resultStack, const ResultList& result)
     1734void B3IRGenerator::unifyValuesWithBlock(const Stack& resultStack, const ResultList& result)
    15101735{
    15111736    ASSERT(result.size() <= resultStack.size());
    15121737
    15131738    for (size_t i = 0; i < result.size(); ++i)
    1514         unify(result[result.size() - 1 - i], resultStack[resultStack.size() - 1 - i]);
    1515 }
    1516 
    1517 static void dumpExpressionStack(const CommaPrinter& comma, const B3IRGenerator::ExpressionList& expressionStack)
    1518 {
    1519     dataLog(comma, "ExpressionStack:");
    1520     for (const auto& expression : expressionStack)
    1521         dataLog(comma, *expression);
    1522 }
    1523 
    1524 void B3IRGenerator::dump(const Vector<ControlEntry>& controlStack, const ExpressionList* expressionStack)
     1739        unify(result[result.size() - 1 - i], resultStack.at(resultStack.size() - 1 - i));
     1740}
     1741
     1742void B3IRGenerator::dump(const Vector<ControlEntry>& controlStack, const Stack* expressionStack)
    15251743{
    15261744    dataLogLn("Constants:");
    … …  
    15351753    for (size_t i = controlStack.size(); i--;) {
    15361754        dataLog("  ", controlStack[i].controlData, ": ");
    1537         CommaPrinter comma(", ", "");
    1538         dumpExpressionStack(comma, *expressionStack);
     1755        expressionStack->dump();
    15391756        expressionStack = &controlStack[i].enclosedExpressionStack;
    15401757        dataLogLn();
    … …  
    15501767}
    15511768
    1552 Expected<std::unique_ptr<InternalFunction>, String> parseAndCompile(CompilationContext& compilationContext, const uint8_t* functionStart, size_t functionLength, const Signature& signature, Vector<UnlinkedWasmToWasmCall>& unlinkedWasmToWasmCalls, const ModuleInformation& info, MemoryMode mode, CompilationMode compilationMode, uint32_t functionIndex, TierUpCount* tierUp, ThrowWasmException throwWasmException)
     1769Expected<std::unique_ptr<InternalFunction>, String> parseAndCompile(CompilationContext& compilationContext, const uint8_t* functionStart, size_t functionLength, const Signature& signature, Vector<UnlinkedWasmToWasmCall>& unlinkedWasmToWasmCalls, unsigned& osrEntryScratchBufferSize, const ModuleInformation& info, MemoryMode mode, CompilationMode compilationMode, uint32_t functionIndex, uint32_t loopIndexForOSREntry, TierUpCount* tierUp, ThrowWasmException throwWasmException)
    15531770{
    15541771    auto result = makeUnique<InternalFunction>();
    … …  
    15741791        : Options::webAssemblyOMGOptimizationLevel());
    15751792
    1576     B3IRGenerator irGenerator(info, procedure, result.get(), unlinkedWasmToWasmCalls, mode, compilationMode, functionIndex, tierUp, throwWasmException);
     1793    B3IRGenerator irGenerator(info, procedure, result.get(), unlinkedWasmToWasmCalls, osrEntryScratchBufferSize, mode, compilationMode, functionIndex, loopIndexForOSREntry, tierUp, throwWasmException);
    15771794    FunctionParser<B3IRGenerator> parser(irGenerator, functionStart, functionLength, signature, info);
    15781795    WASM_FAIL_IF_HELPER_FAILS(parser.parse());

  • trunk/Source/JavaScriptCore/wasm/WasmB3IRGenerator.h

    r248187 r248878  
    5252};
    5353
    54 Expected<std::unique_ptr<InternalFunction>, String> parseAndCompile(CompilationContext&, const uint8_t*, size_t, const Signature&, Vector<UnlinkedWasmToWasmCall>&, const ModuleInformation&, MemoryMode, CompilationMode, uint32_t functionIndex, TierUpCount* = nullptr, ThrowWasmException = nullptr);
     54Expected<std::unique_ptr<InternalFunction>, String> parseAndCompile(CompilationContext&, const uint8_t*, size_t, const Signature&, Vector<UnlinkedWasmToWasmCall>&, unsigned& osrEntryScratchBufferSize, const ModuleInformation&, MemoryMode, CompilationMode, uint32_t functionIndex, uint32_t loopIndexForOSREntry, TierUpCount* = nullptr, ThrowWasmException = nullptr);
    5555
    5656} } // namespace JSC::Wasm

  • trunk/Source/JavaScriptCore/wasm/WasmBBQPlan.cpp

    r248846 r248878  
    268268
    269269        m_unlinkedWasmToWasmCalls[functionIndex] = Vector<UnlinkedWasmToWasmCall>();
    270         TierUpCount* tierUp = Options::useBBQTierUpChecks() ? &m_tierUpCounts[functionIndex] : nullptr;
     270        if (Options::useBBQTierUpChecks())
     271            m_tierUpCounts[functionIndex] = makeUnique<TierUpCount>();
     272        else
     273            m_tierUpCounts[functionIndex] = nullptr;
     274        TierUpCount* tierUp = m_tierUpCounts[functionIndex].get();
    271275        Expected<std::unique_ptr<InternalFunction>, String> parseAndCompileResult;
     276        unsigned osrEntryScratchBufferSize = 0;
    272277
    273278        // FIXME: Some webpages use very large Wasm module, and it exhausts all executable memory in ARM64 devices since the size of executable memory region is only limited to 128MB.
    … …  
    281286            parseAndCompileResult = parseAndCompileAir(m_compilationContexts[functionIndex], function.data.data(), function.data.size(), signature, m_unlinkedWasmToWasmCalls[functionIndex], m_moduleInformation.get(), m_mode, functionIndex, tierUp, m_throwWasmException);
    282287        else
    283             parseAndCompileResult = parseAndCompile(m_compilationContexts[functionIndex], function.data.data(), function.data.size(), signature, m_unlinkedWasmToWasmCalls[functionIndex], m_moduleInformation.get(), m_mode, CompilationMode::BBQMode, functionIndex, tierUp, m_throwWasmException);
     288            parseAndCompileResult = parseAndCompile(m_compilationContexts[functionIndex], function.data.data(), function.data.size(), signature, m_unlinkedWasmToWasmCalls[functionIndex], osrEntryScratchBufferSize, m_moduleInformation.get(), m_mode, CompilationMode::BBQMode, functionIndex, UINT32_MAX, tierUp, m_throwWasmException);
    284289
    285290        if (UNLIKELY(!parseAndCompileResult)) {

  • trunk/Source/JavaScriptCore/wasm/WasmBBQPlan.h

    r235420 r248878  
    104104    }
    105105
    106     Vector<TierUpCount> takeTierUpCounts()
    107     {
    108         RELEASE_ASSERT(!failed() && !hasWork());
    109         return WTFMove(m_tierUpCounts);
    110     }
    111 
    112106    enum class State : uint8_t {
    113107        Initial,
    … …  
    147141    HashMap<uint32_t, std::unique_ptr<InternalFunction>, typename DefaultHash<uint32_t>::Hash, WTF::UnsignedWithZeroKeyHashTraits<uint32_t>> m_embedderToWasmInternalFunctions;
    148142    Vector<CompilationContext> m_compilationContexts;
    149     Vector<TierUpCount> m_tierUpCounts;
     143    Vector<std::unique_ptr<TierUpCount>> m_tierUpCounts;
    150144
    151145    Vector<Vector<UnlinkedWasmToWasmCall>> m_unlinkedWasmToWasmCalls;

  • trunk/Source/JavaScriptCore/wasm/WasmBBQPlanInlines.h

    r248187 r248878  
    4343        RefPtr<Wasm::Callee> embedderEntrypointCallee;
    4444        if (auto embedderToWasmFunction = m_embedderToWasmInternalFunctions.get(internalFunctionIndex)) {
    45             embedderEntrypointCallee = Wasm::Callee::create(CompilationMode::BBQMode, WTFMove(embedderToWasmFunction->entrypoint));
     45            embedderEntrypointCallee = Wasm::EmbedderEntrypointCallee::create(WTFMove(embedderToWasmFunction->entrypoint));
    4646            MacroAssembler::repatchPointer(embedderToWasmFunction->calleeMoveLocation, CalleeBits::boxWasm(embedderEntrypointCallee.get()));
    4747        }
    … …  
    4949        InternalFunction* function = m_wasmInternalFunctions[internalFunctionIndex].get();
    5050        size_t functionIndexSpace = internalFunctionIndex + m_moduleInformation->importFunctionCount();
    51         Ref<Wasm::Callee> wasmEntrypointCallee = Wasm::Callee::create(CompilationMode::BBQMode, WTFMove(function->entrypoint), functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace));
     51        Ref<Wasm::Callee> wasmEntrypointCallee = Wasm::BBQCallee::create(WTFMove(function->entrypoint), functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace), WTFMove(m_tierUpCounts[internalFunctionIndex]));
    5252        MacroAssembler::repatchPointer(function->calleeMoveLocation, CalleeBits::boxWasm(wasmEntrypointCallee.ptr()));
    5353

  • trunk/Source/JavaScriptCore/wasm/WasmCallee.h

    r248187 r248878  
    3333#include "WasmFormat.h"
    3434#include "WasmIndexOrName.h"
     35#include "WasmTierUpCount.h"
    3536#include <wtf/ThreadSafeRefCounted.h>
    3637
    3738namespace JSC { namespace Wasm {
    3839
    39 class Callee final : public ThreadSafeRefCounted<Callee> {
     40class Callee : public ThreadSafeRefCounted<Callee> {
    4041    WTF_MAKE_FAST_ALLOCATED;
    4142public:
    42     static Ref<Callee> create(Wasm::CompilationMode compilationMode, Wasm::Entrypoint&& entrypoint)
    43     {
    44         Callee* callee = new Callee(compilationMode, WTFMove(entrypoint));
    45         return adoptRef(*callee);
    46     }
    47 
    48     static Ref<Callee> create(Wasm::CompilationMode compilationMode, Wasm::Entrypoint&& entrypoint, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name)
    49     {
    50         Callee* callee = new Callee(compilationMode, WTFMove(entrypoint), index, WTFMove(name));
    51         return adoptRef(*callee);
    52     }
    53 
    5443    MacroAssemblerCodePtr<WasmEntryPtrTag> entrypoint() const { return m_entrypoint.compilation->code().retagged<WasmEntryPtrTag>(); }
    5544
    … …  
    6554    }
    6655
    67     JS_EXPORT_PRIVATE ~Callee();
     56    JS_EXPORT_PRIVATE virtual ~Callee();
    6857
    69 private:
     58protected:
    7059    JS_EXPORT_PRIVATE Callee(Wasm::CompilationMode, Wasm::Entrypoint&&);
    7160    JS_EXPORT_PRIVATE Callee(Wasm::CompilationMode, Wasm::Entrypoint&&, size_t, std::pair<const Name*, RefPtr<NameSection>>&&);
    7261
     62private:
    7363    CompilationMode m_compilationMode;
    7464    Wasm::Entrypoint m_entrypoint;
    … …  
    7666};
    7767
     68class OMGCallee final : public Callee {
     69public:
     70    static Ref<OMGCallee> create(Wasm::Entrypoint&& entrypoint, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, Vector<UnlinkedWasmToWasmCall>&& unlinkedCalls)
     71    {
     72        return adoptRef(*new OMGCallee(WTFMove(entrypoint), index, WTFMove(name), WTFMove(unlinkedCalls)));
     73    }
     74
     75    Vector<UnlinkedWasmToWasmCall>& wasmToWasmCallsites() { return m_wasmToWasmCallsites; }
     76
     77private:
     78    OMGCallee(Wasm::Entrypoint&& entrypoint, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, Vector<UnlinkedWasmToWasmCall>&& unlinkedCalls)
     79        : Callee(Wasm::CompilationMode::OMGMode, WTFMove(entrypoint), index, WTFMove(name))
     80        , m_wasmToWasmCallsites(WTFMove(unlinkedCalls))
     81    {
     82    }
     83
     84    Vector<UnlinkedWasmToWasmCall> m_wasmToWasmCallsites;
     85};
     86
     87class OMGForOSREntryCallee final : public Callee {
     88public:
     89    static Ref<OMGForOSREntryCallee> create(Wasm::Entrypoint&& entrypoint, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, unsigned osrEntryScratchBufferSize, uint32_t loopIndex, Vector<UnlinkedWasmToWasmCall>&& unlinkedCalls)
     90    {
     91        return adoptRef(*new OMGForOSREntryCallee(WTFMove(entrypoint), index, WTFMove(name), osrEntryScratchBufferSize, loopIndex, WTFMove(unlinkedCalls)));
     92    }
     93
     94    unsigned osrEntryScratchBufferSize() const { return m_osrEntryScratchBufferSize; }
     95    uint32_t loopIndex() const { return m_loopIndex; }
     96    Vector<UnlinkedWasmToWasmCall>& wasmToWasmCallsites() { return m_wasmToWasmCallsites; }
     97
     98private:
     99    OMGForOSREntryCallee(Wasm::Entrypoint&& entrypoint, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, unsigned osrEntryScratchBufferSize, uint32_t loopIndex, Vector<UnlinkedWasmToWasmCall>&& unlinkedCalls)
     100        : Callee(Wasm::CompilationMode::OMGForOSREntryMode, WTFMove(entrypoint), index, WTFMove(name))
     101        , m_wasmToWasmCallsites(WTFMove(unlinkedCalls))
     102        , m_osrEntryScratchBufferSize(osrEntryScratchBufferSize)
     103        , m_loopIndex(loopIndex)
     104    {
     105    }
     106
     107    Vector<UnlinkedWasmToWasmCall> m_wasmToWasmCallsites;
     108    unsigned m_osrEntryScratchBufferSize;
     109    uint32_t m_loopIndex;
     110};
     111
     112class EmbedderEntrypointCallee final : public Callee {
     113public:
     114    static Ref<EmbedderEntrypointCallee> create(Wasm::Entrypoint&& entrypoint)
     115    {
     116        return adoptRef(*new EmbedderEntrypointCallee(WTFMove(entrypoint)));
     117    }
     118
     119private:
     120    EmbedderEntrypointCallee(Wasm::Entrypoint&& entrypoint)
     121        : Callee(Wasm::CompilationMode::EmbedderEntrypointMode, WTFMove(entrypoint))
     122    {
     123    }
     124};
     125
     126class BBQCallee final : public Callee {
     127public:
     128    static Ref<BBQCallee> create(Wasm::Entrypoint&& entrypoint, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, std::unique_ptr<TierUpCount>&& tierUpCount)
     129    {
     130        return adoptRef(*new BBQCallee(WTFMove(entrypoint), index, WTFMove(name), WTFMove(tierUpCount)));
     131    }
     132
     133    OMGForOSREntryCallee* osrEntryCallee() { return m_osrEntryCallee.get(); }
     134    void setOSREntryCallee(Ref<OMGForOSREntryCallee>&& osrEntryCallee)
     135    {
     136        m_osrEntryCallee = WTFMove(osrEntryCallee);
     137    }
     138
     139    bool didStartCompilingOSREntryCallee() const { return m_didStartCompilingOSREntryCallee; }
     140    void setDidStartCompilingOSREntryCallee(bool value) { m_didStartCompilingOSREntryCallee = value; }
     141
     142    OMGCallee* replacement() { return m_replacement.get(); }
     143    void setReplacement(Ref<OMGCallee>&& replacement)
     144    {
     145        m_replacement = WTFMove(replacement);
     146    }
     147
     148    TierUpCount* tierUpCount() { return m_tierUpCount.get(); }
     149
     150private:
     151    BBQCallee(Wasm::Entrypoint&& entrypoint, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, std::unique_ptr<TierUpCount>&& tierUpCount)
     152        : Callee(Wasm::CompilationMode::BBQMode, WTFMove(entrypoint), index, WTFMove(name))
     153        , m_tierUpCount(WTFMove(tierUpCount))
     154    {
     155    }
     156
     157    RefPtr<OMGForOSREntryCallee> m_osrEntryCallee;
     158    RefPtr<OMGCallee> m_replacement;
     159    std::unique_ptr<TierUpCount> m_tierUpCount;
     160    bool m_didStartCompilingOSREntryCallee { false };
     161};
     162
    78163} } // namespace JSC::Wasm
    79164

  • trunk/Source/JavaScriptCore/wasm/WasmCodeBlock.cpp

    r230096 r248878  
    7272        m_wasmToWasmExitStubs = m_plan->takeWasmToWasmExitStubs();
    7373        m_wasmToWasmCallsites = m_plan->takeWasmToWasmCallsites();
    74         m_tierUpCounts = m_plan->takeTierUpCounts();
    7574
    7675        setCompilationFinished();

  • trunk/Source/JavaScriptCore/wasm/WasmCodeBlock.h

    r230748 r248878  
    9797    }
    9898
     99    Callee& wasmBBQCalleeFromFunctionIndexSpace(unsigned functionIndexSpace)
     100    {
     101        ASSERT(runnable());
     102        RELEASE_ASSERT(functionIndexSpace >= functionImportCount());
     103        unsigned calleeIndex = functionIndexSpace - functionImportCount();
     104        return *m_callees[calleeIndex].get();
     105    }
     106
    99107    MacroAssemblerCodePtr<WasmEntryPtrTag>* entrypointLoadLocationFromFunctionIndexSpace(unsigned functionIndexSpace)
    100108    {
    … …  
    102110        unsigned calleeIndex = functionIndexSpace - functionImportCount();
    103111        return &m_wasmIndirectCallEntryPoints[calleeIndex];
    104     }
    105 
    106     TierUpCount& tierUpCount(uint32_t functionIndex)
    107     {
    108         return m_tierUpCounts[functionIndex];
    109112    }
    110113
    … …  
    116119private:
    117120    friend class OMGPlan;
     121    friend class OMGForOSREntryPlan;
    118122
    119123    CodeBlock(Context*, MemoryMode, ModuleInformation&, CreateEmbedderWrapper&&, ThrowWasmException);
    … …  
    125129    HashMap<uint32_t, RefPtr<Callee>, typename DefaultHash<uint32_t>::Hash, WTF::UnsignedWithZeroKeyHashTraits<uint32_t>> m_embedderCallees;
    126130    Vector<MacroAssemblerCodePtr<WasmEntryPtrTag>> m_wasmIndirectCallEntryPoints;
    127     Vector<TierUpCount> m_tierUpCounts;
    128131    Vector<Vector<UnlinkedWasmToWasmCall>> m_wasmToWasmCallsites;
    129132    Vector<MacroAssemblerCodeRef<WasmEntryPtrTag>> m_wasmToWasmExitStubs;

  • trunk/Source/JavaScriptCore/wasm/WasmCompilationMode.cpp

    r248187 r248878  
    3838    case CompilationMode::OMGMode:
    3939        return "OMG";
     40    case CompilationMode::OMGForOSREntryMode:
     41        return "OMGForOSREntry";
     42    case CompilationMode::EmbedderEntrypointMode:
     43        return "EmbedderEntrypoint";
    4044    }
    4145    RELEASE_ASSERT_NOT_REACHED();

  • trunk/Source/JavaScriptCore/wasm/WasmCompilationMode.h

    r248187 r248878  
    3131    BBQMode,
    3232    OMGMode,
     33    OMGForOSREntryMode,
     34    EmbedderEntrypointMode,
    3335};
    3436

  • trunk/Source/JavaScriptCore/wasm/WasmContext.cpp

    r248876 r248878  
    2525
    2626#include "config.h"
    27 #include "WasmCompilationMode.h"
     27#include "WasmContext.h"
    2828
    29 #include <wtf/Assertions.h>
     29#if ENABLE(WEBASSEMBLY)
    3030
    3131namespace JSC { namespace Wasm {
    3232
    33 const char* makeString(CompilationMode mode)
     33uint64_t* Context::scratchBufferForSize(size_t size)
    3434{
    35     switch (mode) {
    36     case CompilationMode::BBQMode:
    37         return "BBQ";
    38     case CompilationMode::OMGMode:
    39         return "OMG";
     35    if (!size)
     36        return nullptr;
     37
     38    auto locker = holdLock(m_scratchBufferLock);
     39    if (size > m_sizeOfLastScratchBuffer) {
     40        m_sizeOfLastScratchBuffer = size * 2;
     41
     42        auto newBuffer = makeUniqueArray<uint64_t>(m_sizeOfLastScratchBuffer);
     43        RELEASE_ASSERT(newBuffer);
     44        m_scratchBuffers.append(WTFMove(newBuffer));
    4045    }
    41     RELEASE_ASSERT_NOT_REACHED();
    42     return "";
     46    // Scanning scratch buffers for GC is not necessary since while performing OSR entry, we do not perform GC.
     47    return m_scratchBuffers.last().get();
    4348}
    4449
    4550} } // namespace JSC::Wasm
     51
     52#endif // ENABLE(WEBASSEMBLY)

  • trunk/Source/JavaScriptCore/wasm/WasmContext.h

    r243886 r248878  
    2828#if ENABLE(WEBASSEMBLY)
    2929
     30#include <wtf/Lock.h>
     31#include <wtf/UniqueArray.h>
     32#include <wtf/Vector.h>
     33
    3034namespace JSC { namespace Wasm {
    3135
    … …  
    4448    }
    4549
     50    static Instance* tryLoadInstanceFromTLS();
     51
     52    uint64_t* scratchBufferForSize(size_t numberOfSlots);
     53
    4654private:
    4755    Instance* instance { nullptr };
     56    Vector<UniqueArray<uint64_t>> m_scratchBuffers;
     57    size_t m_sizeOfLastScratchBuffer { 0 };
     58    Lock m_scratchBufferLock;
    4859};
    4960

  • trunk/Source/JavaScriptCore/wasm/WasmContextInlines.h

    r243886 r248878  
    6767}
    6868
     69inline Instance* Context::tryLoadInstanceFromTLS()
     70{
     71#if ENABLE(FAST_TLS_JIT)
     72    if (useFastTLS())
     73        return bitwise_cast<Instance*>(_pthread_getspecific_direct(WTF_WASM_CONTEXT_KEY));
     74#endif
     75    return nullptr;
     76}
     77
    6978} } // namespace JSC::Wasm
    7079

  • trunk/Source/JavaScriptCore/wasm/WasmFunctionParser.h

    r246589 r248878  
    4444class FunctionParser : public Parser<void> {
    4545public:
    46     typedef typename Context::ExpressionType ExpressionType;
    47     typedef typename Context::ControlType ControlType;
    48     typedef typename Context::ExpressionList ExpressionList;
     46    using ExpressionType = typename Context::ExpressionType;
     47    using ControlType = typename Context::ControlType;
     48    using ExpressionList = typename Context::ExpressionList;
     49    using Stack = typename Context::Stack;
    4950
    5051    FunctionParser(Context&, const uint8_t* functionStart, size_t functionLength, const Signature&, const ModuleInformation&);
    … …  
    5354
    5455    struct ControlEntry {
    55         ExpressionList enclosedExpressionStack;
     56        Stack enclosedExpressionStack;
    5657        ControlType controlData;
    5758    };
    … …  
    8586
    8687    Context& m_context;
    87     ExpressionList m_expressionStack;
     88    Stack m_expressionStack;
    8889    Vector<ControlEntry> m_controlStack;
    8990    const Signature& m_signature;
    … …  
    9697
    9798    unsigned m_unreachableBlocks { 0 };
     99    unsigned m_loopIndex { 0 };
    98100};
    99101
    … …  
    102104    : Parser(functionStart, functionLength)
    103105    , m_context(context)
     106    , m_expressionStack(context.createStack())
    104107    , m_signature(signature)
    105108    , m_info(info)
    … …  
    137140auto FunctionParser<Context>::parseBody() -> PartialResult
    138141{
    139     m_controlStack.append({ ExpressionList(), m_context.addTopLevel(m_signature.returnType()) });
     142    m_controlStack.append({ m_context.createStack(), m_context.addTopLevel(m_signature.returnType()) });
    140143    uint8_t op;
    141144    while (m_controlStack.size()) {
    … …  
    424427        WASM_PARSER_FAIL_IF(!args.tryReserveCapacity(calleeSignature.argumentCount()), "can't allocate enough memory for call's ", calleeSignature.argumentCount(), " arguments");
    425428        for (size_t i = firstArgumentIndex; i < m_expressionStack.size(); ++i)
    426             args.uncheckedAppend(m_expressionStack[i]);
     429            args.uncheckedAppend(m_expressionStack.at(i));
    427430        m_expressionStack.shrink(firstArgumentIndex);
    428431
    … …  
    454457        size_t firstArgumentIndex = m_expressionStack.size() - argumentCount;
    455458        for (size_t i = firstArgumentIndex; i < m_expressionStack.size(); ++i)
    456             args.uncheckedAppend(m_expressionStack[i]);
     459            args.uncheckedAppend(m_expressionStack.at(i));
    457460        m_expressionStack.shrink(firstArgumentIndex);
    458461
    … …  
    470473        WASM_PARSER_FAIL_IF(!parseResultType(inlineSignature), "can't get block's inline signature");
    471474        m_controlStack.append({ WTFMove(m_expressionStack), m_context.addBlock(inlineSignature) });
    472         m_expressionStack = ExpressionList();
     475        m_expressionStack = m_context.createStack();
    473476        return { };
    474477    }
    … …  
    477480        Type inlineSignature;
    478481        WASM_PARSER_FAIL_IF(!parseResultType(inlineSignature), "can't get loop's inline signature");
    479         m_controlStack.append({ WTFMove(m_expressionStack), m_context.addLoop(inlineSignature) });
    480         m_expressionStack = ExpressionList();
     482        auto expressionStack = WTFMove(m_expressionStack);
     483        auto loop = m_context.addLoop(inlineSignature, expressionStack, m_loopIndex++);
     484        m_controlStack.append({ expressionStack, loop });
     485        m_expressionStack = m_context.createStack();
    481486        return { };
    482487    }
    … …  
    490495        WASM_TRY_ADD_TO_CONTEXT(addIf(condition, inlineSignature, control));
    491496        m_controlStack.append({ WTFMove(m_expressionStack), control });
    492         m_expressionStack = ExpressionList();
     497        m_expressionStack = m_context.createStack();
    493498        return { };
    494499    }

  • trunk/Source/JavaScriptCore/wasm/WasmOMGForOSREntryPlan.cpp

    r248876 r248878  
    11/*
    2  * Copyright (C) 2017-2018 Apple Inc. All rights reserved.
     2 * Copyright (C) 2019 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2525
    2626#include "config.h"
    27 #include "WasmOMGPlan.h"
     27#include "WasmOMGForOSREntryPlan.h"
    2828
    2929#if ENABLE(WEBASSEMBLY)
    … …  
    5151namespace JSC { namespace Wasm {
    5252
    53 namespace WasmOMGPlanInternal {
     53namespace WasmOMGForOSREntryPlanInternal {
    5454static const bool verbose = false;
    5555}
    5656
    57 OMGPlan::OMGPlan(Context* context, Ref<Module>&& module, uint32_t functionIndex, MemoryMode mode, CompletionTask&& task)
     57OMGForOSREntryPlan::OMGForOSREntryPlan(Context* context, Ref<Module>&& module, Ref<BBQCallee>&& callee, uint32_t functionIndex, uint32_t loopIndex, MemoryMode mode, CompletionTask&& task)
    5858    : Base(context, makeRef(const_cast<ModuleInformation&>(module->moduleInformation())), WTFMove(task))
    5959    , m_module(WTFMove(module))
    6060    , m_codeBlock(*m_module->codeBlockFor(mode))
     61    , m_callee(WTFMove(callee))
    6162    , m_functionIndex(functionIndex)
     63    , m_loopIndex(loopIndex)
    6264{
    6365    setMode(mode);
    6466    ASSERT(m_codeBlock->runnable());
    6567    ASSERT(m_codeBlock.ptr() == m_module->codeBlockFor(m_mode));
    66     dataLogLnIf(WasmOMGPlanInternal::verbose, "Starting OMG plan for ", functionIndex, " of module: ", RawPointer(&m_module.get()));
     68    dataLogLnIf(WasmOMGForOSREntryPlanInternal::verbose, "Starting OMGForOSREntry plan for ", functionIndex, " of module: ", RawPointer(&m_module.get()));
    6769}
    6870
    69 void OMGPlan::work(CompilationEffort)
     71void OMGForOSREntryPlan::work(CompilationEffort)
    7072{
    7173    ASSERT(m_codeBlock->runnable());
    … …  
    8284    Vector<UnlinkedWasmToWasmCall> unlinkedCalls;
    8385    CompilationContext context;
    84     auto parseAndCompileResult = parseAndCompile(context, function.data.data(), function.data.size(), signature, unlinkedCalls, m_moduleInformation.get(), m_mode, CompilationMode::OMGMode, m_functionIndex);
     86    unsigned osrEntryScratchBufferSize = 0;
     87    auto parseAndCompileResult = parseAndCompile(context, function.data.data(), function.data.size(), signature, unlinkedCalls, osrEntryScratchBufferSize, m_moduleInformation.get(), m_mode, CompilationMode::OMGForOSREntryMode, m_functionIndex, m_loopIndex);
    8588
    8689    if (UNLIKELY(!parseAndCompileResult)) {
    … …  
    97100
    98101    omgEntrypoint.compilation = makeUnique<B3::Compilation>(
    99         FINALIZE_CODE(linkBuffer, B3CompilationPtrTag, "WebAssembly OMG function[%i] %s name %s", m_functionIndex, signature.toString().ascii().data(), makeString(IndexOrName(functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace))).ascii().data()),
     102        FINALIZE_CODE(linkBuffer, B3CompilationPtrTag, "WebAssembly OMGForOSREntry function[%i] %s name %s", m_functionIndex, signature.toString().ascii().data(), makeString(IndexOrName(functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace))).ascii().data()),
    100103        WTFMove(context.wasmEntrypointByproducts));
    101104
    … …  
    103106
    104107    MacroAssemblerCodePtr<WasmEntryPtrTag> entrypoint;
     108    ASSERT(m_codeBlock.ptr() == m_module->codeBlockFor(mode()));
     109    Ref<OMGForOSREntryCallee> callee = OMGForOSREntryCallee::create(WTFMove(omgEntrypoint), functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace), osrEntryScratchBufferSize, m_loopIndex, WTFMove(unlinkedCalls));
    105110    {
    106         ASSERT(m_codeBlock.ptr() == m_module->codeBlockFor(mode()));
    107         Ref<Callee> callee = Callee::create(CompilationMode::OMGMode, WTFMove(omgEntrypoint), functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace));
    108111        MacroAssembler::repatchPointer(parseAndCompileResult.value()->calleeMoveLocation, CalleeBits::boxWasm(callee.ptr()));
    109         ASSERT(!m_codeBlock->m_optimizedCallees[m_functionIndex]);
    110112        entrypoint = callee->entrypoint();
    111113
    112         // We want to make sure we publish our callee at the same time as we link our callsites. This enables us to ensure we
    113         // always call the fastest code. Any function linked after us will see our new code and the new callsites, which they
    114         // will update. It's also ok if they publish their code before we reset the instruction caches because after we release
    115         // the lock our code is ready to be published too.
    116         LockHolder holder(m_codeBlock->m_lock);
    117         m_codeBlock->m_optimizedCallees[m_functionIndex] = WTFMove(callee);
    118 
    119         for (auto& call : unlinkedCalls) {
     114        auto locker = holdLock(m_codeBlock->m_lock);
     115        for (auto& call : callee->wasmToWasmCallsites()) {
    120116            MacroAssemblerCodePtr<WasmEntryPtrTag> entrypoint;
    121117            if (call.functionIndexSpace < m_module->moduleInformation().importFunctionCount())
    … …  
    126122            MacroAssembler::repatchNearCall(call.callLocation, CodeLocationLabel<WasmEntryPtrTag>(entrypoint));
    127123        }
    128         unlinkedCalls = std::exchange(m_codeBlock->m_wasmToWasmCallsites[m_functionIndex], unlinkedCalls);
    129124    }
     125    resetInstructionCacheOnAllThreads();
     126    WTF::storeStoreFence();
     127    {
     128        auto locker = holdLock(m_codeBlock->m_lock);
     129        {
     130            auto locker = holdLock(m_callee->tierUpCount()->getLock());
     131            m_callee->setOSREntryCallee(callee.copyRef());
     132            m_callee->tierUpCount()->osrEntryTriggers()[m_loopIndex] = TierUpCount::TriggerReason::CompilationDone;
     133            m_callee->tierUpCount()->m_compilationStatusForOMGForOSREntry = TierUpCount::CompilationStatus::Compiled;
     134        }
     135        WTF::storeStoreFence();
     136        // It is possible that a new OMG callee is added while we release m_codeBlock->lock.
     137        // Until we add OMGForOSREntry callee to BBQCallee's m_osrEntryCallee, this new OMG function linking does not happen for this OMGForOSREntry callee.
     138        // We re-link this OMGForOSREntry callee again not to miss that chance.
     139        for (auto& call : callee->wasmToWasmCallsites()) {
     140            MacroAssemblerCodePtr<WasmEntryPtrTag> entrypoint;
     141            if (call.functionIndexSpace < m_module->moduleInformation().importFunctionCount())
     142                entrypoint = m_codeBlock->m_wasmToWasmExitStubs[call.functionIndexSpace].code();
     143            else
     144                entrypoint = m_codeBlock->wasmEntrypointCalleeFromFunctionIndexSpace(call.functionIndexSpace).entrypoint().retagged<WasmEntryPtrTag>();
    130145
    131     // It's important to make sure we do this before we make any of the code we just compiled visible. If we didn't, we could end up
    132     // where we are tiering up some function A to A' and we repatch some function B to call A' instead of A. Another CPU could see
    133     // the updates to B but still not have reset its cache of A', which would lead to all kinds of badness.
    134     resetInstructionCacheOnAllThreads();
    135     WTF::storeStoreFence(); // This probably isn't necessary but it's good to be paranoid.
    136 
    137     m_codeBlock->m_wasmIndirectCallEntryPoints[m_functionIndex] = entrypoint;
    138     {
    139         LockHolder holder(m_codeBlock->m_lock);
    140 
    141         auto repatchCalls = [&] (const Vector<UnlinkedWasmToWasmCall>& callsites) {
    142             for (auto& call : callsites) {
    143                 dataLogLnIf(WasmOMGPlanInternal::verbose, "Considering repatching call at: ", RawPointer(call.callLocation.dataLocation()), " that targets ", call.functionIndexSpace);
    144                 if (call.functionIndexSpace == functionIndexSpace) {
    145                     dataLogLnIf(WasmOMGPlanInternal::verbose, "Repatching call at: ", RawPointer(call.callLocation.dataLocation()), " to ", RawPointer(entrypoint.executableAddress()));
    146                     MacroAssembler::repatchNearCall(call.callLocation, CodeLocationLabel<WasmEntryPtrTag>(entrypoint));
    147                 }
    148             }
    149 
    150         };
    151 
    152         for (unsigned i = 0; i < m_codeBlock->m_wasmToWasmCallsites.size(); ++i) {
    153             if (i != functionIndexSpace)
    154                 repatchCalls(m_codeBlock->m_wasmToWasmCallsites[i]);
     146            MacroAssembler::repatchNearCall(call.callLocation, CodeLocationLabel<WasmEntryPtrTag>(entrypoint));
    155147        }
    156 
    157         // Make sure we repatch any recursive calls.
    158         repatchCalls(unlinkedCalls);
    159148    }
    160 
    161     dataLogLnIf(WasmOMGPlanInternal::verbose, "Finished with tier up count at: ", m_codeBlock->tierUpCount(m_functionIndex).count());
     149    dataLogLnIf(WasmOMGForOSREntryPlanInternal::verbose, "Finished OMGForOSREntry ", m_functionIndex, " with tier up count at: ", m_callee->tierUpCount()->count());
    162150    complete(holdLock(m_lock));
    163 }
    164 
    165 void OMGPlan::runForIndex(Instance* instance, uint32_t functionIndex)
    166 {
    167     Wasm::CodeBlock& codeBlock = *instance->codeBlock();
    168     ASSERT(instance->memory()->mode() == codeBlock.mode());
    169 
    170     if (codeBlock.tierUpCount(functionIndex).shouldStartTierUp()) {
    171         Ref<Plan> plan = adoptRef(*new OMGPlan(instance->context(), Ref<Wasm::Module>(instance->module()), functionIndex, codeBlock.mode(), Plan::dontFinalize()));
    172         ensureWorklist().enqueue(plan.copyRef());
    173         if (UNLIKELY(!Options::useConcurrentJIT()))
    174             plan->waitForCompletion();
    175     }
    176151}
    177152

  • trunk/Source/JavaScriptCore/wasm/WasmOMGForOSREntryPlan.h

    r248876 r248878  
    11/*
    2  * Copyright (C) 2017 Apple Inc. All rights reserved.
     2 * Copyright (C) 2019 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3030#include "WasmContext.h"
    3131#include "WasmModule.h"
     32#include "WasmOperations.h"
    3233#include "WasmPlan.h"
    3334
    3435namespace JSC {
    3536
     37class BBQCallee;
    3638class CallLinkInfo;
    3739
    3840namespace Wasm {
    3941
    40 class OMGPlan final : public Plan {
     42class OMGForOSREntryPlan final : public Plan {
    4143public:
    4244    using Base = Plan;
    … …  
    4648    bool multiThreaded() const override { return false; }
    4749
    48     static void runForIndex(Instance*, uint32_t functionIndex);
     50    // Note: CompletionTask should not hold a reference to the Plan otherwise there will be a reference cycle.
     51    OMGForOSREntryPlan(Context*, Ref<Module>&&, Ref<BBQCallee>&&, uint32_t functionIndex, uint32_t loopIndex, MemoryMode, CompletionTask&&);
    4952
    5053private:
    5154    // For some reason friendship doesn't extend to parent classes...
    5255    using Base::m_lock;
    53 
    54     // Note: CompletionTask should not hold a reference to the Plan otherwise there will be a reference cycle.
    55     OMGPlan(Context*, Ref<Module>&&, uint32_t functionIndex, MemoryMode, CompletionTask&&);
    5656
    5757    bool isComplete() const override { return m_completed; }
    … …  
    6161        runCompletionTasks(locker);
    6262    }
    63    
     63
    6464    Ref<Module> m_module;
    6565    Ref<CodeBlock> m_codeBlock;
     66    Ref<BBQCallee> m_callee;
    6667    bool m_completed { false };
    6768    uint32_t m_functionIndex;
     69    uint32_t m_loopIndex;
    6870};
    6971

  • trunk/Source/JavaScriptCore/wasm/WasmOMGPlan.cpp

    r248846 r248878  
    8181
    8282    Vector<UnlinkedWasmToWasmCall> unlinkedCalls;
     83    unsigned osrEntryScratchBufferSize;
    8384    CompilationContext context;
    84     auto parseAndCompileResult = parseAndCompile(context, function.data.data(), function.data.size(), signature, unlinkedCalls, m_moduleInformation.get(), m_mode, CompilationMode::OMGMode, m_functionIndex);
     85    auto parseAndCompileResult = parseAndCompile(context, function.data.data(), function.data.size(), signature, unlinkedCalls, osrEntryScratchBufferSize, m_moduleInformation.get(), m_mode, CompilationMode::OMGMode, m_functionIndex, UINT32_MAX);
    8586
    8687    if (UNLIKELY(!parseAndCompileResult)) {
    … …  
    105106    {
    106107        ASSERT(m_codeBlock.ptr() == m_module->codeBlockFor(mode()));
    107         Ref<Callee> callee = Callee::create(CompilationMode::OMGMode, WTFMove(omgEntrypoint), functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace));
     108        Ref<OMGCallee> callee = OMGCallee::create(WTFMove(omgEntrypoint), functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace), WTFMove(unlinkedCalls));
    108109        MacroAssembler::repatchPointer(parseAndCompileResult.value()->calleeMoveLocation, CalleeBits::boxWasm(callee.ptr()));
    109110        ASSERT(!m_codeBlock->m_optimizedCallees[m_functionIndex]);
    … …  
    115116        // the lock our code is ready to be published too.
    116117        LockHolder holder(m_codeBlock->m_lock);
    117         m_codeBlock->m_optimizedCallees[m_functionIndex] = WTFMove(callee);
    118 
    119         for (auto& call : unlinkedCalls) {
     118        m_codeBlock->m_optimizedCallees[m_functionIndex] = callee.copyRef();
     119        {
     120            BBQCallee& bbqCallee = *static_cast<BBQCallee*>(m_codeBlock->m_callees[m_functionIndex].get());
     121            auto locker = holdLock(bbqCallee.tierUpCount()->getLock());
     122            bbqCallee.setReplacement(callee.copyRef());
     123            bbqCallee.tierUpCount()->m_compilationStatusForOMG = TierUpCount::CompilationStatus::Compiled;
     124        }
     125        for (auto& call : callee->wasmToWasmCallsites()) {
    120126            MacroAssemblerCodePtr<WasmEntryPtrTag> entrypoint;
    121127            if (call.functionIndexSpace < m_module->moduleInformation().importFunctionCount())
    … …  
    126132            MacroAssembler::repatchNearCall(call.callLocation, CodeLocationLabel<WasmEntryPtrTag>(entrypoint));
    127133        }
    128         unlinkedCalls = std::exchange(m_codeBlock->m_wasmToWasmCallsites[m_functionIndex], unlinkedCalls);
    129134    }
    130135
    … …  
    147152                }
    148153            }
    149 
    150154        };
    151155
    152156        for (unsigned i = 0; i < m_codeBlock->m_wasmToWasmCallsites.size(); ++i) {
    153             if (i != functionIndexSpace)
    154                 repatchCalls(m_codeBlock->m_wasmToWasmCallsites[i]);
     157            repatchCalls(m_codeBlock->m_wasmToWasmCallsites[i]);
     158            if (OMGCallee* replacementCallee = static_cast<BBQCallee*>(m_codeBlock->m_callees[i].get())->replacement())
     159                repatchCalls(replacementCallee->wasmToWasmCallsites());
     160            if (OMGForOSREntryCallee* osrEntryCallee = static_cast<BBQCallee*>(m_codeBlock->m_callees[i].get())->osrEntryCallee())
     161                repatchCalls(osrEntryCallee->wasmToWasmCallsites());
    155162        }
    156 
    157         // Make sure we repatch any recursive calls.
    158         repatchCalls(unlinkedCalls);
    159163    }
    160164
    161     dataLogLnIf(WasmOMGPlanInternal::verbose, "Finished with tier up count at: ", m_codeBlock->tierUpCount(m_functionIndex).count());
     165    dataLogLnIf(WasmOMGPlanInternal::verbose, "Finished OMG ", m_functionIndex, " with tier up count at: ", static_cast<BBQCallee*>(m_codeBlock->m_callees[m_functionIndex].get())->tierUpCount()->count());
    162166    complete(holdLock(m_lock));
    163 }
    164 
    165 void OMGPlan::runForIndex(Instance* instance, uint32_t functionIndex)
    166 {
    167     Wasm::CodeBlock& codeBlock = *instance->codeBlock();
    168     ASSERT(instance->memory()->mode() == codeBlock.mode());
    169 
    170     if (codeBlock.tierUpCount(functionIndex).shouldStartTierUp()) {
    171         Ref<Plan> plan = adoptRef(*new OMGPlan(instance->context(), Ref<Wasm::Module>(instance->module()), functionIndex, codeBlock.mode(), Plan::dontFinalize()));
    172         ensureWorklist().enqueue(plan.copyRef());
    173         if (UNLIKELY(!Options::useConcurrentJIT()))
    174             plan->waitForCompletion();
    175     }
    176167}
    177168

  • trunk/Source/JavaScriptCore/wasm/WasmOMGPlan.h

    r224020 r248878  
    4646    bool multiThreaded() const override { return false; }
    4747
    48     static void runForIndex(Instance*, uint32_t functionIndex);
     48    // Note: CompletionTask should not hold a reference to the Plan otherwise there will be a reference cycle.
     49    OMGPlan(Context*, Ref<Module>&&, uint32_t functionIndex, MemoryMode, CompletionTask&&);
    4950
    5051private:
    5152    // For some reason friendship doesn't extend to parent classes...
    5253    using Base::m_lock;
    53 
    54     // Note: CompletionTask should not hold a reference to the Plan otherwise there will be a reference cycle.
    55     OMGPlan(Context*, Ref<Module>&&, uint32_t functionIndex, MemoryMode, CompletionTask&&);
    5654
    5755    bool isComplete() const override { return m_completed; }

  • trunk/Source/JavaScriptCore/wasm/WasmOSREntryData.h

    r248876 r248878  
    11/*
    2  * Copyright (C) 2017 Apple Inc. All rights reserved.
     2 * Copyright (C) 2019 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2828#if ENABLE(WEBASSEMBLY)
    2929
     30#include "B3ValueRep.h"
     31#include "WasmFormat.h"
     32#include <wtf/Vector.h>
     33
    3034namespace JSC { namespace Wasm {
    3135
    32 class Instance;
    33 
    34 struct Context {
    35     Instance* load() const;
    36     void store(Instance*, void* softStackLimit);
    37 
    38     static bool useFastTLS();
    39 
    40     Instance** pointerToInstance()
     36class OSREntryValue : public B3::ValueRep {
     37public:
     38    OSREntryValue(const B3::ValueRep& valueRep, B3::Type type)
     39        : B3::ValueRep(valueRep)
     40        , m_type(type)
    4141    {
    42         ASSERT(!useFastTLS());
    43         return &instance;
    4442    }
    4543
     44    B3::Type type() const { return m_type; }
     45
    4646private:
    47     Instance* instance { nullptr };
     47    B3::Type m_type;
     48};
     49
     50class OSREntryData {
     51    WTF_MAKE_NONCOPYABLE(OSREntryData);
     52    WTF_MAKE_FAST_ALLOCATED;
     53public:
     54    OSREntryData(uint32_t functionIndex, uint32_t loopIndex)
     55        : m_functionIndex(functionIndex)
     56        , m_loopIndex(loopIndex)
     57    {
     58    }
     59
     60    uint32_t functionIndex() const { return m_functionIndex; }
     61    uint32_t loopIndex() const { return m_loopIndex; }
     62    Vector<OSREntryValue>& values() { return m_values; }
     63
     64private:
     65    uint32_t m_functionIndex;
     66    uint32_t m_loopIndex;
     67    Vector<OSREntryValue> m_values;
    4868};
    4969

  • trunk/Source/JavaScriptCore/wasm/WasmOperations.h

    r248876 r248878  
    2626#pragma once
    2727
    28 namespace JSC { namespace Wasm {
     28#if ENABLE(WEBASSEMBLY)
    2929
    30 enum class CompilationMode : uint8_t {
    31     BBQMode,
    32     OMGMode,
    33 };
     30#include "SlowPathReturnType.h"
    3431
    35 const char* makeString(CompilationMode);
     32namespace JSC {
     33namespace Probe {
     34class Context;
     35} // namespace JSC::Probe
     36namespace Wasm {
     37
     38class Instance;
     39
     40void JIT_OPERATION triggerOSREntryNow(Probe::Context&) WTF_INTERNAL;
     41void JIT_OPERATION triggerTierUpNow(Instance*, uint32_t functionIndex) WTF_INTERNAL;
    3642
    3743} } // namespace JSC::Wasm
     44
     45#endif // ENABLE(WEBASSEMBLY)

  • trunk/Source/JavaScriptCore/wasm/WasmThunks.cpp

    r248686 r248878  
    3737#include "WasmExceptionType.h"
    3838#include "WasmInstance.h"
    39 #include "WasmOMGPlan.h"
     39#include "WasmOperations.h"
    4040
    4141namespace JSC { namespace Wasm {
    … …  
    7878}
    7979
    80 MacroAssemblerCodeRef<JITThunkPtrTag> triggerOMGTierUpThunkGenerator(const AbstractLocker&)
     80MacroAssemblerCodeRef<JITThunkPtrTag> triggerOMGEntryTierUpThunkGenerator(const AbstractLocker&)
    8181{
    8282    // We expect that the user has already put the function index into GPRInfo::argumentGPR1
    … …  
    9191
    9292    jit.loadWasmContextInstance(GPRInfo::argumentGPR0);
    93     typedef void (*Run)(Instance*, uint32_t);
    94     Run run = OMGPlan::runForIndex;
    95     jit.move(MacroAssembler::TrustedImmPtr(tagCFunctionPtr<OperationPtrTag>(run)), GPRInfo::argumentGPR2);
     93    jit.move(MacroAssembler::TrustedImmPtr(tagCFunctionPtr<OperationPtrTag>(triggerTierUpNow)), GPRInfo::argumentGPR2);
    9694    jit.call(GPRInfo::argumentGPR2, OperationPtrTag);
    9795
    … …  
    10199    jit.ret();
    102100    LinkBuffer linkBuffer(jit, GLOBAL_THUNK_ID);
    103     return FINALIZE_CODE(linkBuffer, JITThunkPtrTag, "Trigger OMG tier up");
     101    return FINALIZE_CODE(linkBuffer, JITThunkPtrTag, "Trigger OMG entry tier up");
    104102}
    105103

  • trunk/Source/JavaScriptCore/wasm/WasmThunks.h

    r241923 r248878  
    3535MacroAssemblerCodeRef<JITThunkPtrTag> throwExceptionFromWasmThunkGenerator(const AbstractLocker&);
    3636MacroAssemblerCodeRef<JITThunkPtrTag> throwStackOverflowFromWasmThunkGenerator(const AbstractLocker&);
    37 MacroAssemblerCodeRef<JITThunkPtrTag> triggerOMGTierUpThunkGenerator(const AbstractLocker&);
     37MacroAssemblerCodeRef<JITThunkPtrTag> triggerOMGEntryTierUpThunkGenerator(const AbstractLocker&);
    3838
    3939typedef MacroAssemblerCodeRef<JITThunkPtrTag> (*ThunkGenerator)(const AbstractLocker&);

  • trunk/Source/JavaScriptCore/wasm/WasmTierUpCount.cpp

    r248876 r248878  
    2525
    2626#include "config.h"
    27 #include "WasmCompilationMode.h"
     27#include "WasmTierUpCount.h"
    2828
    29 #include <wtf/Assertions.h>
     29#if ENABLE(WEBASSEMBLY)
     30
     31#include "WasmOSREntryData.h"
    3032
    3133namespace JSC { namespace Wasm {
    3234
    33 const char* makeString(CompilationMode mode)
     35TierUpCount::TierUpCount()
    3436{
    35     switch (mode) {
    36     case CompilationMode::BBQMode:
    37         return "BBQ";
    38     case CompilationMode::OMGMode:
    39         return "OMG";
    40     }
    41     RELEASE_ASSERT_NOT_REACHED();
    42     return "";
     37    setNewThreshold(Options::thresholdForOMGOptimizeAfterWarmUp(), nullptr);
    4338}
    4439
    45 } } // namespace JSC::Wasm
     40TierUpCount::~TierUpCount() = default;
     41
     42OSREntryData& TierUpCount::addOSREntryData(uint32_t functionIndex, uint32_t loopIndex)
     43{
     44    m_osrEntryData.append(makeUnique<OSREntryData>(functionIndex, loopIndex));
     45    return *m_osrEntryData.last().get();
     46}
     47
     48} } // namespace JSC::Table
     49
     50#endif // ENABLE(WEBASSEMBLY)

  • trunk/Source/JavaScriptCore/wasm/WasmTierUpCount.h

    r215898 r248878  
    2828#if ENABLE(WEBASSEMBLY)
    2929
     30#include "CompilationResult.h"
     31#include "ExecutionCounter.h"
    3032#include "Options.h"
    3133#include <wtf/Atomics.h>
     34#include <wtf/SegmentedVector.h>
    3235#include <wtf/StdLibExtras.h>
    3336
    3437namespace JSC { namespace Wasm {
     38
     39class OSREntryData;
    3540
    3641// This class manages the tier up counts for Wasm binaries. The main interesting thing about
    … …  
    3944// don't care too much if the countdown is slightly off. The tier up trigger is atomic, however,
    4045// so tier up will be triggered exactly once.
    41 class TierUpCount {
     46class TierUpCount : public UpperTierExecutionCounter {
    4247    WTF_MAKE_NONCOPYABLE(TierUpCount);
    4348public:
    44     TierUpCount()
    45         : m_count(Options::webAssemblyOMGTierUpCount())
    46         , m_tierUpStarted(false)
     49    enum class TriggerReason : uint8_t {
     50        DontTrigger,
     51        CompilationDone,
     52        StartCompilation,
     53    };
     54
     55    enum class CompilationStatus : uint8_t {
     56        NotCompiled,
     57        StartCompilation,
     58        Compiled,
     59        Failed,
     60    };
     61
     62    TierUpCount();
     63    ~TierUpCount();
     64
     65    static int32_t loopIncrement() { return Options::omgTierUpCounterIncrementForLoop(); }
     66    static int32_t functionEntryIncrement() { return Options::omgTierUpCounterIncrementForEntry(); }
     67
     68    SegmentedVector<TriggerReason, 16>& osrEntryTriggers() { return m_osrEntryTriggers; }
     69    Vector<uint32_t>& outerLoops() { return m_outerLoops; }
     70    Lock& getLock() { return m_lock; }
     71
     72    OSREntryData& addOSREntryData(uint32_t functionIndex, uint32_t loopIndex);
     73
     74    void optimizeAfterWarmUp(uint32_t functionIndex)
    4775    {
     76        dataLogLnIf(Options::verboseOSR(), functionIndex, ": OMG-optimizing after warm-up.");
     77        setNewThreshold(Options::thresholdForOMGOptimizeAfterWarmUp(), nullptr);
    4878    }
    4979
    50     TierUpCount(TierUpCount&& other)
     80    bool checkIfOptimizationThresholdReached()
    5181    {
    52         ASSERT(other.m_count == Options::webAssemblyOMGTierUpCount());
    53         m_count = other.m_count;
     82        return checkIfThresholdCrossedAndSet(nullptr);
    5483    }
    5584
    56     static uint32_t loopDecrement() { return Options::webAssemblyLoopDecrement(); }
    57     static uint32_t functionEntryDecrement() { return Options::webAssemblyFunctionEntryDecrement(); }
    58 
    59     bool shouldStartTierUp()
     85    void dontOptimizeAnytimeSoon(uint32_t functionIndex)
    6086    {
    61         return !m_tierUpStarted.exchange(true);
     87        dataLogLnIf(Options::verboseOSR(), functionIndex, ": Not OMG-optimizing anytime soon.");
     88        deferIndefinitely();
    6289    }
    6390
    64     int32_t count() { return bitwise_cast<int32_t>(m_count); }
     91    void optimizeNextInvocation(uint32_t functionIndex)
     92    {
     93        dataLogLnIf(Options::verboseOSR(), functionIndex, ": OMG-optimizing next invocation.");
     94        setNewThreshold(0, nullptr);
     95    }
    6596
    66 private:
    67     uint32_t m_count;
    68     Atomic<bool> m_tierUpStarted;
     97    void optimizeSoon(uint32_t functionIndex)
     98    {
     99        dataLogLnIf(Options::verboseOSR(), functionIndex, ": OMG-optimizing soon.");
     100        // FIXME: Need adjustment once we get more information about wasm functions.
     101        setNewThreshold(Options::thresholdForOMGOptimizeSoon(), nullptr);
     102    }
     103
     104    void setOptimizationThresholdBasedOnCompilationResult(uint32_t functionIndex, CompilationResult result)
     105    {
     106        switch (result) {
     107        case CompilationSuccessful:
     108            optimizeNextInvocation(functionIndex);
     109            return;
     110        case CompilationFailed:
     111            dontOptimizeAnytimeSoon(functionIndex);
     112            return;
     113        case CompilationDeferred:
     114            optimizeAfterWarmUp(functionIndex);
     115            return;
     116        case CompilationInvalidated:
     117            // This is weird - it will only happen in cases when the DFG code block (i.e.
     118            // the code block that this JITCode belongs to) is also invalidated. So it
     119            // doesn't really matter what we do. But, we do the right thing anyway. Note
     120            // that us counting the reoptimization actually means that we might count it
     121            // twice. But that's generally OK. It's better to overcount reoptimizations
     122            // than it is to undercount them.
     123            optimizeAfterWarmUp(functionIndex);
     124            return;
     125        }
     126        RELEASE_ASSERT_NOT_REACHED();
     127    }
     128
     129    Atomic<bool> m_tierUpStarted { false };
     130    Lock m_lock;
     131    CompilationStatus m_compilationStatusForOMG { CompilationStatus::NotCompiled };
     132    CompilationStatus m_compilationStatusForOMGForOSREntry { CompilationStatus::NotCompiled };
     133    SegmentedVector<TriggerReason, 16> m_osrEntryTriggers;
     134    Vector<uint32_t> m_outerLoops;
     135    Vector<std::unique_ptr<OSREntryData>> m_osrEntryData;
    69136};
    70137   

  • trunk/Source/JavaScriptCore/wasm/WasmValidate.cpp

    r246645 r248878  
    8080    typedef Unexpected<ErrorType> UnexpectedResult;
    8181    typedef Expected<void, ErrorType> Result;
    82     typedef Type ExpressionType;
     82    using ExpressionType = Type;
     83    using ExpressionList = Vector<ExpressionType, 1>;
     84    using Stack = ExpressionList;
    8385    typedef ControlData ControlType;
    84     typedef Vector<ExpressionType, 1> ExpressionList;
    8586    typedef FunctionParser<Validate>::ControlEntry ControlEntry;
    8687
    8788    static constexpr ExpressionType emptyExpression() { return Void; }
     89    Stack createStack() { return Stack(); }
    8890
    8991    template <typename ...Args>
    … …  
    134136    ControlData WARN_UNUSED_RETURN addTopLevel(Type signature);
    135137    ControlData WARN_UNUSED_RETURN addBlock(Type signature);
    136     ControlData WARN_UNUSED_RETURN addLoop(Type signature);
     138    ControlData WARN_UNUSED_RETURN addLoop(Type signature, const Stack&, uint32_t);
    137139    Result WARN_UNUSED_RETURN addIf(ExpressionType condition, Type signature, ControlData& result);
    138     Result WARN_UNUSED_RETURN addElse(ControlData&, const ExpressionList&);
     140    Result WARN_UNUSED_RETURN addElse(ControlData&, const Stack&);
    139141    Result WARN_UNUSED_RETURN addElseToUnreachable(ControlData&);
    140142
    141     Result WARN_UNUSED_RETURN addReturn(ControlData& topLevel, const ExpressionList& returnValues);
    142     Result WARN_UNUSED_RETURN addBranch(ControlData&, ExpressionType condition, const ExpressionList& expressionStack);
    143     Result WARN_UNUSED_RETURN addSwitch(ExpressionType condition, const Vector<ControlData*>& targets, ControlData& defaultTarget, const ExpressionList& expressionStack);
    144     Result WARN_UNUSED_RETURN endBlock(ControlEntry&, ExpressionList& expressionStack);
     143    Result WARN_UNUSED_RETURN addReturn(ControlData& topLevel, const Stack& returnValues);
     144    Result WARN_UNUSED_RETURN addBranch(ControlData&, ExpressionType condition, const Stack& expressionStack);
     145    Result WARN_UNUSED_RETURN addSwitch(ExpressionType condition, const Vector<ControlData*>& targets, ControlData& defaultTarget, const Stack& expressionStack);
     146    Result WARN_UNUSED_RETURN endBlock(ControlEntry&, Stack& expressionStack);
    145147    Result WARN_UNUSED_RETURN addEndToUnreachable(ControlEntry&);
    146148    Result WARN_UNUSED_RETURN addGrowMemory(ExpressionType delta, ExpressionType& result);
    … …  
    162164    }
    163165
    164     void dump(const Vector<ControlEntry>&, const ExpressionList*);
     166    void dump(const Vector<ControlEntry>&, const Stack*);
    165167    void setParser(FunctionParser<Validate>*) { }
    166168
    167169private:
    168     Result WARN_UNUSED_RETURN unify(const ExpressionList&, const ControlData&);
    169 
    170     Result WARN_UNUSED_RETURN checkBranchTarget(ControlData& target, const ExpressionList& expressionStack);
     170    Result WARN_UNUSED_RETURN unify(const Stack&, const ControlData&);
     171
     172    Result WARN_UNUSED_RETURN checkBranchTarget(ControlData& target, const Stack& expressionStack);
    171173
    172174    Vector<Type> m_locals;
    … …  
    300302}
    301303
    302 Validate::ControlType Validate::addLoop(Type signature)
     304Validate::ControlType Validate::addLoop(Type signature, const Stack&, uint32_t)
    303305{
    304306    return ControlData(BlockType::Loop, signature);
    … …  
    320322}
    321323
    322 auto Validate::addElse(ControlType& current, const ExpressionList& values) -> Result
     324auto Validate::addElse(ControlType& current, const Stack& values) -> Result
    323325{
    324326    WASM_FAIL_IF_HELPER_FAILS(unify(values, current));
    … …  
    343345}
    344346
    345 auto Validate::checkBranchTarget(ControlType& target, const ExpressionList& expressionStack) -> Result
     347auto Validate::checkBranchTarget(ControlType& target, const Stack& expressionStack) -> Result
    346348{
    347349    if (target.branchTargetSignature() == Void)
    … …  
    354356}
    355357
    356 auto Validate::addBranch(ControlType& target, ExpressionType condition, const ExpressionList& stack) -> Result
     358auto Validate::addBranch(ControlType& target, ExpressionType condition, const Stack& stack) -> Result
    357359{
    358360    // Void means this is an unconditional branch.
    … …  
    361363}
    362364
    363 auto Validate::addSwitch(ExpressionType condition, const Vector<ControlData*>& targets, ControlData& defaultTarget, const ExpressionList& expressionStack) -> Result
     365auto Validate::addSwitch(ExpressionType condition, const Vector<ControlData*>& targets, ControlData& defaultTarget, const Stack& expressionStack) -> Result
    364366{
    365367    WASM_VALIDATOR_FAIL_IF(condition != I32, "br_table with non-i32 condition ", condition);
    … …  
    384386}
    385387
    386 auto Validate::endBlock(ControlEntry& entry, ExpressionList& stack) -> Result
     388auto Validate::endBlock(ControlEntry& entry, Stack& stack) -> Result
    387389{
    388390    WASM_FAIL_IF_HELPER_FAILS(unify(stack, entry.controlData));
    … …  
    427429}
    428430
    429 auto Validate::unify(const ExpressionList& values, const ControlType& block) -> Result
     431auto Validate::unify(const Stack& values, const ControlType& block) -> Result
    430432{
    431433    if (block.signature() == Void) {
    … …  
    439441}
    440442
    441 static void dumpExpressionStack(const CommaPrinter& comma, const Validate::ExpressionList& expressionStack)
     443static void dumpExpressionStack(const CommaPrinter& comma, const Validate::Stack& expressionStack)
    442444{
    443445    dataLog(comma, " ExpressionStack:");
    … …  
    446448}
    447449
    448 void Validate::dump(const Vector<ControlEntry>& controlStack, const ExpressionList* expressionStack)
     450void Validate::dump(const Vector<ControlEntry>& controlStack, const Stack* expressionStack)
    449451{
    450452    for (size_t i = controlStack.size(); i--;) {

  • trunk/Tools/ChangeLog

    r248859 r248878  
     12019-08-19  Yusuke Suzuki  <ysuzuki@apple.com>
     2
     3        [JSC] OSR entry to Wasm OMG
     4        https://bugs.webkit.org/show_bug.cgi?id=200362
     5
     6        Reviewed by Michael Saboff.
     7
     8        * Scripts/run-jsc-stress-tests:
     9
    1102019-08-19  Zhifei Fang  <zhifei_fang@apple.com>
    211

  • trunk/Tools/Scripts/run-jsc-stress-tests

    r248829 r248878  
    489489# We force all tests to use a smaller (1.5M) stack so that stack overflow tests can run faster.
    490490BASE_OPTIONS = ["--useFTLJIT=false", "--useFunctionDotArguments=true", "--validateExceptionChecks=true", "--useDollarVM=true", "--maxPerThreadStackUsage=1572864"]
    491 EAGER_OPTIONS = ["--thresholdForJITAfterWarmUp=10", "--thresholdForJITSoon=10", "--thresholdForOptimizeAfterWarmUp=20", "--thresholdForOptimizeAfterLongWarmUp=20", "--thresholdForOptimizeSoon=20", "--thresholdForFTLOptimizeAfterWarmUp=20", "--thresholdForFTLOptimizeSoon=20", "--maximumEvalCacheableSourceLength=150000", "--useEagerCodeBlockJettisonTiming=true"]
     491EAGER_OPTIONS = ["--thresholdForJITAfterWarmUp=10", "--thresholdForJITSoon=10", "--thresholdForOptimizeAfterWarmUp=20", "--thresholdForOptimizeAfterLongWarmUp=20", "--thresholdForOptimizeSoon=20", "--thresholdForFTLOptimizeAfterWarmUp=20", "--thresholdForFTLOptimizeSoon=20", "--thresholdForOMGOptimizeAfterWarmUp=20", "--thresholdForOMGOptimizeSoon=20", "--maximumEvalCacheableSourceLength=150000", "--useEagerCodeBlockJettisonTiming=true"]
    492492# NOTE: Tests rely on this using scribbleFreeCells.
    493493NO_CJIT_OPTIONS = ["--useConcurrentJIT=false", "--thresholdForJITAfterWarmUp=100", "--scribbleFreeCells=true"]
    … …  
    10861086    if $mode != "quick"
    10871087        run("wasm-no-cjit-yes-tls-context", "-m", "--useFastTLSForWasmContext=true", *(FTL_OPTIONS + NO_CJIT_OPTIONS))
     1088        run("wasm-eager", "-m", *(FTL_OPTIONS + EAGER_OPTIONS))
    10881089        run("wasm-eager-jettison", "-m", "--forceCodeBlockToJettisonDueToOldAge=true", *FTL_OPTIONS)
    10891090        run("wasm-no-call-ic", "-m", "--useCallICsForWebAssemblyToJSCalls=false", *FTL_OPTIONS)
    … …  
    11041105    if $mode != "quick"
    11051106        run("wasm-no-cjit-yes-tls-context", "-m", "--useFastTLSForWasmContext=true", *(FTL_OPTIONS + NO_CJIT_OPTIONS + optionalTestSpecificOptions))
     1107        run("wasm-eager", "-m", *(FTL_OPTIONS + EAGER_OPTIONS + optionalTestSpecificOptions))
    11061108        run("wasm-eager-jettison", "-m", "--forceCodeBlockToJettisonDueToOldAge=true", *(FTL_OPTIONS + optionalTestSpecificOptions))
    11071109        run("wasm-no-call-ic", "-m", "--useCallICsForWebAssemblyToJSCalls=false", *(FTL_OPTIONS + optionalTestSpecificOptions))
Note: See TracChangeset for help on using the changeset viewer.