Changeset 279256 in webkit

Timestamp:

Jun 24, 2021 5:06:56 PM (3 years ago)

Author:

mark.lam@apple.com

Message:

Use ldp and stp more for saving / restoring registers on ARM64.
​https://bugs.webkit.org/show_bug.cgi?id=227039
rdar://79354736

Reviewed by Saam Barati.

This patch introduces a spooler abstraction in AssemblyHelpers. The spooler
basically batches up load / store operations and emit them as pair instructions
if appropriate.

There are 4 spooler classes:

1. Spooler
   • template base class for LoadRegSpooler and StoreRegSpooler.
   • encapsulates the batching strategy for load / store pairs.

2. LoadRegSpooler - specializes Spooler to handle load pairs.
3. StoreRegSpooler - specializes Spooler to handle store pairs.

4. CopySpooler
   • handles matching loads with stores.
   • tries to emit loads as load pairs if possible.
   • tries to emot stores as store pairs if possible.
   • ensures that pre-requisite loads are emitted before stores are emitted.
   • other than loads, also support constants and registers as sources of values to be stored. This is useful in OSR exit ramps where we may materialize a stack value to store from constants or registers in addition to values we load from the old stack frame or from a scratch buffer.

In this patch, we also do the following:

1. Use spoolers in many places so that we can emit load / store pairs instead of single load / stores. This helps shrink JIT code side, and also potentially improves performance.

2. In DFG::OSRExit::compileExit(), we used to recover constants into a scratch buffer, and then later, load from that scratch buffer to store into the new stack frame(s).

This patch changes it so that we defer constant recovery until the final
loop where we store the recovered value directly into the new stack frame(s).
This saves us the work (and JIT code space) for storing into a scratch buffer
and then reloading from the scratch buffer.

There is one exception: tmp values used by active checkpoints. We need to call
operationMaterializeOSRExitSideState() to materialize the active checkpoint
side state before the final loop where we now recover constants. Hence, we
need these tmp values recovered before hand.

So, we check upfront if we have active checkpoint side state to materialize.
If so, we'll eagerly recover the constants for initializing those tmps.

We also use the CopySpooler in the final loop to emit load / store pairs for
filling in the new stack frame(s).

One more thing: it turns out that the vast majority of constants to be recovered
is simply the undefined value. So, as an optimization, the final loop keeps
the undefined value in a register, and has the spooler store directly from
that register when appropriate. This saves on JIT code to repeatedly materialize
the undefined JSValue constant.

3. In reifyInlinedCallFrames(), replace the use of GPRInfo::nonArgGPR0 with GPRInfo::regT4. nonArgGPRs are sometimes map to certain regTXs on certain ports. Replacing with regT4 makes it easier to ensure that we're not trashing the register when we use more temp registers.

reifyInlinedCallFrames() will be using emitSaveOrCopyLLIntBaselineCalleeSavesFor()
later where we need more temp registers.

4. Move the following functions to AssemblyHelpers.cpp. They don't need to be inline functions. Speedometer2 and JetStream2 shows that making these non inline does not hurt performance:

AssemblyHelpers::emitSave(const RegisterAtOffsetList&);
AssemblyHelpers::emitRestore(const RegisterAtOffsetList&);
AssemblyHelpers::emitSaveCalleeSavesFor(const RegisterAtOffsetList*);
AssemblyHelpers::emitSaveOrCopyCalleeSavesFor(...);
AssemblyHelpers::emitRestoreCalleeSavesFor(const RegisterAtOffsetList*);
AssemblyHelpers::copyLLIntBaselineCalleeSavesFromFrameOrRegisterToEntryFrameCalleeSavesBuffer(...);

Also renamed emitSaveOrCopyCalleeSavesFor() to emitSaveOrCopyLLIntBaselineCalleeSavesFor()
because it is only used with baseline codeBlocks.

Results:
Cummulative LinkBuffer profile sizes shrunk by ~2M in aggregate:

base new

===

BaselineJIT: 83827048 (79.943703 MB) => 83718736 (79.840408 MB)

DFG: 56594836 (53.973042 MB) => 56603508 (53.981312 MB)

InlineCache: 33923900 (32.352352 MB) => 33183156 (31.645924 MB)

FTL: 6770956 (6.457287 MB) => 6568964 (6.264652 MB)

DFGOSRExit: 5212096 (4.970642 MB) => 3728088 (3.555382 MB)

CSSJIT: 748428 (730.886719 KB) => 748428 (730.886719 KB)

FTLOSRExit: 692276 (676.050781 KB) => 656884 (641.488281 KB)

YarrJIT: 445280 (434.843750 KB) => 512988 (500.964844 KB)

FTLThunk: 22908 (22.371094 KB) => 22556 (22.027344 KB)

BoundFunctionThunk: 8400 (8.203125 KB) => 10088 (9.851562 KB)

ExtraCTIThunk: 6952 (6.789062 KB) => 6824 (6.664062 KB)

SpecializedThunk: 4508 (4.402344 KB) => 4508 (4.402344 KB)

Thunk: 3912 (3.820312 KB) => 3784 (3.695312 KB)

LLIntThunk: 2908 (2.839844 KB) => 2908 (2.839844 KB)

VirtualThunk: 1248 (1.218750 KB) => 1248 (1.218750 KB)

DFGThunk: 1084 (1.058594 KB) => 444

DFGOSREntry: 216 => 184

JumpIsland: 0

WasmThunk: 0

Wasm: 0

Uncategorized: 0

Total: 188266956 (179.545361 MB) => 185773296 (177.167221 MB)

Speedometer2 and JetStream2 results shows that performance is neutral for this
patch (as measured on an M1 Mac):

Speedometer2:

---

| subtest | ms | ms | b / a | pValue (significance using False Discovery Rate) |

---

| Elm-TodoMVC |129.037500 |127.212500 |0.985857 | 0.012706 |
| VueJS-TodoMVC |28.312500 |27.525000 |0.972185 | 0.240315 |
| EmberJS-TodoMVC |132.550000 |132.025000 |0.996039 | 0.538034 |
| Flight-TodoMVC |80.762500 |80.875000 |1.001393 | 0.914749 |
| BackboneJS-TodoMVC |51.637500 |51.175000 |0.991043 | 0.285427 |
| Preact-TodoMVC |21.025000 |22.075000 |1.049941 | 0.206140 |
| AngularJS-TodoMVC |142.900000 |142.887500 |0.999913 | 0.990681 |
| Inferno-TodoMVC |69.300000 |69.775000 |1.006854 | 0.505201 |
| Vanilla-ES2015-TodoMVC |71.500000 |71.225000 |0.996154 | 0.608650 |
| Angular2-TypeScript-TodoMVC |43.287500 |43.275000 |0.999711 | 0.987926 |
| VanillaJS-TodoMVC |57.212500 |57.812500 |1.010487 | 0.333357 |
| jQuery-TodoMVC |276.150000 |276.775000 |1.002263 | 0.614404 |
| EmberJS-Debug-TodoMVC |353.612500 |352.762500 |0.997596 | 0.518836 |
| React-TodoMVC |93.637500 |92.637500 |0.989321 | 0.036277 |
| React-Redux-TodoMVC |158.237500 |156.587500 |0.989573 | 0.042154 |
| Vanilla-ES2015-Babel-Webpack-TodoMVC |68.050000 |68.087500 |1.000551 | 0.897149 |

---

a mean = 236.26950
b mean = 236.57964
pValue = 0.7830785938
(Bigger means are better.)
1.001 times better
Results ARE NOT significant

JetStream2:

---

| subtest | pts | pts | b / a | pValue (significance using False Discovery Rate) |

---

| gaussian-blur |542.570057 |542.671885 |1.000188 | 0.982573 |
| HashSet-wasm |57.710498 |64.406371 |1.116025 | 0.401424 |
| gcc-loops-wasm |44.516009 |44.453535 |0.998597 | 0.973651 |
| json-parse-inspector |241.275085 |240.720491 |0.997701 | 0.704732 |
| prepack-wtb |62.640114 |63.754878 |1.017796 | 0.205840 |
| date-format-xparb-SP |416.976817 |448.921409 |1.076610 | 0.052977 |
| WSL |1.555257 |1.570233 |1.009629 | 0.427924 |
| OfflineAssembler |177.052352 |179.746511 |1.015217 | 0.112114 |
| cdjs |192.517586 |194.598906 |1.010811 | 0.025807 |
| UniPoker |514.023694 |526.111500 |1.023516 | 0.269892 |
| json-stringify-inspector |227.584725 |223.619390 |0.982576 | 0.102714 |
| crypto-sha1-SP |980.728788 |984.192104 |1.003531 | 0.838618 |
| Basic |685.148483 |711.590247 |1.038593 | 0.142952 |
| chai-wtb |106.256376 |106.590318 |1.003143 | 0.865894 |
| crypto-aes-SP |722.308829 |728.702310 |1.008851 | 0.486766 |
| Babylon |655.857561 |654.204901 |0.997480 | 0.931520 |
| string-unpack-code-SP |407.837271 |405.710752 |0.994786 | 0.729122 |
| stanford-crypto-aes |456.906021 |449.993856 |0.984872 | 0.272994 |
| raytrace |883.911335 |902.887238 |1.021468 | 0.189785 |
| multi-inspector-code-load |409.997347 |405.643639 |0.989381 | 0.644447 |
| hash-map |593.590160 |601.576332 |1.013454 | 0.249414 |
| stanford-crypto-pbkdf2 |722.178638 |728.283532 |1.008453 | 0.661195 |
| coffeescript-wtb |42.393544 |41.869545 |0.987640 | 0.197441 |
| Box2D |452.034685 |454.104868 |1.004580 | 0.535342 |
| richards-wasm |140.873688 |148.394050 |1.053384 | 0.303651 |
| lebab-wtb |61.671318 |62.119403 |1.007266 | 0.620998 |
| tsf-wasm |108.592794 |119.498398 |1.100427 | 0.504710 |
| base64-SP |629.744643 |603.425565 |0.958207 | 0.049997 |
| navier-stokes |740.588523 |739.951662 |0.999140 | 0.871445 |
| jshint-wtb |51.938359 |52.651104 |1.013723 | 0.217137 |
| regex-dna-SP |459.251148 |463.492489 |1.009235 | 0.371891 |
| async-fs |235.853820 |236.031189 |1.000752 | 0.938459 |
| first-inspector-code-load |275.298325 |274.172125 |0.995909 | 0.623403 |
| segmentation |44.002842 |43.445960 |0.987344 | 0.207134 |
| typescript |26.360161 |26.458820 |1.003743 | 0.609942 |
| octane-code-load |1126.749036 |1087.132024 |0.964840 | 0.524171 |
| float-mm.c |16.691935 |16.721354 |1.001762 | 0.194425 |
| quicksort-wasm |461.630091 |450.161127 |0.975156 | 0.371394 |
| Air |392.442375 |412.201810 |1.050350 | 0.046887 |
| splay |510.111886 |475.131657 |0.931426 | 0.024732 |
| ai-astar |607.966974 |626.573181 |1.030604 | 0.468711 |
| acorn-wtb |67.510766 |68.143956 |1.009379 | 0.481663 |
| gbemu |144.133842 |145.620304 |1.010313 | 0.802154 |
| richards |963.475078 |946.658879 |0.982546 | 0.231189 |
| 3d-cube-SP |549.426784 |550.479154 |1.001915 | 0.831307 |
| espree-wtb |68.707483 |73.762202 |1.073569 | 0.033603 |
| bomb-workers |96.882596 |96.116121 |0.992089 | 0.687952 |
| tagcloud-SP |309.888767 |303.538511 |0.979508 | 0.187768 |
| mandreel |133.667031 |135.009929 |1.010047 | 0.075232 |
| 3d-raytrace-SP |491.967649 |492.528992 |1.001141 | 0.957842 |
| delta-blue |1066.718312 |1080.230772 |1.012667 | 0.549382 |
| ML |139.617293 |140.088630 |1.003376 | 0.661651 |
| regexp |351.773956 |351.075935 |0.998016 | 0.769250 |
| crypto |1510.474663 |1519.218842 |1.005789 | 0.638420 |
| crypto-md5-SP |795.447899 |774.082493 |0.973140 | 0.079728 |
| earley-boyer |812.574545 |870.678372 |1.071506 | 0.044081 |
| octane-zlib |25.162470 |25.660261 |1.019783 | 0.554591 |
| date-format-tofte-SP |395.296135 |398.008992 |1.006863 | 0.650475 |
| n-body-SP |1165.386611 |1150.525110 |0.987248 | 0.227908 |
| pdfjs |189.060252 |191.015628 |1.010343 | 0.633777 |
| FlightPlanner |908.426192 |903.636642 |0.994728 | 0.838821 |
| uglify-js-wtb |34.029399 |34.164342 |1.003965 | 0.655652 |
| babylon-wtb |81.329869 |80.855680 |0.994170 | 0.854393 |
| stanford-crypto-sha256 |826.850533 |838.494164 |1.014082 | 0.579636 |

---

a mean = 237.91084
b mean = 239.92670
pValue = 0.0657710897
(Bigger means are better.)
1.008 times better
Results ARE NOT significant

• CMakeLists.txt:
• JavaScriptCore.xcodeproj/project.pbxproj:
• assembler/MacroAssembler.h:

(JSC::MacroAssembler::pushToSaveByteOffset):

• assembler/MacroAssemblerARM64.h:

(JSC::MacroAssemblerARM64::pushToSaveByteOffset):

• dfg/DFGOSRExit.cpp:

(JSC::DFG::OSRExit::compileExit):

• dfg/DFGOSRExitCompilerCommon.cpp:

(JSC::DFG::reifyInlinedCallFrames):

• dfg/DFGThunks.cpp:

(JSC::DFG::osrExitGenerationThunkGenerator):

• ftl/FTLSaveRestore.cpp:

(JSC::FTL::saveAllRegisters):
(JSC::FTL::restoreAllRegisters):

• ftl/FTLSaveRestore.h:
• ftl/FTLThunks.cpp:

(JSC::FTL::genericGenerationThunkGenerator):
(JSC::FTL::slowPathCallThunkGenerator):

• jit/AssemblyHelpers.cpp:

(JSC::AssemblyHelpers::restoreCalleeSavesFromEntryFrameCalleeSavesBuffer):
(JSC::AssemblyHelpers::copyCalleeSavesToEntryFrameCalleeSavesBufferImpl):
(JSC::AssemblyHelpers::emitSave):
(JSC::AssemblyHelpers::emitRestore):
(JSC::AssemblyHelpers::emitSaveCalleeSavesFor):
(JSC::AssemblyHelpers::emitRestoreCalleeSavesFor):
(JSC::AssemblyHelpers::copyLLIntBaselineCalleeSavesFromFrameOrRegisterToEntryFrameCalleeSavesBuffer):
(JSC::AssemblyHelpers::emitSaveOrCopyLLIntBaselineCalleeSavesFor):

• jit/AssemblyHelpers.h:

(JSC::AssemblyHelpers::copyLLIntBaselineCalleeSavesFromFrameOrRegisterToEntryFrameCalleeSavesBuffer):
(JSC::AssemblyHelpers::emitSave): Deleted.
(JSC::AssemblyHelpers::emitRestore): Deleted.
(JSC::AssemblyHelpers::emitSaveOrCopyCalleeSavesFor): Deleted.

• jit/AssemblyHelpersSpoolers.h: Added.

(JSC::AssemblyHelpers::Spooler::Spooler):
(JSC::AssemblyHelpers::Spooler::handleGPR):
(JSC::AssemblyHelpers::Spooler::finalizeGPR):
(JSC::AssemblyHelpers::Spooler::handleFPR):
(JSC::AssemblyHelpers::Spooler::finalizeFPR):
(JSC::AssemblyHelpers::Spooler::op):
(JSC::AssemblyHelpers::LoadRegSpooler::LoadRegSpooler):
(JSC::AssemblyHelpers::LoadRegSpooler::loadGPR):
(JSC::AssemblyHelpers::LoadRegSpooler::finalizeGPR):
(JSC::AssemblyHelpers::LoadRegSpooler::loadFPR):
(JSC::AssemblyHelpers::LoadRegSpooler::finalizeFPR):
(JSC::AssemblyHelpers::LoadRegSpooler::handlePair):
(JSC::AssemblyHelpers::LoadRegSpooler::handleSingle):
(JSC::AssemblyHelpers::StoreRegSpooler::StoreRegSpooler):
(JSC::AssemblyHelpers::StoreRegSpooler::storeGPR):
(JSC::AssemblyHelpers::StoreRegSpooler::finalizeGPR):
(JSC::AssemblyHelpers::StoreRegSpooler::storeFPR):
(JSC::AssemblyHelpers::StoreRegSpooler::finalizeFPR):
(JSC::AssemblyHelpers::StoreRegSpooler::handlePair):
(JSC::AssemblyHelpers::StoreRegSpooler::handleSingle):
(JSC::RegDispatch<GPRReg>::get):
(JSC::RegDispatch<GPRReg>::temp1):
(JSC::RegDispatch<GPRReg>::temp2):
(JSC::RegDispatch<GPRReg>::regToStore):
(JSC::RegDispatch<GPRReg>::invalid):
(JSC::RegDispatch<GPRReg>::regSize):
(JSC::RegDispatch<GPRReg>::isValidLoadPairImm):
(JSC::RegDispatch<GPRReg>::isValidStorePairImm):
(JSC::RegDispatch<FPRReg>::get):
(JSC::RegDispatch<FPRReg>::temp1):
(JSC::RegDispatch<FPRReg>::temp2):
(JSC::RegDispatch<FPRReg>::regToStore):
(JSC::RegDispatch<FPRReg>::invalid):
(JSC::RegDispatch<FPRReg>::regSize):
(JSC::RegDispatch<FPRReg>::isValidLoadPairImm):
(JSC::RegDispatch<FPRReg>::isValidStorePairImm):
(JSC::AssemblyHelpers::CopySpooler::Source::getReg):
(JSC::AssemblyHelpers::CopySpooler::CopySpooler):
(JSC::AssemblyHelpers::CopySpooler::temp1 const):
(JSC::AssemblyHelpers::CopySpooler::temp2 const):
(JSC::AssemblyHelpers::CopySpooler::regToStore):
(JSC::AssemblyHelpers::CopySpooler::invalid):
(JSC::AssemblyHelpers::CopySpooler::regSize):
(JSC::AssemblyHelpers::CopySpooler::isValidLoadPairImm):
(JSC::AssemblyHelpers::CopySpooler::isValidStorePairImm):
(JSC::AssemblyHelpers::CopySpooler::load):
(JSC::AssemblyHelpers::CopySpooler::move):
(JSC::AssemblyHelpers::CopySpooler::copy):
(JSC::AssemblyHelpers::CopySpooler::store):
(JSC::AssemblyHelpers::CopySpooler::flush):
(JSC::AssemblyHelpers::CopySpooler::loadGPR):
(JSC::AssemblyHelpers::CopySpooler::copyGPR):
(JSC::AssemblyHelpers::CopySpooler::moveConstant):
(JSC::AssemblyHelpers::CopySpooler::storeGPR):
(JSC::AssemblyHelpers::CopySpooler::finalizeGPR):
(JSC::AssemblyHelpers::CopySpooler::loadFPR):
(JSC::AssemblyHelpers::CopySpooler::copyFPR):
(JSC::AssemblyHelpers::CopySpooler::storeFPR):
(JSC::AssemblyHelpers::CopySpooler::finalizeFPR):
(JSC::AssemblyHelpers::CopySpooler::loadPair):
(JSC::AssemblyHelpers::CopySpooler::storePair):

• jit/ScratchRegisterAllocator.cpp:

(JSC::ScratchRegisterAllocator::preserveReusedRegistersByPushing):
(JSC::ScratchRegisterAllocator::restoreReusedRegistersByPopping):
(JSC::ScratchRegisterAllocator::preserveRegistersToStackForCall):
(JSC::ScratchRegisterAllocator::restoreRegistersFromStackForCall):

• jit/ScratchRegisterAllocator.h:
• wasm/WasmAirIRGenerator.cpp:

(JSC::Wasm::AirIRGenerator::addReturn):

• wasm/WasmB3IRGenerator.cpp:

(JSC::Wasm::B3IRGenerator::addReturn):

Location:

trunk/Source/JavaScriptCore

Files:

• CMakeLists.txt (modified) (1 diff)
• ChangeLog (modified) (1 diff)
• JavaScriptCore.xcodeproj/project.pbxproj (modified) (4 diffs)
• assembler/MacroAssembler.h (modified) (1 diff)
• assembler/MacroAssemblerARM64.h (modified) (1 diff)
• dfg/DFGOSRExit.cpp (modified) (10 diffs)
• dfg/DFGOSRExitCompilerCommon.cpp (modified) (3 diffs)
• dfg/DFGThunks.cpp (modified) (3 diffs)
• ftl/FTLSaveRestore.cpp (modified) (3 diffs)
• ftl/FTLSaveRestore.h (modified) (3 diffs)
• ftl/FTLThunks.cpp (modified) (6 diffs)
• jit/AssemblyHelpers.cpp (modified) (5 diffs)
• jit/AssemblyHelpers.h (modified) (4 diffs)
• jit/AssemblyHelpersSpoolers.h (added)
• jit/ScratchRegisterAllocator.cpp (modified) (8 diffs)
• jit/ScratchRegisterAllocator.h (modified) (4 diffs)
• wasm/WasmAirIRGenerator.cpp (modified) (2 diffs)
• wasm/WasmB3IRGenerator.cpp (modified) (2 diffs)

trunk/Source/JavaScriptCore/CMakeLists.txt

@@ -758 +758 @@
 
     jit/AssemblyHelpers.h
+    jit/AssemblyHelpersSpoolers.h
     jit/CCallHelpers.h
     jit/ExecutableAllocator.h

trunk/Source/JavaScriptCore/ChangeLog

@@ +1 @@
+2021-06-24  Mark Lam  <mark.lam@apple.com>
+
+        Use ldp and stp more for saving / restoring registers on ARM64.
+        https://bugs.webkit.org/show_bug.cgi?id=227039
+        rdar://79354736
+
+        Reviewed by Saam Barati.
+
+        This patch introduces a spooler abstraction in AssemblyHelpers.  The spooler
+        basically batches up load / store operations and emit them as pair instructions
+        if appropriate.
+
+        There are 4 spooler classes:
+        a. Spooler
+           - template base class for LoadRegSpooler and StoreRegSpooler.
+           - encapsulates the batching strategy for load / store pairs.
+
+        b. LoadRegSpooler - specializes Spooler to handle load pairs.
+        b. StoreRegSpooler - specializes Spooler to handle store pairs.
+
+        d. CopySpooler
+           - handles matching loads with stores.
+           - tries to emit loads as load pairs if possible.
+           - tries to emot stores as store pairs if possible.
+           - ensures that pre-requisite loads are emitted before stores are emitted.
+           - other than loads, also support constants and registers as sources of values
+             to be stored.  This is useful in OSR exit ramps where we may materialize a
+             stack value to store from constants or registers in addition to values we
+             load from the old stack frame or from a scratch buffer.
+
+        In this patch, we also do the following:
+
+        1. Use spoolers in many places so that we can emit load / store pairs instead of
+           single load / stores.  This helps shrink JIT code side, and also potentially
+           improves performance.
+
+        2. In DFG::OSRExit::compileExit(), we used to recover constants into a scratch
+           buffer, and then later, load from that scratch buffer to store into the
+           new stack frame(s).
+
+           This patch changes it so that we defer constant recovery until the final
+           loop where we store the recovered value directly into the new stack frame(s).
+           This saves us the work (and JIT code space) for storing into a scratch buffer
+           and then reloading from the scratch buffer.
+
+           There is one exception: tmp values used by active checkpoints.  We need to call
+           operationMaterializeOSRExitSideState() to materialize the active checkpoint
+           side state before the final loop where we now recover constants.  Hence, we
+           need these tmp values recovered before hand.
+
+           So, we check upfront if we have active checkpoint side state to materialize.
+           If so, we'll eagerly recover the constants for initializing those tmps.
+
+           We also use the CopySpooler in the final loop to emit load / store pairs for
+           filling in the new stack frame(s).
+
+           One more thing: it turns out that the vast majority of constants to be recovered
+           is simply the undefined value.  So, as an optimization, the final loop keeps
+           the undefined value in a register, and has the spooler store directly from
+           that register when appropriate.  This saves on JIT code to repeatedly materialize
+           the undefined JSValue constant.
+
+        3. In reifyInlinedCallFrames(), replace the use of GPRInfo::nonArgGPR0 with
+           GPRInfo::regT4.  nonArgGPRs are sometimes map to certain regTXs on certain ports.
+           Replacing with regT4 makes it easier to ensure that we're not trashing the
+           register when we use more temp registers.
+
+           reifyInlinedCallFrames() will be using emitSaveOrCopyLLIntBaselineCalleeSavesFor()
+           later where we need more temp registers.
+
+        4. Move the following functions to AssemblyHelpers.cpp.  They don't need to be
+           inline functions.  Speedometer2 and JetStream2 shows that making these non
+           inline does not hurt performance:
+
+                AssemblyHelpers::emitSave(const RegisterAtOffsetList&);
+                AssemblyHelpers::emitRestore(const RegisterAtOffsetList&);
+                AssemblyHelpers::emitSaveCalleeSavesFor(const RegisterAtOffsetList*);
+                AssemblyHelpers::emitSaveOrCopyCalleeSavesFor(...);
+                AssemblyHelpers::emitRestoreCalleeSavesFor(const RegisterAtOffsetList*);
+                AssemblyHelpers::copyLLIntBaselineCalleeSavesFromFrameOrRegisterToEntryFrameCalleeSavesBuffer(...);
+
+           Also renamed emitSaveOrCopyCalleeSavesFor() to emitSaveOrCopyLLIntBaselineCalleeSavesFor()
+           because it is only used with baseline codeBlocks.
+
+        Results:
+        Cummulative LinkBuffer profile sizes shrunk by ~2M in aggregate:
+
+                            base                           new
+                            ====                           ===
+               BaselineJIT: 83827048 (79.943703 MB)     => 83718736 (79.840408 MB)
+                       DFG: 56594836 (53.973042 MB)     => 56603508 (53.981312 MB)
+               InlineCache: 33923900 (32.352352 MB)     => 33183156 (31.645924 MB)
+                       FTL: 6770956 (6.457287 MB)       => 6568964 (6.264652 MB)
+                DFGOSRExit: 5212096 (4.970642 MB)       => 3728088 (3.555382 MB)
+                    CSSJIT: 748428 (730.886719 KB)      => 748428 (730.886719 KB)
+                FTLOSRExit: 692276 (676.050781 KB)      => 656884 (641.488281 KB)
+                   YarrJIT: 445280 (434.843750 KB)      => 512988 (500.964844 KB)
+                  FTLThunk: 22908 (22.371094 KB)        => 22556 (22.027344 KB)
+        BoundFunctionThunk: 8400 (8.203125 KB)          => 10088 (9.851562 KB)
+             ExtraCTIThunk: 6952 (6.789062 KB)          => 6824 (6.664062 KB)
+          SpecializedThunk: 4508 (4.402344 KB)          => 4508 (4.402344 KB)
+                     Thunk: 3912 (3.820312 KB)          => 3784 (3.695312 KB)
+                LLIntThunk: 2908 (2.839844 KB)          => 2908 (2.839844 KB)
+              VirtualThunk: 1248 (1.218750 KB)          => 1248 (1.218750 KB)
+                  DFGThunk: 1084 (1.058594 KB)          => 444
+               DFGOSREntry: 216                         => 184
+                JumpIsland: 0
+                 WasmThunk: 0
+                      Wasm: 0
+             Uncategorized: 0
+                     Total: 188266956 (179.545361 MB)   => 185773296 (177.167221 MB)
+
+        Speedometer2 and JetStream2 results shows that performance is neutral for this
+        patch (as measured on an M1 Mac):
+
+        Speedometer2:
+        ----------------------------------------------------------------------------------------------------------------------------------
+        |               subtest                |     ms      |     ms      |  b / a   | pValue (significance using False Discovery Rate) |
+        ----------------------------------------------------------------------------------------------------------------------------------
+        | Elm-TodoMVC                          |129.037500   |127.212500   |0.985857  | 0.012706                                         |
+        | VueJS-TodoMVC                        |28.312500    |27.525000    |0.972185  | 0.240315                                         |
+        | EmberJS-TodoMVC                      |132.550000   |132.025000   |0.996039  | 0.538034                                         |
+        | Flight-TodoMVC                       |80.762500    |80.875000    |1.001393  | 0.914749                                         |
+        | BackboneJS-TodoMVC                   |51.637500    |51.175000    |0.991043  | 0.285427                                         |
+        | Preact-TodoMVC                       |21.025000    |22.075000    |1.049941  | 0.206140                                         |
+        | AngularJS-TodoMVC                    |142.900000   |142.887500   |0.999913  | 0.990681                                         |
+        | Inferno-TodoMVC                      |69.300000    |69.775000    |1.006854  | 0.505201                                         |
+        | Vanilla-ES2015-TodoMVC               |71.500000    |71.225000    |0.996154  | 0.608650                                         |
+        | Angular2-TypeScript-TodoMVC          |43.287500    |43.275000    |0.999711  | 0.987926                                         |
+        | VanillaJS-TodoMVC                    |57.212500    |57.812500    |1.010487  | 0.333357                                         |
+        | jQuery-TodoMVC                       |276.150000   |276.775000   |1.002263  | 0.614404                                         |
+        | EmberJS-Debug-TodoMVC                |353.612500   |352.762500   |0.997596  | 0.518836                                         |
+        | React-TodoMVC                        |93.637500    |92.637500    |0.989321  | 0.036277                                         |
+        | React-Redux-TodoMVC                  |158.237500   |156.587500   |0.989573  | 0.042154                                         |
+        | Vanilla-ES2015-Babel-Webpack-TodoMVC |68.050000    |68.087500    |1.000551  | 0.897149                                         |
+        ----------------------------------------------------------------------------------------------------------------------------------
+
+        a mean = 236.26950
+        b mean = 236.57964
+        pValue = 0.7830785938
+        (Bigger means are better.)
+        1.001 times better
+        Results ARE NOT significant
+
+        JetStream2:
+        -------------------------------------------------------------------------------------------------------------------------
+        |          subtest          |     pts      |     pts      |  b / a   | pValue (significance using False Discovery Rate) |
+        -------------------------------------------------------------------------------------------------------------------------
+        | gaussian-blur             |542.570057    |542.671885    |1.000188  | 0.982573                                         |
+        | HashSet-wasm              |57.710498     |64.406371     |1.116025  | 0.401424                                         |
+        | gcc-loops-wasm            |44.516009     |44.453535     |0.998597  | 0.973651                                         |
+        | json-parse-inspector      |241.275085    |240.720491    |0.997701  | 0.704732                                         |
+        | prepack-wtb               |62.640114     |63.754878     |1.017796  | 0.205840                                         |
+        | date-format-xparb-SP      |416.976817    |448.921409    |1.076610  | 0.052977                                         |
+        | WSL                       |1.555257      |1.570233      |1.009629  | 0.427924                                         |
+        | OfflineAssembler          |177.052352    |179.746511    |1.015217  | 0.112114                                         |
+        | cdjs                      |192.517586    |194.598906    |1.010811  | 0.025807                                         |
+        | UniPoker                  |514.023694    |526.111500    |1.023516  | 0.269892                                         |
+        | json-stringify-inspector  |227.584725    |223.619390    |0.982576  | 0.102714                                         |
+        | crypto-sha1-SP            |980.728788    |984.192104    |1.003531  | 0.838618                                         |
+        | Basic                     |685.148483    |711.590247    |1.038593  | 0.142952                                         |
+        | chai-wtb                  |106.256376    |106.590318    |1.003143  | 0.865894                                         |
+        | crypto-aes-SP             |722.308829    |728.702310    |1.008851  | 0.486766                                         |
+        | Babylon                   |655.857561    |654.204901    |0.997480  | 0.931520                                         |
+        | string-unpack-code-SP     |407.837271    |405.710752    |0.994786  | 0.729122                                         |
+        | stanford-crypto-aes       |456.906021    |449.993856    |0.984872  | 0.272994                                         |
+        | raytrace                  |883.911335    |902.887238    |1.021468  | 0.189785                                         |
+        | multi-inspector-code-load |409.997347    |405.643639    |0.989381  | 0.644447                                         |
+        | hash-map                  |593.590160    |601.576332    |1.013454  | 0.249414                                         |
+        | stanford-crypto-pbkdf2    |722.178638    |728.283532    |1.008453  | 0.661195                                         |
+        | coffeescript-wtb          |42.393544     |41.869545     |0.987640  | 0.197441                                         |
+        | Box2D                     |452.034685    |454.104868    |1.004580  | 0.535342                                         |
+        | richards-wasm             |140.873688    |148.394050    |1.053384  | 0.303651                                         |
+        | lebab-wtb                 |61.671318     |62.119403     |1.007266  | 0.620998                                         |
+        | tsf-wasm                  |108.592794    |119.498398    |1.100427  | 0.504710                                         |
+        | base64-SP                 |629.744643    |603.425565    |0.958207  | 0.049997                                         |
+        | navier-stokes             |740.588523    |739.951662    |0.999140  | 0.871445                                         |
+        | jshint-wtb                |51.938359     |52.651104     |1.013723  | 0.217137                                         |
+        | regex-dna-SP              |459.251148    |463.492489    |1.009235  | 0.371891                                         |
+        | async-fs                  |235.853820    |236.031189    |1.000752  | 0.938459                                         |
+        | first-inspector-code-load |275.298325    |274.172125    |0.995909  | 0.623403                                         |
+        | segmentation              |44.002842     |43.445960     |0.987344  | 0.207134                                         |
+        | typescript                |26.360161     |26.458820     |1.003743  | 0.609942                                         |
+        | octane-code-load          |1126.749036   |1087.132024   |0.964840  | 0.524171                                         |
+        | float-mm.c                |16.691935     |16.721354     |1.001762  | 0.194425                                         |
+        | quicksort-wasm            |461.630091    |450.161127    |0.975156  | 0.371394                                         |
+        | Air                       |392.442375    |412.201810    |1.050350  | 0.046887                                         |
+        | splay                     |510.111886    |475.131657    |0.931426  | 0.024732                                         |
+        | ai-astar                  |607.966974    |626.573181    |1.030604  | 0.468711                                         |
+        | acorn-wtb                 |67.510766     |68.143956     |1.009379  | 0.481663                                         |
+        | gbemu                     |144.133842    |145.620304    |1.010313  | 0.802154                                         |
+        | richards                  |963.475078    |946.658879    |0.982546  | 0.231189                                         |
+        | 3d-cube-SP                |549.426784    |550.479154    |1.001915  | 0.831307                                         |
+        | espree-wtb                |68.707483     |73.762202     |1.073569  | 0.033603                                         |
+        | bomb-workers              |96.882596     |96.116121     |0.992089  | 0.687952                                         |
+        | tagcloud-SP               |309.888767    |303.538511    |0.979508  | 0.187768                                         |
+        | mandreel                  |133.667031    |135.009929    |1.010047  | 0.075232                                         |
+        | 3d-raytrace-SP            |491.967649    |492.528992    |1.001141  | 0.957842                                         |
+        | delta-blue                |1066.718312   |1080.230772   |1.012667  | 0.549382                                         |
+        | ML                        |139.617293    |140.088630    |1.003376  | 0.661651                                         |
+        | regexp                    |351.773956    |351.075935    |0.998016  | 0.769250                                         |
+        | crypto                    |1510.474663   |1519.218842   |1.005789  | 0.638420                                         |
+        | crypto-md5-SP             |795.447899    |774.082493    |0.973140  | 0.079728                                         |
+        | earley-boyer              |812.574545    |870.678372    |1.071506  | 0.044081                                         |
+        | octane-zlib               |25.162470     |25.660261     |1.019783  | 0.554591                                         |
+        | date-format-tofte-SP      |395.296135    |398.008992    |1.006863  | 0.650475                                         |
+        | n-body-SP                 |1165.386611   |1150.525110   |0.987248  | 0.227908                                         |
+        | pdfjs                     |189.060252    |191.015628    |1.010343  | 0.633777                                         |
+        | FlightPlanner             |908.426192    |903.636642    |0.994728  | 0.838821                                         |
+        | uglify-js-wtb             |34.029399     |34.164342     |1.003965  | 0.655652                                         |
+        | babylon-wtb               |81.329869     |80.855680     |0.994170  | 0.854393                                         |
+        | stanford-crypto-sha256    |826.850533    |838.494164    |1.014082  | 0.579636                                         |
+        -------------------------------------------------------------------------------------------------------------------------
+
+        a mean = 237.91084
+        b mean = 239.92670
+        pValue = 0.0657710897
+        (Bigger means are better.)
+        1.008 times better
+        Results ARE NOT significant
+
+        * CMakeLists.txt:
+        * JavaScriptCore.xcodeproj/project.pbxproj:
+        * assembler/MacroAssembler.h:
+        (JSC::MacroAssembler::pushToSaveByteOffset):
+        * assembler/MacroAssemblerARM64.h:
+        (JSC::MacroAssemblerARM64::pushToSaveByteOffset):
+        * dfg/DFGOSRExit.cpp:
+        (JSC::DFG::OSRExit::compileExit):
+        * dfg/DFGOSRExitCompilerCommon.cpp:
+        (JSC::DFG::reifyInlinedCallFrames):
+        * dfg/DFGThunks.cpp:
+        (JSC::DFG::osrExitGenerationThunkGenerator):
+        * ftl/FTLSaveRestore.cpp:
+        (JSC::FTL::saveAllRegisters):
+        (JSC::FTL::restoreAllRegisters):
+        * ftl/FTLSaveRestore.h:
+        * ftl/FTLThunks.cpp:
+        (JSC::FTL::genericGenerationThunkGenerator):
+        (JSC::FTL::slowPathCallThunkGenerator):
+        * jit/AssemblyHelpers.cpp:
+        (JSC::AssemblyHelpers::restoreCalleeSavesFromEntryFrameCalleeSavesBuffer):
+        (JSC::AssemblyHelpers::copyCalleeSavesToEntryFrameCalleeSavesBufferImpl):
+        (JSC::AssemblyHelpers::emitSave):
+        (JSC::AssemblyHelpers::emitRestore):
+        (JSC::AssemblyHelpers::emitSaveCalleeSavesFor):
+        (JSC::AssemblyHelpers::emitRestoreCalleeSavesFor):
+        (JSC::AssemblyHelpers::copyLLIntBaselineCalleeSavesFromFrameOrRegisterToEntryFrameCalleeSavesBuffer):
+        (JSC::AssemblyHelpers::emitSaveOrCopyLLIntBaselineCalleeSavesFor):
+        * jit/AssemblyHelpers.h:
+        (JSC::AssemblyHelpers::copyLLIntBaselineCalleeSavesFromFrameOrRegisterToEntryFrameCalleeSavesBuffer):
+        (JSC::AssemblyHelpers::emitSave): Deleted.
+        (JSC::AssemblyHelpers::emitRestore): Deleted.
+        (JSC::AssemblyHelpers::emitSaveOrCopyCalleeSavesFor): Deleted.
+        * jit/AssemblyHelpersSpoolers.h: Added.
+        (JSC::AssemblyHelpers::Spooler::Spooler):
+        (JSC::AssemblyHelpers::Spooler::handleGPR):
+        (JSC::AssemblyHelpers::Spooler::finalizeGPR):
+        (JSC::AssemblyHelpers::Spooler::handleFPR):
+        (JSC::AssemblyHelpers::Spooler::finalizeFPR):
+        (JSC::AssemblyHelpers::Spooler::op):
+        (JSC::AssemblyHelpers::LoadRegSpooler::LoadRegSpooler):
+        (JSC::AssemblyHelpers::LoadRegSpooler::loadGPR):
+        (JSC::AssemblyHelpers::LoadRegSpooler::finalizeGPR):
+        (JSC::AssemblyHelpers::LoadRegSpooler::loadFPR):
+        (JSC::AssemblyHelpers::LoadRegSpooler::finalizeFPR):
+        (JSC::AssemblyHelpers::LoadRegSpooler::handlePair):
+        (JSC::AssemblyHelpers::LoadRegSpooler::handleSingle):
+        (JSC::AssemblyHelpers::StoreRegSpooler::StoreRegSpooler):
+        (JSC::AssemblyHelpers::StoreRegSpooler::storeGPR):
+        (JSC::AssemblyHelpers::StoreRegSpooler::finalizeGPR):
+        (JSC::AssemblyHelpers::StoreRegSpooler::storeFPR):
+        (JSC::AssemblyHelpers::StoreRegSpooler::finalizeFPR):
+        (JSC::AssemblyHelpers::StoreRegSpooler::handlePair):
+        (JSC::AssemblyHelpers::StoreRegSpooler::handleSingle):
+        (JSC::RegDispatch<GPRReg>::get):
+        (JSC::RegDispatch<GPRReg>::temp1):
+        (JSC::RegDispatch<GPRReg>::temp2):
+        (JSC::RegDispatch<GPRReg>::regToStore):
+        (JSC::RegDispatch<GPRReg>::invalid):
+        (JSC::RegDispatch<GPRReg>::regSize):
+        (JSC::RegDispatch<GPRReg>::isValidLoadPairImm):
+        (JSC::RegDispatch<GPRReg>::isValidStorePairImm):
+        (JSC::RegDispatch<FPRReg>::get):
+        (JSC::RegDispatch<FPRReg>::temp1):
+        (JSC::RegDispatch<FPRReg>::temp2):
+        (JSC::RegDispatch<FPRReg>::regToStore):
+        (JSC::RegDispatch<FPRReg>::invalid):
+        (JSC::RegDispatch<FPRReg>::regSize):
+        (JSC::RegDispatch<FPRReg>::isValidLoadPairImm):
+        (JSC::RegDispatch<FPRReg>::isValidStorePairImm):
+        (JSC::AssemblyHelpers::CopySpooler::Source::getReg):
+        (JSC::AssemblyHelpers::CopySpooler::CopySpooler):
+        (JSC::AssemblyHelpers::CopySpooler::temp1 const):
+        (JSC::AssemblyHelpers::CopySpooler::temp2 const):
+        (JSC::AssemblyHelpers::CopySpooler::regToStore):
+        (JSC::AssemblyHelpers::CopySpooler::invalid):
+        (JSC::AssemblyHelpers::CopySpooler::regSize):
+        (JSC::AssemblyHelpers::CopySpooler::isValidLoadPairImm):
+        (JSC::AssemblyHelpers::CopySpooler::isValidStorePairImm):
+        (JSC::AssemblyHelpers::CopySpooler::load):
+        (JSC::AssemblyHelpers::CopySpooler::move):
+        (JSC::AssemblyHelpers::CopySpooler::copy):
+        (JSC::AssemblyHelpers::CopySpooler::store):
+        (JSC::AssemblyHelpers::CopySpooler::flush):
+        (JSC::AssemblyHelpers::CopySpooler::loadGPR):
+        (JSC::AssemblyHelpers::CopySpooler::copyGPR):
+        (JSC::AssemblyHelpers::CopySpooler::moveConstant):
+        (JSC::AssemblyHelpers::CopySpooler::storeGPR):
+        (JSC::AssemblyHelpers::CopySpooler::finalizeGPR):
+        (JSC::AssemblyHelpers::CopySpooler::loadFPR):
+        (JSC::AssemblyHelpers::CopySpooler::copyFPR):
+        (JSC::AssemblyHelpers::CopySpooler::storeFPR):
+        (JSC::AssemblyHelpers::CopySpooler::finalizeFPR):
+        (JSC::AssemblyHelpers::CopySpooler::loadPair):
+        (JSC::AssemblyHelpers::CopySpooler::storePair):
+        * jit/ScratchRegisterAllocator.cpp:
+        (JSC::ScratchRegisterAllocator::preserveReusedRegistersByPushing):
+        (JSC::ScratchRegisterAllocator::restoreReusedRegistersByPopping):
+        (JSC::ScratchRegisterAllocator::preserveRegistersToStackForCall):
+        (JSC::ScratchRegisterAllocator::restoreRegistersFromStackForCall):
+        * jit/ScratchRegisterAllocator.h:
+        * wasm/WasmAirIRGenerator.cpp:
+        (JSC::Wasm::AirIRGenerator::addReturn):
+        * wasm/WasmB3IRGenerator.cpp:
+        (JSC::Wasm::B3IRGenerator::addReturn):
+
 2021-06-24  Yusuke Suzuki  <ysuzuki@apple.com>
 

trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj

@@ -1150 +1150 @@
                 6A38CFAA1E32B5AB0060206F /* AsyncStackTrace.h in Headers */ = {isa = PBXBuildFile; fileRef = 6A38CFA81E32B58B0060206F /* AsyncStackTrace.h */; };
                 6AD2CB4D19B9140100065719 /* DebuggerEvalEnabler.h in Headers */ = {isa = PBXBuildFile; fileRef = 6AD2CB4C19B9140100065719 /* DebuggerEvalEnabler.h */; settings = {ATTRIBUTES = (Private, ); }; };
+                6B767E7B26791F270017F8D1 /* AssemblyHelpersSpoolers.h in Headers */ = {isa = PBXBuildFile; fileRef = 6B767E7A26791F270017F8D1 /* AssemblyHelpersSpoolers.h */; settings = {ATTRIBUTES = (Private, ); }; };
                 6BCCEC0425D1FA27000F391D /* VerifierSlotVisitorInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 6BCCEC0325D1FA27000F391D /* VerifierSlotVisitorInlines.h */; };
                 70113D4C1A8DB093003848C4 /* IteratorOperations.h in Headers */ = {isa = PBXBuildFile; fileRef = 70113D4A1A8DB093003848C4 /* IteratorOperations.h */; settings = {ATTRIBUTES = (Private, ); }; };
@@ -4000 +4001 @@
                 6AD2CB4C19B9140100065719 /* DebuggerEvalEnabler.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = DebuggerEvalEnabler.h; sourceTree = "<group>"; };
                 6B731CC02647A8370014646F /* SlowPathCall.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = SlowPathCall.cpp; sourceTree = "<group>"; };
+                6B767E7A26791F270017F8D1 /* AssemblyHelpersSpoolers.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AssemblyHelpersSpoolers.h; sourceTree = "<group>"; };
                 6BA93C9590484C5BAD9316EA /* JSScriptFetcher.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JSScriptFetcher.h; sourceTree = "<group>"; };
                 6BCCEC0325D1FA27000F391D /* VerifierSlotVisitorInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = VerifierSlotVisitorInlines.h; sourceTree = "<group>"; };
@@ -6201 +6203 @@
                                 0F24E53B17EA9F5900ABB217 /* AssemblyHelpers.cpp */,
                                 0F24E53C17EA9F5900ABB217 /* AssemblyHelpers.h */,
+                                6B767E7A26791F270017F8D1 /* AssemblyHelpersSpoolers.h */,
                                 723998F6265DBCDB0057867F /* BaselineJITPlan.cpp */,
                                 723998F5265DBCDB0057867F /* BaselineJITPlan.h */,
@@ -9155 +9158 @@
                                 534E034E1E4D4B1600213F64 /* AccessCase.h in Headers */,
                                 E3BFD0BC1DAF808E0065DEA2 /* AccessCaseSnippetParams.h in Headers */,
+                                6B767E7B26791F270017F8D1 /* AssemblyHelpersSpoolers.h in Headers */,
                                 5370B4F61BF26205005C40FC /* AdaptiveInferredPropertyValueWatchpointBase.h in Headers */,
                                 9168BD872447BA4E0080FFB4 /* AggregateError.h in Headers */,

trunk/Source/JavaScriptCore/assembler/MacroAssembler.h

@@ -338 +338 @@
     }
     
-    static ptrdiff_t pushToSaveByteOffset() { return sizeof(void*); }
+    static constexpr ptrdiff_t pushToSaveByteOffset() { return sizeof(void*); }
 #endif // !CPU(ARM64)
 

trunk/Source/JavaScriptCore/assembler/MacroAssemblerARM64.h

@@ -2535 +2535 @@
     }
 
-    static ptrdiff_t pushToSaveByteOffset() { return 16; }
+    static constexpr ptrdiff_t pushToSaveByteOffset() { return 16; }
 
     // Register move operations:

trunk/Source/JavaScriptCore/dfg/DFGOSRExit.cpp

@@ -29 +29 @@
 #if ENABLE(DFG_JIT)
 
-#include "AssemblyHelpers.h"
+#include "AssemblyHelpersSpoolers.h"
 #include "BytecodeStructs.h"
 #include "CheckpointOSRExitSideState.h"
@@ -563 +563 @@
     // It makes things simpler later.
 
+    bool inlineStackContainsActiveCheckpoint = exit.m_codeOrigin.inlineStackContainsActiveCheckpoint();
+    size_t firstTmpToRestoreEarly = operands.size() - operands.numberOfTmps();
+    if (!inlineStackContainsActiveCheckpoint)
+        firstTmpToRestoreEarly = operands.size(); // Don't eagerly restore.
+
     // The tag registers are needed to materialize recoveries below.
     jit.emitMaterializeTagCheckRegisters();
@@ -569 +574 @@
         const ValueRecovery& recovery = operands[index];
 
-        switch (recovery.technique()) {
+        auto currentTechnique = recovery.technique();
+        switch (currentTechnique) {
         case DisplacedInJSStack:
 #if USE(JSVALUE64)
@@ -595 +601 @@
         case Constant: {
 #if USE(JSVALUE64)
-            jit.move(AssemblyHelpers::TrustedImm64(JSValue::encode(recovery.constant())), GPRInfo::regT0);
-            jit.store64(GPRInfo::regT0, scratch + index);
-#else
+            if (index >= firstTmpToRestoreEarly) {
+                ASSERT(operands.operandForIndex(index).isTmp());
+                jit.move(AssemblyHelpers::TrustedImm64(JSValue::encode(recovery.constant())), GPRInfo::regT0);
+                jit.store64(GPRInfo::regT0, scratch + index);
+            }
+#else // not USE(JSVALUE64)
             jit.store32(
                 AssemblyHelpers::TrustedImm32(recovery.constant().tag()),
@@ -767 +776 @@
         jit.copyCalleeSavesToEntryFrameCalleeSavesBuffer(vm.topEntryFrame);
 
-    if (exit.m_codeOrigin.inlineStackContainsActiveCheckpoint()) {
+    if (inlineStackContainsActiveCheckpoint) {
         EncodedJSValue* tmpScratch = scratch + operands.tmpIndex(0);
         jit.setupArguments<decltype(operationMaterializeOSRExitSideState)>(&vm, &exit, tmpScratch);
@@ -777 +786 @@
     // Do all data format conversions and store the results into the stack.
 
+#if USE(JSVALUE64)
+    constexpr GPRReg srcBufferGPR = GPRInfo::regT2;
+    constexpr GPRReg destBufferGPR = GPRInfo::regT3;
+    constexpr GPRReg undefinedGPR = GPRInfo::regT4;
+    bool undefinedGPRIsInitialized = false;
+
+    jit.move(CCallHelpers::TrustedImmPtr(scratch), srcBufferGPR);
+    jit.move(CCallHelpers::framePointerRegister, destBufferGPR);
+    CCallHelpers::CopySpooler spooler(CCallHelpers::CopySpooler::BufferRegs::AllowModification, jit, srcBufferGPR, destBufferGPR, GPRInfo::regT0, GPRInfo::regT1);
+#endif
     for (size_t index = 0; index < operands.size(); ++index) {
         const ValueRecovery& recovery = operands[index];
@@ -787 +806 @@
 
         switch (recovery.technique()) {
+        case Constant: {
+#if USE(JSVALUE64)
+            EncodedJSValue currentConstant = JSValue::encode(recovery.constant());
+            if (currentConstant == encodedJSUndefined()) {
+                if (!undefinedGPRIsInitialized) {
+                    jit.move(CCallHelpers::TrustedImm64(encodedJSUndefined()), undefinedGPR);
+                    undefinedGPRIsInitialized = true;
+                }
+                spooler.copyGPR(undefinedGPR);
+            } else
+                spooler.moveConstant(currentConstant);
+            spooler.storeGPR(operand.virtualRegister().offset() * sizeof(CPURegister));
+            break;
+#else
+            FALLTHROUGH;
+#endif
+        }
         case DisplacedInJSStack:
         case BooleanDisplacedInJSStack:
@@ -796 +832 @@
         case UnboxedCellInGPR:
         case UnboxedDoubleInFPR:
-        case Constant:
         case InFPR:
 #if USE(JSVALUE64)
@@ -804 +839 @@
         case UnboxedStrictInt52InGPR:
         case StrictInt52DisplacedInJSStack:
-            jit.load64(scratch + index, GPRInfo::regT0);
-            jit.store64(GPRInfo::regT0, AssemblyHelpers::addressFor(operand));
+            spooler.loadGPR(index * sizeof(CPURegister));
+            spooler.storeGPR(operand.virtualRegister().offset() * sizeof(CPURegister));
             break;
 #else // not USE(JSVALUE64)
@@ -834 +869 @@
         }
     }
+#if USE(JSVALUE64)
+    spooler.finalizeGPR();
+#endif
 
     // Now that things on the stack are recovered, do the arguments recovery. We assume that arguments

trunk/Source/JavaScriptCore/dfg/DFGOSRExitCompilerCommon.cpp

@@ -285 +285 @@
             jit.untagPtr(GPRInfo::regT2, GPRInfo::regT3);
             jit.addPtr(AssemblyHelpers::TrustedImm32(inlineCallFrame->returnPCOffset() + sizeof(void*)), GPRInfo::callFrameRegister, GPRInfo::regT2);
-            jit.validateUntaggedPtr(GPRInfo::regT3, GPRInfo::nonArgGPR0);
+            jit.validateUntaggedPtr(GPRInfo::regT3, GPRInfo::regT4);
             jit.tagPtr(GPRInfo::regT2, GPRInfo::regT3);
 #endif
@@ -306 +306 @@
 #if CPU(ARM64E)
             jit.addPtr(AssemblyHelpers::TrustedImm32(inlineCallFrame->returnPCOffset() + sizeof(void*)), GPRInfo::callFrameRegister, GPRInfo::regT2);
-            jit.move(AssemblyHelpers::TrustedImmPtr(jumpTarget.untaggedExecutableAddress()), GPRInfo::nonArgGPR0);
-            jit.tagPtr(GPRInfo::regT2, GPRInfo::nonArgGPR0);
-            jit.storePtr(GPRInfo::nonArgGPR0, AssemblyHelpers::addressForByteOffset(inlineCallFrame->returnPCOffset()));
+            jit.move(AssemblyHelpers::TrustedImmPtr(jumpTarget.untaggedExecutableAddress()), GPRInfo::regT4);
+            jit.tagPtr(GPRInfo::regT2, GPRInfo::regT4);
+            jit.storePtr(GPRInfo::regT4, AssemblyHelpers::addressForByteOffset(inlineCallFrame->returnPCOffset()));
 #else
             jit.storePtr(AssemblyHelpers::TrustedImmPtr(jumpTarget.untaggedExecutableAddress()), AssemblyHelpers::addressForByteOffset(inlineCallFrame->returnPCOffset()));
@@ -319 +319 @@
         // If this inlined frame is a tail call that will return back to the original caller, we need to
         // copy the prior contents of the tag registers already saved for the outer frame to this frame.
-        jit.emitSaveOrCopyCalleeSavesFor(
+        jit.emitSaveOrCopyLLIntBaselineCalleeSavesFor(
             baselineCodeBlock,
             static_cast<VirtualRegister>(inlineCallFrame->stackOffset),
             trueCaller ? AssemblyHelpers::UseExistingTagRegisterContents : AssemblyHelpers::CopyBaselineCalleeSavedRegistersFromBaseFrame,
-            GPRInfo::regT2);
+            GPRInfo::regT2, GPRInfo::regT1, GPRInfo::regT4);
 
         if (callerIsLLInt) {

trunk/Source/JavaScriptCore/dfg/DFGThunks.cpp

@@ -29 +29 @@
 #if ENABLE(DFG_JIT)
 
+#include "AssemblyHelpersSpoolers.h"
 #include "CCallHelpers.h"
 #include "DFGJITCode.h"
@@ -50 +51 @@
     ScratchBuffer* scratchBuffer = vm.scratchBufferForSize(scratchSize);
     EncodedJSValue* buffer = static_cast<EncodedJSValue*>(scratchBuffer->dataBuffer());
-    
-    for (unsigned i = 0; i < GPRInfo::numberOfRegisters; ++i) {
+
+#if CPU(ARM64)
+    constexpr GPRReg bufferGPR = CCallHelpers::memoryTempRegister;
+    constexpr unsigned firstGPR = 0;
+#elif CPU(X86_64)
+    GPRReg bufferGPR = jit.scratchRegister();
+    constexpr unsigned firstGPR = 0;
+#else
+    GPRReg bufferGPR = GPRInfo::toRegister(0);
+    constexpr unsigned firstGPR = 1;
+#endif
+
+    if constexpr (firstGPR) {
+        // We're using the firstGPR as the bufferGPR, and need to save it manually.
+        RELEASE_ASSERT(GPRInfo::numberOfRegisters >= 1);
+        RELEASE_ASSERT(bufferGPR == GPRInfo::toRegister(0));
 #if USE(JSVALUE64)
-        jit.store64(GPRInfo::toRegister(i), buffer + i);
+        jit.store64(bufferGPR, buffer);
 #else
-        jit.store32(GPRInfo::toRegister(i), buffer + i);
+        jit.store32(bufferGPR, buffer);
 #endif
     }
+
+    jit.move(CCallHelpers::TrustedImmPtr(buffer), bufferGPR);
+
+    CCallHelpers::StoreRegSpooler storeSpooler(jit, bufferGPR);
+
+    for (unsigned i = firstGPR; i < GPRInfo::numberOfRegisters; ++i) {
+        ptrdiff_t offset = i * sizeof(CPURegister);
+        storeSpooler.storeGPR({ GPRInfo::toRegister(i), offset });
+    }
+    storeSpooler.finalizeGPR();
+
     for (unsigned i = 0; i < FPRInfo::numberOfRegisters; ++i) {
-        jit.move(MacroAssembler::TrustedImmPtr(buffer + GPRInfo::numberOfRegisters + i), GPRInfo::regT0);
-        jit.storeDouble(FPRInfo::toRegister(i), MacroAssembler::Address(GPRInfo::regT0));
+        ptrdiff_t offset = (GPRInfo::numberOfRegisters + i) * sizeof(double);
+        storeSpooler.storeFPR({ FPRInfo::toRegister(i), offset });
     }
-    
+    storeSpooler.finalizeFPR();
+
     // Set up one argument.
     jit.move(GPRInfo::callFrameRegister, GPRInfo::argumentGPR0);
@@ -69 +96 @@
     MacroAssembler::Call functionCall = jit.call(OperationPtrTag);
 
+    jit.move(CCallHelpers::TrustedImmPtr(buffer), bufferGPR);
+    CCallHelpers::LoadRegSpooler loadSpooler(jit, bufferGPR);
+
+    for (unsigned i = firstGPR; i < GPRInfo::numberOfRegisters; ++i) {
+        ptrdiff_t offset = i * sizeof(CPURegister);
+        loadSpooler.loadGPR({ GPRInfo::toRegister(i), offset });
+    }
+    loadSpooler.finalizeGPR();
+
     for (unsigned i = 0; i < FPRInfo::numberOfRegisters; ++i) {
-        jit.move(MacroAssembler::TrustedImmPtr(buffer + GPRInfo::numberOfRegisters + i), GPRInfo::regT0);
-        jit.loadDouble(MacroAssembler::Address(GPRInfo::regT0), FPRInfo::toRegister(i));
+        ptrdiff_t offset = (GPRInfo::numberOfRegisters + i) * sizeof(double);
+        loadSpooler.loadFPR({ FPRInfo::toRegister(i), offset });
     }
-    for (unsigned i = 0; i < GPRInfo::numberOfRegisters; ++i) {
+    loadSpooler.finalizeFPR();
+
+    if constexpr (firstGPR) {
+        // We're using the firstGPR as the bufferGPR, and need to restore it manually.
+        ASSERT(bufferGPR == GPRInfo::toRegister(0));
 #if USE(JSVALUE64)
-        jit.load64(buffer + i, GPRInfo::toRegister(i));
+        jit.load64(buffer, bufferGPR);
 #else
-        jit.load32(buffer + i, GPRInfo::toRegister(i));
+        jit.load32(buffer, bufferGPR);
 #endif
     }

trunk/Source/JavaScriptCore/ftl/FTLSaveRestore.cpp

@@ -1 +1 @@
 /*
- * Copyright (C) 2013, 2014 Apple Inc. All rights reserved.
+ * Copyright (C) 2013-2021 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -29 +29 @@
 #if ENABLE(FTL_JIT)
 
+#include "AssemblyHelpersSpoolers.h"
 #include "FPRInfo.h"
 #include "GPRInfo.h"
-#include "MacroAssembler.h"
+#include "Reg.h"
 #include "RegisterSet.h"
 
@@ -78 +79 @@
         special = RegisterSet::stackRegisters();
         special.merge(RegisterSet::reservedHardwareRegisters());
-        
+
         first = MacroAssembler::firstRegister();
         while (special.get(first))
             first = MacroAssembler::nextRegister(first);
-        second = MacroAssembler::nextRegister(first);
-        while (special.get(second))
-            second = MacroAssembler::nextRegister(second);
     }
-    
+
+    GPRReg nextRegister(GPRReg current)
+    {
+        auto next = MacroAssembler::nextRegister(current);
+        while (special.get(next))
+            next = MacroAssembler::nextRegister(next);
+        return next;
+    }
+
+    FPRReg nextFPRegister(FPRReg current)
+    {
+        auto next = MacroAssembler::nextFPRegister(current);
+        while (special.get(next))
+            next = MacroAssembler::nextFPRegister(next);
+        return next;
+    }
+
     RegisterSet special;
     GPRReg first;
-    GPRReg second;
 };
 
 } // anonymous namespace
 
-void saveAllRegisters(MacroAssembler& jit, char* scratchMemory)
+void saveAllRegisters(AssemblyHelpers& jit, char* scratchMemory)
 {
     Regs regs;
     
     // Get the first register out of the way, so that we can use it as a pointer.
-    jit.poke64(regs.first, 0);
-    jit.move(MacroAssembler::TrustedImmPtr(scratchMemory), regs.first);
-    
+    GPRReg baseGPR = regs.first;
+#if CPU(ARM64)
+    GPRReg nextGPR = regs.nextRegister(baseGPR);
+    GPRReg firstToSaveGPR = regs.nextRegister(nextGPR);
+    ASSERT(baseGPR == ARM64Registers::x0);
+    ASSERT(nextGPR == ARM64Registers::x1);
+#else
+    GPRReg firstToSaveGPR = regs.nextRegister(baseGPR);
+#endif
+    jit.poke64(baseGPR, 0);
+    jit.move(MacroAssembler::TrustedImmPtr(scratchMemory), baseGPR);
+
+    AssemblyHelpers::StoreRegSpooler spooler(jit, baseGPR);
+
     // Get all of the other GPRs out of the way.
-    for (MacroAssembler::RegisterID reg = regs.second; reg <= MacroAssembler::lastRegister(); reg = MacroAssembler::nextRegister(reg)) {
-        if (regs.special.get(reg))
-            continue;
-        jit.store64(reg, MacroAssembler::Address(regs.first, offsetOfGPR(reg)));
-    }
+    for (GPRReg reg = firstToSaveGPR; reg <= MacroAssembler::lastRegister(); reg = regs.nextRegister(reg))
+        spooler.storeGPR({ reg, static_cast<ptrdiff_t>(offsetOfGPR(reg)) });
+    spooler.finalizeGPR();
     
     // Restore the first register into the second one and save it.
-    jit.peek64(regs.second, 0);
-    jit.store64(regs.second, MacroAssembler::Address(regs.first, offsetOfGPR(regs.first)));
+    jit.peek64(firstToSaveGPR, 0);
+#if CPU(ARM64)
+    jit.storePair64(firstToSaveGPR, nextGPR, baseGPR, AssemblyHelpers::TrustedImm32(offsetOfGPR(baseGPR)));
+#else
+    jit.store64(firstToSaveGPR, MacroAssembler::Address(baseGPR, offsetOfGPR(baseGPR)));
+#endif
     
     // Finally save all FPR's.
-    for (MacroAssembler::FPRegisterID reg = MacroAssembler::firstFPRegister(); reg <= MacroAssembler::lastFPRegister(); reg = MacroAssembler::nextFPRegister(reg)) {
-        if (regs.special.get(reg))
-            continue;
-        jit.storeDouble(reg, MacroAssembler::Address(regs.first, offsetOfFPR(reg)));
-    }
+    for (MacroAssembler::FPRegisterID reg = MacroAssembler::firstFPRegister(); reg <= MacroAssembler::lastFPRegister(); reg = regs.nextFPRegister(reg))
+        spooler.storeFPR({ reg, static_cast<ptrdiff_t>(offsetOfFPR(reg)) });
+    spooler.finalizeFPR();
 }
 
-void restoreAllRegisters(MacroAssembler& jit, char* scratchMemory)
+void restoreAllRegisters(AssemblyHelpers& jit, char* scratchMemory)
 {
     Regs regs;
     
     // Give ourselves a pointer to the scratch memory.
-    jit.move(MacroAssembler::TrustedImmPtr(scratchMemory), regs.first);
+    GPRReg baseGPR = regs.first;
+    jit.move(MacroAssembler::TrustedImmPtr(scratchMemory), baseGPR);
     
+    AssemblyHelpers::LoadRegSpooler spooler(jit, baseGPR);
+
     // Restore all FPR's.
-    for (MacroAssembler::FPRegisterID reg = MacroAssembler::firstFPRegister(); reg <= MacroAssembler::lastFPRegister(); reg = MacroAssembler::nextFPRegister(reg)) {
-        if (regs.special.get(reg))
-            continue;
-        jit.loadDouble(MacroAssembler::Address(regs.first, offsetOfFPR(reg)), reg);
-    }
+    for (MacroAssembler::FPRegisterID reg = MacroAssembler::firstFPRegister(); reg <= MacroAssembler::lastFPRegister(); reg = regs.nextFPRegister(reg))
+        spooler.loadFPR({ reg, static_cast<ptrdiff_t>(offsetOfFPR(reg)) });
+    spooler.finalizeFPR();
     
-    for (MacroAssembler::RegisterID reg = regs.second; reg <= MacroAssembler::lastRegister(); reg = MacroAssembler::nextRegister(reg)) {
-        if (regs.special.get(reg))
-            continue;
-        jit.load64(MacroAssembler::Address(regs.first, offsetOfGPR(reg)), reg);
-    }
-    
-    jit.load64(MacroAssembler::Address(regs.first, offsetOfGPR(regs.first)), regs.first);
+#if CPU(ARM64)
+    GPRReg nextGPR = regs.nextRegister(baseGPR);
+    GPRReg firstToRestoreGPR = regs.nextRegister(nextGPR);
+    ASSERT(baseGPR == ARM64Registers::x0);
+    ASSERT(nextGPR == ARM64Registers::x1);
+#else
+    GPRReg firstToRestoreGPR = regs.nextRegister(baseGPR);
+#endif
+    for (MacroAssembler::RegisterID reg = firstToRestoreGPR; reg <= MacroAssembler::lastRegister(); reg = regs.nextRegister(reg))
+        spooler.loadGPR({ reg, static_cast<ptrdiff_t>(offsetOfGPR(reg)) });
+    spooler.finalizeGPR();
+
+#if CPU(ARM64)
+    jit.loadPair64(baseGPR, AssemblyHelpers::TrustedImm32(offsetOfGPR(baseGPR)), baseGPR, nextGPR);
+#else
+    jit.load64(MacroAssembler::Address(baseGPR, offsetOfGPR(baseGPR)), baseGPR);
+#endif
 }
 

trunk/Source/JavaScriptCore/ftl/FTLSaveRestore.h

@@ -1 +1 @@
 /*
- * Copyright (C) 2013 Apple Inc. All rights reserved.
+ * Copyright (C) 2013-2021 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -34 +34 @@
 namespace JSC {
 
-class MacroAssembler;
+class AssemblyHelpers;
 
 namespace FTL {
@@ -47 +47 @@
 // the registers into the scratch buffer such that RegisterID * sizeof(int64_t) is the
 // offset of every register.
-void saveAllRegisters(MacroAssembler& jit, char* scratchMemory);
+void saveAllRegisters(AssemblyHelpers& jit, char* scratchMemory);
 
-void restoreAllRegisters(MacroAssembler& jit, char* scratchMemory);
+void restoreAllRegisters(AssemblyHelpers& jit, char* scratchMemory);
 
 } } // namespace JSC::FTL

trunk/Source/JavaScriptCore/ftl/FTLThunks.cpp

@@ -29 +29 @@
 #if ENABLE(FTL_JIT)
 
-#include "AssemblyHelpers.h"
+#include "AssemblyHelpersSpoolers.h"
 #include "DFGOSRExitCompilerCommon.h"
 #include "FTLOSRExitCompiler.h"
@@ -57 +57 @@
     
     // Note that the "return address" will be the ID that we pass to the generation function.
-    
-    ptrdiff_t stackMisalignment = MacroAssembler::pushToSaveByteOffset();
-    
+
+    constexpr GPRReg stackPointerRegister = MacroAssembler::stackPointerRegister;
+    constexpr GPRReg framePointerRegister = MacroAssembler::framePointerRegister;
+    constexpr ptrdiff_t pushToSaveByteOffset = MacroAssembler::pushToSaveByteOffset();
+    ptrdiff_t stackMisalignment = pushToSaveByteOffset;
+
     // Pretend that we're a C call frame.
-    jit.pushToSave(MacroAssembler::framePointerRegister);
-    jit.move(MacroAssembler::stackPointerRegister, MacroAssembler::framePointerRegister);
-    stackMisalignment += MacroAssembler::pushToSaveByteOffset();
-    
+    jit.pushToSave(framePointerRegister);
+    jit.move(stackPointerRegister, framePointerRegister);
+    stackMisalignment += pushToSaveByteOffset;
+
     // Now create ourselves enough stack space to give saveAllRegisters() a scratch slot.
     unsigned numberOfRequiredPops = 0;
     do {
-        jit.pushToSave(GPRInfo::regT0);
-        stackMisalignment += MacroAssembler::pushToSaveByteOffset();
+        stackMisalignment += pushToSaveByteOffset;
         numberOfRequiredPops++;
     } while (stackMisalignment % stackAlignmentBytes());
-    
+    jit.subPtr(MacroAssembler::TrustedImm32(numberOfRequiredPops * pushToSaveByteOffset), stackPointerRegister);
+
     ScratchBuffer* scratchBuffer = vm.scratchBufferForSize(requiredScratchMemorySizeInBytes());
     char* buffer = static_cast<char*>(scratchBuffer->dataBuffer());
@@ -78 +81 @@
     saveAllRegisters(jit, buffer);
 
-    jit.loadPtr(GPRInfo::callFrameRegister, GPRInfo::argumentGPR0);
+    jit.loadPtr(framePointerRegister, GPRInfo::argumentGPR0);
     jit.peek(
         GPRInfo::argumentGPR1,
-        (stackMisalignment - MacroAssembler::pushToSaveByteOffset()) / sizeof(void*));
+        (stackMisalignment - pushToSaveByteOffset) / sizeof(void*));
     jit.prepareCallOperation(vm);
     MacroAssembler::Call functionCall = jit.call(OperationPtrTag);
@@ -93 +96 @@
 
     // Prepare for tail call.
-    while (numberOfRequiredPops--)
-        jit.popToRestore(GPRInfo::regT1);
-    jit.popToRestore(MacroAssembler::framePointerRegister);
-
-    // When we came in here, there was an additional thing pushed to the stack. Some clients want it
-    // popped before proceeding.
-    while (extraPopsToRestore--)
-        jit.popToRestore(GPRInfo::regT1);
+
+    jit.loadPtr(MacroAssembler::Address(stackPointerRegister, numberOfRequiredPops * pushToSaveByteOffset), framePointerRegister);
+
+    // When we came in here, there was an additional thing pushed to the stack (extraPopsToRestore).
+    // Some clients want it popped before proceeding. Also add 1 for the pushToSave of the framePointerRegister.
+    numberOfRequiredPops += 1 + extraPopsToRestore;
+    jit.addPtr(MacroAssembler::TrustedImm32(numberOfRequiredPops * pushToSaveByteOffset), stackPointerRegister);
 
     // Put the return address wherever the return instruction wants it. On all platforms, this
@@ -178 +180 @@
     currentOffset += sizeof(void*);
 #endif
-    
+
+    AssemblyHelpers::StoreRegSpooler storeSpooler(jit, MacroAssembler::stackPointerRegister);
+
     for (MacroAssembler::RegisterID reg = MacroAssembler::firstRegister(); reg <= MacroAssembler::lastRegister(); reg = static_cast<MacroAssembler::RegisterID>(reg + 1)) {
         if (!key.usedRegisters().get(reg))
             continue;
-        jit.storePtr(reg, AssemblyHelpers::Address(MacroAssembler::stackPointerRegister, currentOffset));
+        storeSpooler.storeGPR({ reg, static_cast<ptrdiff_t>(currentOffset) });
         currentOffset += sizeof(void*);
     }
-    
+    storeSpooler.finalizeGPR();
+
     for (MacroAssembler::FPRegisterID reg = MacroAssembler::firstFPRegister(); reg <= MacroAssembler::lastFPRegister(); reg = static_cast<MacroAssembler::FPRegisterID>(reg + 1)) {
         if (!key.usedRegisters().get(reg))
             continue;
-        jit.storeDouble(reg, AssemblyHelpers::Address(MacroAssembler::stackPointerRegister, currentOffset));
+        storeSpooler.storeFPR({ reg, static_cast<ptrdiff_t>(currentOffset) });
         currentOffset += sizeof(double);
     }
-    
+    storeSpooler.finalizeFPR();
+
     jit.preserveReturnAddressAfterCall(GPRInfo::nonArgGPR1);
     jit.storePtr(GPRInfo::nonArgGPR1, AssemblyHelpers::Address(MacroAssembler::stackPointerRegister, key.offset()));
@@ -211 +217 @@
     jit.restoreReturnAddressBeforeReturn(GPRInfo::nonPreservedNonReturnGPR);
     
+    AssemblyHelpers::LoadRegSpooler loadSpooler(jit, MacroAssembler::stackPointerRegister);
+
     for (MacroAssembler::FPRegisterID reg = MacroAssembler::lastFPRegister(); ; reg = static_cast<MacroAssembler::FPRegisterID>(reg - 1)) {
         if (key.usedRegisters().get(reg)) {
             currentOffset -= sizeof(double);
-            jit.loadDouble(AssemblyHelpers::Address(MacroAssembler::stackPointerRegister, currentOffset), reg);
+            loadSpooler.loadFPR({ reg, static_cast<ptrdiff_t>(currentOffset) });
         }
         if (reg == MacroAssembler::firstFPRegister())
             break;
     }
-    
+    loadSpooler.finalizeFPR();
+
     for (MacroAssembler::RegisterID reg = MacroAssembler::lastRegister(); ; reg = static_cast<MacroAssembler::RegisterID>(reg - 1)) {
         if (key.usedRegisters().get(reg)) {
             currentOffset -= sizeof(void*);
-            jit.loadPtr(AssemblyHelpers::Address(MacroAssembler::stackPointerRegister, currentOffset), reg);
+            loadSpooler.loadGPR({ reg, static_cast<ptrdiff_t>(currentOffset) });
         }
         if (reg == MacroAssembler::firstRegister())
             break;
     }
-    
+    loadSpooler.finalizeGPR();
+
     jit.ret();
 

trunk/Source/JavaScriptCore/jit/AssemblyHelpers.cpp

@@ -30 +30 @@
 
 #include "AccessCase.h"
+#include "AssemblyHelpersSpoolers.h"
 #include "JITOperations.h"
 #include "JSArrayBufferView.h"
@@ -607 +608 @@
     GPRReg scratch = InvalidGPRReg;
     unsigned scratchGPREntryIndex = 0;
-
-    // Use the first GPR entry's register as our scratch.
+#if CPU(ARM64)
+    // We don't need a second scratch GPR, but we'll also defer restoring this
+    // GPR (in the next slot after the scratch) so that we can restore them together
+    // later using a loadPair64.
+    GPRReg unusedNextSlotGPR = InvalidGPRReg;
+#endif
+
+    // Use the first GPR entry's register as our baseGPR.
     for (unsigned i = 0; i < registerCount; i++) {
         RegisterAtOffset entry = allCalleeSaves->at(i);
-        if (dontRestoreRegisters.get(entry.reg()))
+        if (dontRestoreRegisters.contains(entry.reg()))
             continue;
         if (entry.reg().isGPR()) {
+#if CPU(ARM64)
+            if (i + 1 < registerCount) {
+                RegisterAtOffset entry2 = allCalleeSaves->at(i + 1);
+                if (!dontRestoreRegisters.contains(entry2.reg())
+                    && entry2.reg().isGPR()
+                    && entry2.offset() == entry.offset() + static_cast<ptrdiff_t>(sizeof(CPURegister))) {
+                    scratchGPREntryIndex = i;
+                    scratch = entry.reg().gpr();
+                    unusedNextSlotGPR = entry2.reg().gpr();
+                    break;
+                }
+            }
+#else
             scratchGPREntryIndex = i;
             scratch = entry.reg().gpr();
             break;
+#endif
         }
     }
     ASSERT(scratch != InvalidGPRReg);
+    
+    RegisterSet skipList;
+    skipList.set(dontRestoreRegisters);
+
+    // Skip the scratch register(s). We'll restore them later.
+    skipList.add(scratch);
+#if CPU(ARM64)
+    RELEASE_ASSERT(unusedNextSlotGPR != InvalidGPRReg);
+    skipList.add(unusedNextSlotGPR);
+#endif
 
     loadPtr(&topEntryFrame, scratch);
     addPtr(TrustedImm32(EntryFrame::calleeSaveRegistersBufferOffset()), scratch);
 
+    LoadRegSpooler spooler(*this, scratch);
+
     // Restore all callee saves except for the scratch.
-    for (unsigned i = 0; i < registerCount; i++) {
+    unsigned i = 0;
+    for (; i < registerCount; i++) {
         RegisterAtOffset entry = allCalleeSaves->at(i);
-        if (dontRestoreRegisters.get(entry.reg()))
+        if (skipList.contains(entry.reg()))
             continue;
-        if (i == scratchGPREntryIndex)
+        if (!entry.reg().isGPR())
+            break;
+        spooler.loadGPR(entry);
+    }
+    spooler.finalizeGPR();
+    for (; i < registerCount; i++) {
+        RegisterAtOffset entry = allCalleeSaves->at(i);
+        if (skipList.contains(entry.reg()))
             continue;
-        loadReg(Address(scratch, entry.offset()), entry.reg());
-    }
+        ASSERT(!entry.reg().isGPR());
+        spooler.loadFPR(entry);
+    }
+    spooler.finalizeFPR();
 
     // Restore the callee save value of the scratch.
@@ -639 +682 @@
     ASSERT(entry.reg().isGPR());
     ASSERT(scratch == entry.reg().gpr());
-    loadReg(Address(scratch, entry.offset()), scratch);
+#if CPU(ARM64)
+    RegisterAtOffset entry2 = allCalleeSaves->at(scratchGPREntryIndex + 1);
+    ASSERT_UNUSED(entry2, !dontRestoreRegisters.get(entry2.reg()));
+    ASSERT(entry2.reg().isGPR());
+    ASSERT(unusedNextSlotGPR == entry2.reg().gpr());
+    loadPair64(scratch, TrustedImm32(entry.offset()), scratch, unusedNextSlotGPR);
+#else
+    loadPtr(Address(scratch, entry.offset()), scratch);
+#endif
+
 #else
     UNUSED_PARAM(topEntryFrame);
-#endif
+#endif // NUMBER_OF_CALLEE_SAVES_REGISTERS > 0
 }
 
@@ -1000 +1052 @@
     RegisterSet dontCopyRegisters = RegisterSet::stackRegisters();
     unsigned registerCount = allCalleeSaves->size();
-    
-    for (unsigned i = 0; i < registerCount; i++) {
+
+    StoreRegSpooler spooler(*this, calleeSavesBuffer);
+
+    unsigned i = 0;
+    for (; i < registerCount; i++) {
         RegisterAtOffset entry = allCalleeSaves->at(i);
-        if (dontCopyRegisters.get(entry.reg()))
+        if (dontCopyRegisters.contains(entry.reg()))
             continue;
-        storeReg(entry.reg(), Address(calleeSavesBuffer, entry.offset()));
-    }
+        if (!entry.reg().isGPR())
+            break;
+        spooler.storeGPR(entry);
+    }
+    spooler.finalizeGPR();
+    for (; i < registerCount; i++) {
+        RegisterAtOffset entry = allCalleeSaves->at(i);
+        if (dontCopyRegisters.contains(entry.reg()))
+            continue;
+        spooler.storeFPR(entry);
+    }
+    spooler.finalizeFPR();
+
 #else
     UNUSED_PARAM(calleeSavesBuffer);
@@ -1104 +1170 @@
 }
 
+void AssemblyHelpers::emitSave(const RegisterAtOffsetList& list)
+{
+    StoreRegSpooler spooler(*this, framePointerRegister);
+
+    size_t listSize = list.size();
+    size_t i = 0;
+    for (; i < listSize; i++) {
+        auto entry = list.at(i);
+        if (!entry.reg().isGPR())
+            break;
+        spooler.storeGPR(entry);
+    }
+    spooler.finalizeGPR();
+
+    for (; i < listSize; i++)
+        spooler.storeFPR(list.at(i));
+    spooler.finalizeFPR();
+}
+
+void AssemblyHelpers::emitRestore(const RegisterAtOffsetList& list)
+{
+    LoadRegSpooler spooler(*this, framePointerRegister);
+
+    size_t listSize = list.size();
+    size_t i = 0;
+    for (; i < listSize; i++) {
+        auto entry = list.at(i);
+        if (!entry.reg().isGPR())
+            break;
+        spooler.loadGPR(entry);
+    }
+    spooler.finalizeGPR();
+
+    for (; i < listSize; i++)
+        spooler.loadFPR(list.at(i));
+    spooler.finalizeFPR();
+}
+
+void AssemblyHelpers::emitSaveCalleeSavesFor(const RegisterAtOffsetList* calleeSaves)
+{
+    RegisterSet dontSaveRegisters = RegisterSet(RegisterSet::stackRegisters());
+    unsigned registerCount = calleeSaves->size();
+
+    StoreRegSpooler spooler(*this, framePointerRegister);
+
+    unsigned i = 0;
+    for (; i < registerCount; i++) {
+        RegisterAtOffset entry = calleeSaves->at(i);
+        if (entry.reg().isFPR())
+            break;
+        if (dontSaveRegisters.contains(entry.reg()))
+            continue;
+        spooler.storeGPR(entry);
+    }
+    spooler.finalizeGPR();
+    for (; i < registerCount; i++) {
+        RegisterAtOffset entry = calleeSaves->at(i);
+        if (dontSaveRegisters.contains(entry.reg()))
+            continue;
+        spooler.storeFPR(entry);
+    }
+    spooler.finalizeFPR();
+}
+
+void AssemblyHelpers::emitRestoreCalleeSavesFor(const RegisterAtOffsetList* calleeSaves)
+{
+    RegisterSet dontRestoreRegisters = RegisterSet(RegisterSet::stackRegisters());
+    unsigned registerCount = calleeSaves->size();
+    
+    LoadRegSpooler spooler(*this, framePointerRegister);
+
+    unsigned i = 0;
+    for (; i < registerCount; i++) {
+        RegisterAtOffset entry = calleeSaves->at(i);
+        if (entry.reg().isFPR())
+            break;
+        if (dontRestoreRegisters.get(entry.reg()))
+            continue;
+        spooler.loadGPR(entry);
+    }
+    spooler.finalizeGPR();
+    for (; i < registerCount; i++) {
+        RegisterAtOffset entry = calleeSaves->at(i);
+        if (dontRestoreRegisters.get(entry.reg()))
+            continue;
+        spooler.loadFPR(entry);
+    }
+    spooler.finalizeFPR();
+}
+
+void AssemblyHelpers::copyLLIntBaselineCalleeSavesFromFrameOrRegisterToEntryFrameCalleeSavesBuffer(EntryFrame*& topEntryFrame, const TempRegisterSet& usedRegisters)
+{
+#if NUMBER_OF_CALLEE_SAVES_REGISTERS > 0
+    // Copy saved calleeSaves on stack or unsaved calleeSaves in register to vm calleeSave buffer
+    GPRReg destBufferGPR = usedRegisters.getFreeGPR(0);
+    GPRReg temp1 = usedRegisters.getFreeGPR(1);
+    FPRReg fpTemp1 = usedRegisters.getFreeFPR(0);
+    GPRReg temp2 = isARM64() ? usedRegisters.getFreeGPR(2) : InvalidGPRReg;
+    FPRReg fpTemp2 = isARM64() ? usedRegisters.getFreeFPR(1) : InvalidFPRReg;
+
+    loadPtr(&topEntryFrame, destBufferGPR);
+    addPtr(TrustedImm32(EntryFrame::calleeSaveRegistersBufferOffset()), destBufferGPR);
+
+    CopySpooler spooler(*this, framePointerRegister, destBufferGPR, temp1, temp2, fpTemp1, fpTemp2);
+
+    RegisterAtOffsetList* allCalleeSaves = RegisterSet::vmCalleeSaveRegisterOffsets();
+    const RegisterAtOffsetList* currentCalleeSaves = &RegisterAtOffsetList::llintBaselineCalleeSaveRegisters();
+    RegisterSet dontCopyRegisters = RegisterSet::stackRegisters();
+    unsigned registerCount = allCalleeSaves->size();
+
+    unsigned i = 0;
+    for (; i < registerCount; i++) {
+        RegisterAtOffset entry = allCalleeSaves->at(i);
+        if (dontCopyRegisters.contains(entry.reg()))
+            continue;
+        RegisterAtOffset* currentFrameEntry = currentCalleeSaves->find(entry.reg());
+
+        if (!entry.reg().isGPR())
+            break;
+        if (currentFrameEntry)
+            spooler.loadGPR(currentFrameEntry->offset());
+        else
+            spooler.copyGPR(entry.reg().gpr());
+        spooler.storeGPR(entry.offset());
+    }
+    spooler.finalizeGPR();
+
+    for (; i < registerCount; i++) {
+        RegisterAtOffset entry = allCalleeSaves->at(i);
+        if (dontCopyRegisters.get(entry.reg()))
+            continue;
+        RegisterAtOffset* currentFrameEntry = currentCalleeSaves->find(entry.reg());
+
+        RELEASE_ASSERT(entry.reg().isFPR());
+        if (currentFrameEntry)
+            spooler.loadFPR(currentFrameEntry->offset());
+        else
+            spooler.copyFPR(entry.reg().fpr());
+        spooler.storeFPR(entry.offset());
+    }
+    spooler.finalizeFPR();
+
+#else
+    UNUSED_PARAM(topEntryFrame);
+    UNUSED_PARAM(usedRegisters);
+#endif
+}
+
+void AssemblyHelpers::emitSaveOrCopyLLIntBaselineCalleeSavesFor(CodeBlock* codeBlock, VirtualRegister offsetVirtualRegister, RestoreTagRegisterMode tagRegisterMode, GPRReg temp1, GPRReg temp2, GPRReg temp3)
+{
+    ASSERT_UNUSED(codeBlock, codeBlock);
+    ASSERT(JITCode::isBaselineCode(codeBlock->jitType()));
+    ASSERT(codeBlock->calleeSaveRegisters() == &RegisterAtOffsetList::llintBaselineCalleeSaveRegisters());
+
+    const RegisterAtOffsetList* calleeSaves = &RegisterAtOffsetList::llintBaselineCalleeSaveRegisters();
+    RegisterSet dontSaveRegisters = RegisterSet(RegisterSet::stackRegisters());
+    unsigned registerCount = calleeSaves->size();
+
+    GPRReg dstBufferGPR = temp1;
+    addPtr(TrustedImm32(offsetVirtualRegister.offsetInBytes()), framePointerRegister, dstBufferGPR);
+
+    CopySpooler spooler(*this, framePointerRegister, dstBufferGPR, temp2, temp3);
+
+    for (unsigned i = 0; i < registerCount; i++) {
+        RegisterAtOffset entry = calleeSaves->at(i);
+        if (dontSaveRegisters.get(entry.reg()))
+            continue;
+        RELEASE_ASSERT(entry.reg().isGPR());
+
+#if USE(JSVALUE32_64)
+        UNUSED_PARAM(tagRegisterMode);
+#else
+        if (tagRegisterMode == CopyBaselineCalleeSavedRegistersFromBaseFrame)
+            spooler.loadGPR(entry.offset());
+        else
+#endif
+            spooler.copyGPR(entry.reg().gpr());
+        spooler.storeGPR(entry.offset());
+    }
+    spooler.finalizeGPR();
+}
+
 } // namespace JSC
 

trunk/Source/JavaScriptCore/jit/AssemblyHelpers.h

@@ -296 +296 @@
 #endif
     }
-    
+
+    template<typename Op> class Spooler;
+    class LoadRegSpooler;
+    class StoreRegSpooler;
+    class CopySpooler;
+
     Address addressFor(const RegisterAtOffset& entry)
     {
         return Address(GPRInfo::callFrameRegister, entry.offset());
     }
-    
-    void emitSave(const RegisterAtOffsetList& list)
-    {
-        for (const RegisterAtOffset& entry : list) {
-            if (entry.reg().isGPR())
-                storePtr(entry.reg().gpr(), addressFor(entry));
-            else
-                storeDouble(entry.reg().fpr(), addressFor(entry));
-        }
-    }
-    
-    void emitRestore(const RegisterAtOffsetList& list)
-    {
-        for (const RegisterAtOffset& entry : list) {
-            if (entry.reg().isGPR())
-                loadPtr(addressFor(entry), entry.reg().gpr());
-            else
-                loadDouble(addressFor(entry), entry.reg().fpr());
-        }
-    }
+
+    void emitSave(const RegisterAtOffsetList&);
+    void emitRestore(const RegisterAtOffsetList&);
 
     void emitSaveCalleeSavesFor(CodeBlock* codeBlock)
@@ -327 +315 @@
 
         const RegisterAtOffsetList* calleeSaves = codeBlock->calleeSaveRegisters();
-        RegisterSet dontSaveRegisters = RegisterSet(RegisterSet::stackRegisters());
-        unsigned registerCount = calleeSaves->size();
-
-        for (unsigned i = 0; i < registerCount; i++) {
-            RegisterAtOffset entry = calleeSaves->at(i);
-            if (dontSaveRegisters.get(entry.reg()))
-                continue;
-            storeReg(entry.reg(), Address(framePointerRegister, entry.offset()));
-        }
-    }
+        emitSaveCalleeSavesFor(calleeSaves);
+    }
+
+    void emitSaveCalleeSavesFor(const RegisterAtOffsetList* calleeSaves);
     
     enum RestoreTagRegisterMode { UseExistingTagRegisterContents, CopyBaselineCalleeSavedRegistersFromBaseFrame };
 
-    void emitSaveOrCopyCalleeSavesFor(CodeBlock* codeBlock, VirtualRegister offsetVirtualRegister, RestoreTagRegisterMode tagRegisterMode, GPRReg temp)
-    {
-        ASSERT(codeBlock);
-        ASSERT(JITCode::isBaselineCode(codeBlock->jitType()));
-        
-        const RegisterAtOffsetList* calleeSaves = codeBlock->calleeSaveRegisters();
-        RegisterSet dontSaveRegisters = RegisterSet(RegisterSet::stackRegisters());
-        unsigned registerCount = calleeSaves->size();
-
-#if USE(JSVALUE64)
-        RegisterSet baselineCalleeSaves = RegisterSet::llintBaselineCalleeSaveRegisters();
-#endif
-        
-        for (unsigned i = 0; i < registerCount; i++) {
-            RegisterAtOffset entry = calleeSaves->at(i);
-            if (dontSaveRegisters.get(entry.reg()))
-                continue;
-            RELEASE_ASSERT(entry.reg().isGPR());
-
-            GPRReg registerToWrite;
-
-#if USE(JSVALUE32_64)
-            UNUSED_PARAM(tagRegisterMode);
-            UNUSED_PARAM(temp);
-#else
-            if (tagRegisterMode == CopyBaselineCalleeSavedRegistersFromBaseFrame && baselineCalleeSaves.get(entry.reg())) {
-                registerToWrite = temp;
-                loadPtr(AssemblyHelpers::Address(GPRInfo::callFrameRegister, entry.offset()), registerToWrite);
-            } else
-#endif
-                registerToWrite = entry.reg().gpr();
-
-            storePtr(registerToWrite, Address(framePointerRegister, offsetVirtualRegister.offsetInBytes() + entry.offset()));
-        }
-    }
+    void emitSaveOrCopyLLIntBaselineCalleeSavesFor(CodeBlock*, VirtualRegister offsetVirtualRegister, RestoreTagRegisterMode, GPRReg temp1, GPRReg temp2, GPRReg temp3);
     
     void emitRestoreCalleeSavesFor(CodeBlock* codeBlock)
@@ -384 +332 @@
     }
 
-    void emitRestoreCalleeSavesFor(const RegisterAtOffsetList* calleeSaves)
-    {
-        RegisterSet dontRestoreRegisters = RegisterSet(RegisterSet::stackRegisters());
-        unsigned registerCount = calleeSaves->size();
-        
-        for (unsigned i = 0; i < registerCount; i++) {
-            RegisterAtOffset entry = calleeSaves->at(i);
-            if (dontRestoreRegisters.get(entry.reg()))
-                continue;
-            loadReg(Address(framePointerRegister, entry.offset()), entry.reg());
-        }
-    }
+    void emitRestoreCalleeSavesFor(const RegisterAtOffsetList* calleeSaves);
 
     void emitSaveCalleeSaves()
@@ -466 +403 @@
     void restoreCalleeSavesFromEntryFrameCalleeSavesBuffer(EntryFrame*&);
 
-    void copyLLIntBaselineCalleeSavesFromFrameOrRegisterToEntryFrameCalleeSavesBuffer(EntryFrame*& topEntryFrame, const TempRegisterSet& usedRegisters = { RegisterSet::stubUnavailableRegisters() })
-    {
-#if NUMBER_OF_CALLEE_SAVES_REGISTERS > 0
-        GPRReg temp1 = usedRegisters.getFreeGPR(0);
-        GPRReg temp2 = usedRegisters.getFreeGPR(1);
-        FPRReg fpTemp = usedRegisters.getFreeFPR();
-        ASSERT(temp2 != InvalidGPRReg);
-
-        // Copy saved calleeSaves on stack or unsaved calleeSaves in register to vm calleeSave buffer
-        loadPtr(&topEntryFrame, temp1);
-        addPtr(TrustedImm32(EntryFrame::calleeSaveRegistersBufferOffset()), temp1);
-
-        RegisterAtOffsetList* allCalleeSaves = RegisterSet::vmCalleeSaveRegisterOffsets();
-        const RegisterAtOffsetList* currentCalleeSaves = &RegisterAtOffsetList::llintBaselineCalleeSaveRegisters();
-        RegisterSet dontCopyRegisters = RegisterSet::stackRegisters();
-        unsigned registerCount = allCalleeSaves->size();
-
-        for (unsigned i = 0; i < registerCount; i++) {
-            RegisterAtOffset entry = allCalleeSaves->at(i);
-            if (dontCopyRegisters.get(entry.reg()))
-                continue;
-            RegisterAtOffset* currentFrameEntry = currentCalleeSaves->find(entry.reg());
-
-            if (entry.reg().isGPR()) {
-                GPRReg regToStore;
-                if (currentFrameEntry) {
-                    // Load calleeSave from stack into temp register
-                    regToStore = temp2;
-                    loadPtr(Address(framePointerRegister, currentFrameEntry->offset()), regToStore);
-                } else
-                    // Just store callee save directly
-                    regToStore = entry.reg().gpr();
-
-                storePtr(regToStore, Address(temp1, entry.offset()));
-            } else {
-                FPRReg fpRegToStore;
-                if (currentFrameEntry) {
-                    // Load calleeSave from stack into temp register
-                    fpRegToStore = fpTemp;
-                    loadDouble(Address(framePointerRegister, currentFrameEntry->offset()), fpRegToStore);
-                } else
-                    // Just store callee save directly
-                    fpRegToStore = entry.reg().fpr();
-
-                storeDouble(fpRegToStore, Address(temp1, entry.offset()));
-            }
-        }
-#else
-        UNUSED_PARAM(topEntryFrame);
-        UNUSED_PARAM(usedRegisters);
-#endif
-    }
+    void copyLLIntBaselineCalleeSavesFromFrameOrRegisterToEntryFrameCalleeSavesBuffer(EntryFrame*&, const TempRegisterSet& usedRegisters = { RegisterSet::stubUnavailableRegisters() });
 
     void emitMaterializeTagCheckRegisters()

trunk/Source/JavaScriptCore/jit/ScratchRegisterAllocator.cpp

@@ -1 +1 @@
 /*
- * Copyright (C) 2014-2017 Apple Inc. All rights reserved.
+ * Copyright (C) 2014-2021 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -29 +29 @@
 #if ENABLE(JIT)
 
+#include "AssemblyHelpersSpoolers.h"
 #include "MaxFrameExtentForSlowPathCall.h"
 #include "VM.h"
@@ -102 +103 @@
 FPRReg ScratchRegisterAllocator::allocateScratchFPR() { return allocateScratch<FPRInfo>(); }
 
-ScratchRegisterAllocator::PreservedState ScratchRegisterAllocator::preserveReusedRegistersByPushing(MacroAssembler& jit, ExtraStackSpace extraStackSpace)
+ScratchRegisterAllocator::PreservedState ScratchRegisterAllocator::preserveReusedRegistersByPushing(AssemblyHelpers& jit, ExtraStackSpace extraStackSpace)
 {
     if (!didReuseRegisters())
@@ -125 +126 @@
 }
 
-void ScratchRegisterAllocator::restoreReusedRegistersByPopping(MacroAssembler& jit, const ScratchRegisterAllocator::PreservedState& preservedState)
+void ScratchRegisterAllocator::restoreReusedRegistersByPopping(AssemblyHelpers& jit, const ScratchRegisterAllocator::PreservedState& preservedState)
 {
     RELEASE_ASSERT(preservedState);
@@ -162 +163 @@
 }
 
-unsigned ScratchRegisterAllocator::preserveRegistersToStackForCall(MacroAssembler& jit, const RegisterSet& usedRegisters, unsigned extraBytesAtTopOfStack)
+unsigned ScratchRegisterAllocator::preserveRegistersToStackForCall(AssemblyHelpers& jit, const RegisterSet& usedRegisters, unsigned extraBytesAtTopOfStack)
 {
     RELEASE_ASSERT(extraBytesAtTopOfStack % sizeof(void*) == 0);
@@ -175 +176 @@
         MacroAssembler::stackPointerRegister);
 
+    AssemblyHelpers::StoreRegSpooler spooler(jit, MacroAssembler::stackPointerRegister);
+
     unsigned count = 0;
     for (GPRReg reg = MacroAssembler::firstRegister(); reg <= MacroAssembler::lastRegister(); reg = MacroAssembler::nextRegister(reg)) {
         if (usedRegisters.get(reg)) {
-            jit.storePtr(reg, MacroAssembler::Address(MacroAssembler::stackPointerRegister, extraBytesAtTopOfStack + (count * sizeof(EncodedJSValue))));
-            count++;
-        }
-    }
+            spooler.storeGPR({ reg, static_cast<ptrdiff_t>(extraBytesAtTopOfStack + (count * sizeof(EncodedJSValue))) });
+            count++;
+        }
+    }
+    spooler.finalizeGPR();
+
     for (FPRReg reg = MacroAssembler::firstFPRegister(); reg <= MacroAssembler::lastFPRegister(); reg = MacroAssembler::nextFPRegister(reg)) {
         if (usedRegisters.get(reg)) {
-            jit.storeDouble(reg, MacroAssembler::Address(MacroAssembler::stackPointerRegister, extraBytesAtTopOfStack + (count * sizeof(EncodedJSValue))));
-            count++;
-        }
-    }
+            spooler.storeFPR({ reg, static_cast<ptrdiff_t>(extraBytesAtTopOfStack + (count * sizeof(EncodedJSValue))) });
+            count++;
+        }
+    }
+    spooler.finalizeFPR();
 
     RELEASE_ASSERT(count == usedRegisters.numberOfSetRegisters());
@@ -194 +200 @@
 }
 
-void ScratchRegisterAllocator::restoreRegistersFromStackForCall(MacroAssembler& jit, const RegisterSet& usedRegisters, const RegisterSet& ignore, unsigned numberOfStackBytesUsedForRegisterPreservation, unsigned extraBytesAtTopOfStack)
+void ScratchRegisterAllocator::restoreRegistersFromStackForCall(AssemblyHelpers& jit, const RegisterSet& usedRegisters, const RegisterSet& ignore, unsigned numberOfStackBytesUsedForRegisterPreservation, unsigned extraBytesAtTopOfStack)
 {
     RELEASE_ASSERT(extraBytesAtTopOfStack % sizeof(void*) == 0);
@@ -202 +208 @@
     }
 
+    AssemblyHelpers::LoadRegSpooler spooler(jit, MacroAssembler::stackPointerRegister);
+
     unsigned count = 0;
     for (GPRReg reg = MacroAssembler::firstRegister(); reg <= MacroAssembler::lastRegister(); reg = MacroAssembler::nextRegister(reg)) {
         if (usedRegisters.get(reg)) {
             if (!ignore.get(reg))
-                jit.loadPtr(MacroAssembler::Address(MacroAssembler::stackPointerRegister, extraBytesAtTopOfStack + (sizeof(EncodedJSValue) * count)), reg);
-            count++;
-        }
-    }
+                spooler.loadGPR({ reg, static_cast<ptrdiff_t>(extraBytesAtTopOfStack + (sizeof(EncodedJSValue) * count)) });
+            count++;
+        }
+    }
+    spooler.finalizeGPR();
+
     for (FPRReg reg = MacroAssembler::firstFPRegister(); reg <= MacroAssembler::lastFPRegister(); reg = MacroAssembler::nextFPRegister(reg)) {
         if (usedRegisters.get(reg)) {
             if (!ignore.get(reg))
-                jit.loadDouble(MacroAssembler::Address(MacroAssembler::stackPointerRegister, extraBytesAtTopOfStack + (sizeof(EncodedJSValue) * count)), reg);
-            count++;
-        }
-    }
+                spooler.loadFPR({ reg, static_cast<ptrdiff_t>(extraBytesAtTopOfStack + (sizeof(EncodedJSValue) * count)) });
+            count++;
+        }
+    }
+    spooler.finalizeFPR();
 
     unsigned stackOffset = (usedRegisters.numberOfSetRegisters()) * sizeof(EncodedJSValue);

trunk/Source/JavaScriptCore/jit/ScratchRegisterAllocator.h

@@ -1 +1 @@
 /*
- * Copyright (C) 2012, 2014 Apple Inc. All rights reserved.
+ * Copyright (C) 2012-2021 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -28 +28 @@
 #if ENABLE(JIT)
 
-#include "MacroAssembler.h"
 #include "RegisterSet.h"
 #include "TempRegisterSet.h"
@@ -34 +33 @@
 namespace JSC {
 
+class AssemblyHelpers;
 struct ScratchBuffer;
 
@@ -85 +85 @@
     };
 
-    PreservedState preserveReusedRegistersByPushing(MacroAssembler& jit, ExtraStackSpace);
-    void restoreReusedRegistersByPopping(MacroAssembler& jit, const PreservedState&);
+    PreservedState preserveReusedRegistersByPushing(AssemblyHelpers& jit, ExtraStackSpace);
+    void restoreReusedRegistersByPopping(AssemblyHelpers& jit, const PreservedState&);
     
     RegisterSet usedRegistersForCall() const;
     
     unsigned desiredScratchBufferSizeForCall() const;
-    
-    static unsigned preserveRegistersToStackForCall(MacroAssembler& jit, const RegisterSet& usedRegisters, unsigned extraPaddingInBytes);
-    static void restoreRegistersFromStackForCall(MacroAssembler& jit, const RegisterSet& usedRegisters, const RegisterSet& ignore, unsigned numberOfStackBytesUsedForRegisterPreservation, unsigned extraPaddingInBytes);
+
+    static unsigned preserveRegistersToStackForCall(AssemblyHelpers& jit, const RegisterSet& usedRegisters, unsigned extraPaddingInBytes);
+    static void restoreRegistersFromStackForCall(AssemblyHelpers& jit, const RegisterSet& usedRegisters, const RegisterSet& ignore, unsigned numberOfStackBytesUsedForRegisterPreservation, unsigned extraPaddingInBytes);
 
 private:

trunk/Source/JavaScriptCore/wasm/WasmAirIRGenerator.cpp

@@ -1 +1 @@
 /*
- * Copyright (C) 2019-2020 Apple Inc. All rights reserved.
+ * Copyright (C) 2019-2021 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -2982 +2982 @@
     patch->setGenerator([] (CCallHelpers& jit, const B3::StackmapGenerationParams& params) {
         auto calleeSaves = params.code().calleeSaveRegisterAtOffsetList();
-
-        for (RegisterAtOffset calleeSave : calleeSaves)
-            jit.load64ToReg(CCallHelpers::Address(GPRInfo::callFrameRegister, calleeSave.offset()), calleeSave.reg());
-
+        jit.emitRestore(calleeSaves);
         jit.emitFunctionEpilogue();
         jit.ret();

trunk/Source/JavaScriptCore/wasm/WasmB3IRGenerator.cpp

@@ -1 +1 @@
 /*
- * Copyright (C) 2016-2020 Apple Inc. All rights reserved.
+ * Copyright (C) 2016-2021 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -2285 +2285 @@
     patch->setGenerator([] (CCallHelpers& jit, const B3::StackmapGenerationParams& params) {
         auto calleeSaves = params.code().calleeSaveRegisterAtOffsetList();
-
-        for (RegisterAtOffset calleeSave : calleeSaves)
-            jit.load64ToReg(CCallHelpers::Address(GPRInfo::callFrameRegister, calleeSave.offset()), calleeSave.reg());
-
+        jit.emitRestore(calleeSaves);
         jit.emitFunctionEpilogue();
         jit.ret();