Changeset 277383 in webkit


Ignore:
Timestamp:
May 12, 2021, 12:11:25 PM (4 years ago)
Author:
mark.lam@apple.com
Message:

Implement some common Baseline JIT slow paths using JIT thunks.
​https://bugs.webkit.org/show_bug.cgi?id=225682

Reviewed by Filip Pizlo.

Source/JavaScriptCore:

This patch implements the following changes:

  1. Implement exception handling thunks:
    1. handleExceptionGenerator, which calls operationLookupExceptionHandler().
    2. handleExceptionWithCallFrameRollbackGenerator, which calls operationLookupExceptionHandlerFromCallerFrame().

All the JIT tiers were emitting their own copy of these routines to call these
operation, one per CodeBlock. We now emit 2 thunks for these and have all the
tiers just jump to them.

PolymorphicAccess also now uses the handleExceptionGenerator thunk.

DFG::JITCompiler::compileExceptionHandlers() has one small behavior difference
before it calls operationLookupExceptionHandlerFromCallerFrame(): it first
re-sets the top of stack for the function where we are about to throw a
StackOverflowError from. This re-setting of top of stack is useless because
we're imminently unwinding out of at least this frame for the StackOverflowError.
Hence, it is ok to use the handleExceptionWithCallFrameRollbackGenerator thunk
here as well. Note that no other tiers does this re-setting of top of stack.

FTLLowerDFGToB3 has one case using operationLookupExceptionHandlerFromCallerFrame()
which cannot be refactored to use these thunks because it does additional
work to throw a StackOverflowError. A different thunk will be needed. I left
it alone for now.

  1. Introduce JITThunks::existingCTIStub(ThunkGenerator, NoLockingNecessaryTag) so that a thunk can get a pointer to another thunk without locking the JITThunks lock. Otherwise, deadlock ensues.
  1. Change SlowPathCall to emit and use thunks instead of emitting a blob of code to call a slow path function for every bytecode in a CodeBlock.
  1. Introduce JITThunks::ctiSlowPathFunctionStub() to manage these SlowPathFunction thunks.
  1. Introduce JITThunks::preinitializeAggressiveCTIThunks() to initialize these thunks at VM initialization time. Pre-initializing them has multiple benefits:
    1. the thunks are not scattered through out JIT memory, thereby reducing fragmentation.
    2. we don't spend time at runtime compiling them when the user is interacting with the VM. Conceptually, these thunks can be VM independent and can be shared by VMs process-wide. However, it will require some additional work. For now, the thunks remain bound to a specific VM instance.

These changes are only enabled when ENABLE(EXTRA_CTI_THUNKS), which is currently
only available for ARM64 and non-Windows x86_64.

This patch has passed JSC tests on AS Mac.

With this patch, --dumpLinkBufferStats shows the following changes in emitted
JIT code size (using a single run of the CLI version of JetStream2 on AS Mac):

Base New Diff

BaselineJIT: 89089964 (84.962811 MB) 84624776 (80.704475 MB) 0.95x (reduction)

DFG: 39117360 (37.305222 MB) 36415264 (34.728302 MB) 0.93x (reduction)

Thunk: 23230968 (22.154778 MB) 23130336 (22.058807 MB) 1.00x

InlineCache: 22027416 (21.006981 MB) 21969728 (20.951965 MB) 1.00x

FTL: 6575772 (6.271145 MB) 6097336 (5.814873 MB) 0.93x (reduction)

Wasm: 2302724 (2.196049 MB) 2301956 (2.195316 MB) 1.00x

YarrJIT: 1538956 (1.467663 MB) 1522488 (1.451958 MB) 0.99x

CSSJIT: 0 0

Uncategorized: 0 0

  • CMakeLists.txt:
  • JavaScriptCore.xcodeproj/project.pbxproj:
  • Sources.txt:
  • bytecode/CodeBlock.h:

(JSC::CodeBlock::offsetOfInstructionsRawPointer):

  • bytecode/PolymorphicAccess.cpp:

(JSC::AccessGenerationState::emitExplicitExceptionHandler):

  • dfg/DFGJITCompiler.cpp:

(JSC::DFG::JITCompiler::compileExceptionHandlers):
(JSC::DFG::JITCompiler::link):

  • dfg/DFGJITCompiler.h:
  • ftl/FTLCompile.cpp:

(JSC::FTL::compile):

  • ftl/FTLLink.cpp:

(JSC::FTL::link):

  • jit/JIT.cpp:

(JSC::JIT::link):
(JSC::JIT::privateCompileExceptionHandlers):

  • jit/JIT.h:
  • jit/JITThunks.cpp:

(JSC::JITThunks::existingCTIStub):
(JSC::JITThunks::ctiSlowPathFunctionStub):
(JSC::JITThunks::preinitializeExtraCTIThunks):

  • jit/JITThunks.h:
  • jit/SlowPathCall.cpp: Added.

(JSC::JITSlowPathCall::call):
(JSC::JITSlowPathCall::generateThunk):

  • jit/SlowPathCall.h:
  • jit/ThunkGenerators.cpp:

(JSC::handleExceptionGenerator):
(JSC::handleExceptionWithCallFrameRollbackGenerator):
(JSC::popThunkStackPreservesAndHandleExceptionGenerator):

  • jit/ThunkGenerators.h:
  • runtime/CommonSlowPaths.h:
  • runtime/SlowPathFunction.h: Added.
  • runtime/VM.cpp:

(JSC::VM::VM):

Source/WTF:

Introduce ENABLE(EXTRA_CTI_THUNKS) flag to guard the use of these new thunks.
Currently, the thunks are 64-bit only, and only supported for ARM64 and non-Windows
X86_64. The reason it is not supported for Windows as well is because Windows
only has 4 argument registers. In this patch, the thunks do not use that many
registers yet, but there will be more thunks coming that will require the use
of up to 6 argument registers.

  • wtf/PlatformEnable.h:
Location:
trunk/Source
Files:
2 added
21 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/JavaScriptCore/CMakeLists.txt

    r277027 r277383  
    10641064    runtime/ScriptFetchParameters.h
    10651065    runtime/ScriptFetcher.h
     1066    runtime/SlowPathFunction.h
    10661067    runtime/SlowPathReturnType.h
    10671068    runtime/SmallStrings.h

  • trunk/Source/JavaScriptCore/ChangeLog

    r277381 r277383  
     12021-05-12  Mark Lam  <mark.lam@apple.com>
     2
     3        Implement some common Baseline JIT slow paths using JIT thunks.
     4        https://bugs.webkit.org/show_bug.cgi?id=225682
     5
     6        Reviewed by Filip Pizlo.
     7
     8        This patch implements the following changes:
     9
     10        1. Implement exception handling thunks:
     11           a. handleExceptionGenerator, which calls operationLookupExceptionHandler().
     12           b. handleExceptionWithCallFrameRollbackGenerator, which calls
     13              operationLookupExceptionHandlerFromCallerFrame().
     14
     15           All the JIT tiers were emitting their own copy of these routines to call these
     16           operation, one per CodeBlock.  We now emit 2 thunks for these and have all the
     17           tiers just jump to them.
     18
     19           PolymorphicAccess also now uses the handleExceptionGenerator thunk.
     20
     21           DFG::JITCompiler::compileExceptionHandlers() has one small behavior difference
     22           before it calls operationLookupExceptionHandlerFromCallerFrame(): it first
     23           re-sets the top of stack for the function where we are about to throw a
     24           StackOverflowError from.  This re-setting of top of stack is useless because
     25           we're imminently unwinding out of at least this frame for the StackOverflowError.
     26           Hence, it is ok to use the handleExceptionWithCallFrameRollbackGenerator thunk
     27           here as well.  Note that no other tiers does this re-setting of top of stack.
     28
     29           FTLLowerDFGToB3 has one case using operationLookupExceptionHandlerFromCallerFrame()
     30           which cannot be refactored to use these thunks because it does additional
     31           work to throw a StackOverflowError.  A different thunk will be needed.  I left
     32           it alone for now.
     33
     34        2. Introduce JITThunks::existingCTIStub(ThunkGenerator, NoLockingNecessaryTag) so
     35           that a thunk can get a pointer to another thunk without locking the JITThunks
     36           lock.  Otherwise, deadlock ensues.
     37
     38        3. Change SlowPathCall to emit and use thunks instead of emitting a blob of code
     39           to call a slow path function for every bytecode in a CodeBlock.
     40
     41        4. Introduce JITThunks::ctiSlowPathFunctionStub() to manage these SlowPathFunction
     42           thunks.
     43
     44        5. Introduce JITThunks::preinitializeAggressiveCTIThunks() to initialize these
     45           thunks at VM initialization time.  Pre-initializing them has multiple benefits:
     46           a. the thunks are not scattered through out JIT memory, thereby reducing
     47              fragmentation.
     48           b. we don't spend time at runtime compiling them when the user is interacting
     49              with the VM.  Conceptually, these thunks can be VM independent and can be
     50              shared by VMs process-wide.  However, it will require some additional work.
     51              For now, the thunks remain bound to a specific VM instance.
     52
     53        These changes are only enabled when ENABLE(EXTRA_CTI_THUNKS), which is currently
     54        only available for ARM64 and non-Windows x86_64.
     55
     56        This patch has passed JSC tests on AS Mac.
     57
     58        With this patch, --dumpLinkBufferStats shows the following changes in emitted
     59        JIT code size (using a single run of the CLI version of JetStream2 on AS Mac):
     60
     61                        Base                     New                      Diff
     62
     63           BaselineJIT: 89089964 (84.962811 MB)  84624776 (80.704475 MB)  0.95x (reduction)
     64                   DFG: 39117360 (37.305222 MB)  36415264 (34.728302 MB)  0.93x (reduction)
     65                 Thunk: 23230968 (22.154778 MB)  23130336 (22.058807 MB)  1.00x
     66           InlineCache: 22027416 (21.006981 MB)  21969728 (20.951965 MB)  1.00x
     67                   FTL: 6575772 (6.271145 MB)    6097336 (5.814873 MB)    0.93x (reduction)
     68                  Wasm: 2302724 (2.196049 MB)    2301956 (2.195316 MB)    1.00x
     69               YarrJIT: 1538956 (1.467663 MB)    1522488 (1.451958 MB)    0.99x
     70                CSSJIT: 0                        0
     71         Uncategorized: 0                        0
     72
     73        * CMakeLists.txt:
     74        * JavaScriptCore.xcodeproj/project.pbxproj:
     75        * Sources.txt:
     76        * bytecode/CodeBlock.h:
     77        (JSC::CodeBlock::offsetOfInstructionsRawPointer):
     78        * bytecode/PolymorphicAccess.cpp:
     79        (JSC::AccessGenerationState::emitExplicitExceptionHandler):
     80        * dfg/DFGJITCompiler.cpp:
     81        (JSC::DFG::JITCompiler::compileExceptionHandlers):
     82        (JSC::DFG::JITCompiler::link):
     83        * dfg/DFGJITCompiler.h:
     84        * ftl/FTLCompile.cpp:
     85        (JSC::FTL::compile):
     86        * ftl/FTLLink.cpp:
     87        (JSC::FTL::link):
     88        * jit/JIT.cpp:
     89        (JSC::JIT::link):
     90        (JSC::JIT::privateCompileExceptionHandlers):
     91        * jit/JIT.h:
     92        * jit/JITThunks.cpp:
     93        (JSC::JITThunks::existingCTIStub):
     94        (JSC::JITThunks::ctiSlowPathFunctionStub):
     95        (JSC::JITThunks::preinitializeExtraCTIThunks):
     96        * jit/JITThunks.h:
     97        * jit/SlowPathCall.cpp: Added.
     98        (JSC::JITSlowPathCall::call):
     99        (JSC::JITSlowPathCall::generateThunk):
     100        * jit/SlowPathCall.h:
     101        * jit/ThunkGenerators.cpp:
     102        (JSC::handleExceptionGenerator):
     103        (JSC::handleExceptionWithCallFrameRollbackGenerator):
     104        (JSC::popThunkStackPreservesAndHandleExceptionGenerator):
     105        * jit/ThunkGenerators.h:
     106        * runtime/CommonSlowPaths.h:
     107        * runtime/SlowPathFunction.h: Added.
     108        * runtime/VM.cpp:
     109        (JSC::VM::VM):
     110
    11112021-05-12  Commit Queue  <commit-queue@webkit.org>
    2112

  • trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj

    r276676 r277383  
    19891989                FE7C41961B97FC4B00F4D598 /* PingPongStackOverflowTest.cpp in Sources */ = {isa = PBXBuildFile; fileRef = FEDA50D41B97F442009A3B4F /* PingPongStackOverflowTest.cpp */; };
    19901990                FE80C1971D775CDD008510C0 /* CatchScope.h in Headers */ = {isa = PBXBuildFile; fileRef = FE80C1961D775B27008510C0 /* CatchScope.h */; settings = {ATTRIBUTES = (Private, ); }; };
     1991                FE8C0312264A6911001A44AD /* SlowPathFunction.h in Headers */ = {isa = PBXBuildFile; fileRef = FE8C0311264A6910001A44AD /* SlowPathFunction.h */; settings = {ATTRIBUTES = (Private, ); }; };
    19911992                FE8DE54B23AC1DAD005C9142 /* CacheableIdentifier.h in Headers */ = {isa = PBXBuildFile; fileRef = FE8DE54A23AC1DAD005C9142 /* CacheableIdentifier.h */; };
    19921993                FE8DE54D23AC1E86005C9142 /* CacheableIdentifierInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = FE8DE54C23AC1E86005C9142 /* CacheableIdentifierInlines.h */; };
    … …  
    40114012                6A38CFA81E32B58B0060206F /* AsyncStackTrace.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AsyncStackTrace.h; sourceTree = "<group>"; };
    40124013                6AD2CB4C19B9140100065719 /* DebuggerEvalEnabler.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = DebuggerEvalEnabler.h; sourceTree = "<group>"; };
     4014                6B731CC02647A8370014646F /* SlowPathCall.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = SlowPathCall.cpp; sourceTree = "<group>"; };
    40134015                6BA93C9590484C5BAD9316EA /* JSScriptFetcher.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JSScriptFetcher.h; sourceTree = "<group>"; };
    40144016                6BCCEC0325D1FA27000F391D /* VerifierSlotVisitorInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = VerifierSlotVisitorInlines.h; sourceTree = "<group>"; };
    … …  
    53635365                FE80C1981D775FB4008510C0 /* CatchScope.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = CatchScope.cpp; sourceTree = "<group>"; };
    53645366                FE80C19A1D7768FD008510C0 /* ExceptionEventLocation.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ExceptionEventLocation.cpp; sourceTree = "<group>"; };
     5367                FE8C0311264A6910001A44AD /* SlowPathFunction.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = SlowPathFunction.h; sourceTree = "<group>"; };
    53655368                FE8DE54A23AC1DAD005C9142 /* CacheableIdentifier.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CacheableIdentifier.h; sourceTree = "<group>"; };
    53665369                FE8DE54C23AC1E86005C9142 /* CacheableIdentifierInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CacheableIdentifierInlines.h; sourceTree = "<group>"; };
    … …  
    63036306                                0FEE98421A89227500754E93 /* SetupVarargsFrame.cpp */,
    63046307                                0FEE98401A8865B600754E93 /* SetupVarargsFrame.h */,
     6308                                6B731CC02647A8370014646F /* SlowPathCall.cpp */,
    63056309                                A709F2EF17A0AC0400512E98 /* SlowPathCall.h */,
    63066310                                E3F23A7B1ECF13E500978D99 /* Snippet.h */,
    … …  
    78377841                                0F2B66D617B6B5AB00A7AE3F /* SimpleTypedArrayController.cpp */,
    78387842                                0F2B66D717B6B5AB00A7AE3F /* SimpleTypedArrayController.h */,
     7843                                FE8C0311264A6910001A44AD /* SlowPathFunction.h */,
    78397844                                0F5B4A321C84F0D600F1B17E /* SlowPathReturnType.h */,
    78407845                                93303FE80E6A72B500786E6A /* SmallStrings.cpp */,
    … …  
    1024310248                                86E3C61B167BABEE006D760A /* JSValueInternal.h in Headers */,
    1024410249                                53E1F8F82154715A0001DDBC /* JSValuePrivate.h in Headers */,
     10250                                FE8C0312264A6911001A44AD /* SlowPathFunction.h in Headers */,
    1024510251                                BC18C42C0E16F5CD00B34460 /* JSValueRef.h in Headers */,
    1024610252                                86E3C615167BABD7006D760A /* JSVirtualMachine.h in Headers */,

  • trunk/Source/JavaScriptCore/Sources.txt

    r275856 r277383  
    677677jit/ScratchRegisterAllocator.cpp
    678678jit/SetupVarargsFrame.cpp
     679jit/SlowPathCall.cpp
    679680jit/TagRegistersMode.cpp
    680681jit/TempRegisterSet.cpp

  • trunk/Source/JavaScriptCore/bytecode/CodeBlock.h

    r277153 r277383  
    916916    const void* instructionsRawPointer() { return m_instructionsRawPointer; }
    917917
     918    static ptrdiff_t offsetOfInstructionsRawPointer() { return OBJECT_OFFSETOF(CodeBlock, m_instructionsRawPointer); }
     919
    918920    bool loopHintsAreEligibleForFuzzingEarlyReturn()
    919921    {

  • trunk/Source/JavaScriptCore/bytecode/PolymorphicAccess.cpp

    r277305 r277383  
    4141#include "StructureStubInfo.h"
    4242#include "SuperSampler.h"
     43#include "ThunkGenerators.h"
    4344#include <wtf/CommaPrinter.h>
    4445#include <wtf/ListDump.h>
    … …  
    227228            });
    228229    } else {
     230#if ENABLE(EXTRA_CTI_THUNKS)
     231        CCallHelpers::Jump jumpToExceptionHandler = jit->jump();
     232        VM* vm = &m_vm;
     233        jit->addLinkTask(
     234            [=] (LinkBuffer& linkBuffer) {
     235                linkBuffer.link(jumpToExceptionHandler, CodeLocationLabel(vm->getCTIStub(handleExceptionGenerator).retaggedCode<NoPtrTag>()));
     236            });
     237#else
    229238        jit->setupArguments<decltype(operationLookupExceptionHandler)>(CCallHelpers::TrustedImmPtr(&m_vm));
    230239        jit->prepareCallOperation(m_vm);
    … …  
    235244            });
    236245        jit->jumpToExceptionHandler(m_vm);
     246#endif
    237247    }
    238248}

  • trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.cpp

    r277305 r277383  
    138138void JITCompiler::compileExceptionHandlers()
    139139{
     140#if !ENABLE(EXTRA_CTI_THUNKS)
    140141    if (!m_exceptionChecksWithCallFrameRollback.empty()) {
    141142        m_exceptionChecksWithCallFrameRollback.link(this);
    … …  
    166167        jumpToExceptionHandler(vm());
    167168    }
     169#endif // ENABLE(EXTRA_CTI_THUNKS)
    168170}
    169171
    … …  
    282284            linkBuffer.locationOfNearCall<JSInternalPtrTag>(record.call));
    283285    }
    284    
     286
     287#if ENABLE(EXTRA_CTI_THUNKS)
     288    if (!m_exceptionChecks.empty())
     289        linkBuffer.link(m_exceptionChecks, CodeLocationLabel(vm().getCTIStub(handleExceptionGenerator).retaggedCode<NoPtrTag>()));
     290    if (!m_exceptionChecksWithCallFrameRollback.empty())
     291        linkBuffer.link(m_exceptionChecksWithCallFrameRollback, CodeLocationLabel(vm().getCTIStub(handleExceptionWithCallFrameRollbackGenerator).retaggedCode<NoPtrTag>()));
     292#endif // ENABLE(EXTRA_CTI_THUNKS)
     293
    285294    MacroAssemblerCodeRef<JITThunkPtrTag> osrExitThunk = vm().getCTIStub(osrExitGenerationThunkGenerator);
    286295    auto target = CodeLocationLabel<JITThunkPtrTag>(osrExitThunk.code());

  • trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.h

    r276224 r277383  
    324324    JumpList m_exceptionChecks;
    325325    JumpList m_exceptionChecksWithCallFrameRollback;
    326    
     326
    327327    Vector<Label> m_blockHeads;
    328328

  • trunk/Source/JavaScriptCore/ftl/FTLCompile.cpp

    r277305 r277383  
    4141#include "LinkBuffer.h"
    4242#include "PCToCodeOriginMap.h"
     43#include "ThunkGenerators.h"
    4344#include <wtf/RecursableLambda.h>
    4445
    … …  
    126127    // Emit the exception handler.
    127128    *state.exceptionHandler = jit.label();
     129#if ENABLE(EXTRA_CTI_THUNKS)
     130    CCallHelpers::Jump handler = jit.jump();
     131    VM* vmPtr = &vm;
     132    jit.addLinkTask(
     133        [=] (LinkBuffer& linkBuffer) {
     134            linkBuffer.link(handler, CodeLocationLabel(vmPtr->getCTIStub(handleExceptionGenerator).retaggedCode<NoPtrTag>()));
     135        });
     136#else
    128137    jit.copyCalleeSavesToEntryFrameCalleeSavesBuffer(vm.topEntryFrame);
    129138    jit.move(MacroAssembler::TrustedImmPtr(&vm), GPRInfo::argumentGPR0);
    … …  
    135144            linkBuffer.link(call, FunctionPtr<OperationPtrTag>(operationLookupExceptionHandler));
    136145        });
     146#endif // ENABLE(EXTRA_CTI_THUNKS)
    137147
    138148    state.finalizer->b3CodeLinkBuffer = makeUnique<LinkBuffer>(jit, codeBlock, LinkBuffer::Profile::FTL, JITCompilationCanFail);

  • trunk/Source/JavaScriptCore/ftl/FTLLink.cpp

    r277305 r277383  
    146146            CCallHelpers::Call callArityCheck = jit.call(OperationPtrTag);
    147147
     148#if ENABLE(EXTRA_CTI_THUNKS)
     149            auto jumpToExceptionHandler = jit.branch32(CCallHelpers::LessThan, GPRInfo::returnValueGPR, CCallHelpers::TrustedImm32(0));
     150#else
    148151            auto noException = jit.branch32(CCallHelpers::GreaterThanOrEqual, GPRInfo::returnValueGPR, CCallHelpers::TrustedImm32(0));
    149152            jit.copyCalleeSavesToEntryFrameCalleeSavesBuffer(vm.topEntryFrame);
    … …  
    153156            jit.jumpToExceptionHandler(vm);
    154157            noException.link(&jit);
     158#endif // ENABLE(EXTRA_CTI_THUNKS)
    155159
    156160            if (ASSERT_ENABLED) {
    … …  
    175179            }
    176180            linkBuffer->link(callArityCheck, FunctionPtr<OperationPtrTag>(codeBlock->isConstructor() ? operationConstructArityCheck : operationCallArityCheck));
     181#if ENABLE(EXTRA_CTI_THUNKS)
     182            linkBuffer->link(jumpToExceptionHandler, CodeLocationLabel(vm.getCTIStub(handleExceptionWithCallFrameRollbackGenerator).retaggedCode<NoPtrTag>()));
     183#else
    177184            linkBuffer->link(callLookupExceptionHandlerFromCallerFrame, FunctionPtr<OperationPtrTag>(operationLookupExceptionHandlerFromCallerFrame));
     185#endif
    178186            linkBuffer->link(callArityFixup, FunctionPtr<JITThunkPtrTag>(vm.getCTIStub(arityFixupGenerator).code()));
    179187            linkBuffer->link(mainPathJumps, state.generatedFunction);

  • trunk/Source/JavaScriptCore/jit/JIT.cpp

    r277370 r277383  
    898898    }
    899899
     900#if ENABLE(EXTRA_CTI_THUNKS)
     901    if (!m_exceptionChecks.empty())
     902        patchBuffer.link(m_exceptionChecks, CodeLocationLabel(vm().getCTIStub(handleExceptionGenerator).retaggedCode<NoPtrTag>()));
     903    if (!m_exceptionChecksWithCallFrameRollback.empty())
     904        patchBuffer.link(m_exceptionChecksWithCallFrameRollback, CodeLocationLabel(vm().getCTIStub(handleExceptionWithCallFrameRollbackGenerator).retaggedCode<NoPtrTag>()));
     905#endif
    900906
    901907    for (auto& record : m_nearCalls) {
    … …  
    907913            patchBuffer.link(record.from, record.callee);
    908914    }
    909    
     915
    910916    finalizeInlineCaches(m_getByIds, patchBuffer);
    911917    finalizeInlineCaches(m_getByVals, patchBuffer);
    … …  
    919925
    920926    if (m_byValCompilationInfo.size()) {
     927#if ENABLE(EXTRA_CTI_THUNKS)
     928        CodeLocationLabel exceptionHandler(vm().getCTIStub(handleExceptionGenerator).retaggedCode<ExceptionHandlerPtrTag>());
     929#else
    921930        CodeLocationLabel<ExceptionHandlerPtrTag> exceptionHandler = patchBuffer.locationOf<ExceptionHandlerPtrTag>(m_exceptionHandler);
     931#endif
    922932
    923933        for (const auto& byValCompilationInfo : m_byValCompilationInfo) {
    … …  
    10101020void JIT::privateCompileExceptionHandlers()
    10111021{
     1022#if !ENABLE(EXTRA_CTI_THUNKS)
    10121023    if (!m_exceptionChecksWithCallFrameRollback.empty()) {
    10131024        m_exceptionChecksWithCallFrameRollback.link(this);
    … …  
    10341045        jumpToExceptionHandler(vm());
    10351046    }
     1047#endif // ENABLE(EXTRA_CTI_THUNKS)
    10361048}
    10371049

  • trunk/Source/JavaScriptCore/jit/JIT.h

    r277370 r277383  
    988988        JumpList m_exceptionChecks;
    989989        JumpList m_exceptionChecksWithCallFrameRollback;
     990#if !ENABLE(EXTRA_CTI_THUNKS)
    990991        Label m_exceptionHandler;
     992#endif
    991993
    992994        unsigned m_getByIdIndex { UINT_MAX };

  • trunk/Source/JavaScriptCore/jit/JITThunks.cpp

    r263117 r277383  
    11/*
    2  * Copyright (C) 2012-2020 Apple Inc. All rights reserved.
     2 * Copyright (C) 2012-2021 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2929#if ENABLE(JIT)
    3030
     31#include "CommonSlowPaths.h"
    3132#include "JITCode.h"
    3233#include "JSCJSValueInlines.h"
     34#include "SlowPathCall.h"
    3335#include "ThunkGenerators.h"
    3436#include "VM.h"
    … …  
    142144{
    143145    LockHolder locker(m_lock);
     146    return existingCTIStub(generator, NoLockingNecessary);
     147}
     148
     149MacroAssemblerCodeRef<JITThunkPtrTag> JITThunks::existingCTIStub(ThunkGenerator generator, NoLockingNecessaryTag)
     150{
    144151    CTIStubMap::iterator entry = m_ctiStubMap.find(generator);
    145152    if (entry == m_ctiStubMap.end())
    … …  
    147154    return entry->value;
    148155}
     156
     157#if ENABLE(EXTRA_CTI_THUNKS)
     158
     159MacroAssemblerCodeRef<JITThunkPtrTag> JITThunks::ctiSlowPathFunctionStub(VM& vm, SlowPathFunction slowPathFunction)
     160{
     161    LockHolder locker(m_lock);
     162    auto key = bitwise_cast<ThunkGenerator>(slowPathFunction);
     163    CTIStubMap::AddResult entry = m_ctiStubMap.add(key, MacroAssemblerCodeRef<JITThunkPtrTag>());
     164    if (entry.isNewEntry) {
     165        // Compilation thread can only retrieve existing entries.
     166        ASSERT(!isCompilationThread());
     167        entry.iterator->value = JITSlowPathCall::generateThunk(vm, slowPathFunction);
     168    }
     169    return entry.iterator->value;
     170}
     171
     172#endif // ENABLE(EXTRA_CTI_THUNKS)
    149173
    150174struct JITThunks::HostKeySearcher {
    … …  
    234258}
    235259
     260#if ENABLE(EXTRA_CTI_THUNKS)
     261
     262void JITThunks::preinitializeExtraCTIThunks(VM& vm)
     263{
     264    if (!Options::useJIT())
     265        return;
     266
     267    // These 3 should always be initialized first in the following order because
     268    // the other thunk generators rely on these already being initialized.
     269    ctiStub(vm, handleExceptionGenerator);
     270    ctiStub(vm, handleExceptionWithCallFrameRollbackGenerator);
     271    ctiStub(vm, popThunkStackPreservesAndHandleExceptionGenerator);
     272
     273#define INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(name) ctiSlowPathFunctionStub(vm, slow_path_##name)
     274
     275    // From the BaselineJIT DEFINE_SLOW_OP list:
     276    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(in_by_val);
     277    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(less);
     278    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(lesseq);
     279    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(greater);
     280    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(greatereq);
     281    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(is_callable);
     282    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(is_constructor);
     283    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(typeof);
     284    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(typeof_is_object);
     285    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(typeof_is_function);
     286    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(strcat);
     287    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(push_with_scope);
     288    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(create_lexical_environment);
     289    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(get_by_val_with_this);
     290    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(put_by_id_with_this);
     291    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(put_by_val_with_this);
     292    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(resolve_scope_for_hoisting_func_decl_in_eval);
     293    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(define_data_property);
     294    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(define_accessor_property);
     295    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(unreachable);
     296    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(throw_static_error);
     297    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(new_array_with_spread);
     298    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(new_array_buffer);
     299    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(spread);
     300    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(get_enumerable_length);
     301    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(has_enumerable_property);
     302    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(get_property_enumerator);
     303    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(to_index_string);
     304    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(create_direct_arguments);
     305    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(create_scoped_arguments);
     306    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(create_cloned_arguments);
     307    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(create_arguments_butterfly);
     308    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(create_rest);
     309    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(create_promise);
     310    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(new_promise);
     311    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(create_generator);
     312    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(create_async_generator);
     313    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(new_generator);
     314    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(pow);
     315
     316    // From the BaselineJIT DEFINE_SLOWCASE_SLOW_OP list:
     317    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(unsigned);
     318    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(inc);
     319    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(dec);
     320    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(bitnot);
     321    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(bitand);
     322    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(bitor);
     323    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(bitxor);
     324    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(lshift);
     325    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(rshift);
     326    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(urshift);
     327    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(div);
     328    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(create_this);
     329    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(create_promise);
     330    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(create_generator);
     331    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(create_async_generator);
     332    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(to_this);
     333    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(to_primitive);
     334    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(to_number);
     335    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(to_numeric);
     336    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(to_string);
     337    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(to_object);
     338    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(not);
     339    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(stricteq);
     340    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(nstricteq);
     341    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(get_direct_pname);
     342    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(get_prototype_of);
     343    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(has_enumerable_structure_property);
     344    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(has_own_structure_property);
     345    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(in_structure_property);
     346    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(resolve_scope);
     347    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(check_tdz);
     348    INIT_BASELINE_SLOW_PATH_CALL_ROUTINE(to_property_key);
     349#undef INIT_BASELINE_ROUTINE
     350}
     351
     352#endif // ENABLE(EXTRA_CTI_THUNKS)
     353
    236354} // namespace JSC
    237355

  • trunk/Source/JavaScriptCore/jit/JITThunks.h

    r264488 r277383  
    11/*
    2  * Copyright (C) 2012-2019 Apple Inc. All rights reserved.
     2 * Copyright (C) 2012-2021 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3131#include "Intrinsic.h"
    3232#include "MacroAssemblerCodeRef.h"
     33#include "SlowPathFunction.h"
    3334#include "ThunkGenerator.h"
    3435#include "Weak.h"
    … …  
    5354    ~JITThunks() final;
    5455
     56#if ENABLE(EXTRA_CTI_THUNKS)
     57    void preinitializeExtraCTIThunks(VM&);
     58#endif
     59
    5560    MacroAssemblerCodePtr<JITThunkPtrTag> ctiNativeCall(VM&);
    5661    MacroAssemblerCodePtr<JITThunkPtrTag> ctiNativeConstruct(VM&);
    … …  
    6267    MacroAssemblerCodeRef<JITThunkPtrTag> ctiStub(VM&, ThunkGenerator);
    6368    MacroAssemblerCodeRef<JITThunkPtrTag> existingCTIStub(ThunkGenerator);
     69    MacroAssemblerCodeRef<JITThunkPtrTag> existingCTIStub(ThunkGenerator, NoLockingNecessaryTag);
     70#if ENABLE(EXTRA_CTI_THUNKS)
     71    MacroAssemblerCodeRef<JITThunkPtrTag> ctiSlowPathFunctionStub(VM&, SlowPathFunction);
     72#endif
    6473
    6574    NativeExecutable* hostFunctionStub(VM&, TaggedNativeFunction, TaggedNativeFunction constructor, const String& name);

  • trunk/Source/JavaScriptCore/jit/SlowPathCall.h

    r277370 r277383  
    11/*
    2  * Copyright (C) 2013-2018 Apple Inc. All rights reserved.
     2 * Copyright (C) 2013-2021 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2626#pragma once
    2727
    28 #include "CommonSlowPaths.h"
     28#include "JIT.h"
    2929#include "MacroAssemblerCodeRef.h"
     30#include "SlowPathFunction.h"
    3031
    3132#if ENABLE(JIT)
    … …  
    4243        assertIsCFunctionPtr(slowPathFunction);
    4344    }
     45
     46#if ENABLE(EXTRA_CTI_THUNKS)
     47    void call();
     48    static MacroAssemblerCodeRef<JITThunkPtrTag> generateThunk(VM&, SlowPathFunction);
     49
     50#else // not ENABLE(EXTRA_CTI_THUNKS)
    4451
    4552    JIT::Call call()
    … …  
    6370        static_assert(JIT::regT1 == GPRInfo::returnValueGPR2);
    6471#endif
    65        
     72
    6673        m_jit->exceptionCheck();
    6774        return call;
    6875    }
     76#endif // ENABLE(EXTRA_CTI_THUNKS)
    6977
    7078private:

  • trunk/Source/JavaScriptCore/jit/ThunkGenerators.cpp

    r277305 r277383  
    3434#include "SpecializedThunkJIT.h"
    3535#include <wtf/InlineASM.h>
     36#include <wtf/StdIntExtras.h>
    3637#include <wtf/StringPrintStream.h>
    3738#include <wtf/text/StringImpl.h>
    … …  
    4041
    4142namespace JSC {
     43
     44#if ENABLE(EXTRA_CTI_THUNKS)
     45
     46MacroAssemblerCodeRef<JITThunkPtrTag> handleExceptionGenerator(VM& vm)
     47{
     48    CCallHelpers jit;
     49
     50    jit.copyCalleeSavesToEntryFrameCalleeSavesBuffer(vm.topEntryFrame);
     51
     52    jit.move(CCallHelpers::TrustedImmPtr(&vm), GPRInfo::argumentGPR0);
     53    jit.prepareCallOperation(vm);
     54
     55    CCallHelpers::Call operation = jit.call(OperationPtrTag);
     56
     57    jit.jumpToExceptionHandler(vm);
     58
     59    LinkBuffer patchBuffer(jit, GLOBAL_THUNK_ID, LinkBuffer::Profile::Thunk);
     60    patchBuffer.link(operation, FunctionPtr<OperationPtrTag>(operationLookupExceptionHandler));
     61    return FINALIZE_CODE(patchBuffer, JITThunkPtrTag, "handleException");
     62}
     63
     64MacroAssemblerCodeRef<JITThunkPtrTag> handleExceptionWithCallFrameRollbackGenerator(VM& vm)
     65{
     66    CCallHelpers jit;
     67
     68    jit.copyCalleeSavesToEntryFrameCalleeSavesBuffer(vm.topEntryFrame);
     69
     70    jit.move(CCallHelpers::TrustedImmPtr(&vm), GPRInfo::argumentGPR0);
     71    jit.prepareCallOperation(vm);
     72    CCallHelpers::Call operation = jit.call(OperationPtrTag);
     73    jit.jumpToExceptionHandler(vm);
     74
     75    LinkBuffer patchBuffer(jit, GLOBAL_THUNK_ID, LinkBuffer::Profile::Thunk);
     76    patchBuffer.link(operation, FunctionPtr<OperationPtrTag>(operationLookupExceptionHandlerFromCallerFrame));
     77    return FINALIZE_CODE(patchBuffer, JITThunkPtrTag, "handleExceptionWithCallFrameRollback");
     78}
     79
     80MacroAssemblerCodeRef<JITThunkPtrTag> popThunkStackPreservesAndHandleExceptionGenerator(VM& vm)
     81{
     82    CCallHelpers jit;
     83
     84#if CPU(X86_64)
     85    jit.addPtr(CCallHelpers::TrustedImm32(2 * sizeof(CPURegister)), X86Registers::esp);
     86#elif CPU(ARM64)
     87    jit.popPair(CCallHelpers::framePointerRegister, CCallHelpers::linkRegister);
     88#endif
     89
     90    CCallHelpers::Jump continuation = jit.jump();
     91
     92    LinkBuffer patchBuffer(jit, GLOBAL_THUNK_ID, LinkBuffer::Profile::Thunk);
     93    auto handler = vm.jitStubs->existingCTIStub(handleExceptionGenerator, NoLockingNecessary);
     94    RELEASE_ASSERT(handler);
     95    patchBuffer.link(continuation, CodeLocationLabel(handler.retaggedCode<NoPtrTag>()));
     96    return FINALIZE_CODE(patchBuffer, JITThunkPtrTag, "popThunkStackPreservesAndHandleException");
     97}
     98
     99#endif // ENABLE(EXTRA_CTI_THUNKS)
    42100
    43101template<typename TagType>

  • trunk/Source/JavaScriptCore/jit/ThunkGenerators.h

    r253867 r277383  
    11/*
    2  * Copyright (C) 2010-2019 Apple Inc. All rights reserved.
     2 * Copyright (C) 2010-2021 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3535template<PtrTag> class MacroAssemblerCodeRef;
    3636class VM;
     37
     38#if ENABLE(EXTRA_CTI_THUNKS)
     39
     40MacroAssemblerCodeRef<JITThunkPtrTag> handleExceptionGenerator(VM&);
     41MacroAssemblerCodeRef<JITThunkPtrTag> handleExceptionWithCallFrameRollbackGenerator(VM&);
     42MacroAssemblerCodeRef<JITThunkPtrTag> popThunkStackPreservesAndHandleExceptionGenerator(VM&);
     43
     44#endif // ENABLE(EXTRA_CTI_THUNKS)
     45
    3746
    3847MacroAssemblerCodeRef<JITThunkPtrTag> throwExceptionFromCallSlowPathGenerator(VM&);

  • trunk/Source/JavaScriptCore/runtime/CommonSlowPaths.h

    r276609 r277383  
    11/*
    2  * Copyright (C) 2011-2019 Apple Inc. All rights reserved.
     2 * Copyright (C) 2011-2021 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3333#include "JSImmutableButterfly.h"
    3434#include "ScopedArguments.h"
    35 #include "SlowPathReturnType.h"
     35#include "SlowPathFunction.h"
    3636#include "StackAlignment.h"
    3737#include "VMInlines.h"
    … …  
    300300JSC_DECLARE_COMMON_SLOW_PATH(iterator_next_try_fast_wide32);
    301301
    302 using SlowPathFunction = SlowPathReturnType(JIT_OPERATION_ATTRIBUTES*)(CallFrame*, const Instruction*);
    303 
    304302} // namespace JSC

  • trunk/Source/JavaScriptCore/runtime/VM.cpp

    r277094 r277383  
    568568        getCTIInternalFunctionTrampolineFor(CodeForCall);
    569569        getCTIInternalFunctionTrampolineFor(CodeForConstruct);
    570     }
    571 #endif
     570
     571#if ENABLE(EXTRA_CTI_THUNKS)
     572        jitStubs->preinitializeExtraCTIThunks(*this);
     573#endif
     574    }
     575#endif // ENABLE(JIT)
    572576
    573577    if (Options::forceDebuggerBytecodeGeneration() || Options::alwaysUseShadowChicken())

  • trunk/Source/WTF/ChangeLog

    r277357 r277383  
     12021-05-12  Mark Lam  <mark.lam@apple.com>
     2
     3        Implement some common Baseline JIT slow paths using JIT thunks.
     4        https://bugs.webkit.org/show_bug.cgi?id=225682
     5
     6        Reviewed by Filip Pizlo.
     7
     8        Introduce ENABLE(EXTRA_CTI_THUNKS) flag to guard the use of these new thunks.
     9        Currently, the thunks are 64-bit only, and only supported for ARM64 and non-Windows
     10        X86_64.  The reason it is not supported for Windows as well is because Windows
     11        only has 4 argument registers.  In this patch, the thunks do not use that many
     12        registers yet, but there will be more thunks coming that will require the use
     13        of up to 6 argument registers.
     14
     15        * wtf/PlatformEnable.h:
     16
    1172021-05-11  Chris Dumez  <cdumez@apple.com>
    218

  • trunk/Source/WTF/wtf/PlatformEnable.h

    r276887 r277383  
    11/*
    2  * Copyright (C) 2006-2020 Apple Inc. All rights reserved.
     2 * Copyright (C) 2006-2021 Apple Inc. All rights reserved.
    33 * Copyright (C) 2007-2009 Torch Mobile, Inc.
    44 * Copyright (C) 2010, 2011 Research In Motion Limited. All rights reserved.
    … …  
    833833#endif
    834834
     835#if CPU(ARM64) || (CPU(X86_64) && !OS(WINDOWS))
     836/* The implementation of these thunks can use up to 6 argument registers, and
     837   make use of ARM64 like features. For now, we'll only support them on platforms
     838   that have 6 or more argument registers to use.
     839*/
     840#define ENABLE_EXTRA_CTI_THUNKS 1
     841#endif
     842
    835843#if !defined(ENABLE_BINDING_INTEGRITY) && !OS(WINDOWS)
    836844#define ENABLE_BINDING_INTEGRITY 1
Note: See TracChangeset for help on using the changeset viewer.