⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 287864 in webkit


Ignore:
Timestamp:
Jan 10, 2022, 5:36:00 PM (5 years ago)
Author:
sbarati@apple.com
Message:

Allow loop tier up to the Air tier
​https://bugs.webkit.org/show_bug.cgi?id=234587
<rdar://problem/86968638>

Reviewed by Yusuke Suzuki.

This patch adds loop tier up from LLInt -> Air. To implement this, we use
EntrySwitch to point at each loop header, making each loop an entrypoint.
This is unlike BBQ->OMG tier up, where we compile a special OSR entry OMG
callee. This seems like a good architecture for the Air tier, since we might end
up with slightly worse throughput, but we won't need a different compilation
for loops vs call entrypoints.

This patch also fixes a bug in Air's O0 register allocation where it
didn't properly account for all named registers in an instruction. There
was a silly bug where we asked each arg if it were a temp, instead of
asking the Inst for each of its temps, since an Arg can be an address
but still use temps.

  • b3/air/AirAllocateRegistersAndStackAndGenerateCode.cpp:

(JSC::B3::Air::GenerateAndAllocateRegisters::generate):

  • wasm/WasmAirIRGenerator.cpp:

(JSC::Wasm::AirIRGenerator::emitLoad):
(JSC::Wasm::AirIRGenerator::AirIRGenerator):
(JSC::Wasm::AirIRGenerator::finalizeEntrypoints):
(JSC::Wasm::AirIRGenerator::emitLoopTierUpCheck):
(JSC::Wasm::AirIRGenerator::addLoop):
(JSC::Wasm::parseAndCompileAir):

  • wasm/WasmB3IRGenerator.cpp:

(JSC::Wasm::B3IRGenerator::B3IRGenerator):
(JSC::Wasm::parseAndCompileB3):
(JSC::Wasm::parseAndCompile): Deleted.

  • wasm/WasmB3IRGenerator.h:
  • wasm/WasmBBQPlan.cpp:

(JSC::Wasm::BBQPlan::prepareImpl):
(JSC::Wasm::BBQPlan::work):
(JSC::Wasm::BBQPlan::compileFunction):
(JSC::Wasm::BBQPlan::didCompleteCompilation):
(JSC::Wasm::BBQPlan::initializeCallees):

  • wasm/WasmBBQPlan.h:
  • wasm/WasmCallee.h:
  • wasm/WasmCalleeGroup.h:
  • wasm/WasmFormat.h:
  • wasm/WasmIRGeneratorHelpers.h:

(JSC::Wasm::computeExceptionHandlerAndLoopEntrypointLocations):
(JSC::Wasm::computeExceptionHandlerLocations):

  • wasm/WasmLLIntPlan.cpp:

(JSC::Wasm::LLIntPlan::didCompleteCompilation):

  • wasm/WasmOMGPlan.cpp:

(JSC::Wasm::OMGPlan::work):

  • wasm/WasmOSREntryPlan.cpp:

(JSC::Wasm::OSREntryPlan::work):

  • wasm/WasmSlowPaths.cpp:

(JSC::LLInt::WASM_SLOW_PATH_DECL):

  • wasm/js/JSToWasm.cpp:

(JSC::Wasm::createJSToWasmWrapper):

Location:
trunk/Source/JavaScriptCore
Files:
16 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/JavaScriptCore/ChangeLog

    r287848 r287864  
     12022-01-10  Saam Barati  <sbarati@apple.com>
     2
     3        Allow loop tier up to the Air tier
     4        https://bugs.webkit.org/show_bug.cgi?id=234587
     5        <rdar://problem/86968638>
     6
     7        Reviewed by Yusuke Suzuki.
     8
     9        This patch adds loop tier up from LLInt -> Air. To implement this, we use
     10        EntrySwitch to point at each loop header, making each loop an entrypoint.
     11        This is unlike BBQ->OMG tier up, where we compile a special OSR entry OMG
     12        callee. This seems like a good architecture for the Air tier, since we might end
     13        up with slightly worse throughput, but we won't need a different compilation
     14        for loops vs call entrypoints.
     15       
     16        This patch also fixes a bug in Air's O0 register allocation where it
     17        didn't properly account for all named registers in an instruction. There
     18        was a silly bug where we asked each arg if it were a temp, instead of
     19        asking the Inst for each of its temps, since an Arg can be an address
     20        but still use temps.
     21
     22        * b3/air/AirAllocateRegistersAndStackAndGenerateCode.cpp:
     23        (JSC::B3::Air::GenerateAndAllocateRegisters::generate):
     24        * wasm/WasmAirIRGenerator.cpp:
     25        (JSC::Wasm::AirIRGenerator::emitLoad):
     26        (JSC::Wasm::AirIRGenerator::AirIRGenerator):
     27        (JSC::Wasm::AirIRGenerator::finalizeEntrypoints):
     28        (JSC::Wasm::AirIRGenerator::emitLoopTierUpCheck):
     29        (JSC::Wasm::AirIRGenerator::addLoop):
     30        (JSC::Wasm::parseAndCompileAir):
     31        * wasm/WasmB3IRGenerator.cpp:
     32        (JSC::Wasm::B3IRGenerator::B3IRGenerator):
     33        (JSC::Wasm::parseAndCompileB3):
     34        (JSC::Wasm::parseAndCompile): Deleted.
     35        * wasm/WasmB3IRGenerator.h:
     36        * wasm/WasmBBQPlan.cpp:
     37        (JSC::Wasm::BBQPlan::prepareImpl):
     38        (JSC::Wasm::BBQPlan::work):
     39        (JSC::Wasm::BBQPlan::compileFunction):
     40        (JSC::Wasm::BBQPlan::didCompleteCompilation):
     41        (JSC::Wasm::BBQPlan::initializeCallees):
     42        * wasm/WasmBBQPlan.h:
     43        * wasm/WasmCallee.h:
     44        * wasm/WasmCalleeGroup.h:
     45        * wasm/WasmFormat.h:
     46        * wasm/WasmIRGeneratorHelpers.h:
     47        (JSC::Wasm::computeExceptionHandlerAndLoopEntrypointLocations):
     48        (JSC::Wasm::computeExceptionHandlerLocations):
     49        * wasm/WasmLLIntPlan.cpp:
     50        (JSC::Wasm::LLIntPlan::didCompleteCompilation):
     51        * wasm/WasmOMGPlan.cpp:
     52        (JSC::Wasm::OMGPlan::work):
     53        * wasm/WasmOSREntryPlan.cpp:
     54        (JSC::Wasm::OSREntryPlan::work):
     55        * wasm/WasmSlowPaths.cpp:
     56        (JSC::LLInt::WASM_SLOW_PATH_DECL):
     57        * wasm/js/JSToWasm.cpp:
     58        (JSC::Wasm::createJSToWasmWrapper):
     59
    1602022-01-10  Alex Christensen  <achristensen@webkit.org>
    261

  • trunk/Source/JavaScriptCore/b3/air/AirAllocateRegistersAndStackAndGenerateCode.cpp

    r287806 r287864  
    582582            checkConsistency();
    583583
    584             inst.forEachArg([&] (Arg& arg, Arg::Role role, Bank, Width) {
    585                 if (!arg.isTmp())
    586                     return;
    587 
    588                 Tmp tmp = arg.tmp();
     584            inst.forEachTmp([&] (const Tmp& tmp, Arg::Role role, Bank, Width) {
    589585                if (tmp.isReg() && isDisallowedRegister(tmp.reg()))
    590586                    return;
    … …  
    596592                        m_namedDefdRegs.set(tmp.reg());
    597593                }
     594            });
     595
     596            inst.forEachArg([&] (Arg& arg, Arg::Role role, Bank, Width) {
     597                if (!arg.isTmp())
     598                    return;
     599
     600                Tmp tmp = arg.tmp();
    598601
    599602                // We convert any cold uses that are already in the stack to just point to
    … …  
    694697                    // we move to providing that arg in stack form. This will allow us to fully allocate
    695698                    // this inst when we rewind.
    696                     inst.forEachArg([&] (Arg& arg, Arg::Role, Bank, Width) {
    697                         if (!arg.isTmp())
     699                    inst.forEachTmpFast([&] (Tmp& tmp) {
     700                        if (!tmp.isReg())
    698701                            return;
    699 
    700                         Tmp tmp = arg.tmp();
    701                         if (tmp.isReg() && isDisallowedRegister(tmp.reg()))
     702                        if (isDisallowedRegister(tmp.reg()))
    702703                            return;
    703 
    704                         if (tmp.isReg()) {
    705                             Tmp originalTmp = allocationSnapshot[tmp.reg()];
    706                             if (originalTmp.isReg()) {
    707                                 ASSERT(tmp.reg() == originalTmp.reg());
    708                                 // This means this Inst referred to this reg directly. We leave these as is.
    709                                 return;
    710                             }
    711                             tmp = originalTmp;
    712                         }
    713 
    714                         if (!inst.admitsStack(arg)) {
    715                             arg = tmp;
     704                        Tmp originalTmp = allocationSnapshot[tmp.reg()];
     705                        if (originalTmp.isReg()) {
     706                            ASSERT(tmp.reg() == originalTmp.reg());
     707                            // This means this Inst referred to this reg directly. We leave these as is.
    716708                            return;
    717709                        }
    718 
    719                         auto& entry = m_map[tmp];
    720                         if (Reg reg = entry.reg)
    721                             spill(tmp, reg);
    722 
    723                         arg = Arg::addr(Tmp(GPRInfo::callFrameRegister), entry.spillSlot->offsetFromFP());
     710                        tmp = originalTmp;
     711                    });
     712                    inst.forEachArg([&] (Arg& arg, Arg::Role, Bank, Width) {
     713                        if (arg.isTmp() && !arg.tmp().isReg() && inst.admitsStack(arg)) {
     714                            Tmp tmp = arg.tmp();
     715                            auto& entry = m_map[tmp];
     716                            if (Reg reg = entry.reg)
     717                                spill(tmp, reg);
     718
     719                            arg = Arg::addr(Tmp(GPRInfo::callFrameRegister), entry.spillSlot->offsetFromFP());
     720                        }
    724721                    });
    725722

  • trunk/Source/JavaScriptCore/wasm/WasmAirIRGenerator.cpp

    r287801 r287864  
    323323    } while (0)
    324324
    325     AirIRGenerator(const ModuleInformation&, B3::Procedure&, InternalFunction*, Vector<UnlinkedWasmToWasmCall>&, MemoryMode, unsigned functionIndex, TierUpCount*, const Signature&);
     325    AirIRGenerator(const ModuleInformation&, B3::Procedure&, InternalFunction*, Vector<UnlinkedWasmToWasmCall>&, MemoryMode, unsigned functionIndex, TierUpCount*, const Signature&, unsigned& osrEntryScratchBufferSize);
    326326
    327327    void finalizeEntrypoints();
    … …  
    784784    }
    785785
    786     void emitLoad(Tmp base, size_t offset, TypedTmp result)
     786    void emitLoad(Tmp base, size_t offset, const TypedTmp& result)
    787787    {
    788788        emitLoad(moveOpForValueType(result.type()), toB3Type(result.type()), base, offset, result.tmp());
    … …  
    792792
    793793    void emitEntryTierUpCheck();
    794     void emitLoopTierUpCheck(uint32_t loopIndex, const Stack& enclosingStack, const Stack& newStack);
     794    void emitLoopTierUpCheck(uint32_t loopIndex, const Vector<TypedTmp>& liveValues);
    795795
    796796    void emitWriteBarrierForJSWrapper();
    … …  
    885885    StackMaps m_stackmaps;
    886886    Vector<UnlinkedHandlerInfo> m_exceptionHandlers;
     887
     888    Vector<std::pair<BasicBlock*, Vector<TypedTmp>>> m_loopEntryVariableData;
     889    unsigned& m_osrEntryScratchBufferSize;
    887890};
    888891
    … …  
    930933}
    931934
    932 AirIRGenerator::AirIRGenerator(const ModuleInformation& info, B3::Procedure& procedure, InternalFunction* compilation, Vector<UnlinkedWasmToWasmCall>& unlinkedWasmToWasmCalls, MemoryMode mode, unsigned functionIndex, TierUpCount* tierUp, const Signature& signature)
     935AirIRGenerator::AirIRGenerator(const ModuleInformation& info, B3::Procedure& procedure, InternalFunction* compilation, Vector<UnlinkedWasmToWasmCall>& unlinkedWasmToWasmCalls, MemoryMode mode, unsigned functionIndex, TierUpCount* tierUp, const Signature& signature, unsigned& osrEntryScratchBufferSize)
    933936    : m_info(info)
    934937    , m_mode(mode)
    … …  
    939942    , m_unlinkedWasmToWasmCalls(unlinkedWasmToWasmCalls)
    940943    , m_numImportFunctions(info.importFunctionCount())
     944    , m_osrEntryScratchBufferSize(osrEntryScratchBufferSize)
    941945{
    942946    m_currentBlock = m_code.addBlock();
    … …  
    968972            auto moveLocation = jit.moveWithPatch(MacroAssembler::TrustedImmPtr(nullptr), calleeGPR);
    969973            jit.addLinkTask([compilation, moveLocation] (LinkBuffer& linkBuffer) {
    970                 compilation->calleeMoveLocation = linkBuffer.locationOf<WasmEntryPtrTag>(moveLocation);
     974                compilation->calleeMoveLocations.append(linkBuffer.locationOf<WasmEntryPtrTag>(moveLocation));
    971975            });
    972976            jit.emitPutToCallFrameHeader(calleeGPR, CallFrameSlot::callee);
    … …  
    10731077void AirIRGenerator::finalizeEntrypoints()
    10741078{
    1075     unsigned numEntrypoints = 1 + m_catchEntrypoints.size();
     1079    unsigned numEntrypoints = Checked<unsigned>(1) + m_catchEntrypoints.size() + m_loopEntryVariableData.size();
    10761080    m_proc.setNumEntrypoints(numEntrypoints);
    10771081    m_code.setPrologueForEntrypoint(0, Ref<B3::Air::PrologueGenerator>(*m_prologueGenerator));
     1082    for (unsigned i = 1 + m_catchEntrypoints.size(); i < numEntrypoints; ++i)
     1083        m_code.setPrologueForEntrypoint(i, Ref<B3::Air::PrologueGenerator>(*m_prologueGenerator));
    10781084
    10791085    if (m_catchEntrypoints.size()) {
    … …  
    10951101    successors.append(m_mainEntrypointStart);
    10961102    successors.appendVector(m_catchEntrypoints);
     1103
     1104    for (auto& pair : m_loopEntryVariableData) {
     1105        BasicBlock* loopBody = pair.first;
     1106        BasicBlock* entry = m_code.addBlock();
     1107        successors.append(entry);
     1108        m_currentBlock = entry;
     1109
     1110        auto& temps = pair.second;
     1111        m_osrEntryScratchBufferSize = std::max(m_osrEntryScratchBufferSize, static_cast<unsigned>(temps.size()));
     1112        Tmp basePtr = Tmp(GPRInfo::argumentGPR0);
     1113
     1114        for (size_t i = 0; i < temps.size(); ++i) {
     1115            size_t offset = static_cast<size_t>(i) * sizeof(uint64_t);
     1116            emitLoad(basePtr, offset, temps[i]);
     1117        }
     1118
     1119        append(Jump);
     1120        entry->setSuccessors(loopBody);
     1121    }
    10971122
    10981123    RELEASE_ASSERT(numEntrypoints == successors.size());
    … …  
    30343059}
    30353060
    3036 void AirIRGenerator::emitLoopTierUpCheck(uint32_t loopIndex, const Stack& enclosingStack, const Stack& newStack)
     3061void AirIRGenerator::emitLoopTierUpCheck(uint32_t loopIndex, const Vector<TypedTmp>& liveValues)
    30373062{
    30383063    uint32_t outerLoopIndex = this->outerLoopIndex();
    … …  
    30633088    Vector<ConstrainedTmp> patchArgs;
    30643089    patchArgs.append(countdownPtr);
    3065 
    3066     forEachLiveValue([&] (Tmp tmp) {
    3067         patchArgs.append(ConstrainedTmp(tmp, B3::ValueRep::ColdAny));
    3068     });
    3069     for (TypedExpression value : enclosingStack)
    3070         patchArgs.append(ConstrainedTmp(value.value(), B3::ValueRep::ColdAny));
    3071     for (TypedExpression value : newStack)
    3072         patchArgs.append(ConstrainedTmp(value.value(), B3::ValueRep::ColdAny));
     3090    for (const TypedTmp& tmp : liveValues)
     3091        patchArgs.append(ConstrainedTmp(tmp.tmp(), B3::ValueRep::ColdAny));
    30733092
    30743093    TierUpCount::TriggerReason* forceEntryTrigger = &(m_tierUp->osrEntryTriggers().last());
    … …  
    31113130auto AirIRGenerator::addLoop(BlockSignature signature, Stack& enclosingStack, ControlType& block, Stack& newStack, uint32_t loopIndex) -> PartialResult
    31123131{
     3132    RELEASE_ASSERT(loopIndex == m_loopEntryVariableData.size());
     3133
    31133134    BasicBlock* body = m_code.addBlock();
    31143135    BasicBlock* continuation = m_code.addBlock();
    31153136
    31163137    splitStack(signature, enclosingStack, newStack);
     3138
     3139    Vector<TypedTmp> liveValues;
     3140    forEachLiveValue([&] (TypedTmp tmp) {
     3141        liveValues.append(tmp);
     3142    });
     3143    for (auto variable : enclosingStack)
     3144        liveValues.append(variable);
     3145    for (auto variable : newStack)
     3146        liveValues.append(variable);
     3147
    31173148    ResultList results;
    31183149    results.reserveInitialCapacity(newStack.size());
    … …  
    31253156
    31263157    m_currentBlock = body;
    3127     emitLoopTierUpCheck(loopIndex, enclosingStack, newStack);
     3158    emitLoopTierUpCheck(loopIndex, liveValues);
     3159
     3160    m_loopEntryVariableData.append(std::pair<BasicBlock*, Vector<TypedTmp>>(body, WTFMove(liveValues)));
    31283161
    31293162    return { };
    … …  
    39003933    procedure.setOptLevel(Options::webAssemblyBBQAirOptimizationLevel());
    39013934
    3902     AirIRGenerator irGenerator(info, procedure, result.get(), unlinkedWasmToWasmCalls, mode, functionIndex, tierUp, signature);
     3935    AirIRGenerator irGenerator(info, procedure, result.get(), unlinkedWasmToWasmCalls, mode, functionIndex, tierUp, signature, result->osrEntryScratchBufferSize);
    39033936    FunctionParser<AirIRGenerator> parser(irGenerator, function.data.data(), function.data.size(), signature, info);
    39043937    WASM_FAIL_IF_HELPER_FAILS(parser.parse());

  • trunk/Source/JavaScriptCore/wasm/WasmB3IRGenerator.cpp

    r287801 r287864  
    678678
    679679    {
    680         auto* calleeMoveLocation = &compilation->calleeMoveLocation;
     680        auto* calleeMoveLocations = &compilation->calleeMoveLocations;
    681681        static_assert(CallFrameSlot::codeBlock * sizeof(Register) < WasmCallingConvention::headerSizeInBytes, "We rely on this here for now.");
    682682        static_assert(CallFrameSlot::callee * sizeof(Register) < WasmCallingConvention::headerSizeInBytes, "We rely on this here for now.");
    … …  
    688688                GPRReg result = params[0].gpr();
    689689                MacroAssembler::DataLabelPtr moveLocation = jit.moveWithPatch(MacroAssembler::TrustedImmPtr(nullptr), result);
    690                 jit.addLinkTask([calleeMoveLocation, moveLocation] (LinkBuffer& linkBuffer) {
    691                     *calleeMoveLocation = linkBuffer.locationOf<WasmEntryPtrTag>(moveLocation);
     690                jit.addLinkTask([calleeMoveLocations, moveLocation] (LinkBuffer& linkBuffer) {
     691                    calleeMoveLocations->append(linkBuffer.locationOf<WasmEntryPtrTag>(moveLocation));
    692692                });
    693693            });
    … …  
    31393139}
    31403140
    3141 Expected<std::unique_ptr<InternalFunction>, String> parseAndCompile(CompilationContext& compilationContext, const FunctionData& function, const Signature& signature, Vector<UnlinkedWasmToWasmCall>& unlinkedWasmToWasmCalls, unsigned& osrEntryScratchBufferSize, const ModuleInformation& info, MemoryMode mode, CompilationMode compilationMode, uint32_t functionIndex, uint32_t loopIndexForOSREntry, TierUpCount* tierUp)
     3141Expected<std::unique_ptr<InternalFunction>, String> parseAndCompileB3(CompilationContext& compilationContext, const FunctionData& function, const Signature& signature, Vector<UnlinkedWasmToWasmCall>& unlinkedWasmToWasmCalls, const ModuleInformation& info, MemoryMode mode, CompilationMode compilationMode, uint32_t functionIndex, uint32_t loopIndexForOSREntry, TierUpCount* tierUp)
    31423142{
    31433143    auto result = makeUnique<InternalFunction>();
    … …  
    31763176        : Options::webAssemblyOMGOptimizationLevel());
    31773177
    3178     B3IRGenerator irGenerator(info, procedure, result.get(), unlinkedWasmToWasmCalls, osrEntryScratchBufferSize, mode, compilationMode, functionIndex, loopIndexForOSREntry, tierUp);
     3178    B3IRGenerator irGenerator(info, procedure, result.get(), unlinkedWasmToWasmCalls, result->osrEntryScratchBufferSize, mode, compilationMode, functionIndex, loopIndexForOSREntry, tierUp);
    31793179    FunctionParser<B3IRGenerator> parser(irGenerator, function.data.data(), function.data.size(), signature, info);
    31803180    WASM_FAIL_IF_HELPER_FAILS(parser.parse());

  • trunk/Source/JavaScriptCore/wasm/WasmB3IRGenerator.h

    r287801 r287864  
    5858};
    5959
    60 Expected<std::unique_ptr<InternalFunction>, String> parseAndCompile(CompilationContext&, const FunctionData&, const Signature&, Vector<UnlinkedWasmToWasmCall>&, unsigned& osrEntryScratchBufferSize, const ModuleInformation&, MemoryMode, CompilationMode, uint32_t functionIndex, uint32_t loopIndexForOSREntry, TierUpCount* = nullptr);
     60Expected<std::unique_ptr<InternalFunction>, String> parseAndCompileB3(CompilationContext&, const FunctionData&, const Signature&, Vector<UnlinkedWasmToWasmCall>&, const ModuleInformation&, MemoryMode, CompilationMode, uint32_t functionIndex, uint32_t loopIndexForOSREntry, TierUpCount* = nullptr);
    6161
    6262void computePCToCodeOriginMap(CompilationContext&, LinkBuffer&);

  • trunk/Source/JavaScriptCore/wasm/WasmBBQPlan.cpp

    r287801 r287864  
    6565    if (!tryReserveCapacity(m_wasmInternalFunctions, functions.size(), " WebAssembly functions")
    6666        || !tryReserveCapacity(m_compilationContexts, functions.size(), " compilation contexts")
    67         || !tryReserveCapacity(m_tierUpCounts, functions.size(), " tier-up counts"))
     67        || !tryReserveCapacity(m_tierUpCounts, functions.size(), " tier-up counts")
     68        || !tryReserveCapacity(m_allLoopEntrypoints, functions.size(), " loop entrypoints"))
    6869        return false;
    6970
    … …  
    7273    m_compilationContexts.resize(functions.size());
    7374    m_tierUpCounts.resize(functions.size());
     75    m_allLoopEntrypoints.resize(functions.size());
    7476
    7577    return true;
    … …  
    114116
    115117    Vector<CodeLocationLabel<ExceptionHandlerPtrTag>> exceptionHandlerLocations;
    116     computeExceptionHandlerLocations(exceptionHandlerLocations, function.get(), context, linkBuffer);
     118    Vector<CodeLocationLabel<WasmEntryPtrTag>> loopEntrypointLocations;
     119    computeExceptionHandlerAndLoopEntrypointLocations(exceptionHandlerLocations, loopEntrypointLocations, function.get(), context, linkBuffer);
    117120
    118121    computePCToCodeOriginMap(context, linkBuffer);
    … …  
    127130    MacroAssemblerCodePtr<WasmEntryPtrTag> entrypoint;
    128131    {
    129         Ref<BBQCallee> callee = BBQCallee::create(WTFMove(function->entrypoint), functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace), WTFMove(tierUp), WTFMove(unlinkedWasmToWasmCalls), WTFMove(function->stackmaps), WTFMove(function->exceptionHandlers), WTFMove(exceptionHandlerLocations));
    130         MacroAssembler::repatchPointer(function->calleeMoveLocation, CalleeBits::boxWasm(callee.ptr()));
     132        Ref<BBQCallee> callee = BBQCallee::create(WTFMove(function->entrypoint), functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace), WTFMove(tierUp), WTFMove(unlinkedWasmToWasmCalls), WTFMove(function->stackmaps), WTFMove(function->exceptionHandlers), WTFMove(exceptionHandlerLocations), WTFMove(loopEntrypointLocations), function->osrEntryScratchBufferSize);
     133        for (auto& moveLocation : function->calleeMoveLocations)
     134            MacroAssembler::repatchPointer(moveLocation, CalleeBits::boxWasm(callee.ptr()));
    131135        entrypoint = callee->entrypoint();
    132136
    … …  
    197201    ASSERT_UNUSED(functionIndexSpace, m_moduleInformation->signatureIndexFromFunctionIndexSpace(functionIndexSpace) == signatureIndex);
    198202    Expected<std::unique_ptr<InternalFunction>, String> parseAndCompileResult;
    199     unsigned osrEntryScratchBufferSize = 0;
    200203
    201204    // FIXME: Some webpages use very large Wasm module, and it exhausts all executable memory in ARM64 devices since the size of executable memory region is only limited to 128MB.
    … …  
    209212
    210213    if (forceUsingB3)
    211         parseAndCompileResult = parseAndCompile(context, function, signature, unlinkedWasmToWasmCalls, osrEntryScratchBufferSize, m_moduleInformation.get(), m_mode, CompilationMode::BBQMode, functionIndex, UINT32_MAX, tierUp);
     214        parseAndCompileResult = parseAndCompileB3(context, function, signature, unlinkedWasmToWasmCalls, m_moduleInformation.get(), m_mode, CompilationMode::BBQMode, functionIndex, UINT32_MAX, tierUp);
    212215    else
    213216        parseAndCompileResult = parseAndCompileAir(context, function, signature, unlinkedWasmToWasmCalls, m_moduleInformation.get(), m_mode, functionIndex, tierUp);
    … …  
    242245            }
    243246
    244             computeExceptionHandlerLocations(m_exceptionHandlerLocations[functionIndex], function, context, linkBuffer);
     247            computeExceptionHandlerAndLoopEntrypointLocations(m_exceptionHandlerLocations[functionIndex], m_allLoopEntrypoints[functionIndex], function, context, linkBuffer);
    245248
    246249            computePCToCodeOriginMap(context, linkBuffer);
    … …  
    285288        if (auto embedderToWasmFunction = m_embedderToWasmInternalFunctions.get(internalFunctionIndex)) {
    286289            embedderEntrypointCallee = EmbedderEntrypointCallee::create(WTFMove(embedderToWasmFunction->entrypoint));
    287             MacroAssembler::repatchPointer(embedderToWasmFunction->calleeMoveLocation, CalleeBits::boxWasm(embedderEntrypointCallee.get()));
     290            for (auto& moveLocation : embedderToWasmFunction->calleeMoveLocations)
     291                MacroAssembler::repatchPointer(moveLocation, CalleeBits::boxWasm(embedderEntrypointCallee.get()));
    288292        }
    289293
    290294        InternalFunction* function = m_wasmInternalFunctions[internalFunctionIndex].get();
    291295        size_t functionIndexSpace = internalFunctionIndex + m_moduleInformation->importFunctionCount();
    292         Ref<BBQCallee> wasmEntrypointCallee = BBQCallee::create(WTFMove(function->entrypoint), functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace), WTFMove(m_tierUpCounts[internalFunctionIndex]), WTFMove(m_unlinkedWasmToWasmCalls[internalFunctionIndex]), WTFMove(function->stackmaps), WTFMove(function->exceptionHandlers), WTFMove(m_exceptionHandlerLocations[internalFunctionIndex]));
    293         MacroAssembler::repatchPointer(function->calleeMoveLocation, CalleeBits::boxWasm(wasmEntrypointCallee.ptr()));
     296        Ref<BBQCallee> wasmEntrypointCallee = BBQCallee::create(WTFMove(function->entrypoint), functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace), WTFMove(m_tierUpCounts[internalFunctionIndex]), WTFMove(m_unlinkedWasmToWasmCalls[internalFunctionIndex]), WTFMove(function->stackmaps), WTFMove(function->exceptionHandlers), WTFMove(m_exceptionHandlerLocations[internalFunctionIndex]), WTFMove(m_allLoopEntrypoints[internalFunctionIndex]), function->osrEntryScratchBufferSize);
     297
     298        for (auto& moveLocation : function->calleeMoveLocations)
     299            MacroAssembler::repatchPointer(moveLocation, CalleeBits::boxWasm(wasmEntrypointCallee.ptr()));
    294300
    295301        if (m_compilationContexts[internalFunctionIndex].pcToCodeOriginMap)

  • trunk/Source/JavaScriptCore/wasm/WasmBBQPlan.h

    r287122 r287864  
    8888    Vector<CompilationContext> m_compilationContexts;
    8989    Vector<std::unique_ptr<TierUpCount>> m_tierUpCounts;
     90    Vector<Vector<CodeLocationLabel<WasmEntryPtrTag>>> m_allLoopEntrypoints;
    9091
    9192    RefPtr<CalleeGroup> m_calleeGroup { nullptr };

  • trunk/Source/JavaScriptCore/wasm/WasmCallee.h

    r287459 r287864  
    173173class BBQCallee final : public OptimizingJITCallee {
    174174public:
    175     static Ref<BBQCallee> create(Wasm::Entrypoint&& entrypoint, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, std::unique_ptr<TierUpCount>&& tierUpCount, Vector<UnlinkedWasmToWasmCall>&& unlinkedCalls, StackMaps&& stackmaps, Vector<UnlinkedHandlerInfo>&& exceptionHandlers, Vector<CodeLocationLabel<ExceptionHandlerPtrTag>>&& exceptionHandlerLocations)
    176     {
    177         return adoptRef(*new BBQCallee(WTFMove(entrypoint), index, WTFMove(name), WTFMove(tierUpCount), WTFMove(unlinkedCalls), WTFMove(stackmaps), WTFMove(exceptionHandlers), WTFMove(exceptionHandlerLocations)));
     175    static Ref<BBQCallee> create(Wasm::Entrypoint&& entrypoint, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, std::unique_ptr<TierUpCount>&& tierUpCount, Vector<UnlinkedWasmToWasmCall>&& unlinkedCalls, StackMaps&& stackmaps, Vector<UnlinkedHandlerInfo>&& exceptionHandlers, Vector<CodeLocationLabel<ExceptionHandlerPtrTag>>&& exceptionHandlerLocations, Vector<CodeLocationLabel<WasmEntryPtrTag>>&& loopEntrypoints, unsigned osrEntryScratchBufferSize)
     176    {
     177        return adoptRef(*new BBQCallee(WTFMove(entrypoint), index, WTFMove(name), WTFMove(tierUpCount), WTFMove(unlinkedCalls), WTFMove(stackmaps), WTFMove(exceptionHandlers), WTFMove(exceptionHandlerLocations), WTFMove(loopEntrypoints), osrEntryScratchBufferSize));
    178178    }
    179179
    … …  
    195195    TierUpCount* tierUpCount() { return m_tierUpCount.get(); }
    196196
    197 private:
    198     BBQCallee(Wasm::Entrypoint&& entrypoint, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, std::unique_ptr<TierUpCount>&& tierUpCount, Vector<UnlinkedWasmToWasmCall>&& unlinkedCalls, StackMaps&& stackmaps, Vector<UnlinkedHandlerInfo>&& exceptionHandlers, Vector<CodeLocationLabel<ExceptionHandlerPtrTag>>&& exceptionHandlerLocations)
     197    const Vector<CodeLocationLabel<WasmEntryPtrTag>>& loopEntrypoints() { return m_loopEntrypoints; }
     198
     199    unsigned osrEntryScratchBufferSize() const { return m_osrEntryScratchBufferSize; }
     200
     201private:
     202    BBQCallee(Wasm::Entrypoint&& entrypoint, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, std::unique_ptr<TierUpCount>&& tierUpCount, Vector<UnlinkedWasmToWasmCall>&& unlinkedCalls, StackMaps&& stackmaps, Vector<UnlinkedHandlerInfo>&& exceptionHandlers, Vector<CodeLocationLabel<ExceptionHandlerPtrTag>>&& exceptionHandlerLocations, Vector<CodeLocationLabel<WasmEntryPtrTag>>&& loopEntrypoints, unsigned osrEntryScratchBufferSize)
    199203        : OptimizingJITCallee(Wasm::CompilationMode::BBQMode, WTFMove(entrypoint), index, WTFMove(name), WTFMove(unlinkedCalls), WTFMove(stackmaps), WTFMove(exceptionHandlers), WTFMove(exceptionHandlerLocations))
    200204        , m_tierUpCount(WTFMove(tierUpCount))
     205        , m_loopEntrypoints(WTFMove(loopEntrypoints))
     206        , m_osrEntryScratchBufferSize(osrEntryScratchBufferSize)
    201207    {
    202208    }
    … …  
    205211    RefPtr<OMGCallee> m_replacement;
    206212    std::unique_ptr<TierUpCount> m_tierUpCount;
     213    Vector<CodeLocationLabel<WasmEntryPtrTag>> m_loopEntrypoints;
     214    unsigned m_osrEntryScratchBufferSize { 0 };
    207215    bool m_didStartCompilingOSREntryCallee { false };
    208216};
    … …  
    268276#if ENABLE(WEBASSEMBLY_B3JIT)
    269277    JITCallee* replacement(MemoryMode mode) { return m_replacements[static_cast<uint8_t>(mode)].get(); }
    270     void setReplacement(Ref<JITCallee>&& replacement, MemoryMode mode)
     278    void setReplacement(Ref<OptimizingJITCallee>&& replacement, MemoryMode mode)
    271279    {
    272280        m_replacements[static_cast<uint8_t>(mode)] = WTFMove(replacement);
    … …  
    303311
    304312#if ENABLE(WEBASSEMBLY_B3JIT)
    305     RefPtr<JITCallee> m_replacements[Wasm::NumberOfMemoryModes];
     313    RefPtr<OptimizingJITCallee> m_replacements[Wasm::NumberOfMemoryModes];
    306314    RefPtr<OSREntryCallee> m_osrEntryCallees[Wasm::NumberOfMemoryModes];
    307315#endif

  • trunk/Source/JavaScriptCore/wasm/WasmCalleeGroup.h

    r287379 r287864  
    184184    std::atomic<bool> m_compilationFinished { false };
    185185    String m_errorMessage;
     186public:
    186187    Lock m_lock;
    187188};

  • trunk/Source/JavaScriptCore/wasm/WasmFormat.h

    r285065 r287864  
    444444struct InternalFunction {
    445445    WTF_MAKE_STRUCT_FAST_ALLOCATED;
    446     CodeLocationDataLabelPtr<WasmEntryPtrTag> calleeMoveLocation;
     446    Vector<CodeLocationDataLabelPtr<WasmEntryPtrTag>> calleeMoveLocations;
    447447    StackMaps stackmaps;
    448448    Vector<UnlinkedHandlerInfo> exceptionHandlers;
    449449    Entrypoint entrypoint;
     450    unsigned osrEntryScratchBufferSize { 0 };
    450451};
    451452

  • trunk/Source/JavaScriptCore/wasm/WasmIRGeneratorHelpers.h

    r287801 r287864  
    6464
    6565
    66 static inline void computeExceptionHandlerLocations(Vector<CodeLocationLabel<ExceptionHandlerPtrTag>>& handlers, const InternalFunction* function, const CompilationContext& context, LinkBuffer& linkBuffer)
     66static inline void computeExceptionHandlerAndLoopEntrypointLocations(Vector<CodeLocationLabel<ExceptionHandlerPtrTag>>& handlers, Vector<CodeLocationLabel<WasmEntryPtrTag>>& loopEntrypoints, const InternalFunction* function, const CompilationContext& context, LinkBuffer& linkBuffer)
    6767{
    6868    if (!context.procedure)
    6969        return;
    7070
    71     unsigned entrypointIndex = 0;
     71    unsigned entrypointIndex = 1;
    7272    unsigned numEntrypoints = context.procedure->numEntrypoints();
    7373    for (const UnlinkedHandlerInfo& handlerInfo : function->exceptionHandlers) {
    74         RELEASE_ASSERT(entrypointIndex < numEntrypoints);
    7574        if (handlerInfo.m_type == HandlerType::Delegate) {
    7675            handlers.append({ });
    … …  
    7877        }
    7978
     79        RELEASE_ASSERT(entrypointIndex < numEntrypoints);
     80        handlers.append(linkBuffer.locationOf<ExceptionHandlerPtrTag>(context.procedure->code().entrypointLabel(entrypointIndex)));
    8081        ++entrypointIndex;
    81         handlers.append(linkBuffer.locationOf<ExceptionHandlerPtrTag>(context.procedure->code().entrypointLabel(entrypointIndex)));
    82     }
    83     RELEASE_ASSERT(entrypointIndex == numEntrypoints - 1);
     82    }
     83
     84    for (; entrypointIndex < numEntrypoints; ++entrypointIndex)
     85        loopEntrypoints.append(linkBuffer.locationOf<WasmEntryPtrTag>(context.procedure->code().entrypointLabel(entrypointIndex)));
     86}
     87
     88static inline void computeExceptionHandlerLocations(Vector<CodeLocationLabel<ExceptionHandlerPtrTag>>& handlers, const InternalFunction* function, const CompilationContext& context, LinkBuffer& linkBuffer)
     89{
     90    Vector<CodeLocationLabel<WasmEntryPtrTag>> ignored;
     91    computeExceptionHandlerAndLoopEntrypointLocations(handlers, ignored, function, context, linkBuffer);
    8492}
    8593

  • trunk/Source/JavaScriptCore/wasm/WasmLLIntPlan.cpp

    r287513 r287864  
    169169            // FIXME: remove this repatchPointer - just pass in the callee directly
    170170            // https://bugs.webkit.org/show_bug.cgi?id=166462
    171             if (function->calleeMoveLocation)
    172                 MacroAssembler::repatchPointer(function->calleeMoveLocation, CalleeBits::boxWasm(callee.ptr()));
     171            for (auto& moveLocation : function->calleeMoveLocations)
     172                MacroAssembler::repatchPointer(moveLocation, CalleeBits::boxWasm(callee.ptr()));
    173173
    174174            auto result = m_embedderCallees.add(functionIndex, WTFMove(callee));

  • trunk/Source/JavaScriptCore/wasm/WasmOMGPlan.cpp

    r287801 r287864  
    7373
    7474    Vector<UnlinkedWasmToWasmCall> unlinkedCalls;
    75     unsigned osrEntryScratchBufferSize;
    7675    CompilationContext context;
    77     auto parseAndCompileResult = parseAndCompile(context, function, signature, unlinkedCalls, osrEntryScratchBufferSize, m_moduleInformation.get(), m_mode, CompilationMode::OMGMode, m_functionIndex, UINT32_MAX);
     76    auto parseAndCompileResult = parseAndCompileB3(context, function, signature, unlinkedCalls, m_moduleInformation.get(), m_mode, CompilationMode::OMGMode, m_functionIndex, UINT32_MAX);
    7877
    7978    if (UNLIKELY(!parseAndCompileResult)) {
    … …  
    107106        ASSERT(m_calleeGroup.ptr() == m_module->calleeGroupFor(mode()));
    108107        Ref<OMGCallee> callee = OMGCallee::create(WTFMove(omgEntrypoint), functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace), WTFMove(unlinkedCalls), WTFMove(internalFunction->stackmaps), WTFMove(internalFunction->exceptionHandlers), WTFMove(exceptionHandlerLocations));
    109         MacroAssembler::repatchPointer(internalFunction->calleeMoveLocation, CalleeBits::boxWasm(callee.ptr()));
     108        for (auto& moveLocation : internalFunction->calleeMoveLocations)
     109            MacroAssembler::repatchPointer(moveLocation, CalleeBits::boxWasm(callee.ptr()));
    110110        entrypoint = callee->entrypoint();
    111111

  • trunk/Source/JavaScriptCore/wasm/WasmOSREntryPlan.cpp

    r287801 r287864  
    7878    Vector<UnlinkedWasmToWasmCall> unlinkedCalls;
    7979    CompilationContext context;
    80     unsigned osrEntryScratchBufferSize = 0;
    81     auto parseAndCompileResult = parseAndCompile(context, function, signature, unlinkedCalls, osrEntryScratchBufferSize, m_moduleInformation.get(), m_mode, targetCompilationMode, m_functionIndex, m_loopIndex);
     80    auto parseAndCompileResult = parseAndCompileB3(context, function, signature, unlinkedCalls, m_moduleInformation.get(), m_mode, targetCompilationMode, m_functionIndex, m_loopIndex);
    8281
    8382    if (UNLIKELY(!parseAndCompileResult)) {
    … …  
    106105
    107106    ASSERT(m_calleeGroup.ptr() == m_module->calleeGroupFor(mode()));
    108     Ref<OSREntryCallee> callee = OSREntryCallee::create(targetCompilationMode, WTFMove(omgEntrypoint), functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace), osrEntryScratchBufferSize, m_loopIndex, WTFMove(unlinkedCalls), WTFMove(internalFunction->stackmaps), WTFMove(internalFunction->exceptionHandlers), WTFMove(exceptionHandlerLocations));
     107    Ref<OSREntryCallee> callee = OSREntryCallee::create(targetCompilationMode, WTFMove(omgEntrypoint), functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace), internalFunction->osrEntryScratchBufferSize, m_loopIndex, WTFMove(unlinkedCalls), WTFMove(internalFunction->stackmaps), WTFMove(internalFunction->exceptionHandlers), WTFMove(exceptionHandlerLocations));
    109108    {
    110         MacroAssembler::repatchPointer(internalFunction->calleeMoveLocation, CalleeBits::boxWasm(callee.ptr()));
     109        for (auto& moveLocation : internalFunction->calleeMoveLocations)
     110            MacroAssembler::repatchPointer(moveLocation, CalleeBits::boxWasm(callee.ptr()));
    111111
    112112        Locker locker { m_calleeGroup->m_lock };

  • trunk/Source/JavaScriptCore/wasm/WasmSlowPaths.cpp

    r287459 r287864  
    181181    dataLogLnIf(Options::verboseOSR(), *callee, ": Entered loop_osr with tierUpCounter = ", callee->tierUpCounter());
    182182
    183     unsigned loopOSREntryBytecodeOffset = callee->bytecodeOffset(pc);
    184     const auto& osrEntryData = tierUpCounter.osrEntryDataForLoop(loopOSREntryBytecodeOffset);
    185 
    186183    if (!tierUpCounter.checkIfOptimizationThresholdReached()) {
    187184        dataLogLnIf(Options::verboseOSR(), "    JIT threshold should be lifted.");
    … …  
    189186    }
    190187
    191     const auto doOSREntry = [&](Wasm::OSREntryCallee* osrEntryCallee) {
    192         if (osrEntryCallee->loopIndex() != osrEntryData.loopIndex)
     188    unsigned loopOSREntryBytecodeOffset = callee->bytecodeOffset(pc);
     189    const auto& osrEntryData = tierUpCounter.osrEntryDataForLoop(loopOSREntryBytecodeOffset);
     190
     191    if (Options::wasmLLIntTiersUpToBBQ()) {
     192        if (!jitCompileAndSetHeuristics(callee, instance))
    193193            WASM_RETURN_TWO(nullptr, nullptr);
    194194
    195         size_t osrEntryScratchBufferSize = osrEntryCallee->osrEntryScratchBufferSize();
    196         RELEASE_ASSERT(osrEntryScratchBufferSize == osrEntryData.values.size());
     195        Wasm::BBQCallee* bbqCallee;
     196        {
     197            Locker locker { instance->calleeGroup()->m_lock };
     198            bbqCallee = instance->calleeGroup()->bbqCallee(locker, callee->functionIndex());
     199        }
     200        RELEASE_ASSERT(bbqCallee);
     201
     202        size_t osrEntryScratchBufferSize = bbqCallee->osrEntryScratchBufferSize();
     203        RELEASE_ASSERT(osrEntryScratchBufferSize >= osrEntryData.values.size());
    197204        uint64_t* buffer = instance->context()->scratchBufferForSize(osrEntryScratchBufferSize);
    198205        if (!buffer)
    199206            WASM_RETURN_TWO(nullptr, nullptr);
     207        RELEASE_ASSERT(osrEntryData.loopIndex < bbqCallee->loopEntrypoints().size());
    200208
    201209        uint32_t index = 0;
    … …  
    203211            buffer[index++] = READ(reg).encodedJSValue();
    204212
    205         WASM_RETURN_TWO(buffer, osrEntryCallee->entrypoint().executableAddress());
    206     };
    207 
    208     if (auto* osrEntryCallee = callee->osrEntryCallee(instance->memory()->mode()))
    209         return doOSREntry(osrEntryCallee);
    210 
    211     bool compile = false;
    212     {
    213         Locker locker { tierUpCounter.m_lock };
    214         switch (tierUpCounter.m_loopCompilationStatus) {
    215         case Wasm::LLIntTierUpCounter::CompilationStatus::NotCompiled:
    216             compile = true;
    217             tierUpCounter.m_loopCompilationStatus = Wasm::LLIntTierUpCounter::CompilationStatus::Compiling;
    218             break;
    219         case Wasm::LLIntTierUpCounter::CompilationStatus::Compiling:
    220             tierUpCounter.optimizeAfterWarmUp();
    221             break;
    222         case Wasm::LLIntTierUpCounter::CompilationStatus::Compiled:
    223             break;
     213        WASM_RETURN_TWO(buffer, bbqCallee->loopEntrypoints()[osrEntryData.loopIndex].executableAddress());
     214    } else {
     215        const auto doOSREntry = [&](Wasm::OSREntryCallee* osrEntryCallee) {
     216            if (osrEntryCallee->loopIndex() != osrEntryData.loopIndex)
     217                WASM_RETURN_TWO(nullptr, nullptr);
     218
     219            size_t osrEntryScratchBufferSize = osrEntryCallee->osrEntryScratchBufferSize();
     220            RELEASE_ASSERT(osrEntryScratchBufferSize == osrEntryData.values.size());
     221            uint64_t* buffer = instance->context()->scratchBufferForSize(osrEntryScratchBufferSize);
     222            if (!buffer)
     223                WASM_RETURN_TWO(nullptr, nullptr);
     224
     225            uint32_t index = 0;
     226            for (VirtualRegister reg : osrEntryData.values)
     227                buffer[index++] = READ(reg).encodedJSValue();
     228
     229            WASM_RETURN_TWO(buffer, osrEntryCallee->entrypoint().executableAddress());
     230        };
     231
     232        if (auto* osrEntryCallee = callee->osrEntryCallee(instance->memory()->mode()))
     233            return doOSREntry(osrEntryCallee);
     234
     235        bool compile = false;
     236        {
     237            Locker locker { tierUpCounter.m_lock };
     238            switch (tierUpCounter.m_loopCompilationStatus) {
     239            case Wasm::LLIntTierUpCounter::CompilationStatus::NotCompiled:
     240                compile = true;
     241                tierUpCounter.m_loopCompilationStatus = Wasm::LLIntTierUpCounter::CompilationStatus::Compiling;
     242                break;
     243            case Wasm::LLIntTierUpCounter::CompilationStatus::Compiling:
     244                tierUpCounter.optimizeAfterWarmUp();
     245                break;
     246            case Wasm::LLIntTierUpCounter::CompilationStatus::Compiled:
     247                break;
     248            }
    224249        }
    225     }
    226 
    227     if (compile) {
    228         Ref<Wasm::Plan> plan = adoptRef(*static_cast<Wasm::Plan*>(new Wasm::OSREntryPlan(instance->context(), Ref<Wasm::Module>(instance->module()), Ref<Wasm::Callee>(*callee), callee->functionIndex(), osrEntryData.loopIndex, instance->memory()->mode(), Wasm::Plan::dontFinalize())));
    229         Wasm::ensureWorklist().enqueue(plan.copyRef());
    230         if (UNLIKELY(!Options::useConcurrentJIT()))
    231             plan->waitForCompletion();
    232         else
    233             tierUpCounter.optimizeAfterWarmUp();
    234     }
    235 
    236     if (auto* osrEntryCallee = callee->osrEntryCallee(instance->memory()->mode()))
    237         return doOSREntry(osrEntryCallee);
    238 
    239     WASM_RETURN_TWO(nullptr, nullptr);
     250
     251        if (compile) {
     252            Ref<Wasm::Plan> plan = adoptRef(*static_cast<Wasm::Plan*>(new Wasm::OSREntryPlan(instance->context(), Ref<Wasm::Module>(instance->module()), Ref<Wasm::Callee>(*callee), callee->functionIndex(), osrEntryData.loopIndex, instance->memory()->mode(), Wasm::Plan::dontFinalize())));
     253            Wasm::ensureWorklist().enqueue(plan.copyRef());
     254            if (UNLIKELY(!Options::useConcurrentJIT()))
     255                plan->waitForCompletion();
     256            else
     257                tierUpCounter.optimizeAfterWarmUp();
     258        }
     259
     260        if (auto* osrEntryCallee = callee->osrEntryCallee(instance->memory()->mode()))
     261            return doOSREntry(osrEntryCallee);
     262
     263        WASM_RETURN_TWO(nullptr, nullptr);
     264    }
    240265}
    241266

  • trunk/Source/JavaScriptCore/wasm/js/JSToWasm.cpp

    r284935 r287864  
    179179    MacroAssembler::DataLabelPtr calleeMoveLocation = jit.moveWithPatch(MacroAssembler::TrustedImmPtr(nullptr), GPRInfo::nonPreservedNonReturnGPR);
    180180    jit.emitPutToCallFrameHeader(GPRInfo::nonPreservedNonReturnGPR, CallFrameSlot::callee);
    181     CodeLocationDataLabelPtr<WasmEntryPtrTag>* linkedCalleeMove = &result->calleeMoveLocation;
     181    Vector<CodeLocationDataLabelPtr<WasmEntryPtrTag>>* linkedCalleeMove = &result->calleeMoveLocations;
    182182    jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
    183         *linkedCalleeMove = linkBuffer.locationOf<WasmEntryPtrTag>(calleeMoveLocation);
     183        linkedCalleeMove->append(linkBuffer.locationOf<WasmEntryPtrTag>(calleeMoveLocation));
    184184    });
    185185
Note: See TracChangeset for help on using the changeset viewer.