Changeset 190220 in webkit


Ignore:
Timestamp:
Sep 24, 2015, 2:42:59 PM (10 years ago)
Author:
msaboff@apple.com
Message:

[ES6] Implement tail calls in the DFG
​https://bugs.webkit.org/show_bug.cgi?id=148663

Reviewed by Filip Pizlo.

jsc-tailcall: Implement the tail call opcodes in the DFG
​https://bugs.webkit.org/show_bug.cgi?id=146850

This patch adds support for tail calls in the DFG. This requires a slightly high number of nodes:

  • TailCall and TailCallVarargs are straightforward. They are terminal nodes and have the semantics of an actual tail call.
  • TailCallInlinedCaller and TailCallVarargsInlinedCaller are here to perform a tail call inside an inlined function. They are non terminal nodes, and are performing the call as a regular call after popping an appropriate number of inlined tail call frames.
  • TailCallForwardVarargs and TailCallForwardVarargsInlinedCaller are the extension of TailCallVarargs and TailCallVarargsInlinedCaller to enable the varargs forwarding optimization so that we don't lose performance with a tail call instead of a regular call.

This also required two broad kind of changes:

  • Changes in the JIT itself (DFGSpeculativeJIT) are pretty straightforward since they are just an extension of the baseline JIT changes introduced previously.
  • Changes in the runtime are mostly related with handling inline call frames. The idea here is that we have a special TailCall type for call frames that indicates to the various pieces of code walking the inline call frame that they should (recursively) skip the caller in their analysis.
  • bytecode/CallMode.h:

(JSC::specializationKindFor):

  • bytecode/CodeOrigin.cpp:

(JSC::CodeOrigin::inlineDepthForCallFrame):
(JSC::CodeOrigin::isApproximatelyEqualTo):
(JSC::CodeOrigin::approximateHash):
(JSC::CodeOrigin::inlineStack):

  • bytecode/CodeOrigin.h:
  • bytecode/InlineCallFrame.cpp:

(JSC::InlineCallFrame::dumpInContext):
(WTF::printInternal):

  • bytecode/InlineCallFrame.h:

(JSC::InlineCallFrame::callModeFor):
(JSC::InlineCallFrame::kindFor):
(JSC::InlineCallFrame::varargsKindFor):
(JSC::InlineCallFrame::specializationKindFor):
(JSC::InlineCallFrame::isVarargs):
(JSC::InlineCallFrame::isTail):
(JSC::InlineCallFrame::computeCallerSkippingDeadFrames):
(JSC::InlineCallFrame::getCallerSkippingDeadFrames):
(JSC::InlineCallFrame::getCallerInlineFrameSkippingDeadFrames):

  • dfg/DFGAbstractInterpreterInlines.h:

(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):

  • dfg/DFGArgumentsEliminationPhase.cpp:
  • dfg/DFGBasicBlock.h:

(JSC::DFG::BasicBlock::findTerminal):

  • dfg/DFGByteCodeParser.cpp:

(JSC::DFG::ByteCodeParser::inlineCallFrame):
(JSC::DFG::ByteCodeParser::allInlineFramesAreTailCalls):
(JSC::DFG::ByteCodeParser::currentCodeOrigin):
(JSC::DFG::ByteCodeParser::addCallWithoutSettingResult):
(JSC::DFG::ByteCodeParser::addCall):
(JSC::DFG::ByteCodeParser::getPredictionWithoutOSRExit):
(JSC::DFG::ByteCodeParser::getPrediction):
(JSC::DFG::ByteCodeParser::handleCall):
(JSC::DFG::ByteCodeParser::handleVarargsCall):
(JSC::DFG::ByteCodeParser::emitArgumentPhantoms):
(JSC::DFG::ByteCodeParser::inliningCost):
(JSC::DFG::ByteCodeParser::inlineCall):
(JSC::DFG::ByteCodeParser::attemptToInlineCall):
(JSC::DFG::ByteCodeParser::parseBlock):
(JSC::DFG::ByteCodeParser::InlineStackEntry::InlineStackEntry):
(JSC::DFG::ByteCodeParser::parseCodeBlock):

  • dfg/DFGCapabilities.cpp:

(JSC::DFG::capabilityLevel):

  • dfg/DFGClobberize.h:

(JSC::DFG::clobberize):

  • dfg/DFGDoesGC.cpp:

(JSC::DFG::doesGC):

  • dfg/DFGFixupPhase.cpp:

(JSC::DFG::FixupPhase::fixupNode):

  • dfg/DFGGraph.cpp:

(JSC::DFG::Graph::isLiveInBytecode):

  • dfg/DFGGraph.h:

(JSC::DFG::Graph::forAllLocalsLiveInBytecode):

  • dfg/DFGInPlaceAbstractState.cpp:

(JSC::DFG::InPlaceAbstractState::mergeToSuccessors):

  • dfg/DFGJITCompiler.cpp:

(JSC::DFG::JITCompiler::willCatchExceptionInMachineFrame):

  • dfg/DFGLiveCatchVariablePreservationPhase.cpp:

(JSC::DFG::FlushLiveCatchVariablesInsertionPhase::willCatchException):

  • dfg/DFGNode.h:

(JSC::DFG::Node::hasCallVarargsData):
(JSC::DFG::Node::isTerminal):
(JSC::DFG::Node::hasHeapPrediction):

  • dfg/DFGNodeType.h:
  • dfg/DFGOSRExitCompilerCommon.cpp:

(JSC::DFG::handleExitCounts):
(JSC::DFG::reifyInlinedCallFrames):
(JSC::DFG::osrWriteBarrier):

  • dfg/DFGOSRExitPreparation.cpp:

(JSC::DFG::prepareCodeOriginForOSRExit):

  • dfg/DFGOperations.cpp:
  • dfg/DFGPreciseLocalClobberize.h:

(JSC::DFG::PreciseLocalClobberizeAdaptor::readTop):

  • dfg/DFGPredictionPropagationPhase.cpp:

(JSC::DFG::PredictionPropagationPhase::propagate):

  • dfg/DFGSafeToExecute.h:

(JSC::DFG::safeToExecute):

  • dfg/DFGSpeculativeJIT32_64.cpp:

(JSC::DFG::SpeculativeJIT::emitCall):
(JSC::DFG::SpeculativeJIT::compile):

  • dfg/DFGSpeculativeJIT64.cpp:

(JSC::DFG::SpeculativeJIT::emitCall):
(JSC::DFG::SpeculativeJIT::compile):

  • dfg/DFGValidate.cpp:

(JSC::DFG::Validate::validateSSA):

  • dfg/DFGVarargsForwardingPhase.cpp:
  • interpreter/CallFrame.cpp:

(JSC::CallFrame::bytecodeOffset):

  • interpreter/StackVisitor.cpp:

(JSC::StackVisitor::gotoNextFrame):

Location:
trunk/Source/JavaScriptCore
Files:
33 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/JavaScriptCore/ChangeLog

    r190217 r190220  
     12015-09-24  Michael Saboff  <msaboff@apple.com>
     2
     3        [ES6] Implement tail calls in the DFG
     4        https://bugs.webkit.org/show_bug.cgi?id=148663
     5
     6        Reviewed by Filip Pizlo.
     7
     8        jsc-tailcall: Implement the tail call opcodes in the DFG
     9        https://bugs.webkit.org/show_bug.cgi?id=146850
     10
     11        This patch adds support for tail calls in the DFG. This requires a slightly high number of nodes:
     12
     13         - TailCall and TailCallVarargs are straightforward. They are terminal
     14           nodes and have the semantics of an actual tail call.
     15
     16         - TailCallInlinedCaller and TailCallVarargsInlinedCaller are here to perform a
     17           tail call inside an inlined function. They are non terminal nodes,
     18           and are performing the call as a regular call after popping an
     19           appropriate number of inlined tail call frames.
     20
     21         - TailCallForwardVarargs and TailCallForwardVarargsInlinedCaller are the
     22           extension of TailCallVarargs and TailCallVarargsInlinedCaller to enable
     23           the varargs forwarding optimization so that we don't lose
     24           performance with a tail call instead of a regular call.
     25
     26        This also required two broad kind of changes:
     27
     28         - Changes in the JIT itself (DFGSpeculativeJIT) are pretty
     29           straightforward since they are just an extension of the baseline JIT
     30           changes introduced previously.
     31
     32         - Changes in the runtime are mostly related with handling inline call
     33           frames. The idea here is that we have a special TailCall type for
     34           call frames that indicates to the various pieces of code walking the
     35           inline call frame that they should (recursively) skip the caller in
     36           their analysis.
     37
     38        * bytecode/CallMode.h:
     39        (JSC::specializationKindFor):
     40        * bytecode/CodeOrigin.cpp:
     41        (JSC::CodeOrigin::inlineDepthForCallFrame):
     42        (JSC::CodeOrigin::isApproximatelyEqualTo):
     43        (JSC::CodeOrigin::approximateHash):
     44        (JSC::CodeOrigin::inlineStack):
     45        * bytecode/CodeOrigin.h:
     46        * bytecode/InlineCallFrame.cpp:
     47        (JSC::InlineCallFrame::dumpInContext):
     48        (WTF::printInternal):
     49        * bytecode/InlineCallFrame.h:
     50        (JSC::InlineCallFrame::callModeFor):
     51        (JSC::InlineCallFrame::kindFor):
     52        (JSC::InlineCallFrame::varargsKindFor):
     53        (JSC::InlineCallFrame::specializationKindFor):
     54        (JSC::InlineCallFrame::isVarargs):
     55        (JSC::InlineCallFrame::isTail):
     56        (JSC::InlineCallFrame::computeCallerSkippingDeadFrames):
     57        (JSC::InlineCallFrame::getCallerSkippingDeadFrames):
     58        (JSC::InlineCallFrame::getCallerInlineFrameSkippingDeadFrames):
     59        * dfg/DFGAbstractInterpreterInlines.h:
     60        (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
     61        * dfg/DFGArgumentsEliminationPhase.cpp:
     62        * dfg/DFGBasicBlock.h:
     63        (JSC::DFG::BasicBlock::findTerminal):
     64        * dfg/DFGByteCodeParser.cpp:
     65        (JSC::DFG::ByteCodeParser::inlineCallFrame):
     66        (JSC::DFG::ByteCodeParser::allInlineFramesAreTailCalls):
     67        (JSC::DFG::ByteCodeParser::currentCodeOrigin):
     68        (JSC::DFG::ByteCodeParser::addCallWithoutSettingResult):
     69        (JSC::DFG::ByteCodeParser::addCall):
     70        (JSC::DFG::ByteCodeParser::getPredictionWithoutOSRExit):
     71        (JSC::DFG::ByteCodeParser::getPrediction):
     72        (JSC::DFG::ByteCodeParser::handleCall):
     73        (JSC::DFG::ByteCodeParser::handleVarargsCall):
     74        (JSC::DFG::ByteCodeParser::emitArgumentPhantoms):
     75        (JSC::DFG::ByteCodeParser::inliningCost):
     76        (JSC::DFG::ByteCodeParser::inlineCall):
     77        (JSC::DFG::ByteCodeParser::attemptToInlineCall):
     78        (JSC::DFG::ByteCodeParser::parseBlock):
     79        (JSC::DFG::ByteCodeParser::InlineStackEntry::InlineStackEntry):
     80        (JSC::DFG::ByteCodeParser::parseCodeBlock):
     81        * dfg/DFGCapabilities.cpp:
     82        (JSC::DFG::capabilityLevel):
     83        * dfg/DFGClobberize.h:
     84        (JSC::DFG::clobberize):
     85        * dfg/DFGDoesGC.cpp:
     86        (JSC::DFG::doesGC):
     87        * dfg/DFGFixupPhase.cpp:
     88        (JSC::DFG::FixupPhase::fixupNode):
     89        * dfg/DFGGraph.cpp:
     90        (JSC::DFG::Graph::isLiveInBytecode):
     91        * dfg/DFGGraph.h:
     92        (JSC::DFG::Graph::forAllLocalsLiveInBytecode):
     93        * dfg/DFGInPlaceAbstractState.cpp:
     94        (JSC::DFG::InPlaceAbstractState::mergeToSuccessors):
     95        * dfg/DFGJITCompiler.cpp:
     96        (JSC::DFG::JITCompiler::willCatchExceptionInMachineFrame):
     97        * dfg/DFGLiveCatchVariablePreservationPhase.cpp:
     98        (JSC::DFG::FlushLiveCatchVariablesInsertionPhase::willCatchException):
     99        * dfg/DFGNode.h:
     100        (JSC::DFG::Node::hasCallVarargsData):
     101        (JSC::DFG::Node::isTerminal):
     102        (JSC::DFG::Node::hasHeapPrediction):
     103        * dfg/DFGNodeType.h:
     104        * dfg/DFGOSRExitCompilerCommon.cpp:
     105        (JSC::DFG::handleExitCounts):
     106        (JSC::DFG::reifyInlinedCallFrames):
     107        (JSC::DFG::osrWriteBarrier):
     108        * dfg/DFGOSRExitPreparation.cpp:
     109        (JSC::DFG::prepareCodeOriginForOSRExit):
     110        * dfg/DFGOperations.cpp:
     111        * dfg/DFGPreciseLocalClobberize.h:
     112        (JSC::DFG::PreciseLocalClobberizeAdaptor::readTop):
     113        * dfg/DFGPredictionPropagationPhase.cpp:
     114        (JSC::DFG::PredictionPropagationPhase::propagate):
     115        * dfg/DFGSafeToExecute.h:
     116        (JSC::DFG::safeToExecute):
     117        * dfg/DFGSpeculativeJIT32_64.cpp:
     118        (JSC::DFG::SpeculativeJIT::emitCall):
     119        (JSC::DFG::SpeculativeJIT::compile):
     120        * dfg/DFGSpeculativeJIT64.cpp:
     121        (JSC::DFG::SpeculativeJIT::emitCall):
     122        (JSC::DFG::SpeculativeJIT::compile):
     123        * dfg/DFGValidate.cpp:
     124        (JSC::DFG::Validate::validateSSA):
     125        * dfg/DFGVarargsForwardingPhase.cpp:
     126        * interpreter/CallFrame.cpp:
     127        (JSC::CallFrame::bytecodeOffset):
     128        * interpreter/StackVisitor.cpp:
     129        (JSC::StackVisitor::gotoNextFrame):
     130
    11312015-09-23  Filip Pizlo  <fpizlo@apple.com>
    2132

  • trunk/Source/JavaScriptCore/bytecode/CallMode.h

    r189774 r190220  
    2727#define CallMode_h
    2828
     29#include "CodeSpecializationKind.h"
     30
    2931namespace JSC {
    3032
    … …  
    3234
    3335enum FrameAction { KeepTheFrame = 0, ReuseTheFrame };
     36
     37inline CodeSpecializationKind specializationKindFor(CallMode callMode)
     38{
     39    if (callMode == CallMode::Construct)
     40        return CodeForConstruct;
     41
     42    return CodeForCall;
     43}
    3444
    3545} // namespace JSC

  • trunk/Source/JavaScriptCore/bytecode/CodeOrigin.cpp

    r188585 r190220  
    3838{
    3939    unsigned result = 1;
    40     for (InlineCallFrame* current = inlineCallFrame; current; current = current->caller.inlineCallFrame)
     40    for (InlineCallFrame* current = inlineCallFrame; current; current = current->directCaller.inlineCallFrame)
    4141        result++;
    4242    return result;
    … …  
    7979            return false;
    8080       
    81         a = a.inlineCallFrame->caller;
    82         b = b.inlineCallFrame->caller;
     81        a = a.inlineCallFrame->directCaller;
     82        b = b.inlineCallFrame->directCaller;
    8383    }
    8484}
    … …  
    101101        result += WTF::PtrHash<JSCell*>::hash(codeOrigin.inlineCallFrame->executable.get());
    102102       
    103         codeOrigin = codeOrigin.inlineCallFrame->caller;
     103        codeOrigin = codeOrigin.inlineCallFrame->directCaller;
    104104    }
    105105}
    … …  
    110110    result.last() = *this;
    111111    unsigned index = result.size() - 2;
    112     for (InlineCallFrame* current = inlineCallFrame; current; current = current->caller.inlineCallFrame)
    113         result[index--] = current->caller;
     112    for (InlineCallFrame* current = inlineCallFrame; current; current = current->directCaller.inlineCallFrame)
     113        result[index--] = current->directCaller;
    114114    RELEASE_ASSERT(!result[0].inlineCallFrame);
    115115    return result;

  • trunk/Source/JavaScriptCore/bytecode/CodeOrigin.h

    r190073 r190220  
    2727#define CodeOrigin_h
    2828
     29#include "CallMode.h"
    2930#include "CodeBlockHash.h"
    3031#include "CodeSpecializationKind.h"

  • trunk/Source/JavaScriptCore/bytecode/InlineCallFrame.cpp

    r189518 r190220  
    7878    if (executable->isStrictMode())
    7979        out.print(" (StrictMode)");
    80     out.print(", bc#", caller.bytecodeIndex, ", ", kind);
     80    out.print(", bc#", directCaller.bytecodeIndex, ", ", static_cast<Kind>(kind));
    8181    if (isClosureCall)
    8282        out.print(", closure call");
    … …  
    106106        out.print("Construct");
    107107        return;
     108    case JSC::InlineCallFrame::TailCall:
     109        out.print("TailCall");
     110        return;
    108111    case JSC::InlineCallFrame::CallVarargs:
    109112        out.print("CallVarargs");
    … …  
    111114    case JSC::InlineCallFrame::ConstructVarargs:
    112115        out.print("ConstructVarargs");
     116        return;
     117    case JSC::InlineCallFrame::TailCallVarargs:
     118        out.print("TailCallVarargs");
    113119        return;
    114120    case JSC::InlineCallFrame::GetterCall:

  • trunk/Source/JavaScriptCore/bytecode/InlineCallFrame.h

    r189518 r190220  
    5050        Call,
    5151        Construct,
     52        TailCall,
    5253        CallVarargs,
    5354        ConstructVarargs,
     55        TailCallVarargs,
    5456       
    5557        // For these, the stackOffset incorporates the argument count plus the true return PC
    … …  
    5860        SetterCall
    5961    };
    60    
    61     static Kind kindFor(CodeSpecializationKind kind)
    62     {
    63         switch (kind) {
    64         case CodeForCall:
    65             return Call;
    66         case CodeForConstruct:
    67             return Construct;
    68         }
    69         RELEASE_ASSERT_NOT_REACHED();
    70         return Call;
    71     }
    72    
    73     static Kind varargsKindFor(CodeSpecializationKind kind)
    74     {
    75         switch (kind) {
    76         case CodeForCall:
    77             return CallVarargs;
    78         case CodeForConstruct:
    79             return ConstructVarargs;
    80         }
    81         RELEASE_ASSERT_NOT_REACHED();
    82         return Call;
    83     }
    84    
    85     static CodeSpecializationKind specializationKindFor(Kind kind)
     62
     63    static CallMode callModeFor(Kind kind)
    8664    {
    8765        switch (kind) {
    8866        case Call:
    8967        case CallVarargs:
     68        case GetterCall:
     69        case SetterCall:
     70            return CallMode::Regular;
     71        case TailCall:
     72        case TailCallVarargs:
     73            return CallMode::Tail;
     74        case Construct:
     75        case ConstructVarargs:
     76            return CallMode::Construct;
     77        }
     78        RELEASE_ASSERT_NOT_REACHED();
     79    }
     80
     81    static Kind kindFor(CallMode callMode)
     82    {
     83        switch (callMode) {
     84        case CallMode::Regular:
     85            return Call;
     86        case CallMode::Construct:
     87            return Construct;
     88        case CallMode::Tail:
     89            return TailCall;
     90        }
     91        RELEASE_ASSERT_NOT_REACHED();
     92    }
     93   
     94    static Kind varargsKindFor(CallMode callMode)
     95    {
     96        switch (callMode) {
     97        case CallMode::Regular:
     98            return CallVarargs;
     99        case CallMode::Construct:
     100            return ConstructVarargs;
     101        case CallMode::Tail:
     102            return TailCallVarargs;
     103        }
     104        RELEASE_ASSERT_NOT_REACHED();
     105    }
     106   
     107    static CodeSpecializationKind specializationKindFor(Kind kind)
     108    {
     109        switch (kind) {
     110        case Call:
     111        case CallVarargs:
     112        case TailCall:
     113        case TailCallVarargs:
    90114        case GetterCall:
    91115        case SetterCall:
    … …  
    96120        }
    97121        RELEASE_ASSERT_NOT_REACHED();
    98         return CodeForCall;
    99122    }
    100123   
    … …  
    103126        switch (kind) {
    104127        case CallVarargs:
     128        case TailCallVarargs:
    105129        case ConstructVarargs:
    106130            return true;
    … …  
    109133        }
    110134    }
     135
     136    static bool isTail(Kind kind)
     137    {
     138        switch (kind) {
     139        case TailCall:
     140        case TailCallVarargs:
     141            return true;
     142        default:
     143            return false;
     144        }
     145    }
     146    bool isTail() const
     147    {
     148        return isTail(static_cast<Kind>(kind));
     149    }
     150
     151    static CodeOrigin* computeCallerSkippingDeadFrames(InlineCallFrame* inlineCallFrame)
     152    {
     153        CodeOrigin* codeOrigin;
     154        bool tailCallee;
     155        do {
     156            tailCallee = inlineCallFrame->isTail();
     157            codeOrigin = &inlineCallFrame->directCaller;
     158            inlineCallFrame = codeOrigin->inlineCallFrame;
     159        } while (inlineCallFrame && tailCallee);
     160        if (tailCallee)
     161            return nullptr;
     162        return codeOrigin;
     163    }
     164
     165    CodeOrigin* getCallerSkippingDeadFrames()
     166    {
     167        return computeCallerSkippingDeadFrames(this);
     168    }
     169
     170    InlineCallFrame* getCallerInlineFrameSkippingDeadFrames()
     171    {
     172        CodeOrigin* caller = getCallerSkippingDeadFrames();
     173        return caller ? caller->inlineCallFrame : nullptr;
     174    }
    111175   
    112176    Vector<ValueRecovery> arguments; // Includes 'this'.
    113177    WriteBarrier<ScriptExecutable> executable;
    114178    ValueRecovery calleeRecovery;
    115     CodeOrigin caller;
     179    CodeOrigin directCaller;
    116180
    117181    signed stackOffset : 28;

  • trunk/Source/JavaScriptCore/dfg/DFGAbstractInterpreterInlines.h

    r190076 r190220  
    15511551           
    15521552    case Return:
     1553        m_state.setIsValid(false);
     1554        break;
     1555
     1556    case TailCall:
     1557    case TailCallVarargs:
     1558    case TailCallForwardVarargs:
     1559        clobberWorld(node->origin.semantic, clobberLimit);
    15531560        m_state.setIsValid(false);
    15541561        break;
    … …  
    24432450           
    24442451    case Call:
     2452    case TailCallInlinedCaller:
    24452453    case Construct:
    24462454    case CallVarargs:
    24472455    case CallForwardVarargs:
     2456    case TailCallVarargsInlinedCaller:
    24482457    case ConstructVarargs:
    24492458    case ConstructForwardVarargs:
     2459    case TailCallForwardVarargsInlinedCaller:
    24502460        clobberWorld(node->origin.semantic, clobberLimit);
    24512461        forNode(node).makeHeapTop();

  • trunk/Source/JavaScriptCore/dfg/DFGArgumentsEliminationPhase.cpp

    r188979 r190220  
    168168                case CallVarargs:
    169169                case ConstructVarargs:
     170                case TailCallVarargs:
     171                case TailCallVarargsInlinedCaller:
    170172                    escape(node->child1());
    171173                    escape(node->child3());
    … …  
    561563                   
    562564                case CallVarargs:
    563                 case ConstructVarargs: {
     565                case ConstructVarargs:
     566                case TailCallVarargs:
     567                case TailCallVarargsInlinedCaller: {
    564568                    Node* candidate = node->child2().node();
    565569                    if (!m_candidates.contains(candidate))
    … …  
    586590                        for (Node* argument : arguments)
    587591                            m_graph.m_varArgChildren.append(Edge(argument));
    588                         node->setOpAndDefaultFlags(
    589                             node->op() == CallVarargs ? Call : Construct);
     592                        switch (node->op()) {
     593                        case CallVarargs:
     594                            node->setOpAndDefaultFlags(Call);
     595                            break;
     596                        case ConstructVarargs:
     597                            node->setOpAndDefaultFlags(Construct);
     598                            break;
     599                        case TailCallVarargs:
     600                            node->setOpAndDefaultFlags(TailCall);
     601                            break;
     602                        case TailCallVarargsInlinedCaller:
     603                            node->setOpAndDefaultFlags(TailCallInlinedCaller);
     604                            break;
     605                        default:
     606                            RELEASE_ASSERT_NOT_REACHED();
     607                        }
    590608                        node->children = AdjacencyList(
    591609                            AdjacencyList::Variable,
    … …  
    594612                    }
    595613                   
    596                     node->setOpAndDefaultFlags(
    597                         node->op() == CallVarargs ? CallForwardVarargs : ConstructForwardVarargs);
     614                    switch (node->op()) {
     615                    case CallVarargs:
     616                        node->setOpAndDefaultFlags(CallForwardVarargs);
     617                        break;
     618                    case ConstructVarargs:
     619                        node->setOpAndDefaultFlags(ConstructForwardVarargs);
     620                        break;
     621                    case TailCallVarargs:
     622                        node->setOpAndDefaultFlags(TailCallForwardVarargs);
     623                        break;
     624                    case TailCallVarargsInlinedCaller:
     625                        node->setOpAndDefaultFlags(TailCallForwardVarargsInlinedCaller);
     626                        break;
     627                    default:
     628                        RELEASE_ASSERT_NOT_REACHED();
     629                    }
    598630                    break;
    599631                }

  • trunk/Source/JavaScriptCore/dfg/DFGBasicBlock.h

    r189531 r190220  
    9393            case Switch:
    9494            case Return:
     95            case TailCall:
     96            case TailCallVarargs:
     97            case TailCallForwardVarargs:
    9598            case Unreachable:
    9699                return NodeAndIndex(node, i);

  • trunk/Source/JavaScriptCore/dfg/DFGByteCodeParser.cpp

    r190076 r190220  
    184184        SpeculatedType prediction);
    185185    void handleCall(
    186         int result, NodeType op, InlineCallFrame::Kind, unsigned instructionSize,
     186        int result, NodeType op, CallMode, unsigned instructionSize,
    187187        Node* callTarget, int argCount, int registerOffset, CallLinkStatus);
    188     void handleCall(int result, NodeType op, CodeSpecializationKind, unsigned instructionSize, int callee, int argCount, int registerOffset);
    189     void handleCall(Instruction* pc, NodeType op, CodeSpecializationKind);
    190     void handleVarargsCall(Instruction* pc, NodeType op, CodeSpecializationKind);
     188    void handleCall(int result, NodeType op, CallMode, unsigned instructionSize, int callee, int argCount, int registerOffset);
     189    void handleCall(Instruction* pc, NodeType op, CallMode);
     190    void handleVarargsCall(Instruction* pc, NodeType op, CallMode);
    191191    void emitFunctionChecks(CallVariant, Node* callTarget, VirtualRegister thisArgumnt);
    192192    void emitArgumentPhantoms(int registerOffset, int argumentCountIncludingThis);
    193     unsigned inliningCost(CallVariant, int argumentCountIncludingThis, CodeSpecializationKind); // Return UINT_MAX if it's not an inlining candidate. By convention, intrinsics have a cost of 1.
     193    unsigned inliningCost(CallVariant, int argumentCountIncludingThis, CallMode); // Return UINT_MAX if it's not an inlining candidate. By convention, intrinsics have a cost of 1.
    194194    // Handle inlining. Return true if it succeeded, false if we need to plant a call.
    195195    bool handleInlining(Node* callTargetNode, int resultOperand, const CallLinkStatus&, int registerOffset, VirtualRegister thisArgument, VirtualRegister argumentsArgument, unsigned argumentsOffset, int argumentCountIncludingThis, unsigned nextOffset, NodeType callOp, InlineCallFrame::Kind, SpeculatedType prediction);
    … …  
    649649    }
    650650
     651    bool allInlineFramesAreTailCalls()
     652    {
     653        return !inlineCallFrame() || !inlineCallFrame()->getCallerSkippingDeadFrames();
     654    }
     655
    651656    CodeOrigin currentCodeOrigin()
    652657    {
    … …  
    737742    Node* addCallWithoutSettingResult(
    738743        NodeType op, OpInfo opInfo, Node* callee, int argCount, int registerOffset,
    739         SpeculatedType prediction)
     744        OpInfo prediction)
    740745    {
    741746        addVarArgChild(callee);
    … …  
    750755            addVarArgChild(get(virtualRegisterForArgument(i, registerOffset)));
    751756
    752         return addToGraph(Node::VarArg, op, opInfo, OpInfo(prediction));
     757        return addToGraph(Node::VarArg, op, opInfo, prediction);
    753758    }
    754759   
    … …  
    757762        SpeculatedType prediction)
    758763    {
     764        if (op == TailCall) {
     765            if (allInlineFramesAreTailCalls())
     766                return addCallWithoutSettingResult(op, OpInfo(), callee, argCount, registerOffset, OpInfo());
     767            op = TailCallInlinedCaller;
     768        }
     769
     770
    759771        Node* call = addCallWithoutSettingResult(
    760             op, opInfo, callee, argCount, registerOffset, prediction);
     772            op, opInfo, callee, argCount, registerOffset, OpInfo(prediction));
    761773        VirtualRegister resultReg(result);
    762774        if (resultReg.isValid())
    … …  
    777789    SpeculatedType getPredictionWithoutOSRExit(unsigned bytecodeIndex)
    778790    {
    779         ConcurrentJITLocker locker(m_inlineStackTop->m_profiledBlock->m_lock);
    780         return m_inlineStackTop->m_profiledBlock->valueProfilePredictionForBytecodeOffset(locker, bytecodeIndex);
     791        SpeculatedType prediction;
     792        CodeBlock* profiledBlock = nullptr;
     793
     794        {
     795            ConcurrentJITLocker locker(m_inlineStackTop->m_profiledBlock->m_lock);
     796            prediction = m_inlineStackTop->m_profiledBlock->valueProfilePredictionForBytecodeOffset(locker, bytecodeIndex);
     797
     798            if (prediction == SpecNone) {
     799                // If we have no information about the values this
     800                // node generates, we check if by any chance it is
     801                // a tail call opcode. In that case, we walk up the
     802                // inline frames to find a call higher in the call
     803                // chain and use its prediction. If we only have
     804                // inlined tail call frames, we use SpecFullTop
     805                // to avoid a spurious OSR exit.
     806                Instruction* instruction = m_inlineStackTop->m_profiledBlock->instructions().begin() + bytecodeIndex;
     807                OpcodeID opcodeID = m_vm->interpreter->getOpcodeID(instruction->u.opcode);
     808
     809                switch (opcodeID) {
     810                case op_tail_call:
     811                case op_tail_call_varargs: {
     812                    if (!inlineCallFrame()) {
     813                        prediction = SpecFullTop;
     814                        break;
     815                    }
     816                    CodeOrigin* codeOrigin = inlineCallFrame()->getCallerSkippingDeadFrames();
     817                    if (!codeOrigin) {
     818                        prediction = SpecFullTop;
     819                        break;
     820                    }
     821                    InlineStackEntry* stack = m_inlineStackTop;
     822                    while (stack->m_inlineCallFrame != codeOrigin->inlineCallFrame)
     823                        stack = stack->m_caller;
     824                    bytecodeIndex = codeOrigin->bytecodeIndex;
     825                    profiledBlock = stack->m_profiledBlock;
     826                    break;
     827                }
     828
     829                default:
     830                    break;
     831                }
     832            }
     833        }
     834
     835        if (profiledBlock) {
     836            ConcurrentJITLocker locker(profiledBlock->m_lock);
     837            prediction = profiledBlock->valueProfilePredictionForBytecodeOffset(locker, bytecodeIndex);
     838        }
     839
     840        return prediction;
    781841    }
    782842
    … …  
    784844    {
    785845        SpeculatedType prediction = getPredictionWithoutOSRExit(bytecodeIndex);
    786        
     846
    787847        if (prediction == SpecNone) {
    788848            // We have no information about what values this node generates. Give up
    … …  
    10701130    return shouldContinueParsing
    10711131
    1072 void ByteCodeParser::handleCall(Instruction* pc, NodeType op, CodeSpecializationKind kind)
     1132void ByteCodeParser::handleCall(Instruction* pc, NodeType op, CallMode callMode)
    10731133{
    10741134    ASSERT(OPCODE_LENGTH(op_call) == OPCODE_LENGTH(op_construct));
     1135    ASSERT(OPCODE_LENGTH(op_call) == OPCODE_LENGTH(op_tail_call));
    10751136    handleCall(
    1076         pc[1].u.operand, op, kind, OPCODE_LENGTH(op_call),
     1137        pc[1].u.operand, op, callMode, OPCODE_LENGTH(op_call),
    10771138        pc[2].u.operand, pc[3].u.operand, -pc[4].u.operand);
    10781139}
    10791140
    10801141void ByteCodeParser::handleCall(
    1081     int result, NodeType op, CodeSpecializationKind kind, unsigned instructionSize,
     1142    int result, NodeType op, CallMode callMode, unsigned instructionSize,
    10821143    int callee, int argumentCountIncludingThis, int registerOffset)
    10831144{
    … …  
    10891150   
    10901151    handleCall(
    1091         result, op, InlineCallFrame::kindFor(kind), instructionSize, callTarget,
     1152        result, op, callMode, instructionSize, callTarget,
    10921153        argumentCountIncludingThis, registerOffset, callLinkStatus);
    10931154}
    10941155   
    10951156void ByteCodeParser::handleCall(
    1096     int result, NodeType op, InlineCallFrame::Kind kind, unsigned instructionSize,
     1157    int result, NodeType op, CallMode callMode, unsigned instructionSize,
    10971158    Node* callTarget, int argumentCountIncludingThis, int registerOffset,
    10981159    CallLinkStatus callLinkStatus)
    10991160{
    11001161    handleCall(
    1101         result, op, kind, instructionSize, callTarget, argumentCountIncludingThis,
     1162        result, op, InlineCallFrame::kindFor(callMode), instructionSize, callTarget, argumentCountIncludingThis,
    11021163        registerOffset, callLinkStatus, getPrediction());
    11031164}
    … …  
    11191180        // Oddly, this conflates calls that haven't executed with calls that behaved sufficiently polymorphically
    11201181        // that we cannot optimize them.
    1121        
     1182
    11221183        addCall(result, op, OpInfo(), callTarget, argumentCountIncludingThis, registerOffset, prediction);
    11231184        return;
    … …  
    11371198}
    11381199
    1139 void ByteCodeParser::handleVarargsCall(Instruction* pc, NodeType op, CodeSpecializationKind kind)
     1200void ByteCodeParser::handleVarargsCall(Instruction* pc, NodeType op, CallMode callMode)
    11401201{
    11411202    ASSERT(OPCODE_LENGTH(op_call_varargs) == OPCODE_LENGTH(op_construct_varargs));
     1203    ASSERT(OPCODE_LENGTH(op_call_varargs) == OPCODE_LENGTH(op_tail_call_varargs));
    11421204   
    11431205    int result = pc[1].u.operand;
    … …  
    11621224   
    11631225    if (callLinkStatus.canOptimize()
    1164         && handleInlining(callTarget, result, callLinkStatus, firstFreeReg, VirtualRegister(thisReg), VirtualRegister(arguments), firstVarArgOffset, 0, m_currentIndex + OPCODE_LENGTH(op_call_varargs), op, InlineCallFrame::varargsKindFor(kind), prediction)) {
     1226        && handleInlining(callTarget, result, callLinkStatus, firstFreeReg, VirtualRegister(thisReg), VirtualRegister(arguments), firstVarArgOffset, 0, m_currentIndex + OPCODE_LENGTH(op_call_varargs), op, InlineCallFrame::varargsKindFor(callMode), prediction)) {
    11651227        if (m_graph.compilation())
    11661228            m_graph.compilation()->noticeInlinedCall();
    … …  
    11721234   
    11731235    Node* thisChild = get(VirtualRegister(thisReg));
    1174    
     1236
     1237    if (op == TailCallVarargs) {
     1238        if (allInlineFramesAreTailCalls()) {
     1239            addToGraph(op, OpInfo(data), OpInfo(), callTarget, get(VirtualRegister(arguments)), thisChild);
     1240            return;
     1241        }
     1242        op = TailCallVarargsInlinedCaller;
     1243    }
     1244
    11751245    Node* call = addToGraph(op, OpInfo(data), OpInfo(prediction), callTarget, get(VirtualRegister(arguments)), thisChild);
    11761246    VirtualRegister resultReg(result);
    … …  
    12071277}
    12081278
    1209 unsigned ByteCodeParser::inliningCost(CallVariant callee, int argumentCountIncludingThis, CodeSpecializationKind kind)
     1279unsigned ByteCodeParser::inliningCost(CallVariant callee, int argumentCountIncludingThis, CallMode callMode)
    12101280{
     1281    CodeSpecializationKind kind = specializationKindFor(callMode);
    12111282    if (verbose)
    12121283        dataLog("Considering inlining ", callee, " into ", currentCodeOrigin(), "\n");
    … …  
    12501321        codeBlock, kind, callee.isClosureCall());
    12511322    if (verbose) {
    1252         dataLog("    Kind: ", kind, "\n");
     1323        dataLog("    Call mode: ", callMode, "\n");
    12531324        dataLog("    Is closure call: ", callee.isClosureCall(), "\n");
    12541325        dataLog("    Capability level: ", capabilityLevel, "\n");
    … …  
    13211392    CodeSpecializationKind specializationKind = InlineCallFrame::specializationKindFor(kind);
    13221393   
    1323     ASSERT(inliningCost(callee, argumentCountIncludingThis, specializationKind) != UINT_MAX);
     1394    ASSERT(inliningCost(callee, argumentCountIncludingThis, InlineCallFrame::callModeFor(kind)) != UINT_MAX);
    13241395   
    13251396    CodeBlock* codeBlock = callee.functionExecutable()->baselineCodeBlockFor(specializationKind);
    … …  
    14251496        return;
    14261497    }
    1427    
     1498
    14281499    if (Options::verboseDFGByteCodeParsing())
    14291500        dataLog("    Creating new block after inlining.\n");
    … …  
    15321603    }
    15331604   
    1534     unsigned myInliningCost = inliningCost(callee, argumentCountIncludingThis, specializationKind);
     1605    unsigned myInliningCost = inliningCost(callee, argumentCountIncludingThis, InlineCallFrame::callModeFor(kind));
    15351606    if (myInliningCost > inliningBalance)
    15361607        return false;
    … …  
    30043075            // be true anyway except for op_loop_hint, which emits a Phantom to force this
    30053076            // to be true.
    3006             if (!m_currentBlock->isEmpty())
     3077            // We also don't insert a jump if the block already has a terminal,
     3078            // which could happen after a tail call.
     3079            ASSERT(m_currentBlock->isEmpty() || !m_currentBlock->terminal()
     3080                || m_currentBlock->terminal()->op() == TailCall || m_currentBlock->terminal()->op() == TailCallVarargs);
     3081            if (!m_currentBlock->isEmpty() && !m_currentBlock->terminal())
    30073082                addToGraph(Jump, OpInfo(m_currentIndex));
    30083083            return shouldContinueParsing;
    … …  
    35863661
    35873662        case op_jmp: {
     3663            if (m_currentBlock->terminal()) {
     3664                // We could be the dummy jump to a return after a non-inlined, non-emulated tail call in a ternary operator
     3665                Node* terminal = m_currentBlock->terminal();
     3666                ASSERT_UNUSED(terminal, terminal->op() == TailCall || terminal->op() == TailCallVarargs);
     3667                LAST_OPCODE(op_ret);
     3668            }
    35883669            int relativeOffset = currentInstruction[1].u.operand;
    35893670            addToGraph(Jump, OpInfo(m_currentIndex + relativeOffset));
    … …  
    37573838
    37583839        case op_ret:
     3840            if (m_currentBlock->terminal()) {
     3841                // We could be the dummy return after a non-inlined, non-emulated tail call
     3842                Node* terminal = m_currentBlock->terminal();
     3843                ASSERT_UNUSED(terminal, terminal->op() == TailCall || terminal->op() == TailCallVarargs);
     3844                LAST_OPCODE(op_ret);
     3845            }
    37593846            if (inlineCallFrame()) {
    37603847                flushForReturn();
    … …  
    38083895           
    38093896        case op_call:
    3810             handleCall(currentInstruction, Call, CodeForCall);
     3897            handleCall(currentInstruction, Call, CallMode::Regular);
    38113898            // Verify that handleCall(), which could have inlined the callee, didn't trash m_currentInstruction
    38123899            ASSERT(m_currentInstruction == currentInstruction);
    38133900            NEXT_OPCODE(op_call);
    3814            
     3901
     3902        case op_tail_call:
     3903            flushForReturn();
     3904            handleCall(currentInstruction, TailCall, CallMode::Tail);
     3905            // Verify that handleCall(), which could have inlined the callee, didn't trash m_currentInstruction
     3906            ASSERT(m_currentInstruction == currentInstruction);
     3907            // We let the following op_ret handle cases related to
     3908            // inlining to keep things simple.
     3909            NEXT_OPCODE(op_tail_call);
     3910
    38153911        case op_construct:
    3816             handleCall(currentInstruction, Construct, CodeForConstruct);
     3912            handleCall(currentInstruction, Construct, CallMode::Construct);
    38173913            NEXT_OPCODE(op_construct);
    38183914           
    38193915        case op_call_varargs: {
    3820             handleVarargsCall(currentInstruction, CallVarargs, CodeForCall);
     3916            handleVarargsCall(currentInstruction, CallVarargs, CallMode::Regular);
    38213917            NEXT_OPCODE(op_call_varargs);
    38223918        }
     3919
     3920        case op_tail_call_varargs: {
     3921            flushForReturn();
     3922            handleVarargsCall(currentInstruction, TailCallVarargs, CallMode::Tail);
     3923            NEXT_OPCODE(op_tail_call_varargs);
     3924        }
    38233925           
    38243926        case op_construct_varargs: {
    3825             handleVarargsCall(currentInstruction, ConstructVarargs, CodeForConstruct);
     3927            handleVarargsCall(currentInstruction, ConstructVarargs, CallMode::Construct);
    38263928            NEXT_OPCODE(op_construct_varargs);
    38273929        }
    … …  
    45344636        } else
    45354637            m_inlineCallFrame->isClosureCall = true;
    4536         m_inlineCallFrame->caller = byteCodeParser->currentCodeOrigin();
     4638        m_inlineCallFrame->directCaller = byteCodeParser->currentCodeOrigin();
    45374639        m_inlineCallFrame->arguments.resizeToFit(argumentCountIncludingThis); // Set the number of arguments including this, but don't configure the value recoveries, yet.
    45384640        m_inlineCallFrame->kind = kind;
    … …  
    46054707        Vector<DeferredSourceDump>& deferredSourceDump = m_graph.m_plan.callback->ensureDeferredSourceDump();
    46064708        if (inlineCallFrame()) {
    4607             DeferredSourceDump dump(codeBlock->baselineVersion(), m_codeBlock, JITCode::DFGJIT, inlineCallFrame()->caller);
     4709            DeferredSourceDump dump(codeBlock->baselineVersion(), m_codeBlock, JITCode::DFGJIT, inlineCallFrame()->directCaller);
    46084710            deferredSourceDump.append(dump);
    46094711        } else
    … …  
    46164718            dataLog(
    46174719                " for inlining at ", CodeBlockWithJITType(m_codeBlock, JITCode::DFGJIT),
    4618                 " ", inlineCallFrame()->caller);
     4720                " ", inlineCallFrame()->directCaller);
    46194721        }
    46204722        dataLog(
    … …  
    47004802            }
    47014803           
    4702             m_currentBlock = 0;
     4804            m_currentBlock = nullptr;
    47034805        } while (m_currentIndex < limit);
    47044806    }

  • trunk/Source/JavaScriptCore/dfg/DFGCapabilities.cpp

    r189995 r190220  
    180180    case op_throw_static_error:
    181181    case op_call:
     182    case op_tail_call:
    182183    case op_construct:
    183184    case op_call_varargs:
     185    case op_tail_call_varargs:
    184186    case op_construct_varargs:
    185187    case op_create_direct_arguments:

  • trunk/Source/JavaScriptCore/dfg/DFGClobberize.h

    r189279 r190220  
    383383    case ArrayPop:
    384384    case Call:
     385    case TailCallInlinedCaller:
    385386    case Construct:
    386387    case CallVarargs:
    387388    case CallForwardVarargs:
     389    case TailCallVarargsInlinedCaller:
     390    case TailCallForwardVarargsInlinedCaller:
    388391    case ConstructVarargs:
    389392    case ConstructForwardVarargs:
    … …  
    393396        read(World);
    394397        write(Heap);
     398        return;
     399
     400    case TailCall:
     401    case TailCallVarargs:
     402    case TailCallForwardVarargs:
     403        read(World);
     404        write(SideState);
    395405        return;
    396406       

  • trunk/Source/JavaScriptCore/dfg/DFGDoesGC.cpp

    r189279 r190220  
    121121    case CompareStrictEq:
    122122    case Call:
     123    case TailCallInlinedCaller:
    123124    case Construct:
    124125    case CallVarargs:
     126    case TailCallVarargsInlinedCaller:
    125127    case ConstructVarargs:
    126128    case LoadVarargs:
    127129    case CallForwardVarargs:
    128130    case ConstructForwardVarargs:
     131    case TailCallForwardVarargs:
     132    case TailCallForwardVarargsInlinedCaller:
    129133    case Breakpoint:
    130134    case ProfileWillCall:
    … …  
    151155    case Switch:
    152156    case Return:
     157    case TailCall:
     158    case TailCallVarargs:
    153159    case Throw:
    154160    case CountExecution:

  • trunk/Source/JavaScriptCore/dfg/DFGFixupPhase.cpp

    r190076 r190220  
    13441344        case VarInjectionWatchpoint:
    13451345        case Call:
     1346        case TailCallInlinedCaller:
    13461347        case Construct:
    13471348        case CallVarargs:
     1349        case TailCallVarargsInlinedCaller:
    13481350        case ConstructVarargs:
    13491351        case CallForwardVarargs:
    13501352        case ConstructForwardVarargs:
     1353        case TailCallForwardVarargs:
     1354        case TailCallForwardVarargsInlinedCaller:
    13511355        case LoadVarargs:
    13521356        case ProfileControlFlow:
    … …  
    13661370        case Jump:
    13671371        case Return:
     1372        case TailCall:
     1373        case TailCallVarargs:
    13681374        case Throw:
    13691375        case ThrowReferenceError:

  • trunk/Source/JavaScriptCore/dfg/DFGGraph.cpp

    r190076 r190220  
    962962bool Graph::isLiveInBytecode(VirtualRegister operand, CodeOrigin codeOrigin)
    963963{
     964    CodeOrigin* codeOriginPtr = &codeOrigin;
    964965    for (;;) {
    965966        VirtualRegister reg = VirtualRegister(
    966             operand.offset() - codeOrigin.stackOffset());
    967        
    968         if (operand.offset() < codeOrigin.stackOffset() + JSStack::CallFrameHeaderSize) {
     967            operand.offset() - codeOriginPtr->stackOffset());
     968       
     969        if (operand.offset() < codeOriginPtr->stackOffset() + JSStack::CallFrameHeaderSize) {
    969970            if (reg.isArgument()) {
    970971                RELEASE_ASSERT(reg.offset() < JSStack::CallFrameHeaderSize);
    971972               
    972                 if (codeOrigin.inlineCallFrame->isClosureCall
     973                if (codeOriginPtr->inlineCallFrame->isClosureCall
    973974                    && reg.offset() == JSStack::Callee)
    974975                    return true;
    975976               
    976                 if (codeOrigin.inlineCallFrame->isVarargs()
     977                if (codeOriginPtr->inlineCallFrame->isVarargs()
    977978                    && reg.offset() == JSStack::ArgumentCount)
    978979                    return true;
    … …  
    981982            }
    982983           
    983             return livenessFor(codeOrigin.inlineCallFrame).operandIsLive(
    984                 reg.offset(), codeOrigin.bytecodeIndex);
    985         }
    986        
    987         InlineCallFrame* inlineCallFrame = codeOrigin.inlineCallFrame;
     984            return livenessFor(codeOriginPtr->inlineCallFrame).operandIsLive(
     985                reg.offset(), codeOriginPtr->bytecodeIndex);
     986        }
     987       
     988        InlineCallFrame* inlineCallFrame = codeOriginPtr->inlineCallFrame;
    988989        if (!inlineCallFrame)
    989990            break;
    … …  
    995996            return true;
    996997       
    997         codeOrigin = inlineCallFrame->caller;
     998        codeOriginPtr = inlineCallFrame->getCallerSkippingDeadFrames();
     999
     1000        // The first inline call frame could be an inline tail call
     1001        if (!codeOriginPtr)
     1002            break;
    9981003    }
    9991004   

  • trunk/Source/JavaScriptCore/dfg/DFGGraph.h

    r190076 r190220  
    718718        VirtualRegister exclusionStart;
    719719        VirtualRegister exclusionEnd;
     720
     721        CodeOrigin* codeOriginPtr = &codeOrigin;
    720722       
    721723        for (;;) {
    722             InlineCallFrame* inlineCallFrame = codeOrigin.inlineCallFrame;
     724            InlineCallFrame* inlineCallFrame = codeOriginPtr->inlineCallFrame;
    723725            VirtualRegister stackOffset(inlineCallFrame ? inlineCallFrame->stackOffset : 0);
    724726           
    … …  
    732734            CodeBlock* codeBlock = baselineCodeBlockFor(inlineCallFrame);
    733735            FullBytecodeLiveness& fullLiveness = livenessFor(codeBlock);
    734             const FastBitVector& liveness = fullLiveness.getLiveness(codeOrigin.bytecodeIndex);
     736            const FastBitVector& liveness = fullLiveness.getLiveness(codeOriginPtr->bytecodeIndex);
    735737            for (unsigned relativeLocal = codeBlock->m_numCalleeRegisters; relativeLocal--;) {
    736738                VirtualRegister reg = stackOffset + virtualRegisterForLocal(relativeLocal);
    … …  
    759761                functor(reg);
    760762           
    761             codeOrigin = inlineCallFrame->caller;
     763            codeOriginPtr = inlineCallFrame->getCallerSkippingDeadFrames();
     764
     765            // The first inline call frame could be an inline tail call
     766            if (!codeOriginPtr)
     767                break;
    762768        }
    763769    }

  • trunk/Source/JavaScriptCore/dfg/DFGInPlaceAbstractState.cpp

    r189138 r190220  
    365365       
    366366    case Return:
     367    case TailCall:
     368    case TailCallVarargs:
     369    case TailCallForwardVarargs:
    367370    case Unreachable:
    368371        ASSERT(basicBlock->cfaBranchDirection == InvalidBranchDirection);

  • trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.cpp

    r189995 r190220  
    576576            return false;
    577577
    578         bytecodeIndexToCheck = inlineCallFrame->caller.bytecodeIndex;
    579         codeOrigin = codeOrigin.inlineCallFrame->caller;
     578        bytecodeIndexToCheck = inlineCallFrame->directCaller.bytecodeIndex;
     579        codeOrigin = codeOrigin.inlineCallFrame->directCaller;
    580580    }
    581581

  • trunk/Source/JavaScriptCore/dfg/DFGLiveCatchVariablePreservationPhase.cpp

    r189995 r190220  
    8282                return false;
    8383
    84             bytecodeIndexToCheck = inlineCallFrame->caller.bytecodeIndex;
    85             origin = inlineCallFrame->caller;
     84            bytecodeIndexToCheck = inlineCallFrame->directCaller.bytecodeIndex;
     85            origin = inlineCallFrame->directCaller;
    8686        }
    8787    }

  • trunk/Source/JavaScriptCore/dfg/DFGNode.h

    r190076 r190220  
    10081008        case CallVarargs:
    10091009        case CallForwardVarargs:
     1010        case TailCallVarargs:
     1011        case TailCallForwardVarargs:
     1012        case TailCallVarargsInlinedCaller:
     1013        case TailCallForwardVarargsInlinedCaller:
    10101014        case ConstructVarargs:
    10111015        case ConstructForwardVarargs:
    … …  
    11051109        case Switch:
    11061110        case Return:
     1111        case TailCall:
     1112        case TailCallVarargs:
     1113        case TailCallForwardVarargs:
    11071114        case Unreachable:
    11081115            return true;
    … …  
    12551262        case GetByVal:
    12561263        case Call:
     1264        case TailCallInlinedCaller:
    12571265        case Construct:
    12581266        case CallVarargs:
     1267        case TailCallVarargsInlinedCaller:
    12591268        case ConstructVarargs:
    12601269        case CallForwardVarargs:
     1270        case TailCallForwardVarargsInlinedCaller:
    12611271        case GetByOffset:
    12621272        case MultiGetByOffset:

  • trunk/Source/JavaScriptCore/dfg/DFGNodeType.h

    r189279 r190220  
    246246    macro(ConstructVarargs, NodeResultJS | NodeMustGenerate) \
    247247    macro(ConstructForwardVarargs, NodeResultJS | NodeMustGenerate) \
     248    macro(TailCallInlinedCaller, NodeResultJS | NodeMustGenerate | NodeHasVarArgs) \
     249    macro(TailCallVarargsInlinedCaller, NodeResultJS | NodeMustGenerate) \
     250    macro(TailCallForwardVarargsInlinedCaller, NodeResultJS | NodeMustGenerate) \
    248251    \
    249252    /* Allocations. */\
    … …  
    311314    macro(Switch, NodeMustGenerate) \
    312315    macro(Return, NodeMustGenerate) \
     316    macro(TailCall, NodeMustGenerate | NodeHasVarArgs) \
     317    macro(TailCallVarargs, NodeMustGenerate) \
     318    macro(TailCallForwardVarargs, NodeMustGenerate) \
    313319    macro(Unreachable, NodeMustGenerate) \
    314320    \

  • trunk/Source/JavaScriptCore/dfg/DFGOSRExitCompilerCommon.cpp

    r190129 r190220  
    6565    AssemblyHelpers::JumpList loopThreshold;
    6666   
    67     for (InlineCallFrame* inlineCallFrame = exit.m_codeOrigin.inlineCallFrame; inlineCallFrame; inlineCallFrame = inlineCallFrame->caller.inlineCallFrame) {
     67    for (InlineCallFrame* inlineCallFrame = exit.m_codeOrigin.inlineCallFrame; inlineCallFrame; inlineCallFrame = inlineCallFrame->directCaller.inlineCallFrame) {
    6868        loopThreshold.append(
    6969            jit.branchTest8(
    … …  
    137137void reifyInlinedCallFrames(CCallHelpers& jit, const OSRExitBase& exit)
    138138{
     139    // FIXME: We shouldn't leave holes on the stack when performing an OSR exit
     140    // in presence of inlined tail calls.
     141    // https://bugs.webkit.org/show_bug.cgi?id=147511
    139142    ASSERT(jit.baselineCodeBlock()->jitType() == JITCode::BaselineJIT);
    140143    jit.storePtr(AssemblyHelpers::TrustedImmPtr(jit.baselineCodeBlock()), AssemblyHelpers::addressFor((VirtualRegister)JSStack::CodeBlock));
    141144
    142     CodeOrigin codeOrigin;
    143     for (codeOrigin = exit.m_codeOrigin; codeOrigin.inlineCallFrame; codeOrigin = codeOrigin.inlineCallFrame->caller) {
    144         InlineCallFrame* inlineCallFrame = codeOrigin.inlineCallFrame;
    145         CodeBlock* baselineCodeBlock = jit.baselineCodeBlockFor(codeOrigin);
    146         CodeBlock* baselineCodeBlockForCaller = jit.baselineCodeBlockFor(inlineCallFrame->caller);
    147         void* jumpTarget = nullptr;
     145    const CodeOrigin* codeOrigin;
     146    for (codeOrigin = &exit.m_codeOrigin; codeOrigin && codeOrigin->inlineCallFrame; codeOrigin = codeOrigin->inlineCallFrame->getCallerSkippingDeadFrames()) {
     147        InlineCallFrame* inlineCallFrame = codeOrigin->inlineCallFrame;
     148        CodeBlock* baselineCodeBlock = jit.baselineCodeBlockFor(*codeOrigin);
     149        CodeOrigin* trueCaller = inlineCallFrame->getCallerSkippingDeadFrames();
    148150        void* trueReturnPC = nullptr;
    149        
    150         unsigned callBytecodeIndex = inlineCallFrame->caller.bytecodeIndex;
    151        
    152         switch (inlineCallFrame->kind) {
    153         case InlineCallFrame::Call:
    154         case InlineCallFrame::Construct:
    155         case InlineCallFrame::CallVarargs:
    156         case InlineCallFrame::ConstructVarargs: {
    157             CallLinkInfo* callLinkInfo =
    158                 baselineCodeBlockForCaller->getCallLinkInfoForBytecodeIndex(callBytecodeIndex);
    159             RELEASE_ASSERT(callLinkInfo);
    160            
    161             jumpTarget = callLinkInfo->callReturnLocation().executableAddress();
    162             break;
    163         }
    164            
    165         case InlineCallFrame::GetterCall:
    166         case InlineCallFrame::SetterCall: {
    167             StructureStubInfo* stubInfo =
    168                 baselineCodeBlockForCaller->findStubInfo(CodeOrigin(callBytecodeIndex));
    169             RELEASE_ASSERT(stubInfo);
    170            
     151        GPRReg callerFrameGPR = GPRInfo::callFrameRegister;
     152
     153        if (!trueCaller) {
     154            ASSERT(inlineCallFrame->isTail());
     155            jit.loadPtr(AssemblyHelpers::Address(GPRInfo::callFrameRegister, CallFrame::returnPCOffset()), GPRInfo::regT3);
     156            jit.storePtr(GPRInfo::regT3, AssemblyHelpers::addressForByteOffset(inlineCallFrame->returnPCOffset()));
     157            jit.loadPtr(AssemblyHelpers::Address(GPRInfo::callFrameRegister, CallFrame::callerFrameOffset()), GPRInfo::regT3);
     158            callerFrameGPR = GPRInfo::regT3;
     159        } else {
     160            CodeBlock* baselineCodeBlockForCaller = jit.baselineCodeBlockFor(*trueCaller);
     161            unsigned callBytecodeIndex = trueCaller->bytecodeIndex;
     162            void* jumpTarget = nullptr;
     163
    171164            switch (inlineCallFrame->kind) {
    172             case InlineCallFrame::GetterCall:
    173                 jumpTarget = jit.vm()->getCTIStub(baselineGetterReturnThunkGenerator).code().executableAddress();
    174                 break;
    175             case InlineCallFrame::SetterCall:
    176                 jumpTarget = jit.vm()->getCTIStub(baselineSetterReturnThunkGenerator).code().executableAddress();
    177                 break;
    178             default:
    179                 RELEASE_ASSERT_NOT_REACHED();
     165            case InlineCallFrame::Call:
     166            case InlineCallFrame::Construct:
     167            case InlineCallFrame::CallVarargs:
     168            case InlineCallFrame::ConstructVarargs:
     169            case InlineCallFrame::TailCall:
     170            case InlineCallFrame::TailCallVarargs: {
     171                CallLinkInfo* callLinkInfo =
     172                    baselineCodeBlockForCaller->getCallLinkInfoForBytecodeIndex(callBytecodeIndex);
     173                RELEASE_ASSERT(callLinkInfo);
     174
     175                jumpTarget = callLinkInfo->callReturnLocation().executableAddress();
    180176                break;
    181177            }
    182            
    183             trueReturnPC = stubInfo->callReturnLocation.labelAtOffset(
    184                 stubInfo->patch.deltaCallToDone).executableAddress();
    185             break;
    186         } }
    187 
    188         GPRReg callerFrameGPR;
    189         if (inlineCallFrame->caller.inlineCallFrame) {
    190             jit.addPtr(AssemblyHelpers::TrustedImm32(inlineCallFrame->caller.inlineCallFrame->stackOffset * sizeof(EncodedJSValue)), GPRInfo::callFrameRegister, GPRInfo::regT3);
    191             callerFrameGPR = GPRInfo::regT3;
    192         } else
    193             callerFrameGPR = GPRInfo::callFrameRegister;
    194        
    195         jit.storePtr(AssemblyHelpers::TrustedImmPtr(jumpTarget), AssemblyHelpers::addressForByteOffset(inlineCallFrame->returnPCOffset()));
     178
     179            case InlineCallFrame::GetterCall:
     180            case InlineCallFrame::SetterCall: {
     181                StructureStubInfo* stubInfo =
     182                    baselineCodeBlockForCaller->findStubInfo(CodeOrigin(callBytecodeIndex));
     183                RELEASE_ASSERT(stubInfo);
     184
     185                switch (inlineCallFrame->kind) {
     186                case InlineCallFrame::GetterCall:
     187                    jumpTarget = jit.vm()->getCTIStub(baselineGetterReturnThunkGenerator).code().executableAddress();
     188                    break;
     189                case InlineCallFrame::SetterCall:
     190                    jumpTarget = jit.vm()->getCTIStub(baselineSetterReturnThunkGenerator).code().executableAddress();
     191                    break;
     192                default:
     193                    RELEASE_ASSERT_NOT_REACHED();
     194                    break;
     195                }
     196
     197                trueReturnPC = stubInfo->callReturnLocation.labelAtOffset(
     198                    stubInfo->patch.deltaCallToDone).executableAddress();
     199                break;
     200            } }
     201
     202            if (trueCaller->inlineCallFrame) {
     203                jit.addPtr(
     204                    AssemblyHelpers::TrustedImm32(trueCaller->inlineCallFrame->stackOffset * sizeof(EncodedJSValue)),
     205                    GPRInfo::callFrameRegister,
     206                    GPRInfo::regT3);
     207                callerFrameGPR = GPRInfo::regT3;
     208            }
     209
     210            jit.storePtr(AssemblyHelpers::TrustedImmPtr(jumpTarget), AssemblyHelpers::addressForByteOffset(inlineCallFrame->returnPCOffset()));
     211        }
     212
    196213        if (trueReturnPC)
    197214            jit.storePtr(AssemblyHelpers::TrustedImmPtr(trueReturnPC), AssemblyHelpers::addressFor(inlineCallFrame->stackOffset + virtualRegisterForArgument(inlineCallFrame->arguments.size()).offset()));
    … …  
    203220#if USE(JSVALUE64)
    204221        jit.store64(callerFrameGPR, AssemblyHelpers::addressForByteOffset(inlineCallFrame->callerFrameOffset()));
    205         uint32_t locationBits = CallSiteIndex(codeOrigin.bytecodeIndex).bits();
     222        uint32_t locationBits = CallSiteIndex(codeOrigin->bytecodeIndex).bits();
    206223        jit.store32(AssemblyHelpers::TrustedImm32(locationBits), AssemblyHelpers::tagFor((VirtualRegister)(inlineCallFrame->stackOffset + JSStack::ArgumentCount)));
    207224        if (!inlineCallFrame->isClosureCall)
    … …  
    209226#else // USE(JSVALUE64) // so this is the 32-bit part
    210227        jit.storePtr(callerFrameGPR, AssemblyHelpers::addressForByteOffset(inlineCallFrame->callerFrameOffset()));
    211         Instruction* instruction = baselineCodeBlock->instructions().begin() + codeOrigin.bytecodeIndex;
     228        Instruction* instruction = baselineCodeBlock->instructions().begin() + codeOrigin->bytecodeIndex;
    212229        uint32_t locationBits = CallSiteIndex(instruction).bits();
    213230        jit.store32(AssemblyHelpers::TrustedImm32(locationBits), AssemblyHelpers::tagFor((VirtualRegister)(inlineCallFrame->stackOffset + JSStack::ArgumentCount)));
    … …  
    218235    }
    219236
     237    // Don't need to set the toplevel code origin if we only did inline tail calls
     238    if (codeOrigin) {
    220239#if USE(JSVALUE64)
    221     uint32_t locationBits = CallSiteIndex(codeOrigin.bytecodeIndex).bits();
     240    uint32_t locationBits = CallSiteIndex(codeOrigin->bytecodeIndex).bits();
    222241#else
    223     Instruction* instruction = jit.baselineCodeBlock()->instructions().begin() + codeOrigin.bytecodeIndex;
     242    Instruction* instruction = jit.baselineCodeBlock()->instructions().begin() + codeOrigin->bytecodeIndex;
    224243    uint32_t locationBits = CallSiteIndex(instruction).bits();
    225244#endif
    226     jit.store32(AssemblyHelpers::TrustedImm32(locationBits), AssemblyHelpers::tagFor((VirtualRegister)(JSStack::ArgumentCount)));
     245        jit.store32(AssemblyHelpers::TrustedImm32(locationBits), AssemblyHelpers::tagFor((VirtualRegister)(JSStack::ArgumentCount)));
     246    }
    227247}
    228248

  • trunk/Source/JavaScriptCore/dfg/DFGOSRExitPreparation.cpp

    r189889 r190220  
    4242    DeferGC deferGC(vm.heap);
    4343   
    44     for (; codeOrigin.inlineCallFrame; codeOrigin = codeOrigin.inlineCallFrame->caller) {
     44    for (; codeOrigin.inlineCallFrame; codeOrigin = codeOrigin.inlineCallFrame->directCaller) {
    4545        CodeBlock* codeBlock = codeOrigin.inlineCallFrame->baselineCodeBlock();
    4646        if (codeBlock->jitType() == JSC::JITCode::BaselineJIT)

  • trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp

    r189523 r190220  
    13071307   
    13081308    bool didTryToEnterIntoInlinedLoops = false;
    1309     for (InlineCallFrame* inlineCallFrame = exit->m_codeOrigin.inlineCallFrame; inlineCallFrame; inlineCallFrame = inlineCallFrame->caller.inlineCallFrame) {
     1309    for (InlineCallFrame* inlineCallFrame = exit->m_codeOrigin.inlineCallFrame; inlineCallFrame; inlineCallFrame = inlineCallFrame->directCaller.inlineCallFrame) {
    13101310        if (inlineCallFrame->executable->didTryToEnterInLoop()) {
    13111311            didTryToEnterIntoInlinedLoops = true;

  • trunk/Source/JavaScriptCore/dfg/DFGPreciseLocalClobberize.h

    r184776 r190220  
    112112        case ForwardVarargs:
    113113        case CallForwardVarargs:
    114         case ConstructForwardVarargs: {
     114        case ConstructForwardVarargs:
     115        case TailCallForwardVarargs:
     116        case TailCallForwardVarargsInlinedCaller: {
    115117            InlineCallFrame* inlineCallFrame = m_node->child1()->origin.semantic.inlineCallFrame;
    116118            if (!inlineCallFrame) {
    … …  
    139141       
    140142            // Read all of the inline arguments and call frame headers that we didn't already capture.
    141             for (InlineCallFrame* inlineCallFrame = m_node->origin.semantic.inlineCallFrame; inlineCallFrame; inlineCallFrame = inlineCallFrame->caller.inlineCallFrame) {
     143            for (InlineCallFrame* inlineCallFrame = m_node->origin.semantic.inlineCallFrame; inlineCallFrame; inlineCallFrame = inlineCallFrame->getCallerInlineFrameSkippingDeadFrames()) {
    142144                for (unsigned i = inlineCallFrame->arguments.size(); i-- > 1;)
    143145                    m_read(VirtualRegister(inlineCallFrame->stackOffset + virtualRegisterForArgument(i).offset()));

  • trunk/Source/JavaScriptCore/dfg/DFGPredictionPropagationPhase.cpp

    r189279 r190220  
    195195        case GetDirectPname:
    196196        case Call:
     197        case TailCallInlinedCaller:
    197198        case Construct:
    198199        case CallVarargs:
     200        case TailCallVarargsInlinedCaller:
    199201        case ConstructVarargs:
    200202        case CallForwardVarargs:
    201203        case ConstructForwardVarargs:
     204        case TailCallForwardVarargsInlinedCaller:
    202205        case GetGlobalVar:
    203206        case GetGlobalLexicalVariable:
    … …  
    635638        case PutToArguments:
    636639        case Return:
     640        case TailCall:
     641        case TailCallVarargs:
     642        case TailCallForwardVarargs:
    637643        case Throw:
    638644        case PutById:

  • trunk/Source/JavaScriptCore/dfg/DFGSafeToExecute.h

    r190076 r190220  
    217217    case CompareStrictEq:
    218218    case Call:
     219    case TailCallInlinedCaller:
    219220    case Construct:
    220221    case CallVarargs:
     222    case TailCallVarargsInlinedCaller:
     223    case TailCallForwardVarargsInlinedCaller:
    221224    case ConstructVarargs:
    222225    case LoadVarargs:
    … …  
    263266    case Switch:
    264267    case Return:
     268    case TailCall:
     269    case TailCallVarargs:
     270    case TailCallForwardVarargs:
    265271    case Throw:
    266272    case ThrowReferenceError:

  • trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp

    r190129 r190220  
    3131
    3232#include "ArrayPrototype.h"
     33#include "CallFrameShuffler.h"
    3334#include "DFGAbstractInterpreterInlines.h"
    3435#include "DFGCallArrayAllocatorSlowPathGenerator.h"
    … …  
    633634    bool isVarargs = false;
    634635    bool isForwardVarargs = false;
     636    bool isTail = false;
     637    bool isEmulatedTail = false;
    635638    switch (node->op()) {
    636639    case Call:
    637640        callType = CallLinkInfo::Call;
    638641        break;
     642    case TailCall:
     643        callType = CallLinkInfo::TailCall;
     644        isTail = true;
     645        break;
     646    case TailCallInlinedCaller:
     647        callType = CallLinkInfo::Call;
     648        isEmulatedTail = true;
     649        break;
    639650    case Construct:
    640651        callType = CallLinkInfo::Construct;
    … …  
    644655        isVarargs = true;
    645656        break;
     657    case TailCallVarargs:
     658        callType = CallLinkInfo::TailCallVarargs;
     659        isVarargs = true;
     660        isTail = true;
     661        break;
     662    case TailCallVarargsInlinedCaller:
     663        callType = CallLinkInfo::CallVarargs;
     664        isVarargs = true;
     665        isEmulatedTail = true;
     666        break;
    646667    case ConstructVarargs:
    647668        callType = CallLinkInfo::ConstructVarargs;
    … …  
    652673        isForwardVarargs = true;
    653674        break;
     675    case TailCallForwardVarargs:
     676        callType = CallLinkInfo::TailCallVarargs;
     677        isTail = true;
     678        isForwardVarargs = true;
     679        break;
     680    case TailCallForwardVarargsInlinedCaller:
     681        callType = CallLinkInfo::CallVarargs;
     682        isEmulatedTail = true;
     683        isForwardVarargs = true;
     684        break;
    654685    case ConstructForwardVarargs:
    655686        callType = CallLinkInfo::ConstructVarargs;
    … …  
    662693
    663694    Edge calleeEdge = m_jit.graph().child(node, 0);
     695    GPRReg calleeTagGPR;
     696    GPRReg calleePayloadGPR;
     697    CallFrameShuffleData shuffleData;
    664698   
    665699    // Gotta load the arguments somehow. Varargs is trickier.
    … …  
    756790        int numPassedArgs = node->numChildren() - 1;
    757791
    758         m_jit.store32(MacroAssembler::TrustedImm32(numPassedArgs), m_jit.calleeFramePayloadSlot(JSStack::ArgumentCount));
    759        
    760         for (int i = 0; i < numPassedArgs; i++) {
    761             Edge argEdge = m_jit.graph().m_varArgChildren[node->firstChild() + 1 + i];
    762             JSValueOperand arg(this, argEdge);
    763             GPRReg argTagGPR = arg.tagGPR();
    764             GPRReg argPayloadGPR = arg.payloadGPR();
    765             use(argEdge);
    766            
    767             m_jit.store32(argTagGPR, m_jit.calleeArgumentTagSlot(i));
    768             m_jit.store32(argPayloadGPR, m_jit.calleeArgumentPayloadSlot(i));
    769         }
    770     }
    771 
    772     JSValueOperand callee(this, calleeEdge);
    773     GPRReg calleeTagGPR = callee.tagGPR();
    774     GPRReg calleePayloadGPR = callee.payloadGPR();
    775     use(calleeEdge);
    776     m_jit.store32(calleePayloadGPR, m_jit.calleeFramePayloadSlot(JSStack::Callee));
    777     m_jit.store32(calleeTagGPR, m_jit.calleeFrameTagSlot(JSStack::Callee));
    778 
    779     flushRegisters();
     792        if (node->op() == TailCall) {
     793            JSValueOperand callee(this, calleeEdge);
     794            calleeTagGPR = callee.tagGPR();
     795            calleePayloadGPR = callee.payloadGPR();
     796            use(calleeEdge);
     797
     798            shuffleData.numLocals = m_jit.graph().frameRegisterCount();
     799            shuffleData.callee = ValueRecovery::inPair(calleeTagGPR, calleePayloadGPR);
     800            shuffleData.args.resize(numPassedArgs);
     801
     802            for (int i = 0; i < numPassedArgs; ++i) {
     803                Edge argEdge = m_jit.graph().varArgChild(node, i + 1);
     804                GenerationInfo& info = generationInfo(argEdge.node());
     805                use(argEdge);
     806                shuffleData.args[i] = info.recovery(argEdge->virtualRegister());
     807            }
     808        } else {
     809            m_jit.store32(MacroAssembler::TrustedImm32(numPassedArgs), m_jit.calleeFramePayloadSlot(JSStack::ArgumentCount));
     810       
     811            for (int i = 0; i < numPassedArgs; i++) {
     812                Edge argEdge = m_jit.graph().m_varArgChildren[node->firstChild() + 1 + i];
     813                JSValueOperand arg(this, argEdge);
     814                GPRReg argTagGPR = arg.tagGPR();
     815                GPRReg argPayloadGPR = arg.payloadGPR();
     816                use(argEdge);
     817           
     818                m_jit.store32(argTagGPR, m_jit.calleeArgumentTagSlot(i));
     819                m_jit.store32(argPayloadGPR, m_jit.calleeArgumentPayloadSlot(i));
     820            }
     821        }
     822    }
     823
     824    if (node->op() != TailCall) {
     825        JSValueOperand callee(this, calleeEdge);
     826        calleeTagGPR = callee.tagGPR();
     827        calleePayloadGPR = callee.payloadGPR();
     828        use(calleeEdge);
     829        m_jit.store32(calleePayloadGPR, m_jit.calleeFramePayloadSlot(JSStack::Callee));
     830        m_jit.store32(calleeTagGPR, m_jit.calleeFrameTagSlot(JSStack::Callee));
     831
     832        if (!isTail)
     833            flushRegisters();
     834    }
    780835
    781836    GPRFlushedCallResult resultPayload(this);
    … …  
    787842    JITCompiler::JumpList slowPath;
    788843
    789     CallSiteIndex callSite = m_jit.recordCallSiteAndGenerateExceptionHandlingOSRExitIfNeeded(node->origin.semantic, m_stream->size());
     844    CodeOrigin staticOrigin = node->origin.semantic;
     845    ASSERT(!isTail || !staticOrigin.inlineCallFrame || !staticOrigin.inlineCallFrame->getCallerSkippingDeadFrames());
     846    ASSERT(!isEmulatedTail || (staticOrigin.inlineCallFrame && staticOrigin.inlineCallFrame->getCallerSkippingDeadFrames()));
     847    CodeOrigin dynamicOrigin =
     848        isEmulatedTail ? *staticOrigin.inlineCallFrame->getCallerSkippingDeadFrames() : staticOrigin;
     849    CallSiteIndex callSite = m_jit.recordCallSiteAndGenerateExceptionHandlingOSRExitIfNeeded(dynamicOrigin, m_stream->size());
    790850    m_jit.emitStoreCallSiteIndex(callSite);
    791851   
    792852    CallLinkInfo* info = m_jit.codeBlock()->addCallLinkInfo();
    793853
    794     slowPath.append(m_jit.branchIfNotCell(callee.jsValueRegs()));
     854    slowPath.append(m_jit.branchIfNotCell(JSValueRegs(calleeTagGPR, calleePayloadGPR)));
    795855    slowPath.append(m_jit.branchPtrWithPatch(MacroAssembler::NotEqual, calleePayloadGPR, targetToCheck));
    796856
    797     JITCompiler::Call fastCall = m_jit.nearCall();
     857    if (isTail) {
     858        if (node->op() == TailCall) {
     859            info->setFrameShuffleData(shuffleData);
     860            CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
     861        } else {
     862            m_jit.emitRestoreCalleeSaves();
     863            m_jit.prepareForTailCallSlow();
     864        }
     865    }
     866
     867    JITCompiler::Call fastCall = isTail ? m_jit.nearTailCall() : m_jit.nearCall();
    798868
    799869    JITCompiler::Jump done = m_jit.jump();
    … …  
    801871    slowPath.link(&m_jit);
    802872
    803     // Callee payload needs to be in regT0, tag in regT1
    804     if (calleeTagGPR == GPRInfo::regT0) {
    805         if (calleePayloadGPR == GPRInfo::regT1)
    806             m_jit.swap(GPRInfo::regT1, GPRInfo::regT0);
    807         else {
     873    if (node->op() == TailCall) {
     874        CallFrameShuffler callFrameShuffler(m_jit, shuffleData);
     875        callFrameShuffler.setCalleeJSValueRegs(JSValueRegs(
     876            GPRInfo::regT1, GPRInfo::regT0));
     877        callFrameShuffler.prepareForSlowPath();
     878    } else {
     879        // Callee payload needs to be in regT0, tag in regT1
     880        if (calleeTagGPR == GPRInfo::regT0) {
     881            if (calleePayloadGPR == GPRInfo::regT1)
     882                m_jit.swap(GPRInfo::regT1, GPRInfo::regT0);
     883            else {
     884                m_jit.move(calleeTagGPR, GPRInfo::regT1);
     885                m_jit.move(calleePayloadGPR, GPRInfo::regT0);
     886            }
     887        } else {
     888            m_jit.move(calleePayloadGPR, GPRInfo::regT0);
    808889            m_jit.move(calleeTagGPR, GPRInfo::regT1);
    809             m_jit.move(calleePayloadGPR, GPRInfo::regT0);
    810         }
    811     } else {
    812         m_jit.move(calleePayloadGPR, GPRInfo::regT0);
    813         m_jit.move(calleeTagGPR, GPRInfo::regT1);
    814     }
     890        }
     891
     892        if (isTail)
     893            m_jit.emitRestoreCalleeSaves();
     894    }
     895
    815896    m_jit.move(MacroAssembler::TrustedImmPtr(info), GPRInfo::regT2);
    816897    JITCompiler::Call slowCall = m_jit.nearCall();
    … …  
    818899    done.link(&m_jit);
    819900
    820     m_jit.setupResults(resultPayloadGPR, resultTagGPR);
    821 
    822     jsValueResult(resultTagGPR, resultPayloadGPR, node, DataFormatJS, UseChildrenCalledExplicitly);
     901    if (isTail)
     902        m_jit.abortWithReason(JITDidReturnFromTailCall);
     903    else {
     904        m_jit.setupResults(resultPayloadGPR, resultTagGPR);
     905
     906        jsValueResult(resultTagGPR, resultPayloadGPR, node, DataFormatJS, UseChildrenCalledExplicitly);
     907        // After the calls are done, we need to reestablish our stack
     908        // pointer. We rely on this for varargs calls, calls with arity
     909        // mismatch (the callframe is slided) and tail calls.
     910        m_jit.addPtr(TrustedImm32(m_jit.graph().stackPointerOffset() * sizeof(Register)), GPRInfo::callFrameRegister, JITCompiler::stackPointerRegister);
     911    }
    823912
    824913    info->setUpCall(callType, node->origin.semantic, calleePayloadGPR);
    825914    m_jit.addJSCall(fastCall, slowCall, targetToCheck, info);
    826    
    827     // After the calls are done, we need to reestablish our stack
    828     // pointer. We rely on this for varargs calls, calls with arity
    829     // mismatch (the callframe is slided) and tail calls.
    830     m_jit.addPtr(TrustedImm32(m_jit.graph().stackPointerOffset() * sizeof(Register)), GPRInfo::callFrameRegister, JITCompiler::stackPointerRegister);
    831915}
    832916
    … …  
    42544338
    42554339    case Call:
     4340    case TailCall:
     4341    case TailCallInlinedCaller:
    42564342    case Construct:
    42574343    case CallVarargs:
     4344    case TailCallVarargs:
     4345    case TailCallVarargsInlinedCaller:
     4346    case ConstructVarargs:
    42584347    case CallForwardVarargs:
    4259     case ConstructVarargs:
     4348    case TailCallForwardVarargs:
     4349    case TailCallForwardVarargsInlinedCaller:
    42604350    case ConstructForwardVarargs:
    42614351        emitCall(node);

  • trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp

    r190129 r190220  
    3030
    3131#include "ArrayPrototype.h"
     32#include "CallFrameShuffler.h"
    3233#include "DFGAbstractInterpreterInlines.h"
    3334#include "DFGCallArrayAllocatorSlowPathGenerator.h"
    … …  
    614615    bool isVarargs = false;
    615616    bool isForwardVarargs = false;
     617    bool isTail = false;
     618    bool isEmulatedTail = false;
    616619    switch (node->op()) {
    617620    case Call:
    618621        callType = CallLinkInfo::Call;
    619622        break;
     623    case TailCall:
     624        callType = CallLinkInfo::TailCall;
     625        isTail = true;
     626        break;
     627    case TailCallInlinedCaller:
     628        callType = CallLinkInfo::Call;
     629        isEmulatedTail = true;
     630        break;
    620631    case Construct:
    621632        callType = CallLinkInfo::Construct;
    … …  
    625636        isVarargs = true;
    626637        break;
     638    case TailCallVarargs:
     639        callType = CallLinkInfo::TailCallVarargs;
     640        isVarargs = true;
     641        isTail = true;
     642        break;
     643    case TailCallVarargsInlinedCaller:
     644        callType = CallLinkInfo::CallVarargs;
     645        isVarargs = true;
     646        isEmulatedTail = true;
     647        break;
    627648    case ConstructVarargs:
    628649        callType = CallLinkInfo::ConstructVarargs;
    … …  
    637658        isForwardVarargs = true;
    638659        break;
     660    case TailCallForwardVarargs:
     661        callType = CallLinkInfo::TailCallVarargs;
     662        isTail = true;
     663        isForwardVarargs = true;
     664        break;
     665    case TailCallForwardVarargsInlinedCaller:
     666        callType = CallLinkInfo::CallVarargs;
     667        isEmulatedTail = true;
     668        isForwardVarargs = true;
     669        break;
    639670    default:
    640671        DFG_CRASH(m_jit.graph(), node, "bad node type");
    … …  
    642673    }
    643674
    644     Edge calleeEdge = m_jit.graph().child(node, 0);
     675    GPRReg calleeGPR;
     676    CallFrameShuffleData shuffleData;
    645677   
    646678    // Gotta load the arguments somehow. Varargs is trickier.
    … …  
    733765
    734766        m_jit.store32(MacroAssembler::TrustedImm32(numPassedArgs), JITCompiler::calleeFramePayloadSlot(JSStack::ArgumentCount));
    735    
    736         for (int i = 0; i < numPassedArgs; i++) {
    737             Edge argEdge = m_jit.graph().m_varArgChildren[node->firstChild() + 1 + i];
    738             JSValueOperand arg(this, argEdge);
    739             GPRReg argGPR = arg.gpr();
    740             use(argEdge);
    741        
    742             m_jit.store64(argGPR, JITCompiler::calleeArgumentSlot(i));
    743         }
    744     }
    745 
    746     JSValueOperand callee(this, calleeEdge);
    747     GPRReg calleeGPR = callee.gpr();
    748     callee.use();
    749     m_jit.store64(calleeGPR, JITCompiler::calleeFrameSlot(JSStack::Callee));
    750    
    751     flushRegisters();
    752 
    753     GPRFlushedCallResult result(this);
    754     GPRReg resultGPR = result.gpr();
     767
     768        if (node->op() == TailCall) {
     769            Edge calleeEdge = m_jit.graph().child(node, 0);
     770            JSValueOperand callee(this, calleeEdge);
     771            calleeGPR = callee.gpr();
     772            callee.use();
     773
     774            shuffleData.numLocals = m_jit.graph().frameRegisterCount();
     775            shuffleData.callee = ValueRecovery::inGPR(calleeGPR, DataFormatJS);
     776            shuffleData.args.resize(numPassedArgs);
     777
     778            for (int i = 0; i < numPassedArgs; ++i) {
     779                Edge argEdge = m_jit.graph().varArgChild(node, i + 1);
     780                GenerationInfo& info = generationInfo(argEdge.node());
     781                use(argEdge);
     782                shuffleData.args[i] = info.recovery(argEdge->virtualRegister());
     783            }
     784
     785            shuffleData.setupCalleeSaveRegisters(m_jit.codeBlock());
     786        } else {
     787            m_jit.store32(MacroAssembler::TrustedImm32(numPassedArgs), JITCompiler::calleeFramePayloadSlot(JSStack::ArgumentCount));
     788
     789            for (int i = 0; i < numPassedArgs; i++) {
     790                Edge argEdge = m_jit.graph().m_varArgChildren[node->firstChild() + 1 + i];
     791                JSValueOperand arg(this, argEdge);
     792                GPRReg argGPR = arg.gpr();
     793                use(argEdge);
     794               
     795                m_jit.store64(argGPR, JITCompiler::calleeArgumentSlot(i));
     796            }
     797        }
     798    }
     799
     800    if (node->op() != TailCall) {
     801        Edge calleeEdge = m_jit.graph().child(node, 0);
     802        JSValueOperand callee(this, calleeEdge);
     803        calleeGPR = callee.gpr();
     804        callee.use();
     805        m_jit.store64(calleeGPR, JITCompiler::calleeFrameSlot(JSStack::Callee));
     806
     807        flushRegisters();
     808    }
     809
     810    CodeOrigin staticOrigin = node->origin.semantic;
     811    ASSERT(!isTail || !staticOrigin.inlineCallFrame || !staticOrigin.inlineCallFrame->getCallerSkippingDeadFrames());
     812    ASSERT(!isEmulatedTail || (staticOrigin.inlineCallFrame && staticOrigin.inlineCallFrame->getCallerSkippingDeadFrames()));
     813    CodeOrigin dynamicOrigin =
     814    isEmulatedTail ? *staticOrigin.inlineCallFrame->getCallerSkippingDeadFrames() : staticOrigin;
     815
     816    CallSiteIndex callSite = m_jit.recordCallSiteAndGenerateExceptionHandlingOSRExitIfNeeded(dynamicOrigin, m_stream->size());
     817    m_jit.emitStoreCallSiteIndex(callSite);
     818   
     819    CallLinkInfo* callLinkInfo = m_jit.codeBlock()->addCallLinkInfo();
    755820
    756821    JITCompiler::DataLabelPtr targetToCheck;
    757     JITCompiler::Jump slowPath;
    758 
    759     CallSiteIndex callSite = m_jit.recordCallSiteAndGenerateExceptionHandlingOSRExitIfNeeded(node->origin.semantic, m_stream->size());
    760     m_jit.emitStoreCallSiteIndex(callSite);
    761    
    762     CallLinkInfo* callLinkInfo = m_jit.codeBlock()->addCallLinkInfo();
    763    
    764     slowPath = m_jit.branchPtrWithPatch(MacroAssembler::NotEqual, calleeGPR, targetToCheck, MacroAssembler::TrustedImmPtr(0));
    765 
    766     JITCompiler::Call fastCall = m_jit.nearCall();
     822    JITCompiler::Jump slowPath = m_jit.branchPtrWithPatch(MacroAssembler::NotEqual, calleeGPR, targetToCheck, MacroAssembler::TrustedImmPtr(0));
     823
     824    if (isTail) {
     825        if (node->op() == TailCall) {
     826            callLinkInfo->setFrameShuffleData(shuffleData);
     827            CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
     828        } else {
     829            m_jit.emitRestoreCalleeSaves();
     830            m_jit.prepareForTailCallSlow();
     831        }
     832    }
     833
     834    JITCompiler::Call fastCall = isTail ? m_jit.nearTailCall() : m_jit.nearCall();
    767835
    768836    JITCompiler::Jump done = m_jit.jump();
    769    
     837
    770838    slowPath.link(&m_jit);
    771    
    772     m_jit.move(calleeGPR, GPRInfo::regT0); // Callee needs to be in regT0
     839
     840    if (node->op() == TailCall) {
     841        CallFrameShuffler callFrameShuffler(m_jit, shuffleData);
     842        callFrameShuffler.setCalleeJSValueRegs(JSValueRegs(GPRInfo::regT0));
     843        callFrameShuffler.prepareForSlowPath();
     844    } else {
     845        m_jit.move(calleeGPR, GPRInfo::regT0); // Callee needs to be in regT0
     846
     847        if (isTail)
     848            m_jit.emitRestoreCalleeSaves(); // This needs to happen after we moved calleeGPR to regT0
     849    }
     850
    773851    m_jit.move(MacroAssembler::TrustedImmPtr(callLinkInfo), GPRInfo::regT2); // Link info needs to be in regT2
    774852    JITCompiler::Call slowCall = m_jit.nearCall();
    775    
     853
    776854    done.link(&m_jit);
    777    
    778     m_jit.move(GPRInfo::returnValueGPR, resultGPR);
    779    
    780     jsValueResult(resultGPR, m_currentNode, DataFormatJS, UseChildrenCalledExplicitly);
    781    
     855
     856    if (isTail)
     857        m_jit.abortWithReason(JITDidReturnFromTailCall);
     858    else {
     859        GPRFlushedCallResult result(this);
     860        GPRReg resultGPR = result.gpr();
     861        m_jit.move(GPRInfo::returnValueGPR, resultGPR);
     862
     863        jsValueResult(resultGPR, m_currentNode, DataFormatJS, UseChildrenCalledExplicitly);
     864
     865        // After the calls are done, we need to reestablish our stack
     866        // pointer. We rely on this for varargs calls, calls with arity
     867        // mismatch (the callframe is slided) and tail calls.
     868        m_jit.addPtr(TrustedImm32(m_jit.graph().stackPointerOffset() * sizeof(Register)), GPRInfo::callFrameRegister, JITCompiler::stackPointerRegister);
     869    }
     870
    782871    callLinkInfo->setUpCall(callType, m_currentNode->origin.semantic,  calleeGPR);   
    783872    m_jit.addJSCall(fastCall, slowCall, targetToCheck, callLinkInfo);
    784 
    785     // After the calls are done, we need to reestablish our stack
    786     // pointer. We rely on this for varargs calls, calls with arity
    787     // mismatch (the callframe is slided) and tail calls.
    788     m_jit.addPtr(TrustedImm32(m_jit.graph().stackPointerOffset() * sizeof(Register)), GPRInfo::callFrameRegister, JITCompiler::stackPointerRegister);
    789873}
    790874
    … …  
    42494333
    42504334    case Call:
     4335    case TailCall:
     4336    case TailCallInlinedCaller:
    42514337    case Construct:
    42524338    case CallVarargs:
     4339    case TailCallVarargs:
     4340    case TailCallVarargsInlinedCaller:
    42534341    case CallForwardVarargs:
    42544342    case ConstructVarargs:
    42554343    case ConstructForwardVarargs:
     4344    case TailCallForwardVarargs:
     4345    case TailCallForwardVarargsInlinedCaller:
    42564346        emitCall(node);
    42574347        break;
    4258        
     4348
    42594349    case LoadVarargs: {
    42604350        LoadVarargsData* data = node->loadVarargsData();

  • trunk/Source/JavaScriptCore/dfg/DFGValidate.cpp

    r190076 r190220  
    571571                case ForwardVarargs:
    572572                case CallForwardVarargs:
     573                case TailCallForwardVarargs:
     574                case TailCallForwardVarargsInlinedCaller:
    573575                case ConstructForwardVarargs:
    574576                case GetMyArgumentByVal:

  • trunk/Source/JavaScriptCore/dfg/DFGVarargsForwardingPhase.cpp

    r184288 r190220  
    136136            case CallVarargs:
    137137            case ConstructVarargs:
     138            case TailCallVarargs:
     139            case TailCallVarargsInlinedCaller:
    138140                if (node->child1() == candidate || node->child3() == candidate) {
    139141                    if (verbose)
    … …  
    283285                node->setOpAndDefaultFlags(ConstructForwardVarargs);
    284286                break;
    285                
     287
     288            case TailCallVarargs:
     289                if (node->child2() != candidate)
     290                    break;
     291                node->setOpAndDefaultFlags(TailCallForwardVarargs);
     292                break;
     293
     294            case TailCallVarargsInlinedCaller:
     295                if (node->child2() != candidate)
     296                    break;
     297                node->setOpAndDefaultFlags(TailCallForwardVarargsInlinedCaller);
     298                break;
     299
    286300            case SetLocal:
    287301                // This is super odd. We don't have to do anything here, since in DFG IR, the phantom

  • trunk/Source/JavaScriptCore/interpreter/CallFrame.cpp

    r188932 r190220  
    9696        CodeOrigin codeOrigin = this->codeOrigin();
    9797        for (InlineCallFrame* inlineCallFrame = codeOrigin.inlineCallFrame; inlineCallFrame;) {
    98             codeOrigin = inlineCallFrame->caller;
     98            codeOrigin = inlineCallFrame->directCaller;
    9999            inlineCallFrame = codeOrigin.inlineCallFrame;
    100100        }

  • trunk/Source/JavaScriptCore/interpreter/StackVisitor.cpp

    r189995 r190220  
    6161    if (m_frame.isInlinedFrame()) {
    6262        InlineCallFrame* inlineCallFrame = m_frame.inlineCallFrame();
    63         CodeOrigin* callerCodeOrigin = &inlineCallFrame->caller;
    64         readInlinedFrame(m_frame.callFrame(), callerCodeOrigin);
     63        CodeOrigin* callerCodeOrigin = inlineCallFrame->getCallerSkippingDeadFrames();
     64        if (!callerCodeOrigin) {
     65            while (inlineCallFrame) {
     66                readInlinedFrame(m_frame.callFrame(), &inlineCallFrame->directCaller);
     67                inlineCallFrame = m_frame.inlineCallFrame();
     68            }
     69            m_frame.m_VMEntryFrame = m_frame.m_CallerVMEntryFrame;
     70            readFrame(m_frame.callerFrame());
     71        } else
     72            readInlinedFrame(m_frame.callFrame(), callerCodeOrigin);
    6573        return;
    6674    }
Note: See TracChangeset for help on using the changeset viewer.