Changeset 222563 in webkit

Timestamp:

Sep 27, 2017 11:37:41 AM (7 years ago)

Author:

Yusuke Suzuki

Message:

[DFG] Support ArrayPush with multiple args
​https://bugs.webkit.org/show_bug.cgi?id=175823

Reviewed by Saam Barati.

JSTests:

• microbenchmarks/array-push-0.js: Added.

(arrayPush0):

• microbenchmarks/array-push-1.js: Added.

(arrayPush1):

• microbenchmarks/array-push-2.js: Added.

(arrayPush2):

• microbenchmarks/array-push-3.js: Added.

(arrayPush3):

• stress/array-push-multiple-contiguous.js: Added.

(shouldBe):
(test):

• stress/array-push-multiple-double-nan.js: Added.

(shouldBe):
(test):

• stress/array-push-multiple-double.js: Added.

(shouldBe):
(test):

• stress/array-push-multiple-int32.js: Added.

(shouldBe):
(test):

• stress/array-push-multiple-many-contiguous.js: Added.

(shouldBe):
(test):

• stress/array-push-multiple-many-double.js: Added.

(shouldBe):
(test):

• stress/array-push-multiple-many-int32.js: Added.

(shouldBe):
(test):

• stress/array-push-multiple-many-storage.js: Added.

(shouldBe):
(test):

• stress/array-push-multiple-storage.js: Added.

(shouldBe):
(test):

Source/JavaScriptCore:

This patch implements ArrayPush(with multiple arguments) in DFG and FTL. Previously, they are not handled
by ArrayPush. Then they go to generic direct call to Array#push and it does in slow path. This patch
extends ArrayPush to push multiple arguments in a bulk push manner.

The problem of ArrayPush is that we need to perform ArrayPush atomically: If OSR exit occurs in the middle
of ArrayPush, we incorrectly push pushed elements twice. Once we start pushing values, we should not exit.
But we do not want to iterate elements twice, once for type checks and once for actually pushing it. It
could move elements between registers and memory back and forth.

This patch achieves the above goal by separating type checks from ArrayPush. When starting ArrayPush, type
checks for elements are already done by separately emitted Check nodes.

We also add JSArray::pushInline for DFG operations just calling JSArray::push. And we also use it in
arrayProtoFuncPush's fast path.

This patch significantly improves performance of push(multiple args).

baseline patched

Microbenchmarks:

array-push-0 461.8455+-28.9995 ^{151.3438+-6.5653} definitely 3.0516x faster
array-push-1 133.8845+-7.0349 ? 136.1775+-5.8327 ? might be 1.0171x slower
array-push-2 675.6555+-13.4645 ^{145.8747+-6.4621} definitely 4.6318x faster
array-push-3 849.5284+-15.2540 ^{253.4421+-9.1249} definitely 3.3520x faster

baseline patched

SixSpeed:

spread-literal.es5 90.3482+-6.6514 ^{24.8123+-2.3304} definitely 3.6413x faster

• dfg/DFGByteCodeParser.cpp:

(JSC::DFG::ByteCodeParser::handleIntrinsicCall):

• dfg/DFGFixupPhase.cpp:

(JSC::DFG::FixupPhase::fixupNode):

• dfg/DFGNodeType.h:
• dfg/DFGOperations.cpp:
• dfg/DFGOperations.h:
• dfg/DFGSpeculativeJIT.cpp:

(JSC::DFG::SpeculativeJIT::compileArrayPush):

• dfg/DFGSpeculativeJIT.h:

(JSC::DFG::SpeculativeJIT::callOperation):

• dfg/DFGSpeculativeJIT32_64.cpp:

(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGSpeculativeJIT64.cpp:

(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGStoreBarrierInsertionPhase.cpp:
• ftl/FTLLowerDFGToB3.cpp:

(JSC::FTL::DFG::LowerDFGToB3::compileArrayPush):

• jit/JITOperations.h:
• runtime/ArrayPrototype.cpp:

(JSC::arrayProtoFuncPush):

• runtime/JSArray.cpp:

(JSC::JSArray::push):

• runtime/JSArray.h:
• runtime/JSArrayInlines.h:

(JSC::JSArray::pushInline):

Location:

trunk

Files:

• JSTests/ChangeLog (modified) (1 diff)
• JSTests/microbenchmarks/array-push-0.js (added)
• JSTests/microbenchmarks/array-push-1.js (added)
• JSTests/microbenchmarks/array-push-2.js (added)
• JSTests/microbenchmarks/array-push-3.js (added)
• JSTests/stress/array-push-multiple-contiguous.js (added)
• JSTests/stress/array-push-multiple-double-nan.js (added)
• JSTests/stress/array-push-multiple-double.js (added)
• JSTests/stress/array-push-multiple-int32.js (added)
• JSTests/stress/array-push-multiple-many-contiguous.js (added)
• JSTests/stress/array-push-multiple-many-double.js (added)
• JSTests/stress/array-push-multiple-many-int32.js (added)
• JSTests/stress/array-push-multiple-many-storage.js (added)
• JSTests/stress/array-push-multiple-storage.js (added)
• Source/JavaScriptCore/ChangeLog (modified) (1 diff)
• Source/JavaScriptCore/dfg/DFGByteCodeParser.cpp (modified) (2 diffs)
• Source/JavaScriptCore/dfg/DFGFixupPhase.cpp (modified) (1 diff)
• Source/JavaScriptCore/dfg/DFGNodeType.h (modified) (1 diff)
• Source/JavaScriptCore/dfg/DFGOperations.cpp (modified) (2 diffs)
• Source/JavaScriptCore/dfg/DFGOperations.h (modified) (1 diff)
• Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp (modified) (1 diff)
• Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h (modified) (3 diffs)
• Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp (modified) (1 diff)
• Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp (modified) (1 diff)
• Source/JavaScriptCore/dfg/DFGStoreBarrierInsertionPhase.cpp (modified) (1 diff)
• Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp (modified) (1 diff)
• Source/JavaScriptCore/jit/JITOperations.h (modified) (1 diff)
• Source/JavaScriptCore/runtime/ArrayPrototype.cpp (modified) (1 diff)
• Source/JavaScriptCore/runtime/JSArray.cpp (modified) (2 diffs)
• Source/JavaScriptCore/runtime/JSArray.h (modified) (2 diffs)
• Source/JavaScriptCore/runtime/JSArrayInlines.h (modified) (1 diff)

trunk/JSTests/ChangeLog

@@ +1 @@
+2017-09-25  Yusuke Suzuki  <utatane.tea@gmail.com>
+
+        [DFG] Support ArrayPush with multiple args
+        https://bugs.webkit.org/show_bug.cgi?id=175823
+
+        Reviewed by Saam Barati.
+
+        * microbenchmarks/array-push-0.js: Added.
+        (arrayPush0):
+        * microbenchmarks/array-push-1.js: Added.
+        (arrayPush1):
+        * microbenchmarks/array-push-2.js: Added.
+        (arrayPush2):
+        * microbenchmarks/array-push-3.js: Added.
+        (arrayPush3):
+        * stress/array-push-multiple-contiguous.js: Added.
+        (shouldBe):
+        (test):
+        * stress/array-push-multiple-double-nan.js: Added.
+        (shouldBe):
+        (test):
+        * stress/array-push-multiple-double.js: Added.
+        (shouldBe):
+        (test):
+        * stress/array-push-multiple-int32.js: Added.
+        (shouldBe):
+        (test):
+        * stress/array-push-multiple-many-contiguous.js: Added.
+        (shouldBe):
+        (test):
+        * stress/array-push-multiple-many-double.js: Added.
+        (shouldBe):
+        (test):
+        * stress/array-push-multiple-many-int32.js: Added.
+        (shouldBe):
+        (test):
+        * stress/array-push-multiple-many-storage.js: Added.
+        (shouldBe):
+        (test):
+        * stress/array-push-multiple-storage.js: Added.
+        (shouldBe):
+        (test):
+
 2017-09-26  Commit Queue  <commit-queue@webkit.org>
 

trunk/Source/JavaScriptCore/ChangeLog

@@ +1 @@
+2017-09-25  Yusuke Suzuki  <utatane.tea@gmail.com>
+
+        [DFG] Support ArrayPush with multiple args
+        https://bugs.webkit.org/show_bug.cgi?id=175823
+
+        Reviewed by Saam Barati.
+
+        This patch implements ArrayPush(with multiple arguments) in DFG and FTL. Previously, they are not handled
+        by ArrayPush. Then they go to generic direct call to Array#push and it does in slow path. This patch
+        extends ArrayPush to push multiple arguments in a bulk push manner.
+
+        The problem of ArrayPush is that we need to perform ArrayPush atomically: If OSR exit occurs in the middle
+        of ArrayPush, we incorrectly push pushed elements twice. Once we start pushing values, we should not exit.
+        But we do not want to iterate elements twice, once for type checks and once for actually pushing it. It
+        could move elements between registers and memory back and forth.
+
+        This patch achieves the above goal by separating type checks from ArrayPush. When starting ArrayPush, type
+        checks for elements are already done by separately emitted Check nodes.
+
+        We also add JSArray::pushInline for DFG operations just calling JSArray::push. And we also use it in
+        arrayProtoFuncPush's fast path.
+
+        This patch significantly improves performance of `push(multiple args)`.
+
+                                            baseline                  patched
+            Microbenchmarks:
+                array-push-0            461.8455+-28.9995    ^    151.3438+-6.5653        ^ definitely 3.0516x faster
+                array-push-1            133.8845+-7.0349     ?    136.1775+-5.8327        ? might be 1.0171x slower
+                array-push-2            675.6555+-13.4645    ^    145.8747+-6.4621        ^ definitely 4.6318x faster
+                array-push-3            849.5284+-15.2540    ^    253.4421+-9.1249        ^ definitely 3.3520x faster
+
+                                            baseline                  patched
+            SixSpeed:
+                spread-literal.es5       90.3482+-6.6514     ^     24.8123+-2.3304        ^ definitely 3.6413x faster
+
+        * dfg/DFGByteCodeParser.cpp:
+        (JSC::DFG::ByteCodeParser::handleIntrinsicCall):
+        * dfg/DFGFixupPhase.cpp:
+        (JSC::DFG::FixupPhase::fixupNode):
+        * dfg/DFGNodeType.h:
+        * dfg/DFGOperations.cpp:
+        * dfg/DFGOperations.h:
+        * dfg/DFGSpeculativeJIT.cpp:
+        (JSC::DFG::SpeculativeJIT::compileArrayPush):
+        * dfg/DFGSpeculativeJIT.h:
+        (JSC::DFG::SpeculativeJIT::callOperation):
+        * dfg/DFGSpeculativeJIT32_64.cpp:
+        (JSC::DFG::SpeculativeJIT::compile):
+        * dfg/DFGSpeculativeJIT64.cpp:
+        (JSC::DFG::SpeculativeJIT::compile):
+        * dfg/DFGStoreBarrierInsertionPhase.cpp:
+        * ftl/FTLLowerDFGToB3.cpp:
+        (JSC::FTL::DFG::LowerDFGToB3::compileArrayPush):
+        * jit/JITOperations.h:
+        * runtime/ArrayPrototype.cpp:
+        (JSC::arrayProtoFuncPush):
+        * runtime/JSArray.cpp:
+        (JSC::JSArray::push):
+        * runtime/JSArray.h:
+        * runtime/JSArrayInlines.h:
+        (JSC::JSArray::pushInline):
+
 2017-09-26  Joseph Pecoraro  <pecoraro@apple.com>
 

trunk/Source/JavaScriptCore/dfg/DFGByteCodeParser.cpp

@@ -2335 +2335 @@
         
     case ArrayPushIntrinsic: {
-        if (argumentCountIncludingThis != 2)
+        if (static_cast<unsigned>(argumentCountIncludingThis) >= MIN_SPARSE_ARRAY_INDEX)
             return false;
         
@@ -2347 +2347 @@
         case Array::ArrayStorage: {
             insertChecks();
-            Node* arrayPush = addToGraph(ArrayPush, OpInfo(arrayMode.asWord()), OpInfo(prediction), get(virtualRegisterForArgument(0, registerOffset)), get(virtualRegisterForArgument(1, registerOffset)));
+
+            addVarArgChild(nullptr); // For storage.
+            for (int i = 0; i < argumentCountIncludingThis; ++i)
+                addVarArgChild(get(virtualRegisterForArgument(i, registerOffset)));
+            Node* arrayPush = addToGraph(Node::VarArg, ArrayPush, OpInfo(arrayMode.asWord()), OpInfo(prediction));
             set(VirtualRegister(resultOperand), arrayPush);
             

trunk/Source/JavaScriptCore/dfg/DFGFixupPhase.cpp

@@ -1010 +1010 @@
             // array-related - so if refine() turned this into a "Generic" ArrayPush then
             // that would break things.
-            node->setArrayMode(
-                node->arrayMode().refine(
-                    m_graph, node,
-                    node->child1()->prediction() & SpecCell,
-                    SpecInt32Only,
-                    node->child2()->prediction()));
-            blessArrayOperation(node->child1(), Edge(), node->child3());
-            fixEdge<KnownCellUse>(node->child1());
-            
-            switch (node->arrayMode().type()) {
-            case Array::Int32:
-                fixEdge<Int32Use>(node->child2());
-                break;
-            case Array::Double:
-                fixEdge<DoubleRepRealUse>(node->child2());
-                break;
-            case Array::Contiguous:
-            case Array::ArrayStorage:
-                speculateForBarrier(node->child2());
-                break;
-            default:
-                break;
+            Edge& storageEdge = m_graph.varArgChild(node, 0);
+            Edge& arrayEdge = m_graph.varArgChild(node, 1);
+            unsigned elementOffset = 2;
+            unsigned elementCount = node->numChildren() - elementOffset;
+            for (unsigned i = 0; i < elementCount; ++i) {
+                Edge& element = m_graph.varArgChild(node, i + elementOffset);
+                node->setArrayMode(
+                    node->arrayMode().refine(
+                        m_graph, node,
+                        arrayEdge->prediction() & SpecCell,
+                        SpecInt32Only,
+                        element->prediction()));
+            }
+            blessArrayOperation(arrayEdge, Edge(), storageEdge);
+            fixEdge<KnownCellUse>(arrayEdge);
+
+            // Convert `array.push()` to GetArrayLength.
+            ASSERT(node->arrayMode().supportsSelfLength());
+            if (!elementCount) {
+                node->setOpAndDefaultFlags(GetArrayLength);
+                node->child1() = arrayEdge;
+                node->child2() = storageEdge;
+                fixEdge<KnownCellUse>(node->child1());
+                break;
+            }
+
+            // We do not want to perform osr exit and retry for ArrayPush. We insert Check with appropriate type,
+            // and ArrayPush uses the edge as known typed edge. Therefore, ArrayPush do not need to perform type checks.
+            for (unsigned i = 0; i < elementCount; ++i) {
+                Edge& element = m_graph.varArgChild(node, i + elementOffset);
+                switch (node->arrayMode().type()) {
+                case Array::Int32:
+                    insertCheck<Int32Use>(element.node());
+                    fixEdge<KnownInt32Use>(element);
+                    break;
+                case Array::Double:
+                    insertCheck<DoubleRepRealUse>(element.node());
+                    fixEdge<DoubleRepUse>(element);
+                    break;
+                case Array::Contiguous:
+                case Array::ArrayStorage:
+                    speculateForBarrier(element);
+                    break;
+                default:
+                    RELEASE_ASSERT_NOT_REACHED();
+                }
+                ASSERT(shouldNotHaveTypeCheck(element.useKind()));
             }
             break;

trunk/Source/JavaScriptCore/dfg/DFGNodeType.h

@@ -266 +266 @@
     \
     /* Optimizations for array mutation. */\
-    macro(ArrayPush, NodeResultJS | NodeMustGenerate) \
+    macro(ArrayPush, NodeResultJS | NodeMustGenerate | NodeHasVarArgs) \
     macro(ArrayPop, NodeResultJS | NodeMustGenerate) \
     macro(ArraySlice, NodeResultJS | NodeMustGenerate | NodeHasVarArgs) \

trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp

@@ -870 +870 @@
     NativeCallFrameTracer tracer(vm, exec);
     
-    array->push(exec, JSValue::decode(encodedValue));
+    array->pushInline(exec, JSValue::decode(encodedValue));
     return JSValue::encode(jsNumber(array->length()));
 }
@@ -879 +879 @@
     NativeCallFrameTracer tracer(vm, exec);
     
-    array->push(exec, JSValue(JSValue::EncodeAsDouble, value));
+    array->pushInline(exec, JSValue(JSValue::EncodeAsDouble, value));
+    return JSValue::encode(jsNumber(array->length()));
+}
+
+EncodedJSValue JIT_OPERATION operationArrayPushMultiple(ExecState* exec, JSArray* array, void* buffer, int32_t elementCount)
+{
+    VM& vm = exec->vm();
+    NativeCallFrameTracer tracer(&vm, exec);
+    auto scope = DECLARE_THROW_SCOPE(vm);
+
+    // We assume that multiple JSArray::push calls with ArrayWithInt32/ArrayWithContiguous do not cause JS traps.
+    // If it can cause any JS interactions, we can call the caller JS function of this function and overwrite the
+    // content of ScratchBuffer. If the IndexingType is now ArrayWithInt32/ArrayWithContiguous, we can ensure
+    // that there is no indexed accessors in this object and its prototype chain.
+    //
+    // ArrayWithArrayStorage is also OK. It can have indexed accessors. But if you define an indexed accessor, the array's length
+    // becomes larger than that index. So Array#push never overlaps with this accessor. So accessors are never called unless
+    // the IndexingType is ArrayWithSlowPutArrayStorage which could have an indexed accessor in a prototype chain.
+    RELEASE_ASSERT(!shouldUseSlowPut(array->indexingType()));
+
+    EncodedJSValue* values = static_cast<EncodedJSValue*>(buffer);
+    for (int32_t i = 0; i < elementCount; ++i) {
+        array->pushInline(exec, JSValue::decode(values[i]));
+        RETURN_IF_EXCEPTION(scope, encodedJSValue());
+    }
+    return JSValue::encode(jsNumber(array->length()));
+}
+
+EncodedJSValue JIT_OPERATION operationArrayPushDoubleMultiple(ExecState* exec, JSArray* array, void* buffer, int32_t elementCount)
+{
+    VM& vm = exec->vm();
+    NativeCallFrameTracer tracer(&vm, exec);
+    auto scope = DECLARE_THROW_SCOPE(vm);
+
+    // We assume that multiple JSArray::push calls with ArrayWithDouble do not cause JS traps.
+    // If it can cause any JS interactions, we can call the caller JS function of this function and overwrite the
+    // content of ScratchBuffer. If the IndexingType is now ArrayWithDouble, we can ensure
+    // that there is no indexed accessors in this object and its prototype chain.
+    ASSERT(array->indexingType() == ArrayWithDouble);
+
+    double* values = static_cast<double*>(buffer);
+    for (int32_t i = 0; i < elementCount; ++i) {
+        array->pushInline(exec, JSValue(JSValue::EncodeAsDouble, values[i]));
+        RETURN_IF_EXCEPTION(scope, encodedJSValue());
+    }
     return JSValue::encode(jsNumber(array->length()));
 }

trunk/Source/JavaScriptCore/dfg/DFGOperations.h

@@ -137 +137 @@
 void JIT_OPERATION operationDefineAccessorPropertySymbol(ExecState*, JSObject*, Symbol*, JSObject*, JSObject*, int32_t) WTF_INTERNAL;
 EncodedJSValue JIT_OPERATION operationArrayPush(ExecState*, EncodedJSValue encodedValue, JSArray*) WTF_INTERNAL;
+EncodedJSValue JIT_OPERATION operationArrayPushMultiple(ExecState*, JSArray*, void* buffer, int32_t elementCount) WTF_INTERNAL;
 EncodedJSValue JIT_OPERATION operationArrayPushDouble(ExecState*, double value, JSArray*) WTF_INTERNAL;
+EncodedJSValue JIT_OPERATION operationArrayPushDoubleMultiple(ExecState*, JSArray*, void* buffer, int32_t elementCount) WTF_INTERNAL;
 EncodedJSValue JIT_OPERATION operationArrayPop(ExecState*, JSArray*) WTF_INTERNAL;
 EncodedJSValue JIT_OPERATION operationArrayPopAndRecoverLength(ExecState*, JSArray*) WTF_INTERNAL;

trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp

@@ -7867 +7867 @@
 }
 
+void SpeculativeJIT::compileArrayPush(Node* node)
+{
+    ASSERT(node->arrayMode().isJSArray());
+
+    Edge& storageEdge = m_jit.graph().varArgChild(node, 0);
+    Edge& arrayEdge = m_jit.graph().varArgChild(node, 1);
+
+    SpeculateCellOperand base(this, arrayEdge);
+    GPRTemporary storageLength(this);
+
+    GPRReg baseGPR = base.gpr();
+    GPRReg storageLengthGPR = storageLength.gpr();
+
+    StorageOperand storage(this, storageEdge);
+    GPRReg storageGPR = storage.gpr();
+    unsigned elementOffset = 2;
+    unsigned elementCount = node->numChildren() - elementOffset;
+
+#if USE(JSVALUE32_64)
+    GPRTemporary tag(this);
+    GPRReg tagGPR = tag.gpr();
+    JSValueRegs resultRegs { tagGPR, storageLengthGPR };
+#else
+    JSValueRegs resultRegs { storageLengthGPR };
+#endif
+
+    auto getStorageBufferAddress = [&] (GPRReg storageGPR, GPRReg indexGPR, int32_t offset, GPRReg bufferGPR) {
+#if USE(JSVALUE32_64)
+        static_assert(sizeof(JSValue) == 8 && 1 << 3 == 8, "This is strongly assumed in the code below.");
+        m_jit.move(indexGPR, bufferGPR);
+        m_jit.lshift32(TrustedImm32(3), bufferGPR);
+        m_jit.add32(storageGPR, bufferGPR);
+        if (offset)
+            m_jit.add32(TrustedImm32(offset), bufferGPR);
+#else
+        m_jit.getEffectiveAddress64(MacroAssembler::BaseIndex(storageGPR, indexGPR, MacroAssembler::TimesEight, offset), bufferGPR);
+#endif
+    };
+
+    switch (node->arrayMode().type()) {
+    case Array::Int32:
+    case Array::Contiguous: {
+        if (elementCount == 1) {
+            Edge& element = m_jit.graph().varArgChild(node, elementOffset);
+            JSValueOperand value(this, element, ManualOperandSpeculation);
+            JSValueRegs valueRegs = value.jsValueRegs();
+
+            if (node->arrayMode().type() == Array::Int32)
+                RELEASE_ASSERT(!needsTypeCheck(element, SpecInt32Only));
+
+            m_jit.load32(MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()), storageLengthGPR);
+            MacroAssembler::Jump slowPath = m_jit.branch32(MacroAssembler::AboveOrEqual, storageLengthGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
+            m_jit.storeValue(valueRegs, MacroAssembler::BaseIndex(storageGPR, storageLengthGPR, MacroAssembler::TimesEight));
+            m_jit.add32(TrustedImm32(1), storageLengthGPR);
+            m_jit.store32(storageLengthGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
+            m_jit.boxInt32(storageLengthGPR, resultRegs);
+
+            addSlowPathGenerator(
+                slowPathCall(slowPath, this, operationArrayPush, resultRegs, valueRegs, baseGPR));
+
+            jsValueResult(resultRegs, node);
+            return;
+        }
+
+        GPRTemporary buffer(this);
+        GPRReg bufferGPR = buffer.gpr();
+
+        m_jit.load32(MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()), storageLengthGPR);
+        m_jit.move(storageLengthGPR, bufferGPR);
+        m_jit.add32(TrustedImm32(elementCount), bufferGPR);
+        MacroAssembler::Jump slowPath = m_jit.branch32(MacroAssembler::Above, bufferGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
+
+        m_jit.store32(bufferGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
+        getStorageBufferAddress(storageGPR, storageLengthGPR, 0, bufferGPR);
+        m_jit.add32(TrustedImm32(elementCount), storageLengthGPR);
+        m_jit.boxInt32(storageLengthGPR, resultRegs);
+        auto storageDone = m_jit.jump();
+
+        slowPath.link(&m_jit);
+
+        size_t scratchSize = sizeof(EncodedJSValue) * elementCount;
+        ScratchBuffer* scratchBuffer = m_jit.vm()->scratchBufferForSize(scratchSize);
+        m_jit.move(TrustedImmPtr(static_cast<EncodedJSValue*>(scratchBuffer->dataBuffer())), bufferGPR);
+        m_jit.move(TrustedImmPtr(scratchBuffer->addressOfActiveLength()), storageLengthGPR);
+        m_jit.storePtr(TrustedImmPtr(scratchSize), MacroAssembler::Address(storageLengthGPR));
+
+        storageDone.link(&m_jit);
+        for (unsigned elementIndex = 0; elementIndex < elementCount; ++elementIndex) {
+            Edge& element = m_jit.graph().varArgChild(node, elementIndex + elementOffset);
+            JSValueOperand value(this, element, ManualOperandSpeculation);
+            JSValueRegs valueRegs = value.jsValueRegs();
+
+            if (node->arrayMode().type() == Array::Int32)
+                RELEASE_ASSERT(!needsTypeCheck(element, SpecInt32Only));
+
+            m_jit.storeValue(valueRegs, MacroAssembler::Address(bufferGPR, sizeof(EncodedJSValue) * elementIndex));
+            value.use();
+        }
+
+        MacroAssembler::Jump fastPath = m_jit.branchPtr(MacroAssembler::NotEqual, bufferGPR, TrustedImmPtr(static_cast<EncodedJSValue*>(scratchBuffer->dataBuffer())));
+
+        addSlowPathGenerator(slowPathCall(m_jit.jump(), this, operationArrayPushMultiple, resultRegs, baseGPR, bufferGPR, TrustedImm32(elementCount)));
+
+        m_jit.move(TrustedImmPtr(scratchBuffer->addressOfActiveLength()), bufferGPR);
+        m_jit.storePtr(TrustedImmPtr(nullptr), MacroAssembler::Address(bufferGPR));
+
+        base.use();
+        storage.use();
+
+        fastPath.link(&m_jit);
+        jsValueResult(resultRegs, node, DataFormatJS, UseChildrenCalledExplicitly);
+        return;
+    }
+
+    case Array::Double: {
+        if (elementCount == 1) {
+            Edge& element = m_jit.graph().varArgChild(node, elementOffset);
+            SpeculateDoubleOperand value(this, element);
+            FPRReg valueFPR = value.fpr();
+
+            RELEASE_ASSERT(!needsTypeCheck(element, SpecDoubleReal));
+
+            m_jit.load32(MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()), storageLengthGPR);
+            MacroAssembler::Jump slowPath = m_jit.branch32(MacroAssembler::AboveOrEqual, storageLengthGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
+            m_jit.storeDouble(valueFPR, MacroAssembler::BaseIndex(storageGPR, storageLengthGPR, MacroAssembler::TimesEight));
+            m_jit.add32(TrustedImm32(1), storageLengthGPR);
+            m_jit.store32(storageLengthGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
+            m_jit.boxInt32(storageLengthGPR, resultRegs);
+
+            addSlowPathGenerator(
+                slowPathCall(slowPath, this, operationArrayPushDouble, resultRegs, valueFPR, baseGPR));
+
+            jsValueResult(resultRegs, node);
+            return;
+        }
+
+        GPRTemporary buffer(this);
+        GPRReg bufferGPR = buffer.gpr();
+
+        m_jit.load32(MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()), storageLengthGPR);
+        m_jit.move(storageLengthGPR, bufferGPR);
+        m_jit.add32(TrustedImm32(elementCount), bufferGPR);
+        MacroAssembler::Jump slowPath = m_jit.branch32(MacroAssembler::Above, bufferGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
+
+        m_jit.store32(bufferGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
+        getStorageBufferAddress(storageGPR, storageLengthGPR, 0, bufferGPR);
+        m_jit.add32(TrustedImm32(elementCount), storageLengthGPR);
+        m_jit.boxInt32(storageLengthGPR, resultRegs);
+        auto storageDone = m_jit.jump();
+
+        slowPath.link(&m_jit);
+
+        size_t scratchSize = sizeof(double) * elementCount;
+        ScratchBuffer* scratchBuffer = m_jit.vm()->scratchBufferForSize(scratchSize);
+        m_jit.move(TrustedImmPtr(static_cast<EncodedJSValue*>(scratchBuffer->dataBuffer())), bufferGPR);
+        m_jit.move(TrustedImmPtr(scratchBuffer->addressOfActiveLength()), storageLengthGPR);
+        m_jit.storePtr(TrustedImmPtr(scratchSize), MacroAssembler::Address(storageLengthGPR));
+
+        storageDone.link(&m_jit);
+        for (unsigned elementIndex = 0; elementIndex < elementCount; ++elementIndex) {
+            Edge& element = m_jit.graph().varArgChild(node, elementIndex + elementOffset);
+            SpeculateDoubleOperand value(this, element);
+            FPRReg valueFPR = value.fpr();
+
+            RELEASE_ASSERT(!needsTypeCheck(element, SpecDoubleReal));
+
+            m_jit.storeDouble(valueFPR, MacroAssembler::Address(bufferGPR, sizeof(double) * elementIndex));
+            value.use();
+        }
+
+        MacroAssembler::Jump fastPath = m_jit.branchPtr(MacroAssembler::NotEqual, bufferGPR, TrustedImmPtr(static_cast<EncodedJSValue*>(scratchBuffer->dataBuffer())));
+
+        addSlowPathGenerator(slowPathCall(m_jit.jump(), this, operationArrayPushDoubleMultiple, resultRegs, baseGPR, bufferGPR, TrustedImm32(elementCount)));
+
+        m_jit.move(TrustedImmPtr(scratchBuffer->addressOfActiveLength()), bufferGPR);
+        m_jit.storePtr(TrustedImmPtr(nullptr), MacroAssembler::Address(bufferGPR));
+
+        base.use();
+        storage.use();
+
+        fastPath.link(&m_jit);
+        jsValueResult(resultRegs, node, DataFormatJS, UseChildrenCalledExplicitly);
+        return;
+    }
+
+    case Array::ArrayStorage: {
+        // This ensures that the result of ArrayPush is Int32 in AI.
+        int32_t largestPositiveInt32Length = 0x7fffffff - elementCount;
+        if (elementCount == 1) {
+            Edge& element = m_jit.graph().varArgChild(node, elementOffset);
+            JSValueOperand value(this, element);
+            JSValueRegs valueRegs = value.jsValueRegs();
+
+            m_jit.load32(MacroAssembler::Address(storageGPR, ArrayStorage::lengthOffset()), storageLengthGPR);
+
+            // Refuse to handle bizarre lengths.
+            speculationCheck(Uncountable, JSValueRegs(), 0, m_jit.branch32(MacroAssembler::Above, storageLengthGPR, TrustedImm32(largestPositiveInt32Length)));
+
+            MacroAssembler::Jump slowPath = m_jit.branch32(MacroAssembler::AboveOrEqual, storageLengthGPR, MacroAssembler::Address(storageGPR, ArrayStorage::vectorLengthOffset()));
+
+            m_jit.storeValue(valueRegs, MacroAssembler::BaseIndex(storageGPR, storageLengthGPR, MacroAssembler::TimesEight, ArrayStorage::vectorOffset()));
+
+            m_jit.add32(TrustedImm32(1), storageLengthGPR);
+            m_jit.store32(storageLengthGPR, MacroAssembler::Address(storageGPR, ArrayStorage::lengthOffset()));
+            m_jit.add32(TrustedImm32(1), MacroAssembler::Address(storageGPR, OBJECT_OFFSETOF(ArrayStorage, m_numValuesInVector)));
+            m_jit.boxInt32(storageLengthGPR, resultRegs);
+
+            addSlowPathGenerator(
+                slowPathCall(slowPath, this, operationArrayPush, resultRegs, valueRegs, baseGPR));
+
+            jsValueResult(resultRegs, node);
+            return;
+        }
+
+        GPRTemporary buffer(this);
+        GPRReg bufferGPR = buffer.gpr();
+
+        m_jit.load32(MacroAssembler::Address(storageGPR, ArrayStorage::lengthOffset()), storageLengthGPR);
+
+        // Refuse to handle bizarre lengths.
+        speculationCheck(Uncountable, JSValueRegs(), 0, m_jit.branch32(MacroAssembler::Above, storageLengthGPR, TrustedImm32(largestPositiveInt32Length)));
+
+        m_jit.move(storageLengthGPR, bufferGPR);
+        m_jit.add32(TrustedImm32(elementCount), bufferGPR);
+        MacroAssembler::Jump slowPath = m_jit.branch32(MacroAssembler::Above, bufferGPR, MacroAssembler::Address(storageGPR, ArrayStorage::vectorLengthOffset()));
+
+        m_jit.store32(bufferGPR, MacroAssembler::Address(storageGPR, ArrayStorage::lengthOffset()));
+        getStorageBufferAddress(storageGPR, storageLengthGPR, ArrayStorage::vectorOffset(), bufferGPR);
+        m_jit.add32(TrustedImm32(elementCount), MacroAssembler::Address(storageGPR, OBJECT_OFFSETOF(ArrayStorage, m_numValuesInVector)));
+        m_jit.add32(TrustedImm32(elementCount), storageLengthGPR);
+        m_jit.boxInt32(storageLengthGPR, resultRegs);
+        auto storageDone = m_jit.jump();
+
+        slowPath.link(&m_jit);
+
+        size_t scratchSize = sizeof(EncodedJSValue) * elementCount;
+        ScratchBuffer* scratchBuffer = m_jit.vm()->scratchBufferForSize(scratchSize);
+        m_jit.move(TrustedImmPtr(static_cast<EncodedJSValue*>(scratchBuffer->dataBuffer())), bufferGPR);
+        m_jit.move(TrustedImmPtr(scratchBuffer->addressOfActiveLength()), storageLengthGPR);
+        m_jit.storePtr(TrustedImmPtr(scratchSize), MacroAssembler::Address(storageLengthGPR));
+
+        storageDone.link(&m_jit);
+        for (unsigned elementIndex = 0; elementIndex < elementCount; ++elementIndex) {
+            Edge& element = m_jit.graph().varArgChild(node, elementIndex + elementOffset);
+            JSValueOperand value(this, element);
+            JSValueRegs valueRegs = value.jsValueRegs();
+
+            m_jit.storeValue(valueRegs, MacroAssembler::Address(bufferGPR, sizeof(EncodedJSValue) * elementIndex));
+            value.use();
+        }
+
+        MacroAssembler::Jump fastPath = m_jit.branchPtr(MacroAssembler::NotEqual, bufferGPR, TrustedImmPtr(static_cast<EncodedJSValue*>(scratchBuffer->dataBuffer())));
+
+        addSlowPathGenerator(
+            slowPathCall(m_jit.jump(), this, operationArrayPushMultiple, resultRegs, baseGPR, bufferGPR, TrustedImm32(elementCount)));
+
+        m_jit.move(TrustedImmPtr(scratchBuffer->addressOfActiveLength()), bufferGPR);
+        m_jit.storePtr(TrustedImmPtr(nullptr), MacroAssembler::Address(bufferGPR));
+
+        base.use();
+        storage.use();
+
+        fastPath.link(&m_jit);
+        jsValueResult(resultRegs, node, DataFormatJS, UseChildrenCalledExplicitly);
+        return;
+    }
+
+    default:
+        RELEASE_ASSERT_NOT_REACHED();
+    }
+}
+
 void SpeculativeJIT::compileNotifyWrite(Node* node)
 {

trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h

@@ -1614 +1614 @@
         return appendCallSetResult(operation, result);
     }
-    JITCompiler::Call callOperation(J_JITOperation_EJA operation, GPRReg result, GPRReg arg1, GPRReg arg2)
+    JITCompiler::Call callOperation(J_JITOperation_EJA operation, JSValueRegs result, JSValueRegs arg1, GPRReg arg2)
+    {
+        m_jit.setupArgumentsWithExecState(arg1.gpr(), arg2);
+        return appendCallSetResult(operation, result.payloadGPR());
+    }
+    JITCompiler::Call callOperation(J_JITOperation_EJA operation, GPRReg result, JSValueRegs arg1, GPRReg arg2)
+    {
+        m_jit.setupArgumentsWithExecState(arg1.gpr(), arg2);
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(J_JITOperation_EP operation, GPRReg result, GPRReg arg1)
+    {
+        m_jit.setupArgumentsWithExecState(arg1);
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(J_JITOperation_EZ operation, GPRReg result, GPRReg arg1)
+    {
+        m_jit.setupArgumentsWithExecState(arg1);
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(J_JITOperation_EZ operation, GPRReg result, int32_t arg1)
+    {
+        m_jit.setupArgumentsWithExecState(TrustedImm32(arg1));
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(J_JITOperation_EZZ operation, GPRReg result, int32_t arg1, GPRReg arg2)
+    {
+        m_jit.setupArgumentsWithExecState(TrustedImm32(arg1), arg2);
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(P_JITOperation_EQZ operation, GPRReg result, GPRReg arg1, GPRReg arg2)
     {
         m_jit.setupArgumentsWithExecState(arg1, arg2);
         return appendCallSetResult(operation, result);
     }
-    JITCompiler::Call callOperation(J_JITOperation_EP operation, GPRReg result, GPRReg arg1)
+    JITCompiler::Call callOperation(P_JITOperation_EQZ operation, GPRReg result, GPRReg arg1, TrustedImm32 arg2)
+    {
+        m_jit.setupArgumentsWithExecState(arg1, arg2);
+        return appendCallSetResult(operation, result);
+    }
+
+    JITCompiler::Call callOperation(J_JITOperation_EZIcfZ operation, GPRReg result, int32_t arg1, InlineCallFrame* inlineCallFrame, GPRReg arg2)
+    {
+        m_jit.setupArgumentsWithExecState(TrustedImm32(arg1), TrustedImmPtr(inlineCallFrame), arg2);
+        return appendCallSetResult(operation, result);
+    }
+
+    JITCompiler::Call callOperation(P_JITOperation_EJS operation, GPRReg result, GPRReg value, size_t index)
+    {
+        m_jit.setupArgumentsWithExecState(value, TrustedImmPtr(index));
+        return appendCallSetResult(operation, result);
+    }
+
+    JITCompiler::Call callOperation(P_JITOperation_EStJ operation, GPRReg result, RegisteredStructure structure, GPRReg arg2)
+    {
+        m_jit.setupArgumentsWithExecState(TrustedImmPtr(structure), arg2);
+        return appendCallSetResult(operation, result);
+    }
+
+    JITCompiler::Call callOperation(C_JITOperation_EGJ operation, GPRReg result, JSGlobalObject* globalObject, GPRReg arg1)
+    {
+        m_jit.setupArgumentsWithExecState(TrustedImmPtr::weakPointer(m_jit.graph(), globalObject), arg1);
+        return appendCallSetResult(operation, result);
+    }
+
+    JITCompiler::Call callOperation(C_JITOperation_EGJ operation, GPRReg result, JSGlobalObject* globalObject, JSValueRegs arg1)
+    {
+        return callOperation(operation, result, globalObject, arg1.gpr());
+    }
+
+    JITCompiler::Call callOperation(C_JITOperation_EJ operation, GPRReg result, GPRReg arg1)
     {
         m_jit.setupArgumentsWithExecState(arg1);
         return appendCallSetResult(operation, result);
     }
-    JITCompiler::Call callOperation(J_JITOperation_EZ operation, GPRReg result, GPRReg arg1)
+    JITCompiler::Call callOperation(C_JITOperation_EJ operation, GPRReg result, JSValueRegs arg1)
+    {
+        m_jit.setupArgumentsWithExecState(arg1.gpr());
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(C_JITOperation_EJJ operation, GPRReg result, GPRReg arg1, GPRReg arg2)
+    {
+        m_jit.setupArgumentsWithExecState(arg1, arg2);
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(C_JITOperation_EJJC operation, GPRReg result, GPRReg arg1, GPRReg arg2, GPRReg arg3)
+    {
+        m_jit.setupArgumentsWithExecState(arg1, arg2, arg3);
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(C_JITOperation_EJJJ operation, GPRReg result, GPRReg arg1, GPRReg arg2, GPRReg arg3)
+    {
+        m_jit.setupArgumentsWithExecState(arg1, arg2, arg3);
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(C_JITOperation_EJZ operation, GPRReg result, GPRReg arg1, GPRReg arg2)
+    {
+        m_jit.setupArgumentsWithExecState(arg1, arg2);
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(C_JITOperation_EJZC operation, GPRReg result, GPRReg arg1, GPRReg arg2, GPRReg arg3)
+    {
+        m_jit.setupArgumentsWithExecState(arg1, arg2, arg3);
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(S_JITOperation_J operation, GPRReg result, GPRReg arg1)
+    {
+        m_jit.setupArguments(arg1);
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(S_JITOperation_EJ operation, GPRReg result, GPRReg arg1)
     {
         m_jit.setupArgumentsWithExecState(arg1);
         return appendCallSetResult(operation, result);
     }
-    JITCompiler::Call callOperation(J_JITOperation_EZ operation, GPRReg result, int32_t arg1)
-    {
-        m_jit.setupArgumentsWithExecState(TrustedImm32(arg1));
-        return appendCallSetResult(operation, result);
-    }
-    JITCompiler::Call callOperation(J_JITOperation_EZZ operation, GPRReg result, int32_t arg1, GPRReg arg2)
-    {
-        m_jit.setupArgumentsWithExecState(TrustedImm32(arg1), arg2);
-        return appendCallSetResult(operation, result);
-    }
-    JITCompiler::Call callOperation(P_JITOperation_EQZ operation, GPRReg result, GPRReg arg1, GPRReg arg2)
+    JITCompiler::Call callOperation(S_JITOperation_EJ operation, GPRReg result, JSValueRegs arg1)
+    {
+        return callOperation(operation, result, arg1.gpr());
+    }
+    JITCompiler::Call callOperation(J_JITOperation_EJ operation, JSValueRegs result, JSValueRegs arg1)
+    {
+        return callOperation(operation, result.payloadGPR(), arg1.payloadGPR());
+    }
+    JITCompiler::Call callOperation(J_JITOperation_EJ operation, GPRReg result, JSValueRegs arg1)
+    {
+        return callOperation(operation, result, arg1.payloadGPR());
+    }
+    JITCompiler::Call callOperation(J_JITOperation_EJ operation, GPRReg result, GPRReg arg1)
+    {
+        m_jit.setupArgumentsWithExecState(arg1);
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(S_JITOperation_EJI operation, GPRReg result, GPRReg arg1, UniquedStringImpl* uid)
+    {
+        m_jit.setupArgumentsWithExecState(arg1, TrustedImmPtr(uid));
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(S_JITOperation_EJI operation, GPRReg result, JSValueRegs arg1, UniquedStringImpl* uid)
+    {
+        return callOperation(operation, result, arg1.gpr(), uid);
+    }
+    JITCompiler::Call callOperation(S_JITOperation_EJJ operation, GPRReg result, GPRReg arg1, GPRReg arg2)
     {
         m_jit.setupArgumentsWithExecState(arg1, arg2);
         return appendCallSetResult(operation, result);
     }
-    JITCompiler::Call callOperation(P_JITOperation_EQZ operation, GPRReg result, GPRReg arg1, TrustedImm32 arg2)
+    JITCompiler::Call callOperation(S_JITOperation_EJJ operation, GPRReg result,  JSValueRegs arg1, JSValueRegs arg2)
+    {
+        return callOperation(operation, result, arg1.gpr(), arg2.gpr());
+    }
+    JITCompiler::Call callOperation(S_JITOperation_EGJJ operation, GPRReg result, GPRReg arg1, GPRReg arg2, GPRReg arg3)
+    {
+        m_jit.setupArgumentsWithExecState(arg1, arg2, arg3);
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(S_JITOperation_EGReoJ operation, GPRReg result, GPRReg arg1, GPRReg arg2, GPRReg arg3)
+    {
+        m_jit.setupArgumentsWithExecState(arg1, arg2, arg3);
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(S_JITOperation_EGReoJss operation, GPRReg result, GPRReg arg1, GPRReg arg2, GPRReg arg3)
+    {
+        m_jit.setupArgumentsWithExecState(arg1, arg2, arg3);
+        return appendCallSetResult(operation, result);
+    }
+
+    JITCompiler::Call callOperation(J_JITOperation_EPP operation, GPRReg result, GPRReg arg1, GPRReg arg2)
     {
         m_jit.setupArgumentsWithExecState(arg1, arg2);
         return appendCallSetResult(operation, result);
     }
-
-    JITCompiler::Call callOperation(J_JITOperation_EZIcfZ operation, GPRReg result, int32_t arg1, InlineCallFrame* inlineCallFrame, GPRReg arg2)
-    {
-        m_jit.setupArgumentsWithExecState(TrustedImm32(arg1), TrustedImmPtr(inlineCallFrame), arg2);
-        return appendCallSetResult(operation, result);
-    }
-
-    JITCompiler::Call callOperation(P_JITOperation_EJS operation, GPRReg result, GPRReg value, size_t index)
-    {
-        m_jit.setupArgumentsWithExecState(value, TrustedImmPtr(index));
-        return appendCallSetResult(operation, result);
-    }
-
-    JITCompiler::Call callOperation(P_JITOperation_EStJ operation, GPRReg result, RegisteredStructure structure, GPRReg arg2)
-    {
-        m_jit.setupArgumentsWithExecState(TrustedImmPtr(structure), arg2);
-        return appendCallSetResult(operation, result);
-    }
-
-    JITCompiler::Call callOperation(C_JITOperation_EGJ operation, GPRReg result, JSGlobalObject* globalObject, GPRReg arg1)
-    {
-        m_jit.setupArgumentsWithExecState(TrustedImmPtr::weakPointer(m_jit.graph(), globalObject), arg1);
-        return appendCallSetResult(operation, result);
-    }
-
-    JITCompiler::Call callOperation(C_JITOperation_EGJ operation, GPRReg result, JSGlobalObject* globalObject, JSValueRegs arg1)
-    {
-        return callOperation(operation, result, globalObject, arg1.gpr());
-    }
-
-    JITCompiler::Call callOperation(C_JITOperation_EJ operation, GPRReg result, GPRReg arg1)
-    {
-        m_jit.setupArgumentsWithExecState(arg1);
-        return appendCallSetResult(operation, result);
-    }
-    JITCompiler::Call callOperation(C_JITOperation_EJ operation, GPRReg result, JSValueRegs arg1)
-    {
-        m_jit.setupArgumentsWithExecState(arg1.gpr());
-        return appendCallSetResult(operation, result);
-    }
-    JITCompiler::Call callOperation(C_JITOperation_EJJ operation, GPRReg result, GPRReg arg1, GPRReg arg2)
+    JITCompiler::Call callOperation(J_JITOperation_EPPP operation, GPRReg result, GPRReg arg1, GPRReg arg2, GPRReg arg3)
+    {
+        m_jit.setupArgumentsWithExecState(arg1, arg2, arg3);
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(J_JITOperation_EGP operation, GPRReg result, GPRReg arg1, GPRReg arg2)
     {
         m_jit.setupArgumentsWithExecState(arg1, arg2);
         return appendCallSetResult(operation, result);
     }
-    JITCompiler::Call callOperation(C_JITOperation_EJJC operation, GPRReg result, GPRReg arg1, GPRReg arg2, GPRReg arg3)
+    JITCompiler::Call callOperation(J_JITOperation_EAPZ operation, JSValueRegs result, GPRReg arg1, GPRReg arg2, TrustedImm32 arg3)
     {
         m_jit.setupArgumentsWithExecState(arg1, arg2, arg3);
-        return appendCallSetResult(operation, result);
-    }
-    JITCompiler::Call callOperation(C_JITOperation_EJJJ operation, GPRReg result, GPRReg arg1, GPRReg arg2, GPRReg arg3)
+        return appendCallSetResult(operation, result.payloadGPR());
+    }
+    JITCompiler::Call callOperation(J_JITOperation_EAPZ operation, GPRReg result, GPRReg arg1, GPRReg arg2, TrustedImm32 arg3)
     {
         m_jit.setupArgumentsWithExecState(arg1, arg2, arg3);
-        return appendCallSetResult(operation, result);
-    }
-    JITCompiler::Call callOperation(C_JITOperation_EJZ operation, GPRReg result, GPRReg arg1, GPRReg arg2)
-    {
-        m_jit.setupArgumentsWithExecState(arg1, arg2);
-        return appendCallSetResult(operation, result);
-    }
-    JITCompiler::Call callOperation(C_JITOperation_EJZC operation, GPRReg result, GPRReg arg1, GPRReg arg2, GPRReg arg3)
-    {
-        m_jit.setupArgumentsWithExecState(arg1, arg2, arg3);
-        return appendCallSetResult(operation, result);
-    }
-    JITCompiler::Call callOperation(S_JITOperation_J operation, GPRReg result, GPRReg arg1)
-    {
-        m_jit.setupArguments(arg1);
-        return appendCallSetResult(operation, result);
-    }
-    JITCompiler::Call callOperation(S_JITOperation_EJ operation, GPRReg result, GPRReg arg1)
-    {
-        m_jit.setupArgumentsWithExecState(arg1);
-        return appendCallSetResult(operation, result);
-    }
-    JITCompiler::Call callOperation(S_JITOperation_EJ operation, GPRReg result, JSValueRegs arg1)
-    {
-        return callOperation(operation, result, arg1.gpr());
-    }
-    JITCompiler::Call callOperation(J_JITOperation_EJ operation, JSValueRegs result, JSValueRegs arg1)
-    {
-        return callOperation(operation, result.payloadGPR(), arg1.payloadGPR());
-    }
-    JITCompiler::Call callOperation(J_JITOperation_EJ operation, GPRReg result, JSValueRegs arg1)
-    {
-        return callOperation(operation, result, arg1.payloadGPR());
-    }
-    JITCompiler::Call callOperation(J_JITOperation_EJ operation, GPRReg result, GPRReg arg1)
-    {
-        m_jit.setupArgumentsWithExecState(arg1);
-        return appendCallSetResult(operation, result);
-    }
-    JITCompiler::Call callOperation(S_JITOperation_EJI operation, GPRReg result, GPRReg arg1, UniquedStringImpl* uid)
-    {
-        m_jit.setupArgumentsWithExecState(arg1, TrustedImmPtr(uid));
-        return appendCallSetResult(operation, result);
-    }
-    JITCompiler::Call callOperation(S_JITOperation_EJI operation, GPRReg result, JSValueRegs arg1, UniquedStringImpl* uid)
-    {
-        return callOperation(operation, result, arg1.gpr(), uid);
-    }
-    JITCompiler::Call callOperation(S_JITOperation_EJJ operation, GPRReg result, GPRReg arg1, GPRReg arg2)
-    {
-        m_jit.setupArgumentsWithExecState(arg1, arg2);
-        return appendCallSetResult(operation, result);
-    }
-    JITCompiler::Call callOperation(S_JITOperation_EJJ operation, GPRReg result,  JSValueRegs arg1, JSValueRegs arg2)
-    {
-        return callOperation(operation, result, arg1.gpr(), arg2.gpr());
-    }
-    JITCompiler::Call callOperation(S_JITOperation_EGJJ operation, GPRReg result, GPRReg arg1, GPRReg arg2, GPRReg arg3)
-    {
-        m_jit.setupArgumentsWithExecState(arg1, arg2, arg3);
-        return appendCallSetResult(operation, result);
-    }
-    JITCompiler::Call callOperation(S_JITOperation_EGReoJ operation, GPRReg result, GPRReg arg1, GPRReg arg2, GPRReg arg3)
-    {
-        m_jit.setupArgumentsWithExecState(arg1, arg2, arg3);
-        return appendCallSetResult(operation, result);
-    }
-    JITCompiler::Call callOperation(S_JITOperation_EGReoJss operation, GPRReg result, GPRReg arg1, GPRReg arg2, GPRReg arg3)
-    {
-        m_jit.setupArgumentsWithExecState(arg1, arg2, arg3);
-        return appendCallSetResult(operation, result);
-    }
-
-    JITCompiler::Call callOperation(J_JITOperation_EPP operation, GPRReg result, GPRReg arg1, GPRReg arg2)
-    {
-        m_jit.setupArgumentsWithExecState(arg1, arg2);
-        return appendCallSetResult(operation, result);
-    }
-    JITCompiler::Call callOperation(J_JITOperation_EPPP operation, GPRReg result, GPRReg arg1, GPRReg arg2, GPRReg arg3)
-    {
-        m_jit.setupArgumentsWithExecState(arg1, arg2, arg3);
-        return appendCallSetResult(operation, result);
-    }
-    JITCompiler::Call callOperation(J_JITOperation_EGP operation, GPRReg result, GPRReg arg1, GPRReg arg2)
-    {
-        m_jit.setupArgumentsWithExecState(arg1, arg2);
         return appendCallSetResult(operation, result);
     }
@@ -2113 +2128 @@
     {
         m_jit.setupArgumentsWithExecState(arg1, arg2);
+        return appendCallSetResult(operation, result.payloadGPR(), result.tagGPR());
+    }
+    JITCompiler::Call callOperation(J_JITOperation_EAPZ operation, JSValueRegs result, GPRReg arg1, GPRReg arg2, TrustedImm32 arg3)
+    {
+        m_jit.setupArgumentsWithExecState(arg1, arg2, arg3);
         return appendCallSetResult(operation, result.payloadGPR(), result.tagGPR());
     }
@@ -2928 +2948 @@
     void compileArraySlice(Node*);
     void compileArrayIndexOf(Node*);
+    void compileArrayPush(Node*);
     void compileNotifyWrite(Node*);
     bool compileRegExpExec(Node*);

trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp

@@ -3377 +3377 @@
         
     case ArrayPush: {
-        ASSERT(node->arrayMode().isJSArray());
-        
-        SpeculateCellOperand base(this, node->child1());
-        GPRTemporary storageLength(this);
-        
-        GPRReg baseGPR = base.gpr();
-        GPRReg storageLengthGPR = storageLength.gpr();
-        
-        StorageOperand storage(this, node->child3());
-        GPRReg storageGPR = storage.gpr();
-        
-        switch (node->arrayMode().type()) {
-        case Array::Int32: {
-            SpeculateInt32Operand value(this, node->child2());
-            GPRReg valuePayloadGPR = value.gpr();
-            
-            m_jit.load32(MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()), storageLengthGPR);
-            MacroAssembler::Jump slowPath = m_jit.branch32(MacroAssembler::AboveOrEqual, storageLengthGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
-            m_jit.store32(TrustedImm32(JSValue::Int32Tag), MacroAssembler::BaseIndex(storageGPR, storageLengthGPR, MacroAssembler::TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.tag)));
-            m_jit.store32(valuePayloadGPR, MacroAssembler::BaseIndex(storageGPR, storageLengthGPR, MacroAssembler::TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.payload)));
-            m_jit.add32(TrustedImm32(1), storageLengthGPR);
-            m_jit.store32(storageLengthGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
-            m_jit.move(TrustedImm32(JSValue::Int32Tag), storageGPR);
-            
-            addSlowPathGenerator(
-                slowPathCall(
-                    slowPath, this, operationArrayPush,
-                    JSValueRegs(storageGPR, storageLengthGPR),
-                    TrustedImm32(JSValue::Int32Tag), valuePayloadGPR, baseGPR));
-        
-            jsValueResult(storageGPR, storageLengthGPR, node);
-            break;
-        }
-            
-        case Array::Contiguous: {
-            JSValueOperand value(this, node->child2());
-            GPRReg valueTagGPR = value.tagGPR();
-            GPRReg valuePayloadGPR = value.payloadGPR();
-
-            m_jit.load32(MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()), storageLengthGPR);
-            MacroAssembler::Jump slowPath = m_jit.branch32(MacroAssembler::AboveOrEqual, storageLengthGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
-            m_jit.store32(valueTagGPR, MacroAssembler::BaseIndex(storageGPR, storageLengthGPR, MacroAssembler::TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.tag)));
-            m_jit.store32(valuePayloadGPR, MacroAssembler::BaseIndex(storageGPR, storageLengthGPR, MacroAssembler::TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.payload)));
-            m_jit.add32(TrustedImm32(1), storageLengthGPR);
-            m_jit.store32(storageLengthGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
-            m_jit.move(TrustedImm32(JSValue::Int32Tag), storageGPR);
-            
-            addSlowPathGenerator(
-                slowPathCall(
-                    slowPath, this, operationArrayPush,
-                    JSValueRegs(storageGPR, storageLengthGPR),
-                    JSValueRegs(valueTagGPR, valuePayloadGPR), baseGPR));
-        
-            jsValueResult(storageGPR, storageLengthGPR, node);
-            break;
-        }
-            
-        case Array::Double: {
-            SpeculateDoubleOperand value(this, node->child2());
-            FPRReg valueFPR = value.fpr();
-
-            DFG_TYPE_CHECK(
-                JSValueRegs(), node->child2(), SpecDoubleReal,
-                m_jit.branchDouble(MacroAssembler::DoubleNotEqualOrUnordered, valueFPR, valueFPR));
-            
-            m_jit.load32(MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()), storageLengthGPR);
-            MacroAssembler::Jump slowPath = m_jit.branch32(MacroAssembler::AboveOrEqual, storageLengthGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
-            m_jit.storeDouble(valueFPR, MacroAssembler::BaseIndex(storageGPR, storageLengthGPR, MacroAssembler::TimesEight));
-            m_jit.add32(TrustedImm32(1), storageLengthGPR);
-            m_jit.store32(storageLengthGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
-            m_jit.move(TrustedImm32(JSValue::Int32Tag), storageGPR);
-            
-            addSlowPathGenerator(
-                slowPathCall(
-                    slowPath, this, operationArrayPushDouble,
-                    JSValueRegs(storageGPR, storageLengthGPR),
-                    valueFPR, baseGPR));
-        
-            jsValueResult(storageGPR, storageLengthGPR, node);
-            break;
-        }
-            
-        case Array::ArrayStorage: {
-            JSValueOperand value(this, node->child2());
-            GPRReg valueTagGPR = value.tagGPR();
-            GPRReg valuePayloadGPR = value.payloadGPR();
-
-            m_jit.load32(MacroAssembler::Address(storageGPR, ArrayStorage::lengthOffset()), storageLengthGPR);
-        
-            // Refuse to handle bizarre lengths.
-            speculationCheck(Uncountable, JSValueRegs(), 0, m_jit.branch32(MacroAssembler::Above, storageLengthGPR, TrustedImm32(0x7ffffffe)));
-        
-            MacroAssembler::Jump slowPath = m_jit.branch32(MacroAssembler::AboveOrEqual, storageLengthGPR, MacroAssembler::Address(storageGPR, ArrayStorage::vectorLengthOffset()));
-        
-            m_jit.store32(valueTagGPR, MacroAssembler::BaseIndex(storageGPR, storageLengthGPR, MacroAssembler::TimesEight, ArrayStorage::vectorOffset() + OBJECT_OFFSETOF(JSValue, u.asBits.tag)));
-            m_jit.store32(valuePayloadGPR, MacroAssembler::BaseIndex(storageGPR, storageLengthGPR, MacroAssembler::TimesEight, ArrayStorage::vectorOffset() + OBJECT_OFFSETOF(JSValue, u.asBits.payload)));
-        
-            m_jit.add32(TrustedImm32(1), storageLengthGPR);
-            m_jit.store32(storageLengthGPR, MacroAssembler::Address(storageGPR, ArrayStorage::lengthOffset()));
-            m_jit.add32(TrustedImm32(1), MacroAssembler::Address(storageGPR, OBJECT_OFFSETOF(ArrayStorage, m_numValuesInVector)));
-            m_jit.move(TrustedImm32(JSValue::Int32Tag), storageGPR);
-        
-            addSlowPathGenerator(slowPathCall(slowPath, this, operationArrayPush, JSValueRegs(storageGPR, storageLengthGPR),
-                JSValueRegs(valueTagGPR, valuePayloadGPR), baseGPR));
-        
-            jsValueResult(storageGPR, storageLengthGPR, node);
-            break;
-        }
-            
-        default:
-            CRASH();
-            break;
-        }
+        compileArrayPush(node);
         break;
     }

trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp

@@ -3656 +3656 @@
         
     case ArrayPush: {
-        ASSERT(node->arrayMode().isJSArray());
-        
-        SpeculateCellOperand base(this, node->child1());
-        GPRTemporary storageLength(this);
-        
-        GPRReg baseGPR = base.gpr();
-        GPRReg storageLengthGPR = storageLength.gpr();
-        
-        StorageOperand storage(this, node->child3());
-        GPRReg storageGPR = storage.gpr();
-
-        switch (node->arrayMode().type()) {
-        case Array::Int32:
-        case Array::Contiguous: {
-            JSValueOperand value(this, node->child2(), ManualOperandSpeculation);
-            GPRReg valueGPR = value.gpr();
-
-            if (node->arrayMode().type() == Array::Int32) {
-                DFG_TYPE_CHECK(
-                    JSValueRegs(valueGPR), node->child2(), SpecInt32Only,
-                    m_jit.branch64(
-                        MacroAssembler::Below, valueGPR, GPRInfo::tagTypeNumberRegister));
-            }
-
-            m_jit.load32(MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()), storageLengthGPR);
-            MacroAssembler::Jump slowPath = m_jit.branch32(MacroAssembler::AboveOrEqual, storageLengthGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
-            m_jit.store64(valueGPR, MacroAssembler::BaseIndex(storageGPR, storageLengthGPR, MacroAssembler::TimesEight));
-            m_jit.add32(TrustedImm32(1), storageLengthGPR);
-            m_jit.store32(storageLengthGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
-            m_jit.or64(GPRInfo::tagTypeNumberRegister, storageLengthGPR);
-            
-            addSlowPathGenerator(
-                slowPathCall(
-                    slowPath, this, operationArrayPush, storageLengthGPR,
-                    valueGPR, baseGPR));
-        
-            jsValueResult(storageLengthGPR, node);
-            break;
-        }
-            
-        case Array::Double: {
-            SpeculateDoubleOperand value(this, node->child2());
-            FPRReg valueFPR = value.fpr();
-
-            DFG_TYPE_CHECK(
-                JSValueRegs(), node->child2(), SpecDoubleReal,
-                m_jit.branchDouble(MacroAssembler::DoubleNotEqualOrUnordered, valueFPR, valueFPR));
-            
-            m_jit.load32(MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()), storageLengthGPR);
-            MacroAssembler::Jump slowPath = m_jit.branch32(MacroAssembler::AboveOrEqual, storageLengthGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
-            m_jit.storeDouble(valueFPR, MacroAssembler::BaseIndex(storageGPR, storageLengthGPR, MacroAssembler::TimesEight));
-            m_jit.add32(TrustedImm32(1), storageLengthGPR);
-            m_jit.store32(storageLengthGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
-            m_jit.or64(GPRInfo::tagTypeNumberRegister, storageLengthGPR);
-            
-            addSlowPathGenerator(
-                slowPathCall(
-                    slowPath, this, operationArrayPushDouble, storageLengthGPR,
-                    valueFPR, baseGPR));
-        
-            jsValueResult(storageLengthGPR, node);
-            break;
-        }
-            
-        case Array::ArrayStorage: {
-            JSValueOperand value(this, node->child2());
-            GPRReg valueGPR = value.gpr();
-
-            m_jit.load32(MacroAssembler::Address(storageGPR, ArrayStorage::lengthOffset()), storageLengthGPR);
-        
-            // Refuse to handle bizarre lengths.
-            speculationCheck(Uncountable, JSValueRegs(), 0, m_jit.branch32(MacroAssembler::Above, storageLengthGPR, TrustedImm32(0x7ffffffe)));
-        
-            MacroAssembler::Jump slowPath = m_jit.branch32(MacroAssembler::AboveOrEqual, storageLengthGPR, MacroAssembler::Address(storageGPR, ArrayStorage::vectorLengthOffset()));
-        
-            m_jit.store64(valueGPR, MacroAssembler::BaseIndex(storageGPR, storageLengthGPR, MacroAssembler::TimesEight, ArrayStorage::vectorOffset()));
-        
-            m_jit.add32(TrustedImm32(1), storageLengthGPR);
-            m_jit.store32(storageLengthGPR, MacroAssembler::Address(storageGPR, ArrayStorage::lengthOffset()));
-            m_jit.add32(TrustedImm32(1), MacroAssembler::Address(storageGPR, OBJECT_OFFSETOF(ArrayStorage, m_numValuesInVector)));
-            m_jit.or64(GPRInfo::tagTypeNumberRegister, storageLengthGPR);
-        
-            addSlowPathGenerator(
-                slowPathCall(
-                    slowPath, this, operationArrayPush, NoResult, storageLengthGPR,
-                    valueGPR, baseGPR));
-        
-            jsValueResult(storageLengthGPR, node);
-            break;
-        }
-            
-        default:
-            CRASH();
-            break;
-        }
+        compileArrayPush(node);
         break;
     }

trunk/Source/JavaScriptCore/dfg/DFGStoreBarrierInsertionPhase.cpp

@@ -251 +251 @@
                 switch (m_node->arrayMode().type()) {
                 case Array::Contiguous:
-                case Array::ArrayStorage:
-                    considerBarrier(m_node->child1(), m_node->child2());
+                case Array::ArrayStorage: {
+                    unsigned elementOffset = 2;
+                    unsigned elementCount = m_node->numChildren() - elementOffset;
+                    Edge& arrayEdge = m_graph.varArgChild(m_node, 1);
+                    for (unsigned i = 0; i < elementCount; ++i) {
+                        Edge& element = m_graph.varArgChild(m_node, i + elementOffset);
+                        considerBarrier(arrayEdge, element);
+                    }
                     break;
+                }
                 default:
                     break;

trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp

@@ -4083 +4083 @@
     void compileArrayPush()
     {
-        LValue base = lowCell(m_node->child1());
-        LValue storage = lowStorage(m_node->child3());
-        
+        LValue base = lowCell(m_graph.varArgChild(m_node, 1));
+        LValue storage = lowStorage(m_graph.varArgChild(m_node, 0));
+        unsigned elementOffset = 2;
+        unsigned elementCount = m_node->numChildren() - elementOffset;
+
         switch (m_node->arrayMode().type()) {
         case Array::Int32:
         case Array::Contiguous:
         case Array::Double: {
-            LValue value;
-            Output::StoreType storeType;
-            
-            if (m_node->arrayMode().type() != Array::Double) {
-                value = lowJSValue(m_node->child2(), ManualOperandSpeculation);
-                if (m_node->arrayMode().type() == Array::Int32) {
-                    FTL_TYPE_CHECK(
-                        jsValueValue(value), m_node->child2(), SpecInt32Only, isNotInt32(value));
+            IndexedAbstractHeap& heap = m_heaps.forArrayType(m_node->arrayMode().type());
+
+            if (elementCount == 1) {
+                LValue value;
+                Output::StoreType storeType;
+                
+                Edge& element = m_graph.varArgChild(m_node, elementOffset);
+                if (m_node->arrayMode().type() != Array::Double) {
+                    value = lowJSValue(element, ManualOperandSpeculation);
+                    if (m_node->arrayMode().type() == Array::Int32)
+                        RELEASE_ASSERT(!m_interpreter.needsTypeCheck(element, SpecInt32Only));
+                    storeType = Output::Store64;
+                } else {
+                    value = lowDouble(element);
+                    RELEASE_ASSERT(!m_interpreter.needsTypeCheck(element, SpecDoubleReal));
+                    storeType = Output::StoreDouble;
                 }
-                storeType = Output::Store64;
-            } else {
-                value = lowDouble(m_node->child2());
-                FTL_TYPE_CHECK(
-                    doubleValue(value), m_node->child2(), SpecDoubleReal,
-                    m_out.doubleNotEqualOrUnordered(value, value));
-                storeType = Output::StoreDouble;
+
+                LValue prevLength = m_out.load32(storage, m_heaps.Butterfly_publicLength);
+                
+                LBasicBlock fastPath = m_out.newBlock();
+                LBasicBlock slowPath = m_out.newBlock();
+                LBasicBlock continuation = m_out.newBlock();
+                
+                m_out.branch(
+                    m_out.aboveOrEqual(
+                        prevLength, m_out.load32(storage, m_heaps.Butterfly_vectorLength)),
+                    unsure(slowPath), unsure(fastPath));
+                
+                LBasicBlock lastNext = m_out.appendTo(fastPath, slowPath);
+                m_out.store(
+                    value, m_out.baseIndex(heap, storage, m_out.zeroExtPtr(prevLength)), storeType);
+                LValue newLength = m_out.add(prevLength, m_out.int32One);
+                m_out.store32(newLength, storage, m_heaps.Butterfly_publicLength);
+                
+                ValueFromBlock fastResult = m_out.anchor(boxInt32(newLength));
+                m_out.jump(continuation);
+                
+                m_out.appendTo(slowPath, continuation);
+                LValue operation;
+                if (m_node->arrayMode().type() != Array::Double)
+                    operation = m_out.operation(operationArrayPush);
+                else
+                    operation = m_out.operation(operationArrayPushDouble);
+                ValueFromBlock slowResult = m_out.anchor(
+                    vmCall(Int64, operation, m_callFrame, value, base));
+                m_out.jump(continuation);
+                
+                m_out.appendTo(continuation, lastNext);
+                setJSValue(m_out.phi(Int64, fastResult, slowResult));
+                return;
             }
-            
-            IndexedAbstractHeap& heap = m_heaps.forArrayType(m_node->arrayMode().type());
 
             LValue prevLength = m_out.load32(storage, m_heaps.Butterfly_publicLength);
-            
+            LValue newLength = m_out.add(prevLength, m_out.constInt32(elementCount));
+
             LBasicBlock fastPath = m_out.newBlock();
             LBasicBlock slowPath = m_out.newBlock();
+            LBasicBlock setup = m_out.newBlock();
+            LBasicBlock slowCallPath = m_out.newBlock();
             LBasicBlock continuation = m_out.newBlock();
-            
-            m_out.branch(
-                m_out.aboveOrEqual(
-                    prevLength, m_out.load32(storage, m_heaps.Butterfly_vectorLength)),
-                unsure(slowPath), unsure(fastPath));
-            
+
+            LValue beyondVectorLength = m_out.above(newLength, m_out.load32(storage, m_heaps.Butterfly_vectorLength));
+
+            m_out.branch(beyondVectorLength, unsure(slowPath), unsure(fastPath));
+
             LBasicBlock lastNext = m_out.appendTo(fastPath, slowPath);
-            m_out.store(
-                value, m_out.baseIndex(heap, storage, m_out.zeroExtPtr(prevLength)), storeType);
-            LValue newLength = m_out.add(prevLength, m_out.int32One);
             m_out.store32(newLength, storage, m_heaps.Butterfly_publicLength);
-            
+            ValueFromBlock fastBufferResult = m_out.anchor(m_out.baseIndex(storage, m_out.zeroExtPtr(prevLength), ScaleEight));
+            m_out.jump(setup);
+
+            m_out.appendTo(slowPath, setup);
+            size_t scratchSize = sizeof(EncodedJSValue) * elementCount;
+            static_assert(sizeof(EncodedJSValue) == sizeof(double), "");
+            ASSERT(scratchSize);
+            ScratchBuffer* scratchBuffer = vm().scratchBufferForSize(scratchSize);
+            m_out.storePtr(m_out.constIntPtr(scratchSize), m_out.absolute(scratchBuffer->addressOfActiveLength()));
+            ValueFromBlock slowBufferResult = m_out.anchor(m_out.constIntPtr(static_cast<EncodedJSValue*>(scratchBuffer->dataBuffer())));
+            m_out.jump(setup);
+
+            m_out.appendTo(setup, slowCallPath);
+            LValue buffer = m_out.phi(pointerType(), fastBufferResult, slowBufferResult);
+            for (unsigned elementIndex = 0; elementIndex < elementCount; ++elementIndex) {
+                Edge& element = m_graph.varArgChild(m_node, elementIndex + elementOffset);
+
+                LValue value;
+                Output::StoreType storeType;
+                if (m_node->arrayMode().type() != Array::Double) {
+                    value = lowJSValue(element, ManualOperandSpeculation);
+                    if (m_node->arrayMode().type() == Array::Int32)
+                        RELEASE_ASSERT(!m_interpreter.needsTypeCheck(element, SpecInt32Only));
+                    storeType = Output::Store64;
+                } else {
+                    value = lowDouble(element);
+                    RELEASE_ASSERT(!m_interpreter.needsTypeCheck(element, SpecDoubleReal));
+                    storeType = Output::StoreDouble;
+                }
+
+                m_out.store(value, m_out.baseIndex(heap, buffer, m_out.constInt32(elementIndex), jsNumber(elementIndex)), storeType);
+            }
             ValueFromBlock fastResult = m_out.anchor(boxInt32(newLength));
-            m_out.jump(continuation);
-            
-            m_out.appendTo(slowPath, continuation);
+
+            m_out.branch(beyondVectorLength, unsure(slowCallPath), unsure(continuation));
+
+            m_out.appendTo(slowCallPath, continuation);
             LValue operation;
             if (m_node->arrayMode().type() != Array::Double)
-                operation = m_out.operation(operationArrayPush);
+                operation = m_out.operation(operationArrayPushMultiple);
             else
-                operation = m_out.operation(operationArrayPushDouble);
-            ValueFromBlock slowResult = m_out.anchor(
-                vmCall(Int64, operation, m_callFrame, value, base));
+                operation = m_out.operation(operationArrayPushDoubleMultiple);
+            ValueFromBlock slowResult = m_out.anchor(vmCall(Int64, operation, m_callFrame, base, buffer, m_out.constInt32(elementCount)));
+            m_out.storePtr(m_out.constIntPtr(0), m_out.absolute(scratchBuffer->addressOfActiveLength()));
             m_out.jump(continuation);
-            
+
             m_out.appendTo(continuation, lastNext);
             setJSValue(m_out.phi(Int64, fastResult, slowResult));
             return;
         }
-            
+
         default:
             DFG_CRASH(m_graph, m_node, "Bad array type");

trunk/Source/JavaScriptCore/jit/JITOperations.h

@@ -142 +142 @@
 typedef EncodedJSValue (JIT_OPERATION *J_JITOperation_EJC)(ExecState*, EncodedJSValue, JSCell*);
 typedef EncodedJSValue (JIT_OPERATION *J_JITOperation_EJA)(ExecState*, EncodedJSValue, JSArray*);
+typedef EncodedJSValue (JIT_OPERATION *J_JITOperation_EAPZ)(ExecState*, JSArray*, void*, int32_t);
 typedef EncodedJSValue (JIT_OPERATION *J_JITOperation_EJArp)(ExecState*, EncodedJSValue, ArithProfile*);
 typedef EncodedJSValue (JIT_OPERATION *J_JITOperation_EJI)(ExecState*, EncodedJSValue, UniquedStringImpl*);

trunk/Source/JavaScriptCore/runtime/ArrayPrototype.cpp

@@ -765 +765 @@
     JSValue thisValue = exec->thisValue().toThis(exec, StrictMode);
 
-    if (isJSArray(thisValue) && exec->argumentCount() == 1) {
+    if (LIKELY(isJSArray(thisValue) && exec->argumentCount() == 1)) {
         JSArray* array = asArray(thisValue);
         scope.release();
-        array->push(exec, exec->uncheckedArgument(0));
+        array->pushInline(exec, exec->uncheckedArgument(0));
         return JSValue::encode(jsNumber(array->length()));
     }

trunk/Source/JavaScriptCore/runtime/JSArray.cpp

@@ -41 +41 @@
 namespace JSC {
 
-static const char* const LengthExceededTheMaximumArrayLengthError = "Length exceeded the maximum array length";
+const char* const LengthExceededTheMaximumArrayLengthError = "Length exceeded the maximum array length";
 
 STATIC_ASSERT_IS_TRIVIALLY_DESTRUCTIBLE(JSArray);
@@ -726 +726 @@
 //  - we always are writing beyond the current array bounds, so it is always necessary to update m_length & m_numValuesInVector.
 //  - pushing to an array of length 2^32-1 stores the property, but throws a range error.
-void JSArray::push(ExecState* exec, JSValue value)
-{
-    VM& vm = exec->vm();
-    auto scope = DECLARE_THROW_SCOPE(vm);
-
-    Butterfly* butterfly = m_butterfly.getMayBeNull();
-    
-    switch (indexingType()) {
-    case ArrayClass: {
-        createInitialUndecided(vm, 0);
-        FALLTHROUGH;
-    }
-        
-    case ArrayWithUndecided: {
-        convertUndecidedForValue(vm, value);
-        scope.release();
-        push(exec, value);
-        return;
-    }
-        
-    case ArrayWithInt32: {
-        if (!value.isInt32()) {
-            convertInt32ForValue(vm, value);
-            scope.release();
-            push(exec, value);
-            return;
-        }
-
-        unsigned length = butterfly->publicLength();
-        ASSERT(length <= butterfly->vectorLength());
-        if (length < butterfly->vectorLength()) {
-            butterfly->contiguousInt32()[length].setWithoutWriteBarrier(value);
-            butterfly->setPublicLength(length + 1);
-            return;
-        }
-        
-        if (UNLIKELY(length > MAX_ARRAY_INDEX)) {
-            methodTable(vm)->putByIndex(this, exec, length, value, true);
-            if (!scope.exception())
-                throwException(exec, scope, createRangeError(exec, ASCIILiteral(LengthExceededTheMaximumArrayLengthError)));
-            return;
-        }
-
-        scope.release();
-        putByIndexBeyondVectorLengthWithoutAttributes<Int32Shape>(exec, length, value);
-        return;
-    }
-
-    case ArrayWithContiguous: {
-        unsigned length = butterfly->publicLength();
-        ASSERT(length <= butterfly->vectorLength());
-        if (length < butterfly->vectorLength()) {
-            butterfly->contiguous()[length].set(vm, this, value);
-            butterfly->setPublicLength(length + 1);
-            return;
-        }
-        
-        if (UNLIKELY(length > MAX_ARRAY_INDEX)) {
-            methodTable(vm)->putByIndex(this, exec, length, value, true);
-            if (!scope.exception())
-                throwException(exec, scope, createRangeError(exec, ASCIILiteral(LengthExceededTheMaximumArrayLengthError)));
-            return;
-        }
-
-        scope.release();
-        putByIndexBeyondVectorLengthWithoutAttributes<ContiguousShape>(exec, length, value);
-        return;
-    }
-        
-    case ArrayWithDouble: {
-        if (!value.isNumber()) {
-            convertDoubleToContiguous(vm);
-            scope.release();
-            push(exec, value);
-            return;
-        }
-        double valueAsDouble = value.asNumber();
-        if (valueAsDouble != valueAsDouble) {
-            convertDoubleToContiguous(vm);
-            scope.release();
-            push(exec, value);
-            return;
-        }
-
-        unsigned length = butterfly->publicLength();
-        ASSERT(length <= butterfly->vectorLength());
-        if (length < butterfly->vectorLength()) {
-            butterfly->contiguousDouble()[length] = valueAsDouble;
-            butterfly->setPublicLength(length + 1);
-            return;
-        }
-        
-        if (UNLIKELY(length > MAX_ARRAY_INDEX)) {
-            methodTable(vm)->putByIndex(this, exec, length, value, true);
-            if (!scope.exception())
-                throwException(exec, scope, createRangeError(exec, ASCIILiteral(LengthExceededTheMaximumArrayLengthError)));
-            return;
-        }
-
-        scope.release();
-        putByIndexBeyondVectorLengthWithoutAttributes<DoubleShape>(exec, length, value);
-        return;
-    }
-        
-    case ArrayWithSlowPutArrayStorage: {
-        unsigned oldLength = length();
-        bool putResult = false;
-        if (attemptToInterceptPutByIndexOnHole(exec, oldLength, value, true, putResult)) {
-            if (!scope.exception() && oldLength < 0xFFFFFFFFu) {
-                scope.release();
-                setLength(exec, oldLength + 1, true);
-            }
-            return;
-        }
-        FALLTHROUGH;
-    }
-        
-    case ArrayWithArrayStorage: {
-        ArrayStorage* storage = butterfly->arrayStorage();
-
-        // Fast case - push within vector, always update m_length & m_numValuesInVector.
-        unsigned length = storage->length();
-        if (length < storage->vectorLength()) {
-            storage->m_vector[length].set(vm, this, value);
-            storage->setLength(length + 1);
-            ++storage->m_numValuesInVector;
-            return;
-        }
-
-        // Pushing to an array of invalid length (2^31-1) stores the property, but throws a range error.
-        if (UNLIKELY(storage->length() > MAX_ARRAY_INDEX)) {
-            methodTable(vm)->putByIndex(this, exec, storage->length(), value, true);
-            // Per ES5.1 15.4.4.7 step 6 & 15.4.5.1 step 3.d.
-            if (!scope.exception())
-                throwException(exec, scope, createRangeError(exec, ASCIILiteral(LengthExceededTheMaximumArrayLengthError)));
-            return;
-        }
-
-        // Handled the same as putIndex.
-        scope.release();
-        putByIndexBeyondVectorLengthWithArrayStorage(exec, storage->length(), value, true, storage);
-        return;
-    }
-        
-    default:
-        RELEASE_ASSERT_NOT_REACHED();
-    }
+NEVER_INLINE void JSArray::push(ExecState* exec, JSValue value)
+{
+    pushInline(exec, value);
 }
 

trunk/Source/JavaScriptCore/runtime/JSArray.h

@@ -32 +32 @@
 class LLIntOffsetsExtractor;
 
+extern const char* const LengthExceededTheMaximumArrayLengthError;
+
 class JSArray : public JSNonFinalObject {
     friend class LLIntOffsetsExtractor;
@@ -91 +93 @@
     JS_EXPORT_PRIVATE bool setLength(ExecState*, unsigned, bool throwException = false);
 
+    void pushInline(ExecState*, JSValue);
     JS_EXPORT_PRIVATE void push(ExecState*, JSValue);
     JS_EXPORT_PRIVATE JSValue pop(ExecState*);

trunk/Source/JavaScriptCore/runtime/JSArrayInlines.h

@@ -81 +81 @@
 }
 
+ALWAYS_INLINE void JSArray::pushInline(ExecState* exec, JSValue value)
+{
+    VM& vm = exec->vm();
+    auto scope = DECLARE_THROW_SCOPE(vm);
+
+    Butterfly* butterfly = m_butterfly.getMayBeNull();
+
+    switch (indexingType()) {
+    case ArrayClass: {
+        createInitialUndecided(vm, 0);
+        FALLTHROUGH;
+    }
+
+    case ArrayWithUndecided: {
+        convertUndecidedForValue(vm, value);
+        scope.release();
+        push(exec, value);
+        return;
+    }
+
+    case ArrayWithInt32: {
+        if (!value.isInt32()) {
+            convertInt32ForValue(vm, value);
+            scope.release();
+            push(exec, value);
+            return;
+        }
+
+        unsigned length = butterfly->publicLength();
+        ASSERT(length <= butterfly->vectorLength());
+        if (length < butterfly->vectorLength()) {
+            butterfly->contiguousInt32()[length].setWithoutWriteBarrier(value);
+            butterfly->setPublicLength(length + 1);
+            return;
+        }
+
+        if (UNLIKELY(length > MAX_ARRAY_INDEX)) {
+            methodTable(vm)->putByIndex(this, exec, length, value, true);
+            if (!scope.exception())
+                throwException(exec, scope, createRangeError(exec, ASCIILiteral(LengthExceededTheMaximumArrayLengthError)));
+            return;
+        }
+
+        scope.release();
+        putByIndexBeyondVectorLengthWithoutAttributes<Int32Shape>(exec, length, value);
+        return;
+    }
+
+    case ArrayWithContiguous: {
+        unsigned length = butterfly->publicLength();
+        ASSERT(length <= butterfly->vectorLength());
+        if (length < butterfly->vectorLength()) {
+            butterfly->contiguous()[length].set(vm, this, value);
+            butterfly->setPublicLength(length + 1);
+            return;
+        }
+
+        if (UNLIKELY(length > MAX_ARRAY_INDEX)) {
+            methodTable(vm)->putByIndex(this, exec, length, value, true);
+            if (!scope.exception())
+                throwException(exec, scope, createRangeError(exec, ASCIILiteral(LengthExceededTheMaximumArrayLengthError)));
+            return;
+        }
+
+        scope.release();
+        putByIndexBeyondVectorLengthWithoutAttributes<ContiguousShape>(exec, length, value);
+        return;
+    }
+
+    case ArrayWithDouble: {
+        if (!value.isNumber()) {
+            convertDoubleToContiguous(vm);
+            scope.release();
+            push(exec, value);
+            return;
+        }
+        double valueAsDouble = value.asNumber();
+        if (valueAsDouble != valueAsDouble) {
+            convertDoubleToContiguous(vm);
+            scope.release();
+            push(exec, value);
+            return;
+        }
+
+        unsigned length = butterfly->publicLength();
+        ASSERT(length <= butterfly->vectorLength());
+        if (length < butterfly->vectorLength()) {
+            butterfly->contiguousDouble()[length] = valueAsDouble;
+            butterfly->setPublicLength(length + 1);
+            return;
+        }
+
+        if (UNLIKELY(length > MAX_ARRAY_INDEX)) {
+            methodTable(vm)->putByIndex(this, exec, length, value, true);
+            if (!scope.exception())
+                throwException(exec, scope, createRangeError(exec, ASCIILiteral(LengthExceededTheMaximumArrayLengthError)));
+            return;
+        }
+
+        scope.release();
+        putByIndexBeyondVectorLengthWithoutAttributes<DoubleShape>(exec, length, value);
+        return;
+    }
+
+    case ArrayWithSlowPutArrayStorage: {
+        unsigned oldLength = length();
+        bool putResult = false;
+        if (attemptToInterceptPutByIndexOnHole(exec, oldLength, value, true, putResult)) {
+            if (!scope.exception() && oldLength < 0xFFFFFFFFu) {
+                scope.release();
+                setLength(exec, oldLength + 1, true);
+            }
+            return;
+        }
+        FALLTHROUGH;
+    }
+
+    case ArrayWithArrayStorage: {
+        ArrayStorage* storage = butterfly->arrayStorage();
+
+        // Fast case - push within vector, always update m_length & m_numValuesInVector.
+        unsigned length = storage->length();
+        if (length < storage->vectorLength()) {
+            storage->m_vector[length].set(vm, this, value);
+            storage->setLength(length + 1);
+            ++storage->m_numValuesInVector;
+            return;
+        }
+
+        // Pushing to an array of invalid length (2^31-1) stores the property, but throws a range error.
+        if (UNLIKELY(storage->length() > MAX_ARRAY_INDEX)) {
+            methodTable(vm)->putByIndex(this, exec, storage->length(), value, true);
+            // Per ES5.1 15.4.4.7 step 6 & 15.4.5.1 step 3.d.
+            if (!scope.exception())
+                throwException(exec, scope, createRangeError(exec, ASCIILiteral(LengthExceededTheMaximumArrayLengthError)));
+            return;
+        }
+
+        // Handled the same as putIndex.
+        scope.release();
+        putByIndexBeyondVectorLengthWithArrayStorage(exec, storage->length(), value, true, storage);
+        return;
+    }
+
+    default:
+        RELEASE_ASSERT_NOT_REACHED();
+    }
+}
+
 } // namespace JSC