Changeset 280760 in webkit

Timestamp:

Aug 7, 2021, 2:38:59 PM (5 years ago)

Author:

keith_miller@apple.com

Message:

for-in should only emit one loop in bytecode
​https://bugs.webkit.org/show_bug.cgi?id=227989

Reviewed by Yusuke Suzuki.

JSTests:

• microbenchmarks/for-in-double-array-with-own-named.js: Added.

(test):

• microbenchmarks/for-in-double-array.js: Added.

(test):

• microbenchmarks/for-in-getters.js: Added.

(test):

• microbenchmarks/for-in-int32-array-with-own-named.js: Added.

(test):

• microbenchmarks/for-in-int32-array.js: Added.

(test):

• microbenchmarks/for-in-int32-object-with-own-named-and-getters.js: Added.

(test):

• microbenchmarks/for-in-int32-object-with-own-named.js: Added.

(test):

• microbenchmarks/for-in-object-with-own-named.js: Added.

(sum):
(opaqueSet):

• microbenchmarks/for-in-string-array.js: Added.

(test):

• microbenchmarks/for-of-iterate-array-map-set.js: Added.

(sum):
(let.generator):

• stress/for-in-array-mode.js:

(test):

• stress/for-in-base-reassigned-later.js:
• stress/for-in-delete-during-iteration.js:
• stress/for-in-primitive-index-on-prototype.js: Added.

(test):

• stress/for-in-tests.js:
• stress/has-own-property-structure-for-in-loop-correctness.js:

(test5):

Source/JavaScriptCore:

This patch redesigns how we implement for-in loops. Before this patch we would emit three copies of the for-in loop body. One for the indexed properties, one for the named-own properties, and one for generic properties (anything else). This had a couple of problems. Firstly, it meant bytecode size grew exponentially to number of nested for-in loops. This in turn meant DFG/FTL compilation took much longer.

Going off our experience with fast for-of, this patch turns for-in loops specializations into
a "fused" opcode that internally switches on the enumeration mode it currently sees. For example, if we are enumerating an own-named property, the new enumerator_get_by_val bytecode will check the enumerator cell's cached structure matches the base's then load the property offset directly.

There are four new opcodes this patch adds, which replace the various operations we had for the specialized loops previously. The new opcodes are EnumeratorGetByVal, EnumeratorInByVal, EnumeratorHasOwnProperty, and EnumeratorNext. The first three correspond to GetByVal, InByVal, and HasOwnProperty respectively. The EnumeratorNext opcode has three results in bytecode, the next enumeration value's mode, the index of the property name, and the property name string itself. When enumeration is done EnumeratorNext returns JS null as the property name string. Since the DFG doesn't support tuples yet this opcode is spilt into four new nodes. The first computes the updated index and mode for the next enumeration key, which is encoded into a single JS number. Then there are two nodes that extract the mode and index. Finally, the last new node produces the property name string or null based on the extracted mode and index.

Since, in most benchmarks, any given enumeration opcode tends to profile exactly one enumeration mode. This patch focuses primarily on reimplementing all the optimizations we have for any one specific mode. This means there are still potential optimizations for the multi-mode flavors of each new opcode.

The main optimizations implemented for each new opcode are:

EnumeratorNext:
1) IndexedMode loops are loaded and checked for presence inline (DFG/FTL).
2) NamedMode is computed inline as long as the cached structure on the enumerator cell matches the base (Baseline+). This can only differ if there's a transition.
3) property names are extracted from the cached buffer inline (Baseline+).

EnumeratorGetByVal:
EnumeratorInByVal:
EnumeratorHasOwnProperty:
1) IndexedMode has all the optimizations of a normal XByVal on indexed properties (DFG/FTL).
2) NamedMode will extract the value directly from the inline/out-of-line offset if the structure matches the enumerator's (Baseline+).

There are also a few interesting changes worth mentioning here:
1) If a for-in loop would produce an empty enumerator we now always
return the VMs empty enumerator. This has two benefits, most importantly, it distingishes between an unprofiled for-in loop and empty enumeration, which prevents OSR exit loops. Also, it means that the various Enumerator opcodes no longer need to handle undefined/null when toObjecting the base value.

2) The enumerator now contains a bit set of all the modes it will produce. This removes a few extra branches when speculating on the modes we will see in EnumeratorNext.

3) In the DFG, enumerator GetByVal relies on compileGetByVal to set the result it also passes a prefix callback which emits code after the various cases set up their operands but before code is emitting to help satisfy the branch over register allocation validation. Also, the array mode branch in compileGetByVal passes the data format that it would prefer, which for normal GetByVal is returned. For EnumeratorGetByVal, that preference is completely ignored and it always returns DataFormatJS.

• assembler/MacroAssemblerARM64.h:

(JSC::MacroAssemblerARM64::or8):

• assembler/MacroAssemblerX86Common.h:

(JSC::MacroAssemblerX86Common::or8):

• assembler/MacroAssemblerX86_64.h:

(JSC::MacroAssemblerX86_64::rshift64):
(JSC::MacroAssemblerX86_64::or8): Deleted.

• builtins/BuiltinNames.h:
• bytecode/BytecodeList.rb:
• bytecode/BytecodeUseDef.cpp:

(JSC::computeUsesForBytecodeIndexImpl):
(JSC::computeDefsForBytecodeIndexImpl):

• bytecode/CodeBlock.cpp:

(JSC::CodeBlock::finishCreation):

• bytecode/LinkTimeConstant.h:
• bytecode/Opcode.h:
• bytecompiler/BytecodeGenerator.cpp:

(JSC::BytecodeGenerator::recordHasOwnPropertyInForInLoop):
(JSC::BytecodeGenerator::emitInByVal):
(JSC::BytecodeGenerator::emitGetByVal):
(JSC::BytecodeGenerator::emitEnumeratorNext):
(JSC::BytecodeGenerator::emitEnumeratorHasOwnProperty):
(JSC::BytecodeGenerator::pushForInScope):
(JSC::BytecodeGenerator::popForInScope):
(JSC::rewriteOp):
(JSC::ForInContext::finalize):
(JSC::BytecodeGenerator::findForInContext):
(JSC::BytecodeGenerator::recordHasOwnStructurePropertyInForInLoop): Deleted.
(JSC::BytecodeGenerator::emitGetEnumerableLength): Deleted.
(JSC::BytecodeGenerator::emitHasEnumerableIndexedProperty): Deleted.
(JSC::BytecodeGenerator::emitHasEnumerableStructureProperty): Deleted.
(JSC::BytecodeGenerator::emitHasEnumerableProperty): Deleted.
(JSC::BytecodeGenerator::emitHasOwnStructureProperty): Deleted.
(JSC::BytecodeGenerator::emitEnumeratorStructurePropertyName): Deleted.
(JSC::BytecodeGenerator::emitEnumeratorGenericPropertyName): Deleted.
(JSC::BytecodeGenerator::emitToIndexString): Deleted.
(JSC::BytecodeGenerator::pushIndexedForInScope): Deleted.
(JSC::BytecodeGenerator::popIndexedForInScope): Deleted.
(JSC::BytecodeGenerator::pushStructureForInScope): Deleted.
(JSC::BytecodeGenerator::popStructureForInScope): Deleted.
(JSC::StructureForInContext::finalize): Deleted.
(JSC::IndexedForInContext::finalize): Deleted.
(JSC::BytecodeGenerator::findStructureForInContext): Deleted.

• bytecompiler/BytecodeGenerator.h:

(JSC::ForInContext::isValid const):
(JSC::ForInContext::invalidate):
(JSC::ForInContext::local const):
(JSC::ForInContext::propertyName const):
(JSC::ForInContext::propertyOffset const):
(JSC::ForInContext::enumerator const):
(JSC::ForInContext::mode const):
(JSC::ForInContext::ForInContext):
(JSC::ForInContext::bodyBytecodeStartOffset const):
(JSC::ForInContext::type const): Deleted.
(JSC::ForInContext::isIndexedForInContext const): Deleted.
(JSC::ForInContext::isStructureForInContext const): Deleted.
(JSC::ForInContext::asIndexedForInContext): Deleted.
(JSC::ForInContext::asStructureForInContext): Deleted.
(JSC::StructureForInContext::StructureForInContext): Deleted.
(JSC::StructureForInContext::index const): Deleted.
(JSC::StructureForInContext::property const): Deleted.
(JSC::StructureForInContext::enumerator const): Deleted.
(JSC::StructureForInContext::baseVariable const): Deleted.
(JSC::StructureForInContext::addGetInst): Deleted.
(JSC::StructureForInContext::addInInst): Deleted.
(JSC::StructureForInContext::addHasOwnPropertyJump): Deleted.
(JSC::IndexedForInContext::IndexedForInContext): Deleted.
(JSC::IndexedForInContext::index const): Deleted.
(JSC::IndexedForInContext::addGetInst): Deleted.

• bytecompiler/NodesCodegen.cpp:

(JSC::HasOwnPropertyFunctionCallDotNode::emitBytecode):
(JSC::ForInNode::emitBytecode):

• dfg/DFGAbstractInterpreterInlines.h:

(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):

• dfg/DFGArrayMode.h:

(JSC::DFG::ArrayMode::isSaneChain const):

• dfg/DFGBackwardsPropagationPhase.cpp:

(JSC::DFG::BackwardsPropagationPhase::propagate):

• dfg/DFGByteCodeParser.cpp:

(JSC::DFG::ByteCodeParser::parseBlock):

• dfg/DFGCFAPhase.cpp:

(JSC::DFG::CFAPhase::injectOSR):

• dfg/DFGCapabilities.cpp:

(JSC::DFG::capabilityLevel):

• dfg/DFGClobberize.h:

(JSC::DFG::clobberize):

• dfg/DFGDoesGC.cpp:

(JSC::DFG::doesGC):

• dfg/DFGFixupPhase.cpp:

(JSC::DFG::FixupPhase::fixupNode):
(JSC::DFG::FixupPhase::setJSArraySaneChainIfPossible):

• dfg/DFGGraph.cpp:

(JSC::DFG::Graph::dump):

• dfg/DFGIntegerRangeOptimizationPhase.cpp:
• dfg/DFGMayExit.cpp:
• dfg/DFGNode.h:

(JSC::DFG::Node::hasHeapPrediction):
(JSC::DFG::Node::hasStorageChild const):
(JSC::DFG::Node::storageChildIndex):
(JSC::DFG::Node::hasArrayMode):
(JSC::DFG::Node::hasEnumeratorMetadata const):
(JSC::DFG::Node::enumeratorMetadata):

• dfg/DFGNodeType.h:
• dfg/DFGOpInfo.h:

(JSC::DFG::OpInfo::OpInfo):

• dfg/DFGOperations.cpp:

(JSC::DFG::JSC_DEFINE_JIT_OPERATION):

• dfg/DFGOperations.h:
• dfg/DFGPredictionPropagationPhase.cpp:
• dfg/DFGSSALoweringPhase.cpp:

(JSC::DFG::SSALoweringPhase::handleNode):

• dfg/DFGSafeToExecute.h:

(JSC::DFG::safeToExecute):

• dfg/DFGSpeculativeJIT.cpp:

(JSC::DFG::JSValueRegsTemporary::JSValueRegsTemporary):
(JSC::DFG::SpeculativeJIT::compileGetByValOnString):
(JSC::DFG::SpeculativeJIT::setIntTypedArrayLoadResult):
(JSC::DFG::SpeculativeJIT::compileGetByValOnIntTypedArray):
(JSC::DFG::SpeculativeJIT::compileGetByValOnFloatTypedArray):
(JSC::DFG::SpeculativeJIT::compileGetByValForObjectWithString):
(JSC::DFG::SpeculativeJIT::compileGetByValForObjectWithSymbol):
(JSC::DFG::SpeculativeJIT::compileGetByValOnDirectArguments):
(JSC::DFG::SpeculativeJIT::compileGetByValOnScopedArguments):
(JSC::DFG::SpeculativeJIT::compileEnumeratorNextUpdateIndexAndMode):
(JSC::DFG::SpeculativeJIT::compileEnumeratorNextExtractIndex):
(JSC::DFG::SpeculativeJIT::compileEnumeratorNextExtractMode):
(JSC::DFG::SpeculativeJIT::compileEnumeratorNextUpdatePropertyName):
(JSC::DFG::SpeculativeJIT::compileEnumeratorGetByVal):
(JSC::DFG::SpeculativeJIT::compileEnumeratorHasProperty):
(JSC::DFG::SpeculativeJIT::compileEnumeratorInByVal):
(JSC::DFG::SpeculativeJIT::compileEnumeratorHasOwnProperty):
(JSC::DFG::SpeculativeJIT::compileHasIndexedProperty):
(JSC::DFG::SpeculativeJIT::compileGetEnumerableLength): Deleted.
(JSC::DFG::SpeculativeJIT::compileHasEnumerableProperty): Deleted.
(JSC::DFG::SpeculativeJIT::compileToIndexString): Deleted.
(JSC::DFG::SpeculativeJIT::compileHasEnumerableStructureProperty): Deleted.
(JSC::DFG::SpeculativeJIT::compileHasOwnStructurePropertyImpl): Deleted.
(JSC::DFG::SpeculativeJIT::compileHasOwnStructureProperty): Deleted.
(JSC::DFG::SpeculativeJIT::compileInStructureProperty): Deleted.
(JSC::DFG::SpeculativeJIT::compileGetEnumeratorPname): Deleted.
(JSC::DFG::SpeculativeJIT::compileGetDirectPname): Deleted.

• dfg/DFGSpeculativeJIT.h:

(JSC::DFG::SpeculativeJIT::allocate):
(JSC::DFG::JSValueOperand::regs):
(JSC::DFG::JSValueOperand::gpr):
(JSC::DFG::StorageOperand::StorageOperand):
(JSC::DFG::StorageOperand::~StorageOperand):
(JSC::DFG::StorageOperand::emplace):
(JSC::DFG::JSValueRegsTemporary::operator bool):
(JSC::DFG::JSValueRegsTemporary::JSValueRegsTemporary):

• dfg/DFGSpeculativeJIT32_64.cpp:

(JSC::DFG::SpeculativeJIT::compileGetByVal):
(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGSpeculativeJIT64.cpp:

(JSC::DFG::SpeculativeJIT::compileGetByVal):
(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGTypeCheckHoistingPhase.cpp:

(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks):
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks):

• ftl/FTLAbstractHeapRepository.h:
• ftl/FTLCapabilities.cpp:

(JSC::FTL::canCompile):

• ftl/FTLLowerDFGToB3.cpp:

(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileGetByValImpl):
(JSC::FTL::DFG::LowerDFGToB3::compileGetByVal):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharAtImpl):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt):
(JSC::FTL::DFG::LowerDFGToB3::compileCompareStrictEq):

• ftl/FTLOutput.h:

(JSC::FTL::Output::phi):

• generator/DSL.rb:
• interpreter/Register.h:
• interpreter/RegisterInlines.h:

(JSC::Register::operator=):

• jit/AssemblyHelpers.h:
• jit/JIT.cpp:

(JSC::JIT::privateCompileMainPass):
(JSC::JIT::privateCompileSlowCases):

• jit/JIT.h:
• jit/JITOpcodes.cpp:

(JSC::JIT::privateCompileHasIndexedProperty):
(JSC::JIT::emit_op_has_structure_propertyImpl): Deleted.
(JSC::JIT::emit_op_has_enumerable_structure_property): Deleted.
(JSC::JIT::emit_op_has_own_structure_property): Deleted.
(JSC::JIT::emit_op_in_structure_property): Deleted.
(JSC::JIT::emit_op_has_enumerable_indexed_property): Deleted.
(JSC::JIT::emitSlow_op_has_enumerable_indexed_property): Deleted.
(JSC::JIT::emit_op_get_direct_pname): Deleted.
(JSC::JIT::emit_op_enumerator_structure_pname): Deleted.
(JSC::JIT::emit_op_enumerator_generic_pname): Deleted.

• jit/JITOpcodes32_64.cpp:

(JSC::JIT::privateCompileHasIndexedProperty):
(JSC::JIT::emit_op_has_structure_propertyImpl): Deleted.
(JSC::JIT::emit_op_has_enumerable_structure_property): Deleted.
(JSC::JIT::emit_op_has_own_structure_property): Deleted.
(JSC::JIT::emit_op_in_structure_property): Deleted.
(JSC::JIT::emit_op_has_enumerable_indexed_property): Deleted.
(JSC::JIT::emitSlow_op_has_enumerable_indexed_property): Deleted.
(JSC::JIT::emit_op_get_direct_pname): Deleted.
(JSC::JIT::emit_op_enumerator_structure_pname): Deleted.
(JSC::JIT::emit_op_enumerator_generic_pname): Deleted.

• jit/JITPropertyAccess.cpp:

(JSC::JIT::generateGetByValSlowCase):
(JSC::JIT::emitSlow_op_get_by_val):
(JSC::JIT::emit_op_enumerator_next):
(JSC::JIT::emit_op_enumerator_get_by_val):
(JSC::JIT::emitSlow_op_enumerator_get_by_val):
(JSC::JIT::emit_enumerator_has_propertyImpl):
(JSC::JIT::emit_op_enumerator_in_by_val):
(JSC::JIT::emit_op_enumerator_has_own_property):

• jit/JITPropertyAccess32_64.cpp:

(JSC::JIT::emit_op_enumerator_next):
(JSC::JIT::emit_op_enumerator_get_by_val):
(JSC::JIT::emitSlow_op_enumerator_get_by_val):
(JSC::JIT::emit_op_enumerator_in_by_val):
(JSC::JIT::emit_op_enumerator_has_own_property):

• llint/LowLevelInterpreter.asm:
• llint/LowLevelInterpreter64.asm:
• runtime/CommonSlowPaths.cpp:

(JSC::JSC_DEFINE_COMMON_SLOW_PATH):

• runtime/CommonSlowPaths.h:
• runtime/FileBasedFuzzerAgent.cpp:

(JSC::FileBasedFuzzerAgent::getPredictionInternal):

• runtime/FileBasedFuzzerAgentBase.cpp:

(JSC::FileBasedFuzzerAgentBase::opcodeAliasForLookupKey):

• runtime/JSGlobalObject.cpp:

(JSC::JSGlobalObject::init):

• runtime/JSPropertyNameEnumerator.cpp:

(JSC::JSPropertyNameEnumerator::JSPropertyNameEnumerator):
(JSC::JSPropertyNameEnumerator::computeNext):

• runtime/JSPropertyNameEnumerator.h:

(JSC::propertyNameEnumerator):

• runtime/PredictionFileCreatingFuzzerAgent.cpp:

(JSC::PredictionFileCreatingFuzzerAgent::getPredictionInternal):

Location:

trunk

Files:

• JSTests/ChangeLog (modified) (1 diff)
• JSTests/microbenchmarks/for-in-double-array-with-own-named.js (added)
• JSTests/microbenchmarks/for-in-double-array.js (added)
• JSTests/microbenchmarks/for-in-getters.js (added)
• JSTests/microbenchmarks/for-in-int32-array-with-own-named.js (added)
• JSTests/microbenchmarks/for-in-int32-array.js (added)
• JSTests/microbenchmarks/for-in-int32-object-with-own-named-and-getters.js (added)
• JSTests/microbenchmarks/for-in-int32-object-with-own-named.js (added)
• JSTests/microbenchmarks/for-in-object-with-own-named.js (added)
• JSTests/microbenchmarks/for-in-string-array.js (added)
• JSTests/microbenchmarks/for-of-iterate-array-map-set.js (added)
• JSTests/stress/for-in-array-mode.js (modified) (2 diffs)
• JSTests/stress/for-in-base-reassigned-later.js (modified) (1 diff)
• JSTests/stress/for-in-delete-during-iteration.js (modified) (2 diffs)
• JSTests/stress/for-in-primitive-index-on-prototype.js (added)
• JSTests/stress/for-in-tests.js (modified) (1 diff)
• JSTests/stress/has-own-property-structure-for-in-loop-correctness.js (modified) (1 diff)
• Source/JavaScriptCore/ChangeLog (modified) (1 diff)
• Source/JavaScriptCore/assembler/MacroAssemblerARM64.h (modified) (1 diff)
• Source/JavaScriptCore/assembler/MacroAssemblerX86Common.h (modified) (2 diffs)
• Source/JavaScriptCore/assembler/MacroAssemblerX86_64.h (modified) (2 diffs)
• Source/JavaScriptCore/builtins/BuiltinNames.h (modified) (1 diff)
• Source/JavaScriptCore/bytecode/BytecodeList.rb (modified) (3 diffs)
• Source/JavaScriptCore/bytecode/BytecodeUseDef.cpp (modified) (7 diffs)
• Source/JavaScriptCore/bytecode/CodeBlock.cpp (modified) (3 diffs)
• Source/JavaScriptCore/bytecode/LinkTimeConstant.h (modified) (1 diff)
• Source/JavaScriptCore/bytecode/Opcode.h (modified) (4 diffs)
• Source/JavaScriptCore/bytecompiler/BytecodeGenerator.cpp (modified) (14 diffs)
• Source/JavaScriptCore/bytecompiler/BytecodeGenerator.h (modified) (7 diffs)
• Source/JavaScriptCore/bytecompiler/NodesCodegen.cpp (modified) (6 diffs)
• Source/JavaScriptCore/dfg/DFGAbstractInterpreterInlines.h (modified) (4 diffs)
• Source/JavaScriptCore/dfg/DFGArrayMode.h (modified) (1 diff)
• Source/JavaScriptCore/dfg/DFGBackwardsPropagationPhase.cpp (modified) (1 diff)
• Source/JavaScriptCore/dfg/DFGByteCodeParser.cpp (modified) (2 diffs)
• Source/JavaScriptCore/dfg/DFGCFAPhase.cpp (modified) (1 diff)
• Source/JavaScriptCore/dfg/DFGCapabilities.cpp (modified) (2 diffs)
• Source/JavaScriptCore/dfg/DFGClobberize.h (modified) (6 diffs)
• Source/JavaScriptCore/dfg/DFGDoesGC.cpp (modified) (4 diffs)
• Source/JavaScriptCore/dfg/DFGFixupPhase.cpp (modified) (8 diffs)
• Source/JavaScriptCore/dfg/DFGGraph.cpp (modified) (1 diff)
• Source/JavaScriptCore/dfg/DFGIntegerRangeOptimizationPhase.cpp (modified) (2 diffs)
• Source/JavaScriptCore/dfg/DFGMayExit.cpp (modified) (1 diff)
• Source/JavaScriptCore/dfg/DFGNode.h (modified) (7 diffs)
• Source/JavaScriptCore/dfg/DFGNodeType.h (modified) (1 diff)
• Source/JavaScriptCore/dfg/DFGOpInfo.h (modified) (2 diffs)
• Source/JavaScriptCore/dfg/DFGOperations.cpp (modified) (2 diffs)
• Source/JavaScriptCore/dfg/DFGOperations.h (modified) (2 diffs)
• Source/JavaScriptCore/dfg/DFGPredictionPropagationPhase.cpp (modified) (7 diffs)
• Source/JavaScriptCore/dfg/DFGSSALoweringPhase.cpp (modified) (1 diff)
• Source/JavaScriptCore/dfg/DFGSafeToExecute.h (modified) (5 diffs)
• Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp (modified) (28 diffs)
• Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h (modified) (16 diffs)
• Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp (modified) (4 diffs)
• Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp (modified) (5 diffs)
• Source/JavaScriptCore/dfg/DFGTypeCheckHoistingPhase.cpp (modified) (2 diffs)
• Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h (modified) (1 diff)
• Source/JavaScriptCore/ftl/FTLCapabilities.cpp (modified) (1 diff)
• Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp (modified) (27 diffs)
• Source/JavaScriptCore/ftl/FTLOutput.h (modified) (3 diffs)
• Source/JavaScriptCore/generator/DSL.rb (modified) (1 diff)
• Source/JavaScriptCore/interpreter/Register.h (modified) (1 diff)
• Source/JavaScriptCore/interpreter/RegisterInlines.h (modified) (1 diff)
• Source/JavaScriptCore/jit/AssemblyHelpers.h (modified) (2 diffs)
• Source/JavaScriptCore/jit/JIT.cpp (modified) (6 diffs)
• Source/JavaScriptCore/jit/JIT.h (modified) (5 diffs)
• Source/JavaScriptCore/jit/JITOpcodes.cpp (modified) (4 diffs)
• Source/JavaScriptCore/jit/JITOpcodes32_64.cpp (modified) (3 diffs)
• Source/JavaScriptCore/jit/JITPropertyAccess.cpp (modified) (4 diffs)
• Source/JavaScriptCore/jit/JITPropertyAccess32_64.cpp (modified) (1 diff)
• Source/JavaScriptCore/llint/LowLevelInterpreter.asm (modified) (2 diffs)
• Source/JavaScriptCore/llint/LowLevelInterpreter64.asm (modified) (1 diff)
• Source/JavaScriptCore/runtime/CommonSlowPaths.cpp (modified) (2 diffs)
• Source/JavaScriptCore/runtime/CommonSlowPaths.h (modified) (1 diff)
• Source/JavaScriptCore/runtime/FileBasedFuzzerAgent.cpp (modified) (1 diff)
• Source/JavaScriptCore/runtime/FileBasedFuzzerAgentBase.cpp (modified) (2 diffs)
• Source/JavaScriptCore/runtime/JSGlobalObject.cpp (modified) (1 diff)
• Source/JavaScriptCore/runtime/JSPropertyNameEnumerator.cpp (modified) (2 diffs)
• Source/JavaScriptCore/runtime/JSPropertyNameEnumerator.h (modified) (7 diffs)
• Source/JavaScriptCore/runtime/PredictionFileCreatingFuzzerAgent.cpp (modified) (2 diffs)

trunk/JSTests/ChangeLog

@@ +1 @@
+2021-08-07  Keith Miller  <keith_miller@apple.com>
+
+        for-in should only emit one loop in bytecode
+        https://bugs.webkit.org/show_bug.cgi?id=227989
+
+        Reviewed by Yusuke Suzuki.
+
+        * microbenchmarks/for-in-double-array-with-own-named.js: Added.
+        (test):
+        * microbenchmarks/for-in-double-array.js: Added.
+        (test):
+        * microbenchmarks/for-in-getters.js: Added.
+        (test):
+        * microbenchmarks/for-in-int32-array-with-own-named.js: Added.
+        (test):
+        * microbenchmarks/for-in-int32-array.js: Added.
+        (test):
+        * microbenchmarks/for-in-int32-object-with-own-named-and-getters.js: Added.
+        (test):
+        * microbenchmarks/for-in-int32-object-with-own-named.js: Added.
+        (test):
+        * microbenchmarks/for-in-object-with-own-named.js: Added.
+        (sum):
+        (opaqueSet):
+        * microbenchmarks/for-in-string-array.js: Added.
+        (test):
+        * microbenchmarks/for-of-iterate-array-map-set.js: Added.
+        (sum):
+        (let.generator):
+        * stress/for-in-array-mode.js:
+        (test):
+        * stress/for-in-base-reassigned-later.js:
+        * stress/for-in-delete-during-iteration.js:
+        * stress/for-in-primitive-index-on-prototype.js: Added.
+        (test):
+        * stress/for-in-tests.js:
+        * stress/has-own-property-structure-for-in-loop-correctness.js:
+        (test5):
+
 2021-08-05  Mikhail R. Gadelha  <mikhail@igalia.com>
 

trunk/JSTests/stress/for-in-array-mode.js

@@ -27 +27 @@
 "(arr) {" + "\n" +
 "    var sum = 0;" + "\n" +
-"    for (var i in arr)" + "\n" +
+"    for (var i in arr) {" + "\n" +
 "        sum += arr[i];" + "\n" +
+"    } " +
 "    return sum;" + "\n" +
 "}";
@@ -76 +77 @@
         if ((actual != expected) && (actual != previousResult)) {
             print("FAIL: " + name + ": expected: " + expected + ", actual: " + actual + ", starting @ loop iteration " + i);
+            $vm.breakpoint();
             previousResult = actual;
             failed = true;

trunk/JSTests/stress/for-in-base-reassigned-later.js

@@ -15 +15 @@
     var result = foo({0:0, 1:1, a:2, b:3});
     if (result != 7)
-        throw "Error: bad result: " + result;
+        throw new Error("bad result: " + result);
 }

trunk/JSTests/stress/for-in-delete-during-iteration.js

@@ -13 +13 @@
     noInline(foo);
     for (var i = 0; i < 10000; ++i) {
-        if (foo() !== "1235")
-            throw new Error("bad result");
+        var result = foo();
+        if (result !== "1235")
+            throw new Error("bad result got: " + result);
     }
     foo(null);
@@ -37 +38 @@
     noInline(foo);
     for (var i = 0; i < 10000; ++i) {
-        if (foo() !== "xz")
-            throw new Error("bad result");
+        var result = foo();
+        if (result !== "xz")
+            throw new Error("bad result: " + result);
     }
 })();

trunk/JSTests/stress/for-in-tests.js

@@ -106 +106 @@
     }
 })();
+
 (function() {
     var foo = function(a, b) {

trunk/JSTests/stress/has-own-property-structure-for-in-loop-correctness.js

@@ -133 +133 @@
 
     for (let i = 0; i < 1000; ++i) {
-        assert(count(o) === 0);
+        var c = count(o);
+        if (c)
+            print(c);
+        assert(c === 0);
     }
 }

trunk/Source/JavaScriptCore/ChangeLog

@@ +1 @@
+2021-08-07  Keith Miller  <keith_miller@apple.com>
+
+        for-in should only emit one loop in bytecode
+        https://bugs.webkit.org/show_bug.cgi?id=227989
+
+        Reviewed by Yusuke Suzuki.
+
+        This patch redesigns how we implement for-in loops. Before this patch we would emit three copies of the for-in loop body. One for the indexed properties, one for the named-own properties, and one for generic properties (anything else). This had a couple of problems. Firstly, it meant bytecode size grew exponentially to number of nested for-in loops. This in turn meant DFG/FTL compilation took much longer.
+
+        Going off our experience with fast for-of, this patch turns for-in loops specializations into
+        a "fused" opcode that internally switches on the enumeration mode it currently sees. For example, if we are enumerating an own-named property, the new enumerator_get_by_val bytecode will check the enumerator cell's cached structure matches the base's then load the property offset directly.
+
+        There are four new opcodes this patch adds, which replace the various operations we had for the specialized loops previously. The new opcodes are EnumeratorGetByVal, EnumeratorInByVal, EnumeratorHasOwnProperty, and EnumeratorNext. The first three correspond to GetByVal, InByVal, and HasOwnProperty respectively. The EnumeratorNext opcode has three results in bytecode, the next enumeration value's mode, the index of the property name, and the property name string itself. When enumeration is done EnumeratorNext returns JS null as the property name string. Since the DFG doesn't support tuples yet this opcode is spilt into four new nodes. The first computes the updated index and mode for the next enumeration key, which is encoded into a single JS number. Then there are two nodes that extract the mode and index. Finally, the last new node produces the property name string or null based on the extracted mode and index.
+
+        Since, in most benchmarks, any given enumeration opcode tends to profile exactly one enumeration mode. This patch focuses primarily on reimplementing all the optimizations we have for any one specific mode. This means there are still potential optimizations for the multi-mode flavors of each new opcode.
+
+        The main optimizations implemented for each new opcode are:
+
+        EnumeratorNext:
+        1) IndexedMode loops are loaded and checked for presence inline (DFG/FTL).
+        2) NamedMode is computed inline as long as the cached structure on the enumerator cell matches the base (Baseline+). This can only differ if there's a transition.
+        3) property names are extracted from the cached buffer inline (Baseline+).
+
+        EnumeratorGetByVal:
+        EnumeratorInByVal:
+        EnumeratorHasOwnProperty:
+        1) IndexedMode has all the optimizations of a normal XByVal on indexed properties (DFG/FTL).
+        2) NamedMode will extract the value directly from the inline/out-of-line offset if the structure matches the enumerator's (Baseline+).
+
+        There are also a few interesting changes worth mentioning here:
+        1) If a for-in loop would produce an empty enumerator we now always
+        return the VMs empty enumerator. This has two benefits, most importantly, it distingishes between an unprofiled for-in loop and empty enumeration, which prevents OSR exit loops. Also, it means that the various Enumerator opcodes no longer need to handle undefined/null when `toObject`ing the base value.
+
+        2) The enumerator now contains a bit set of all the modes it will produce. This removes a few extra branches when speculating on the modes we will see in EnumeratorNext.
+
+        3) In the DFG, enumerator GetByVal relies on compileGetByVal to set the result it also passes a prefix callback which emits code after the various cases set up their operands but before code is emitting to help satisfy the branch over register allocation validation. Also, the array mode branch in compileGetByVal passes the data format that it would prefer, which for normal GetByVal is returned. For EnumeratorGetByVal, that preference is completely ignored and it always returns DataFormatJS.
+
+        * assembler/MacroAssemblerARM64.h:
+        (JSC::MacroAssemblerARM64::or8):
+        * assembler/MacroAssemblerX86Common.h:
+        (JSC::MacroAssemblerX86Common::or8):
+        * assembler/MacroAssemblerX86_64.h:
+        (JSC::MacroAssemblerX86_64::rshift64):
+        (JSC::MacroAssemblerX86_64::or8): Deleted.
+        * builtins/BuiltinNames.h:
+        * bytecode/BytecodeList.rb:
+        * bytecode/BytecodeUseDef.cpp:
+        (JSC::computeUsesForBytecodeIndexImpl):
+        (JSC::computeDefsForBytecodeIndexImpl):
+        * bytecode/CodeBlock.cpp:
+        (JSC::CodeBlock::finishCreation):
+        * bytecode/LinkTimeConstant.h:
+        * bytecode/Opcode.h:
+        * bytecompiler/BytecodeGenerator.cpp:
+        (JSC::BytecodeGenerator::recordHasOwnPropertyInForInLoop):
+        (JSC::BytecodeGenerator::emitInByVal):
+        (JSC::BytecodeGenerator::emitGetByVal):
+        (JSC::BytecodeGenerator::emitEnumeratorNext):
+        (JSC::BytecodeGenerator::emitEnumeratorHasOwnProperty):
+        (JSC::BytecodeGenerator::pushForInScope):
+        (JSC::BytecodeGenerator::popForInScope):
+        (JSC::rewriteOp):
+        (JSC::ForInContext::finalize):
+        (JSC::BytecodeGenerator::findForInContext):
+        (JSC::BytecodeGenerator::recordHasOwnStructurePropertyInForInLoop): Deleted.
+        (JSC::BytecodeGenerator::emitGetEnumerableLength): Deleted.
+        (JSC::BytecodeGenerator::emitHasEnumerableIndexedProperty): Deleted.
+        (JSC::BytecodeGenerator::emitHasEnumerableStructureProperty): Deleted.
+        (JSC::BytecodeGenerator::emitHasEnumerableProperty): Deleted.
+        (JSC::BytecodeGenerator::emitHasOwnStructureProperty): Deleted.
+        (JSC::BytecodeGenerator::emitEnumeratorStructurePropertyName): Deleted.
+        (JSC::BytecodeGenerator::emitEnumeratorGenericPropertyName): Deleted.
+        (JSC::BytecodeGenerator::emitToIndexString): Deleted.
+        (JSC::BytecodeGenerator::pushIndexedForInScope): Deleted.
+        (JSC::BytecodeGenerator::popIndexedForInScope): Deleted.
+        (JSC::BytecodeGenerator::pushStructureForInScope): Deleted.
+        (JSC::BytecodeGenerator::popStructureForInScope): Deleted.
+        (JSC::StructureForInContext::finalize): Deleted.
+        (JSC::IndexedForInContext::finalize): Deleted.
+        (JSC::BytecodeGenerator::findStructureForInContext): Deleted.
+        * bytecompiler/BytecodeGenerator.h:
+        (JSC::ForInContext::isValid const):
+        (JSC::ForInContext::invalidate):
+        (JSC::ForInContext::local const):
+        (JSC::ForInContext::propertyName const):
+        (JSC::ForInContext::propertyOffset const):
+        (JSC::ForInContext::enumerator const):
+        (JSC::ForInContext::mode const):
+        (JSC::ForInContext::ForInContext):
+        (JSC::ForInContext::bodyBytecodeStartOffset const):
+        (JSC::ForInContext::type const): Deleted.
+        (JSC::ForInContext::isIndexedForInContext const): Deleted.
+        (JSC::ForInContext::isStructureForInContext const): Deleted.
+        (JSC::ForInContext::asIndexedForInContext): Deleted.
+        (JSC::ForInContext::asStructureForInContext): Deleted.
+        (JSC::StructureForInContext::StructureForInContext): Deleted.
+        (JSC::StructureForInContext::index const): Deleted.
+        (JSC::StructureForInContext::property const): Deleted.
+        (JSC::StructureForInContext::enumerator const): Deleted.
+        (JSC::StructureForInContext::baseVariable const): Deleted.
+        (JSC::StructureForInContext::addGetInst): Deleted.
+        (JSC::StructureForInContext::addInInst): Deleted.
+        (JSC::StructureForInContext::addHasOwnPropertyJump): Deleted.
+        (JSC::IndexedForInContext::IndexedForInContext): Deleted.
+        (JSC::IndexedForInContext::index const): Deleted.
+        (JSC::IndexedForInContext::addGetInst): Deleted.
+        * bytecompiler/NodesCodegen.cpp:
+        (JSC::HasOwnPropertyFunctionCallDotNode::emitBytecode):
+        (JSC::ForInNode::emitBytecode):
+        * dfg/DFGAbstractInterpreterInlines.h:
+        (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
+        * dfg/DFGArrayMode.h:
+        (JSC::DFG::ArrayMode::isSaneChain const):
+        * dfg/DFGBackwardsPropagationPhase.cpp:
+        (JSC::DFG::BackwardsPropagationPhase::propagate):
+        * dfg/DFGByteCodeParser.cpp:
+        (JSC::DFG::ByteCodeParser::parseBlock):
+        * dfg/DFGCFAPhase.cpp:
+        (JSC::DFG::CFAPhase::injectOSR):
+        * dfg/DFGCapabilities.cpp:
+        (JSC::DFG::capabilityLevel):
+        * dfg/DFGClobberize.h:
+        (JSC::DFG::clobberize):
+        * dfg/DFGDoesGC.cpp:
+        (JSC::DFG::doesGC):
+        * dfg/DFGFixupPhase.cpp:
+        (JSC::DFG::FixupPhase::fixupNode):
+        (JSC::DFG::FixupPhase::setJSArraySaneChainIfPossible):
+        * dfg/DFGGraph.cpp:
+        (JSC::DFG::Graph::dump):
+        * dfg/DFGIntegerRangeOptimizationPhase.cpp:
+        * dfg/DFGMayExit.cpp:
+        * dfg/DFGNode.h:
+        (JSC::DFG::Node::hasHeapPrediction):
+        (JSC::DFG::Node::hasStorageChild const):
+        (JSC::DFG::Node::storageChildIndex):
+        (JSC::DFG::Node::hasArrayMode):
+        (JSC::DFG::Node::hasEnumeratorMetadata const):
+        (JSC::DFG::Node::enumeratorMetadata):
+        * dfg/DFGNodeType.h:
+        * dfg/DFGOpInfo.h:
+        (JSC::DFG::OpInfo::OpInfo):
+        * dfg/DFGOperations.cpp:
+        (JSC::DFG::JSC_DEFINE_JIT_OPERATION):
+        * dfg/DFGOperations.h:
+        * dfg/DFGPredictionPropagationPhase.cpp:
+        * dfg/DFGSSALoweringPhase.cpp:
+        (JSC::DFG::SSALoweringPhase::handleNode):
+        * dfg/DFGSafeToExecute.h:
+        (JSC::DFG::safeToExecute):
+        * dfg/DFGSpeculativeJIT.cpp:
+        (JSC::DFG::JSValueRegsTemporary::JSValueRegsTemporary):
+        (JSC::DFG::SpeculativeJIT::compileGetByValOnString):
+        (JSC::DFG::SpeculativeJIT::setIntTypedArrayLoadResult):
+        (JSC::DFG::SpeculativeJIT::compileGetByValOnIntTypedArray):
+        (JSC::DFG::SpeculativeJIT::compileGetByValOnFloatTypedArray):
+        (JSC::DFG::SpeculativeJIT::compileGetByValForObjectWithString):
+        (JSC::DFG::SpeculativeJIT::compileGetByValForObjectWithSymbol):
+        (JSC::DFG::SpeculativeJIT::compileGetByValOnDirectArguments):
+        (JSC::DFG::SpeculativeJIT::compileGetByValOnScopedArguments):
+        (JSC::DFG::SpeculativeJIT::compileEnumeratorNextUpdateIndexAndMode):
+        (JSC::DFG::SpeculativeJIT::compileEnumeratorNextExtractIndex):
+        (JSC::DFG::SpeculativeJIT::compileEnumeratorNextExtractMode):
+        (JSC::DFG::SpeculativeJIT::compileEnumeratorNextUpdatePropertyName):
+        (JSC::DFG::SpeculativeJIT::compileEnumeratorGetByVal):
+        (JSC::DFG::SpeculativeJIT::compileEnumeratorHasProperty):
+        (JSC::DFG::SpeculativeJIT::compileEnumeratorInByVal):
+        (JSC::DFG::SpeculativeJIT::compileEnumeratorHasOwnProperty):
+        (JSC::DFG::SpeculativeJIT::compileHasIndexedProperty):
+        (JSC::DFG::SpeculativeJIT::compileGetEnumerableLength): Deleted.
+        (JSC::DFG::SpeculativeJIT::compileHasEnumerableProperty): Deleted.
+        (JSC::DFG::SpeculativeJIT::compileToIndexString): Deleted.
+        (JSC::DFG::SpeculativeJIT::compileHasEnumerableStructureProperty): Deleted.
+        (JSC::DFG::SpeculativeJIT::compileHasOwnStructurePropertyImpl): Deleted.
+        (JSC::DFG::SpeculativeJIT::compileHasOwnStructureProperty): Deleted.
+        (JSC::DFG::SpeculativeJIT::compileInStructureProperty): Deleted.
+        (JSC::DFG::SpeculativeJIT::compileGetEnumeratorPname): Deleted.
+        (JSC::DFG::SpeculativeJIT::compileGetDirectPname): Deleted.
+        * dfg/DFGSpeculativeJIT.h:
+        (JSC::DFG::SpeculativeJIT::allocate):
+        (JSC::DFG::JSValueOperand::regs):
+        (JSC::DFG::JSValueOperand::gpr):
+        (JSC::DFG::StorageOperand::StorageOperand):
+        (JSC::DFG::StorageOperand::~StorageOperand):
+        (JSC::DFG::StorageOperand::emplace):
+        (JSC::DFG::JSValueRegsTemporary::operator bool):
+        (JSC::DFG::JSValueRegsTemporary::JSValueRegsTemporary):
+        * dfg/DFGSpeculativeJIT32_64.cpp:
+        (JSC::DFG::SpeculativeJIT::compileGetByVal):
+        (JSC::DFG::SpeculativeJIT::compile):
+        * dfg/DFGSpeculativeJIT64.cpp:
+        (JSC::DFG::SpeculativeJIT::compileGetByVal):
+        (JSC::DFG::SpeculativeJIT::compile):
+        * dfg/DFGTypeCheckHoistingPhase.cpp:
+        (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks):
+        (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks):
+        * ftl/FTLAbstractHeapRepository.h:
+        * ftl/FTLCapabilities.cpp:
+        (JSC::FTL::canCompile):
+        * ftl/FTLLowerDFGToB3.cpp:
+        (JSC::FTL::DFG::LowerDFGToB3::compileNode):
+        (JSC::FTL::DFG::LowerDFGToB3::compileGetByValImpl):
+        (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal):
+        (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAtImpl):
+        (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt):
+        (JSC::FTL::DFG::LowerDFGToB3::compileCompareStrictEq):
+        * ftl/FTLOutput.h:
+        (JSC::FTL::Output::phi):
+        * generator/DSL.rb:
+        * interpreter/Register.h:
+        * interpreter/RegisterInlines.h:
+        (JSC::Register::operator=):
+        * jit/AssemblyHelpers.h:
+        * jit/JIT.cpp:
+        (JSC::JIT::privateCompileMainPass):
+        (JSC::JIT::privateCompileSlowCases):
+        * jit/JIT.h:
+        * jit/JITOpcodes.cpp:
+        (JSC::JIT::privateCompileHasIndexedProperty):
+        (JSC::JIT::emit_op_has_structure_propertyImpl): Deleted.
+        (JSC::JIT::emit_op_has_enumerable_structure_property): Deleted.
+        (JSC::JIT::emit_op_has_own_structure_property): Deleted.
+        (JSC::JIT::emit_op_in_structure_property): Deleted.
+        (JSC::JIT::emit_op_has_enumerable_indexed_property): Deleted.
+        (JSC::JIT::emitSlow_op_has_enumerable_indexed_property): Deleted.
+        (JSC::JIT::emit_op_get_direct_pname): Deleted.
+        (JSC::JIT::emit_op_enumerator_structure_pname): Deleted.
+        (JSC::JIT::emit_op_enumerator_generic_pname): Deleted.
+        * jit/JITOpcodes32_64.cpp:
+        (JSC::JIT::privateCompileHasIndexedProperty):
+        (JSC::JIT::emit_op_has_structure_propertyImpl): Deleted.
+        (JSC::JIT::emit_op_has_enumerable_structure_property): Deleted.
+        (JSC::JIT::emit_op_has_own_structure_property): Deleted.
+        (JSC::JIT::emit_op_in_structure_property): Deleted.
+        (JSC::JIT::emit_op_has_enumerable_indexed_property): Deleted.
+        (JSC::JIT::emitSlow_op_has_enumerable_indexed_property): Deleted.
+        (JSC::JIT::emit_op_get_direct_pname): Deleted.
+        (JSC::JIT::emit_op_enumerator_structure_pname): Deleted.
+        (JSC::JIT::emit_op_enumerator_generic_pname): Deleted.
+        * jit/JITPropertyAccess.cpp:
+        (JSC::JIT::generateGetByValSlowCase):
+        (JSC::JIT::emitSlow_op_get_by_val):
+        (JSC::JIT::emit_op_enumerator_next):
+        (JSC::JIT::emit_op_enumerator_get_by_val):
+        (JSC::JIT::emitSlow_op_enumerator_get_by_val):
+        (JSC::JIT::emit_enumerator_has_propertyImpl):
+        (JSC::JIT::emit_op_enumerator_in_by_val):
+        (JSC::JIT::emit_op_enumerator_has_own_property):
+        * jit/JITPropertyAccess32_64.cpp:
+        (JSC::JIT::emit_op_enumerator_next):
+        (JSC::JIT::emit_op_enumerator_get_by_val):
+        (JSC::JIT::emitSlow_op_enumerator_get_by_val):
+        (JSC::JIT::emit_op_enumerator_in_by_val):
+        (JSC::JIT::emit_op_enumerator_has_own_property):
+        * llint/LowLevelInterpreter.asm:
+        * llint/LowLevelInterpreter64.asm:
+        * runtime/CommonSlowPaths.cpp:
+        (JSC::JSC_DEFINE_COMMON_SLOW_PATH):
+        * runtime/CommonSlowPaths.h:
+        * runtime/FileBasedFuzzerAgent.cpp:
+        (JSC::FileBasedFuzzerAgent::getPredictionInternal):
+        * runtime/FileBasedFuzzerAgentBase.cpp:
+        (JSC::FileBasedFuzzerAgentBase::opcodeAliasForLookupKey):
+        * runtime/JSGlobalObject.cpp:
+        (JSC::JSGlobalObject::init):
+        * runtime/JSPropertyNameEnumerator.cpp:
+        (JSC::JSPropertyNameEnumerator::JSPropertyNameEnumerator):
+        (JSC::JSPropertyNameEnumerator::computeNext):
+        * runtime/JSPropertyNameEnumerator.h:
+        (JSC::propertyNameEnumerator):
+        * runtime/PredictionFileCreatingFuzzerAgent.cpp:
+        (JSC::PredictionFileCreatingFuzzerAgent::getPredictionInternal):
+
 2021-08-06  Yusuke Suzuki  <ysuzuki@apple.com>
 

trunk/Source/JavaScriptCore/assembler/MacroAssemblerARM64.h

@@ -1070 +1070 @@
     }
 
+    void or8(RegisterID src, AbsoluteAddress address)
+    {
+        load8(address.m_ptr, getCachedDataTempRegisterIDAndInvalidate());
+        or32(src, dataTempRegister, dataTempRegister);
+        store8(dataTempRegister, address.m_ptr);
+    }
+
     void or8(TrustedImm32 imm, AbsoluteAddress address)
     {

trunk/Source/JavaScriptCore/assembler/MacroAssemblerX86Common.h

@@ -626 +626 @@
     }
 
+    void or8(RegisterID src, AbsoluteAddress address)
+    {
+        move(TrustedImmPtr(address.m_ptr), scratchRegister());
+        or8(src, Address(scratchRegister()));
+    }
+
     void or32(Address src, RegisterID dest)
     {
@@ -664 +670 @@
     {
         m_assembler.orb_im(static_cast<int8_t>(imm.m_value), address.offset, address.base, address.index, address.scale);
+    }
+
+    void or8(TrustedImm32 imm, AbsoluteAddress address)
+    {
+        move(TrustedImmPtr(address.m_ptr), scratchRegister());
+        or8(imm, Address(scratchRegister()));
     }
 

trunk/Source/JavaScriptCore/assembler/MacroAssemblerX86_64.h

@@ -98 +98 @@
     }
 
-    void or8(TrustedImm32 imm, AbsoluteAddress address)
-    {
-        move(TrustedImmPtr(address.m_ptr), scratchRegister());
-        or8(imm, Address(scratchRegister()));
-    }
-
     void sub32(TrustedImm32 imm, AbsoluteAddress address)
     {
@@ -586 +580 @@
             swap(src, X86Registers::ecx);
         }
+    }
+
+    void rshift64(RegisterID src, TrustedImm32 imm, RegisterID dest)
+    {
+        move(src, dest);
+        rshift64(imm, dest);
     }
 

trunk/Source/JavaScriptCore/builtins/BuiltinNames.h

@@ -182 +182 @@
     macro(createPrivateSymbol) \
     macro(entries) \
-    macro(outOfLineReactionCounts)
+    macro(outOfLineReactionCounts) \
+    macro(emptyPropertyNameEnumerator)
 
 

trunk/Source/JavaScriptCore/bytecode/BytecodeList.rb

@@ -30 +30 @@
     :ECMAMode,
     :ErrorTypeWithExtension,
+    :EnumeratorMetadata,
     :GetByIdMode,
     :GetByIdModeMetadata,
@@ -1155 +1156 @@
     }
 
-op :get_enumerable_length,
-    args: {
-        dst: VirtualRegister,
-        base: VirtualRegister,
-    }
-
-op :has_enumerable_indexed_property,
-    args: {
-        dst: VirtualRegister,
-        base: VirtualRegister,
-        property: VirtualRegister,
+op :get_property_enumerator,
+    args: {
+        dst: VirtualRegister,
+        base: VirtualRegister,
+    }
+
+op :enumerator_next,
+    args: {
+        # out
+        propertyName: VirtualRegister,
+        # in/out
+        mode: VirtualRegister, # Will always be a JS UInt32 representing a JSForInMode.
+        index: VirtualRegister, # Gets reset to zero every time mode changes.
+        # in
+        base: VirtualRegister,
+        enumerator: VirtualRegister,
     },
     metadata: {
         arrayProfile: ArrayProfile,
-    }
-
-op :has_enumerable_structure_property,
-    args: {
-        dst: VirtualRegister,
-        base: VirtualRegister,
-        property: VirtualRegister,
-        enumerator: VirtualRegister,
-    }
-
-op :has_own_structure_property,
-    args: {
-        dst: VirtualRegister,
-        base: VirtualRegister,
-        property: VirtualRegister,
-        enumerator: VirtualRegister,
-    }
-
-op :in_structure_property,
-    args: {
-        dst: VirtualRegister,
-        base: VirtualRegister,
-        property: VirtualRegister,
-        enumerator: VirtualRegister,
-    }
-
-op :has_enumerable_property,
-    args: {
-        dst: VirtualRegister,
-        base: VirtualRegister,
-        property: VirtualRegister,
-    }
-
-op :get_direct_pname,
-    args: {
-        dst: VirtualRegister,
-        base: VirtualRegister,
-        property: VirtualRegister,
+        enumeratorMetadata: EnumeratorMetadata,
+    }
+
+op :enumerator_get_by_val,
+    args: {
+        dst: VirtualRegister,
+        base: VirtualRegister,
+        mode: VirtualRegister,
+        propertyName: VirtualRegister,
         index: VirtualRegister,
         enumerator: VirtualRegister,
@@ -1212 +1189 @@
     metadata: {
         profile: ValueProfile,
-    }
-
-op :get_property_enumerator,
-    args: {
-        dst: VirtualRegister,
-        base: VirtualRegister,
-    }
-
-op :enumerator_structure_pname,
-    args: {
-        dst: VirtualRegister,
+        arrayProfile: ArrayProfile,
+        enumeratorMetadata: EnumeratorMetadata,
+    }
+
+op :enumerator_in_by_val,
+    args: {
+        dst: VirtualRegister,
+        base: VirtualRegister,
+        mode: VirtualRegister,
+        propertyName: VirtualRegister,
+        index: VirtualRegister,
         enumerator: VirtualRegister,
+    },
+    metadata: {
+        arrayProfile: ArrayProfile,
+        enumeratorMetadata: EnumeratorMetadata,
+    }
+
+op :enumerator_has_own_property,
+    args: {
+        dst: VirtualRegister,
+        base: VirtualRegister,
+        mode: VirtualRegister,
+        propertyName: VirtualRegister,
         index: VirtualRegister,
-    }
-
-op :enumerator_generic_pname,
-    args: {
-        dst: VirtualRegister,
         enumerator: VirtualRegister,
-        index: VirtualRegister,
-    }
-
-op :to_index_string,
-    args: {
-        dst: VirtualRegister,
-        index: VirtualRegister,
+    },
+    metadata: {
+        arrayProfile: ArrayProfile,
+        enumeratorMetadata: EnumeratorMetadata,
     }
 

trunk/Source/JavaScriptCore/bytecode/BytecodeUseDef.cpp

@@ -164 +164 @@
     USES(OpSpread, argument)
     USES(OpGetPropertyEnumerator, base)
-    USES(OpGetEnumerableLength, base)
     USES(OpNewFuncExp, scope)
     USES(OpNewGeneratorFuncExp, scope)
     USES(OpNewAsyncFuncExp, scope)
-    USES(OpToIndexString, index)
     USES(OpCreateLexicalEnvironment, scope, symbolTable, initialValue)
     USES(OpCreateGeneratorFrameEnvironment, scope, symbolTable, initialValue)
@@ -222 +220 @@
     USES(OpNewArrayBuffer, immutableButterfly)
 
-    USES(OpHasEnumerableIndexedProperty, base, property)
-    USES(OpHasEnumerableStructureProperty, base, property, enumerator)
-    USES(OpHasEnumerableProperty, base, property)
-    USES(OpEnumeratorStructurePname, enumerator, index)
-    USES(OpEnumeratorGenericPname, enumerator, index)
     USES(OpGetByVal, base, property)
     USES(OpGetPrivateName, base, property)
@@ -266 +259 @@
     USES(OpGetByValWithThis, base, thisValue, property)
     USES(OpInstanceofCustom, value, constructor, hasInstanceValue)
-    USES(OpHasOwnStructureProperty, base, property, enumerator)
-    USES(OpInStructureProperty, base, property, enumerator)
 
     case op_call_varargs: {
@@ -285 +276 @@
     }
 
-    USES(OpGetDirectPname, base, property, index, enumerator)
-
     USES(OpSwitchString, scrutinee)
     USES(OpSwitchChar, scrutinee)
@@ -295 +284 @@
 
     USES(OpYield, argument)
+
+    USES(OpEnumeratorNext, mode, index, base, enumerator)
+    USES(OpEnumeratorGetByVal, base, mode, propertyName, index, enumerator)
+    USES(OpEnumeratorInByVal, base, mode, propertyName, index, enumerator)
+    USES(OpEnumeratorHasOwnProperty, base, mode, propertyName, index, enumerator)
 
     case op_iterator_open: {
@@ -425 +419 @@
     // These all have a single destination for the first argument.
     DEFS(OpArgumentCount, dst)
-    DEFS(OpToIndexString, dst)
-    DEFS(OpGetEnumerableLength, dst)
-    DEFS(OpHasEnumerableIndexedProperty, dst)
-    DEFS(OpHasEnumerableStructureProperty, dst)
-    DEFS(OpHasEnumerableProperty, dst)
-    DEFS(OpHasOwnStructureProperty, dst)
-    DEFS(OpInStructureProperty, dst)
-    DEFS(OpGetDirectPname, dst)
     DEFS(OpGetPropertyEnumerator, dst)
-    DEFS(OpEnumeratorStructurePname, dst)
-    DEFS(OpEnumeratorGenericPname, dst)
     DEFS(OpGetParentScope, dst)
     DEFS(OpPushWithScope, dst)
@@ -568 +552 @@
     DEFS(OpCatch, exception, thrownValue)
 
+    DEFS(OpEnumeratorNext, propertyName, mode, index)
+    DEFS(OpEnumeratorGetByVal, dst)
+    DEFS(OpEnumeratorInByVal, dst)
+    DEFS(OpEnumeratorHasOwnProperty, dst)
+
     case op_iterator_open: {
         auto bytecode = instruction->as<OpIteratorOpen>();

trunk/Source/JavaScriptCore/bytecode/CodeBlock.cpp

@@ -473 +473 @@
         m_bytecodeCost += opcodeLengths[opcodeID];
         switch (opcodeID) {
-        LINK(OpHasEnumerableIndexedProperty)
-
         LINK(OpCallVarargs, profile)
         LINK(OpTailCallVarargs, profile)
@@ -482 +480 @@
         LINK(OpGetPrivateName, profile)
 
-        LINK(OpGetDirectPname, profile)
         LINK(OpGetByIdWithThis, profile)
         LINK(OpTryGetById, profile)
@@ -503 +500 @@
 
         LINK(OpGetById, profile)
+
+        LINK(OpEnumeratorNext, profile)
+        LINK(OpEnumeratorInByVal, profile)
+        LINK(OpEnumeratorHasOwnProperty, profile)
+        LINK(OpEnumeratorGetByVal, profile)
 
         LINK(OpCall, profile)

trunk/Source/JavaScriptCore/bytecode/LinkTimeConstant.h

@@ -112 +112 @@
     v(hasOwnPropertyFunction, nullptr) \
     v(createPrivateSymbol, nullptr) \
+    v(emptyPropertyNameEnumerator, nullptr) \
 
 

trunk/Source/JavaScriptCore/bytecode/Opcode.h

@@ -1 +1 @@
 /*
- * Copyright (C) 2008, 2009, 2013, 2014 Apple Inc. All rights reserved.
+ * Copyright (C) 2008-2021 Apple Inc. All rights reserved.
  * Copyright (C) 2008 Cameron Zwarich <cwzwarich@uwaterloo.ca>
  *
@@ -100 +100 @@
     macro(OpConstructVarargs) \
     macro(OpGetByVal) \
-    macro(OpGetDirectPname) \
+    macro(OpEnumeratorGetByVal) \
     macro(OpGetById) \
     macro(OpGetByIdWithThis) \
@@ -128 +128 @@
 
 #define FOR_EACH_OPCODE_WITH_ARRAY_PROFILE(macro) \
-    macro(OpHasEnumerableIndexedProperty) \
     macro(OpCallVarargs) \
     macro(OpTailCallVarargs) \
@@ -137 +136 @@
     macro(OpPutByVal) \
     macro(OpPutByValDirect) \
+    macro(OpEnumeratorNext) \
+    macro(OpEnumeratorGetByVal) \
+    macro(OpEnumeratorInByVal) \
+    macro(OpEnumeratorHasOwnProperty) \
 
 #define FOR_EACH_OPCODE_WITH_ARRAY_ALLOCATION_PROFILE(macro) \

trunk/Source/JavaScriptCore/bytecompiler/BytecodeGenerator.cpp

@@ -1504 +1504 @@
 }
 
-void BytecodeGenerator::recordHasOwnStructurePropertyInForInLoop(StructureForInContext& structureContext, unsigned branchOffset, Label& genericPath)
+void BytecodeGenerator::recordHasOwnPropertyInForInLoop(ForInContext& context, unsigned branchOffset, Label& genericPath)
 {
     RELEASE_ASSERT(genericPath.isBound());
     RELEASE_ASSERT(!genericPath.isForward());
-    structureContext.addHasOwnPropertyJump(branchOffset, genericPath.location());
+    context.addHasOwnPropertyJump(branchOffset, genericPath.location());
 }
 
@@ -2582 +2582 @@
             continue;
 
-        if (!context.isStructureForInContext())
-            break;
-
-        StructureForInContext& structureContext = context.asStructureForInContext();
-        OpInStructureProperty::emit<OpcodeSize::Wide32>(this, dst, base, property, structureContext.enumerator());
-        structureContext.addInInst(m_lastInstruction.offset(), property->index());
+        OpEnumeratorInByVal::emit<OpcodeSize::Wide32>(this, dst, base, context.mode(), property, context.propertyOffset(), context.enumerator());
+        context.addInInst(m_lastInstruction.offset(), property->index());
         return dst;
     }
@@ -2735 +2731 @@
             continue;
 
-        if (context.isIndexedForInContext()) {
-            auto& indexedContext = context.asIndexedForInContext();
-            kill(dst);
-            if (OpGetByVal::checkWithoutMetadataID<OpcodeSize::Narrow>(this, dst, base, property))
-                OpGetByVal::emitWithSmallestSizeRequirement<OpcodeSize::Narrow>(this, dst, base, indexedContext.index());
-            else if (OpGetByVal::checkWithoutMetadataID<OpcodeSize::Wide16>(this, dst, base, property))
-                OpGetByVal::emitWithSmallestSizeRequirement<OpcodeSize::Wide16>(this, dst, base, indexedContext.index());
-            else
-                OpGetByVal::emit<OpcodeSize::Wide32>(this, dst, base, indexedContext.index());
-            indexedContext.addGetInst(m_lastInstruction.offset(), property->index());
-            return dst;
-        }
-
-        // We cannot do the above optimization here since OpGetDirectPname => OpGetByVal conversion involves different metadata ID allocation.
-        StructureForInContext& structureContext = context.asStructureForInContext();
-        OpGetDirectPname::emit<OpcodeSize::Wide32>(this, kill(dst), base, property, structureContext.index(), structureContext.enumerator());
-
-        structureContext.addGetInst(m_lastInstruction.offset(), property->index());
+        // FIXME: We should have a better bytecode rewriter that can resize chunks.
+        OpEnumeratorGetByVal::emit<OpcodeSize::Wide32>(this, kill(dst), base, context.mode(), property, context.propertyOffset(), context.enumerator());
+        context.addGetInst(m_lastInstruction.offset(), property->index());
         return dst;
     }
@@ -4553 +4534 @@
 }
 
-RegisterID* BytecodeGenerator::emitGetEnumerableLength(RegisterID* dst, RegisterID* base)
-{
-    OpGetEnumerableLength::emit(this, dst, base);
-    return dst;
-}
-
-RegisterID* BytecodeGenerator::emitHasEnumerableIndexedProperty(RegisterID* dst, RegisterID* base, RegisterID* propertyName)
-{
-    OpHasEnumerableIndexedProperty::emit(this, dst, base, propertyName);
-    return dst;
-}
-
-RegisterID* BytecodeGenerator::emitHasEnumerableStructureProperty(RegisterID* dst, RegisterID* base, RegisterID* propertyName, RegisterID* enumerator)
-{
-    OpHasEnumerableStructureProperty::emit(this, dst, base, propertyName, enumerator);
-    return dst;
-}
-
-RegisterID* BytecodeGenerator::emitHasEnumerableProperty(RegisterID* dst, RegisterID* base, RegisterID* propertyName)
-{
-    OpHasEnumerableProperty::emit(this, dst, base, propertyName);
-    return dst;
-}
-
-RegisterID* BytecodeGenerator::emitHasOwnStructureProperty(RegisterID* dst, RegisterID* base, RegisterID* propertyName, RegisterID* enumerator)
-{
-    OpHasOwnStructureProperty::emit(this, dst, base, propertyName, enumerator);
+void BytecodeGenerator::emitEnumeratorNext(RegisterID* propertyName, RegisterID* mode, RegisterID* index, RegisterID* base, RegisterID* enumerator)
+{
+    OpEnumeratorNext::emit(this, propertyName, mode, index, base, enumerator);
+}
+
+RegisterID* BytecodeGenerator::emitEnumeratorHasOwnProperty(RegisterID* dst, RegisterID* base, RegisterID* mode, RegisterID* propertyName, RegisterID* index, RegisterID* enumerator)
+{
+    OpEnumeratorHasOwnProperty::emit(this, dst, base, mode, propertyName, index, enumerator);
     return dst;
 }
@@ -4589 +4551 @@
 }
 
-RegisterID* BytecodeGenerator::emitEnumeratorStructurePropertyName(RegisterID* dst, RegisterID* enumerator, RegisterID* index)
-{
-    OpEnumeratorStructurePname::emit(this, dst, enumerator, index);
-    return dst;
-}
-
-RegisterID* BytecodeGenerator::emitEnumeratorGenericPropertyName(RegisterID* dst, RegisterID* enumerator, RegisterID* index)
-{
-    OpEnumeratorGenericPname::emit(this, dst, enumerator, index);
-    return dst;
-}
-
-RegisterID* BytecodeGenerator::emitToIndexString(RegisterID* dst, RegisterID* index)
-{
-    OpToIndexString::emit(this, dst, index);
-    return dst;
-}
-
 RegisterID* BytecodeGenerator::emitIsCellWithType(RegisterID* dst, RegisterID* src, JSType type)
 {
@@ -4647 +4591 @@
     OpIsEmpty::emit(this, dst, src);
     return dst;
-}
-
-void BytecodeGenerator::pushIndexedForInScope(RegisterID* localRegister, RegisterID* indexRegister)
-{
-    if (!localRegister)
-        return;
-    unsigned bodyBytecodeStartOffset = instructions().size();
-    m_forInContextStack.append(adoptRef(*new IndexedForInContext(localRegister, indexRegister, bodyBytecodeStartOffset)));
-}
-
-void BytecodeGenerator::popIndexedForInScope(RegisterID* localRegister)
-{
-    if (!localRegister)
-        return;
-    unsigned bodyBytecodeEndOffset = instructions().size();
-    m_forInContextStack.last()->asIndexedForInContext().finalize(*this, m_codeBlock.get(), bodyBytecodeEndOffset);
-    m_forInContextStack.removeLast();
 }
 
@@ -4769 +4696 @@
 }
 
-void BytecodeGenerator::pushStructureForInScope(RegisterID* localRegister, RegisterID* indexRegister, RegisterID* propertyRegister, RegisterID* enumeratorRegister, std::optional<Variable> baseVariable)
+void BytecodeGenerator::pushForInScope(RegisterID* localRegister, RegisterID* propertyNameRegister, RegisterID* propertyOffsetRegister, RegisterID* enumeratorRegister, RegisterID* modeRegister, std::optional<Variable> baseVariable)
 {
     if (!localRegister)
         return;
     unsigned bodyBytecodeStartOffset = instructions().size();
-    m_forInContextStack.append(adoptRef(*new StructureForInContext(localRegister, indexRegister, propertyRegister, enumeratorRegister, baseVariable, bodyBytecodeStartOffset)));
-}
-
-void BytecodeGenerator::popStructureForInScope(RegisterID* localRegister)
+    m_forInContextStack.append(adoptRef(*new ForInContext(localRegister, propertyNameRegister, propertyOffsetRegister, enumeratorRegister, modeRegister, baseVariable, bodyBytecodeStartOffset)));
+}
+
+void BytecodeGenerator::popForInScope(RegisterID* localRegister)
 {
     if (!localRegister)
         return;
     unsigned bodyBytecodeEndOffset = instructions().size();
-    m_forInContextStack.last()->asStructureForInContext().finalize(*this, m_codeBlock.get(), bodyBytecodeEndOffset);
+    m_forInContextStack.last()->finalize(*this, m_codeBlock.get(), bodyBytecodeEndOffset);
     m_forInContextStack.removeLast();
 }
@@ -5385 +5312 @@
 }
 
+template <typename OldOpType, typename NewOpType, typename TupleType>
+ALWAYS_INLINE void rewriteOp(BytecodeGenerator& generator, TupleType& instTuple)
+{
+    unsigned instIndex = std::get<0>(instTuple);
+    int propertyRegIndex = std::get<1>(instTuple);
+    auto instruction = generator.m_writer.ref(instIndex);
+    auto end = instIndex + instruction->size();
+    ASSERT(instruction->isWide32());
+
+    generator.m_writer.seek(instIndex);
+
+    auto bytecode = instruction->as<OldOpType>();
+
+    generator.disablePeepholeOptimization();
+
+    // Change the opcode to get_by_val.
+    // 1. dst stays the same.
+    // 2. base stays the same.
+    // 3. property gets switched to the original property.
+
+    static_assert(sizeof(NewOpType) <= sizeof(OldOpType));
+    NewOpType::emit(&generator, bytecode.m_dst, bytecode.m_base, VirtualRegister(propertyRegIndex));
+
+    // 4. nop out the remaining bytes
+    while (generator.m_writer.position() < end)
+        OpNop::emit<OpcodeSize::Narrow>(&generator);
+}
+
 void ForInContext::finalize(BytecodeGenerator& generator, UnlinkedCodeBlockGenerator* codeBlock, unsigned bodyBytecodeEndOffset)
 {
@@ -5399 +5354 @@
     // reassigned from its original value.
 
-    for (unsigned offset = bodyBytecodeStartOffset(); isValid() && offset < bodyBytecodeEndOffset;) {
+    bool escaped = false;
+    for (unsigned offset = bodyBytecodeStartOffset(); !escaped && offset < bodyBytecodeEndOffset;) {
         auto instruction = generator.instructions().at(offset);
         ASSERT(!instruction->is<OpEnter>());
@@ -5405 +5361 @@
             computeDefsForBytecodeIndex(codeBlock, instruction.ptr(), checkpoint, [&] (VirtualRegister operand) {
                 if (local()->virtualRegister() == operand)
-                    invalidate();
+                    escaped = true;
             });
         }
         offset += instruction->size();
     }
-}
-
-template <typename OldOpType, typename TupleType>
-ALWAYS_INLINE void rewriteOp(BytecodeGenerator& generator, TupleType& instTuple)
-{
-    unsigned instIndex = std::get<0>(instTuple);
-    int propertyRegIndex = std::get<1>(instTuple);
-    auto instruction = generator.m_writer.ref(instIndex);
-    auto end = instIndex + instruction->size();
-    ASSERT(instruction->isWide32());
-
-    generator.m_writer.seek(instIndex);
-
-    auto bytecode = instruction->as<OldOpType>();
-
-    generator.disablePeepholeOptimization();
-
-    // Change the opcode to get_by_val.
-    // 1. dst stays the same.
-    // 2. base stays the same.
-    // 3. property gets switched to the original property.
-
-    if constexpr (std::is_same<OldOpType, OpGetDirectPname>::value) {
-        static_assert(sizeof(OpGetByVal) <= sizeof(OpGetDirectPname));
-        OpGetByVal::emit<OpcodeSize::Wide32>(&generator, bytecode.m_dst, bytecode.m_base, VirtualRegister(propertyRegIndex));
-    } else if constexpr (std::is_same<OldOpType, OpInStructureProperty>::value) {
-        static_assert(sizeof(OpInByVal) <= sizeof(OpInStructureProperty));
-        OpInByVal::emit<OpcodeSize::Wide32>(&generator, bytecode.m_dst, bytecode.m_base, VirtualRegister(propertyRegIndex));
-    } else
-        RELEASE_ASSERT_NOT_REACHED();
-
-    // 4. nop out the remaining bytes
-    while (generator.m_writer.position() < end)
-        OpNop::emit<OpcodeSize::Narrow>(&generator);
-}
-
-void StructureForInContext::finalize(BytecodeGenerator& generator, UnlinkedCodeBlockGenerator* codeBlock, unsigned bodyBytecodeEndOffset)
-{
-    Base::finalize(generator, codeBlock, bodyBytecodeEndOffset);
-    if (isValid())
+
+    if (!escaped)
         return;
 
@@ -5456 +5374 @@
 
     for (const auto& instTuple : m_getInsts)
-        rewriteOp<OpGetDirectPname>(generator, instTuple);
+        rewriteOp<OpEnumeratorGetByVal, OpGetByVal>(generator, instTuple);
 
     for (const auto& instTuple : m_inInsts)
-        rewriteOp<OpInStructureProperty>(generator, instTuple);
+        rewriteOp<OpEnumeratorInByVal, OpInByVal>(generator, instTuple);
 
     for (const auto& hasOwnPropertyTuple : m_hasOwnPropertyJumpInsts) {
@@ -5485 +5403 @@
         generator.m_lastOpcodeID = lastOpcodeID;
         generator.m_lastInstruction = lastInstruction;
-    }
-}
-
-void IndexedForInContext::finalize(BytecodeGenerator& generator, UnlinkedCodeBlockGenerator* codeBlock, unsigned bodyBytecodeEndOffset)
-{
-    Base::finalize(generator, codeBlock, bodyBytecodeEndOffset);
-    if (isValid())
-        return;
-
-    for (const auto& instPair : m_getInsts) {
-        unsigned instIndex = instPair.first;
-        int propertyRegIndex = instPair.second;
-        generator.m_writer.ref(instIndex)->cast<OpGetByVal>()->setProperty(VirtualRegister(propertyRegIndex), []() {
-            ASSERT_NOT_REACHED();
-            return VirtualRegister();
-        });
     }
 }
@@ -5529 +5431 @@
 }
 
-StructureForInContext* BytecodeGenerator::findStructureForInContext(RegisterID* property)
+ForInContext* BytecodeGenerator::findForInContext(RegisterID* property)
 {
     for (size_t i = m_forInContextStack.size(); i--; ) {
@@ -5536 +5438 @@
             continue;
 
-        if (!context.isStructureForInContext())
-            break;
-
-        return &context.asStructureForInContext();
+        return &context;
     }
 

trunk/Source/JavaScriptCore/bytecompiler/BytecodeGenerator.h

@@ -62 +62 @@
     class JSImmutableButterfly;
     class Identifier;
-    class IndexedForInContext;
-    class StructureForInContext;
+    class ForInContext;
 
     enum ExpectedFunction {
@@ -257 +256 @@
         WTF_MAKE_NONCOPYABLE(ForInContext);
     public:
-        virtual ~ForInContext() = default;
-
-        bool isValid() const { return m_isValid; }
-        void invalidate() { m_isValid = false; }
-
-        enum class Type : uint8_t {
-            IndexedForIn,
-            StructureForIn
-        };
-
-        Type type() const { return m_type; }
-        bool isIndexedForInContext() const { return m_type == Type::IndexedForIn; }
-        bool isStructureForInContext() const { return m_type == Type::StructureForIn; }
-
-        IndexedForInContext& asIndexedForInContext()
-        {
-            ASSERT(isIndexedForInContext());
-            return *reinterpret_cast<IndexedForInContext*>(this);
-        }
-
-        StructureForInContext& asStructureForInContext()
-        {
-            ASSERT(isStructureForInContext());
-            return *reinterpret_cast<StructureForInContext*>(this);
-        }
-
-        RegisterID* local() const { return m_localRegister.get(); }
-
-    protected:
-        ForInContext(RegisterID* localRegister, Type type, unsigned bodyBytecodeStartOffset)
-            : m_localRegister(localRegister)
-            , m_type(type)
-            , m_bodyBytecodeStartOffset(bodyBytecodeStartOffset)
-        { }
-
-        unsigned bodyBytecodeStartOffset() const { return m_bodyBytecodeStartOffset; }
-
-        void finalize(BytecodeGenerator&, UnlinkedCodeBlockGenerator*, unsigned bodyBytecodeEndOffset);
-
-    private:
-        RefPtr<RegisterID> m_localRegister;
-        bool m_isValid { true };
-        Type m_type;
-        unsigned m_bodyBytecodeStartOffset;
-    };
-
-    class StructureForInContext : public ForInContext {
-        using Base = ForInContext;
-    public:
         using GetInst = std::tuple<unsigned, int>;
         using InInst = GetInst;
         using HasOwnPropertyJumpInst = std::tuple<unsigned, unsigned>;
 
-        StructureForInContext(RegisterID* localRegister, RegisterID* indexRegister, RegisterID* propertyRegister, RegisterID* enumeratorRegister, std::optional<Variable> baseVariable, unsigned bodyBytecodeStartOffset)
-            : ForInContext(localRegister, Type::StructureForIn, bodyBytecodeStartOffset)
-            , m_indexRegister(indexRegister)
-            , m_propertyRegister(propertyRegister)
-            , m_enumeratorRegister(enumeratorRegister)
+        bool isValid() const { return m_isValid; }
+        void invalidate() { m_isValid = false; }
+
+        RegisterID* local() const { return m_localRegister.get(); }
+        RegisterID* propertyName() const { return m_propertyName.get(); }
+        RegisterID* propertyOffset() const { return m_propertyOffset.get(); }
+        RegisterID* enumerator() const { return m_enumerator.get(); }
+        RegisterID* mode() const { return m_mode.get(); }
+        const std::optional<Variable>& baseVariable() const { return m_baseVariable; }
+
+        void addGetInst(unsigned instIndex, int propertyRegIndex)
+        {
+            m_getInsts.append(GetInst { instIndex, propertyRegIndex });
+        }
+
+        void addInInst(unsigned instIndex, int propertyRegIndex)
+        {
+            m_inInsts.append(InInst { instIndex, propertyRegIndex });
+        }
+
+        void addHasOwnPropertyJump(unsigned branchInstIndex, unsigned genericPathTarget)
+        {
+            m_hasOwnPropertyJumpInsts.append(HasOwnPropertyJumpInst { branchInstIndex, genericPathTarget });
+        }
+
+        ForInContext(RegisterID* localRegister, RegisterID* propertyName, RegisterID* propertyOffset, RegisterID* enumerator, RegisterID* mode, std::optional<Variable> baseVariable, unsigned bodyBytecodeStartOffset)
+            : m_localRegister(localRegister)
+            , m_propertyName(propertyName)
+            , m_propertyOffset(propertyOffset)
+            , m_enumerator(enumerator)
+            , m_mode(mode)
             , m_baseVariable(baseVariable)
-        {
-        }
-
-        RegisterID* index() const { return m_indexRegister.get(); }
-        RegisterID* property() const { return m_propertyRegister.get(); }
-        RegisterID* enumerator() const { return m_enumeratorRegister.get(); }
-        const std::optional<Variable>& baseVariable() const { return m_baseVariable; }
-
-        void addGetInst(unsigned instIndex, int propertyRegIndex)
-        {
-            m_getInsts.append(GetInst { instIndex, propertyRegIndex });
-        }
-
-        void addInInst(unsigned instIndex, int propertyRegIndex)
-        {
-            m_inInsts.append(InInst { instIndex, propertyRegIndex });
-        }
-
-        void addHasOwnPropertyJump(unsigned branchInstIndex, unsigned genericPathTarget)
-        {
-            m_hasOwnPropertyJumpInsts.append(HasOwnPropertyJumpInst { branchInstIndex, genericPathTarget });
-        }
+            , m_bodyBytecodeStartOffset(bodyBytecodeStartOffset)
+        { }
+
+        unsigned bodyBytecodeStartOffset() const { return m_bodyBytecodeStartOffset; }
 
         void finalize(BytecodeGenerator&, UnlinkedCodeBlockGenerator*, unsigned bodyBytecodeEndOffset);
 
     private:
-        RefPtr<RegisterID> m_indexRegister;
-        RefPtr<RegisterID> m_propertyRegister;
-        RefPtr<RegisterID> m_enumeratorRegister;
+        RefPtr<RegisterID> m_localRegister;
+        RefPtr<RegisterID> m_propertyName;
+        RefPtr<RegisterID> m_propertyOffset;
+        RefPtr<RegisterID> m_enumerator;
+        RefPtr<RegisterID> m_mode;
         std::optional<Variable> m_baseVariable;
+        bool m_isValid { true };
+        unsigned m_bodyBytecodeStartOffset;
+        Vector<InInst> m_inInsts;
         Vector<GetInst> m_getInsts;
-        Vector<InInst> m_inInsts;
         Vector<HasOwnPropertyJumpInst> m_hasOwnPropertyJumpInsts;
-    };
-
-    class IndexedForInContext : public ForInContext {
-        using Base = ForInContext;
-    public:
-        IndexedForInContext(RegisterID* localRegister, RegisterID* indexRegister, unsigned bodyBytecodeStartOffset)
-            : ForInContext(localRegister, Type::IndexedForIn, bodyBytecodeStartOffset)
-            , m_indexRegister(indexRegister)
-        {
-        }
-
-        RegisterID* index() const { return m_indexRegister.get(); }
-
-        void finalize(BytecodeGenerator&, UnlinkedCodeBlockGenerator*, unsigned bodyBytecodeEndOffset);
-        void addGetInst(unsigned instIndex, int propertyIndex) { m_getInsts.append({ instIndex, propertyIndex }); }
-
-    private:
-        RefPtr<RegisterID> m_indexRegister;
-        Vector<std::pair<unsigned, int>> m_getInsts;
     };
 
@@ -400 +343 @@
 
         friend class FinallyContext;
-        friend class IndexedForInContext;
-        friend class StructureForInContext;
+        friend class ForInContext;
         friend class StrictModeScope;
 
-        template <typename OldOpType, typename TupleType>
+        template <typename OldOpType, typename NewOpType, typename TupleType>
         friend void rewriteOp(BytecodeGenerator&, TupleType&);
 
@@ -682 +624 @@
         void hoistSloppyModeFunctionIfNecessary(const Identifier& functionName);
 
-        StructureForInContext* findStructureForInContext(RegisterID* property);
+        ForInContext* findForInContext(RegisterID* property);
 
     private:
@@ -918 +860 @@
         void emitJumpIfNotFunctionApply(RegisterID* cond, Label& target);
         unsigned emitWideJumpIfNotFunctionHasOwnProperty(RegisterID* cond, Label& target);
-        void recordHasOwnStructurePropertyInForInLoop(StructureForInContext&, unsigned branchOffset, Label& genericPath);
+        void recordHasOwnPropertyInForInLoop(ForInContext&, unsigned branchOffset, Label& genericPath);
 
         template<typename BinOp, typename JmpOp>
@@ -929 +871 @@
         void emitCheckTraps();
 
-        RegisterID* emitHasEnumerableIndexedProperty(RegisterID* dst, RegisterID* base, RegisterID* propertyName);
-        RegisterID* emitHasEnumerableStructureProperty(RegisterID* dst, RegisterID* base, RegisterID* propertyName, RegisterID* enumerator);
-        RegisterID* emitHasEnumerableProperty(RegisterID* dst, RegisterID* base, RegisterID* propertyName);
-        RegisterID* emitHasOwnStructureProperty(RegisterID* dst, RegisterID* base, RegisterID* propertyName, RegisterID* enumerator);
         RegisterID* emitGetPropertyEnumerator(RegisterID* dst, RegisterID* base);
-        RegisterID* emitGetEnumerableLength(RegisterID* dst, RegisterID* base);
-        RegisterID* emitGetStructurePropertyEnumerator(RegisterID* dst, RegisterID* base, RegisterID* length);
-        RegisterID* emitGetGenericPropertyEnumerator(RegisterID* dst, RegisterID* base, RegisterID* length, RegisterID* structureEnumerator);
-        RegisterID* emitEnumeratorStructurePropertyName(RegisterID* dst, RegisterID* enumerator, RegisterID* index);
-        RegisterID* emitEnumeratorGenericPropertyName(RegisterID* dst, RegisterID* enumerator, RegisterID* index);
-        RegisterID* emitToIndexString(RegisterID* dst, RegisterID* index);
+        void emitEnumeratorNext(RegisterID* propertyName, RegisterID* mode, RegisterID* index, RegisterID* base, RegisterID* enumerator);
+        RegisterID* emitEnumeratorHasOwnProperty(RegisterID* dst, RegisterID* base, RegisterID* mode, RegisterID* propertyName, RegisterID* index, RegisterID* enumerator);
 
         RegisterID* emitIsCellWithType(RegisterID* dst, RegisterID* src, JSType);
@@ -1058 +992 @@
         void emitOptionalCheck(RegisterID* src);
 
-        void pushIndexedForInScope(RegisterID* local, RegisterID* index);
-        void popIndexedForInScope(RegisterID* local);
-        void pushStructureForInScope(RegisterID* local, RegisterID* index, RegisterID* property, RegisterID* enumerator, std::optional<Variable> base);
-        void popStructureForInScope(RegisterID* local);
+        void pushForInScope(RegisterID* local, RegisterID* propertyName, RegisterID* propertyOffset, RegisterID* enumerator, RegisterID* mode, std::optional<Variable> base);
+        void popForInScope(RegisterID* local);
 
         LabelScope* breakTarget(const Identifier&);

trunk/Source/JavaScriptCore/bytecompiler/NodesCodegen.cpp

@@ -2132 +2132 @@
     ExpressionNode* argument = m_args->m_listNode->m_expr;
     RELEASE_ASSERT(argument->isResolveNode());
-    StructureForInContext* structureContext = nullptr;
+    ForInContext* context = nullptr;
     Variable argumentVariable = generator.variable(static_cast<ResolveNode*>(argument)->identifier());
     if (argumentVariable.isLocal()) {
         RegisterID* property = argumentVariable.local();
-        structureContext = generator.findStructureForInContext(property);
+        context = generator.findForInContext(property);
     }
 
     auto canUseFastHasOwnProperty = [&] {
-        if (!structureContext)
+        if (!context)
             return false;
-        if (!structureContext->baseVariable())
+        if (!context->baseVariable())
             return false;
         if (m_base->isResolveNode())
-            return generator.variable(static_cast<ResolveNode*>(m_base)->identifier()) == structureContext->baseVariable().value();
+            return generator.variable(static_cast<ResolveNode*>(m_base)->identifier()) == context->baseVariable().value();
         if (m_base->isThisNode()) {
             // After generator.ensureThis (which must be invoked in |base|'s materialization), we can ensure that |this| is in local this-register.
             ASSERT(base);
-            return generator.variable(generator.propertyNames().builtinNames().thisPrivateName(), ThisResolutionType::Local) == structureContext->baseVariable().value();
+            return generator.variable(generator.propertyNames().builtinNames().thisPrivateName(), ThisResolutionType::Local) == context->baseVariable().value();
         }
         return false;
@@ -2161 +2161 @@
 
         unsigned branchInsnOffset = generator.emitWideJumpIfNotFunctionHasOwnProperty(function.get(), realCall.get());
-        generator.emitHasOwnStructureProperty(returnValue.get(), base.get(), generator.emitNode(argument), structureContext->enumerator());
+        generator.emitEnumeratorHasOwnProperty(returnValue.get(), base.get(), context->mode(), generator.emitNode(argument), context->propertyOffset(), context->enumerator());
         generator.emitJump(end.get());
 
@@ -2173 +2173 @@
         generator.emitLabel(end.get());
 
-        generator.recordHasOwnStructurePropertyInForInLoop(*structureContext, branchInsnOffset, realCall);
+        generator.recordHasOwnPropertyInForInLoop(*context, branchInsnOffset, realCall);
     } else {
         CallArguments callArguments(generator, m_args);
@@ -4185 +4185 @@
         generator.emitLoad(dst, jsUndefined());
 
-    Ref<Label> end = generator.newLabel();
-
     RegisterID* forLoopSymbolTable = nullptr;
     generator.pushLexicalScope(this, BytecodeGenerator::ScopeType::LetConstScope, BytecodeGenerator::TDZCheckOptimization::Optimize, BytecodeGenerator::NestedScopeType::IsNested, &forLoopSymbolTable);
@@ -4194 +4192 @@
 
     RefPtr<RegisterID> base = generator.newTemporary();
-    RefPtr<RegisterID> length;
-    RefPtr<RegisterID> enumerator;
 
     generator.emitNode(base.get(), m_expr);
     RefPtr<RegisterID> local = this->tryGetBoundLocal(generator);
-    RefPtr<RegisterID> enumeratorIndex;
+
 
     std::optional<Variable> baseVariable;
@@ -4210 +4206 @@
     }
 
-    // Pause at the assignment expression for each for..in iteration.
-    generator.emitDebugHook(m_lexpr);
-
     int profilerStartOffset = m_statement->startOffset();
     int profilerEndOffset = m_statement->endOffset() + (m_statement->isBlock() ? 1 : 0);
 
-    enumerator = generator.emitGetPropertyEnumerator(generator.newTemporary(), base.get());
-
-    BytecodeGenerator::PreservedTDZStack preservedTDZStack;
-    generator.preserveTDZStack(preservedTDZStack);
-
-    // Indexed property loop.
     {
+        RefPtr<RegisterID> enumerator = generator.newTemporary();
+        RefPtr<RegisterID> mode = generator.emitLoad(generator.newTemporary(), jsNumber(static_cast<unsigned>(JSPropertyNameEnumerator::InitMode)));
+        RefPtr<RegisterID> index = generator.emitLoad(generator.newTemporary(), jsNumber(0));
+        RefPtr<RegisterID> propertyName = generator.newTemporary();
         Ref<LabelScope> scope = generator.newLabelScope(LabelScope::Loop);
-        Ref<Label> loopStart = generator.newLabel();
-        Ref<Label> loopEnd = generator.newLabel();
-
-        length = generator.emitGetEnumerableLength(generator.newTemporary(), enumerator.get());
-        RefPtr<RegisterID> i = generator.emitLoad(generator.newTemporary(), jsNumber(0));
-        RefPtr<RegisterID> propertyName = generator.newTemporary();
-
-        generator.emitLabel(loopStart.get());
+
+        enumerator = generator.emitGetPropertyEnumerator(generator.newTemporary(), base.get());
+        generator.moveLinkTimeConstant(propertyName.get(), LinkTimeConstant::emptyPropertyNameEnumerator);
+        generator.emitEqualityOp<OpStricteq>(propertyName.get(), enumerator.get(), propertyName.get());
+        generator.emitJumpIfTrue(propertyName.get(), scope->breakTarget());
+
+        generator.emitLabel(*scope->continueTarget());
         generator.emitLoopHint();
-
-        RefPtr<RegisterID> result = generator.emitBinaryOp<OpLess>(generator.newTemporary(), i.get(), length.get());
-        generator.emitJumpIfFalse(result.get(), loopEnd.get());
-        generator.emitHasEnumerableIndexedProperty(result.get(), base.get(), i.get());
-        generator.emitJumpIfFalse(result.get(), *scope->continueTarget());
-
-        generator.emitToIndexString(propertyName.get(), i.get());
+        generator.prepareLexicalScopeForNextForLoopIteration(this, forLoopSymbolTable);
+        generator.emitDebugHook(m_lexpr); // Pause at the assignment expression for each for..in iteration.
+
+        // FIXME: We should have a way to see if anyone is actually using the propertyName for something other than a get_by_val. If not, we could eliminate the toString in this opcode.
+        generator.emitEnumeratorNext(propertyName.get(), mode.get(), index.get(), base.get(), enumerator.get());
+
+        // Note, choosing undefined or null helps please DFG's Abstract Interpreter as it doesn't distinguish null and undefined as types (via SpecOther).
+        generator.emitJumpIfTrue(generator.emitIsUndefinedOrNull(generator.newTemporary(), propertyName.get()), scope->breakTarget());
+
         this->emitLoopHeader(generator, propertyName.get());
 
         generator.emitProfileControlFlow(profilerStartOffset);
 
-        generator.pushIndexedForInScope(local.get(), i.get());
+        generator.pushForInScope(local.get(), propertyName.get(), index.get(), enumerator.get(), mode.get(), baseVariable);
         generator.emitNode(dst, m_statement);
-        generator.popIndexedForInScope(local.get());
+        generator.popForInScope(local.get());
 
         generator.emitProfileControlFlow(profilerEndOffset);
-
-        generator.emitLabel(*scope->continueTarget());
-        generator.prepareLexicalScopeForNextForLoopIteration(this, forLoopSymbolTable);
-        generator.emitInc(i.get());
-        generator.emitDebugHook(m_lexpr); // Pause at the assignment expression for each for..in iteration.
-        generator.emitJump(loopStart.get());
+        generator.emitJump(*scope->continueTarget());
 
         generator.emitLabel(scope->breakTarget());
-        generator.emitJump(end.get());
-        generator.emitLabel(loopEnd.get());
-    }
-    generator.restoreTDZStack(preservedTDZStack);
-
-    // Structure property loop.
-    {
-        Ref<LabelScope> scope = generator.newLabelScope(LabelScope::Loop);
-        Ref<Label> loopStart = generator.newLabel();
-        Ref<Label> loopEnd = generator.newLabel();
-
-        enumeratorIndex = generator.emitLoad(generator.newTemporary(), jsNumber(0));
-        RefPtr<RegisterID> propertyName = generator.newTemporary();
-        generator.emitEnumeratorStructurePropertyName(propertyName.get(), enumerator.get(), enumeratorIndex.get());
-
-        generator.emitLabel(loopStart.get());
-        generator.emitLoopHint();
-
-        RefPtr<RegisterID> result = generator.emitIsNull(generator.newTemporary(), propertyName.get());
-        generator.emitJumpIfTrue(result.get(), loopEnd.get());
-        generator.emitHasEnumerableStructureProperty(result.get(), base.get(), propertyName.get(), enumerator.get());
-        generator.emitJumpIfFalse(result.get(), *scope->continueTarget());
-
-        this->emitLoopHeader(generator, propertyName.get());
-
-        generator.emitProfileControlFlow(profilerStartOffset);
-
-        generator.pushStructureForInScope(local.get(), enumeratorIndex.get(), propertyName.get(), enumerator.get(), baseVariable);
-        generator.emitNode(dst, m_statement);
-        generator.popStructureForInScope(local.get());
-
-        generator.emitProfileControlFlow(profilerEndOffset);
-
-        generator.emitLabel(*scope->continueTarget());
-        generator.prepareLexicalScopeForNextForLoopIteration(this, forLoopSymbolTable);
-        generator.emitInc(enumeratorIndex.get());
-        generator.emitEnumeratorStructurePropertyName(propertyName.get(), enumerator.get(), enumeratorIndex.get());
-        generator.emitDebugHook(m_lexpr); // Pause at the assignment expression for each for..in iteration.
-        generator.emitJump(loopStart.get());
-        
-        generator.emitLabel(scope->breakTarget());
-        generator.emitJump(end.get());
-        generator.emitLabel(loopEnd.get());
-    }
-    generator.restoreTDZStack(preservedTDZStack);
-
-    // Generic property loop.
-    {
-        Ref<LabelScope> scope = generator.newLabelScope(LabelScope::Loop);
-        Ref<Label> loopStart = generator.newLabel();
-        Ref<Label> loopEnd = generator.newLabel();
-
-        RefPtr<RegisterID> propertyName = generator.newTemporary();
-
-        generator.emitEnumeratorGenericPropertyName(propertyName.get(), enumerator.get(), enumeratorIndex.get());
-
-        generator.emitLabel(loopStart.get());
-        generator.emitLoopHint();
-
-        RefPtr<RegisterID> result = generator.emitIsNull(generator.newTemporary(), propertyName.get());
-        generator.emitJumpIfTrue(result.get(), loopEnd.get());
-
-        generator.emitHasEnumerableProperty(result.get(), base.get(), propertyName.get());
-        generator.emitJumpIfFalse(result.get(), *scope->continueTarget());
-
-        this->emitLoopHeader(generator, propertyName.get());
-
-        generator.emitProfileControlFlow(profilerStartOffset);
-
-        generator.emitNode(dst, m_statement);
-
-        generator.emitLabel(*scope->continueTarget());
-        generator.prepareLexicalScopeForNextForLoopIteration(this, forLoopSymbolTable);
-        generator.emitInc(enumeratorIndex.get());
-        generator.emitEnumeratorGenericPropertyName(propertyName.get(), enumerator.get(), enumeratorIndex.get());
-        generator.emitDebugHook(m_lexpr); // Pause at the assignment expression for each for..in iteration.
-        generator.emitJump(loopStart.get());
-
-        generator.emitLabel(scope->breakTarget());
-        generator.emitJump(end.get());
-        generator.emitLabel(loopEnd.get());
-    }
-
-    generator.emitLabel(end.get());
+    }
+
     generator.popLexicalScope(this);
     generator.emitProfileControlFlow(profilerEndOffset);

trunk/Source/JavaScriptCore/dfg/DFGAbstractInterpreterInlines.h

@@ -2152 +2152 @@
         setForNode(node, m_vm.stringStructure.get());
         break;
-            
+
+    case EnumeratorGetByVal: {
+        // FIXME: This should be able to do code motion for OwnStructureMode but we'd need to teach AI about the enumerator's cached structure for that to be profitable.
+        clobberWorld();
+        makeHeapTopForNode(node);
+        break;
+    }
+
     case GetByVal:
     case AtomicsAdd:
@@ -4245 +4252 @@
         break;
     }
-            
-    case GetEnumerableLength: {
-        setNonCellTypeForNode(node, SpecInt32Only);
-        break;
-    }
-    case HasEnumerableProperty: {
-        setNonCellTypeForNode(node, SpecBoolean);
-        clobberWorld();
-        break;
-    }
-    case InStructureProperty:
-    case HasOwnStructureProperty:
-    case HasEnumerableStructureProperty: {
-        setNonCellTypeForNode(node, SpecBoolean);
-        clobberWorld();
-        break;
-    }
-    case HasIndexedProperty:
-    case HasEnumerableIndexedProperty: {
+
+    case HasIndexedProperty: {
         ArrayMode mode = node->arrayMode();
         switch (mode.type()) {
@@ -4282 +4272 @@
         break;
     }
-    case GetDirectPname: {
-        clobberWorld();
-        makeHeapTopForNode(node);
-        break;
-    }
+
     case GetPropertyEnumerator: {
         setTypeForNode(node, SpecCell);
@@ -4292 +4278 @@
         break;
     }
-    case GetEnumeratorStructurePname: {
+
+    case EnumeratorNextUpdateIndexAndMode: {
+        ArrayMode arrayMode = node->arrayMode();
+        if (arrayMode.isSaneChain())
+            ASSERT(node->enumeratorMetadata() == JSPropertyNameEnumerator::IndexedMode);
+        else if (node->enumeratorMetadata() != JSPropertyNameEnumerator::OwnStructureMode || m_graph.varArgChild(node, 0).useKind() != CellUse)
+            clobberWorld();
+
+        setNonCellTypeForNode(node, SpecBytecodeNumber);
+        break;
+    }
+
+    case EnumeratorNextExtractMode: {
+        if (node->child1()->enumeratorMetadata() == JSPropertyNameEnumerator::IndexedMode) {
+            setConstant(node, jsNumber(static_cast<uint8_t>(JSPropertyNameEnumerator::IndexedMode)));
+            break;
+        }
+
+        if (node->child1()->enumeratorMetadata() == JSPropertyNameEnumerator::OwnStructureMode && m_graph.varArgChild(node->child1().node(), 0).useKind() == CellUse) {
+            setConstant(node, jsNumber(static_cast<uint8_t>(JSPropertyNameEnumerator::OwnStructureMode)));
+            break;
+        }
+
+        FALLTHROUGH;
+    }
+    case EnumeratorNextExtractIndex: {
+        setNonCellTypeForNode(node, SpecInt32Only);
+        break;
+    }
+
+    case EnumeratorNextUpdatePropertyName: {
         setTypeForNode(node, SpecString | SpecOther);
         break;
     }
-    case GetEnumeratorGenericPname: {
-        setTypeForNode(node, SpecString | SpecOther);
-        break;
-    }
-    case ToIndexString: {
-        setTypeForNode(node, SpecString);
+
+    case EnumeratorInByVal:
+    case EnumeratorHasOwnProperty: {
+        // FIXME: We can determine when the property definitely exists based on abstract
+        // value information.
+        clobberWorld();
+        setNonCellTypeForNode(node, SpecBoolean);
         break;
     }

trunk/Source/JavaScriptCore/dfg/DFGArrayMode.h

@@ -288 +288 @@
     {
         return speculation() == Array::OutOfBoundsSaneChain;
+    }
+
+    bool isSaneChain() const
+    {
+        return isInBoundsSaneChain() || isOutOfBoundsSaneChain();
     }
 

trunk/Source/JavaScriptCore/dfg/DFGBackwardsPropagationPhase.cpp

@@ -391 +391 @@
             break;
         }
-            
+
+        case EnumeratorGetByVal:
         case GetByVal: {
             m_graph.varArgChild(node, 0)->mergeFlags(NodeBytecodeUsesAsValue);

trunk/Source/JavaScriptCore/dfg/DFGByteCodeParser.cpp

@@ -8388 +8388 @@
         }
 
-        case op_get_enumerable_length: {
-            auto bytecode = currentInstruction->as<OpGetEnumerableLength>();
-            set(bytecode.m_dst, addToGraph(GetEnumerableLength, get(bytecode.m_base)));
-            NEXT_OPCODE(op_get_enumerable_length);
-        }
-
-        case op_has_enumerable_structure_property: {
-            auto bytecode = currentInstruction->as<OpHasEnumerableStructureProperty>();
-            set(bytecode.m_dst, addToGraph(HasEnumerableStructureProperty,
-                get(bytecode.m_base),
-                get(bytecode.m_property),
-                get(bytecode.m_enumerator)));
-            NEXT_OPCODE(op_has_enumerable_structure_property);
-        }
-
-        case op_has_enumerable_property: {
-            auto bytecode = currentInstruction->as<OpHasEnumerableProperty>();
-            set(bytecode.m_dst, addToGraph(HasEnumerableProperty, get(bytecode.m_base), get(bytecode.m_property)));
-            NEXT_OPCODE(op_has_enumerable_property);
-        }
-
-        case op_has_own_structure_property: {
-            auto bytecode = currentInstruction->as<OpHasOwnStructureProperty>();
-            set(bytecode.m_dst, addToGraph(HasOwnStructureProperty,
-                get(bytecode.m_base),
-                get(bytecode.m_property),
-                get(bytecode.m_enumerator)));
-            NEXT_OPCODE(op_has_own_structure_property);
-        }
-
-        case op_in_structure_property: {
-            auto bytecode = currentInstruction->as<OpInStructureProperty>();
-            set(bytecode.m_dst, addToGraph(InStructureProperty,
-                get(bytecode.m_base),
-                get(bytecode.m_property),
-                get(bytecode.m_enumerator)));
-            NEXT_OPCODE(op_in_structure_property);
-        }
-
-        case op_has_enumerable_indexed_property: {
-            auto bytecode = currentInstruction->as<OpHasEnumerableIndexedProperty>();
-            Node* base = get(bytecode.m_base);
-            ArrayMode arrayMode = getArrayMode(bytecode.metadata(codeBlock).m_arrayProfile, Array::Read);
-            Node* property = get(bytecode.m_property);
-            addVarArgChild(base);
-            addVarArgChild(property);
-            addVarArgChild(nullptr);
-            Node* hasIterableProperty = addToGraph(Node::VarArg, HasEnumerableIndexedProperty, OpInfo(arrayMode.asWord()));
-            m_exitOK = false; // HasIndexedProperty must be treated as if it clobbers exit state, since FixupPhase may make it generic.
-            set(bytecode.m_dst, hasIterableProperty);
-            NEXT_OPCODE(op_has_enumerable_indexed_property);
-        }
-
-        case op_get_direct_pname: {
-            auto bytecode = currentInstruction->as<OpGetDirectPname>();
-            SpeculatedType prediction = getPredictionWithoutOSRExit();
-            
-            Node* base = get(bytecode.m_base);
-            Node* property = get(bytecode.m_property);
-            Node* index = get(bytecode.m_index);
-            Node* enumerator = get(bytecode.m_enumerator);
-
-            addVarArgChild(base);
-            addVarArgChild(property);
-            addVarArgChild(index);
-            addVarArgChild(enumerator);
-            set(bytecode.m_dst, addToGraph(Node::VarArg, GetDirectPname, OpInfo(0), OpInfo(prediction)));
-
-            NEXT_OPCODE(op_get_direct_pname);
-        }
-
         case op_get_property_enumerator: {
             auto bytecode = currentInstruction->as<OpGetPropertyEnumerator>();
@@ -8465 +8394 @@
         }
 
-        case op_enumerator_structure_pname: {
-            auto bytecode = currentInstruction->as<OpEnumeratorStructurePname>();
-            set(bytecode.m_dst, addToGraph(GetEnumeratorStructurePname,
-                get(bytecode.m_enumerator),
-                get(bytecode.m_index)));
-            NEXT_OPCODE(op_enumerator_structure_pname);
-        }
-
-        case op_enumerator_generic_pname: {
-            auto bytecode = currentInstruction->as<OpEnumeratorGenericPname>();
-            set(bytecode.m_dst, addToGraph(GetEnumeratorGenericPname,
-                get(bytecode.m_enumerator),
-                get(bytecode.m_index)));
-            NEXT_OPCODE(op_enumerator_generic_pname);
-        }
-            
-        case op_to_index_string: {
-            auto bytecode = currentInstruction->as<OpToIndexString>();
-            set(bytecode.m_dst, addToGraph(ToIndexString, get(bytecode.m_index)));
-            NEXT_OPCODE(op_to_index_string);
+        case op_enumerator_next: {
+            auto bytecode = currentInstruction->as<OpEnumeratorNext>();
+            auto& metadata = bytecode.metadata(codeBlock);
+            ArrayMode arrayMode = getArrayMode(metadata.m_arrayProfile, Array::Read);
+            Node* base = get(bytecode.m_base);
+            Node* index = get(bytecode.m_index);
+            Node* enumerator = get(bytecode.m_enumerator);
+            Node* mode = get(bytecode.m_mode);
+
+            auto seenModes = OptionSet<JSPropertyNameEnumerator::Mode>::fromRaw(metadata.m_enumeratorMetadata);
+
+            if (!seenModes)
+                addToGraph(ForceOSRExit);
+
+            addVarArgChild(base);
+            addVarArgChild(index);
+            addVarArgChild(mode);
+            addVarArgChild(enumerator);
+            addVarArgChild(nullptr); // storage for IndexedMode only.
+            Node* updatedIndexAndMode = addToGraph(Node::VarArg, EnumeratorNextUpdateIndexAndMode, OpInfo(arrayMode.asWord()), OpInfo(seenModes));
+
+            Node* updatedMode = addToGraph(EnumeratorNextExtractMode, updatedIndexAndMode);
+            set(bytecode.m_mode, updatedMode);
+
+            Node* updatedIndex = addToGraph(EnumeratorNextExtractIndex, updatedIndexAndMode);
+            set(bytecode.m_index, updatedIndex);
+
+            set(bytecode.m_propertyName, addToGraph(EnumeratorNextUpdatePropertyName, OpInfo(), OpInfo(seenModes), updatedIndex, updatedMode, enumerator));
+
+            NEXT_OPCODE(op_enumerator_next);
+        }
+
+        case op_enumerator_get_by_val: {
+            auto bytecode = currentInstruction->as<OpEnumeratorGetByVal>();
+            auto& metadata = bytecode.metadata(codeBlock);
+            ArrayMode arrayMode = getArrayMode(metadata.m_arrayProfile, Array::Read);
+            SpeculatedType speculation = getPredictionWithoutOSRExit();
+
+            Node* base = get(bytecode.m_base);
+            Node* propertyName = get(bytecode.m_propertyName);
+            Node* index = get(bytecode.m_index);
+            Node* mode = get(bytecode.m_mode);
+            Node* enumerator = get(bytecode.m_enumerator);
+
+            auto seenModes = OptionSet<JSPropertyNameEnumerator::Mode>::fromRaw(metadata.m_enumeratorMetadata);
+            if (!seenModes)
+                addToGraph(ForceOSRExit);
+
+            if (seenModes == JSPropertyNameEnumerator::IndexedMode) {
+                Node* badMode = addToGraph(ArithBitAnd, mode, jsConstant(jsNumber(JSPropertyNameEnumerator::GenericMode | JSPropertyNameEnumerator::OwnStructureMode)));
+
+                // We know the ArithBitAnd cannot have effects so it's ok to exit here.
+                m_exitOK = true;
+                addToGraph(ExitOK);
+
+                addToGraph(CheckIsConstant, OpInfo(m_graph.freezeStrong(jsNumber(0))), badMode);
+
+                addVarArgChild(base);
+                addVarArgChild(index); // Use index so we'll use the normal indexed optimizations.
+                addVarArgChild(nullptr); // For property storage to match GetByVal.
+                set(bytecode.m_dst, addToGraph(Node::VarArg, GetByVal, OpInfo(arrayMode.asWord()), OpInfo(speculation)));
+
+                addToGraph(Phantom, propertyName);
+                addToGraph(Phantom, enumerator);
+                NEXT_OPCODE(op_enumerator_get_by_val);
+            }
+
+            // FIXME: Checking for a BadConstantValue causes us to always use the Generic variant if we switched from IndexedMode -> IndexedMode + OwnStructureMode even though that might be fine.
+            if (!seenModes.containsAny({ JSPropertyNameEnumerator::GenericMode, JSPropertyNameEnumerator::HasSeenOwnStructureModeStructureMismatch })
+                && !m_inlineStackTop->m_exitProfile.hasExitSite(m_currentIndex, BadConstantValue)) {
+                Node* modeTest = addToGraph(SameValue, mode, jsConstant(jsNumber(static_cast<uint8_t>(JSPropertyNameEnumerator::GenericMode))));
+                addToGraph(CheckIsConstant, OpInfo(m_graph.freezeStrong(jsBoolean(false))), modeTest);
+
+                addVarArgChild(base);
+                addVarArgChild(index); // Use index so we'll use the normal indexed optimizations.
+                addVarArgChild(nullptr); // For property storage to match GetByVal.
+                addVarArgChild(index);
+                addVarArgChild(mode);
+                addVarArgChild(enumerator);
+                set(bytecode.m_dst, addToGraph(Node::VarArg, EnumeratorGetByVal, OpInfo(arrayMode.asWord()), OpInfo(speculation)));
+
+                addToGraph(Phantom, propertyName);
+                NEXT_OPCODE(op_enumerator_get_by_val);
+            }
+
+            addVarArgChild(base);
+            addVarArgChild(propertyName);
+            addVarArgChild(nullptr); // For property storage to match GetByVal.
+            addVarArgChild(index);
+            addVarArgChild(mode);
+            addVarArgChild(enumerator);
+            set(bytecode.m_dst, addToGraph(Node::VarArg, EnumeratorGetByVal, OpInfo(arrayMode.asWord()), OpInfo(speculation)));
+
+            NEXT_OPCODE(op_enumerator_get_by_val);
+        }
+
+        case op_enumerator_in_by_val: {
+            auto bytecode = currentInstruction->as<OpEnumeratorInByVal>();
+            auto& metadata = bytecode.metadata(codeBlock);
+            ArrayMode arrayMode = getArrayMode(metadata.m_arrayProfile, Array::Read);
+
+            addVarArgChild(get(bytecode.m_base));
+            addVarArgChild(get(bytecode.m_propertyName));
+            addVarArgChild(get(bytecode.m_index));
+            addVarArgChild(get(bytecode.m_mode));
+            addVarArgChild(get(bytecode.m_enumerator));
+            set(bytecode.m_dst, addToGraph(Node::VarArg, EnumeratorInByVal, OpInfo(arrayMode.asWord()), OpInfo(metadata.m_enumeratorMetadata)));
+
+            NEXT_OPCODE(op_enumerator_in_by_val);
+        }
+
+        case op_enumerator_has_own_property: {
+            auto bytecode = currentInstruction->as<OpEnumeratorHasOwnProperty>();
+            auto& metadata = bytecode.metadata(codeBlock);
+            ArrayMode arrayMode = getArrayMode(metadata.m_arrayProfile, Array::Read);
+
+            addVarArgChild(get(bytecode.m_base));
+            addVarArgChild(get(bytecode.m_propertyName));
+            addVarArgChild(get(bytecode.m_index));
+            addVarArgChild(get(bytecode.m_mode));
+            addVarArgChild(get(bytecode.m_enumerator));
+            set(bytecode.m_dst, addToGraph(Node::VarArg, EnumeratorHasOwnProperty, OpInfo(arrayMode.asWord()), OpInfo(metadata.m_enumeratorMetadata)));
+
+            NEXT_OPCODE(op_enumerator_has_own_property);
         }
 

trunk/Source/JavaScriptCore/dfg/DFGCFAPhase.cpp

@@ -165 +165 @@
         
         // This merges snapshot of stack values while CFA phase want to have proven types and values. This is somewhat tricky.
-        // But this is OK as long as DFG OSR entry validates the inputs with *proven* AbstracValue values. And it turns out that this
+        // But this is OK as long as DFG OSR entry validates the inputs with *proven* AbstractValue values. And it turns out that this
         // type widening is critical to navier-stokes. Without it, navier-stokes has more strict constraint on OSR entry and
         // fails OSR entry repeatedly.

trunk/Source/JavaScriptCore/dfg/DFGCapabilities.cpp

@@ -255 +255 @@
     case op_get_scope:
     case op_get_from_scope:
-    case op_get_enumerable_length:
-    case op_has_enumerable_indexed_property:
-    case op_has_enumerable_structure_property:
-    case op_has_enumerable_property:
-    case op_has_own_structure_property:
-    case op_in_structure_property:
-    case op_get_direct_pname:
     case op_get_property_enumerator:
-    case op_enumerator_structure_pname:
-    case op_enumerator_generic_pname:
-    case op_to_index_string:
     case op_new_func:
     case op_new_func_exp:
@@ -300 +290 @@
     case op_check_private_brand:
     case op_switch_string:
+    case op_enumerator_next:
+    case op_enumerator_get_by_val:
+    case op_enumerator_in_by_val:
+    case op_enumerator_has_own_property:
         return CanCompileAndInline;
 

trunk/Source/JavaScriptCore/dfg/DFGClobberize.h

@@ -148 +148 @@
         case CheckArrayOrEmpty:
             break;
+        case EnumeratorNextUpdateIndexAndMode:
+        case EnumeratorGetByVal:
+        case EnumeratorInByVal:
+        case EnumeratorHasOwnProperty:
         case GetIndexedPropertyStorage:
         case GetArrayLength:
@@ -165 +169 @@
         case ArrayIndexOf:
         case HasIndexedProperty:
-        case HasEnumerableIndexedProperty:
         case AtomicsAdd:
         case AtomicsAnd:
@@ -338 +341 @@
         return;
 
-    case GetEnumerableLength: {
-        read(Heap);
-        write(SideState);
-        return;
-    }
-
-    case ToIndexString:
-    case GetEnumeratorStructurePname:
-    case GetEnumeratorGenericPname: {
+    case EnumeratorNextUpdatePropertyName: {
+        def(PureValue(node, node->enumeratorMetadata().toRaw()));
+        return;
+    }
+
+    case EnumeratorNextExtractMode:
+    case EnumeratorNextExtractIndex: {
         def(PureValue(node));
         return;
     }
 
-    case HasIndexedProperty:
-    case HasEnumerableIndexedProperty: {
+    case EnumeratorNextUpdateIndexAndMode: {
+        read(JSObject_butterfly);
+        if (node->enumeratorMetadata() == JSPropertyNameEnumerator::OwnStructureMode && graph.varArgChild(node, 0).useKind() == CellUse) {
+            read(NamedProperties);
+            read(JSCell_structureID);
+            return;
+        }
+
+        if (node->enumeratorMetadata() != JSPropertyNameEnumerator::IndexedMode) {
+            clobberTop();
+            return;
+        }
+
+        ArrayMode mode = node->arrayMode();
+        switch (mode.type()) {
+        case Array::ForceExit: {
+            write(SideState);
+            return;
+        }
+        case Array::Int32: {
+            if (mode.isSaneChain()) {
+                read(Butterfly_publicLength);
+                read(IndexedInt32Properties);
+                def(HeapLocation(HasIndexedPropertyLoc, IndexedInt32Properties, graph.varArgChild(node, 0), graph.varArgChild(node, 1)), LazyNode(node));
+                return;
+            }
+            break;
+        }
+
+        case Array::Double: {
+            if (mode.isSaneChain()) {
+                read(Butterfly_publicLength);
+                read(IndexedDoubleProperties);
+                def(HeapLocation(HasIndexedPropertyLoc, IndexedDoubleProperties, graph.varArgChild(node, 0), graph.varArgChild(node, 1)), LazyNode(node));
+                return;
+            }
+            break;
+        }
+
+        case Array::Contiguous: {
+            if (mode.isSaneChain()) {
+                read(Butterfly_publicLength);
+                read(IndexedContiguousProperties);
+                def(HeapLocation(HasIndexedPropertyLoc, IndexedContiguousProperties, graph.varArgChild(node, 0), graph.varArgChild(node, 1)), LazyNode(node));
+                return;
+            }
+            break;
+        }
+
+        case Array::ArrayStorage: {
+            if (mode.isInBounds()) {
+                read(Butterfly_vectorLength);
+                read(IndexedArrayStorageProperties);
+                return;
+            }
+            break;
+        }
+
+        default:
+            break;
+        }
+
+        clobberTop();
+        return;
+    }
+
+    case HasIndexedProperty: {
         read(JSObject_butterfly);
         ArrayMode mode = node->arrayMode();
@@ -706 +772 @@
     case ToPropertyKey:
     case InByVal:
+    case EnumeratorInByVal:
+    case EnumeratorHasOwnProperty:
     case InById:
     case HasPrivateName:
@@ -717 +785 @@
     case ResolveScope:
     case ToObject:
-    case HasEnumerableStructureProperty:
-    case HasEnumerableProperty:
-    case HasOwnStructureProperty:
-    case InStructureProperty:
     case GetPropertyEnumerator:
-    case GetDirectPname:
     case InstanceOfCustom:
     case ToNumber:
@@ -905 +968 @@
         for (unsigned i = data->limit; i--;)
             write(AbstractHeap(Stack, data->start + static_cast<int>(i)));
+        return;
+    }
+
+    case EnumeratorGetByVal: {
+        clobberTop();
         return;
     }

trunk/Source/JavaScriptCore/dfg/DFGDoesGC.cpp

@@ -206 +206 @@
     case GetByOffset:
     case GetGetterSetterByOffset:
-    case GetEnumerableLength:
     case FiatInt52:
     case BooleanToNumber:
@@ -298 +297 @@
     case GetByIdWithThis:
     case GetByValWithThis:
-    case GetDirectPname:
     case GetDynamicVar:
     case GetMapBucket:
     case HasIndexedProperty:
-    case HasEnumerableIndexedProperty:
-    case HasEnumerableStructureProperty:
-    case HasEnumerableProperty:
     case HasOwnProperty:
-    case HasOwnStructureProperty:
-    case InStructureProperty:
     case InById:
     case InByVal:
@@ -393 +386 @@
     case ThrowStaticError:
     case GetPropertyEnumerator:
-    case GetEnumeratorStructurePname:
-    case GetEnumeratorGenericPname:
-    case ToIndexString:
+    case EnumeratorInByVal:
+    case EnumeratorHasOwnProperty:
+    case EnumeratorNextUpdatePropertyName:
+    case EnumeratorNextUpdateIndexAndMode:
     case MaterializeNewObject:
     case MaterializeNewInternalFieldObject:
@@ -505 +499 @@
     case GetIndexedPropertyStorage:
     case GetByVal:
+    case EnumeratorGetByVal:
         if (node->arrayMode().type() == Array::String)
             return true;
+        return false;
+
+    case EnumeratorNextExtractMode:
+    case EnumeratorNextExtractIndex:
         return false;
 

trunk/Source/JavaScriptCore/dfg/DFGFixupPhase.cpp

@@ -1025 +1025 @@
         }
 
+        case EnumeratorGetByVal: {
+            fixEdge<KnownInt32Use>(m_graph.varArgChild(node, 3));
+            fixEdge<KnownInt32Use>(m_graph.varArgChild(node, 4));
+            fixEdge<KnownCellUse>(m_graph.varArgChild(node, 5));
+            FALLTHROUGH;
+        }
+
         case GetByVal: {
             if (!node->prediction()) {
@@ -1063 +1070 @@
                 node->setArrayMode(ArrayMode(Array::Generic, node->arrayMode().action()));
                 break;
+
+
+            case Array::ForceExit: {
+                // Don't force OSR because we have only seen OwnStructureMode.
+                // FIXME: WE should have a better way to do this...
+                if (node->op() == EnumeratorGetByVal)
+                    node->setArrayMode(node->arrayMode().withType(Array::Generic));
+                break;
+            }
             default:
                 break;
@@ -1074 +1090 @@
             case Array::Double:
             case Array::Int32: {
-                std::optional<Array::Speculation> saneChainSpeculation;
-                if (arrayMode.isJSArrayWithOriginalStructure()) {
-                    // Check if InBoundsSaneChain will work on a per-type basis. Note that:
-                    //
-                    // 1) We don't want double arrays to sometimes return undefined, since
-                    // that would require a change to the return type and it would pessimise
-                    // things a lot. So, we'd only want to do that if we actually had
-                    // evidence that we could read from a hole. That's pretty annoying.
-                    // Likely the best way to handle that case is with an equivalent of
-                    // InBoundsSaneChain for OutOfBounds. For now we just detect when Undefined and
-                    // NaN are indistinguishable according to backwards propagation, and just
-                    // use InBoundsSaneChain in that case. This happens to catch a lot of cases.
-                    //
-                    // 2) We don't want int32 array loads to have to do a hole check just to
-                    // coerce to Undefined, since that would mean twice the checks. We want to
-                    // be able to say we always return Int32. FIXME: Maybe this should be profiling
-                    // based?
-                    //
-                    // This has two implications. First, we have to do more checks than we'd
-                    // like. It's unfortunate that we have to do the hole check. Second,
-                    // some accesses that hit a hole will now need to take the full-blown
-                    // out-of-bounds slow path. We can fix that with:
-                    // https://bugs.webkit.org/show_bug.cgi?id=144668
-                    
-                    switch (arrayMode.type()) {
-                    case Array::Int32:
-                        if (is64Bit() && arrayMode.speculation() == Array::OutOfBounds && !m_graph.hasExitSite(node->origin.semantic, NegativeIndex))
-                            saneChainSpeculation = Array::OutOfBoundsSaneChain;
-                        break;
-                    case Array::Contiguous:
-                        // This is happens to be entirely natural. We already would have
-                        // returned any JSValue, and now we'll return Undefined. We still do
-                        // the check but it doesn't require taking any kind of slow path.
-                        if (is64Bit() && arrayMode.speculation() == Array::OutOfBounds && !m_graph.hasExitSite(node->origin.semantic, NegativeIndex))
-                            saneChainSpeculation = Array::OutOfBoundsSaneChain;
-                        else if (arrayMode.speculation() == Array::InBounds)
-                            saneChainSpeculation = Array::InBoundsSaneChain;
-                        break;
-                        
-                    case Array::Double:
-                        if (!(node->flags() & NodeBytecodeUsesAsOther) && arrayMode.speculation() == Array::InBounds) {
-                            // Holes look like NaN already, so if the user doesn't care
-                            // about the difference between Undefined and NaN then we can
-                            // do this.
-                            saneChainSpeculation = Array::InBoundsSaneChain;
-                        } else if (is64Bit() && arrayMode.speculation() == Array::OutOfBounds && !m_graph.hasExitSite(node->origin.semantic, NegativeIndex))
-                            saneChainSpeculation = Array::OutOfBoundsSaneChain;
-                        break;
-                        
-                    default:
-                        break;
-                    }
-                }
-
-                if (saneChainSpeculation)
-                    setSaneChainIfPossible(node, *saneChainSpeculation);
-
+                setJSArraySaneChainIfPossible(node);
                 break;
             }
@@ -1179 +1139 @@
                 break;
             }
-            
+
+            // Don't set a non-JSValue result for EnumeratorGetByVal as the indexing mode doesn't tell us about the type of OwnStructure
+            // properties. This isn't an issue for IndexedMode only as BytecodeParsing will emit a normal GetByVal if we've profiled that.
+            if (node->op() == EnumeratorGetByVal)
+                break;
             switch (arrayMode.type()) {
             case Array::Double:
@@ -2276 +2240 @@
         }
 
-        case GetEnumerableLength: {
-            fixEdge<CellUse>(node->child1());
-            break;
-        }
-        case HasEnumerableProperty: {
-            fixEdge<CellUse>(node->child2());
-            break;
-        }
-        case HasEnumerableStructureProperty: {
-            fixEdge<StringUse>(node->child2());
-            fixEdge<KnownCellUse>(node->child3());
-            break;
-        }
-        case HasOwnStructureProperty:
-        case InStructureProperty: {
-            fixEdge<CellUse>(node->child1());
-            fixEdge<KnownStringUse>(node->child2());
-            fixEdge<KnownCellUse>(node->child3());
-            break;
-        }
-        case HasIndexedProperty:
-        case HasEnumerableIndexedProperty: {
+        case HasIndexedProperty: {
             node->setArrayMode(
                 node->arrayMode().refine(
@@ -2316 +2259 @@
             break;
         }
-        case GetDirectPname: {
-            Edge& base = m_graph.varArgChild(node, 0);
-            Edge& property = m_graph.varArgChild(node, 1);
-            Edge& index = m_graph.varArgChild(node, 2);
-            Edge& enumerator = m_graph.varArgChild(node, 3);
-            fixEdge<CellUse>(base);
-            fixEdge<KnownCellUse>(property);
-            fixEdge<Int32Use>(index);
-            fixEdge<KnownCellUse>(enumerator);
-            break;
-        }
+
         case GetPropertyEnumerator: {
             if (node->child1()->shouldSpeculateCell())
@@ -2332 +2265 @@
             break;
         }
-        case GetEnumeratorStructurePname: {
-            fixEdge<KnownCellUse>(node->child1());
-            fixEdge<Int32Use>(node->child2());
-            break;
-        }
-        case GetEnumeratorGenericPname: {
-            fixEdge<KnownCellUse>(node->child1());
-            fixEdge<Int32Use>(node->child2());
-            break;
-        }
-        case ToIndexString: {
-            fixEdge<Int32Use>(node->child1());
-            break;
-        }
+
+        case EnumeratorNextUpdateIndexAndMode: {
+            Edge& base = m_graph.varArgChild(node, 0);
+            Edge& index = m_graph.varArgChild(node, 1);
+            Edge& storageEdge = m_graph.varArgChild(node, node->storageChildIndex());
+
+            if (node->enumeratorMetadata() == JSPropertyNameEnumerator::IndexedMode) {
+                node->setArrayMode(
+                    node->arrayMode().refine(
+                        m_graph, node,
+                        m_graph.varArgChild(node, 0)->prediction(),
+                        m_graph.varArgChild(node, 1)->prediction()));
+
+                fixEdge<CellUse>(base);
+                blessArrayOperation(base, index, storageEdge);
+
+                ArrayMode arrayMode = node->arrayMode();
+                if (arrayMode.benefitsFromOriginalArray())
+                    setSaneChainIfPossible(node, arrayMode.speculation() == Array::InBounds ? Array::InBoundsSaneChain : Array::OutOfBoundsSaneChain);
+            }
+
+            if (node->enumeratorMetadata() == JSPropertyNameEnumerator::OwnStructureMode && !m_graph.hasExitSite(node->origin.semantic, BadCache))
+                fixEdge<CellUse>(base);
+
+            fixEdge<KnownInt32Use>(index);
+            fixEdge<KnownInt32Use>(m_graph.varArgChild(node, 2));
+            fixEdge<KnownCellUse>(m_graph.varArgChild(node, 3));
+            break;
+        }
+
+        case EnumeratorNextExtractIndex:
+        case EnumeratorNextExtractMode: {
+            break;
+        }
+
+        case EnumeratorNextUpdatePropertyName: {
+            fixEdge<KnownInt32Use>(node->child1());
+            fixEdge<KnownInt32Use>(node->child2());
+            fixEdge<KnownCellUse>(node->child3());
+            break;
+        }
+
+        case EnumeratorHasOwnProperty:
+        case EnumeratorInByVal: {
+            if (m_graph.varArgChild(node, 0)->shouldSpeculateCell())
+                fixEdge<CellUse>(m_graph.varArgChild(node, 0));
+            fixEdge<KnownInt32Use>(m_graph.varArgChild(node, 2));
+            fixEdge<KnownInt32Use>(m_graph.varArgChild(node, 3));
+            fixEdge<KnownCellUse>(m_graph.varArgChild(node, 4));
+            break;
+        }
+
         case ProfileType: {
             // We want to insert type checks based on the instructionTypeSet of the TypeLocation, not the globalTypeSet.
@@ -3691 +3662 @@
     }
 
+    void setJSArraySaneChainIfPossible(Node* node)
+    {
+        ArrayMode arrayMode = node->arrayMode();
+        std::optional<Array::Speculation> saneChainSpeculation;
+        if (arrayMode.isJSArrayWithOriginalStructure()) {
+            // Check if InBoundsSaneChain will work on a per-type basis. Note that:
+            //
+            // 1) We don't want double arrays to sometimes return undefined, since
+            // that would require a change to the return type and it would pessimise
+            // things a lot. So, we'd only want to do that if we actually had
+            // evidence that we could read from a hole. That's pretty annoying.
+            // Likely the best way to handle that case is with an equivalent of
+            // InBoundsSaneChain for OutOfBounds. For now we just detect when Undefined and
+            // NaN are indistinguishable according to backwards propagation, and just
+            // use InBoundsSaneChain in that case. This happens to catch a lot of cases.
+            //
+            // 2) We don't want int32 array loads to have to do a hole check just to
+            // coerce to Undefined, since that would mean twice the checks. We want to
+            // be able to say we always return Int32. FIXME: Maybe this should be profiling
+            // based?
+            //
+            // This has two implications. First, we have to do more checks than we'd
+            // like. It's unfortunate that we have to do the hole check. Second,
+            // some accesses that hit a hole will now need to take the full-blown
+            // out-of-bounds slow path. We can fix that with:
+            // https://bugs.webkit.org/show_bug.cgi?id=144668
+
+            switch (arrayMode.type()) {
+            case Array::Int32:
+                if (is64Bit() && arrayMode.speculation() == Array::OutOfBounds && !m_graph.hasExitSite(node->origin.semantic, NegativeIndex))
+                    saneChainSpeculation = Array::OutOfBoundsSaneChain;
+                break;
+            case Array::Contiguous:
+                // This is happens to be entirely natural. We already would have
+                // returned any JSValue, and now we'll return Undefined. We still do
+                // the check but it doesn't require taking any kind of slow path.
+                if (is64Bit() && arrayMode.speculation() == Array::OutOfBounds && !m_graph.hasExitSite(node->origin.semantic, NegativeIndex))
+                    saneChainSpeculation = Array::OutOfBoundsSaneChain;
+                else if (arrayMode.speculation() == Array::InBounds)
+                    saneChainSpeculation = Array::InBoundsSaneChain;
+                break;
+
+            case Array::Double:
+                if (!(node->flags() & NodeBytecodeUsesAsOther) && arrayMode.speculation() == Array::InBounds) {
+                    // Holes look like NaN already, so if the user doesn't care
+                    // about the difference between Undefined and NaN then we can
+                    // do this.
+                    saneChainSpeculation = Array::InBoundsSaneChain;
+                } else if (is64Bit() && arrayMode.speculation() == Array::OutOfBounds && !m_graph.hasExitSite(node->origin.semantic, NegativeIndex))
+                    saneChainSpeculation = Array::OutOfBoundsSaneChain;
+                break;
+
+            default:
+                break;
+            }
+        }
+
+        if (saneChainSpeculation)
+            setSaneChainIfPossible(node, *saneChainSpeculation);
+    }
+
     void setSaneChainIfPossible(Node* node, Array::Speculation speculation)
     {

trunk/Source/JavaScriptCore/dfg/DFGGraph.cpp

@@ -388 +388 @@
     if (node->hasPutByIdStatus())
         out.print(comma, *node->putByIdStatus());
+    if (node->hasEnumeratorMetadata())
+        out.print(comma, "enumeratorModes = ", node->enumeratorMetadata().toRaw());
     if (node->isJump())
         out.print(comma, "T:", *node->targetBlock());

trunk/Source/JavaScriptCore/dfg/DFGIntegerRangeOptimizationPhase.cpp

@@ -1345 +1345 @@
                 }
 
+                case EnumeratorGetByVal:
                 case GetByVal: {
                     if (node->arrayMode().type() != Array::Undecided)
@@ -1381 +1382 @@
     {
         switch (node->op()) {
+        // FIXME: Teach this about EnumeratorNextExtractIndex.
         case CheckInBounds: {
             setRelationship(Relationship::safeCreate(node->child1().node(), node->child2().node(), Relationship::LessThan));

trunk/Source/JavaScriptCore/dfg/DFGMayExit.cpp

@@ -112 +112 @@
     case FilterCheckPrivateBrandStatus:
     case FilterSetPrivateBrandStatus:
+    case EnumeratorNextExtractMode:
+    case EnumeratorNextExtractIndex:
         break;
 
+    case EnumeratorNextUpdatePropertyName:
     case StrCat:
     case Call:

trunk/Source/JavaScriptCore/dfg/DFGNode.h

@@ -53 +53 @@
 #include "GetByVariant.h"
 #include "JSCJSValue.h"
+#include "JSPropertyNameEnumerator.h"
 #include "Operands.h"
 #include "PrivateFieldPutKind.h"
@@ -1765 +1766 @@
         case ArithCeil:
         case ArithTrunc:
-        case GetDirectPname:
         case GetById:
         case GetByIdFlush:
@@ -1773 +1773 @@
         case GetPrototypeOf:
         case TryGetById:
+        case EnumeratorGetByVal:
         case GetByVal:
         case GetByValWithThis:
@@ -1942 +1943 @@
     }
 
+    bool hasStorageChild() const
+    {
+        switch (op()) {
+        case StringCharAt:
+        case StringCharCodeAt:
+        case StringCodePointAt:
+        case EnumeratorGetByVal:
+        case GetByVal:
+        case PutByValDirect:
+        case PutByVal:
+        case PutByValAlias:
+        case AtomicsAdd:
+        case AtomicsAnd:
+        case AtomicsCompareExchange:
+        case AtomicsExchange:
+        case AtomicsLoad:
+        case AtomicsOr:
+        case AtomicsStore:
+        case AtomicsSub:
+        case AtomicsXor:
+        case ArrayPush:
+        case ArrayPop:
+        case GetArrayLength:
+        case HasIndexedProperty:
+        case EnumeratorNextUpdateIndexAndMode:
+        case ArrayIndexOf:
+            return true;
+        default:
+            break;
+        }
+        return false;
+    }
+
+    unsigned storageChildIndex()
+    {
+        ASSERT(hasStorageChild());
+        switch (op()) {
+        case StringCharAt:
+        case StringCharCodeAt:
+        case StringCodePointAt:
+        case EnumeratorGetByVal:
+        case GetByVal:
+            return 2;
+        case PutByValDirect:
+        case PutByVal:
+        case PutByValAlias:
+            return 3;
+        case AtomicsAdd:
+        case AtomicsAnd:
+        case AtomicsCompareExchange:
+        case AtomicsExchange:
+        case AtomicsLoad:
+        case AtomicsOr:
+        case AtomicsStore:
+        case AtomicsSub:
+        case AtomicsXor:
+            return 2 + numExtraAtomicsArgs(op());
+        case ArrayPush:
+            return 0;
+
+        case ArrayPop:
+        case GetArrayLength:
+            return 2;
+
+        case HasIndexedProperty:
+            return 2;
+
+        case EnumeratorNextUpdateIndexAndMode:
+            return 4;
+
+        default:
+            // FIXME: Add ArrayIndexOf but it does non-trivial things to determine storage edge.
+            RELEASE_ASSERT_NOT_REACHED();
+        }
+    }
+
     bool hasUidOperand()
     {
@@ -2206 +2283 @@
         case PutByValAlias:
         case GetByVal:
+        case EnumeratorNextUpdateIndexAndMode:
+        case EnumeratorGetByVal:
+        case EnumeratorInByVal:
+        case EnumeratorHasOwnProperty:
         case StringCharAt:
         case StringCharCodeAt:
@@ -2217 +2298 @@
         case ArrayIndexOf:
         case HasIndexedProperty:
-        case HasEnumerableIndexedProperty:
         case AtomicsAdd:
         case AtomicsAnd:
@@ -3175 +3255 @@
     }
 
+    bool hasEnumeratorMetadata() const
+    {
+        switch (op()) {
+        case EnumeratorNextUpdateIndexAndMode:
+        case EnumeratorNextUpdatePropertyName:
+        case EnumeratorHasOwnProperty:
+        case EnumeratorInByVal:
+            return true;
+        default:
+            return false;
+        }
+    }
+
+    OptionSet<JSPropertyNameEnumerator::Mode> enumeratorMetadata()
+    {
+        ASSERT(hasEnumeratorMetadata());
+        return OptionSet<JSPropertyNameEnumerator::Mode>::fromRaw(m_opInfo2.as<unsigned>());
+    }
+
     void dumpChildren(PrintStream& out)
     {

trunk/Source/JavaScriptCore/dfg/DFGNodeType.h

@@ -502 +502 @@
     macro(FencedStoreBarrier, NodeMustGenerate) \
     \
-    /* For-in enumeration opcodes */\
-    macro(GetEnumerableLength, NodeMustGenerate | NodeResultJS) \
     /* Must generate because of Proxies on the prototype chain */ \
     macro(HasIndexedProperty, NodeMustGenerate | NodeResultBoolean | NodeHasVarArgs) \
-    macro(HasEnumerableIndexedProperty, NodeMustGenerate | NodeResultBoolean | NodeHasVarArgs) \
-    macro(HasEnumerableStructureProperty, NodeResultBoolean) \
-    macro(HasEnumerableProperty, NodeResultBoolean) \
-    macro(HasOwnStructureProperty, NodeResultBoolean | NodeMustGenerate) \
-    macro(InStructureProperty, NodeMustGenerate | NodeResultBoolean) \
-    macro(GetDirectPname, NodeMustGenerate | NodeHasVarArgs | NodeResultJS) \
+    /* For-in enumeration opcodes */\
     macro(GetPropertyEnumerator, NodeMustGenerate | NodeResultJS) \
-    macro(GetEnumeratorStructurePname, NodeMustGenerate | NodeResultJS) \
-    macro(GetEnumeratorGenericPname, NodeMustGenerate | NodeResultJS) \
-    macro(ToIndexString, NodeResultJS) \
+    macro(EnumeratorNextUpdateIndexAndMode, NodeResultJS | NodeHasVarArgs) \
+    macro(EnumeratorNextExtractMode, NodeResultInt32) \
+    macro(EnumeratorNextExtractIndex, NodeResultInt32) \
+    macro(EnumeratorNextUpdatePropertyName, NodeResultJS) \
+    macro(EnumeratorGetByVal, NodeResultJS | NodeHasVarArgs | NodeMustGenerate) \
+    macro(EnumeratorInByVal, NodeResultBoolean | NodeHasVarArgs | NodeMustGenerate) \
+    macro(EnumeratorHasOwnProperty, NodeResultBoolean | NodeHasVarArgs | NodeMustGenerate) \
     /* Nodes for JSMap and JSSet */ \
     macro(MapHash, NodeResultInt32) \

trunk/Source/JavaScriptCore/dfg/DFGOpInfo.h

@@ -30 +30 @@
 #include "HeapCell.h"
 #include "PrivateFieldPutKind.h"
+#include <wtf/OptionSet.h>
 #include <wtf/StdLibExtras.h>
 
@@ -51 +52 @@
     explicit OpInfo(ECMAMode ecmaMode) : m_value(ecmaMode.value()) { }
     explicit OpInfo(PrivateFieldPutKind putKind) : m_value(putKind.value()) { }
+    template<typename EnumType>
+    explicit OpInfo(OptionSet<EnumType> optionSet) : m_value(optionSet.toRaw()) { }
 
     template <typename T>

trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp

@@ -2400 +2400 @@
 }
 
-JSC_DEFINE_JIT_OPERATION(operationHasEnumerableProperty, EncodedJSValue, (JSGlobalObject* globalObject, EncodedJSValue encodedBaseValue, JSCell* property))
-{
-    VM& vm = globalObject->vm();
-    CallFrame* callFrame = DECLARE_CALL_FRAME(vm);
-    JITOperationPrologueCallFrameTracer tracer(vm, callFrame);
-    auto scope = DECLARE_THROW_SCOPE(vm);
-
-    JSValue baseValue = JSValue::decode(encodedBaseValue);
-    if (baseValue.isUndefinedOrNull())
-        return JSValue::encode(jsBoolean(false));
-
-    JSObject* base = baseValue.toObject(globalObject);
-    EXCEPTION_ASSERT(!scope.exception() || !base);
-    if (!base)
-        return JSValue::encode(JSValue());
-    auto propertyName = asString(property)->toIdentifier(globalObject);
-    RETURN_IF_EXCEPTION(scope, { });
-    RELEASE_AND_RETURN(scope, JSValue::encode(jsBoolean(base->hasEnumerableProperty(globalObject, propertyName))));
-}
-
-JSC_DEFINE_JIT_OPERATION(operationInStructureProperty, EncodedJSValue, (JSGlobalObject* globalObject, JSCell* base, JSString* property))
-{
-    VM& vm = globalObject->vm();
-    CallFrame* callFrame = DECLARE_CALL_FRAME(vm);
-    JITOperationPrologueCallFrameTracer tracer(vm, callFrame);
-
-    return JSValue::encode(jsBoolean(CommonSlowPaths::opInByVal(globalObject, base, property)));
-}
-
-JSC_DEFINE_JIT_OPERATION(operationHasOwnStructureProperty, EncodedJSValue, (JSGlobalObject* globalObject, JSCell* base, JSString* property))
-{
-    VM& vm = globalObject->vm();
-    CallFrame* callFrame = DECLARE_CALL_FRAME(vm);
-    JITOperationPrologueCallFrameTracer tracer(vm, callFrame);
-
-    auto scope = DECLARE_THROW_SCOPE(vm);
-    auto propertyName = property->toIdentifier(globalObject);
-    RETURN_IF_EXCEPTION(scope, { });
-    scope.release();
-    return JSValue::encode(jsBoolean(objectPrototypeHasOwnProperty(globalObject, base, propertyName)));
-}
-
 JSC_DEFINE_JIT_OPERATION(operationHasIndexedProperty, size_t, (JSGlobalObject* globalObject, JSCell* baseCell, int32_t subscript))
 {
@@ -2498 +2456 @@
 }
 
-JSC_DEFINE_JIT_OPERATION(operationToIndexString, JSCell*, (JSGlobalObject* globalObject, int32_t index))
-{
-    VM& vm = globalObject->vm();
-    CallFrame* callFrame = DECLARE_CALL_FRAME(vm);
-    JITOperationPrologueCallFrameTracer tracer(vm, callFrame);
-    return jsString(vm, Identifier::from(vm, index).string());
+JSC_DEFINE_JIT_OPERATION(operationEnumeratorNextUpdateIndexAndMode, EncodedJSValue, (JSGlobalObject* globalObject, EncodedJSValue baseValue, uint32_t index, int32_t modeNumber, JSPropertyNameEnumerator* enumerator))
+{
+    VM& vm = globalObject->vm();
+    CallFrame* callFrame = DECLARE_CALL_FRAME(vm);
+    JITOperationPrologueCallFrameTracer tracer(vm, callFrame);
+    auto scope = DECLARE_THROW_SCOPE(vm);
+
+    JSPropertyNameEnumerator::Mode mode = static_cast<JSPropertyNameEnumerator::Mode>(modeNumber);
+    if (JSValue::decode(baseValue).isUndefinedOrNull()) {
+        ASSERT(mode == JSPropertyNameEnumerator::InitMode);
+        ASSERT(!index);
+        ASSERT(!enumerator->sizeOfPropertyNames());
+        mode = JSPropertyNameEnumerator::GenericMode;
+    } else {
+        JSObject* base = JSValue::decode(baseValue).toObject(globalObject);
+        RETURN_IF_EXCEPTION(scope, { });
+
+        constexpr bool shouldAllocateIndexedNameString = false;
+        enumerator->computeNext(globalObject, base, index, mode, shouldAllocateIndexedNameString);
+        RETURN_IF_EXCEPTION(scope, { });
+    }
+
+#if USE(JSVALUE64)
+    JSValue result = bitwise_cast<JSValue>(static_cast<uint64_t>(mode) << 32 | index | JSValue::DoubleEncodeOffset);
+#else
+    JSValue result = JSValue(mode, index);
+#endif
+    ASSERT(result.isDouble());
+    return JSValue::encode(result);
+}
+
+JSC_DEFINE_JIT_OPERATION(operationEnumeratorNextUpdatePropertyName, EncodedJSValue, (JSGlobalObject* globalObject, uint32_t index, int32_t modeNumber, JSPropertyNameEnumerator* enumerator))
+{
+    VM& vm = globalObject->vm();
+    CallFrame* callFrame = DECLARE_CALL_FRAME(vm);
+    JITOperationPrologueCallFrameTracer tracer(vm, callFrame);
+
+    if (modeNumber == JSPropertyNameEnumerator::IndexedMode) {
+        if (index < enumerator->indexedLength())
+            return JSValue::encode(jsString(vm, Identifier::from(vm, index).string()));
+        return JSValue::encode(jsNull());
+    }
+
+    JSString* result = enumerator->propertyNameAtIndex(index);
+    if (!result)
+        return JSValue::encode(jsNull());
+
+    return JSValue::encode(result);
+}
+
+JSC_DEFINE_JIT_OPERATION(operationEnumeratorRecoverNameAndGetByVal, EncodedJSValue, (JSGlobalObject* globalObject, JSCell* base, uint32_t index, JSPropertyNameEnumerator* enumerator))
+{
+    VM& vm = globalObject->vm();
+    CallFrame* callFrame = DECLARE_CALL_FRAME(vm);
+    JITOperationPrologueCallFrameTracer tracer(vm, callFrame);
+    auto scope = DECLARE_THROW_SCOPE(vm);
+
+    JSString* string = enumerator->propertyNameAtIndex(index);
+    PropertyName propertyName = string->toIdentifier(globalObject);
+    // This should only really return for TerminationException since we know string is backed by a UUID.
+    RETURN_IF_EXCEPTION(scope, { });
+    JSObject* object = base->toObject(globalObject);
+    RETURN_IF_EXCEPTION(scope, { });
+
+    RELEASE_AND_RETURN(scope, JSValue::encode(object->get(globalObject, propertyName)));
+}
+
+JSC_DEFINE_JIT_OPERATION(operationEnumeratorInByVal, EncodedJSValue, (JSGlobalObject* globalObject, EncodedJSValue baseValue, EncodedJSValue propertyNameValue, uint32_t index, int32_t modeNumber))
+{
+    VM& vm = globalObject->vm();
+    CallFrame* callFrame = DECLARE_CALL_FRAME(vm);
+    JITOperationPrologueCallFrameTracer tracer(vm, callFrame);
+    auto scope = DECLARE_THROW_SCOPE(vm);
+
+    JSValue base = JSValue::decode(baseValue);
+    RETURN_IF_EXCEPTION(scope, { });
+    if (modeNumber == JSPropertyNameEnumerator::IndexedMode && base.isObject())
+        return JSValue::encode(jsBoolean(jsCast<JSObject*>(base)->hasProperty(globalObject, index)));
+
+    JSString* propertyName = jsSecureCast<JSString*>(vm, JSValue::decode(propertyNameValue));
+    RELEASE_AND_RETURN(scope, JSValue::encode(jsBoolean(CommonSlowPaths::opInByVal(globalObject, base, propertyName))));
+}
+
+JSC_DEFINE_JIT_OPERATION(operationEnumeratorHasOwnProperty, EncodedJSValue, (JSGlobalObject* globalObject, EncodedJSValue baseValue, EncodedJSValue propertyNameValue, uint32_t index, int32_t modeNumber))
+{
+    VM& vm = globalObject->vm();
+    CallFrame* callFrame = DECLARE_CALL_FRAME(vm);
+    JITOperationPrologueCallFrameTracer tracer(vm, callFrame);
+    auto scope = DECLARE_THROW_SCOPE(vm);
+
+    JSValue base = JSValue::decode(baseValue);
+    RETURN_IF_EXCEPTION(scope, { });
+    if (modeNumber == JSPropertyNameEnumerator::IndexedMode && base.isObject())
+        return JSValue::encode(jsBoolean(jsCast<JSObject*>(base)->hasOwnProperty(globalObject, index)));
+
+    JSString* propertyName = jsSecureCast<JSString*>(vm, JSValue::decode(propertyNameValue));
+    auto identifier = propertyName->toIdentifier(globalObject);
+    RETURN_IF_EXCEPTION(scope, { });
+    RELEASE_AND_RETURN(scope, JSValue::encode(jsBoolean(objectPrototypeHasOwnProperty(globalObject, base, identifier))));
 }
 

trunk/Source/JavaScriptCore/dfg/DFGOperations.h

@@ -37 +37 @@
 class JSBigInt;
 struct UnlinkedStringJumpTable;
+class JSPropertyNameEnumerator;
 
 namespace DFG {
@@ -105 +106 @@
 JSC_DECLARE_JIT_OPERATION(operationHasIndexedProperty, size_t, (JSGlobalObject*, JSCell*, int32_t));
 JSC_DECLARE_JIT_OPERATION(operationHasEnumerableIndexedProperty, size_t, (JSGlobalObject*, JSCell*, int32_t));
-JSC_DECLARE_JIT_OPERATION(operationHasEnumerableProperty, EncodedJSValue, (JSGlobalObject*, EncodedJSValue, JSCell*));
-JSC_DECLARE_JIT_OPERATION(operationHasOwnStructureProperty, EncodedJSValue, (JSGlobalObject*, JSCell*, JSString*));
-JSC_DECLARE_JIT_OPERATION(operationInStructureProperty, EncodedJSValue, (JSGlobalObject*, JSCell*, JSString*));
 JSC_DECLARE_JIT_OPERATION(operationGetPropertyEnumerator, JSCell*, (JSGlobalObject*, EncodedJSValue));
 JSC_DECLARE_JIT_OPERATION(operationGetPropertyEnumeratorCell, JSCell*, (JSGlobalObject*, JSCell*));
-JSC_DECLARE_JIT_OPERATION(operationToIndexString, JSCell*, (JSGlobalObject*, int32_t));
+JSC_DECLARE_JIT_OPERATION(operationEnumeratorNextUpdateIndexAndMode, EncodedJSValue, (JSGlobalObject*, EncodedJSValue, uint32_t, int32_t, JSPropertyNameEnumerator*));
+JSC_DECLARE_JIT_OPERATION(operationEnumeratorNextUpdatePropertyName, EncodedJSValue, (JSGlobalObject*, uint32_t, int32_t, JSPropertyNameEnumerator*));
+JSC_DECLARE_JIT_OPERATION(operationEnumeratorInByVal, EncodedJSValue, (JSGlobalObject*, EncodedJSValue, EncodedJSValue, uint32_t, int32_t));
+JSC_DECLARE_JIT_OPERATION(operationEnumeratorHasOwnProperty, EncodedJSValue, (JSGlobalObject*, EncodedJSValue, EncodedJSValue, uint32_t, int32_t));
+JSC_DECLARE_JIT_OPERATION(operationEnumeratorRecoverNameAndGetByVal, EncodedJSValue, (JSGlobalObject*, JSCell*, uint32_t, JSPropertyNameEnumerator*));
+
 JSC_DECLARE_JIT_OPERATION(operationNewRegexpWithLastIndex, JSCell*, (JSGlobalObject*, JSCell*, EncodedJSValue));
 JSC_DECLARE_JIT_OPERATION(operationNewArray, char*, (JSGlobalObject*, Structure*, void*, size_t));

trunk/Source/JavaScriptCore/dfg/DFGPredictionPropagationPhase.cpp

@@ -474 +474 @@
             if (!child1->prediction())
                 break;
-            
+
             Edge child2 = m_graph.child(node, 1);
             ArrayMode arrayMode = node->arrayMode().refine(
@@ -481 +481 @@
                 child2->prediction(),
                 SpecNone);
-            
+
             switch (arrayMode.type()) {
             case Array::Int32:
@@ -863 +863 @@
         }
 
+        case EnumeratorGetByVal:
         case ArrayPop:
         case ArrayPush:
@@ -883 +884 @@
         case GetPrivateNameById:
         case MultiGetByOffset:
-        case GetDirectPname:
         case Call:
         case DirectCall:
@@ -1206 +1206 @@
             break;
 
-        case GetEnumerableLength: {
+        case EnumeratorNextExtractMode:
+        case EnumeratorNextExtractIndex: {
             setPrediction(SpecInt32Only);
             break;
         }
+
+        case EnumeratorInByVal:
+        case EnumeratorHasOwnProperty:
         case InByVal:
         case InById:
@@ -1215 +1219 @@
         case HasPrivateBrand:
         case HasOwnProperty:
-        case HasOwnStructureProperty:
-        case InStructureProperty:
-        case HasIndexedProperty:
-        case HasEnumerableIndexedProperty:
-        case HasEnumerableStructureProperty:
-        case HasEnumerableProperty: {
+        case HasIndexedProperty: {
             setPrediction(SpecBoolean);
             break;
@@ -1228 +1227 @@
             break;
         }
-        case GetEnumeratorStructurePname: {
-            setPrediction(SpecCell | SpecOther);
-            break;
-        }
-        case GetEnumeratorGenericPname: {
-            setPrediction(SpecCell | SpecOther);
-            break;
-        }
-        case ToIndexString: {
-            setPrediction(SpecString);
-            break;
-        }
+
+        case EnumeratorNextUpdateIndexAndMode: {
+            setPrediction(SpecFullNumber);
+            break;
+        }
+
+        case EnumeratorNextUpdatePropertyName: {
+            setPrediction(SpecString | SpecOther);
+            break;
+        }
+
         case ParseInt: {
             // We expect this node to almost always produce an int32. However,

trunk/Source/JavaScriptCore/dfg/DFGSSALoweringPhase.cpp

@@ -83 +83 @@
 
         case HasIndexedProperty:
-        case HasEnumerableIndexedProperty:
             lowerBoundsCheck(m_graph.child(m_node, 0), m_graph.child(m_node, 1), m_graph.child(m_node, 2));
             break;
 
+        case EnumeratorGetByVal:
         case GetByVal: {
             lowerBoundsCheck(m_graph.varArgChild(m_node, 0), m_graph.varArgChild(m_node, 1), m_graph.varArgChild(m_node, 2));

trunk/Source/JavaScriptCore/dfg/DFGSafeToExecute.h

@@ -297 +297 @@
     case FiatInt52:
     case HasIndexedProperty:
-    case HasEnumerableIndexedProperty:
-    case GetEnumeratorStructurePname:
-    case GetEnumeratorGenericPname:
-    case ToIndexString:
     case CheckStructureImmediate:
     case GetMyArgumentByVal:
@@ -373 +369 @@
         return false;
 
+    case EnumeratorGetByVal:
     case GetByVal:
     case GetIndexedPropertyStorage:
@@ -497 +494 @@
     }
 
+    case EnumeratorNextUpdateIndexAndMode:
+    // These technically don't have effects but they'll only ever follow a EnumeratorNextUpdateIndexAndMode so we might as well return false.
+    case EnumeratorNextExtractMode:
+    case EnumeratorNextExtractIndex:
+    case EnumeratorNextUpdatePropertyName:
     case ToThis:
     case CreateThis:
@@ -596 +598 @@
     case InByVal:
     case InById:
+    case EnumeratorInByVal:
+    case EnumeratorHasOwnProperty:
     case HasPrivateName:
     case HasPrivateBrand:
@@ -640 +644 @@
     case MultiPutByOffset:
     case MultiDeleteByOffset:
-    case GetEnumerableLength:
-    case HasEnumerableStructureProperty:
-    case HasEnumerableProperty:
-    case HasOwnStructureProperty:
-    case InStructureProperty:
-    case GetDirectPname:
     case GetPropertyEnumerator:
     case PhantomNewObject:

trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp

@@ -1638 +1638 @@
 }
 
+JSValueRegsTemporary::JSValueRegsTemporary(SpeculativeJIT* jit, GPRReg specificPayload)
 #if USE(JSVALUE64)
-template<typename T>
-JSValueRegsTemporary::JSValueRegsTemporary(SpeculativeJIT* jit, ReuseTag, T& operand, WhichValueWord)
-    : m_gpr(jit, Reuse, operand)
-{
-}
+    : m_gpr(jit, specificPayload)
 #else
-template<typename T>
-JSValueRegsTemporary::JSValueRegsTemporary(SpeculativeJIT* jit, ReuseTag, T& operand, WhichValueWord resultWord)
-{
-    if (resultWord == PayloadWord) {
-        m_payloadGPR = GPRTemporary(jit, Reuse, operand);
-        m_tagGPR = GPRTemporary(jit);
-    } else {
-        m_payloadGPR = GPRTemporary(jit);
-        m_tagGPR = GPRTemporary(jit, Reuse, operand);
-    }
-}
+    : m_payloadGPR(jit, specificPayload)
+    , m_tagGPR(jit)
 #endif
+{
+}
 
 #if USE(JSVALUE64)
@@ -2598 +2588 @@
 }
 
-void SpeculativeJIT::compileGetByValOnString(Node* node)
+void SpeculativeJIT::compileGetByValOnString(Node* node, const ScopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>& prefix)
 {
     SpeculateCellOperand base(this, m_graph.child(node, 0));
@@ -2607 +2597 @@
     GPRReg storageReg = storage.gpr();
 
-    GPRTemporary scratch(this);
-    GPRReg scratchReg = scratch.gpr();
-#if USE(JSVALUE32_64)
-    GPRTemporary resultTag;
-    GPRReg resultTagReg = InvalidGPRReg;
-    if (node->arrayMode().isOutOfBounds()) {
-        GPRTemporary realResultTag(this);
-        resultTag.adopt(realResultTag);
-        resultTagReg = resultTag.gpr();
-    }
-#endif
+    JSValueRegs resultRegs;
+    DataFormat format;
+    std::tie(resultRegs, format) = prefix(node->arrayMode().isOutOfBounds() ? DataFormatJS : DataFormatCell);
+    GPRReg scratchReg = resultRegs.payloadGPR();
 
     // unsigned comparison so we can filter out negative indices and indices that are too large
@@ -2653 +2636 @@
 
     if (node->arrayMode().isOutOfBounds()) {
+        ASSERT(format == DataFormatJS);
 #if USE(JSVALUE32_64)
-        m_jit.move(TrustedImm32(JSValue::CellTag), resultTagReg);
+        m_jit.move(TrustedImm32(JSValue::CellTag), resultRegs.tagGPR());
 #endif
 
@@ -2672 +2656 @@
             m_jit.graph().registerAndWatchStructureTransition(objectPrototypeStructure);
 
-#if USE(JSVALUE64)
             addSlowPathGenerator(makeUnique<SaneStringGetByValSlowPathGenerator>(
-                outOfBounds, this, JSValueRegs(scratchReg), TrustedImmPtr::weakPointer(m_graph, globalObject), baseReg, propertyReg));
-#else
-            addSlowPathGenerator(makeUnique<SaneStringGetByValSlowPathGenerator>(
-                outOfBounds, this, JSValueRegs(resultTagReg, scratchReg), TrustedImmPtr::weakPointer(m_graph, globalObject), baseReg, propertyReg));
-#endif
+                outOfBounds, this, resultRegs, TrustedImmPtr::weakPointer(m_graph, globalObject), baseReg, propertyReg));
         } else {
-#if USE(JSVALUE64)
             addSlowPathGenerator(
                 slowPathCall(
                     outOfBounds, this, operationGetByValStringInt,
-                    scratchReg, TrustedImmPtr::weakPointer(m_graph, globalObject), baseReg, propertyReg));
-#else
-            addSlowPathGenerator(
-                slowPathCall(
-                    outOfBounds, this, operationGetByValStringInt,
-                    JSValueRegs(resultTagReg, scratchReg), TrustedImmPtr::weakPointer(m_graph, globalObject), baseReg, propertyReg));
-#endif
-        }
-        
-#if USE(JSVALUE64)
-        jsValueResult(scratchReg, m_currentNode);
-#else
-        jsValueResult(resultTagReg, scratchReg, m_currentNode);
-#endif
-    } else
-        cellResult(scratchReg, m_currentNode);
+                    resultRegs, TrustedImmPtr::weakPointer(m_graph, globalObject), baseReg, propertyReg));
+        }
+        
+        jsValueResult(resultRegs, m_currentNode);
+    } else {
+        if (format == DataFormatJS)
+            jsValueResult(resultRegs, m_currentNode);
+        else {
+            ASSERT(format == DataFormatCell);
+            cellResult(resultRegs.payloadGPR(), m_currentNode);
+        }
+    }
 }
 
@@ -3352 +3326 @@
 }
 
-void SpeculativeJIT::setIntTypedArrayLoadResult(Node* node, GPRReg resultReg, TypedArrayType type, bool canSpeculate)
-{
-    if (elementSize(type) < 4 || isSigned(type)) {
+void SpeculativeJIT::setIntTypedArrayLoadResult(Node* node, JSValueRegs resultRegs, TypedArrayType type, bool canSpeculate, bool shouldBox)
+{
+    bool isUInt32 = elementSize(type) == 4 && !isSigned(type);
+    GPRReg resultReg = resultRegs.payloadGPR();
+
+    if (shouldBox) {
+        if (isUInt32) {
+            FPRTemporary fresult(this);
+            m_jit.convertInt32ToDouble(resultReg, fresult.fpr());
+            JITCompiler::Jump positive = m_jit.branch32(MacroAssembler::GreaterThanOrEqual, resultReg, TrustedImm32(0));
+            m_jit.addDouble(JITCompiler::AbsoluteAddress(&AssemblyHelpers::twoToThe32), fresult.fpr());
+            positive.link(&m_jit);
+            m_jit.boxDouble(fresult.fpr(), resultRegs);
+        } else
+            m_jit.boxInt32(resultRegs.payloadGPR(), resultRegs);
+        jsValueResult(resultRegs, node);
+        return;
+    }
+
+    if (!isUInt32) {
+        ASSERT(elementSize(type) < 4 || isSigned(type));
         strictInt32Result(resultReg, node);
         return;
     }
-    
-    ASSERT(elementSize(type) == 4 && !isSigned(type));
+
     if (node->shouldSpeculateInt32() && canSpeculate) {
         speculationCheck(Overflow, JSValueRegs(), nullptr, m_jit.branch32(MacroAssembler::LessThan, resultReg, TrustedImm32(0)));
@@ -3383 +3374 @@
 }
 
-void SpeculativeJIT::compileGetByValOnIntTypedArray(Node* node, TypedArrayType type)
+void SpeculativeJIT::compileGetByValOnIntTypedArray(Node* node, TypedArrayType type, const ScopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>& prefix)
 {
     ASSERT(isInt(type));
@@ -3395 +3386 @@
     GPRReg storageReg = storage.gpr();
 
-    GPRTemporary result(this);
-    GPRReg resultReg = result.gpr();
+    JSValueRegs resultRegs;
+    DataFormat format;
+    std::tie(resultRegs, format) = prefix(DataFormatInt32);
+    bool shouldBox = format == DataFormatJS;
 
     emitTypedArrayBoundsCheck(node, baseReg, propertyReg);
-    loadFromIntTypedArray(storageReg, propertyReg, resultReg, type);
+    loadFromIntTypedArray(storageReg, propertyReg, resultRegs.payloadGPR(), type);
     constexpr bool canSpeculate = true;
-    setIntTypedArrayLoadResult(node, resultReg, type, canSpeculate);
+    setIntTypedArrayLoadResult(node, resultRegs, type, canSpeculate, shouldBox);
 }
 
@@ -3639 +3632 @@
 }
 
-void SpeculativeJIT::compileGetByValOnFloatTypedArray(Node* node, TypedArrayType type)
+void SpeculativeJIT::compileGetByValOnFloatTypedArray(Node* node, TypedArrayType type, const ScopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>& prefix)
 {
     ASSERT(isFloat(type));
@@ -3650 +3643 @@
     GPRReg propertyReg = property.gpr();
     GPRReg storageReg = storage.gpr();
+
+    JSValueRegs resultRegs;
+    DataFormat format;
+    std::tie(resultRegs, format) = prefix(DataFormatDouble);
 
     FPRTemporary result(this);
@@ -3667 +3664 @@
     }
     
-    doubleResult(resultReg, node);
+    if (format == DataFormatJS) {
+        m_jit.boxDouble(resultReg, resultRegs);
+        jsValueResult(resultRegs, node);
+    } else {
+        ASSERT(format == DataFormatDouble);
+        doubleResult(resultReg, node);
+    }
 }
 
@@ -3706 +3709 @@
 }
 
-void SpeculativeJIT::compileGetByValForObjectWithString(Node* node)
+void SpeculativeJIT::compileGetByValForObjectWithString(Node* node, const ScopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>& prefix)
 {
     SpeculateCellOperand arg1(this, m_graph.varArgChild(node, 0));
@@ -3714 +3717 @@
     GPRReg arg2GPR = arg2.gpr();
 
+    JSValueRegs resultRegs;
+    std::tie(resultRegs, std::ignore) = prefix(DataFormatJS);
+
     speculateObject(m_graph.varArgChild(node, 0), arg1GPR);
     speculateString(m_graph.varArgChild(node, 1), arg2GPR);
 
     flushRegisters();
-    JSValueRegsFlushedCallResult result(this);
-    JSValueRegs resultRegs = result.regs();
     callOperation(operationGetByValObjectString, resultRegs, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), arg1GPR, arg2GPR);
     m_jit.exceptionCheck();
@@ -3726 +3730 @@
 }
 
-void SpeculativeJIT::compileGetByValForObjectWithSymbol(Node* node)
+void SpeculativeJIT::compileGetByValForObjectWithSymbol(Node* node, const ScopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>& prefix)
 {
     SpeculateCellOperand arg1(this, m_graph.varArgChild(node, 0));
@@ -3734 +3738 @@
     GPRReg arg2GPR = arg2.gpr();
 
+    JSValueRegs resultRegs;
+    std::tie(resultRegs, std::ignore) = prefix(DataFormatJS);
+
     speculateObject(m_graph.varArgChild(node, 0), arg1GPR);
     speculateSymbol(m_graph.varArgChild(node, 1), arg2GPR);
 
     flushRegisters();
-    JSValueRegsFlushedCallResult result(this);
-    JSValueRegs resultRegs = result.regs();
     callOperation(operationGetByValObjectSymbol, resultRegs, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), arg1GPR, arg2GPR);
     m_jit.exceptionCheck();
@@ -8041 +8046 @@
 }
 
-void SpeculativeJIT::compileGetByValOnDirectArguments(Node* node)
+void SpeculativeJIT::compileGetByValOnDirectArguments(Node* node, const ScopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>& prefix)
 {
     SpeculateCellOperand base(this, m_graph.varArgChild(node, 0));
     SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
-    JSValueRegsTemporary result(this);
-    GPRTemporary scratch(this);
     
     GPRReg baseReg = base.gpr();
     GPRReg propertyReg = property.gpr();
-    JSValueRegs resultRegs = result.regs();
-    GPRReg scratchReg = scratch.gpr();
+
+    JSValueRegs resultRegs;
+    std::tie(resultRegs, std::ignore) = prefix(DataFormatJS);
+    GPRReg scratchReg = resultRegs.payloadGPR();
     
     if (!m_compileOkay)
@@ -8082 +8087 @@
 }
 
-void SpeculativeJIT::compileGetByValOnScopedArguments(Node* node)
+void SpeculativeJIT::compileGetByValOnScopedArguments(Node* node, const ScopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>& prefix)
 {
     SpeculateCellOperand base(this, m_graph.varArgChild(node, 0));
     SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
-    JSValueRegsTemporary result(this);
     GPRTemporary scratch(this);
     GPRTemporary scratch2(this);
@@ -8092 +8096 @@
     GPRReg baseReg = base.gpr();
     GPRReg propertyReg = property.gpr();
-    JSValueRegs resultRegs = result.regs();
     GPRReg scratchReg = scratch.gpr();
     GPRReg scratch2Reg = scratch2.gpr();
@@ -8098 +8101 @@
     if (!m_compileOkay)
         return;
+
+    JSValueRegs resultRegs;
+    std::tie(resultRegs, std::ignore) = prefix(DataFormatJS);
     
     m_jit.loadPtr(
@@ -13445 +13451 @@
 }
 
-void SpeculativeJIT::compileGetEnumerableLength(Node* node)
-{
-    SpeculateCellOperand enumerator(this, node->child1());
-    GPRTemporary result(this, Reuse, enumerator);
+void SpeculativeJIT::compileEnumeratorNextUpdateIndexAndMode(Node* node)
+{
+    Edge baseEdge = m_graph.varArgChild(node, 0);
+    SpeculateStrictInt32Operand index(this, m_graph.varArgChild(node, 1));
+    SpeculateStrictInt32Operand mode(this, m_graph.varArgChild(node, 2));
+    SpeculateCellOperand enumerator(this, m_graph.varArgChild(node, 3));
+
+    GPRReg indexGPR = index.gpr();
+    GPRReg modeGPR = mode.gpr();
     GPRReg enumeratorGPR = enumerator.gpr();
-    GPRReg resultGPR = result.gpr();
-
-    m_jit.load32(MacroAssembler::Address(enumeratorGPR, JSPropertyNameEnumerator::indexedLengthOffset()), resultGPR);
-    strictInt32Result(resultGPR, node);
-}
-
-void SpeculativeJIT::compileHasEnumerableProperty(Node* node)
-{
-    JSValueOperand base(this, node->child1());
-    SpeculateCellOperand property(this, node->child2());
-
-    JSValueRegs baseRegs = base.jsValueRegs();
-    GPRReg propertyGPR = property.gpr();
+
+    if (node->enumeratorMetadata() == JSPropertyNameEnumerator::IndexedMode) {
+        JSValueRegsTemporary result(this);
+        GPRTemporary scratch(this, Reuse, index);
+        JSValueRegs resultRegs = result.regs();
+
+        speculationCheck(BadCache, JSValueSource(), node, m_jit.branch32(MacroAssembler::NotEqual, MacroAssembler::Address(enumeratorGPR, JSPropertyNameEnumerator::endGenericPropertyIndexOffset()), TrustedImm32(0)));
+
+        MacroAssembler::Label incrementLoop;
+        MacroAssembler::Jump done;
+        compileHasIndexedProperty(node, operationHasEnumerableIndexedProperty, scopedLambda<std::tuple<GPRReg, GPRReg>()>([&] {
+
+            m_jit.move(indexGPR, scratch.gpr());
+            MacroAssembler::Jump initMode = m_jit.branchTest32(MacroAssembler::Zero, modeGPR);
+
+            incrementLoop = m_jit.label();
+            m_jit.add32(TrustedImm32(1), scratch.gpr());
+
+            initMode.link(&m_jit);
+            done = m_jit.branch32(MacroAssembler::AboveOrEqual, scratch.gpr(), MacroAssembler::Address(enumeratorGPR, JSPropertyNameEnumerator::indexedLengthOffset()));
+            return std::make_pair(scratch.gpr(), resultRegs.payloadGPR());
+        }));
+        m_jit.branchTest32(MacroAssembler::Zero, resultRegs.payloadGPR()).linkTo(incrementLoop, &m_jit);
+
+        done.link(&m_jit);
+
+#if USE(JSVALUE64)
+        m_jit.move(TrustedImm64(JSValue::DoubleEncodeOffset | static_cast<uint64_t>(JSPropertyNameEnumerator::IndexedMode) << 32), resultRegs.payloadGPR());
+        m_jit.or64(scratch.gpr(), resultRegs.payloadGPR());
+#else
+        m_jit.move(TrustedImm32(JSPropertyNameEnumerator::IndexedMode), resultRegs.tagGPR());
+#endif
+
+        jsValueResult(resultRegs, node);
+        return;
+    }
+
+    if (node->enumeratorMetadata() == JSPropertyNameEnumerator::OwnStructureMode && baseEdge.useKind() == CellUse) {
+        SpeculateCellOperand base(this, baseEdge);
+        JSValueRegsTemporary result(this);
+        GPRReg baseGPR = base.gpr();
+        JSValueRegs resultRegs = result.regs();
+
+        // Has the same structure as the enumerator.
+        m_jit.load32(MacroAssembler::Address(baseGPR, JSCell::structureIDOffset()), resultRegs.payloadGPR());
+        speculationCheck(BadCache, JSValueSource(), node, m_jit.branch32(MacroAssembler::NotEqual, resultRegs.payloadGPR(), MacroAssembler::Address(enumeratorGPR, JSPropertyNameEnumerator::cachedStructureIDOffset())));
+
+        m_jit.load32(MacroAssembler::Address(enumeratorGPR, JSPropertyNameEnumerator::modeSetOffset()), resultRegs.payloadGPR());
+        speculationCheck(BadCache, JSValueSource(), node, m_jit.branch32(MacroAssembler::NotEqual, TrustedImm32(JSPropertyNameEnumerator::OwnStructureMode), resultRegs.payloadGPR()));
+
+        m_jit.move(indexGPR, resultRegs.payloadGPR());
+        MacroAssembler::Jump initMode = m_jit.branchTest32(MacroAssembler::Zero, modeGPR);
+
+        m_jit.add32(MacroAssembler::TrustedImm32(1), resultRegs.payloadGPR());
+
+        initMode.link(&m_jit);
+#if USE(JSVALUE64)
+        m_jit.or64(TrustedImm64(JSValue::DoubleEncodeOffset | static_cast<uint64_t>(JSPropertyNameEnumerator::OwnStructureMode) << 32), resultRegs.payloadGPR());
+#else
+        m_jit.move(TrustedImm32(JSPropertyNameEnumerator::OwnStructureMode), resultRegs.tagGPR());
+#endif
+
+        jsValueResult(resultRegs, node);
+        return;
+    }
+
+    JSValueOperand base(this, baseEdge);
+    JSValueRegs baseRegs = base.regs();
 
     flushRegisters();
     JSValueRegsFlushedCallResult result(this);
     JSValueRegs resultRegs = result.regs();
-    callOperation(operationHasEnumerableProperty, resultRegs, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseRegs, propertyGPR);
+    callOperation(operationEnumeratorNextUpdateIndexAndMode, resultRegs, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseRegs, indexGPR, modeGPR, enumeratorGPR);
     m_jit.exceptionCheck();
-    blessedBooleanResult(resultRegs.payloadGPR(), node);
-}
-
-void SpeculativeJIT::compileToIndexString(Node* node)
-{
-    SpeculateInt32Operand index(this, node->child1());
-    GPRReg indexGPR = index.gpr();
-
-    flushRegisters();
-    GPRFlushedCallResult result(this);
+    jsValueResult(resultRegs, node);
+}
+
+void SpeculativeJIT::compileEnumeratorNextExtractIndex(Node* node)
+{
+    JSValueOperand updatedPair(this, node->child1());
+    JSValueRegs pairRegs = updatedPair.jsValueRegs();
+    GPRReg payloadGPR = pairRegs.payloadGPR();
+
+    GPRTemporary result(this);
     GPRReg resultGPR = result.gpr();
-    callOperation(operationToIndexString, resultGPR, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), indexGPR);
-    m_jit.exceptionCheck();
-    cellResult(resultGPR, node);
+
+    m_jit.and32(TrustedImm32(std::numeric_limits<uint32_t>::max()), payloadGPR, resultGPR);
+
+    strictInt32Result(resultGPR, node);
+}
+
+void SpeculativeJIT::compileEnumeratorNextExtractMode(Node* node)
+{
+    JSValueOperand updatedPair(this, node->child1());
+    JSValueRegs pairRegs = updatedPair.jsValueRegs();
+    GPRReg pairGPR = is64Bit() ? pairRegs.payloadGPR() : pairRegs.tagGPR();
+
+    GPRTemporary result(this);
+    GPRReg resultGPR = result.gpr();
+
+#if CPU(ARM64) && USE(JSVALUE64)
+    m_jit.extractUnsignedBitfield64(pairGPR, TrustedImm32(32), TrustedImm32(WTF::fastLog2(static_cast<unsigned>(JSPropertyNameEnumerator::enumerationModeMask + 1))), resultGPR);
+#else
+#if USE(JSVALUE64)
+    m_jit.rshift64(pairGPR, TrustedImm32(32), resultGPR);
+#else
+    m_jit.move(pairGPR, resultGPR);
+#endif
+    m_jit.and32(TrustedImm32(JSPropertyNameEnumerator::enumerationModeMask), resultGPR);
+#endif
+
+    strictInt32Result(resultGPR, node);
+}
+
+void SpeculativeJIT::compileEnumeratorNextUpdatePropertyName(Node* node)
+{
+    SpeculateStrictInt32Operand indexOperand(this, node->child1());
+    SpeculateStrictInt32Operand modeOperand(this, node->child2());
+    SpeculateCellOperand enumeratorOperand(this, node->child3());
+    JSValueRegsTemporary resultTemp(this);
+
+    GPRReg index = indexOperand.gpr();
+    GPRReg mode = modeOperand.gpr();
+    GPRReg enumerator = enumeratorOperand.gpr();
+    JSValueRegs resultRegs = resultTemp.regs();
+
+    OptionSet seenModes = node->enumeratorMetadata();
+
+    MacroAssembler::JumpList doneCases;
+    MacroAssembler::Jump operationCall;
+
+    bool needsOperation = seenModes.contains(JSPropertyNameEnumerator::IndexedMode);
+
+    // Make sure we flush on all code paths if we could call the operation.
+    if (needsOperation)
+        flushRegisters();
+
+    if (seenModes.containsAny({ JSPropertyNameEnumerator::OwnStructureMode, JSPropertyNameEnumerator::GenericMode })) {
+
+        if (needsOperation)
+            operationCall = m_jit.branchTest32(MacroAssembler::NonZero, mode, TrustedImm32(JSPropertyNameEnumerator::IndexedMode));
+
+        auto outOfBounds = m_jit.branch32(MacroAssembler::AboveOrEqual, index, MacroAssembler::Address(enumerator, JSPropertyNameEnumerator::endGenericPropertyIndexOffset()));
+
+        m_jit.loadPtr(MacroAssembler::Address(enumerator, JSPropertyNameEnumerator::cachedPropertyNamesVectorOffset()), resultRegs.payloadGPR());
+        m_jit.loadPtr(MacroAssembler::BaseIndex(resultRegs.payloadGPR(), index, MacroAssembler::ScalePtr), resultRegs.payloadGPR());
+#if USE(JSVALUE32_64)
+        m_jit.move(TrustedImm32(JSValue::CellTag), resultRegs.tagGPR());
+#endif
+        doneCases.append(m_jit.jump());
+
+        outOfBounds.link(&m_jit);
+        m_jit.moveTrustedValue(jsNull(), resultRegs);
+    }
+
+    if (needsOperation) {
+        if (operationCall.isSet()) {
+            doneCases.append(m_jit.jump());
+            operationCall.link(&m_jit);
+        }
+        callOperation(operationEnumeratorNextUpdatePropertyName, resultRegs, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), index, mode, enumerator);
+        m_jit.exceptionCheck();
+    }
+
+    doneCases.link(&m_jit);
+    jsValueResult(resultRegs, node);
+}
+
+void SpeculativeJIT::compileEnumeratorGetByVal(Node* node)
+{
+    Edge baseEdge = m_graph.varArgChild(node, 0);
+    auto generate = [&] (GPRReg baseCellGPR) {
+        MacroAssembler::JumpList doneCases;
+        JSValueRegsTemporary result;
+        JSValueRegs resultRegs;
+        GPRReg indexGPR;
+        GPRReg enumeratorGPR;
+        MacroAssembler::Jump badStructureSlowPath;
+
+        compileGetByVal(node, scopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat)>([&] (DataFormat) {
+            Edge storageEdge = m_graph.varArgChild(node, 2);
+            StorageOperand storage;
+            if (storageEdge)
+                storage.emplace(this, storageEdge);
+            SpeculateStrictInt32Operand index(this, m_graph.varArgChild(node, 3));
+            SpeculateStrictInt32Operand mode(this, m_graph.varArgChild(node, 4));
+            SpeculateCellOperand enumerator(this, m_graph.varArgChild(node, 5));
+
+            GPRReg modeGPR = mode.gpr();
+            indexGPR = index.gpr();
+            enumeratorGPR = enumerator.gpr();
+
+            result = JSValueRegsTemporary(this);
+            resultRegs = result.regs();
+            GPRReg scratchGPR = resultRegs.payloadGPR();
+
+            bool haveStorage = !!storageEdge;
+            GPRTemporary storageTemporary;
+            GPRReg storageGPR;
+            if (!haveStorage) {
+                storageTemporary = GPRTemporary(this, Reuse, enumerator);
+                storageGPR = storageTemporary.gpr();
+            } else
+                storageGPR = storage.gpr();
+
+            MacroAssembler::JumpList notFastNamedCases;
+
+            // FIXME: We shouldn't generate this code if we know base is not an object.
+            notFastNamedCases.append(m_jit.branchTest32(MacroAssembler::NonZero, modeGPR, TrustedImm32(JSPropertyNameEnumerator::IndexedMode | JSPropertyNameEnumerator::GenericMode)));
+            {
+                if (!m_state.forNode(baseEdge).isType(SpecCell))
+                    notFastNamedCases.append(m_jit.branchIfNotCell(baseCellGPR));
+
+                // Check the structure
+                // FIXME: If we know there's only one structure for base we can just embed it here.
+                m_jit.load32(MacroAssembler::Address(baseCellGPR, JSCell::structureIDOffset()), scratchGPR);
+
+                auto badStructure = m_jit.branch32(
+                    MacroAssembler::NotEqual,
+                    scratchGPR,
+                    MacroAssembler::Address(
+                        enumeratorGPR, JSPropertyNameEnumerator::cachedStructureIDOffset()));
+
+                // FIXME: Maybe we should have a better way to represent Indexed+Named?
+                if (m_graph.varArgChild(node, 1).node() == m_graph.varArgChild(node, 3).node())
+                    badStructureSlowPath = badStructure;
+                else
+                    notFastNamedCases.append(badStructure);
+
+                // Compute the offset
+                // If index is less than the enumerator's cached inline storage, then it's an inline access
+                MacroAssembler::Jump outOfLineAccess = m_jit.branch32(MacroAssembler::AboveOrEqual,
+                    indexGPR, MacroAssembler::Address(enumeratorGPR, JSPropertyNameEnumerator::cachedInlineCapacityOffset()));
+
+                m_jit.loadValue(MacroAssembler::BaseIndex(baseCellGPR, indexGPR, MacroAssembler::TimesEight, JSObject::offsetOfInlineStorage()), resultRegs);
+
+                doneCases.append(m_jit.jump());
+
+                // Otherwise it's out of line
+                outOfLineAccess.link(&m_jit);
+                m_jit.move(indexGPR, scratchGPR);
+                m_jit.sub32(MacroAssembler::Address(enumeratorGPR, JSPropertyNameEnumerator::cachedInlineCapacityOffset()), scratchGPR);
+                m_jit.neg32(scratchGPR);
+                m_jit.signExtend32ToPtr(scratchGPR, scratchGPR);
+                if (!haveStorage)
+                    m_jit.loadPtr(MacroAssembler::Address(baseCellGPR, JSObject::butterflyOffset()), storageGPR);
+                int32_t offsetOfFirstProperty = static_cast<int32_t>(offsetInButterfly(firstOutOfLineOffset)) * sizeof(EncodedJSValue);
+                m_jit.loadValue(MacroAssembler::BaseIndex(storageGPR, scratchGPR, MacroAssembler::TimesEight, offsetOfFirstProperty), resultRegs);
+                doneCases.append(m_jit.jump());
+            }
+
+            notFastNamedCases.link(&m_jit);
+            return std::make_pair(resultRegs, DataFormatJS);
+        }));
+
+        // We rely on compileGetByVal to call jsValueResult for us.
+        // FIXME: This is kinda hacky...
+        ASSERT(generationInfo(node).jsValueRegs() == resultRegs && generationInfo(node).registerFormat() == DataFormatJS);
+
+        if (badStructureSlowPath.isSet())
+            addSlowPathGenerator(slowPathCall(badStructureSlowPath, this, operationEnumeratorRecoverNameAndGetByVal, resultRegs, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseCellGPR, indexGPR, enumeratorGPR));
+
+        doneCases.link(&m_jit);
+    };
+
+    if (isCell(baseEdge.useKind())) {
+        // Use manual operand speculation since Fixup may have picked a UseKind more restrictive than CellUse.
+        speculate(node, baseEdge);
+        SpeculateCellOperand baseOperand(this, baseEdge, ManualOperandSpeculation);
+        generate(baseOperand.gpr());
+    } else {
+        JSValueOperand baseOperand(this, baseEdge);
+        generate(baseOperand.gpr());
+    }
+}
+
+template<typename SlowPathFunctionType>
+void SpeculativeJIT::compileEnumeratorHasProperty(Node* node, SlowPathFunctionType slowPathFunction)
+{
+    Edge baseEdge = m_graph.varArgChild(node, 0);
+    auto generate = [&] (auto base, GPRReg baseCellGPR) {
+        JSValueOperand propertyName(this, m_graph.varArgChild(node, 1));
+        SpeculateStrictInt32Operand index(this, m_graph.varArgChild(node, 2));
+        SpeculateStrictInt32Operand mode(this, m_graph.varArgChild(node, 3));
+        SpeculateCellOperand enumerator(this, m_graph.varArgChild(node, 4));
+
+        JSValueRegs propertyNameRegs = propertyName.regs();
+        GPRReg indexGPR = index.gpr();
+        GPRReg modeGPR = mode.gpr();
+        GPRReg enumeratorGPR = enumerator.gpr();
+
+        JSValueRegsTemporary result(this);
+        JSValueRegs resultRegs = result.regs();
+
+        MacroAssembler::JumpList operationCases;
+
+        if (m_state.forNode(baseEdge).m_type & ~SpecCell)
+            operationCases.append(m_jit.branchIfNotCell(base));
+
+        // FIXME: We shouldn't generate this code if we know base is not a cell.
+        operationCases.append(m_jit.branchTest32(MacroAssembler::Zero, modeGPR, TrustedImm32(JSPropertyNameEnumerator::OwnStructureMode)));
+
+        m_jit.load32(MacroAssembler::Address(baseCellGPR, JSCell::structureIDOffset()), resultRegs.payloadGPR());
+        operationCases.append(m_jit.branch32(MacroAssembler::NotEqual, resultRegs.payloadGPR(), MacroAssembler::Address(enumeratorGPR, JSPropertyNameEnumerator::cachedStructureIDOffset())));
+
+        moveTrueTo(resultRegs.payloadGPR());
+        MacroAssembler::Jump done = m_jit.jump();
+
+        operationCases.link(&m_jit);
+
+        flushRegisters();
+#if USE(JSVALUE32_64)
+        m_jit.move(TrustedImm32(JSValue::CellTag), resultRegs.tagGPR());
+        auto baseRegs = JSValueRegs(baseCellGPR, resultRegs.tagGPR());
+#else
+        auto baseRegs = base;
+#endif
+        callOperation(slowPathFunction, resultRegs, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseRegs, propertyNameRegs, indexGPR, modeGPR);
+        m_jit.exceptionCheck();
+
+        done.link(&m_jit);
+
+        blessedBooleanResult(resultRegs.payloadGPR(), node);
+    };
+
+    if (isCell(baseEdge.useKind())) {
+        SpeculateCellOperand base(this, baseEdge);
+        generate(base.gpr(), base.gpr());
+    } else {
+        JSValueOperand base(this, baseEdge);
+        generate(base.regs(), base.regs().payloadGPR());
+    }
+}
+
+void SpeculativeJIT::compileEnumeratorInByVal(Node* node)
+{
+    compileEnumeratorHasProperty(node, operationEnumeratorInByVal);
+}
+
+void SpeculativeJIT::compileEnumeratorHasOwnProperty(Node* node)
+{
+    compileEnumeratorHasProperty(node, operationEnumeratorHasOwnProperty);
 }
 
@@ -13625 +13945 @@
 }
 
-void SpeculativeJIT::compileHasEnumerableStructureProperty(Node* node)
-{
-    JSValueOperand base(this, node->child1());
-    SpeculateCellOperand property(this, node->child2());
-    SpeculateCellOperand enumerator(this, node->child3());
-    JSValueRegsTemporary result(this);
-
-    JSValueRegs baseRegs = base.jsValueRegs();
-    GPRReg propertyGPR = property.gpr();
-    JSValueRegs resultRegs = result.regs();
-
-    CCallHelpers::JumpList wrongStructure;
-
-    wrongStructure.append(m_jit.branchIfNotCell(baseRegs));
-
-    m_jit.load32(MacroAssembler::Address(baseRegs.payloadGPR(), JSCell::structureIDOffset()), resultRegs.payloadGPR());
-    wrongStructure.append(m_jit.branch32(MacroAssembler::NotEqual,
-        resultRegs.payloadGPR(),
-        MacroAssembler::Address(enumerator.gpr(), JSPropertyNameEnumerator::cachedStructureIDOffset())));
-
-    moveTrueTo(resultRegs.payloadGPR());
-
-    addSlowPathGenerator(slowPathCall(wrongStructure, this, operationHasEnumerableProperty, resultRegs, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseRegs, propertyGPR));
-    blessedBooleanResult(resultRegs.payloadGPR(), node);
-}
-
-template <typename Function>
-void SpeculativeJIT::compileHasOwnStructurePropertyImpl(Node* node, Function slowPath)
-{
-    SpeculateCellOperand base(this, node->child1());
-    SpeculateCellOperand property(this, node->child2());
-    SpeculateCellOperand enumerator(this, node->child3());
-    JSValueRegsTemporary result(this);
-
-    GPRReg baseGPR = base.gpr();
-    GPRReg propertyGPR = property.gpr();
-    JSValueRegs resultRegs = result.regs();
-
-    CCallHelpers::JumpList wrongStructure;
-
-    m_jit.load32(MacroAssembler::Address(baseGPR, JSCell::structureIDOffset()), resultRegs.payloadGPR());
-    wrongStructure.append(m_jit.branch32(MacroAssembler::NotEqual,
-        resultRegs.payloadGPR(),
-        MacroAssembler::Address(enumerator.gpr(), JSPropertyNameEnumerator::cachedStructureIDOffset())));
-
-    moveTrueTo(resultRegs.payloadGPR());
-
-    addSlowPathGenerator(slowPathCall(wrongStructure, this, slowPath, resultRegs, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseGPR, propertyGPR));
-    blessedBooleanResult(resultRegs.payloadGPR(), node);
-}
-
-void SpeculativeJIT::compileHasOwnStructureProperty(Node* node)
-{
-    compileHasOwnStructurePropertyImpl(node, operationHasOwnStructureProperty);
-}
-
-void SpeculativeJIT::compileInStructureProperty(Node* node)
-{
-    compileHasOwnStructurePropertyImpl(node, operationInStructureProperty);
-}
-
 void SpeculativeJIT::compileGetPropertyEnumerator(Node* node)
 {
@@ -13710 +13969 @@
     m_jit.exceptionCheck();
     cellResult(resultGPR, node);
-}
-
-void SpeculativeJIT::compileGetEnumeratorPname(Node* node)
-{
-    ASSERT(node->op() == GetEnumeratorStructurePname || node->op() == GetEnumeratorGenericPname);
-    SpeculateCellOperand enumerator(this, node->child1());
-    SpeculateStrictInt32Operand index(this, node->child2());
-    GPRTemporary scratch(this);
-    JSValueRegsTemporary result(this);
-
-    GPRReg enumeratorGPR = enumerator.gpr();
-    GPRReg indexGPR = index.gpr();
-    GPRReg scratchGPR = scratch.gpr();
-    JSValueRegs resultRegs = result.regs();
-
-    MacroAssembler::Jump inBounds = m_jit.branch32(MacroAssembler::Below, indexGPR,
-        MacroAssembler::Address(enumeratorGPR, (node->op() == GetEnumeratorStructurePname)
-            ? JSPropertyNameEnumerator::endStructurePropertyIndexOffset()
-            : JSPropertyNameEnumerator::endGenericPropertyIndexOffset()));
-
-    m_jit.moveValue(jsNull(), resultRegs);
-
-    MacroAssembler::Jump done = m_jit.jump();
-    inBounds.link(&m_jit);
-
-    m_jit.loadPtr(MacroAssembler::Address(enumeratorGPR, JSPropertyNameEnumerator::cachedPropertyNamesVectorOffset()), scratchGPR);
-    m_jit.loadPtr(MacroAssembler::BaseIndex(scratchGPR, indexGPR, MacroAssembler::ScalePtr), resultRegs.payloadGPR());
-#if USE(JSVALUE32_64)
-    m_jit.move(MacroAssembler::TrustedImm32(JSValue::CellTag), resultRegs.tagGPR());
-#endif
-
-    done.link(&m_jit);
-    jsValueResult(resultRegs, node);
 }
 
@@ -14937 +15163 @@
 }
 
-void SpeculativeJIT::compileHasIndexedProperty(Node* node, S_JITOperation_GCZ slowPathOperation)
+void SpeculativeJIT::compileHasIndexedProperty(Node* node, S_JITOperation_GCZ slowPathOperation, const ScopedLambda<std::tuple<GPRReg, GPRReg>()>& prefix)
 {
     SpeculateCellOperand base(this, m_graph.varArgChild(node, 0));
-    SpeculateStrictInt32Operand index(this, m_graph.varArgChild(node, 1));
-    GPRTemporary result(this);
 
     GPRReg baseGPR = base.gpr();
-    GPRReg indexGPR = index.gpr();
-    GPRReg resultGPR = result.gpr();
+    GPRReg indexGPR = InvalidGPRReg;
+    GPRReg resultGPR = InvalidGPRReg;
 
     MacroAssembler::JumpList slowCases;
@@ -14952 +15176 @@
     case Array::Int32:
     case Array::Contiguous: {
-        ASSERT(!!m_graph.varArgChild(node, 2));
-        StorageOperand storage(this, m_graph.varArgChild(node, 2));
+        ASSERT(!!m_graph.varArgChild(node, node->storageChildIndex()));
+        StorageOperand storage(this, m_graph.varArgChild(node, node->storageChildIndex()));
         GPRTemporary scratch(this);
 
         GPRReg storageGPR = storage.gpr();
         GPRReg scratchGPR = scratch.gpr();
+
+        std::tie(indexGPR, resultGPR) = prefix();
 
         MacroAssembler::Jump outOfBounds = m_jit.branch32(MacroAssembler::AboveOrEqual, indexGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
@@ -14986 +15212 @@
     }
     case Array::Double: {
-        ASSERT(!!m_graph.varArgChild(node, 2));
-        StorageOperand storage(this, m_graph.varArgChild(node, 2));
+        ASSERT(!!m_graph.varArgChild(node, node->storageChildIndex()));
+        StorageOperand storage(this, m_graph.varArgChild(node, node->storageChildIndex()));
         FPRTemporary scratch(this);
         FPRReg scratchFPR = scratch.fpr();
         GPRReg storageGPR = storage.gpr();
+
+        std::tie(indexGPR, resultGPR) = prefix();
 
         MacroAssembler::Jump outOfBounds = m_jit.branch32(MacroAssembler::AboveOrEqual, indexGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
@@ -15015 +15243 @@
     }
     case Array::ArrayStorage: {
-        ASSERT(!!m_graph.varArgChild(node, 2));
-        StorageOperand storage(this, m_graph.varArgChild(node, 2));
+        ASSERT(!!m_graph.varArgChild(node, node->storageChildIndex()));
+        StorageOperand storage(this, m_graph.varArgChild(node, node->storageChildIndex()));
         GPRTemporary scratch(this);
 
         GPRReg storageGPR = storage.gpr();
         GPRReg scratchGPR = scratch.gpr();
+
+        std::tie(indexGPR, resultGPR) = prefix();
 
         MacroAssembler::Jump outOfBounds = m_jit.branch32(MacroAssembler::AboveOrEqual, indexGPR, MacroAssembler::Address(storageGPR, ArrayStorage::vectorLengthOffset()));
@@ -15050 +15280 @@
         // FIXME: Optimize TypedArrays in HasIndexedProperty IC
         // https://bugs.webkit.org/show_bug.cgi?id=221183
+        std::tie(indexGPR, resultGPR) = prefix();
         slowCases.append(m_jit.jump());
         break;
@@ -15056 +15287 @@
 
     addSlowPathGenerator(slowPathCall(slowCases, this, slowPathOperation, resultGPR, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseGPR, indexGPR));
-
-    unblessedBooleanResult(resultGPR, node);
-}
-
-void SpeculativeJIT::compileGetDirectPname(Node* node)
-{
-    Edge& baseEdge = m_jit.graph().varArgChild(node, 0);
-    Edge& propertyEdge = m_jit.graph().varArgChild(node, 1);
-    Edge& indexEdge = m_jit.graph().varArgChild(node, 2);
-
-    SpeculateCellOperand base(this, baseEdge);
-    SpeculateCellOperand property(this, propertyEdge);
-    GPRReg baseGPR = base.gpr();
-    GPRReg propertyGPR = property.gpr();
-
-    Edge& enumeratorEdge = m_jit.graph().varArgChild(node, 3);
-    SpeculateStrictInt32Operand index(this, indexEdge);
-    SpeculateCellOperand enumerator(this, enumeratorEdge);
-    GPRTemporary scratch(this);
-    JSValueRegsTemporary result(this);
-
-    GPRReg indexGPR = index.gpr();
-    GPRReg enumeratorGPR = enumerator.gpr();
-    GPRReg scratchGPR = scratch.gpr();
-    JSValueRegs resultRegs = result.regs();
-
-    MacroAssembler::JumpList slowPath;
-
-    // Check the structure
-    m_jit.load32(MacroAssembler::Address(baseGPR, JSCell::structureIDOffset()), scratchGPR);
-    slowPath.append(
-        m_jit.branch32(
-            MacroAssembler::NotEqual,
-            scratchGPR,
-            MacroAssembler::Address(
-                enumeratorGPR, JSPropertyNameEnumerator::cachedStructureIDOffset())));
-
-    // Compute the offset
-    // If index is less than the enumerator's cached inline storage, then it's an inline access
-    MacroAssembler::Jump outOfLineAccess = m_jit.branch32(MacroAssembler::AboveOrEqual,
-        indexGPR, MacroAssembler::Address(enumeratorGPR, JSPropertyNameEnumerator::cachedInlineCapacityOffset()));
-
-    m_jit.loadValue(MacroAssembler::BaseIndex(baseGPR, indexGPR, MacroAssembler::TimesEight, JSObject::offsetOfInlineStorage()), resultRegs);
-
-    MacroAssembler::Jump done = m_jit.jump();
-
-    // Otherwise it's out of line
-    outOfLineAccess.link(&m_jit);
-    m_jit.loadPtr(MacroAssembler::Address(baseGPR, JSObject::butterflyOffset()), resultRegs.payloadGPR());
-    m_jit.move(indexGPR, scratchGPR);
-    m_jit.sub32(MacroAssembler::Address(enumeratorGPR, JSPropertyNameEnumerator::cachedInlineCapacityOffset()), scratchGPR);
-    m_jit.neg32(scratchGPR);
-    m_jit.signExtend32ToPtr(scratchGPR, scratchGPR);
-    int32_t offsetOfFirstProperty = static_cast<int32_t>(offsetInButterfly(firstOutOfLineOffset)) * sizeof(EncodedJSValue);
-    m_jit.loadValue(MacroAssembler::BaseIndex(resultRegs.payloadGPR(), scratchGPR, MacroAssembler::TimesEight, offsetOfFirstProperty), resultRegs);
-
-    done.link(&m_jit);
-
-    addSlowPathGenerator(slowPathCall(slowPath, this, operationGetByValCell, resultRegs, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseGPR, CCallHelpers::CellValue(propertyGPR)));
-
-    jsValueResult(resultRegs, node);
 }
 

trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h

@@ -48 +48 @@
 
 class GPRTemporary;
+class JSValueRegsTemporary;
 class JSValueOperand;
 class SlowPathGenerator;
@@ -247 +248 @@
     GPRReg allocate(GPRReg specific)
     {
+        if (specific == InvalidGPRReg)
+            allocate();
 #if ENABLE(DFG_REGISTER_ALLOCATION_VALIDATION)
         m_jit.addRegisterAllocationAtOffset(m_jit.debugOffset());
@@ -1346 +1349 @@
     }
     
+    // We use a scopedLambda to placate register allocation validation.
+    void compileGetByVal(Node*, const ScopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>& prefix);
+
     void compileGetCharCodeAt(Node*);
-    void compileGetByValOnString(Node*);
+    void compileGetByValOnString(Node*, const ScopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>& prefix);
     void compileFromCharCode(Node*); 
 
-    void compileGetByValOnDirectArguments(Node*);
-    void compileGetByValOnScopedArguments(Node*);
+    void compileGetByValOnDirectArguments(Node*, const ScopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>& prefix);
+    void compileGetByValOnScopedArguments(Node*, const ScopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>& prefix);
 
     void compileGetPrivateName(Node*);
@@ -1425 +1431 @@
     void emitTypedArrayBoundsCheck(Node*, GPRReg baseGPR, GPRReg indexGPR);
     void compileGetTypedArrayByteOffset(Node*);
-    void compileGetByValOnIntTypedArray(Node*, TypedArrayType);
+    void compileGetByValOnIntTypedArray(Node*, TypedArrayType, const ScopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>& prefix);
     void compilePutByValForIntTypedArray(GPRReg base, GPRReg property, Node*, TypedArrayType);
-    void compileGetByValOnFloatTypedArray(Node*, TypedArrayType);
+    void compileGetByValOnFloatTypedArray(Node*, TypedArrayType, const ScopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>& prefix);
     void compilePutByValForFloatTypedArray(GPRReg base, GPRReg property, Node*, TypedArrayType);
-    void compileGetByValForObjectWithString(Node*);
-    void compileGetByValForObjectWithSymbol(Node*);
+    void compileGetByValForObjectWithString(Node*, const ScopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>& prefix);
+    void compileGetByValForObjectWithSymbol(Node*, const ScopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>& prefix);
     void compilePutByValForCellWithString(Node*, Edge& child1, Edge& child2, Edge& child3);
     void compilePutByValForCellWithSymbol(Node*, Edge& child1, Edge& child2, Edge& child3);
@@ -1459 +1465 @@
         Edge valueUse);
     void loadFromIntTypedArray(GPRReg storageReg, GPRReg propertyReg, GPRReg resultReg, TypedArrayType);
-    void setIntTypedArrayLoadResult(Node*, GPRReg resultReg, TypedArrayType, bool canSpeculate);
+    void setIntTypedArrayLoadResult(Node*, JSValueRegs resultRegs, TypedArrayType, bool canSpeculate, bool shouldBox);
     template <typename ClassType> void compileNewFunctionCommon(GPRReg, RegisteredStructure, GPRReg, GPRReg, GPRReg, MacroAssembler::JumpList&, size_t, FunctionExecutable*);
     void compileNewFunction(Node*);
@@ -1525 +1531 @@
     void compileThrow(Node*);
     void compileThrowStaticError(Node*);
-    void compileGetEnumerableLength(Node*);
-    void compileHasEnumerableStructureProperty(Node*);
-    void compileHasEnumerableProperty(Node*);
-    void compileToIndexString(Node*);
+
+    void compileEnumeratorNextUpdateIndexAndMode(Node*);
+    void compileEnumeratorNextExtractMode(Node*);
+    void compileEnumeratorNextExtractIndex(Node*);
+    void compileEnumeratorNextUpdatePropertyName(Node*);
+    void compileEnumeratorGetByVal(Node*);
+    template<typename SlowPathFunctionType>
+    void compileEnumeratorHasProperty(Node*, SlowPathFunctionType);
+    void compileEnumeratorInByVal(Node*);
+    void compileEnumeratorHasOwnProperty(Node*);
+
     void compilePutByIdFlush(Node*);
     void compilePutById(Node*);
     void compilePutByIdDirect(Node*);
     void compilePutByIdWithThis(Node*);
-    template <typename Function>
-    void compileHasOwnStructurePropertyImpl(Node*, Function);
-    void compileHasOwnStructureProperty(Node*);
-    void compileInStructureProperty(Node*);
-    void compileGetDirectPname(Node*);
     void compileGetPropertyEnumerator(Node*);
-    void compileGetEnumeratorPname(Node*);
     void compileGetExecutable(Node*);
     void compileGetGetter(Node*);
@@ -1569 +1576 @@
     void compileLogShadowChickenPrologue(Node*);
     void compileLogShadowChickenTail(Node*);
-    void compileHasIndexedProperty(Node*, S_JITOperation_GCZ);
+    void compileHasIndexedProperty(Node*, S_JITOperation_GCZ, const ScopedLambda<std::tuple<GPRReg, GPRReg>()>& prefix);
     void compileExtractCatchLocal(Node*);
     void compileClearCatchLocals(Node*);
@@ -1927 +1934 @@
     }
 
+    JSValueRegs regs() { return jsValueRegs(); }
+
 #if USE(JSVALUE64)
     GPRReg gpr()
@@ -1966 +1975 @@
     }
 
-    GPRReg gpr(WhichValueWord which)
+    GPRReg gpr(WhichValueWord which = PayloadWord)
     {
         return jsValueRegs().gpr(which);
@@ -2004 +2013 @@
     WTF_MAKE_FAST_ALLOCATED;
 public:
+    StorageOperand() = default;
+
     explicit StorageOperand(SpeculativeJIT* jit, Edge edge)
         : m_jit(jit)
@@ -2009 +2020 @@
         , m_gprOrInvalid(InvalidGPRReg)
     {
+        emplace(jit, edge);
+    }
+    
+    ~StorageOperand()
+    {
+        if (m_gprOrInvalid != InvalidGPRReg)
+            m_jit->unlock(m_gprOrInvalid);
+    }
+    
+    Edge edge() const
+    {
+        return m_edge;
+    }
+    
+    Node* node() const
+    {
+        return edge().node();
+    }
+    
+    GPRReg gpr()
+    {
+        if (m_gprOrInvalid == InvalidGPRReg)
+            m_gprOrInvalid = m_jit->fillStorage(edge());
+        return m_gprOrInvalid;
+    }
+
+    void emplace(SpeculativeJIT* jit, Edge edge)
+    {
+        m_jit = jit;
+        m_edge = edge;
+        ASSERT(m_gprOrInvalid == InvalidGPRReg);
         ASSERT(m_jit);
         ASSERT(edge.useKind() == UntypedUse || edge.useKind() == KnownCellUse);
@@ -2015 +2057 @@
     }
     
-    ~StorageOperand()
-    {
-        ASSERT(m_gprOrInvalid != InvalidGPRReg);
-        m_jit->unlock(m_gprOrInvalid);
-    }
-    
-    Edge edge() const
-    {
-        return m_edge;
-    }
-    
-    Node* node() const
-    {
-        return edge().node();
-    }
-    
-    GPRReg gpr()
-    {
-        if (m_gprOrInvalid == InvalidGPRReg)
-            m_gprOrInvalid = m_jit->fillStorage(edge());
-        return m_gprOrInvalid;
-    }
-    
     void use()
     {
@@ -2044 +2063 @@
     
 private:
-    SpeculativeJIT* m_jit;
+    SpeculativeJIT* m_jit { nullptr };
     Edge m_edge;
-    GPRReg m_gprOrInvalid;
+    GPRReg m_gprOrInvalid { InvalidGPRReg };
 };
 
@@ -2135 +2154 @@
     JSValueRegsTemporary();
     JSValueRegsTemporary(SpeculativeJIT*);
+    JSValueRegsTemporary(SpeculativeJIT*, GPRReg specificPayload);
+#if USE(JSVALUE32_64)
+    JSValueRegsTemporary(SpeculativeJIT*, GPRReg specificPayload, GPRReg specificTag);
+#endif
     template<typename T>
     JSValueRegsTemporary(SpeculativeJIT*, ReuseTag, T& operand, WhichValueWord resultRegWord = PayloadWord);
@@ -2140 +2163 @@
     ~JSValueRegsTemporary();
     
+    explicit operator bool() { return !!regs(); }
+
+    JSValueRegsTemporary& operator=(JSValueRegsTemporary&&) = default;
+
     JSValueRegs regs();
 
@@ -2150 +2177 @@
 #endif
 };
+
+#if USE(JSVALUE64)
+template<typename T>
+JSValueRegsTemporary::JSValueRegsTemporary(SpeculativeJIT* jit, ReuseTag, T& operand, WhichValueWord)
+    : m_gpr(jit, Reuse, operand)
+{
+}
+#else
+template<typename T>
+JSValueRegsTemporary::JSValueRegsTemporary(SpeculativeJIT* jit, ReuseTag, T& operand, WhichValueWord resultWord)
+{
+    if (resultWord == PayloadWord) {
+        m_payloadGPR = GPRTemporary(jit, Reuse, operand);
+        m_tagGPR = GPRTemporary(jit);
+    } else {
+        m_payloadGPR = GPRTemporary(jit);
+        m_tagGPR = GPRTemporary(jit, Reuse, operand);
+    }
+}
+#endif
 
 class FPRTemporary {

trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp

@@ -1813 +1813 @@
 }
 
+void SpeculativeJIT::compileGetByVal(Node* node, const ScopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>& prefix)
+{
+    switch (node->arrayMode().type()) {
+    case Array::SelectUsingPredictions:
+    case Array::ForceExit:
+        RELEASE_ASSERT_NOT_REACHED();
+#if COMPILER_QUIRK(CONSIDERS_UNREACHABLE_CODE)
+        terminateSpeculativeExecution(InadequateCoverage, JSValueRegs(), 0);
+#endif
+        break;
+    case Array::Undecided: {
+        SpeculateStrictInt32Operand index(this, m_graph.varArgChild(node, 1));
+
+        GPRReg indexGPR = index.gpr();
+
+        JSValueRegs resultRegs;
+        std::tie(resultRegs, std::ignore) = prefix(DataFormatJS);
+
+        speculationCheck(OutOfBounds, JSValueRegs(), node,
+            m_jit.branch32(MacroAssembler::LessThan, indexGPR, MacroAssembler::TrustedImm32(0)));
+
+        m_jit.move(MacroAssembler::TrustedImm32(JSValue::UndefinedTag), resultRegs.tagGPR());
+        m_jit.move(MacroAssembler::TrustedImm32(0), resultRegs.payloadGPR());
+        jsValueResult(resultRegs, node);
+        break;
+    }
+    case Array::Generic: {
+        if (m_graph.m_slowGetByVal.contains(node)) {
+            if (m_graph.varArgChild(node, 0).useKind() == ObjectUse) {
+                if (m_graph.varArgChild(node, 1).useKind() == StringUse) {
+                    compileGetByValForObjectWithString(node, prefix);
+                    break;
+                }
+
+                if (m_graph.varArgChild(node, 1).useKind() == SymbolUse) {
+                    compileGetByValForObjectWithSymbol(node, prefix);
+                    break;
+                }
+            }
+
+            SpeculateCellOperand base(this, m_graph.varArgChild(node, 0)); // Save a register, speculate cell. We'll probably be right.
+            JSValueOperand property(this, m_graph.varArgChild(node, 1));
+            GPRReg baseGPR = base.gpr();
+            JSValueRegs propertyRegs = property.jsValueRegs();
+
+            JSValueRegs resultRegs;
+            std::tie(resultRegs, std::ignore) = prefix(DataFormatJS);
+
+            flushRegisters();
+            callOperation(operationGetByValCell, resultRegs, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseGPR, propertyRegs);
+            m_jit.exceptionCheck();
+
+            jsValueResult(resultRegs, node);
+            break;
+        }
+
+        speculate(node, m_graph.varArgChild(node, 0));
+        speculate(node, m_graph.varArgChild(node, 1));
+
+        JSValueOperand base(this, m_graph.varArgChild(node, 0), ManualOperandSpeculation);
+        JSValueOperand property(this, m_graph.varArgChild(node, 1), ManualOperandSpeculation);
+
+        JSValueRegs baseRegs = base.jsValueRegs();
+        JSValueRegs propertyRegs = property.jsValueRegs();
+
+        JSValueRegs resultRegs;
+        std::tie(resultRegs, std::ignore) = prefix(DataFormatJS);
+
+        CodeOrigin codeOrigin = node->origin.semantic;
+        CallSiteIndex callSite = m_jit.recordCallSiteAndGenerateExceptionHandlingOSRExitIfNeeded(codeOrigin, m_stream->size());
+        RegisterSet usedRegisters = this->usedRegisters();
+
+        JITCompiler::JumpList slowCases;
+        if (!m_state.forNode(m_graph.varArgChild(node, 0)).isType(SpecCell))
+            slowCases.append(m_jit.branchIfNotCell(baseRegs.tagGPR()));
+
+        JITGetByValGenerator gen(
+            m_jit.codeBlock(), JITType::DFGJIT, codeOrigin, callSite, AccessType::GetByVal, usedRegisters,
+            baseRegs, propertyRegs, resultRegs, InvalidGPRReg);
+
+        if (m_state.forNode(m_graph.varArgChild(node, 1)).isType(SpecString))
+            gen.stubInfo()->propertyIsString = true;
+        else if (m_state.forNode(m_graph.varArgChild(node, 1)).isType(SpecInt32Only))
+            gen.stubInfo()->propertyIsInt32 = true;
+        else if (m_state.forNode(m_graph.varArgChild(node, 1)).isType(SpecSymbol))
+            gen.stubInfo()->propertyIsSymbol = true;
+
+        gen.generateFastPath(m_jit);
+
+        slowCases.append(gen.slowPathJump());
+
+        std::unique_ptr<SlowPathGenerator> slowPath = slowPathCall(
+            slowCases, this, operationGetByValOptimize,
+            resultRegs, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(codeOrigin)), gen.stubInfo(), nullptr, baseRegs, propertyRegs);
+
+        m_jit.addGetByVal(gen, slowPath.get());
+        addSlowPathGenerator(WTFMove(slowPath));
+
+        jsValueResult(resultRegs, node);
+        break;
+    }
+    case Array::Int32:
+    case Array::Contiguous: {
+        if (node->arrayMode().isInBounds()) {
+            SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
+            StorageOperand storage(this, m_graph.varArgChild(node, 2));
+
+            GPRReg propertyReg = property.gpr();
+            GPRReg storageReg = storage.gpr();
+
+            if (!m_compileOkay)
+                return;
+
+            JSValueRegs resultRegs;
+            DataFormat format;
+            std::tie(resultRegs, format) = prefix(node->arrayMode().type() == Array::Int32 ? DataFormatInt32 : DataFormatJS);
+
+            speculationCheck(OutOfBounds, JSValueRegs(), 0, m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, Butterfly::offsetOfPublicLength())));
+
+            if (format == DataFormatInt32) {
+                ASSERT(!node->arrayMode().isInBoundsSaneChain());
+
+                speculationCheck(
+                    OutOfBounds, JSValueRegs(), 0,
+                    m_jit.branch32(
+                        MacroAssembler::Equal,
+                        MacroAssembler::BaseIndex(
+                            storageReg, propertyReg, MacroAssembler::TimesEight, TagOffset),
+                        TrustedImm32(JSValue::EmptyValueTag)));
+                m_jit.load32(
+                    MacroAssembler::BaseIndex(
+                        storageReg, propertyReg, MacroAssembler::TimesEight, PayloadOffset),
+                    resultRegs.payloadGPR());
+                strictInt32Result(resultRegs.payloadGPR(), node);
+                break;
+            }
+
+            ASSERT(format == DataFormatJS);
+            m_jit.load32(
+                MacroAssembler::BaseIndex(
+                    storageReg, propertyReg, MacroAssembler::TimesEight, TagOffset),
+                resultRegs.tagGPR());
+            m_jit.load32(
+                MacroAssembler::BaseIndex(
+                    storageReg, propertyReg, MacroAssembler::TimesEight, PayloadOffset),
+                resultRegs.payloadGPR());
+            if (node->arrayMode().isInBoundsSaneChain()) {
+                JITCompiler::Jump notHole = m_jit.branchIfNotEmpty(resultRegs.tagGPR());
+                m_jit.move(TrustedImm32(JSValue::UndefinedTag), resultRegs.tagGPR());
+                m_jit.move(TrustedImm32(0), resultRegs.payloadGPR());
+                notHole.link(&m_jit);
+            } else {
+                speculationCheck(
+                    LoadFromHole, JSValueRegs(), nullptr,
+                    m_jit.branchIfEmpty(resultRegs.tagGPR()));
+            }
+            jsValueResult(resultRegs, node);
+            break;
+        }
+
+        SpeculateCellOperand base(this, m_graph.varArgChild(node, 0));
+        SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
+        StorageOperand storage(this, m_graph.varArgChild(node, 2));
+
+        GPRReg baseReg = base.gpr();
+        GPRReg propertyReg = property.gpr();
+        GPRReg storageReg = storage.gpr();
+
+        if (!m_compileOkay)
+            return;
+
+        JSValueRegs resultRegs;
+        std::tie(resultRegs, std::ignore) = prefix(DataFormatJS);
+
+        MacroAssembler::JumpList slowCases;
+
+        slowCases.append(m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, Butterfly::offsetOfPublicLength())));
+
+        m_jit.loadValue(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight), resultRegs);
+        slowCases.append(m_jit.branchIfEmpty(resultRegs.tagGPR()));
+
+        addSlowPathGenerator(
+            slowPathCall(
+                slowCases, this, operationGetByValObjectInt,
+                resultRegs, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseReg, propertyReg));
+
+        jsValueResult(resultRegs, node);
+        break;
+    }
+    case Array::Double: {
+        if (node->arrayMode().isInBounds()) {
+            SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
+            StorageOperand storage(this, m_graph.varArgChild(node, 2));
+
+            GPRReg propertyReg = property.gpr();
+            GPRReg storageReg = storage.gpr();
+
+            if (!m_compileOkay)
+                return;
+
+            FPRTemporary result(this);
+
+            JSValueRegs resultRegs;
+            DataFormat format;
+            std::tie(resultRegs, format) = prefix(DataFormatDouble);
+
+            speculationCheck(OutOfBounds, JSValueRegs(), 0, m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, Butterfly::offsetOfPublicLength())));
+
+            m_jit.loadDouble(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight), result.fpr());
+            if (!node->arrayMode().isInBoundsSaneChain())
+                speculationCheck(LoadFromHole, JSValueRegs(), 0, m_jit.branchIfNaN(result.fpr()));
+            if (format == DataFormatJS) {
+                boxDouble(result.fpr(), resultRegs);
+                jsValueResult(resultRegs, node);
+            } else {
+                ASSERT(format == DataFormatDouble && !resultRegs);
+                doubleResult(result.fpr(), node);
+            }
+            break;
+        }
+
+        SpeculateCellOperand base(this, m_graph.varArgChild(node, 0));
+        SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
+        StorageOperand storage(this, m_graph.varArgChild(node, 2));
+
+        GPRReg baseReg = base.gpr();
+        GPRReg propertyReg = property.gpr();
+        GPRReg storageReg = storage.gpr();
+
+        if (!m_compileOkay)
+            return;
+
+        FPRTemporary temp(this);
+        FPRReg tempReg = temp.fpr();
+
+        JSValueRegs resultRegs;
+        std::tie(resultRegs, std::ignore) = prefix(DataFormatJS);
+
+        MacroAssembler::JumpList slowCases;
+
+        slowCases.append(m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, Butterfly::offsetOfPublicLength())));
+
+        m_jit.loadDouble(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight), tempReg);
+        slowCases.append(m_jit.branchIfNaN(tempReg));
+        boxDouble(tempReg, resultRegs);
+
+        addSlowPathGenerator(
+            slowPathCall(
+                slowCases, this, operationGetByValObjectInt,
+                resultRegs, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseReg, propertyReg));
+
+        jsValueResult(resultRegs, node);
+        break;
+    }
+    case Array::ArrayStorage:
+    case Array::SlowPutArrayStorage: {
+        if (node->arrayMode().isInBounds()) {
+            SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
+            StorageOperand storage(this, m_graph.varArgChild(node, 2));
+            GPRReg propertyReg = property.gpr();
+            GPRReg storageReg = storage.gpr();
+
+            if (!m_compileOkay)
+                return;
+
+            JSValueRegs resultRegs;
+            std::tie(resultRegs, std::ignore) = prefix(DataFormatJS);
+
+            speculationCheck(OutOfBounds, JSValueRegs(), 0, m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, ArrayStorage::vectorLengthOffset())));
+
+            m_jit.load32(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight, ArrayStorage::vectorOffset() + OBJECT_OFFSETOF(JSValue, u.asBits.tag)), resultRegs.tagGPR());
+            speculationCheck(LoadFromHole, JSValueRegs(), 0, m_jit.branchIfEmpty(resultRegs.tagGPR()));
+            m_jit.load32(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight, ArrayStorage::vectorOffset() + OBJECT_OFFSETOF(JSValue, u.asBits.payload)), resultRegs.payloadGPR());
+
+            jsValueResult(resultRegs, node);
+            break;
+        }
+
+        SpeculateCellOperand base(this, m_graph.varArgChild(node, 0));
+        SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
+        StorageOperand storage(this, m_graph.varArgChild(node, 2));
+        GPRReg propertyReg = property.gpr();
+        GPRReg storageReg = storage.gpr();
+        GPRReg baseReg = base.gpr();
+
+        if (!m_compileOkay)
+            return;
+
+        JSValueRegs resultRegs;
+        std::tie(resultRegs, std::ignore) = prefix(DataFormatJS);
+
+        JITCompiler::Jump outOfBounds = m_jit.branch32(
+            MacroAssembler::AboveOrEqual, propertyReg,
+            MacroAssembler::Address(storageReg, ArrayStorage::vectorLengthOffset()));
+
+        m_jit.load32(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight, ArrayStorage::vectorOffset() + OBJECT_OFFSETOF(JSValue, u.asBits.tag)), resultRegs.tagGPR());
+        JITCompiler::Jump hole = m_jit.branchIfEmpty(resultRegs.tagGPR());
+        m_jit.load32(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight, ArrayStorage::vectorOffset() + OBJECT_OFFSETOF(JSValue, u.asBits.payload)), resultRegs.payloadGPR());
+
+        JITCompiler::JumpList slowCases;
+        slowCases.append(outOfBounds);
+        slowCases.append(hole);
+        addSlowPathGenerator(
+            slowPathCall(
+                slowCases, this, operationGetByValObjectInt,
+                resultRegs, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)),
+                baseReg, propertyReg));
+
+        jsValueResult(resultRegs, node);
+        break;
+    }
+    case Array::String:
+        compileGetByValOnString(node, prefix);
+        break;
+    case Array::DirectArguments:
+        compileGetByValOnDirectArguments(node, prefix);
+        break;
+    case Array::ScopedArguments:
+        compileGetByValOnScopedArguments(node, prefix);
+        break;
+    default: {
+        TypedArrayType type = node->arrayMode().typedArrayType();
+        if (isInt(type))
+            compileGetByValOnIntTypedArray(node, type, prefix);
+        else
+            compileGetByValOnFloatTypedArray(node, type, prefix);
+    } }
+}
+
 void SpeculativeJIT::compile(Node* node)
 {
@@ -2234 +2563 @@
     case StringCharAt: {
         // Relies on StringCharAt node having same basic layout as GetByVal
-        compileGetByValOnString(node);
+        JSValueRegsTemporary result;
+        compileGetByValOnString(node, scopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>([&] (DataFormat preferredFormat) {
+            result = JSValueRegsTemporary(this);
+            ASSERT(preferredFormat & DataFormatJS);
+            return std::make_pair(result.regs(), preferredFormat);
+        }));
         break;
     }
@@ -2266 +2600 @@
 
     case GetByVal: {
-        switch (node->arrayMode().type()) {
-        case Array::SelectUsingPredictions:
-        case Array::ForceExit:
-            RELEASE_ASSERT_NOT_REACHED();
-#if COMPILER_QUIRK(CONSIDERS_UNREACHABLE_CODE)
-            terminateSpeculativeExecution(InadequateCoverage, JSValueRegs(), 0);
-#endif
-            break;
-        case Array::Undecided: {
-            SpeculateStrictInt32Operand index(this, m_graph.varArgChild(node, 1));
-            GPRTemporary resultTag(this, Reuse, index);
-            GPRTemporary resultPayload(this);
-
-            GPRReg indexGPR = index.gpr();
-            GPRReg resultTagGPR = resultTag.gpr();
-            GPRReg resultPayloadGPR = resultPayload.gpr();
-
-            speculationCheck(OutOfBounds, JSValueRegs(), node,
-                m_jit.branch32(MacroAssembler::LessThan, indexGPR, MacroAssembler::TrustedImm32(0)));
-
-            use(m_graph.varArgChild(node, 0));
-            index.use();
-
-            m_jit.move(MacroAssembler::TrustedImm32(JSValue::UndefinedTag), resultTagGPR);
-            m_jit.move(MacroAssembler::TrustedImm32(0), resultPayloadGPR);
-            jsValueResult(resultTagGPR, resultPayloadGPR, node, UseChildrenCalledExplicitly);
-            break;
-        }
-        case Array::Generic: {
-            if (m_graph.m_slowGetByVal.contains(node)) {
-                if (m_graph.varArgChild(node, 0).useKind() == ObjectUse) {
-                    if (m_graph.varArgChild(node, 1).useKind() == StringUse) {
-                        compileGetByValForObjectWithString(node);
-                        break;
-                    }
-
-                    if (m_graph.varArgChild(node, 1).useKind() == SymbolUse) {
-                        compileGetByValForObjectWithSymbol(node);
-                        break;
-                    }
-                }
-
-                SpeculateCellOperand base(this, m_graph.varArgChild(node, 0)); // Save a register, speculate cell. We'll probably be right.
-                JSValueOperand property(this, m_graph.varArgChild(node, 1));
-                GPRReg baseGPR = base.gpr();
-                JSValueRegs propertyRegs = property.jsValueRegs();
-                
-                flushRegisters();
-                JSValueRegsFlushedCallResult result(this);
-                JSValueRegs resultRegs = result.regs();
-                callOperation(operationGetByValCell, resultRegs, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseGPR, propertyRegs);
-                m_jit.exceptionCheck();
-                
-                jsValueResult(resultRegs, node);
+        JSValueRegsTemporary jsValueResult;
+        GPRTemporary oneRegResult;
+        compileGetByVal(node, scopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>([&] (DataFormat preferredFormat) {
+            JSValueRegs resultRegs;
+            switch (preferredFormat) {
+            case DataFormatDouble:
+                break;
+            case DataFormatInt32:
+            case DataFormatCell:
+                oneRegResult = GPRTemporary(this);
+                resultRegs = JSValueRegs::payloadOnly(oneRegResult.gpr());
+                break;
+            default: {
+                ASSERT(preferredFormat == DataFormatJS);
+                jsValueResult = JSValueRegsTemporary(this);
+                resultRegs = jsValueResult.regs();
                 break;
             }
-
-            speculate(node, m_graph.varArgChild(node, 0));
-            speculate(node, m_graph.varArgChild(node, 1));
-
-            JSValueOperand base(this, m_graph.varArgChild(node, 0), ManualOperandSpeculation);
-            JSValueOperand property(this, m_graph.varArgChild(node, 1), ManualOperandSpeculation);
-            GPRTemporary resultTag(this, Reuse, property, TagWord);
-            GPRTemporary resultPayload(this, Reuse, property, PayloadWord);
-
-            JSValueRegs baseRegs = base.jsValueRegs();
-            JSValueRegs propertyRegs = property.jsValueRegs();
-            JSValueRegs resultRegs(resultTag.gpr(), resultPayload.gpr());
-
-            CodeOrigin codeOrigin = node->origin.semantic;
-            CallSiteIndex callSite = m_jit.recordCallSiteAndGenerateExceptionHandlingOSRExitIfNeeded(codeOrigin, m_stream->size());
-            RegisterSet usedRegisters = this->usedRegisters();
-
-            JITCompiler::JumpList slowCases;
-            if (!m_state.forNode(m_graph.varArgChild(node, 0)).isType(SpecCell))
-                slowCases.append(m_jit.branchIfNotCell(baseRegs.tagGPR()));
-
-            JITGetByValGenerator gen(
-                m_jit.codeBlock(), JITType::DFGJIT, codeOrigin, callSite, AccessType::GetByVal, usedRegisters,
-                baseRegs, propertyRegs, resultRegs, InvalidGPRReg);
-
-            if (m_state.forNode(m_graph.varArgChild(node, 1)).isType(SpecString))
-                gen.stubInfo()->propertyIsString = true;
-            else if (m_state.forNode(m_graph.varArgChild(node, 1)).isType(SpecInt32Only))
-                gen.stubInfo()->propertyIsInt32 = true;
-            else if (m_state.forNode(m_graph.varArgChild(node, 1)).isType(SpecSymbol))
-                gen.stubInfo()->propertyIsSymbol = true;
-
-            gen.generateFastPath(m_jit);
-            
-            slowCases.append(gen.slowPathJump());
-
-            std::unique_ptr<SlowPathGenerator> slowPath = slowPathCall(
-                slowCases, this, operationGetByValOptimize,
-                resultRegs, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(codeOrigin)), gen.stubInfo(), nullptr, baseRegs, propertyRegs);
-            
-            m_jit.addGetByVal(gen, slowPath.get());
-            addSlowPathGenerator(WTFMove(slowPath));
-
-            jsValueResult(resultRegs, node);
-            break;
-        }
-        case Array::Int32:
-        case Array::Contiguous: {
-            if (node->arrayMode().isInBounds()) {
-                SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
-                StorageOperand storage(this, m_graph.varArgChild(node, 2));
-            
-                GPRReg propertyReg = property.gpr();
-                GPRReg storageReg = storage.gpr();
-            
-                if (!m_compileOkay)
-                    return;
-            
-                speculationCheck(OutOfBounds, JSValueRegs(), 0, m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, Butterfly::offsetOfPublicLength())));
-            
-                GPRTemporary resultPayload(this);
-                if (node->arrayMode().type() == Array::Int32) {
-                    ASSERT(!node->arrayMode().isInBoundsSaneChain());
-                    
-                    speculationCheck(
-                        OutOfBounds, JSValueRegs(), 0,
-                        m_jit.branch32(
-                            MacroAssembler::Equal,
-                            MacroAssembler::BaseIndex(
-                                storageReg, propertyReg, MacroAssembler::TimesEight, TagOffset),
-                            TrustedImm32(JSValue::EmptyValueTag)));
-                    m_jit.load32(
-                        MacroAssembler::BaseIndex(
-                            storageReg, propertyReg, MacroAssembler::TimesEight, PayloadOffset),
-                        resultPayload.gpr());
-                    strictInt32Result(resultPayload.gpr(), node);
-                    break;
-                }
-                
-                GPRTemporary resultTag(this);
-                m_jit.load32(
-                    MacroAssembler::BaseIndex(
-                        storageReg, propertyReg, MacroAssembler::TimesEight, TagOffset),
-                    resultTag.gpr());
-                m_jit.load32(
-                    MacroAssembler::BaseIndex(
-                        storageReg, propertyReg, MacroAssembler::TimesEight, PayloadOffset),
-                    resultPayload.gpr());
-                if (node->arrayMode().isInBoundsSaneChain()) {
-                    JITCompiler::Jump notHole = m_jit.branchIfNotEmpty(resultTag.gpr());
-                    m_jit.move(TrustedImm32(JSValue::UndefinedTag), resultTag.gpr());
-                    m_jit.move(TrustedImm32(0), resultPayload.gpr());
-                    notHole.link(&m_jit);
-                } else {
-                    speculationCheck(
-                        LoadFromHole, JSValueRegs(), 0,
-                        m_jit.branchIfEmpty(resultTag.gpr()));
-                }
-                jsValueResult(resultTag.gpr(), resultPayload.gpr(), node);
-                break;
-            }
-
-            SpeculateCellOperand base(this, m_graph.varArgChild(node, 0));
-            SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
-            StorageOperand storage(this, m_graph.varArgChild(node, 2));
-            
-            GPRReg baseReg = base.gpr();
-            GPRReg propertyReg = property.gpr();
-            GPRReg storageReg = storage.gpr();
-            
-            if (!m_compileOkay)
-                return;
-            
-            GPRTemporary resultTag(this);
-            GPRTemporary resultPayload(this);
-            GPRReg resultTagReg = resultTag.gpr();
-            GPRReg resultPayloadReg = resultPayload.gpr();
-            
-            MacroAssembler::JumpList slowCases;
-
-            slowCases.append(m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, Butterfly::offsetOfPublicLength())));
-            
-            m_jit.load32(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.tag)), resultTagReg);
-            m_jit.load32(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.payload)), resultPayloadReg);
-            slowCases.append(m_jit.branchIfEmpty(resultTagReg));
-            
-            addSlowPathGenerator(
-                slowPathCall(
-                    slowCases, this, operationGetByValObjectInt,
-                    JSValueRegs(resultTagReg, resultPayloadReg), TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseReg, propertyReg));
-            
-            jsValueResult(resultTagReg, resultPayloadReg, node);
-            break;
-        }
-        case Array::Double: {
-            if (node->arrayMode().isInBounds()) {
-                SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
-                StorageOperand storage(this, m_graph.varArgChild(node, 2));
-            
-                GPRReg propertyReg = property.gpr();
-                GPRReg storageReg = storage.gpr();
-            
-                if (!m_compileOkay)
-                    return;
-            
-                speculationCheck(OutOfBounds, JSValueRegs(), 0, m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, Butterfly::offsetOfPublicLength())));
-            
-                FPRTemporary result(this);
-                m_jit.loadDouble(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight), result.fpr());
-                if (!node->arrayMode().isInBoundsSaneChain())
-                    speculationCheck(LoadFromHole, JSValueRegs(), 0, m_jit.branchIfNaN(result.fpr()));
-                doubleResult(result.fpr(), node);
-                break;
-            }
-
-            SpeculateCellOperand base(this, m_graph.varArgChild(node, 0));
-            SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
-            StorageOperand storage(this, m_graph.varArgChild(node, 2));
-            
-            GPRReg baseReg = base.gpr();
-            GPRReg propertyReg = property.gpr();
-            GPRReg storageReg = storage.gpr();
-            
-            if (!m_compileOkay)
-                return;
-            
-            GPRTemporary resultTag(this);
-            GPRTemporary resultPayload(this);
-            FPRTemporary temp(this);
-            GPRReg resultTagReg = resultTag.gpr();
-            GPRReg resultPayloadReg = resultPayload.gpr();
-            FPRReg tempReg = temp.fpr();
-            
-            MacroAssembler::JumpList slowCases;
-            
-            slowCases.append(m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, Butterfly::offsetOfPublicLength())));
-            
-            m_jit.loadDouble(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight), tempReg);
-            slowCases.append(m_jit.branchIfNaN(tempReg));
-            boxDouble(tempReg, resultTagReg, resultPayloadReg);
-
-            addSlowPathGenerator(
-                slowPathCall(
-                    slowCases, this, operationGetByValObjectInt,
-                    JSValueRegs(resultTagReg, resultPayloadReg), TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseReg, propertyReg));
-            
-            jsValueResult(resultTagReg, resultPayloadReg, node);
-            break;
-        }
-        case Array::ArrayStorage:
-        case Array::SlowPutArrayStorage: {
-            if (node->arrayMode().isInBounds()) {
-                SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
-                StorageOperand storage(this, m_graph.varArgChild(node, 2));
-                GPRReg propertyReg = property.gpr();
-                GPRReg storageReg = storage.gpr();
-        
-                if (!m_compileOkay)
-                    return;
-
-                speculationCheck(OutOfBounds, JSValueRegs(), 0, m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, ArrayStorage::vectorLengthOffset())));
-
-                GPRTemporary resultTag(this);
-                GPRTemporary resultPayload(this);
-
-                m_jit.load32(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight, ArrayStorage::vectorOffset() + OBJECT_OFFSETOF(JSValue, u.asBits.tag)), resultTag.gpr());
-                speculationCheck(LoadFromHole, JSValueRegs(), 0, m_jit.branchIfEmpty(resultTag.gpr()));
-                m_jit.load32(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight, ArrayStorage::vectorOffset() + OBJECT_OFFSETOF(JSValue, u.asBits.payload)), resultPayload.gpr());
-            
-                jsValueResult(resultTag.gpr(), resultPayload.gpr(), node);
-                break;
-            }
-
-            SpeculateCellOperand base(this, m_graph.varArgChild(node, 0));
-            SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
-            StorageOperand storage(this, m_graph.varArgChild(node, 2));
-            GPRReg propertyReg = property.gpr();
-            GPRReg storageReg = storage.gpr();
-            GPRReg baseReg = base.gpr();
-
-            if (!m_compileOkay)
-                return;
-
-            GPRTemporary resultTag(this);
-            GPRTemporary resultPayload(this);
-            GPRReg resultTagReg = resultTag.gpr();
-            GPRReg resultPayloadReg = resultPayload.gpr();
-
-            JITCompiler::Jump outOfBounds = m_jit.branch32(
-                MacroAssembler::AboveOrEqual, propertyReg,
-                MacroAssembler::Address(storageReg, ArrayStorage::vectorLengthOffset()));
-
-            m_jit.load32(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight, ArrayStorage::vectorOffset() + OBJECT_OFFSETOF(JSValue, u.asBits.tag)), resultTagReg);
-            JITCompiler::Jump hole = m_jit.branchIfEmpty(resultTag.gpr());
-            m_jit.load32(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight, ArrayStorage::vectorOffset() + OBJECT_OFFSETOF(JSValue, u.asBits.payload)), resultPayloadReg);
-            
-            JITCompiler::JumpList slowCases;
-            slowCases.append(outOfBounds);
-            slowCases.append(hole);
-            addSlowPathGenerator(
-                slowPathCall(
-                    slowCases, this, operationGetByValObjectInt,
-                    JSValueRegs(resultTagReg, resultPayloadReg), TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)),
-                    baseReg, propertyReg));
-
-            jsValueResult(resultTagReg, resultPayloadReg, node);
-            break;
-        }
-        case Array::String:
-            compileGetByValOnString(node);
-            break;
-        case Array::DirectArguments:
-            compileGetByValOnDirectArguments(node);
-            break;
-        case Array::ScopedArguments:
-            compileGetByValOnScopedArguments(node);
-            break;
-        default: {
-            TypedArrayType type = node->arrayMode().typedArrayType();
-            if (isInt(type))
-                compileGetByValOnIntTypedArray(node, type);
-            else
-                compileGetByValOnFloatTypedArray(node, type);
-        } }
+            };
+            return std::make_pair(resultRegs, preferredFormat);
+        }));
         break;
     }
@@ -4113 +4148 @@
     }
 
-    case GetEnumerableLength: {
-        compileGetEnumerableLength(node);
-        break;
-    }
-    case HasEnumerableStructureProperty: {
-        compileHasEnumerableStructureProperty(node);
-        break;
-    }
-    case HasEnumerableProperty: {
-        compileHasEnumerableProperty(node);
-        break;
-    }
-    case HasOwnStructureProperty: {
-        compileHasOwnStructureProperty(node);
-        break;
-    }
-    case InStructureProperty: {
-        compileInStructureProperty(node);
-        break;
-    }
     case HasIndexedProperty: {
-        compileHasIndexedProperty(node, operationHasIndexedProperty);
-        break;
-    }
-    case HasEnumerableIndexedProperty: {
-        compileHasIndexedProperty(node, operationHasEnumerableIndexedProperty);
-        break;
-    }
-    case GetDirectPname: {
-        compileGetDirectPname(node);
-        break;
-    }
+        SpeculateStrictInt32Operand index(this, m_graph.varArgChild(node, 1));
+        GPRTemporary result(this, Reuse, index);
+
+        compileHasIndexedProperty(node, operationHasIndexedProperty, scopedLambda<std::tuple<GPRReg, GPRReg>()>([&] { return std::make_pair(index.gpr(), result.gpr()); }));
+        unblessedBooleanResult(result.gpr(), node);
+        break;
+    }
+
     case GetPropertyEnumerator: {
         compileGetPropertyEnumerator(node);
         break;
     }
-    case GetEnumeratorStructurePname:
-    case GetEnumeratorGenericPname: {
-        compileGetEnumeratorPname(node);
-        break;
-    }
-    case ToIndexString: {
-        compileToIndexString(node);
-        break;
-    }
+
+    case EnumeratorNextUpdateIndexAndMode: {
+        compileEnumeratorNextUpdateIndexAndMode(node);
+        break;
+    }
+
+    case EnumeratorNextExtractMode: {
+        compileEnumeratorNextExtractMode(node);
+        break;
+    }
+
+    case EnumeratorNextExtractIndex: {
+        compileEnumeratorNextExtractIndex(node);
+        break;
+    }
+
+    case EnumeratorNextUpdatePropertyName: {
+        compileEnumeratorNextUpdatePropertyName(node);
+        break;
+    }
+
+    case EnumeratorGetByVal: {
+        compileEnumeratorGetByVal(node);
+        break;
+    }
+
+    case EnumeratorInByVal: {
+        compileEnumeratorInByVal(node);
+        break;
+    }
+
+    case EnumeratorHasOwnProperty: {
+        compileEnumeratorHasOwnProperty(node);
+        break;
+    }
+
     case ProfileType: {
         compileProfileType(node);

trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp

@@ -2247 +2247 @@
 }
 
+void SpeculativeJIT::compileGetByVal(Node* node, const ScopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>& prefix)
+{
+    switch (node->arrayMode().type()) {
+    case Array::AnyTypedArray:
+    case Array::ForceExit:
+    case Array::SelectUsingArguments:
+    case Array::SelectUsingPredictions:
+    case Array::Unprofiled:
+        DFG_CRASH(m_jit.graph(), node, "Bad array mode type");
+        break;
+    case Array::Undecided: {
+        SpeculateStrictInt32Operand index(this, m_graph.varArgChild(node, 1));
+        GPRReg indexGPR = index.gpr();
+
+        JSValueRegs resultRegs;
+        std::tie(resultRegs, std::ignore) = prefix(DataFormatJS);
+
+        speculationCheck(OutOfBounds, JSValueRegs(), node,
+            m_jit.branch32(MacroAssembler::LessThan, indexGPR, MacroAssembler::TrustedImm32(0)));
+
+        m_jit.move(MacroAssembler::TrustedImm64(JSValue::ValueUndefined), resultRegs.gpr());
+
+        jsValueResult(resultRegs, node);
+        break;
+    }
+    case Array::BigInt64Array:
+    case Array::BigUint64Array:
+    case Array::Generic: {
+        if (m_graph.m_slowGetByVal.contains(node)) {
+            if (m_graph.varArgChild(node, 0).useKind() == ObjectUse) {
+                if (m_graph.varArgChild(node, 1).useKind() == StringUse) {
+                    compileGetByValForObjectWithString(node, prefix);
+                    return;
+                }
+
+                if (m_graph.varArgChild(node, 1).useKind() == SymbolUse) {
+                    compileGetByValForObjectWithSymbol(node, prefix);
+                    return;
+                }
+            }
+
+            JSValueOperand base(this, m_graph.varArgChild(node, 0));
+            JSValueOperand property(this, m_graph.varArgChild(node, 1));
+            GPRReg baseGPR = base.gpr();
+            GPRReg propertyGPR = property.gpr();
+
+            JSValueRegs resultRegs;
+            std::tie(resultRegs, std::ignore) = prefix(DataFormatJS);
+
+            flushRegisters();
+            callOperation(operationGetByVal, resultRegs, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseGPR, propertyGPR);
+            m_jit.exceptionCheck();
+
+            jsValueResult(resultRegs, node);
+            return;
+        }
+
+        JSValueOperand base(this, m_graph.varArgChild(node, 0), ManualOperandSpeculation);
+        JSValueOperand property(this, m_graph.varArgChild(node, 1), ManualOperandSpeculation);
+        GPRReg baseGPR = base.gpr();
+        GPRReg propertyGPR = property.gpr();
+
+        GPRTemporary stubInfo;
+        GPRReg stubInfoGPR = InvalidGPRReg;
+        if (JITCode::useDataIC(JITType::DFGJIT)) {
+            stubInfo = GPRTemporary(this);
+            stubInfoGPR = stubInfo.gpr();
+        }
+
+        speculate(node, m_graph.varArgChild(node, 0));
+        speculate(node, m_graph.varArgChild(node, 1));
+
+        JSValueRegs resultRegs;
+        std::tie(resultRegs, std::ignore) = prefix(DataFormatJS);
+        GPRReg resultGPR = resultRegs.gpr();
+
+        CodeOrigin codeOrigin = node->origin.semantic;
+        CallSiteIndex callSite = m_jit.recordCallSiteAndGenerateExceptionHandlingOSRExitIfNeeded(codeOrigin, m_stream->size());
+        RegisterSet usedRegisters = this->usedRegisters();
+
+        JITCompiler::JumpList slowCases;
+        if (!m_state.forNode(m_graph.varArgChild(node, 0)).isType(SpecCell))
+            slowCases.append(m_jit.branchIfNotCell(baseGPR));
+
+        JITGetByValGenerator gen(
+            m_jit.codeBlock(), JITType::DFGJIT, codeOrigin, callSite, AccessType::GetByVal, usedRegisters,
+            JSValueRegs(baseGPR), JSValueRegs(propertyGPR), JSValueRegs(resultGPR), stubInfoGPR);
+
+        if (m_state.forNode(m_graph.varArgChild(node, 1)).isType(SpecString))
+            gen.stubInfo()->propertyIsString = true;
+        else if (m_state.forNode(m_graph.varArgChild(node, 1)).isType(SpecInt32Only))
+            gen.stubInfo()->propertyIsInt32 = true;
+        else if (m_state.forNode(m_graph.varArgChild(node, 1)).isType(SpecSymbol))
+            gen.stubInfo()->propertyIsSymbol = true;
+
+        gen.generateFastPath(m_jit);
+
+        if (!JITCode::useDataIC(JITType::DFGJIT))
+            slowCases.append(gen.slowPathJump());
+
+        std::unique_ptr<SlowPathGenerator> slowPath;
+        if (JITCode::useDataIC(JITType::DFGJIT)) {
+            slowPath = slowPathICCall(
+                slowCases, this, gen.stubInfo(), stubInfoGPR, CCallHelpers::Address(stubInfoGPR, StructureStubInfo::offsetOfSlowOperation()), operationGetByValOptimize,
+                resultGPR, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(codeOrigin)), stubInfoGPR, nullptr, baseGPR, propertyGPR);
+        } else {
+            slowPath = slowPathCall(
+                slowCases, this, operationGetByValOptimize,
+                resultGPR, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(codeOrigin)), gen.stubInfo(), nullptr, baseGPR, propertyGPR);
+        }
+
+        m_jit.addGetByVal(gen, slowPath.get());
+        addSlowPathGenerator(WTFMove(slowPath));
+
+        jsValueResult(resultGPR, node);
+        break;
+    }
+    case Array::Int32:
+    case Array::Contiguous: {
+        if (node->arrayMode().isInBounds()) {
+            SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
+            StorageOperand storage(this, m_graph.varArgChild(node, 2));
+
+            GPRReg propertyReg = property.gpr();
+            GPRReg storageReg = storage.gpr();
+
+            if (!m_compileOkay)
+                return;
+
+            JSValueRegs resultRegs;
+            DataFormat format;
+            std::tie(resultRegs, format) = prefix(node->arrayMode().type() == Array::Int32 ? DataFormatJSInt32 : DataFormatJS);
+            GPRReg result = resultRegs.gpr();
+
+            speculationCheck(OutOfBounds, JSValueRegs(), nullptr, m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, Butterfly::offsetOfPublicLength())));
+
+            m_jit.load64(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight), result);
+            if (node->arrayMode().isInBoundsSaneChain()) {
+                ASSERT(node->arrayMode().type() == Array::Contiguous);
+                JITCompiler::Jump notHole = m_jit.branchIfNotEmpty(result);
+                m_jit.move(TrustedImm64(JSValue::encode(jsUndefined())), result);
+                notHole.link(&m_jit);
+            } else {
+                speculationCheck(
+                    LoadFromHole, JSValueRegs(), nullptr,
+                    m_jit.branchIfEmpty(result));
+            }
+            jsValueResult(result, node, format);
+            break;
+        }
+
+        SpeculateCellOperand base(this, m_graph.varArgChild(node, 0));
+        SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
+        StorageOperand storage(this, m_graph.varArgChild(node, 2));
+
+        GPRReg baseReg = base.gpr();
+        GPRReg propertyReg = property.gpr();
+        GPRReg storageReg = storage.gpr();
+
+        if (!m_compileOkay)
+            return;
+
+        JSValueRegs resultRegs;
+        std::tie(resultRegs, std::ignore) = prefix(DataFormatJS);
+        GPRReg resultReg = resultRegs.gpr();
+
+        MacroAssembler::JumpList slowCases;
+
+        slowCases.append(m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, Butterfly::offsetOfPublicLength())));
+
+        m_jit.load64(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight), resultReg);
+
+        if (node->arrayMode().isOutOfBoundsSaneChain()) {
+            auto done = m_jit.branchIfNotEmpty(resultReg);
+            slowCases.link(&m_jit);
+            speculationCheck(NegativeIndex, JSValueRegs(), nullptr, m_jit.branch32(MacroAssembler::LessThan, propertyReg, CCallHelpers::TrustedImm32(0)));
+            m_jit.move(CCallHelpers::TrustedImm64(JSValue::encode(jsUndefined())), resultReg);
+            done.link(&m_jit);
+        } else {
+            slowCases.append(m_jit.branchIfEmpty(resultReg));
+            addSlowPathGenerator(
+                slowPathCall(
+                    slowCases, this, operationGetByValObjectInt,
+                    resultReg, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseReg, propertyReg));
+        }
+
+
+        jsValueResult(resultReg, node);
+        break;
+    }
+
+    case Array::Double: {
+        if (node->arrayMode().isInBounds()) {
+            SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
+            StorageOperand storage(this, m_graph.varArgChild(node, 2));
+
+            GPRReg propertyReg = property.gpr();
+            GPRReg storageReg = storage.gpr();
+
+            if (!m_compileOkay)
+                return;
+
+            FPRTemporary result(this);
+            FPRReg resultReg = result.fpr();
+
+            JSValueRegs resultRegs;
+            DataFormat format;
+            std::tie(resultRegs, format) = prefix(DataFormatDouble);
+
+            speculationCheck(OutOfBounds, JSValueRegs(), nullptr, m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, Butterfly::offsetOfPublicLength())));
+
+            m_jit.loadDouble(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight), resultReg);
+            if (!node->arrayMode().isInBoundsSaneChain())
+                speculationCheck(LoadFromHole, JSValueRegs(), nullptr, m_jit.branchIfNaN(resultReg));
+            if (format == DataFormatJS) {
+                boxDouble(resultReg, resultRegs);
+                jsValueResult(resultRegs, node);
+            } else {
+                ASSERT(format == DataFormatDouble && !resultRegs);
+                doubleResult(resultReg, node);
+            }
+            break;
+        }
+
+        bool resultIsUnboxed = node->arrayMode().isOutOfBoundsSaneChain() && !(node->flags() & NodeBytecodeUsesAsOther);
+
+        SpeculateCellOperand base(this, m_graph.varArgChild(node, 0));
+        SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
+        StorageOperand storage(this, m_graph.varArgChild(node, 2));
+
+        GPRReg baseReg = base.gpr();
+        GPRReg propertyReg = property.gpr();
+        GPRReg storageReg = storage.gpr();
+
+        if (!m_compileOkay)
+            return;
+
+        FPRTemporary temp(this);
+        FPRReg tempReg = temp.fpr();
+
+        JSValueRegs resultRegs;
+        DataFormat format;
+        std::tie(resultRegs, format) = prefix(resultIsUnboxed ? DataFormatDouble : DataFormatJS);
+
+        MacroAssembler::JumpList slowCases;
+
+        slowCases.append(m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, Butterfly::offsetOfPublicLength())));
+
+        m_jit.loadDouble(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight), tempReg);
+        if (node->arrayMode().isOutOfBoundsSaneChain()) {
+            if (format == DataFormatDouble) {
+                auto done = m_jit.jump();
+                slowCases.link(&m_jit);
+                speculationCheck(NegativeIndex, JSValueRegs(), nullptr, m_jit.branch32(MacroAssembler::LessThan, propertyReg, CCallHelpers::TrustedImm32(0)));
+                static const double NaN = PNaN;
+                m_jit.loadDouble(TrustedImmPtr(&NaN), tempReg);
+                done.link(&m_jit);
+                ASSERT(!resultRegs);
+                doubleResult(tempReg, node);
+            } else {
+                slowCases.append(m_jit.branchIfNaN(tempReg));
+                boxDouble(tempReg, resultRegs.gpr());
+                auto done = m_jit.jump();
+                slowCases.link(&m_jit);
+                speculationCheck(NegativeIndex, JSValueRegs(), nullptr, m_jit.branch32(MacroAssembler::LessThan, propertyReg, CCallHelpers::TrustedImm32(0)));
+                m_jit.move(CCallHelpers::TrustedImm64(JSValue::encode(jsUndefined())), resultRegs.gpr());
+                done.link(&m_jit);
+                jsValueResult(resultRegs.gpr(), node);
+            }
+        } else {
+            slowCases.append(m_jit.branchIfNaN(tempReg));
+            boxDouble(tempReg, resultRegs.gpr());
+            addSlowPathGenerator(
+                slowPathCall(
+                    slowCases, this, operationGetByValObjectInt,
+                    resultRegs.gpr(), TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseReg, propertyReg));
+            jsValueResult(resultRegs.gpr(), node);
+        }
+
+        break;
+    }
+
+    case Array::ArrayStorage:
+    case Array::SlowPutArrayStorage: {
+        if (node->arrayMode().isInBounds()) {
+            SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
+            StorageOperand storage(this, m_graph.varArgChild(node, 2));
+
+            GPRReg propertyReg = property.gpr();
+            GPRReg storageReg = storage.gpr();
+
+            if (!m_compileOkay)
+                return;
+
+            JSValueRegs resultRegs;
+            std::tie(resultRegs, std::ignore) = prefix(DataFormatJS);
+            GPRReg resultGPR = resultRegs.gpr();
+
+            speculationCheck(OutOfBounds, JSValueRegs(), nullptr, m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, ArrayStorage::vectorLengthOffset())));
+
+
+            m_jit.load64(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight, ArrayStorage::vectorOffset()), resultGPR);
+            speculationCheck(LoadFromHole, JSValueRegs(), nullptr, m_jit.branchIfEmpty(resultGPR));
+
+            jsValueResult(resultGPR, node);
+            break;
+        }
+
+        SpeculateCellOperand base(this, m_graph.varArgChild(node, 0));
+        SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
+        StorageOperand storage(this, m_graph.varArgChild(node, 2));
+
+        GPRReg baseReg = base.gpr();
+        GPRReg propertyReg = property.gpr();
+        GPRReg storageReg = storage.gpr();
+
+        if (!m_compileOkay)
+            return;
+
+        JSValueRegs resultRegs;
+        std::tie(resultRegs, std::ignore) = prefix(DataFormatJS);
+        GPRReg resultReg = resultRegs.gpr();
+
+        MacroAssembler::JumpList slowCases;
+
+        slowCases.append(m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, ArrayStorage::vectorLengthOffset())));
+
+        m_jit.load64(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight, ArrayStorage::vectorOffset()), resultReg);
+        slowCases.append(m_jit.branchIfEmpty(resultReg));
+
+        addSlowPathGenerator(
+            slowPathCall(
+                slowCases, this, operationGetByValObjectInt,
+                resultReg, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseReg, propertyReg));
+
+        jsValueResult(resultReg, node);
+        break;
+    }
+    case Array::String:
+        compileGetByValOnString(node, prefix);
+        break;
+    case Array::DirectArguments:
+        compileGetByValOnDirectArguments(node, prefix);
+        break;
+    case Array::ScopedArguments:
+        compileGetByValOnScopedArguments(node, prefix);
+        break;
+    case Array::Int8Array:
+    case Array::Int16Array:
+    case Array::Int32Array:
+    case Array::Uint8Array:
+    case Array::Uint8ClampedArray:
+    case Array::Uint16Array:
+    case Array::Uint32Array:
+    case Array::Float32Array:
+    case Array::Float64Array: {
+        TypedArrayType type = node->arrayMode().typedArrayType();
+        if (isInt(type))
+            compileGetByValOnIntTypedArray(node, type, prefix);
+        else
+            compileGetByValOnFloatTypedArray(node, type, prefix);
+    } }
+}
+
 void SpeculativeJIT::compile(Node* node)
 {
@@ -2714 +3078 @@
     case StringCharAt: {
         // Relies on StringCharAt node having same basic layout as GetByVal
-        compileGetByValOnString(node);
+        JSValueRegsTemporary result;
+        compileGetByValOnString(node, scopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>([&] (DataFormat preferredFormat) {
+            result = JSValueRegsTemporary(this);
+            ASSERT(preferredFormat == DataFormatJS || preferredFormat == DataFormatCell);
+            return std::make_pair(result.regs(), preferredFormat);
+        }));
         break;
     }
@@ -2747 +3116 @@
 
     case GetByVal: {
-        switch (node->arrayMode().type()) {
-        case Array::AnyTypedArray:
-        case Array::ForceExit:
-        case Array::SelectUsingArguments:
-        case Array::SelectUsingPredictions:
-        case Array::Unprofiled:
-            DFG_CRASH(m_jit.graph(), node, "Bad array mode type");
-            break;
-        case Array::Undecided: {
-            SpeculateStrictInt32Operand index(this, m_graph.varArgChild(node, 1));
-            GPRTemporary result(this, Reuse, index);
-            GPRReg indexGPR = index.gpr();
-            GPRReg resultGPR = result.gpr();
-
-            speculationCheck(OutOfBounds, JSValueRegs(), node,
-                m_jit.branch32(MacroAssembler::LessThan, indexGPR, MacroAssembler::TrustedImm32(0)));
-
-            use(m_graph.varArgChild(node, 0));
-            index.use();
-
-            m_jit.move(MacroAssembler::TrustedImm64(JSValue::ValueUndefined), resultGPR);
-            jsValueResult(resultGPR, node, UseChildrenCalledExplicitly);
-            break;
-        }
-        case Array::BigInt64Array:
-        case Array::BigUint64Array:
-        case Array::Generic: {
-            if (m_graph.m_slowGetByVal.contains(node)) {
-                if (m_graph.varArgChild(node, 0).useKind() == ObjectUse) {
-                    if (m_graph.varArgChild(node, 1).useKind() == StringUse) {
-                        compileGetByValForObjectWithString(node);
-                        break;
-                    }
-
-                    if (m_graph.varArgChild(node, 1).useKind() == SymbolUse) {
-                        compileGetByValForObjectWithSymbol(node);
-                        break;
-                    }
-                }
-
-                JSValueOperand base(this, m_graph.varArgChild(node, 0));
-                JSValueOperand property(this, m_graph.varArgChild(node, 1));
-                GPRReg baseGPR = base.gpr();
-                GPRReg propertyGPR = property.gpr();
-                
-                flushRegisters();
-                GPRFlushedCallResult result(this);
-                callOperation(operationGetByVal, result.gpr(), TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseGPR, propertyGPR);
-                m_jit.exceptionCheck();
-                
-                jsValueResult(result.gpr(), node);
+        JSValueRegsTemporary result;
+        compileGetByVal(node, scopedLambda<std::tuple<JSValueRegs, DataFormat>(DataFormat preferredFormat)>([&] (DataFormat preferredFormat) {
+            JSValueRegs resultRegs;
+            switch (preferredFormat) {
+            case DataFormatDouble:
+                break;
+            default: {
+                result = JSValueRegsTemporary(this);
+                resultRegs = result.regs();
                 break;
             }
-
-            std::optional<GPRTemporary> stubInfo;
-            JSValueOperand base(this, m_graph.varArgChild(node, 0), ManualOperandSpeculation);
-            JSValueOperand property(this, m_graph.varArgChild(node, 1), ManualOperandSpeculation);
-            GPRTemporary result(this, Reuse, property);
-
-            GPRReg stubInfoGPR = InvalidGPRReg;
-            if (JITCode::useDataIC(JITType::DFGJIT)) {
-                stubInfo.emplace(this);
-                stubInfoGPR = stubInfo->gpr();
-            }
-            GPRReg baseGPR = base.gpr();
-            GPRReg propertyGPR = property.gpr();
-            GPRReg resultGPR = result.gpr();
-
-            speculate(node, m_graph.varArgChild(node, 0));
-            speculate(node, m_graph.varArgChild(node, 1));
-
-            CodeOrigin codeOrigin = node->origin.semantic;
-            CallSiteIndex callSite = m_jit.recordCallSiteAndGenerateExceptionHandlingOSRExitIfNeeded(codeOrigin, m_stream->size());
-            RegisterSet usedRegisters = this->usedRegisters();
-
-            JITCompiler::JumpList slowCases;
-            if (!m_state.forNode(m_graph.varArgChild(node, 0)).isType(SpecCell))
-                slowCases.append(m_jit.branchIfNotCell(baseGPR));
-
-            JITGetByValGenerator gen(
-                m_jit.codeBlock(), JITType::DFGJIT, codeOrigin, callSite, AccessType::GetByVal, usedRegisters,
-                JSValueRegs(baseGPR), JSValueRegs(propertyGPR), JSValueRegs(resultGPR), stubInfoGPR);
-
-            if (m_state.forNode(m_graph.varArgChild(node, 1)).isType(SpecString))
-                gen.stubInfo()->propertyIsString = true;
-            else if (m_state.forNode(m_graph.varArgChild(node, 1)).isType(SpecInt32Only))
-                gen.stubInfo()->propertyIsInt32 = true;
-            else if (m_state.forNode(m_graph.varArgChild(node, 1)).isType(SpecSymbol))
-                gen.stubInfo()->propertyIsSymbol = true;
-
-            gen.generateFastPath(m_jit);
-            
-            if (!JITCode::useDataIC(JITType::DFGJIT))
-                slowCases.append(gen.slowPathJump());
-
-            std::unique_ptr<SlowPathGenerator> slowPath;
-            if (JITCode::useDataIC(JITType::DFGJIT)) {
-                slowPath = slowPathICCall(
-                    slowCases, this, gen.stubInfo(), stubInfoGPR, CCallHelpers::Address(stubInfoGPR, StructureStubInfo::offsetOfSlowOperation()), operationGetByValOptimize,
-                    resultGPR, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(codeOrigin)), stubInfoGPR, nullptr, baseGPR, propertyGPR);
-            } else {
-                slowPath = slowPathCall(
-                    slowCases, this, operationGetByValOptimize,
-                    resultGPR, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(codeOrigin)), gen.stubInfo(), nullptr, baseGPR, propertyGPR);
-            }
-            
-            m_jit.addGetByVal(gen, slowPath.get());
-            addSlowPathGenerator(WTFMove(slowPath));
-
-            jsValueResult(resultGPR, node);
-            break;
-        }
-        case Array::Int32:
-        case Array::Contiguous: {
-            if (node->arrayMode().isInBounds()) {
-                SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
-                StorageOperand storage(this, m_graph.varArgChild(node, 2));
-
-                GPRReg propertyReg = property.gpr();
-                GPRReg storageReg = storage.gpr();
-
-                if (!m_compileOkay)
-                    return;
-                
-                speculationCheck(OutOfBounds, JSValueRegs(), nullptr, m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, Butterfly::offsetOfPublicLength())));
-                
-                GPRTemporary result(this);
-
-                m_jit.load64(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight), result.gpr());
-                if (node->arrayMode().isInBoundsSaneChain()) {
-                    ASSERT(node->arrayMode().type() == Array::Contiguous);
-                    JITCompiler::Jump notHole = m_jit.branchIfNotEmpty(result.gpr());
-                    m_jit.move(TrustedImm64(JSValue::encode(jsUndefined())), result.gpr());
-                    notHole.link(&m_jit);
-                } else {
-                    speculationCheck(
-                        LoadFromHole, JSValueRegs(), nullptr,
-                        m_jit.branchIfEmpty(result.gpr()));
-                }
-                jsValueResult(result.gpr(), node, node->arrayMode().type() == Array::Int32 ? DataFormatJSInt32 : DataFormatJS);
-                break;
-            }
-            
-            SpeculateCellOperand base(this, m_graph.varArgChild(node, 0));
-            SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
-            StorageOperand storage(this, m_graph.varArgChild(node, 2));
-            
-            GPRReg baseReg = base.gpr();
-            GPRReg propertyReg = property.gpr();
-            GPRReg storageReg = storage.gpr();
-            
-            if (!m_compileOkay)
-                return;
-            
-            GPRTemporary result(this);
-            GPRReg resultReg = result.gpr();
-            
-            MacroAssembler::JumpList slowCases;
-            
-            slowCases.append(m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, Butterfly::offsetOfPublicLength())));
-
-            m_jit.load64(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight), resultReg);
-
-            if (node->arrayMode().isOutOfBoundsSaneChain()) {
-                auto done = m_jit.branchIfNotEmpty(resultReg);
-                slowCases.link(&m_jit);
-                speculationCheck(NegativeIndex, JSValueRegs(), nullptr, m_jit.branch32(MacroAssembler::LessThan, propertyReg, CCallHelpers::TrustedImm32(0)));
-                m_jit.move(CCallHelpers::TrustedImm64(JSValue::encode(jsUndefined())), resultReg);
-                done.link(&m_jit);
-            } else {
-                slowCases.append(m_jit.branchIfEmpty(resultReg));
-                addSlowPathGenerator(
-                    slowPathCall(
-                        slowCases, this, operationGetByValObjectInt,
-                        result.gpr(), TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseReg, propertyReg));
-            }
-            
-            
-            jsValueResult(resultReg, node);
-            break;
-        }
-
-        case Array::Double: {
-            if (node->arrayMode().isInBounds()) {
-                SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
-                StorageOperand storage(this, m_graph.varArgChild(node, 2));
-
-                GPRReg propertyReg = property.gpr();
-                GPRReg storageReg = storage.gpr();
-
-                if (!m_compileOkay)
-                    return;
-
-                FPRTemporary result(this);
-                FPRReg resultReg = result.fpr();
-
-                speculationCheck(OutOfBounds, JSValueRegs(), nullptr, m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, Butterfly::offsetOfPublicLength())));
-
-                m_jit.loadDouble(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight), resultReg);
-                if (!node->arrayMode().isInBoundsSaneChain())
-                    speculationCheck(LoadFromHole, JSValueRegs(), nullptr, m_jit.branchIfNaN(resultReg));
-                doubleResult(resultReg, node);
-                break;
-            }
-
-            bool resultIsUnboxed = node->arrayMode().isOutOfBoundsSaneChain() && !(node->flags() & NodeBytecodeUsesAsOther);
-
-            SpeculateCellOperand base(this, m_graph.varArgChild(node, 0));
-            SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
-            StorageOperand storage(this, m_graph.varArgChild(node, 2));
-            
-            GPRReg baseReg = base.gpr();
-            GPRReg propertyReg = property.gpr();
-            GPRReg storageReg = storage.gpr();
-            
-            if (!m_compileOkay)
-                return;
-            
-            std::optional<GPRTemporary> result;
-            std::optional<GPRReg> resultReg;
-            if (!resultIsUnboxed) {
-                result.emplace(this);
-                resultReg = result->gpr();
-            }
-            FPRTemporary temp(this);
-            FPRReg tempReg = temp.fpr();
-            
-            MacroAssembler::JumpList slowCases;
-            
-            slowCases.append(m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, Butterfly::offsetOfPublicLength())));
-
-            m_jit.loadDouble(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight), tempReg);
-            if (node->arrayMode().isOutOfBoundsSaneChain()) {
-                if (resultIsUnboxed) {
-                    auto done = m_jit.jump();
-                    slowCases.link(&m_jit);
-                    speculationCheck(NegativeIndex, JSValueRegs(), nullptr, m_jit.branch32(MacroAssembler::LessThan, propertyReg, CCallHelpers::TrustedImm32(0)));
-                    static const double NaN = PNaN;
-                    m_jit.loadDouble(TrustedImmPtr(&NaN), tempReg);
-                    done.link(&m_jit);
-                    doubleResult(tempReg, node);
-                } else {
-                    slowCases.append(m_jit.branchIfNaN(tempReg));
-                    boxDouble(tempReg, *resultReg);
-                    auto done = m_jit.jump();
-                    slowCases.link(&m_jit);
-                    speculationCheck(NegativeIndex, JSValueRegs(), nullptr, m_jit.branch32(MacroAssembler::LessThan, propertyReg, CCallHelpers::TrustedImm32(0)));
-                    m_jit.move(CCallHelpers::TrustedImm64(JSValue::encode(jsUndefined())), *resultReg);
-                    done.link(&m_jit);
-                    jsValueResult(*resultReg, node);
-                }
-            } else {
-                slowCases.append(m_jit.branchIfNaN(tempReg));
-                boxDouble(tempReg, *resultReg);
-                addSlowPathGenerator(
-                    slowPathCall(
-                        slowCases, this, operationGetByValObjectInt,
-                        *resultReg, TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseReg, propertyReg));
-                jsValueResult(*resultReg, node);
-            }
-
-            break;
-        }
-
-        case Array::ArrayStorage:
-        case Array::SlowPutArrayStorage: {
-            if (node->arrayMode().isInBounds()) {
-                SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
-                StorageOperand storage(this, m_graph.varArgChild(node, 2));
-            
-                GPRReg propertyReg = property.gpr();
-                GPRReg storageReg = storage.gpr();
-            
-                if (!m_compileOkay)
-                    return;
-            
-                speculationCheck(OutOfBounds, JSValueRegs(), nullptr, m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, ArrayStorage::vectorLengthOffset())));
-            
-                GPRTemporary result(this);
-                m_jit.load64(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight, ArrayStorage::vectorOffset()), result.gpr());
-                speculationCheck(LoadFromHole, JSValueRegs(), nullptr, m_jit.branchIfEmpty(result.gpr()));
-            
-                jsValueResult(result.gpr(), node);
-                break;
-            }
-
-            SpeculateCellOperand base(this, m_graph.varArgChild(node, 0));
-            SpeculateStrictInt32Operand property(this, m_graph.varArgChild(node, 1));
-            StorageOperand storage(this, m_graph.varArgChild(node, 2));
-            
-            GPRReg baseReg = base.gpr();
-            GPRReg propertyReg = property.gpr();
-            GPRReg storageReg = storage.gpr();
-            
-            if (!m_compileOkay)
-                return;
-            
-            GPRTemporary result(this);
-            GPRReg resultReg = result.gpr();
-            
-            MacroAssembler::JumpList slowCases;
-            
-            slowCases.append(m_jit.branch32(MacroAssembler::AboveOrEqual, propertyReg, MacroAssembler::Address(storageReg, ArrayStorage::vectorLengthOffset())));
-    
-            m_jit.load64(MacroAssembler::BaseIndex(storageReg, propertyReg, MacroAssembler::TimesEight, ArrayStorage::vectorOffset()), resultReg);
-            slowCases.append(m_jit.branchIfEmpty(resultReg));
-    
-            addSlowPathGenerator(
-                slowPathCall(
-                    slowCases, this, operationGetByValObjectInt,
-                    result.gpr(), TrustedImmPtr::weakPointer(m_graph, m_graph.globalObjectFor(node->origin.semantic)), baseReg, propertyReg));
-            
-            jsValueResult(resultReg, node);
-            break;
-        }
-        case Array::String:
-            compileGetByValOnString(node);
-            break;
-        case Array::DirectArguments:
-            compileGetByValOnDirectArguments(node);
-            break;
-        case Array::ScopedArguments:
-            compileGetByValOnScopedArguments(node);
-            break;
-        case Array::Int8Array:
-        case Array::Int16Array:
-        case Array::Int32Array:
-        case Array::Uint8Array:
-        case Array::Uint8ClampedArray:
-        case Array::Uint16Array:
-        case Array::Uint32Array:
-        case Array::Float32Array:
-        case Array::Float64Array: {
-            TypedArrayType type = node->arrayMode().typedArrayType();
-            if (isInt(type))
-                compileGetByValOnIntTypedArray(node, type);
-            else
-                compileGetByValOnFloatTypedArray(node, type);
-        } }
+            };
+            return std::make_pair(resultRegs, preferredFormat);
+        }));
         break;
     }
@@ -3582 +3626 @@
         }
         constexpr bool canSpeculate = false;
-        setIntTypedArrayLoadResult(node, resultGPR, type, canSpeculate);
+        constexpr bool shouldBox = false;
+        setIntTypedArrayLoadResult(node, JSValueRegs(resultGPR), type, canSpeculate, shouldBox);
         break;
     }
@@ -5270 +5315 @@
         break;
     }
-        
-    case GetEnumerableLength: {
-        compileGetEnumerableLength(node);
-        break;
-    }
-    case HasEnumerableStructureProperty: {
-        compileHasEnumerableStructureProperty(node);
-        break;
-    }
-    case HasEnumerableProperty: {
-        compileHasEnumerableProperty(node);
-        break;
-    }
-    case HasOwnStructureProperty: {
-        compileHasOwnStructureProperty(node);
-        break;
-    }
-    case InStructureProperty: {
-        compileInStructureProperty(node);
-        break;
-    }
+
     case HasIndexedProperty: {
-        compileHasIndexedProperty(node, operationHasIndexedProperty);
-        break;
-    }
-    case HasEnumerableIndexedProperty: {
-        compileHasIndexedProperty(node, operationHasEnumerableIndexedProperty);
-        break;
-    }
-    case GetDirectPname: {
-        compileGetDirectPname(node);
-        break;
-    }
+        SpeculateStrictInt32Operand index(this, m_graph.varArgChild(node, 1));
+        GPRTemporary result(this, Reuse, index);
+
+        compileHasIndexedProperty(node, operationHasIndexedProperty, scopedLambda<std::tuple<GPRReg, GPRReg>()>([&] { return std::make_pair(index.gpr(), result.gpr()); }));
+        unblessedBooleanResult(result.gpr(), node);
+        break;
+    }
+
     case GetPropertyEnumerator: {
         compileGetPropertyEnumerator(node);
         break;
     }
-    case GetEnumeratorStructurePname:
-    case GetEnumeratorGenericPname: {
-        compileGetEnumeratorPname(node);
-        break;
-    }
-    case ToIndexString: {
-        compileToIndexString(node);
-        break;
-    }
+
+    case EnumeratorNextUpdateIndexAndMode: {
+        compileEnumeratorNextUpdateIndexAndMode(node);
+        break;
+    }
+
+    case EnumeratorNextExtractMode: {
+        compileEnumeratorNextExtractMode(node);
+        break;
+    }
+
+    case EnumeratorNextExtractIndex: {
+        compileEnumeratorNextExtractIndex(node);
+        break;
+    }
+
+    case EnumeratorNextUpdatePropertyName: {
+        compileEnumeratorNextUpdatePropertyName(node);
+        break;
+    }
+
+    case EnumeratorGetByVal: {
+        compileEnumeratorGetByVal(node);
+        break;
+    }
+
+    case EnumeratorInByVal: {
+        compileEnumeratorInByVal(node);
+        break;
+    }
+
+    case EnumeratorHasOwnProperty: {
+        compileEnumeratorHasOwnProperty(node);
+        break;
+    }
+
     case ProfileType: {
         compileProfileType(node);

trunk/Source/JavaScriptCore/dfg/DFGTypeCheckHoistingPhase.cpp

@@ -274 +274 @@
                 case NukeStructureAndSetButterfly:
                 case GetButterfly:
+                case EnumeratorGetByVal:
                 case GetByVal:
                 case PutByValDirect:
@@ -353 +354 @@
                 case ReallocatePropertyStorage:
                 case GetButterfly:
+                case EnumeratorGetByVal:
                 case GetByVal:
                 case PutByValDirect:

trunk/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h

@@ -115 +115 @@
     macro(JSPropertyNameEnumerator_endStructurePropertyIndex, JSPropertyNameEnumerator::endStructurePropertyIndexOffset()) \
     macro(JSPropertyNameEnumerator_indexLength, JSPropertyNameEnumerator::indexedLengthOffset()) \
+    macro(JSPropertyNameEnumerator_modeSet, JSPropertyNameEnumerator::modeSetOffset()) \
     macro(JSRopeString_flags, JSRopeString::offsetOfFlags()) \
     macro(JSRopeString_length, JSRopeString::offsetOfLength()) \

trunk/Source/JavaScriptCore/ftl/FTLCapabilities.cpp

@@ -287 +287 @@
     case BooleanToNumber:
     case HasIndexedProperty:
-    case HasEnumerableIndexedProperty:
-    case HasEnumerableStructureProperty:
-    case HasEnumerableProperty:
-    case HasOwnStructureProperty:
-    case InStructureProperty:
-    case GetDirectPname:
-    case GetEnumerableLength:
     case GetIndexedPropertyStorage:
     case GetPropertyEnumerator:
-    case GetEnumeratorStructurePname:
-    case GetEnumeratorGenericPname:
-    case ToIndexString:
+    case EnumeratorNextUpdateIndexAndMode:
+    case EnumeratorNextExtractMode:
+    case EnumeratorNextExtractIndex:
+    case EnumeratorNextUpdatePropertyName:
+    case EnumeratorGetByVal:
+    case EnumeratorInByVal:
+    case EnumeratorHasOwnProperty:
     case BottomValue:
     case PhantomNewObject:

trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp

@@ -1524 +1524 @@
             break;
         case HasIndexedProperty:
-            compileHasIndexedProperty(operationHasIndexedProperty);
-            break;
-        case HasEnumerableIndexedProperty:
-            compileHasIndexedProperty(operationHasEnumerableIndexedProperty);
-            break;
-        case HasEnumerableStructureProperty:
-            compileHasEnumerableStructureProperty();
-            break;
-        case HasEnumerableProperty:
-            compileHasEnumerableProperty();
-            break;
-        case HasOwnStructureProperty:
-            compileHasOwnStructureProperty();
-            break;
-        case InStructureProperty:
-            compileInStructureProperty();
-            break;
-        case GetDirectPname:
-            compileGetDirectPname();
-            break;
-        case GetEnumerableLength:
-            compileGetEnumerableLength();
+            compileHasIndexedProperty();
             break;
         case GetPropertyEnumerator:
             compileGetPropertyEnumerator();
             break;
-        case GetEnumeratorStructurePname:
-            compileGetEnumeratorStructurePname();
-            break;
-        case GetEnumeratorGenericPname:
-            compileGetEnumeratorGenericPname();
-            break;
-        case ToIndexString:
-            compileToIndexString();
+        case EnumeratorNextUpdateIndexAndMode:
+            compileEnumeratorNextUpdateIndexAndMode();
+            break;
+        case EnumeratorNextExtractIndex:
+            compileEnumeratorNextExtractIndex();
+            break;
+        case EnumeratorNextExtractMode:
+            compileEnumeratorNextExtractMode();
+            break;
+        case EnumeratorNextUpdatePropertyName:
+            compileEnumeratorNextUpdatePropertyName();
+            break;
+        case EnumeratorGetByVal:
+            compileEnumeratorGetByVal();
+            break;
+        case EnumeratorInByVal:
+            compileEnumeratorInByVal();
+            break;
+        case EnumeratorHasOwnProperty:
+            compileEnumeratorHasOwnProperty();
             break;
         case CheckStructureImmediate:
@@ -5021 +5012 @@
     }
     
-    void compileGetByVal()
+    LValue compileGetByValImpl()
     {
         JSGlobalObject* globalObject = m_graph.globalObjectFor(m_origin.semantic);
@@ -5048 +5039 @@
                 if (m_node->arrayMode().type() == Array::Contiguous)
                     ensureStillAliveHere(base);
-                setJSValue(result);
-                return;
+                return result;
             }
 
@@ -5081 +5071 @@
             if (m_node->arrayMode().type() == Array::Contiguous)
                 ensureStillAliveHere(base);
-            setJSValue(m_out.phi(Int64, fastResult, slowResult));
-            return;
+            return m_out.phi(Int64, fastResult, slowResult);
         }
             
@@ -5101 +5090 @@
                         m_out.doubleNotEqualOrUnordered(result, result));
                 }
-                setDouble(result);
-                break;
+
+                return result;
             }
 
@@ -5142 +5131 @@
             m_out.appendTo(continuation, lastNext);
             if (resultIsUnboxed)
-                setDouble(m_out.phi(Double, fastResult, slowResult));
-            else
-                setJSValue(m_out.phi(Int64, fastResult, slowResult));
-            return;
+                return m_out.phi(Double, fastResult, slowResult);
+
+            return m_out.phi(Int64, fastResult, slowResult);
         }
 
@@ -5152 +5140 @@
 
             speculate(OutOfBounds, noValue(), m_node, m_out.lessThan(index, m_out.int32Zero));
-            setJSValue(m_out.constInt64(JSValue::ValueUndefined));
-            return;
+            return m_out.constInt64(JSValue::ValueUndefined);
         }
             
@@ -5170 +5157 @@
                 TypedPointer address = m_out.baseIndex(
                     m_heaps.DirectArguments_storage, base, m_out.zeroExtPtr(index));
-                setJSValue(m_out.load64(address));
-                return;
+                return m_out.load64(address);
             }
 
@@ -5193 +5179 @@
 
             m_out.appendTo(continuation, lastNext);
-            setJSValue(m_out.phi(Int64, fastResult, slowResult));
-            return;
+            return m_out.phi(Int64, fastResult, slowResult);
         }
             
@@ -5247 +5232 @@
             
             m_out.appendTo(continuation, lastNext);
-            setJSValue(m_out.phi(Int64, namedResult, overflowResult));
-            return;
+            return m_out.phi(Int64, namedResult, overflowResult);
         }
             
@@ -5257 +5241 @@
                 if (m_graph.varArgChild(m_node, 0).useKind() == ObjectUse) {
                     if (m_graph.varArgChild(m_node, 1).useKind() == StringUse) {
-                        setJSValue(vmCall(
+                        return vmCall(
                             Int64, operationGetByValObjectString, weakPointer(globalObject),
-                            lowObject(m_graph.varArgChild(m_node, 0)), lowString(m_graph.varArgChild(m_node, 1))));
-                        return;
+                            lowObject(m_graph.varArgChild(m_node, 0)), lowString(m_graph.varArgChild(m_node, 1)));
                     }
 
                     if (m_graph.varArgChild(m_node, 1).useKind() == SymbolUse) {
-                        setJSValue(vmCall(
+                        return vmCall(
                             Int64, operationGetByValObjectSymbol, weakPointer(globalObject),
-                            lowObject(m_graph.varArgChild(m_node, 0)), lowSymbol(m_graph.varArgChild(m_node, 1))));
-                        return;
+                            lowObject(m_graph.varArgChild(m_node, 0)), lowSymbol(m_graph.varArgChild(m_node, 1)));
                     }
                 }
 
-                setJSValue(vmCall(
+                return vmCall(
                     Int64, operationGetByVal, weakPointer(globalObject),
-                    lowJSValue(m_graph.varArgChild(m_node, 0)), lowJSValue(m_graph.varArgChild(m_node, 1))));
-                return;
+                    lowJSValue(m_graph.varArgChild(m_node, 0)), lowJSValue(m_graph.varArgChild(m_node, 1)));
             }
 
@@ -5374 +5355 @@
             });
 
-            setJSValue(patchpoint);
-            return;
+            return patchpoint;
         }
 
@@ -5391 +5371 @@
                 // We have to keep base alive to keep content in storage alive.
                 ensureStillAliveHere(base);
-                setJSValue(result);
-                return;
+                return result;
             }
 
@@ -5418 +5397 @@
             // We have to keep base alive to keep content in storage alive.
             ensureStillAliveHere(base);
-            setJSValue(m_out.phi(Int64, fastResult, slowResult));
-            return;
+            return m_out.phi(Int64, fastResult, slowResult);
         }
             
         case Array::String: {
-            compileStringCharAt();
-            return;
+            return compileStringCharAtImpl();
         }
             
@@ -5449 +5426 @@
                     // We have to keep base alive since that keeps storage alive.
                     ensureStillAliveHere(base);
-                    constexpr bool canSpeculate = true;
-                    setIntTypedArrayLoadResult(result, type, canSpeculate);
-                    return;
+                    return result;
                 }
             
@@ -5470 +5445 @@
                 // We have to keep base alive since that keeps storage alive.
                 ensureStillAliveHere(base);
-                setDouble(result);
-                return;
+                return result;
             }
         }
@@ -5481 +5455 @@
         case Array::Unprofiled:
             DFG_CRASH(m_graph, m_node, "Bad array type");
+            return nullptr;
+        }
+    }
+
+    void compileGetByVal()
+    {
+        LValue result = compileGetByValImpl();
+        TypedArrayType type = m_node->arrayMode().typedArrayType();
+        if (isInt(type)) {
+            constexpr bool canSpeculate = true;
+            setIntTypedArrayLoadResult(result, type, canSpeculate);
             return;
         }
+
+        if (result->type() == Double)
+            setDouble(result);
+        else
+            setJSValue(result);
     }
     
@@ -8572 +8562 @@
     }
     
-    void compileStringCharAt()
+    LValue compileStringCharAtImpl()
     {
         LValue base = lowString(m_graph.child(m_node, 0));
@@ -8680 +8670 @@
         // We have to keep base alive since that keeps storage alive.
         ensureStillAliveHere(base);
-        setJSValue(m_out.phi(Int64, results));
+        return m_out.phi(Int64, results);
+    }
+
+    void compileStringCharAt()
+    {
+        setJSValue(compileStringCharAtImpl());
     }
     
@@ -12907 +12902 @@
     }
     
-    void compileHasIndexedProperty(S_JITOperation_GCZ slowPathOperation)
+    LValue compileHasIndexedPropertyImpl(LValue index, S_JITOperation_GCZ slowPathOperation)
     {
         JSGlobalObject* globalObject = m_graph.globalObjectFor(m_origin.semantic);
         LValue base = lowCell(m_graph.varArgChild(m_node, 0));
-        LValue index = lowInt32(m_graph.varArgChild(m_node, 1));
         ArrayMode mode = m_node->arrayMode();
 
@@ -12917 +12911 @@
         case Array::Int32:
         case Array::Contiguous: {
-            LValue storage = lowStorage(m_graph.varArgChild(m_node, 2));
+            LValue storage = lowStorage(m_graph.varArgChild(m_node, m_node->storageChildIndex()));
 
             IndexedAbstractHeap& heap = mode.type() == Array::Int32 ?
@@ -12952 +12946 @@
 
             m_out.appendTo(continuation, lastNext);
-            setBoolean(m_out.phi(Int32, checkHoleResult, slowResult));
-            return;
+            return m_out.phi(Int32, checkHoleResult, slowResult);
         }
         case Array::Double: {
-            LValue storage = lowStorage(m_graph.varArgChild(m_node, 2));
+            LValue storage = lowStorage(m_graph.varArgChild(m_node, m_node->storageChildIndex()));
             
             IndexedAbstractHeap& heap = m_heaps.indexedDoubleProperties;
@@ -12990 +12983 @@
             
             m_out.appendTo(continuation, lastNext);
-            setBoolean(m_out.phi(Int32, checkHoleResult, slowResult));
-            return;
+            return m_out.phi(Int32, checkHoleResult, slowResult);
         }
 
         case Array::ArrayStorage: {
-            LValue storage = lowStorage(m_graph.varArgChild(m_node, 2));
+            LValue storage = lowStorage(m_graph.varArgChild(m_node, m_node->storageChildIndex()));
 
             LBasicBlock slowCase = m_out.newBlock();
@@ -13027 +13019 @@
 
             m_out.appendTo(continuation, lastNext);
-            setBoolean(m_out.phi(Int32, checkHoleResult, slowResult));
-            break;
+            return m_out.phi(Int32, checkHoleResult, slowResult);
         }
 
         default: {
-            setBoolean(m_out.notZero64(vmCall(Int64, slowPathOperation, weakPointer(globalObject), base, index)));
-            break;
-        }
-        }
-    }
-
-    void compileHasEnumerableProperty()
-    {
-        JSGlobalObject* globalObject = m_graph.globalObjectFor(m_origin.semantic);
-        LValue base = lowJSValue(m_node->child1());
-        LValue property = lowCell(m_node->child2());
-        setJSValue(vmCall(Int64, operationHasEnumerableProperty, weakPointer(globalObject), base, property));
-    }
-
-    template <typename SlowPathCall>
-    void compileHasStructurePropertyImpl(LValue base, SlowPathCall slowPathCall)
-    {
-        JSGlobalObject* globalObject = m_graph.globalObjectFor(m_origin.semantic);
-        LValue property = lowString(m_node->child2());
-        LValue enumerator = lowCell(m_node->child3());
-
-        LBasicBlock isCellCase = m_out.newBlock();
-        LBasicBlock correctStructure = m_out.newBlock();
-        LBasicBlock slowPath = m_out.newBlock();
-        LBasicBlock continuation = m_out.newBlock();
-
-        m_out.branch(isCell(base, provenType(m_node->child1())),
-            usually(isCellCase), rarely(slowPath));
-
-        LBasicBlock lastNext = m_out.appendTo(isCellCase, correctStructure);
-
-        m_out.branch(m_out.notEqual(
-            m_out.load32(base, m_heaps.JSCell_structureID),
-            m_out.load32(enumerator, m_heaps.JSPropertyNameEnumerator_cachedStructureID)),
-            rarely(slowPath), usually(correctStructure));
-
-        m_out.appendTo(correctStructure, slowPath);
-        ValueFromBlock correctStructureResult = m_out.anchor(m_out.booleanTrue);
-        m_out.jump(continuation);
-
-        m_out.appendTo(slowPath, continuation);
-        ValueFromBlock slowPathResult = m_out.anchor(
-            m_out.equal(
-                m_out.constInt64(JSValue::encode(jsBoolean(true))), 
-                vmCall(Int64, slowPathCall, weakPointer(globalObject), base, property)));
-        m_out.jump(continuation);
-
-        m_out.appendTo(continuation, lastNext);
-        setBoolean(m_out.phi(Int32, correctStructureResult, slowPathResult));
-    }
-
-    void compileHasEnumerableStructureProperty()
-    {
-        compileHasStructurePropertyImpl(lowJSValue(m_node->child1()), operationHasEnumerableProperty);
-    }
-
-    void compileHasOwnStructureProperty()
-    {
-        compileHasStructurePropertyImpl(lowCell(m_node->child1()), operationHasOwnStructureProperty);
-    }
-
-    void compileInStructureProperty()
-    {
-        compileHasStructurePropertyImpl(lowCell(m_node->child1()), operationInStructureProperty);
-    }
-
-    void compileGetDirectPname()
-    {
-        JSGlobalObject* globalObject = m_graph.globalObjectFor(m_origin.semantic);
-        LValue base = lowCell(m_graph.varArgChild(m_node, 0));
-        LValue property = lowCell(m_graph.varArgChild(m_node, 1));
-        LValue index = lowInt32(m_graph.varArgChild(m_node, 2));
-        LValue enumerator = lowCell(m_graph.varArgChild(m_node, 3));
-
-        LBasicBlock checkOffset = m_out.newBlock();
-        LBasicBlock inlineLoad = m_out.newBlock();
-        LBasicBlock outOfLineLoad = m_out.newBlock();
-        LBasicBlock slowCase = m_out.newBlock();
-        LBasicBlock continuation = m_out.newBlock();
-
-        m_out.branch(m_out.notEqual(
-            m_out.load32(base, m_heaps.JSCell_structureID),
-            m_out.load32(enumerator, m_heaps.JSPropertyNameEnumerator_cachedStructureID)),
-            rarely(slowCase), usually(checkOffset));
-
-        LBasicBlock lastNext = m_out.appendTo(checkOffset, inlineLoad);
-        m_out.branch(m_out.aboveOrEqual(index, m_out.load32(enumerator, m_heaps.JSPropertyNameEnumerator_cachedInlineCapacity)),
-            unsure(outOfLineLoad), unsure(inlineLoad));
-
-        m_out.appendTo(inlineLoad, outOfLineLoad);
-        ValueFromBlock inlineResult = m_out.anchor(
-            m_out.load64(m_out.baseIndex(m_heaps.properties.atAnyNumber(), 
-                base, m_out.zeroExt(index, Int64), ScaleEight, JSObject::offsetOfInlineStorage())));
-        m_out.jump(continuation);
-
-        m_out.appendTo(outOfLineLoad, slowCase);
-        LValue storage = m_out.loadPtr(base, m_heaps.JSObject_butterfly);
-        LValue realIndex = m_out.signExt32To64(
-            m_out.neg(m_out.sub(index, m_out.load32(enumerator, m_heaps.JSPropertyNameEnumerator_cachedInlineCapacity))));
-        int32_t offsetOfFirstProperty = static_cast<int32_t>(offsetInButterfly(firstOutOfLineOffset)) * sizeof(EncodedJSValue);
-        ValueFromBlock outOfLineResult = m_out.anchor(
-            m_out.load64(m_out.baseIndex(m_heaps.properties.atAnyNumber(), storage, realIndex, ScaleEight, offsetOfFirstProperty)));
-        m_out.jump(continuation);
-
-        m_out.appendTo(slowCase, continuation);
-        ValueFromBlock slowCaseResult = m_out.anchor(
-            vmCall(Int64, operationGetByVal, weakPointer(globalObject), base, property));
-        m_out.jump(continuation);
-
-        m_out.appendTo(continuation, lastNext);
-        setJSValue(m_out.phi(Int64, inlineResult, outOfLineResult, slowCaseResult));
-    }
-
-    void compileGetEnumerableLength()
-    {
-        LValue enumerator = lowCell(m_node->child1());
-        setInt32(m_out.load32(enumerator, m_heaps.JSPropertyNameEnumerator_indexLength));
+            return m_out.notZero64(vmCall(Int64, slowPathOperation, weakPointer(globalObject), base, index));
+        }
+        }
+    }
+
+    void compileHasIndexedProperty()
+    {
+        LValue index = lowInt32(m_graph.varArgChild(m_node, 1));
+        setBoolean(compileHasIndexedPropertyImpl(index, operationHasIndexedProperty));
     }
 
@@ -13160 +13043 @@
     }
 
-    void compileGetEnumeratorStructurePname()
-    {
-        LValue enumerator = lowCell(m_node->child1());
-        LValue index = lowInt32(m_node->child2());
-
-        LBasicBlock inBounds = m_out.newBlock();
-        LBasicBlock outOfBounds = m_out.newBlock();
-        LBasicBlock continuation = m_out.newBlock();
-
-        m_out.branch(m_out.below(index, m_out.load32(enumerator, m_heaps.JSPropertyNameEnumerator_endStructurePropertyIndex)),
-            usually(inBounds), rarely(outOfBounds));
-
-        LBasicBlock lastNext = m_out.appendTo(inBounds, outOfBounds);
-        LValue storage = m_out.loadPtr(enumerator, m_heaps.JSPropertyNameEnumerator_cachedPropertyNamesVector);
-        ValueFromBlock inBoundsResult = m_out.anchor(
-            m_out.loadPtr(m_out.baseIndex(m_heaps.JSPropertyNameEnumerator_cachedPropertyNamesVectorContents, storage, m_out.zeroExtPtr(index))));
-        m_out.jump(continuation);
-
-        m_out.appendTo(outOfBounds, continuation);
-        ValueFromBlock outOfBoundsResult = m_out.anchor(m_out.constInt64(JSValue::ValueNull));
-        m_out.jump(continuation);
-        
-        m_out.appendTo(continuation, lastNext);
-        setJSValue(m_out.phi(Int64, inBoundsResult, outOfBoundsResult));
-    }
-
-    void compileGetEnumeratorGenericPname()
-    {
-        LValue enumerator = lowCell(m_node->child1());
-        LValue index = lowInt32(m_node->child2());
-
-        LBasicBlock inBounds = m_out.newBlock();
-        LBasicBlock outOfBounds = m_out.newBlock();
-        LBasicBlock continuation = m_out.newBlock();
-
-        m_out.branch(m_out.below(index, m_out.load32(enumerator, m_heaps.JSPropertyNameEnumerator_endGenericPropertyIndex)),
-            usually(inBounds), rarely(outOfBounds));
-
-        LBasicBlock lastNext = m_out.appendTo(inBounds, outOfBounds);
-        LValue storage = m_out.loadPtr(enumerator, m_heaps.JSPropertyNameEnumerator_cachedPropertyNamesVector);
-        ValueFromBlock inBoundsResult = m_out.anchor(
-            m_out.loadPtr(m_out.baseIndex(m_heaps.JSPropertyNameEnumerator_cachedPropertyNamesVectorContents, storage, m_out.zeroExtPtr(index))));
-        m_out.jump(continuation);
-
-        m_out.appendTo(outOfBounds, continuation);
-        ValueFromBlock outOfBoundsResult = m_out.anchor(m_out.constInt64(JSValue::ValueNull));
-        m_out.jump(continuation);
-        
-        m_out.appendTo(continuation, lastNext);
-        setJSValue(m_out.phi(Int64, inBoundsResult, outOfBoundsResult));
-    }
-    
-    void compileToIndexString()
+    void compileEnumeratorNextUpdateIndexAndMode()
+    {
+        JSGlobalObject* globalObject = m_graph.globalObjectFor(m_origin.semantic);
+        LValue index = lowInt32(m_graph.varArgChild(m_node, 1));
+        LValue mode = lowInt32(m_graph.varArgChild(m_node, 2));
+        LValue enumerator = lowCell(m_graph.varArgChild(m_node, 3));
+
+        if (m_node->enumeratorMetadata() == JSPropertyNameEnumerator::IndexedMode) {
+            speculate(BadCache, noValue(), m_node, m_out.notZero32(m_out.load32(enumerator, m_heaps.JSPropertyNameEnumerator_endGenericPropertyIndex)));
+
+            LBasicBlock increment = m_out.newBlock();
+            LBasicBlock checkHasProperty = m_out.newBlock();
+            LBasicBlock continuation = m_out.newBlock();
+
+            ValueFromBlock initialIndex = m_out.anchor(index);
+            ValueFromBlock initialIndexForHasProperty = m_out.anchor(index);
+            m_out.branch(m_out.isZero32(mode), unsure(checkHasProperty), unsure(increment));
+
+            m_out.appendTo(increment);
+            LValue indexPhi = m_out.phi(Int32, initialIndex);
+            LValue incrementedIndex = m_out.add(indexPhi, m_out.int32One);
+            ValueFromBlock incrementedIndexResult = m_out.anchor(incrementedIndex);
+            ValueFromBlock finalIncrementedIndex = m_out.anchor(incrementedIndex);
+            m_out.branch(m_out.lessThan(incrementedIndex, m_out.load32(enumerator, m_heaps.JSPropertyNameEnumerator_indexLength)), unsure(checkHasProperty), unsure(continuation));
+
+            m_out.appendTo(checkHasProperty);
+            LValue propertyIndex = m_out.phi(Int32, initialIndexForHasProperty, incrementedIndexResult);
+            m_out.addIncomingToPhi(indexPhi, m_out.anchor(propertyIndex));
+            ValueFromBlock finalPropertyIndex = m_out.anchor(propertyIndex);
+            LValue hasProperty = compileHasIndexedPropertyImpl(propertyIndex, operationHasEnumerableIndexedProperty);
+            m_out.branch(hasProperty, unsure(continuation), unsure(increment));
+
+            m_out.appendTo(continuation);
+            LValue finalIndex = m_out.phi(Int32, finalIncrementedIndex, finalPropertyIndex);
+            setJSValue(m_out.bitOr(m_out.zeroExt(finalIndex, Int64), m_out.constInt64(JSValue::NumberTag | static_cast<uint64_t>(JSPropertyNameEnumerator::IndexedMode) << 32)));
+            return;
+        }
+
+        Edge& baseEdge = m_graph.varArgChild(m_node, 0);
+        if (m_node->enumeratorMetadata() == JSPropertyNameEnumerator::OwnStructureMode && baseEdge.useKind() == CellUse) {
+            LValue base = lowCell(baseEdge);
+            speculate(BadCache, noValue(), m_node, m_out.notEqual(m_out.load32(base, m_heaps.JSCell_structureID), m_out.load32(enumerator, m_heaps.JSPropertyNameEnumerator_cachedStructureID)));
+
+            speculate(BadCache, noValue(), m_node, m_out.notEqual(m_out.load32(enumerator, m_heaps.JSPropertyNameEnumerator_modeSet), m_out.constInt32(JSPropertyNameEnumerator::OwnStructureMode)));
+
+            LBasicBlock increment = m_out.newBlock();
+            LBasicBlock continuation = m_out.newBlock();
+
+            ValueFromBlock initialIndex = m_out.anchor(index);
+            m_out.branch(m_out.isZero32(mode), unsure(continuation), unsure(increment));
+
+            m_out.appendTo(increment);
+            ValueFromBlock incrementedIndex = m_out.anchor(m_out.add(index, m_out.int32One));
+            m_out.jump(continuation);
+
+            m_out.appendTo(continuation);
+            index = m_out.phi(Int32, initialIndex, incrementedIndex);
+            setJSValue(m_out.bitOr(m_out.zeroExt(index, Int64), m_out.constInt64(JSValue::DoubleEncodeOffset | static_cast<uint64_t>(JSPropertyNameEnumerator::OwnStructureMode) << 32)));
+            return;
+        }
+
+        LValue base = lowJSValue(baseEdge);
+        setJSValue(vmCall(Int64, operationEnumeratorNextUpdateIndexAndMode, weakPointer(globalObject), base, index, mode, enumerator));
+    }
+
+    void compileEnumeratorNextExtractIndex()
+    {
+        LValue boxedPair = lowJSValue(m_node->child1());
+
+        setInt32(m_out.castToInt32(boxedPair));
+    }
+
+    void compileEnumeratorNextExtractMode()
+    {
+        LValue boxedPair = lowJSValue(m_node->child1());
+
+        LValue highBits = m_out.castToInt32(m_out.lShr(boxedPair, m_out.constInt32(32)));
+        setInt32(m_out.bitAnd(highBits, m_out.constInt32(JSPropertyNameEnumerator::enumerationModeMask)));
+    }
+
+    // FIXME: We should probably have a method of value recovery for this node since it's "effect" free but always live in bytecode.
+    void compileEnumeratorNextUpdatePropertyName()
     {
         JSGlobalObject* globalObject = m_graph.globalObjectFor(m_origin.semantic);
         LValue index = lowInt32(m_node->child1());
-        setJSValue(vmCall(Int64, operationToIndexString, weakPointer(globalObject), index));
+        LValue mode = lowInt32(m_node->child2());
+        LValue enumerator = lowCell(m_node->child3());
+
+        LBasicBlock operationBlock = nullptr;
+        LBasicBlock continuation = nullptr;
+
+        Vector<ValueFromBlock, 3> results;
+        OptionSet seenModes = m_node->enumeratorMetadata();
+
+        if (seenModes.containsAny({ JSPropertyNameEnumerator::OwnStructureMode, JSPropertyNameEnumerator::GenericMode })) {
+            LBasicBlock checkIndex = nullptr;
+            LBasicBlock outOfBoundsBlock = m_out.newBlock();
+            LBasicBlock loadPropertyNameBlock = m_out.newBlock();
+            continuation = m_out.newBlock();
+
+            if (seenModes.contains(JSPropertyNameEnumerator::IndexedMode)) {
+                checkIndex = m_out.newBlock();
+                operationBlock = m_out.newBlock();
+                m_out.branch(m_out.testIsZero32(mode, m_out.constInt32(JSPropertyNameEnumerator::IndexedMode)), unsure(checkIndex), unsure(operationBlock));
+            }
+
+            {
+                if (checkIndex)
+                    m_out.appendTo(checkIndex);
+                LValue outOfBounds = m_out.aboveOrEqual(index, m_out.load32(enumerator, m_heaps.JSPropertyNameEnumerator_endGenericPropertyIndex));
+                m_out.branch(outOfBounds, unsure(outOfBoundsBlock), unsure(loadPropertyNameBlock));
+            }
+
+            {
+                m_out.appendTo(outOfBoundsBlock);
+                results.append(m_out.anchor(m_out.constInt64(JSValue::encode(jsNull()))));
+                m_out.jump(continuation);
+            }
+
+            {
+                m_out.appendTo(loadPropertyNameBlock);
+                LValue namesVector = m_out.loadPtr(enumerator, m_heaps.JSPropertyNameEnumerator_cachedPropertyNamesVector);
+                results.append(m_out.anchor(m_out.zeroExtPtr(m_out.loadPtr(m_out.baseIndex(m_heaps.WriteBarrierBuffer_bufferContents.atAnyIndex(), namesVector, m_out.zeroExt(index, Int64), ScalePtr)))));
+                m_out.jump(continuation);
+            }
+        }
+
+        if (seenModes.contains(JSPropertyNameEnumerator::IndexedMode)) {
+            if (operationBlock)
+                m_out.appendTo(operationBlock);
+            results.append(m_out.anchor(vmCall(Int64, operationEnumeratorNextUpdatePropertyName, weakPointer(globalObject), index, mode, enumerator)));
+            if (continuation)
+                m_out.jump(continuation);
+        }
+
+        if (continuation)
+            m_out.appendTo(continuation);
+
+        ASSERT(results.size());
+        LValue result = m_out.phi(Int64, results);
+        setJSValue(result);
+    }
+
+    void compileEnumeratorGetByVal()
+    {
+        Edge baseEdge = m_graph.varArgChild(m_node, 0);
+        Edge propertyNameEdge = m_graph.varArgChild(m_node, 1);
+        Edge storageEdge = m_graph.varArgChild(m_node, 2);
+        Edge indexEdge = m_graph.varArgChild(m_node, 3);
+
+        LValue base = DFG::isCell(baseEdge.useKind()) ? lowCell(baseEdge, ManualOperandSpeculation) : lowJSValue(baseEdge);
+
+        LValue storage = storageEdge ? lowStorage(storageEdge) : nullptr;
+        // We have to materialize the propertyName here in case it requires boxing since the IC path won't dominate subsequent uses in subsequent nodes.
+        // If it's an Int32 and we use it as such this boxing will be DCE'd by b3 later anyway.
+        lowJSValue(propertyNameEdge, ManualOperandSpeculation);
+
+
+        LValue index = lowInt32(indexEdge);
+        LValue mode = lowInt32(m_graph.varArgChild(m_node, 4));
+        LValue enumerator = lowCell(m_graph.varArgChild(m_node, 5));
+
+        LBasicBlock checkIsCellBlock = m_out.newBlock();
+        LBasicBlock checkStructureBlock = m_out.newBlock();
+        LBasicBlock checkInlineOrOutOfLineBlock = m_out.newBlock();
+        LBasicBlock inlineLoadBlock = m_out.newBlock();
+        LBasicBlock outOfLineLoadBlock = m_out.newBlock();
+        LBasicBlock genericICBlock = m_out.newBlock();
+        LBasicBlock continuation = m_out.newBlock();
+
+        Vector<ValueFromBlock, 4> results;
+
+        LValue isNotNamed = m_out.testNonZero32(mode, m_out.constInt32(JSPropertyNameEnumerator::IndexedMode | JSPropertyNameEnumerator::GenericMode));
+        m_out.branch(isNotNamed, unsure(genericICBlock), unsure(checkIsCellBlock));
+
+        m_out.appendTo(checkIsCellBlock);
+        m_out.branch(isCell(base, provenType(baseEdge)), usually(checkStructureBlock), rarely(genericICBlock));
+
+        m_out.appendTo(checkStructureBlock);
+        LValue structureID;
+        auto structure = m_state.forNode(baseEdge.node()).m_structure.onlyStructure();
+        if (structure)
+            structureID = m_out.constInt32(structure->id());
+        else
+            structureID = m_out.load32(base, m_heaps.JSCell_structureID);
+
+        LValue hasEnumeratorStructure = m_out.equal(structureID, m_out.load32(enumerator, m_heaps.JSPropertyNameEnumerator_cachedStructureID));
+
+        if (indexEdge.node() == propertyNameEdge.node()) {
+            JSGlobalObject* globalObject = m_graph.globalObjectFor(m_origin.semantic);
+            LBasicBlock badStructureSlowPath = m_out.newBlock();
+            m_out.branch(hasEnumeratorStructure, usually(checkInlineOrOutOfLineBlock), rarely(genericICBlock));
+
+            m_out.appendTo(badStructureSlowPath);
+            results.append(m_out.anchor(vmCall(Int64, operationEnumeratorRecoverNameAndGetByVal, weakPointer(globalObject), base, index, enumerator)));
+        } else
+            m_out.branch(hasEnumeratorStructure, usually(checkInlineOrOutOfLineBlock), rarely(genericICBlock));
+
+        m_out.appendTo(checkInlineOrOutOfLineBlock);
+        LValue inlineCapacity = nullptr;
+        bool hasNoOutOfLineProperties = false;
+        if (structure) {
+            hasNoOutOfLineProperties = !structure->outOfLineCapacity();
+            inlineCapacity = m_out.constInt32(structure->inlineCapacity());
+        }
+        if (!inlineCapacity)
+            inlineCapacity = m_out.load32(enumerator, m_heaps.JSPropertyNameEnumerator_cachedInlineCapacity);
+        LValue isInline = hasNoOutOfLineProperties ? m_out.int32One : m_out.below(index, inlineCapacity);
+        m_out.branch(isInline, unsure(inlineLoadBlock), unsure(outOfLineLoadBlock));
+
+        m_out.appendTo(inlineLoadBlock);
+        results.append(m_out.anchor(
+            m_out.load64(m_out.baseIndex(m_heaps.properties.atAnyNumber(),
+                base, m_out.zeroExt(index, Int64), ScaleEight, JSObject::offsetOfInlineStorage()))));
+        m_out.jump(continuation);
+
+        m_out.appendTo(outOfLineLoadBlock);
+        if (!storage)
+            storage = m_out.loadPtr(base, m_heaps.JSObject_butterfly);
+
+        LValue realIndex = m_out.signExt32To64(
+            m_out.neg(m_out.sub(index, inlineCapacity)));
+        int32_t offsetOfFirstProperty = static_cast<int32_t>(offsetInButterfly(firstOutOfLineOffset)) * sizeof(EncodedJSValue);
+        results.append(m_out.anchor(
+            m_out.load64(m_out.baseIndex(m_heaps.properties.atAnyNumber(), storage, realIndex, ScaleEight, offsetOfFirstProperty))));
+        m_out.jump(continuation);
+
+        m_out.appendTo(genericICBlock);
+        LValue genericResult = compileGetByValImpl();
+        TypedArrayType type = m_node->arrayMode().typedArrayType();
+        if (isInt(type)) {
+            if (elementSize(type) < 4 || isSigned(type))
+                genericResult = boxInt32(genericResult);
+            else
+                genericResult = strictInt52ToJSValue(genericResult);
+        } else if (genericResult->type() == Double)
+            genericResult = boxDouble(genericResult);
+
+        results.append(m_out.anchor(genericResult));
+        m_out.jump(continuation);
+
+        m_out.appendTo(continuation);
+        ASSERT(results.size());
+        LValue result = m_out.phi(Int64, results);
+        setJSValue(result);
+    }
+
+    template<typename SlowPathFunctionType>
+    void compileEnumeratorHasProperty(SlowPathFunctionType slowPathFunction)
+    {
+        JSGlobalObject* globalObject = m_graph.globalObjectFor(m_origin.semantic);
+        Edge baseEdge = m_graph.varArgChild(m_node, 0);
+        LValue base = baseEdge.useKind() == CellUse ? lowCell(baseEdge) : lowJSValue(baseEdge);
+        LValue propertyName = lowJSValue(m_graph.varArgChild(m_node, 1));
+        LValue index = lowInt32(m_graph.varArgChild(m_node, 2));
+        LValue mode = lowInt32(m_graph.varArgChild(m_node, 3));
+        LValue enumerator = lowCell(m_graph.varArgChild(m_node, 4));
+
+        LBasicBlock isNamedBlock = m_out.newBlock();
+        LBasicBlock isCellBlock = m_out.newBlock();
+        LBasicBlock matchesStructureBlock = m_out.newBlock();
+        LBasicBlock operationBlock = m_out.newBlock();
+        LBasicBlock continuation = m_out.newBlock();
+
+        m_out.branch(m_out.testNonZero32(mode, m_out.constInt32(JSPropertyNameEnumerator::OwnStructureMode)), unsure(isNamedBlock), unsure(operationBlock));
+        m_out.appendTo(isNamedBlock);
+
+        m_out.branch(isCell(base, provenType(baseEdge)), unsure(isCellBlock), unsure(operationBlock));
+
+        m_out.appendTo(isCellBlock);
+        LValue structureID = m_out.load32(base, m_heaps.JSCell_structureID);
+        m_out.branch(m_out.equal(structureID, m_out.load32(enumerator, m_heaps.JSPropertyNameEnumerator_cachedStructureID)),
+            usually(matchesStructureBlock), rarely(operationBlock));
+
+        m_out.appendTo(matchesStructureBlock);
+        ValueFromBlock namedResult = m_out.anchor(m_out.booleanTrue);
+        m_out.jump(continuation);
+
+        m_out.appendTo(operationBlock);
+        ValueFromBlock operationResult = m_out.anchor(unboxBoolean(vmCall(Int64, slowPathFunction, weakPointer(globalObject), base, propertyName, index, mode)));
+        m_out.jump(continuation);
+
+        m_out.appendTo(continuation);
+        setBoolean(m_out.phi(Int32, namedResult, operationResult));
+    }
+
+    void compileEnumeratorInByVal()
+    {
+        compileEnumeratorHasProperty(operationEnumeratorInByVal);
+    }
+
+    void compileEnumeratorHasOwnProperty()
+    {
+        compileEnumeratorHasProperty(operationEnumeratorHasOwnProperty);
     }
     
@@ -17345 +17465 @@
             return;
         }
-        
+
         if (m_node->shouldSpeculateInt32() && canSpeculate) {
             speculate(
@@ -20297 +20417 @@
     }
 
+    // This only works for ints right now.
+    PatchpointValue* dataLogForDebugging(const char* prefix, LValue value)
+    {
+        PatchpointValue* result = m_out.patchpoint(Void);
+        result->effects.writesLocalState = true;
+        result->append(value, ValueRep::reg(GPRInfo::regT0));
+        Type valueType = value->type();
+        result->setGenerator([=] (CCallHelpers& jit, const StackmapGenerationParams&) {
+            AllowMacroScratchRegisterUsage allowScratch(jit);
+
+            jit.probeDebug([=] (Probe::Context& context) {
+                if (valueType == Int32)
+                    dataLogLn(prefix, context.gpr<int32_t>(GPRInfo::regT0));
+                else
+                    dataLogLn(prefix, context.gpr<JSValue>(GPRInfo::regT0));
+            });
+        });
+
+        return result;
+    }
+
     AvailabilityMap& availabilityMap() { return m_availabilityCalculator.m_availability; }
     

trunk/Source/JavaScriptCore/ftl/FTLOutput.h

@@ -1 +1 @@
 /*
- * Copyright (C) 2013-2019 Apple Inc. All rights reserved.
+ * Copyright (C) 2013-2021 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -146 +146 @@
     template<typename... Params>
     void addIncomingToPhi(LValue phi, ValueFromBlock, Params... theRest);
+    template<typename... Params>
+    void addIncomingToPhiIfSet(LValue phi, Params... theRest);
     
     LValue opaque(LValue);
@@ -481 +483 @@
 inline LValue Output::phi(LType type, const VectorType& vector)
 {
+    ASSERT(vector.size());
     LValue phiNode = phi(type);
     for (const ValueFromBlock& valueFromBlock : vector)

trunk/Source/JavaScriptCore/generator/DSL.rb

@@ -139 +139 @@
 #include "Instruction.h"
 #include "IterationModeMetadata.h"
+#include "JSPropertyNameEnumerator.h"
 #include "Opcode.h"
 #include "PrivateFieldPutKind.h"

trunk/Source/JavaScriptCore/interpreter/Register.h

@@ -54 +54 @@
         ALWAYS_INLINE Register& operator=(CodeBlock*);
         ALWAYS_INLINE Register& operator=(JSScope*);
-        ALWAYS_INLINE Register& operator=(JSObject*);
+        ALWAYS_INLINE Register& operator=(JSCell*);
         ALWAYS_INLINE Register& operator=(EncodedJSValue);
 

trunk/Source/JavaScriptCore/interpreter/RegisterInlines.h

@@ -64 +64 @@
 }
 
-ALWAYS_INLINE Register& Register::operator=(JSObject* object)
+ALWAYS_INLINE Register& Register::operator=(JSCell* object)
 {
     u.value = JSValue::encode(JSValue(object));

trunk/Source/JavaScriptCore/jit/AssemblyHelpers.h

@@ -189 +189 @@
 #endif
     }
-    
+
     void loadValue(Address address, JSValueRegs regs)
     {
@@ -890 +890 @@
             Below, Address(cellGPR, JSCell::typeInfoTypeOffset()), TrustedImm32(ObjectType));
     }
-    
+
     // Note that first and last are inclusive.
     Jump branchIfType(GPRReg cellGPR, JSTypeRange range)

trunk/Source/JavaScriptCore/jit/JIT.cpp

@@ -309 +309 @@
         DEFINE_SLOW_OP(new_array_buffer)
         DEFINE_SLOW_OP(spread)
-        DEFINE_SLOW_OP(get_enumerable_length)
-        DEFINE_SLOW_OP(has_enumerable_property)
         DEFINE_SLOW_OP(get_property_enumerator)
-        DEFINE_SLOW_OP(to_index_string)
         DEFINE_SLOW_OP(create_direct_arguments)
         DEFINE_SLOW_OP(create_scoped_arguments)
@@ -366 +363 @@
         DEFINE_OP(op_get_by_id_direct)
         DEFINE_OP(op_get_by_val)
+        DEFINE_OP(op_enumerator_next)
+        DEFINE_OP(op_enumerator_get_by_val)
+        DEFINE_OP(op_enumerator_in_by_val)
+        DEFINE_OP(op_enumerator_has_own_property)
         DEFINE_OP(op_get_private_name)
         DEFINE_OP(op_set_private_brand)
@@ -474 +475 @@
         DEFINE_OP(op_put_to_arguments)
 
-        DEFINE_OP(op_has_enumerable_indexed_property)
-        DEFINE_OP(op_has_enumerable_structure_property)
-        DEFINE_OP(op_has_own_structure_property)
-        DEFINE_OP(op_in_structure_property)
-        DEFINE_OP(op_get_direct_pname)
-        DEFINE_OP(op_enumerator_structure_pname)
-        DEFINE_OP(op_enumerator_generic_pname)
-            
         DEFINE_OP(op_log_shadow_chicken_prologue)
         DEFINE_OP(op_log_shadow_chicken_tail)
@@ -579 +572 @@
         DEFINE_SLOWCASE_OP(op_get_by_id_direct)
         DEFINE_SLOWCASE_OP(op_get_by_val)
+        DEFINE_SLOWCASE_OP(op_enumerator_get_by_val)
         DEFINE_SLOWCASE_OP(op_get_private_name)
         DEFINE_SLOWCASE_OP(op_set_private_brand)
@@ -610 +604 @@
         DEFINE_SLOWCASE_OP(op_del_by_id)
         DEFINE_SLOWCASE_OP(op_sub)
-        DEFINE_SLOWCASE_OP(op_has_enumerable_indexed_property)
 #if !ENABLE(EXTRA_CTI_THUNKS)
         DEFINE_SLOWCASE_OP(op_get_from_scope)
@@ -643 +636 @@
         DEFINE_SLOWCASE_SLOW_OP(stricteq)
         DEFINE_SLOWCASE_SLOW_OP(nstricteq)
-        DEFINE_SLOWCASE_SLOW_OP(get_direct_pname)
         DEFINE_SLOWCASE_SLOW_OP(get_prototype_of)
-        DEFINE_SLOWCASE_SLOW_OP(has_enumerable_structure_property)
-        DEFINE_SLOWCASE_SLOW_OP(has_own_structure_property)
-        DEFINE_SLOWCASE_SLOW_OP(in_structure_property)
 #if !ENABLE(EXTRA_CTI_THUNKS)
         DEFINE_SLOWCASE_SLOW_OP(resolve_scope)

trunk/Source/JavaScriptCore/jit/JIT.h

@@ -454 +454 @@
         void emitGetVirtualRegister(VirtualRegister src, JSValueRegs dst);
         void emitPutVirtualRegister(VirtualRegister dst, JSValueRegs src);
+        void emitStore(VirtualRegister, const JSValue constant, RegisterID base = callFrameRegister);
 
         int32_t getOperandConstantInt(VirtualRegister src);
@@ -470 +471 @@
 
         void emitStore(VirtualRegister, RegisterID tag, RegisterID payload, RegisterID base = callFrameRegister);
-        void emitStore(VirtualRegister, const JSValue constant, RegisterID base = callFrameRegister);
         void emitStoreInt32(VirtualRegister, RegisterID payload, bool indexIsInt32 = false);
         void emitStoreInt32(VirtualRegister, TrustedImm32 payload, bool indexIsInt32 = false);
@@ -671 +671 @@
         void emit_op_unsigned(const Instruction*);
         void emit_op_urshift(const Instruction*);
-        template <typename OpCodeType>
-        void emit_op_has_structure_propertyImpl(const Instruction*);
-        void emit_op_has_enumerable_indexed_property(const Instruction*);
-        void emit_op_has_enumerable_structure_property(const Instruction*);
-        void emit_op_has_own_structure_property(const Instruction*);
-        void emit_op_in_structure_property(const Instruction*);
-        void emit_op_get_direct_pname(const Instruction*);
-        void emit_op_enumerator_structure_pname(const Instruction*);
-        void emit_op_enumerator_generic_pname(const Instruction*);
         void emit_op_get_internal_field(const Instruction*);
         void emit_op_put_internal_field(const Instruction*);
@@ -685 +676 @@
         void emit_op_log_shadow_chicken_tail(const Instruction*);
         void emit_op_to_property_key(const Instruction*);
+
+        template<typename OpcodeType>
+        void generateGetByValSlowCase(const OpcodeType&, Vector<SlowCaseEntry>::iterator&);
+
+        void emit_op_enumerator_next(const Instruction*);
+        void emit_op_enumerator_get_by_val(const Instruction*);
+        void emitSlow_op_enumerator_get_by_val(const Instruction*, Vector<SlowCaseEntry>::iterator&);
+
+        template<typename OpcodeType, typename SlowPathFunctionType>
+        void emit_enumerator_has_propertyImpl(const Instruction*, const OpcodeType&, SlowPathFunctionType);
+        void emit_op_enumerator_in_by_val(const Instruction*);
+        void emit_op_enumerator_has_own_property(const Instruction*);
 
         void emitSlow_op_add(const Instruction*, Vector<SlowCaseEntry>::iterator&);
@@ -736 +739 @@
         void emitSlow_op_put_private_name(const Instruction*, Vector<SlowCaseEntry>::iterator&);
         void emitSlow_op_sub(const Instruction*, Vector<SlowCaseEntry>::iterator&);
-        void emitSlow_op_has_enumerable_indexed_property(const Instruction*, Vector<SlowCaseEntry>::iterator&);
 
         void emit_op_resolve_scope(const Instruction*);

trunk/Source/JavaScriptCore/jit/JITOpcodes.cpp

@@ -1693 +1693 @@
 #if USE(JSVALUE64)
 
-template <typename OpCodeType>
-void JIT::emit_op_has_structure_propertyImpl(const Instruction* currentInstruction)
-{
-    auto bytecode = currentInstruction->as<OpCodeType>();
-    VirtualRegister dst = bytecode.m_dst;
-    VirtualRegister base = bytecode.m_base;
-    VirtualRegister enumerator = bytecode.m_enumerator;
-
-    emitGetVirtualRegister(base, regT0);
-    emitGetVirtualRegister(enumerator, regT1);
-    emitJumpSlowCaseIfNotJSCell(regT0, base);
-
-    load32(Address(regT0, JSCell::structureIDOffset()), regT0);
-    addSlowCase(branch32(NotEqual, regT0, Address(regT1, JSPropertyNameEnumerator::cachedStructureIDOffset())));
-    
-    move(TrustedImm64(JSValue::encode(jsBoolean(true))), regT0);
-    emitPutVirtualRegister(dst);
-}
-
-void JIT::emit_op_has_enumerable_structure_property(const Instruction* currentInstruction)
-{
-    emit_op_has_structure_propertyImpl<OpHasEnumerableStructureProperty>(currentInstruction);
-}
-
-void JIT::emit_op_has_own_structure_property(const Instruction* currentInstruction)
-{
-    emit_op_has_structure_propertyImpl<OpHasOwnStructureProperty>(currentInstruction);
-}
-
-void JIT::emit_op_in_structure_property(const Instruction* currentInstruction)
-{
-    emit_op_has_structure_propertyImpl<OpInStructureProperty>(currentInstruction);
-}
-
 void JIT::privateCompileHasIndexedProperty(ByValInfo* byValInfo, ReturnAddressPtr returnAddress, JITArrayMode arrayMode)
 {
     const Instruction* currentInstruction = m_codeBlock->instructions().at(byValInfo->bytecodeIndex).ptr();
-    
+
     PatchableJump badType;
-    
+
     // FIXME: Add support for other types like TypedArrays and Arguments.
     // See https://bugs.webkit.org/show_bug.cgi?id=135033 and https://bugs.webkit.org/show_bug.cgi?id=135034.
@@ -1740 +1706 @@
 
     LinkBuffer patchBuffer(*this, m_codeBlock, LinkBuffer::Profile::InlineCache);
-    
+
     patchBuffer.link(badType, byValInfo->slowPathTarget);
     patchBuffer.link(slowCases, byValInfo->slowPathTarget);
@@ -1749 +1715 @@
         m_codeBlock, patchBuffer, JITStubRoutinePtrTag,
         "Baseline has_indexed_property stub for %s, return point %p", toCString(*m_codeBlock).data(), returnAddress.untaggedValue());
-    
+
     if (JITCode::useDataIC(JITType::BaselineJIT)) {
         byValInfo->m_badTypeJumpTarget = CodeLocationLabel<JITStubRoutinePtrTag>(byValInfo->stubRoutine->code().code());
@@ -1757 +1723 @@
         MacroAssembler::repatchCall(CodeLocationCall<ReturnAddressPtrTag>(MacroAssemblerCodePtr<ReturnAddressPtrTag>(returnAddress)), FunctionPtr<OperationPtrTag>(operationHasIndexedPropertyGeneric));
     }
-}
-
-void JIT::emit_op_has_enumerable_indexed_property(const Instruction* currentInstruction)
-{
-    auto bytecode = currentInstruction->as<OpHasEnumerableIndexedProperty>();
-    auto& metadata = bytecode.metadata(m_codeBlock);
-    VirtualRegister dst = bytecode.m_dst;
-    VirtualRegister base = bytecode.m_base;
-    VirtualRegister property = bytecode.m_property;
-    ArrayProfile* profile = &metadata.m_arrayProfile;
-    ByValInfo* byValInfo = m_codeBlock->addByValInfo(m_bytecodeIndex);
-    
-    emitGetVirtualRegisters(base, regT0, property, regT1);
-
-    emitJumpSlowCaseIfNotInt(regT1);
-
-    // This is technically incorrect - we're zero-extending an int32. On the hot path this doesn't matter.
-    // We check the value as if it was a uint32 against the m_vectorLength - which will always fail if
-    // number was signed since m_vectorLength is always less than intmax (since the total allocation
-    // size is always less than 4Gb). As such zero extending will have been correct (and extending the value
-    // to 64-bits is necessary since it's used in the address calculation. We zero extend rather than sign
-    // extending since it makes it easier to re-tag the value in the slow case.
-    zeroExtend32ToWord(regT1, regT1);
-
-    emitJumpSlowCaseIfNotJSCell(regT0, base);
-    emitArrayProfilingSiteWithCell(regT0, regT2, profile);
-    and32(TrustedImm32(IndexingShapeMask), regT2);
-
-    JITArrayMode mode = chooseArrayMode(profile);
-    PatchableJump badType;
-
-    // FIXME: Add support for other types like TypedArrays and Arguments.
-    // See https://bugs.webkit.org/show_bug.cgi?id=135033 and https://bugs.webkit.org/show_bug.cgi?id=135034.
-    JumpList slowCases = emitLoadForArrayMode(currentInstruction, mode, badType, byValInfo);
-    
-    move(TrustedImm64(JSValue::encode(jsBoolean(true))), regT0);
-
-    if (!JITCode::useDataIC(JITType::BaselineJIT))
-        addSlowCase(badType);
-    addSlowCase(slowCases);
-    
-    Label done = label();
-    
-    emitPutVirtualRegister(dst);
-
-    Label nextHotPath = label();
-    
-    m_byValCompilationInfo.append(ByValCompilationInfo(byValInfo, m_bytecodeIndex, PatchableJump(), badType, mode, profile, done, nextHotPath));
-}
-
-void JIT::emitSlow_op_has_enumerable_indexed_property(const Instruction* currentInstruction, Vector<SlowCaseEntry>::iterator& iter)
-{
-    linkAllSlowCases(iter);
-
-    auto bytecode = currentInstruction->as<OpHasEnumerableIndexedProperty>();
-    VirtualRegister dst = bytecode.m_dst;
-    VirtualRegister base = bytecode.m_base;
-    VirtualRegister property = bytecode.m_property;
-    ByValInfo* byValInfo = m_byValCompilationInfo[m_byValInstructionIndex].byValInfo;
-
-    Label slowPath = label();
-    
-    emitGetVirtualRegister(base, regT0);
-    emitGetVirtualRegister(property, regT1);
-
-    Call call;
-    if (JITCode::useDataIC(JITType::BaselineJIT)) {
-        byValInfo->m_slowOperation = operationHasIndexedPropertyDefault;
-        move(TrustedImmPtr(byValInfo), GPRInfo::nonArgGPR0);
-        callOperation<decltype(operationHasIndexedPropertyDefault)>(Address(GPRInfo::nonArgGPR0, ByValInfo::offsetOfSlowOperation()), dst, TrustedImmPtr(m_codeBlock->globalObject()), regT0, regT1, GPRInfo::nonArgGPR0);
-    } else
-        call = callOperation(operationHasIndexedPropertyDefault, dst, TrustedImmPtr(m_codeBlock->globalObject()), regT0, regT1, byValInfo);
-
-    m_byValCompilationInfo[m_byValInstructionIndex].slowPathTarget = slowPath;
-    m_byValCompilationInfo[m_byValInstructionIndex].returnAddress = call;
-    m_byValInstructionIndex++;
-}
-
-void JIT::emit_op_get_direct_pname(const Instruction* currentInstruction)
-{
-    auto bytecode = currentInstruction->as<OpGetDirectPname>();
-    VirtualRegister dst = bytecode.m_dst;
-    VirtualRegister base = bytecode.m_base;
-    VirtualRegister index = bytecode.m_index;
-    VirtualRegister enumerator = bytecode.m_enumerator;
-
-    // Check that base is a cell
-    emitGetVirtualRegister(base, regT0);
-    emitJumpSlowCaseIfNotJSCell(regT0, base);
-
-    // Check the structure
-    emitGetVirtualRegister(enumerator, regT2);
-    load32(Address(regT0, JSCell::structureIDOffset()), regT1);
-    addSlowCase(branch32(NotEqual, regT1, Address(regT2, JSPropertyNameEnumerator::cachedStructureIDOffset())));
-
-    // Compute the offset
-    emitGetVirtualRegister(index, regT1);
-    // If index is less than the enumerator's cached inline storage, then it's an inline access
-    Jump outOfLineAccess = branch32(AboveOrEqual, regT1, Address(regT2, JSPropertyNameEnumerator::cachedInlineCapacityOffset()));
-    addPtr(TrustedImm32(JSObject::offsetOfInlineStorage()), regT0);
-    signExtend32ToPtr(regT1, regT1);
-    load64(BaseIndex(regT0, regT1, TimesEight), regT0);
-    
-    Jump done = jump();
-
-    // Otherwise it's out of line
-    outOfLineAccess.link(this);
-    loadPtr(Address(regT0, JSObject::butterflyOffset()), regT0);
-    sub32(Address(regT2, JSPropertyNameEnumerator::cachedInlineCapacityOffset()), regT1);
-    neg32(regT1);
-    signExtend32ToPtr(regT1, regT1);
-    int32_t offsetOfFirstProperty = static_cast<int32_t>(offsetInButterfly(firstOutOfLineOffset)) * sizeof(EncodedJSValue);
-    load64(BaseIndex(regT0, regT1, TimesEight, offsetOfFirstProperty), regT0);
-    
-    done.link(this);
-    emitValueProfilingSite(bytecode.metadata(m_codeBlock), regT0);
-    emitPutVirtualRegister(dst, regT0);
-}
-
-void JIT::emit_op_enumerator_structure_pname(const Instruction* currentInstruction)
-{
-    auto bytecode = currentInstruction->as<OpEnumeratorStructurePname>();
-    VirtualRegister dst = bytecode.m_dst;
-    VirtualRegister enumerator = bytecode.m_enumerator;
-    VirtualRegister index = bytecode.m_index;
-
-    emitGetVirtualRegister(index, regT0);
-    emitGetVirtualRegister(enumerator, regT1);
-    Jump inBounds = branch32(Below, regT0, Address(regT1, JSPropertyNameEnumerator::endStructurePropertyIndexOffset()));
-
-    move(TrustedImm64(JSValue::encode(jsNull())), regT0);
-
-    Jump done = jump();
-    inBounds.link(this);
-
-    loadPtr(Address(regT1, JSPropertyNameEnumerator::cachedPropertyNamesVectorOffset()), regT1);
-    signExtend32ToPtr(regT0, regT0);
-    load64(BaseIndex(regT1, regT0, TimesEight), regT0);
-
-    done.link(this);
-    emitPutVirtualRegister(dst);
-}
-
-void JIT::emit_op_enumerator_generic_pname(const Instruction* currentInstruction)
-{
-    auto bytecode = currentInstruction->as<OpEnumeratorGenericPname>();
-    VirtualRegister dst = bytecode.m_dst;
-    VirtualRegister enumerator = bytecode.m_enumerator;
-    VirtualRegister index = bytecode.m_index;
-
-    emitGetVirtualRegister(index, regT0);
-    emitGetVirtualRegister(enumerator, regT1);
-    Jump inBounds = branch32(Below, regT0, Address(regT1, JSPropertyNameEnumerator::endGenericPropertyIndexOffset()));
-
-    move(TrustedImm64(JSValue::encode(jsNull())), regT0);
-
-    Jump done = jump();
-    inBounds.link(this);
-
-    loadPtr(Address(regT1, JSPropertyNameEnumerator::cachedPropertyNamesVectorOffset()), regT1);
-    signExtend32ToPtr(regT0, regT0);
-    load64(BaseIndex(regT1, regT0, TimesEight), regT0);
-    
-    done.link(this);
-    emitPutVirtualRegister(dst);
 }
 

trunk/Source/JavaScriptCore/jit/JITOpcodes32_64.cpp

@@ -1143 +1143 @@
 }
 
-template <typename OpCodeType>
-void JIT::emit_op_has_structure_propertyImpl(const Instruction* currentInstruction)
-{
-    auto bytecode = currentInstruction->as<OpCodeType>();
-    VirtualRegister dst = bytecode.m_dst;
-    VirtualRegister base = bytecode.m_base;
-    VirtualRegister enumerator = bytecode.m_enumerator;
-
-    emitLoadPayload(base, regT0);
-    emitJumpSlowCaseIfNotJSCell(base);
-
-    emitLoadPayload(enumerator, regT1);
-
-    load32(Address(regT0, JSCell::structureIDOffset()), regT0);
-    addSlowCase(branch32(NotEqual, regT0, Address(regT1, JSPropertyNameEnumerator::cachedStructureIDOffset())));
-    
-    move(TrustedImm32(1), regT0);
-    emitStoreBool(dst, regT0);
-}
-
-void JIT::emit_op_has_enumerable_structure_property(const Instruction* currentInstruction)
-{
-    emit_op_has_structure_propertyImpl<OpHasEnumerableStructureProperty>(currentInstruction);
-}
-
-void JIT::emit_op_has_own_structure_property(const Instruction* currentInstruction)
-{
-    emit_op_has_structure_propertyImpl<OpHasOwnStructureProperty>(currentInstruction);
-}
-
-void JIT::emit_op_in_structure_property(const Instruction* currentInstruction)
-{
-    emit_op_has_structure_propertyImpl<OpInStructureProperty>(currentInstruction);
-}
-
 void JIT::privateCompileHasIndexedProperty(ByValInfo* byValInfo, ReturnAddressPtr returnAddress, JITArrayMode arrayMode)
 {
     const Instruction* currentInstruction = m_codeBlock->instructions().at(byValInfo->bytecodeIndex).ptr();
-    
+
     PatchableJump badType;
-    
+
     // FIXME: Add support for other types like TypedArrays and Arguments.
     // See https://bugs.webkit.org/show_bug.cgi?id=135033 and https://bugs.webkit.org/show_bug.cgi?id=135034.
@@ -1191 +1156 @@
 
     LinkBuffer patchBuffer(*this, m_codeBlock, LinkBuffer::Profile::InlineCache);
-    
+
     patchBuffer.link(badType, byValInfo->slowPathTarget);
     patchBuffer.link(slowCases, byValInfo->slowPathTarget);
@@ -1200 +1165 @@
         m_codeBlock, patchBuffer, JITStubRoutinePtrTag,
         "Baseline has_indexed_property stub for %s, return point %p", toCString(*m_codeBlock).data(), returnAddress.untaggedValue());
-    
+
     MacroAssembler::repatchJump(byValInfo->m_badTypeJump, CodeLocationLabel<JITStubRoutinePtrTag>(byValInfo->stubRoutine->code().code()));
     MacroAssembler::repatchCall(CodeLocationCall<ReturnAddressPtrTag>(MacroAssemblerCodePtr<ReturnAddressPtrTag>(returnAddress)), FunctionPtr<OperationPtrTag>(operationHasIndexedPropertyGeneric));
-}
-
-void JIT::emit_op_has_enumerable_indexed_property(const Instruction* currentInstruction)
-{
-    auto bytecode = currentInstruction->as<OpHasEnumerableIndexedProperty>();
-    auto& metadata = bytecode.metadata(m_codeBlock);
-    VirtualRegister dst = bytecode.m_dst;
-    VirtualRegister base = bytecode.m_base;
-    VirtualRegister property = bytecode.m_property;
-    ArrayProfile* profile = &metadata.m_arrayProfile;
-    ByValInfo* byValInfo = m_codeBlock->addByValInfo(m_bytecodeIndex);
-    
-    emitLoadPayload(base, regT0);
-    emitJumpSlowCaseIfNotJSCell(base);
-
-    emitLoad(property, regT3, regT1);
-    addSlowCase(branchIfNotInt32(regT3));
-
-    // This is technically incorrect - we're zero-extending an int32. On the hot path this doesn't matter.
-    // We check the value as if it was a uint32 against the m_vectorLength - which will always fail if
-    // number was signed since m_vectorLength is always less than intmax (since the total allocation
-    // size is always less than 4Gb). As such zero extending will have been correct (and extending the value
-    // to 64-bits is necessary since it's used in the address calculation. We zero extend rather than sign
-    // extending since it makes it easier to re-tag the value in the slow case.
-    zeroExtend32ToWord(regT1, regT1);
-
-    emitArrayProfilingSiteWithCell(regT0, regT2, profile);
-    and32(TrustedImm32(IndexingShapeMask), regT2);
-
-    JITArrayMode mode = chooseArrayMode(profile);
-    PatchableJump badType;
-
-    // FIXME: Add support for other types like TypedArrays and Arguments.
-    // See https://bugs.webkit.org/show_bug.cgi?id=135033 and https://bugs.webkit.org/show_bug.cgi?id=135034.
-    JumpList slowCases = emitLoadForArrayMode(currentInstruction, mode, badType, byValInfo);
-    move(TrustedImm32(1), regT0);
-
-    addSlowCase(badType);
-    addSlowCase(slowCases);
-    
-    Label done = label();
-    
-    emitStoreBool(dst, regT0);
-
-    Label nextHotPath = label();
-    
-    m_byValCompilationInfo.append(ByValCompilationInfo(byValInfo, m_bytecodeIndex, PatchableJump(), badType, mode, profile, done, nextHotPath));
-}
-
-void JIT::emitSlow_op_has_enumerable_indexed_property(const Instruction* currentInstruction, Vector<SlowCaseEntry>::iterator& iter)
-{
-    linkAllSlowCases(iter);
-
-    auto bytecode = currentInstruction->as<OpHasEnumerableIndexedProperty>();
-    VirtualRegister dst = bytecode.m_dst;
-    VirtualRegister base = bytecode.m_base;
-    VirtualRegister property = bytecode.m_property;
-    ByValInfo* byValInfo = m_byValCompilationInfo[m_byValInstructionIndex].byValInfo;
-
-    Label slowPath = label();
-    
-    emitLoad(base, regT1, regT0);
-    emitLoad(property, regT3, regT2);
-    Call call = callOperation(operationHasIndexedPropertyDefault, dst, m_codeBlock->globalObject(), JSValueRegs(regT1, regT0), JSValueRegs(regT3, regT2), byValInfo);
-
-    m_byValCompilationInfo[m_byValInstructionIndex].slowPathTarget = slowPath;
-    m_byValCompilationInfo[m_byValInstructionIndex].returnAddress = call;
-    m_byValInstructionIndex++;
-}
-
-void JIT::emit_op_get_direct_pname(const Instruction* currentInstruction)
-{
-    auto bytecode = currentInstruction->as<OpGetDirectPname>();
-    VirtualRegister dst = bytecode.m_dst;
-    VirtualRegister base = bytecode.m_base;
-    VirtualRegister index = bytecode.m_index;
-    VirtualRegister enumerator = bytecode.m_enumerator;
-
-    // Check that base is a cell
-    emitLoadPayload(base, regT0);
-    emitJumpSlowCaseIfNotJSCell(base);
-
-    // Check the structure
-    emitLoadPayload(enumerator, regT1);
-    load32(Address(regT0, JSCell::structureIDOffset()), regT2);
-    addSlowCase(branch32(NotEqual, regT2, Address(regT1, JSPropertyNameEnumerator::cachedStructureIDOffset())));
-
-    // Compute the offset
-    emitLoadPayload(index, regT2);
-    // If index is less than the enumerator's cached inline storage, then it's an inline access
-    Jump outOfLineAccess = branch32(AboveOrEqual, regT2, Address(regT1, JSPropertyNameEnumerator::cachedInlineCapacityOffset()));
-    addPtr(TrustedImm32(JSObject::offsetOfInlineStorage()), regT0);
-    load32(BaseIndex(regT0, regT2, TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.tag)), regT1);
-    load32(BaseIndex(regT0, regT2, TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.payload)), regT0);
-    
-    Jump done = jump();
-
-    // Otherwise it's out of line
-    outOfLineAccess.link(this);
-    loadPtr(Address(regT0, JSObject::butterflyOffset()), regT0);
-    sub32(Address(regT1, JSPropertyNameEnumerator::cachedInlineCapacityOffset()), regT2);
-    neg32(regT2);
-    int32_t offsetOfFirstProperty = static_cast<int32_t>(offsetInButterfly(firstOutOfLineOffset)) * sizeof(EncodedJSValue);
-    load32(BaseIndex(regT0, regT2, TimesEight, offsetOfFirstProperty + OBJECT_OFFSETOF(JSValue, u.asBits.tag)), regT1);
-    load32(BaseIndex(regT0, regT2, TimesEight, offsetOfFirstProperty + OBJECT_OFFSETOF(JSValue, u.asBits.payload)), regT0);
-    
-    done.link(this);
-    emitValueProfilingSite(bytecode.metadata(m_codeBlock), JSValueRegs(regT1, regT0));
-    emitStore(dst, regT1, regT0);
-}
-
-void JIT::emit_op_enumerator_structure_pname(const Instruction* currentInstruction)
-{
-    auto bytecode = currentInstruction->as<OpEnumeratorStructurePname>();
-    VirtualRegister dst = bytecode.m_dst;
-    VirtualRegister enumerator = bytecode.m_enumerator;
-    VirtualRegister index = bytecode.m_index;
-
-    emitLoadPayload(index, regT0);
-    emitLoadPayload(enumerator, regT1);
-    Jump inBounds = branch32(Below, regT0, Address(regT1, JSPropertyNameEnumerator::endStructurePropertyIndexOffset()));
-
-    move(TrustedImm32(JSValue::NullTag), regT2);
-    move(TrustedImm32(0), regT0);
-
-    Jump done = jump();
-    inBounds.link(this);
-
-    loadPtr(Address(regT1, JSPropertyNameEnumerator::cachedPropertyNamesVectorOffset()), regT1);
-    loadPtr(BaseIndex(regT1, regT0, ScalePtr), regT0);
-    move(TrustedImm32(JSValue::CellTag), regT2);
-
-    done.link(this);
-    emitStore(dst, regT2, regT0);
-}
-
-void JIT::emit_op_enumerator_generic_pname(const Instruction* currentInstruction)
-{
-    auto bytecode = currentInstruction->as<OpEnumeratorGenericPname>();
-    VirtualRegister dst = bytecode.m_dst;
-    VirtualRegister enumerator = bytecode.m_enumerator;
-    VirtualRegister index = bytecode.m_index;
-
-    emitLoadPayload(index, regT0);
-    emitLoadPayload(enumerator, regT1);
-    Jump inBounds = branch32(Below, regT0, Address(regT1, JSPropertyNameEnumerator::endGenericPropertyIndexOffset()));
-
-    move(TrustedImm32(JSValue::NullTag), regT2);
-    move(TrustedImm32(0), regT0);
-
-    Jump done = jump();
-    inBounds.link(this);
-
-    loadPtr(Address(regT1, JSPropertyNameEnumerator::cachedPropertyNamesVectorOffset()), regT1);
-    loadPtr(BaseIndex(regT1, regT0, ScalePtr), regT0);
-    move(TrustedImm32(JSValue::CellTag), regT2);
-    
-    done.link(this);
-    emitStore(dst, regT2, regT0);
 }
 

trunk/Source/JavaScriptCore/jit/JITPropertyAccess.cpp

@@ -37 +37 @@
 #include "LinkBuffer.h"
 #include "PrivateFieldPutKind.h"
+#include "ProbeContext.h"
 #include "SlowPathCall.h"
 #include "StructureStubInfo.h"
@@ -87 +88 @@
 }
 
-void JIT::emitSlow_op_get_by_val(const Instruction* currentInstruction, Vector<SlowCaseEntry>::iterator& iter)
+template<typename OpcodeType>
+void JIT::generateGetByValSlowCase(const OpcodeType& bytecode, Vector<SlowCaseEntry>::iterator& iter)
 {
     if (hasAnySlowCases(iter)) {
-        auto bytecode = currentInstruction->as<OpGetByVal>();
         VirtualRegister dst = bytecode.m_dst;
         auto& metadata = bytecode.metadata(m_codeBlock);
@@ -135 +136 @@
         gen.reportSlowPathCall(coldPathBegin, call);
     }
+}
+
+void JIT::emitSlow_op_get_by_val(const Instruction* currentInstruction, Vector<SlowCaseEntry>::iterator& iter)
+{
+    generateGetByValSlowCase(currentInstruction->as<OpGetByVal>(), iter);
 }
 
@@ -2978 +2984 @@
 template void JIT::emit_op_put_by_val<OpPutByVal>(const Instruction*);
 
+void JIT::emit_op_enumerator_next(const Instruction* currentInstruction)
+{
+    auto bytecode = currentInstruction->as<OpEnumeratorNext>();
+    auto& metadata = bytecode.metadata(m_codeBlock);
+
+    VirtualRegister base = bytecode.m_base;
+    VirtualRegister mode = bytecode.m_mode;
+    VirtualRegister index = bytecode.m_index;
+    VirtualRegister propertyName = bytecode.m_propertyName;
+    VirtualRegister enumerator = bytecode.m_enumerator;
+
+    JumpList done;
+    JumpList operationCases;
+
+    GPRReg modeGPR = regT0;
+    GPRReg indexGPR = regT1;
+    GPRReg baseGPR = regT2;
+
+    // This is the most common mode set we tend to see, so special case it if we profile it in the LLInt.
+    if (metadata.m_enumeratorMetadata == JSPropertyNameEnumerator::OwnStructureMode) {
+        GPRReg enumeratorGPR = regT3;
+        emitGetVirtualRegister(enumerator, enumeratorGPR);
+        operationCases.append(branchTest32(NonZero, Address(enumeratorGPR, JSPropertyNameEnumerator::modeSetOffset()), TrustedImm32(~JSPropertyNameEnumerator::OwnStructureMode)));
+        emitGetVirtualRegister(base, baseGPR);
+        load32(Address(enumeratorGPR, JSPropertyNameEnumerator::cachedStructureIDOffset()), indexGPR);
+        operationCases.append(branch32(NotEqual, indexGPR, Address(baseGPR, JSCell::structureIDOffset())));
+
+        emitGetVirtualRegister(mode, modeGPR);
+        emitGetVirtualRegister(index, indexGPR);
+        Jump notInit = branchTest32(Zero, modeGPR);
+        // Need to use add64 since this is a JSValue int32.
+        add64(TrustedImm32(1), indexGPR);
+        emitPutVirtualRegister(index, indexGPR);
+        notInit.link(this);
+        storeTrustedValue(jsNumber(static_cast<uint8_t>(JSPropertyNameEnumerator::OwnStructureMode)), addressFor(mode));
+
+        Jump outOfBounds = branch32(AboveOrEqual, indexGPR, Address(enumeratorGPR, JSPropertyNameEnumerator::endStructurePropertyIndexOffset()));
+        loadPtr(Address(enumeratorGPR, JSPropertyNameEnumerator::cachedPropertyNamesVectorOffset()), enumeratorGPR);
+        // We need to clear the high bits from the number encoding.
+        and32(TrustedImm32(-1), indexGPR);
+        loadPtr(BaseIndex(enumeratorGPR, indexGPR, ScalePtr), enumeratorGPR);
+
+        emitPutVirtualRegister(propertyName, enumeratorGPR);
+        done.append(jump());
+
+        outOfBounds.link(this);
+        storeTrustedValue(jsNull(), addressFor(propertyName));
+        done.append(jump());
+    }
+
+    operationCases.link(this);
+
+    JITSlowPathCall slowPathCall(this, currentInstruction, slow_path_enumerator_next);
+    slowPathCall.call();
+
+    done.link(this);
+}
+
+void JIT::emit_op_enumerator_get_by_val(const Instruction* currentInstruction)
+{
+    auto bytecode = currentInstruction->as<OpEnumeratorGetByVal>();
+    auto& metadata = bytecode.metadata(m_codeBlock);
+    VirtualRegister dst = bytecode.m_dst;
+    VirtualRegister mode = bytecode.m_mode;
+    VirtualRegister base = bytecode.m_base;
+    VirtualRegister index = bytecode.m_index;
+    VirtualRegister propertyName = bytecode.m_propertyName;
+    VirtualRegister enumerator = bytecode.m_enumerator;
+    ArrayProfile* profile = &metadata.m_arrayProfile;
+
+    JumpList doneCases;
+
+    auto resultGPR = regT0;
+
+    emitGetVirtualRegister(base, regT0);
+    emitGetVirtualRegister(mode, regT2);
+    emitGetVirtualRegister(propertyName, regT1);
+
+    or8(regT2, AbsoluteAddress(&metadata.m_enumeratorMetadata));
+
+    addSlowCase(branchIfNotCell(regT0));
+    // This is always an int32 encoded value.
+    Jump isNotOwnStructureMode = branchTest32(NonZero, regT2, TrustedImm32(JSPropertyNameEnumerator::IndexedMode | JSPropertyNameEnumerator::GenericMode));
+
+    // Check the structure
+    emitGetVirtualRegister(enumerator, regT2);
+    load32(Address(regT0, JSCell::structureIDOffset()), regT3);
+    Jump structureMismatch = branch32(NotEqual, regT3, Address(regT2, JSPropertyNameEnumerator::cachedStructureIDOffset()));
+
+    // Compute the offset.
+    emitGetVirtualRegister(index, regT3);
+    // If index is less than the enumerator's cached inline storage, then it's an inline access
+    Jump outOfLineAccess = branch32(AboveOrEqual, regT3, Address(regT2, JSPropertyNameEnumerator::cachedInlineCapacityOffset()));
+    addPtr(TrustedImm32(JSObject::offsetOfInlineStorage()), regT0);
+    signExtend32ToPtr(regT3, regT3);
+    load64(BaseIndex(regT0, regT3, TimesEight), resultGPR);
+
+    doneCases.append(jump());
+
+    // Otherwise it's out of line
+    outOfLineAccess.link(this);
+    loadPtr(Address(regT0, JSObject::butterflyOffset()), regT0);
+    sub32(Address(regT2, JSPropertyNameEnumerator::cachedInlineCapacityOffset()), regT3);
+    neg32(regT3);
+    signExtend32ToPtr(regT3, regT3);
+    int32_t offsetOfFirstProperty = static_cast<int32_t>(offsetInButterfly(firstOutOfLineOffset)) * sizeof(EncodedJSValue);
+    load64(BaseIndex(regT0, regT3, TimesEight, offsetOfFirstProperty), resultGPR);
+    doneCases.append(jump());
+
+    structureMismatch.link(this);
+    store8(TrustedImm32(JSPropertyNameEnumerator::HasSeenOwnStructureModeStructureMismatch), &metadata.m_enumeratorMetadata);
+
+    isNotOwnStructureMode.link(this);
+    Jump isNotIndexed = branchTest32(Zero, regT2, TrustedImm32(JSPropertyNameEnumerator::IndexedMode));
+    // Replace the string with the index.
+    emitGetVirtualRegister(index, regT1);
+
+    isNotIndexed.link(this);
+    emitArrayProfilingSiteWithCell(regT0, regT2, profile);
+
+    JITGetByValGenerator gen(
+        m_codeBlock, JITType::BaselineJIT, CodeOrigin(m_bytecodeIndex), CallSiteIndex(m_bytecodeIndex), AccessType::GetByVal, RegisterSet::stubUnavailableRegisters(),
+        JSValueRegs(regT0), JSValueRegs(regT1), JSValueRegs(resultGPR), regT2);
+    gen.generateFastPath(*this);
+    if (!JITCode::useDataIC(JITType::BaselineJIT))
+        addSlowCase(gen.slowPathJump());
+    else
+        addSlowCase();
+    m_getByVals.append(gen);
+
+    doneCases.link(this);
+
+    emitValueProfilingSite(metadata, JSValueRegs(resultGPR));
+    emitPutVirtualRegister(dst);
+}
+
+void JIT::emitSlow_op_enumerator_get_by_val(const Instruction* currentInstruction, Vector<SlowCaseEntry>::iterator& iter)
+{
+    generateGetByValSlowCase(currentInstruction->as<OpEnumeratorGetByVal>(), iter);
+}
+
+template <typename OpcodeType, typename SlowPathFunctionType>
+void JIT::emit_enumerator_has_propertyImpl(const Instruction* currentInstruction, const OpcodeType& bytecode, SlowPathFunctionType generalCase)
+{
+    auto& metadata = bytecode.metadata(m_codeBlock);
+    VirtualRegister dst = bytecode.m_dst;
+    VirtualRegister base = bytecode.m_base;
+    VirtualRegister enumerator = bytecode.m_enumerator;
+    VirtualRegister mode = bytecode.m_mode;
+
+    JumpList slowCases;
+
+    emitGetVirtualRegister(mode, regT0);
+    or8(regT0, AbsoluteAddress(&metadata.m_enumeratorMetadata));
+
+    slowCases.append(branchTest32(Zero, regT0, TrustedImm32(JSPropertyNameEnumerator::OwnStructureMode)));
+
+    emitGetVirtualRegister(base, regT0);
+
+    slowCases.append(branchIfNotCell(regT0));
+
+    emitGetVirtualRegister(enumerator, regT1);
+    load32(Address(regT0, JSCell::structureIDOffset()), regT0);
+    slowCases.append(branch32(NotEqual, regT0, Address(regT1, JSPropertyNameEnumerator::cachedStructureIDOffset())));
+
+    move(TrustedImm64(JSValue::encode(jsBoolean(true))), regT0);
+    emitPutVirtualRegister(dst);
+    Jump done = jump();
+
+    slowCases.link(this);
+
+    JITSlowPathCall slowPathCall(this, currentInstruction, generalCase);
+    slowPathCall.call();
+
+    done.link(this);
+}
+
+void JIT::emit_op_enumerator_in_by_val(const Instruction* currentInstruction)
+{
+    emit_enumerator_has_propertyImpl(currentInstruction, currentInstruction->as<OpEnumeratorInByVal>(), slow_path_enumerator_in_by_val);
+}
+
+void JIT::emit_op_enumerator_has_own_property(const Instruction* currentInstruction)
+{
+    emit_enumerator_has_propertyImpl(currentInstruction, currentInstruction->as<OpEnumeratorHasOwnProperty>(), slow_path_enumerator_has_own_property);
+}
+
 #else // USE(JSVALUE64)
 

trunk/Source/JavaScriptCore/jit/JITPropertyAccess32_64.cpp

@@ -1506 +1506 @@
 template void JIT::emit_op_put_by_val<OpPutByVal>(const Instruction*);
 
+void JIT::emit_op_enumerator_next(const Instruction* currentInstruction)
+{
+    JITSlowPathCall slowPathCall(this, currentInstruction, slow_path_enumerator_next);
+    slowPathCall.call();
+}
+
+void JIT::emit_op_enumerator_get_by_val(const Instruction* currentInstruction)
+{
+    JITSlowPathCall slowPathCall(this, currentInstruction, slow_path_enumerator_get_by_val);
+    slowPathCall.call();
+}
+
+void JIT::emitSlow_op_enumerator_get_by_val(const Instruction*, Vector<SlowCaseEntry>::iterator&)
+{
+    UNREACHABLE_FOR_PLATFORM();
+}
+
+void JIT::emit_op_enumerator_in_by_val(const Instruction* currentInstruction)
+{
+    JITSlowPathCall slowPathCall(this, currentInstruction, slow_path_enumerator_in_by_val);
+    slowPathCall.call();
+}
+
+void JIT::emit_op_enumerator_has_own_property(const Instruction* currentInstruction)
+{
+    JITSlowPathCall slowPathCall(this, currentInstruction, slow_path_enumerator_has_own_property);
+    slowPathCall.call();
+}
+
 } // namespace JSC
 

trunk/Source/JavaScriptCore/llint/LowLevelInterpreter.asm

@@ -1993 +1993 @@
 slowPathOp(define_accessor_property)
 slowPathOp(define_data_property)
-slowPathOp(enumerator_generic_pname)
-slowPathOp(enumerator_structure_pname)
 slowPathOp(get_by_val_with_this)
-slowPathOp(get_direct_pname)
-slowPathOp(get_enumerable_length)
 slowPathOp(get_property_enumerator)
-slowPathOp(has_enumerable_indexed_property)
-slowPathOp(has_enumerable_property)
+slowPathOp(enumerator_next)
+slowPathOp(enumerator_get_by_val)
+slowPathOp(enumerator_in_by_val)
+slowPathOp(enumerator_has_own_property)
 
 if not JSVALUE64
-    slowPathOp(has_enumerable_structure_property)
-    slowPathOp(has_own_structure_property)
-    slowPathOp(in_structure_property)
     slowPathOp(get_prototype_of)
 end
@@ -2022 +2017 @@
 slowPathOp(strcat)
 slowPathOp(throw_static_error)
-slowPathOp(to_index_string)
 slowPathOp(typeof)
 slowPathOp(typeof_is_object)

trunk/Source/JavaScriptCore/llint/LowLevelInterpreter64.asm

@@ -3142 +3142 @@
 end)
 
-macro hasStructurePropertyImpl(size, get, dispatch, return, slowPathCall)
-    get(m_base, t0)
-    loadConstantOrVariable(size, t0, t1)
-    btqnz t1, notCellMask, .slowPath
-
-    loadVariable(get, m_enumerator, t0)
-    loadi JSCell::m_structureID[t1], t1
-    bineq t1, JSPropertyNameEnumerator::m_cachedStructureID[t0], .slowPath
-
-    return(ValueTrue)
-
-.slowPath:
-    callSlowPath(slowPathCall)
-    dispatch()
-end
-
-llintOpWithReturn(op_has_enumerable_structure_property, OpHasEnumerableStructureProperty, macro (size, get, dispatch, return)
-    hasStructurePropertyImpl(size, get, dispatch,  return, _slow_path_has_enumerable_structure_property)
-end)
-
-llintOpWithReturn(op_has_own_structure_property, OpHasOwnStructureProperty, macro (size, get, dispatch, return)
-    hasStructurePropertyImpl(size, get, dispatch,  return, _slow_path_has_own_structure_property)
-end)
-
-llintOpWithReturn(op_in_structure_property, OpInStructureProperty, macro (size, get, dispatch, return)
-    hasStructurePropertyImpl(size, get, dispatch,  return, _slow_path_in_structure_property)
-end)
-
 op(fuzzer_return_early_from_loop_hint, macro ()
     loadp CodeBlock[cfr], t0

trunk/Source/JavaScriptCore/runtime/CommonSlowPaths.cpp

@@ -963 +963 @@
 }
 
-JSC_DEFINE_COMMON_SLOW_PATH(slow_path_get_enumerable_length)
-{
-    BEGIN();
-    auto bytecode = pc->as<OpGetEnumerableLength>();
-    JSValue enumeratorValue = GET_C(bytecode.m_base).jsValue();
-    if (enumeratorValue.isUndefinedOrNull())
-        RETURN(jsNumber(0));
-
-    JSPropertyNameEnumerator* enumerator = jsCast<JSPropertyNameEnumerator*>(enumeratorValue.asCell());
-
-    RETURN(jsNumber(enumerator->indexedLength()));
-}
-
-JSC_DEFINE_COMMON_SLOW_PATH(slow_path_has_enumerable_indexed_property)
-{
-    BEGIN();
-    auto bytecode = pc->as<OpHasEnumerableIndexedProperty>();
-    auto& metadata = bytecode.metadata(codeBlock);
-    JSObject* base = GET_C(bytecode.m_base).jsValue().toObject(globalObject);
-    CHECK_EXCEPTION();
-    JSValue property = GET(bytecode.m_property).jsValue();
-    metadata.m_arrayProfile.observeStructure(base->structure(vm));
-    ASSERT(property.isUInt32AsAnyInt());
-    RETURN(jsBoolean(base->hasEnumerableProperty(globalObject, property.asUInt32AsAnyInt())));
-}
-
-JSC_DEFINE_COMMON_SLOW_PATH(slow_path_has_enumerable_structure_property)
-{
-    BEGIN();
-    auto bytecode = pc->as<OpHasEnumerableStructureProperty>();
-    JSObject* base = GET_C(bytecode.m_base).jsValue().toObject(globalObject);
-    CHECK_EXCEPTION();
-    JSValue property = GET(bytecode.m_property).jsValue();
-    RELEASE_ASSERT(property.isString());
-#if USE(JSVALUE32_64)
-    JSPropertyNameEnumerator* enumerator = jsCast<JSPropertyNameEnumerator*>(GET(bytecode.m_enumerator).jsValue().asCell());
-    if (base->structure(vm)->id() == enumerator->cachedStructureID())
-        RETURN(jsBoolean(true));
-#endif
-    JSString* string = asString(property);
-    auto propertyName = string->toIdentifier(globalObject);
-    CHECK_EXCEPTION();
-    RETURN(jsBoolean(base->hasEnumerableProperty(globalObject, propertyName)));
-}
-
-JSC_DEFINE_COMMON_SLOW_PATH(slow_path_has_own_structure_property)
-{
-    BEGIN();
-    auto bytecode = pc->as<OpHasOwnStructureProperty>();
-    JSValue base = GET_C(bytecode.m_base).jsValue();
-
-#if USE(JSVALUE32_64)
-    JSPropertyNameEnumerator* enumerator = jsCast<JSPropertyNameEnumerator*>(GET(bytecode.m_enumerator).jsValue().asCell());
-    if (base.isCell() && base.asCell()->structure(vm)->id() == enumerator->cachedStructureID())
-        RETURN(jsBoolean(true));
-#endif
-
-    JSValue property = GET(bytecode.m_property).jsValue();
-    RELEASE_ASSERT(property.isString());
-    JSString* string = asString(property);
-    auto propertyName = string->toIdentifier(globalObject);
-    CHECK_EXCEPTION();
-
-    RETURN(jsBoolean(objectPrototypeHasOwnProperty(globalObject, base, propertyName)));
-}
-
-JSC_DEFINE_COMMON_SLOW_PATH(slow_path_in_structure_property)
-{
-    BEGIN();
-    auto bytecode = pc->as<OpInStructureProperty>();
-    JSValue base = GET_C(bytecode.m_base).jsValue();
-#if USE(JSVALUE32_64)
-    JSPropertyNameEnumerator* enumerator = jsCast<JSPropertyNameEnumerator*>(GET(bytecode.m_enumerator).jsValue().asCell());
-    if (base.isCell() && base.asCell()->structure(vm)->id() == enumerator->cachedStructureID())
-        RETURN(jsBoolean(true));
-#endif
-    JSValue property = GET(bytecode.m_property).jsValue();
-    RELEASE_ASSERT(property.isString());
-    RETURN(jsBoolean(CommonSlowPaths::opInByVal(globalObject, base, asString(property))));
-}
-
-JSC_DEFINE_COMMON_SLOW_PATH(slow_path_has_enumerable_property)
-{
-    BEGIN();
-    auto bytecode = pc->as<OpHasEnumerableProperty>();
-    JSObject* base = GET_C(bytecode.m_base).jsValue().toObject(globalObject);
-    CHECK_EXCEPTION();
-    JSValue property = GET(bytecode.m_property).jsValue();
-    ASSERT(property.isString());
-    JSString* string = asString(property);
-    auto propertyName = string->toIdentifier(globalObject);
-    CHECK_EXCEPTION();
-    RETURN(jsBoolean(base->hasEnumerableProperty(globalObject, propertyName)));
-}
-
-JSC_DEFINE_COMMON_SLOW_PATH(slow_path_get_direct_pname)
-{
-    BEGIN();
-    auto bytecode = pc->as<OpGetDirectPname>();
-    JSValue baseValue = GET_C(bytecode.m_base).jsValue();
-    JSValue property = GET(bytecode.m_property).jsValue();
-    ASSERT(property.isString());
-    JSString* string = asString(property);
-    auto propertyName = string->toIdentifier(globalObject);
-    CHECK_EXCEPTION();
-    RETURN(baseValue.get(globalObject, propertyName));
-}
-
 JSC_DEFINE_COMMON_SLOW_PATH(slow_path_get_property_enumerator)
 {
@@ -1085 +977 @@
 }
 
-JSC_DEFINE_COMMON_SLOW_PATH(slow_path_enumerator_structure_pname)
-{
-    BEGIN();
-    auto bytecode = pc->as<OpEnumeratorStructurePname>();
-    JSPropertyNameEnumerator* enumerator = jsCast<JSPropertyNameEnumerator*>(GET(bytecode.m_enumerator).jsValue().asCell());
-    uint32_t index = GET(bytecode.m_index).jsValue().asUInt32AsAnyInt();
-
-    JSString* propertyName = nullptr;
-    if (index < enumerator->endStructurePropertyIndex())
-        propertyName = enumerator->propertyNameAtIndex(index);
-    RETURN(propertyName ? propertyName : jsNull());
-}
-
-JSC_DEFINE_COMMON_SLOW_PATH(slow_path_enumerator_generic_pname)
-{
-    BEGIN();
-    auto bytecode = pc->as<OpEnumeratorGenericPname>();
-    JSPropertyNameEnumerator* enumerator = jsCast<JSPropertyNameEnumerator*>(GET(bytecode.m_enumerator).jsValue().asCell());
-    uint32_t index = GET(bytecode.m_index).jsValue().asUInt32AsAnyInt();
-
-    JSString* propertyName = nullptr;
-    if (enumerator->endStructurePropertyIndex() <= index && index < enumerator->endGenericPropertyIndex())
-        propertyName = enumerator->propertyNameAtIndex(index);
-    RETURN(propertyName ? propertyName : jsNull());
-}
-
-JSC_DEFINE_COMMON_SLOW_PATH(slow_path_to_index_string)
-{
-    BEGIN();
-    auto bytecode = pc->as<OpToIndexString>();
-    JSValue indexValue = GET(bytecode.m_index).jsValue();
-    ASSERT(indexValue.isUInt32AsAnyInt());
-    RETURN(jsString(vm, Identifier::from(vm, indexValue.asUInt32AsAnyInt()).string()));
+JSC_DEFINE_COMMON_SLOW_PATH(slow_path_enumerator_next)
+{
+    BEGIN();
+    auto bytecode = pc->as<OpEnumeratorNext>();
+    auto& metadata = bytecode.metadata(codeBlock);
+    Register& modeRegister = GET(bytecode.m_mode);
+    Register& indexRegister = GET(bytecode.m_index);
+    Register& nameRegister = GET(bytecode.m_propertyName);
+
+    auto mode = static_cast<JSPropertyNameEnumerator::Mode>(modeRegister.jsValue().asUInt32());
+    uint32_t index = indexRegister.jsValue().asUInt32();
+
+    JSPropertyNameEnumerator* enumerator = jsCast<JSPropertyNameEnumerator*>(GET(bytecode.m_enumerator).jsValue());
+    JSValue baseValue = GET(bytecode.m_base).jsValue();
+    ASSERT(!baseValue.isUndefinedOrNull());
+    JSObject* base = baseValue.toObject(globalObject);
+    CHECK_EXCEPTION();
+    metadata.m_arrayProfile.observeStructureID(base->structureID());
+
+    JSString* name = enumerator->computeNext(globalObject, base, index, mode);
+    CHECK_EXCEPTION();
+
+    metadata.m_enumeratorMetadata |= static_cast<uint8_t>(mode);
+    modeRegister = jsNumber(static_cast<uint8_t>(mode));
+    indexRegister = jsNumber(index);
+    nameRegister = name ? name : jsNull();
+    END();
+}
+
+JSC_DEFINE_COMMON_SLOW_PATH(slow_path_enumerator_get_by_val)
+{
+    BEGIN();
+    auto bytecode = pc->as<OpEnumeratorGetByVal>();
+    JSValue baseValue = GET_C(bytecode.m_base).jsValue();
+    auto& metadata = bytecode.metadata(codeBlock);
+    auto mode = static_cast<JSPropertyNameEnumerator::Mode>(GET(bytecode.m_mode).jsValue().asUInt32());
+    metadata.m_enumeratorMetadata |= static_cast<uint8_t>(mode);
+
+    JSPropertyNameEnumerator* enumerator = jsCast<JSPropertyNameEnumerator*>(GET(bytecode.m_enumerator).jsValue());
+    unsigned index = GET(bytecode.m_index).jsValue().asInt32();
+    switch (mode) {
+    case JSPropertyNameEnumerator::IndexedMode: {
+        if (LIKELY(baseValue.isCell()))
+            metadata.m_arrayProfile.observeStructureID(baseValue.asCell()->structureID());
+        RETURN_PROFILED(baseValue.get(globalObject, static_cast<unsigned>(index)));
+    }
+    case JSPropertyNameEnumerator::OwnStructureMode: {
+        if (LIKELY(baseValue.isCell()) && baseValue.asCell()->structureID() == enumerator->cachedStructureID()) {
+            // We'll only match the structure ID if the base is an object.
+            ASSERT(index < enumerator->endStructurePropertyIndex());
+            RETURN_PROFILED(baseValue.getObject()->getDirect(index < enumerator->cachedInlineCapacity() ? index : index - enumerator->cachedInlineCapacity() + firstOutOfLineOffset));
+        } else
+            metadata.m_enumeratorMetadata |= static_cast<uint8_t>(JSPropertyNameEnumerator::HasSeenOwnStructureModeStructureMismatch);
+        FALLTHROUGH;
+    }
+
+    case JSPropertyNameEnumerator::GenericMode: {
+        if (baseValue.isCell() && mode != JSPropertyNameEnumerator::OwnStructureMode)
+            metadata.m_arrayProfile.observeStructureID(baseValue.asCell()->structureID());
+        JSString* string = asString(GET(bytecode.m_propertyName).jsValue());
+        auto propertyName = string->toIdentifier(globalObject);
+        CHECK_EXCEPTION();
+        RETURN_PROFILED(baseValue.get(globalObject, propertyName));
+    }
+
+    default:
+        RELEASE_ASSERT_NOT_REACHED();
+        break;
+    };
+    RELEASE_ASSERT_NOT_REACHED();
+}
+
+JSC_DEFINE_COMMON_SLOW_PATH(slow_path_enumerator_in_by_val)
+{
+    BEGIN();
+    auto bytecode = pc->as<OpEnumeratorInByVal>();
+    JSValue baseValue = GET_C(bytecode.m_base).jsValue();
+    auto& metadata = bytecode.metadata(codeBlock);
+    auto mode = static_cast<JSPropertyNameEnumerator::Mode>(GET(bytecode.m_mode).jsValue().asUInt32());
+    metadata.m_enumeratorMetadata |= static_cast<uint8_t>(mode);
+
+    CHECK_EXCEPTION();
+    JSPropertyNameEnumerator* enumerator = jsCast<JSPropertyNameEnumerator*>(GET(bytecode.m_enumerator).jsValue());
+    if (auto* base = baseValue.getObject()) {
+        if (mode == JSPropertyNameEnumerator::OwnStructureMode && base->structureID() == enumerator->cachedStructureID())
+            RETURN(jsBoolean(true));
+
+        if (mode == JSPropertyNameEnumerator::IndexedMode)
+            RETURN(jsBoolean(base->hasProperty(globalObject, GET(bytecode.m_index).jsValue().asUInt32())));
+    }
+
+    JSString* string = asString(GET(bytecode.m_propertyName).jsValue());
+    RETURN(jsBoolean(CommonSlowPaths::opInByVal(globalObject, baseValue, string, &metadata.m_arrayProfile)));
+}
+
+JSC_DEFINE_COMMON_SLOW_PATH(slow_path_enumerator_has_own_property)
+{
+    BEGIN();
+    auto bytecode = pc->as<OpEnumeratorHasOwnProperty>();
+    JSValue baseValue = GET_C(bytecode.m_base).jsValue();
+    auto& metadata = bytecode.metadata(codeBlock);
+    auto mode = static_cast<JSPropertyNameEnumerator::Mode>(GET(bytecode.m_mode).jsValue().asUInt32());
+    metadata.m_enumeratorMetadata |= static_cast<uint8_t>(mode);
+
+    JSPropertyNameEnumerator* enumerator = jsCast<JSPropertyNameEnumerator*>(GET(bytecode.m_enumerator).jsValue());
+    if (auto* base = baseValue.getObject()) {
+        if (mode == JSPropertyNameEnumerator::OwnStructureMode && base->structureID() == enumerator->cachedStructureID())
+            RETURN(jsBoolean(true));
+
+        if (mode == JSPropertyNameEnumerator::IndexedMode)
+            RETURN(jsBoolean(base->hasOwnProperty(globalObject, GET(bytecode.m_index).jsValue().asUInt32())));
+    }
+
+    JSString* string = asString(GET(bytecode.m_propertyName).jsValue());
+    auto propertyName = string->toIdentifier(globalObject);
+    CHECK_EXCEPTION();
+    RETURN(jsBoolean(objectPrototypeHasOwnProperty(globalObject, baseValue, propertyName)));
 }
 

trunk/Source/JavaScriptCore/runtime/CommonSlowPaths.h

@@ -255 +255 @@
 JSC_DECLARE_COMMON_SLOW_PATH(slow_path_to_primitive);
 JSC_DECLARE_COMMON_SLOW_PATH(slow_path_to_property_key);
-JSC_DECLARE_COMMON_SLOW_PATH(slow_path_get_enumerable_length);
-JSC_DECLARE_COMMON_SLOW_PATH(slow_path_has_enumerable_indexed_property);
-JSC_DECLARE_COMMON_SLOW_PATH(slow_path_has_enumerable_structure_property);
-JSC_DECLARE_COMMON_SLOW_PATH(slow_path_has_enumerable_property);
-JSC_DECLARE_COMMON_SLOW_PATH(slow_path_has_own_structure_property);
-JSC_DECLARE_COMMON_SLOW_PATH(slow_path_in_structure_property);
-JSC_DECLARE_COMMON_SLOW_PATH(slow_path_get_direct_pname);
 JSC_DECLARE_COMMON_SLOW_PATH(slow_path_get_property_enumerator);
-JSC_DECLARE_COMMON_SLOW_PATH(slow_path_enumerator_structure_pname);
-JSC_DECLARE_COMMON_SLOW_PATH(slow_path_enumerator_generic_pname);
-JSC_DECLARE_COMMON_SLOW_PATH(slow_path_to_index_string);
+JSC_DECLARE_COMMON_SLOW_PATH(slow_path_enumerator_next);
+JSC_DECLARE_COMMON_SLOW_PATH(slow_path_enumerator_get_by_val);
+JSC_DECLARE_COMMON_SLOW_PATH(slow_path_enumerator_in_by_val);
+JSC_DECLARE_COMMON_SLOW_PATH(slow_path_enumerator_has_own_property);
 JSC_DECLARE_COMMON_SLOW_PATH(slow_path_profile_type_clear_log);
 JSC_DECLARE_COMMON_SLOW_PATH(slow_path_unreachable);

trunk/Source/JavaScriptCore/runtime/FileBasedFuzzerAgent.cpp

@@ -91 +91 @@
 
     case op_get_by_val_with_this:
-    case op_get_direct_pname:
     case op_construct:
     case op_construct_varargs:

trunk/Source/JavaScriptCore/runtime/FileBasedFuzzerAgentBase.cpp

@@ -1 +1 @@
 /*
- * Copyright (C) 2019 Apple Inc. All rights reserved.
+ * Copyright (C) 2019-2021 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -53 +53 @@
     if (opcodeId == op_call_varargs || opcodeId == op_call_eval || opcodeId == op_tail_call || opcodeId == op_tail_call_varargs)
         return op_call;
-    if (opcodeId == op_get_direct_pname || opcodeId == op_get_by_val_with_this)
+    if (opcodeId == op_enumerator_get_by_val || opcodeId == op_get_by_val_with_this)
         return op_get_by_val;
     if (opcodeId == op_construct_varargs)

trunk/Source/JavaScriptCore/runtime/JSGlobalObject.cpp

@@ -1462 +1462 @@
 #endif
 
+    m_linkTimeConstants[static_cast<unsigned>(LinkTimeConstant::emptyPropertyNameEnumerator)].initLater([] (const Initializer<JSCell>& init) {
+        init.set(init.vm.emptyPropertyNameEnumerator());
+    });
+
     if (Options::exposeProfilersOnGlobalObject()) {
 #if ENABLE(SAMPLING_PROFILER)

trunk/Source/JavaScriptCore/runtime/JSPropertyNameEnumerator.cpp

@@ -57 +57 @@
     , m_cachedInlineCapacity(structure ? structure->inlineCapacity() : 0)
 {
+    if (m_indexedLength)
+        m_modeSet |= JSPropertyNameEnumerator::IndexedMode;
+    if (m_endStructurePropertyIndex)
+        m_modeSet |= JSPropertyNameEnumerator::OwnStructureMode;
+    if (m_endGenericPropertyIndex - m_endStructurePropertyIndex)
+        m_modeSet |= JSPropertyNameEnumerator::GenericMode;
 }
 
@@ -149 +155 @@
 }
 
+JSString* JSPropertyNameEnumerator::computeNext(JSGlobalObject* globalObject, JSObject* base, uint32_t& index, Mode& mode, bool shouldAllocateIndexedNameString)
+{
+    VM& vm = globalObject->vm();
+    auto scope = DECLARE_THROW_SCOPE(vm);
+
+    ASSERT(indexedLength() || sizeOfPropertyNames());
+
+    index++;
+    switch (mode) {
+    case InitMode: {
+        mode = IndexedMode;
+        index = 0;
+        FALLTHROUGH;
+    }
+
+    case JSPropertyNameEnumerator::IndexedMode: {
+        while (index < indexedLength() && !base->hasEnumerableProperty(globalObject, index)) {
+            RETURN_IF_EXCEPTION(scope, nullptr);
+            index++;
+        }
+        scope.assertNoException();
+
+        if (index < indexedLength())
+            return shouldAllocateIndexedNameString ? jsString(vm, Identifier::from(vm, index).string()) : nullptr;
+
+        if (!sizeOfPropertyNames())
+            return nullptr;
+
+        mode = OwnStructureMode;
+        index = 0;
+        FALLTHROUGH;
+    }
+
+    case JSPropertyNameEnumerator::OwnStructureMode:
+    case JSPropertyNameEnumerator::GenericMode: {
+        JSString* name = nullptr;
+        while (true) {
+            if (index >= sizeOfPropertyNames())
+                break;
+            name = propertyNameAtIndex(index);
+            if (!name)
+                break;
+            if (index < endStructurePropertyIndex() && base->structureID() == cachedStructureID())
+                break;
+            auto id = Identifier::fromString(vm, name->value(globalObject));
+            RETURN_IF_EXCEPTION(scope, nullptr);
+            if (base->hasEnumerableProperty(globalObject, id))
+                break;
+            RETURN_IF_EXCEPTION(scope, nullptr);
+            name = nullptr;
+            index++;
+        }
+        scope.assertNoException();
+
+        if (index >= endStructurePropertyIndex() && index < sizeOfPropertyNames())
+            mode = GenericMode;
+        return name;
+    }
+
+    default:
+        break;
+    }
+    RELEASE_ASSERT_NOT_REACHED();
+    return nullptr;
+}
+
 } // namespace JSC

trunk/Source/JavaScriptCore/runtime/JSPropertyNameEnumerator.h

@@ -29 +29 @@
 #include "Operations.h"
 #include "PropertyNameArray.h"
-#include "Structure.h"
+#include "StructureChain.h"
 
 namespace JSC {
@@ -37 +37 @@
     using Base = JSCell;
     static constexpr unsigned StructureFlags = Base::StructureFlags | StructureIsImmortal;
+
+    enum Mode : uint8_t {
+        InitMode = 0,
+        IndexedMode = 1 << 0,
+        OwnStructureMode = 1 << 1,
+        GenericMode = 1 << 2,
+        // Profiling Only
+        HasSeenOwnStructureModeStructureMismatch = 1 << 3,
+    };
+
+    static constexpr uint8_t enumerationModeMask = (GenericMode << 1) - 1;
 
     template<typename CellType, SubspaceAccess mode>
@@ -75 +86 @@
     uint32_t cachedInlineCapacity() const { return m_cachedInlineCapacity; }
     uint32_t sizeOfPropertyNames() const { return endGenericPropertyIndex(); }
+    uint32_t modeSet() const { return m_modeSet; }
     static ptrdiff_t cachedStructureIDOffset() { return OBJECT_OFFSETOF(JSPropertyNameEnumerator, m_cachedStructureID); }
     static ptrdiff_t indexedLengthOffset() { return OBJECT_OFFSETOF(JSPropertyNameEnumerator, m_indexedLength); }
@@ -80 +92 @@
     static ptrdiff_t endGenericPropertyIndexOffset() { return OBJECT_OFFSETOF(JSPropertyNameEnumerator, m_endGenericPropertyIndex); }
     static ptrdiff_t cachedInlineCapacityOffset() { return OBJECT_OFFSETOF(JSPropertyNameEnumerator, m_cachedInlineCapacity); }
+    static ptrdiff_t modeSetOffset() { return OBJECT_OFFSETOF(JSPropertyNameEnumerator, m_modeSet); }
     static ptrdiff_t cachedPropertyNamesVectorOffset()
     {
         return OBJECT_OFFSETOF(JSPropertyNameEnumerator, m_propertyNames);
     }
+
+    JSString* computeNext(JSGlobalObject*, JSObject* base, uint32_t& currentIndex, Mode&, bool shouldAllocateIndexedNameString = true);
 
     DECLARE_VISIT_CHILDREN;
@@ -100 +115 @@
     uint32_t m_endGenericPropertyIndex;
     uint32_t m_cachedInlineCapacity;
+    uint32_t m_modeSet { 0 };
 };
 
@@ -135 +151 @@
     }
 
-    enumerator = JSPropertyNameEnumerator::create(vm, structureAfterGettingPropertyNames, indexedLength, numberStructureProperties, WTFMove(propertyNames));
+    if (!indexedLength && !propertyNames.size())
+        enumerator = vm.emptyPropertyNameEnumerator();
+    else
+        enumerator = JSPropertyNameEnumerator::create(vm, structureAfterGettingPropertyNames, indexedLength, numberStructureProperties, WTFMove(propertyNames));
     if (!indexedLength && successfullyNormalizedChain && structureAfterGettingPropertyNames == structure) {
         enumerator->setCachedPrototypeChain(vm, structure->prototypeChain(globalObject, base));
@@ -144 +163 @@
 }
 
+using EnumeratorMetadata = std::underlying_type_t<JSPropertyNameEnumerator::Mode>;
+
 } // namespace JSC

trunk/Source/JavaScriptCore/runtime/PredictionFileCreatingFuzzerAgent.cpp

@@ -1 +1 @@
 /*
- * Copyright (C) 2019 Apple Inc. All rights reserved.
+ * Copyright (C) 2019-2021 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -53 +53 @@
     case op_get_by_id_with_this:
     case op_get_by_val_with_this:
-    case op_get_direct_pname:
+    case op_enumerator_get_by_val:
     case op_construct:
     case op_construct_varargs: