⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 205462 in webkit


Ignore:
Timestamp:
Sep 5, 2016, 6:02:22 PM (10 years ago)
Author:
fpizlo@apple.com
Message:

Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
​https://bugs.webkit.org/show_bug.cgi?id=160125

Reviewed by Geoffrey Garen and Keith Miller.
JSTests:


Most of the things I did properly covered by existing tests, but I found some simple cases of
unshifting that had sketchy coverage.

  • stress/array-storage-array-unshift.js: Added.
  • stress/contiguous-array-unshift.js: Added.
  • stress/double-array-unshift.js: Added.
  • stress/int32-array-unshift.js: Added.

Source/bmalloc:


I needed to tryMemalign, so I added such a thing.

  • bmalloc/Allocator.cpp:

(bmalloc::Allocator::allocate):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):

  • bmalloc/Allocator.h:
  • bmalloc/Cache.h:

(bmalloc::Cache::tryAllocate):

  • bmalloc/bmalloc.h:

(bmalloc::api::tryMemalign):

Source/JavaScriptCore:

In order to make the GC concurrent (bug 149432), we would either need to enable concurrent
copying or we would need to not copy. Concurrent copying carries a 1-2% throughput overhead
from the barriers alone. Considering that MarkedSpace does a decent job of avoiding
fragmentation, it's unlikely that it's worth paying 1-2% throughput for copying. So, we want
to get rid of copied space. This change moves copied space's biggest client over to marked
space.

Moving butterflies to marked space means having them use the new Auxiliary HeapCell
allocation path. This is a fairly mechanical change, but it caused performance regressions
everywhere, so this change also fixes MarkedSpace's performance issues.

At a high level the mechanical changes are:

  • We use AuxiliaryBarrier instead of CopyBarrier.


  • We use tryAllocateAuxiliary instead of tryAllocateStorage. I got rid of the silly CheckedBoolean stuff, since it's so much more trouble than it's worth.


  • The JITs have to emit inlined marked space allocations instead of inline copy space allocations.


  • Everyone has to get used to zeroing their butterflies after allocation instead of relying on them being pre-zeroed by the GC. Copied space would zero things for you, while marked space doesn't.


That's about 1/3 of this change. But this led to performance problems, which I fixed with
optimizations that amounted to a major MarkedSpace rewrite:

  • MarkedSpace always causes internal fragmentation for array allocations because the vector length we choose when we resize usually leads to a cell size that doesn't correspond to any size class. I got around this by making array allocations usually round up vectorLength to the maximum allowed by the size class that we would have allocated in. Also, ensureLengthSlow() and friends first make sure that the requested length can't just be fulfilled with the current allocation size. This safeguard means that not every array allocation has to do size class queries. For example, the fast path of new Array(length) never does any size class queries, under the assumption that (1) the speed gained from avoiding an ensureLengthSlow() call, which then just changes the vectorLength by doing the size class query, is too small to offset the speed lost by doing the query on every allocation and (2) new Array(length) is a pretty good hint that resizing is not very likely.


  • Size classes in MarkedSpace were way too precise, which led to external fragmentation. This changes MarkedSpace size classes to use a linear progression for very small sizes followed by a geometric progression that naturally transitions to a hyperbolic progression. We want hyperbolic sizes when we get close to blockSize: for example the largest size we want is payloadSize / 2 rounded down, to ensure we get exactly two cells with minimal slop. The next size down should be payloadSize / 3 rounded down, and so on. After the last precise size (80 bytes), we proceed using a geometric progression, but round up each size to minimize slop at the end of the block. This naturally causes the geometric progression to turn hyperbolic for large sizes. The size class configuration happens at VM start-up, so it can be controlled with runtime options. I found that a base of 1.4 works pretty well.


  • Large allocations caused massive internal fragmentation, since the smallest large allocation had to use exactly blockSize, and the largest small allocation used blockSize / 2. The next size up - the first large allocation size to require two blocks - also had 50% internal fragmentation. This is because we required large allocations to be blockSize aligned, so that MarkedBlock::blockFor() would work. I decided to rewrite all of that. Cells no longer have to be owned by a MarkedBlock. They can now alternatively be owned by a LargeAllocation. These two things are abstracted as CellContainer. You know that a cell is owned by a LargeAllocation if the MarkedBlock::atomSize / 2 bit is set. Basically, large allocations are deliberately misaligned by 8 bytes. This actually works out great since (1) typed arrays won't use large allocations anyway since they have their own malloc fallback and (2) large array butterflies already have a 8 byte header, which means that the 8 byte base misalignment aligns the large array payload on a 16 byte boundary. I took extreme care to make sure that the isLargeAllocation bit checks are as rare as possible; for example, ExecState::vm() skips the check because we know that callees must be small allocations. It's also possible to use template tricks to do one check for cell container kind, and then invoke a function specialized for MarkedBlock or a function specialized for LargeAllocation. LargeAllocation includes stubs for all MarkedBlock methods that get used from functions that are template-specialized like this. That's mostly to speed up the GC marking code. Most other code can use CellContainer API or HeapCell API directly. That's another thing: HeapCell, the common base of JSCell and auxiliary allocations, is now smart enough to do a lot of things for you, like HeapCell::vm(), HeapCell::heap(), HeapCell::isLargeAllocation(), and HeapCell::cellContainer(). The size cutoff for large allocations is runtime-configurable, so long as you don't choose something so small that callees end up large. I found that 400 bytes is roughly optimal. This means that the MarkedBlock size classes end up being:


16, 32, 48, 64, 80, 112, 160, 224, 320


The next size class would have been 432, but that's above the 400 byte cutoff. All of this
is configurable with --sizeClassProgression and --largeAllocationCutoff. You can see what
size classes you end up with by doing --dumpSizeClasses=true.


  • Copied space uses 64KB blocks, while marked space used to use 16KB blocks. Allocating a lot of stuff in 16KB blocks was slower than allocating it in 64KB blocks because the GC had a lot of per-block overhead. I removed this overhead: It's now 2x faster to scan all MarkedBlocks because the list that contains the interesting meta-data is allocated on the side, for better locality during a sequential walk. It's no longer necessary to scan MarkedBlocks to find WeakSets, since the sets of WeakSets for eden scan and full scan are maintained on-the-fly. It's no longer necessary to scan all MarkedBlocks to clear mark bits because we now use versioned mark bits: to clear then, just increment the 64-bit heap version. It's no longer necessary to scan retired MarkedBlocks while allocating because marking retires them on-the-fly. It's no longer necessary to sort all blocks in the IncrementalSweeper's snapshot because blocks now know if they are in the snapshot. Put together, these optimizations allowed me to reduce block size to 16KB without losing much performance. There is some small perf loss on JetStream/splay, but not enough to hurt JetStream overall. I tried reducing block sizes further, to 4KB, since that is a progression on membuster. That's not possible yet, since there is still enough per-block overhead yet that such a reduction hurts JetStream too much. I filed a bug about improving this further: ​https://bugs.webkit.org/show_bug.cgi?id=161581.


  • Even after all of that, copying butterflies was still faster because it allowed us to skip sweeping dead space. A good GC allocates over dead bytes without explicitly freeing them, so the GC pause is O(size of live), not O(size of live + dead). O(dead) is usually much larger than O(live), especially in an eden collection. Copying satisfies this premise while mark+sweep does not. So, I invented a new kind of allocator: bump'n'pop. Previously, our MarkedSpace allocator was a freelist pop. That's simple and easy to inline but requires that we walk the block to build a free list. This means walking dead space. The new allocator allows totally free MarkedBlocks to simply set up a bump-pointer arena instead. The allocator is a hybrid of bump-pointer and freelist pop. It tries bump first. The bump pointer always bumps by cellSize, so the result of filling a block with bumping looks as if we had used freelist popping to fill it. Additionally, each MarkedBlock now has a bit to quickly tell if the block is entirely free. This makes sweeping O(1) whenever a MarkedBlock is completely empty, which is the common case because of the generational hypothesis: the number of objects that survive an eden collection is a tiny fraction of the number of objects that had been allocated, and this fraction is so small that there are typically fewer than one survivors per MarkedBlock. This change was enough to make this change a net win over tip-of-tree.


  • FTL now shares the same allocation fast paths as everything else, which is great, because bump'n'pop has gnarly control flow. We don't really want B3 to have to think about that control flow, since it won't be able to improve the machine code we write ourselves. GC fast paths are best written in assembly. So, I've empowered B3 to have even better support for Patchpoint terminals. It's now totally fine for a Patchpoint terminal to be non-Void. So, the new FTL allocation fast paths are just Patchpoint terminals that call through to AssemblyHelpers::emitAllocate(). B3 still reasons about things like constant-folding the size class calculation and constant-hoisting the allocator. Also, I gave the FTL the ability to constant-fold some allocator logic (in case we first assume that we're doing a variable-length allocation but then realize that the length is known). I think it makes sense to have constant folding rules in FTL::Output, or whatever the B3 IR builder is, since this makes lowering easier (you can constant fold during lowering more easily) and it reduces the amount of malloc traffic. In the future, we could teach B3 how to better constant-fold this code. That would require allowing loads to be constant-folded, which is doable but hella tricky.


  • It used to be that if a logical object allocation required two physical allocations (first the butterfly and then the cell), then the JIT would emit the code in such a way that a failure in the second fast path would cause us to forget the successful first physical allocation. This was pointlessly wasteful. It turns out that it's very cheap to devote a register to storing either the butterfly or null, because the butterfly register is anyway going to be free inside the first allocation. The only overhead here is zeroing the butterfly register. With that in place, we can just pass the butterfly-or-null to the slow path, which can then either allocate a butterfly or not. So now we never waste a successful allocation. This patch implements such a solution both in DFG (where it's easy to do this since we control registers already) and in FTL (where it's annoying, because mutable "butterfly-or-null" variables are hard to say in SSA; also I realized that we had code duplicated the JSArray allocation utility, so I deduplicated it). This came up because in one version of this patch, this wastage would resonate with some Kraken benchmark: the benchmark would always allocate N small things followed by one bigger thing. The problem was I accidentally adjusted the various fixed overheads in MarkedBlock in such a way that the JSObject size class, which both the small and big thing shared for their cell, could hold exactly N cells per MarkedBlock. Then the benchmark would always call slow path when it allocated the big thing. So, it would end up having to allocate the big thing's large butterfly twice, every single time! Ouch!


  • It used to be that we zeroed CopiedBlocks using memset, and so array allocations enjoyed amortization of the cost of zeroing. This doesn't work anymore - it's now up to the client of the allocator to initialize the object to whatever state they need. It used to be that we would just use a dumb loop. I initially changed this so that we would end up in memset for large allocations, but this didn't actually help performance that much. I got a much better result by playing with different memsets written in assembly. First I wrote one using non-temporal stores. That was a small speed-up over memset. Then I tried the classic "rep stos" approach, and holy cow that version was fast. It's a ~20% speed-up on array allocation microbenchmarks. So, this patch adds code paths to do "rep stos" on x86_64, or memset, or use a loop, as appropriate, for both "contiguous" arrays (holes are zero) and double arrays (holes are PNaN). Note that the JIT always emits either a loop or a flat slab of stores (if the size is known), but those paths in the JIT won't trigger for NewArrayWithSize() if the size is large, since that takes us to the operationNewArrayWithSize() slow path, which calls into JSArray::create(). That's why the optimizations here are all in JSArray::create() - that's the hot place for large arrays that need to be filled with holes.


All of this put together gives us neutral perf on JetStream, membuster, and PLT3, a ~1%
regression on Speedometer, and up to a 4% regression Kraken. The Kraken regression is
because Kraken was allocating exactly 1024 element arrays at a rate of 400MB/sec. This is a
best-case scenario for bump allocation. I think that we should fix bmalloc to make up the
difference, but take the hit for now because it's a crazy corner case. By comparison, the
alternative approach of using a copy barrier would have cost us 1-2%. That's the real
apples-to-apples comparison if your premise is that we should have a concurrent GC. After we
finish removing copied space, we will be barrier-ready for concurrent GC: we already have a
marking barrier and we simply won't need a copying barrier. This change gets us there for
the purposes of our benchmarks, since the remaining clients of copied space are not very
important. On the other hand, if we keep copying, then getting barrier-ready would mean
adding back the copy barrier, which costs more perf.

We might get bigger speed-ups once we remove CopiedSpace altogether. That requires moving
typed arrays and a few other weird things over to Aux MarkedSpace.

This also includes some header sanitization. The introduction of AuxiliaryBarrier, HeapCell,
and CellContainer meant that I had to include those files from everywhere. Fortunately,
just including JSCInlines.h (instead of manually including the files that includes) is
usually enough. So, I made most of JSC's cpp files include JSCInlines.h, which is something
that we were already basically doing. In places where JSCInlines.h would be too much, I just
included HeapInlines.h. This got weird, because we previously included HeapInlines.h from
JSObject.h. That's bad because it led to some circular dependencies, so I fixed it - but that
meant having to manually include HeapInlines.h from the places that previously got it
implicitly via JSObject.h. But that led to more problems for some reason: I started getting
build errors because non-JSC files were having trouble including Opcode.h. That's just silly,
since Opcode.h is meant to be an internal JSC header. So, I made it an internal header and
made it impossible to include it from outside JSC. This was a lot of work, but it was
necessary to get the patch to build on all ports. It's also a net win. There were many places
in WebCore that were transitively including a *ton* of JSC headers just because of the
JSObject.h->HeapInlines.h edge and a bunch of dependency edges that arose from some public
(for WebCore) JSC headers needing Interpreter.h or Opcode.h for bad reasons.

  • API/JSManagedValue.mm:

(-[JSManagedValue initWithValue:]):

  • API/JSTypedArray.cpp:
  • API/ObjCCallbackFunction.mm:
  • API/tests/testapi.mm:

(testObjectiveCAPI):
(testWeakValue): Deleted.

  • CMakeLists.txt:
  • JavaScriptCore.xcodeproj/project.pbxproj:
  • Scripts/builtins/builtins_generate_combined_implementation.py:

(BuiltinsCombinedImplementationGenerator.generate_secondary_header_includes):

  • Scripts/builtins/builtins_generate_internals_wrapper_implementation.py:

(BuiltinsInternalsWrapperImplementationGenerator.generate_secondary_header_includes):

  • Scripts/builtins/builtins_generate_separate_implementation.py:

(BuiltinsSeparateImplementationGenerator.generate_secondary_header_includes):

  • assembler/AbstractMacroAssembler.h:

(JSC::AbstractMacroAssembler::JumpList::link):
(JSC::AbstractMacroAssembler::JumpList::linkTo):

  • assembler/MacroAssembler.h:
  • assembler/MacroAssemblerARM64.h:

(JSC::MacroAssemblerARM64::add32):

  • assembler/MacroAssemblerCodeRef.cpp: Added.

(JSC::MacroAssemblerCodePtr::createLLIntCodePtr):
(JSC::MacroAssemblerCodePtr::dumpWithName):
(JSC::MacroAssemblerCodePtr::dump):
(JSC::MacroAssemblerCodeRef::createLLIntCodeRef):
(JSC::MacroAssemblerCodeRef::dump):

  • assembler/MacroAssemblerCodeRef.h:

(JSC::MacroAssemblerCodePtr::createLLIntCodePtr): Deleted.
(JSC::MacroAssemblerCodePtr::dumpWithName): Deleted.
(JSC::MacroAssemblerCodePtr::dump): Deleted.
(JSC::MacroAssemblerCodeRef::createLLIntCodeRef): Deleted.
(JSC::MacroAssemblerCodeRef::dump): Deleted.

  • b3/B3BasicBlock.cpp:

(JSC::B3::BasicBlock::appendBoolConstant):

  • b3/B3BasicBlock.h:
  • b3/B3DuplicateTails.cpp:
  • b3/B3StackmapGenerationParams.h:
  • b3/testb3.cpp:

(JSC::B3::testPatchpointTerminalReturnValue):
(JSC::B3::run):

  • bindings/ScriptValue.cpp:
  • bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp:
  • bytecode/BytecodeBasicBlock.cpp:
  • bytecode/BytecodeLivenessAnalysis.cpp:
  • bytecode/BytecodeUseDef.h:
  • bytecode/CallLinkInfo.cpp:

(JSC::CallLinkInfo::callTypeFor):

  • bytecode/CallLinkInfo.h:

(JSC::CallLinkInfo::callTypeFor): Deleted.

  • bytecode/CallLinkStatus.cpp:
  • bytecode/CodeBlock.cpp:

(JSC::CodeBlock::finishCreation):
(JSC::CodeBlock::clearLLIntGetByIdCache):
(JSC::CodeBlock::predictedMachineCodeSize):

  • bytecode/CodeBlock.h:

(JSC::CodeBlock::jitCodeMap): Deleted.
(JSC::clearLLIntGetByIdCache): Deleted.

  • bytecode/ExecutionCounter.h:
  • bytecode/Instruction.h:
  • bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp:

(JSC::LLIntPrototypeLoadAdaptiveStructureWatchpoint::fireInternal):

  • bytecode/ObjectAllocationProfile.h:

(JSC::ObjectAllocationProfile::isNull):
(JSC::ObjectAllocationProfile::initialize):

  • bytecode/Opcode.h:

(JSC::padOpcodeName):

  • bytecode/PolymorphicAccess.cpp:

(JSC::AccessCase::generateImpl):
(JSC::PolymorphicAccess::regenerate):

  • bytecode/PolymorphicAccess.h:
  • bytecode/PreciseJumpTargets.cpp:
  • bytecode/StructureStubInfo.cpp:
  • bytecode/StructureStubInfo.h:
  • bytecode/UnlinkedCodeBlock.cpp:

(JSC::UnlinkedCodeBlock::vm): Deleted.

  • bytecode/UnlinkedCodeBlock.h:
  • bytecode/UnlinkedInstructionStream.cpp:
  • bytecode/UnlinkedInstructionStream.h:
  • dfg/DFGOperations.cpp:
  • dfg/DFGSpeculativeJIT.cpp:

(JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
(JSC::DFG::SpeculativeJIT::compileMakeRope):
(JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):

  • dfg/DFGSpeculativeJIT.h:

(JSC::DFG::SpeculativeJIT::emitAllocateJSCell):
(JSC::DFG::SpeculativeJIT::emitAllocateJSObject):

  • dfg/DFGSpeculativeJIT32_64.cpp:

(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):

  • dfg/DFGSpeculativeJIT64.cpp:

(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):

  • dfg/DFGStrengthReductionPhase.cpp:

(JSC::DFG::StrengthReductionPhase::handleNode):

  • ftl/FTLAbstractHeapRepository.h:
  • ftl/FTLCompile.cpp:
  • ftl/FTLJITFinalizer.cpp:
  • ftl/FTLLowerDFGToB3.cpp:

(JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments):
(JSC::FTL::DFG::LowerDFGToB3::compileCreateRest):
(JSC::FTL::DFG::LowerDFGToB3::allocateArrayWithSize):
(JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize):
(JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
(JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
(JSC::FTL::DFG::LowerDFGToB3::initializeArrayElements):
(JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
(JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell):
(JSC::FTL::DFG::LowerDFGToB3::allocateCell):
(JSC::FTL::DFG::LowerDFGToB3::allocateObject):
(JSC::FTL::DFG::LowerDFGToB3::allocatorForSize):
(JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedObject):
(JSC::FTL::DFG::LowerDFGToB3::allocateJSArray):
(JSC::FTL::DFG::LowerDFGToB3::compileAllocateArrayWithSize): Deleted.

  • ftl/FTLOutput.cpp:

(JSC::FTL::Output::constBool):
(JSC::FTL::Output::add):
(JSC::FTL::Output::shl):
(JSC::FTL::Output::aShr):
(JSC::FTL::Output::lShr):
(JSC::FTL::Output::zeroExt):
(JSC::FTL::Output::equal):
(JSC::FTL::Output::notEqual):
(JSC::FTL::Output::above):
(JSC::FTL::Output::aboveOrEqual):
(JSC::FTL::Output::below):
(JSC::FTL::Output::belowOrEqual):
(JSC::FTL::Output::greaterThan):
(JSC::FTL::Output::greaterThanOrEqual):
(JSC::FTL::Output::lessThan):
(JSC::FTL::Output::lessThanOrEqual):
(JSC::FTL::Output::select):
(JSC::FTL::Output::appendSuccessor):
(JSC::FTL::Output::addIncomingToPhi):

  • ftl/FTLOutput.h:
  • ftl/FTLValueFromBlock.h:

(JSC::FTL::ValueFromBlock::operator bool):
(JSC::FTL::ValueFromBlock::ValueFromBlock): Deleted.

  • ftl/FTLWeightedTarget.h:

(JSC::FTL::WeightedTarget::frequentedBlock):

  • heap/CellContainer.h: Added.

(JSC::CellContainer::CellContainer):
(JSC::CellContainer::operator bool):
(JSC::CellContainer::isMarkedBlock):
(JSC::CellContainer::isLargeAllocation):
(JSC::CellContainer::markedBlock):
(JSC::CellContainer::largeAllocation):

  • heap/CellContainerInlines.h: Added.

(JSC::CellContainer::isMarked):
(JSC::CellContainer::isMarkedOrNewlyAllocated):
(JSC::CellContainer::noteMarked):
(JSC::CellContainer::cellSize):
(JSC::CellContainer::weakSet):
(JSC::CellContainer::flipIfNecessary):

  • heap/ConservativeRoots.cpp:

(JSC::ConservativeRoots::ConservativeRoots):
(JSC::ConservativeRoots::~ConservativeRoots):
(JSC::ConservativeRoots::grow):
(JSC::ConservativeRoots::genericAddPointer):
(JSC::ConservativeRoots::genericAddSpan):

  • heap/ConservativeRoots.h:

(JSC::ConservativeRoots::roots):

  • heap/CopyToken.h:
  • heap/FreeList.cpp: Added.

(JSC::FreeList::dump):

  • heap/FreeList.h: Added.

(JSC::FreeList::FreeList):
(JSC::FreeList::list):
(JSC::FreeList::bump):
(JSC::FreeList::operator==):
(JSC::FreeList::operator!=):
(JSC::FreeList::operator bool):
(JSC::FreeList::allocationWillFail):
(JSC::FreeList::allocationWillSucceed):

  • heap/GCTypeMap.h: Added.

(JSC::GCTypeMap::operator[]):

  • heap/Heap.cpp:

(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalizeUnconditionalFinalizers):
(JSC::Heap::markRoots):
(JSC::Heap::copyBackingStores):
(JSC::Heap::gatherStackRoots):
(JSC::Heap::gatherJSStackRoots):
(JSC::Heap::gatherScratchBufferRoots):
(JSC::Heap::clearLivenessData):
(JSC::Heap::visitSmallStrings):
(JSC::Heap::visitConservativeRoots):
(JSC::Heap::removeDeadCompilerWorklistEntries):
(JSC::Heap::gatherExtraHeapSnapshotData):
(JSC::Heap::removeDeadHeapSnapshotNodes):
(JSC::Heap::visitProtectedObjects):
(JSC::Heap::visitArgumentBuffers):
(JSC::Heap::visitException):
(JSC::Heap::visitStrongHandles):
(JSC::Heap::visitHandleStack):
(JSC::Heap::visitSamplingProfiler):
(JSC::Heap::traceCodeBlocksAndJITStubRoutines):
(JSC::Heap::converge):
(JSC::Heap::visitWeakHandles):
(JSC::Heap::updateObjectCounts):
(JSC::Heap::clearUnmarkedExecutables):
(JSC::Heap::deleteUnmarkedCompiledCode):
(JSC::Heap::collectAllGarbage):
(JSC::Heap::collect):
(JSC::Heap::collectWithoutAnySweep):
(JSC::Heap::collectImpl):
(JSC::Heap::suspendCompilerThreads):
(JSC::Heap::willStartCollection):
(JSC::Heap::flushOldStructureIDTables):
(JSC::Heap::flushWriteBarrierBuffer):
(JSC::Heap::stopAllocation):
(JSC::Heap::prepareForMarking):
(JSC::Heap::reapWeakHandles):
(JSC::Heap::pruneStaleEntriesFromWeakGCMaps):
(JSC::Heap::sweepArrayBuffers):
(JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor):
(JSC::MarkedBlockSnapshotFunctor::operator()):
(JSC::Heap::snapshotMarkedSpace):
(JSC::Heap::deleteSourceProviderCaches):
(JSC::Heap::notifyIncrementalSweeper):
(JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks):
(JSC::Heap::resetAllocators):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::didFinishCollection):
(JSC::Heap::resumeCompilerThreads):
(JSC::Zombify::visit):
(JSC::Heap::forEachCodeBlockImpl):

  • heap/Heap.h:

(JSC::Heap::allocatorForObjectWithoutDestructor):
(JSC::Heap::allocatorForObjectWithDestructor):
(JSC::Heap::allocatorForAuxiliaryData):
(JSC::Heap::jitStubRoutines):
(JSC::Heap::codeBlockSet):
(JSC::Heap::storageAllocator): Deleted.

  • heap/HeapCell.h:

(JSC::HeapCell::isZapped): Deleted.

  • heap/HeapCellInlines.h: Added.

(JSC::HeapCell::isLargeAllocation):
(JSC::HeapCell::cellContainer):
(JSC::HeapCell::markedBlock):
(JSC::HeapCell::largeAllocation):
(JSC::HeapCell::heap):
(JSC::HeapCell::vm):
(JSC::HeapCell::cellSize):
(JSC::HeapCell::allocatorAttributes):
(JSC::HeapCell::destructionMode):
(JSC::HeapCell::cellKind):

  • heap/HeapInlines.h:

(JSC::Heap::heap):
(JSC::Heap::isLive):
(JSC::Heap::isMarked):
(JSC::Heap::testAndSetMarked):
(JSC::Heap::setMarked):
(JSC::Heap::cellSize):
(JSC::Heap::forEachCodeBlock):
(JSC::Heap::allocateObjectOfType):
(JSC::Heap::subspaceForObjectOfType):
(JSC::Heap::allocatorForObjectOfType):
(JSC::Heap::allocateAuxiliary):
(JSC::Heap::tryAllocateAuxiliary):
(JSC::Heap::tryReallocateAuxiliary):
(JSC::Heap::isPointerGCObject): Deleted.
(JSC::Heap::isValueGCObject): Deleted.

  • heap/HeapOperation.cpp: Added.

(WTF::printInternal):

  • heap/HeapOperation.h:
  • heap/HeapUtil.h: Added.

(JSC::HeapUtil::findGCObjectPointersForMarking):
(JSC::HeapUtil::isPointerGCObjectJSCell):
(JSC::HeapUtil::isValueGCObject):

  • heap/IncrementalSweeper.cpp:

(JSC::IncrementalSweeper::sweepNextBlock):

  • heap/IncrementalSweeper.h:
  • heap/LargeAllocation.cpp: Added.

(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::LargeAllocation):
(JSC::LargeAllocation::lastChanceToFinalize):
(JSC::LargeAllocation::shrink):
(JSC::LargeAllocation::visitWeakSet):
(JSC::LargeAllocation::reapWeakSet):
(JSC::LargeAllocation::flip):
(JSC::LargeAllocation::isEmpty):
(JSC::LargeAllocation::sweep):
(JSC::LargeAllocation::destroy):
(JSC::LargeAllocation::dump):

  • heap/LargeAllocation.h: Added.

(JSC::LargeAllocation::fromCell):
(JSC::LargeAllocation::cell):
(JSC::LargeAllocation::isLargeAllocation):
(JSC::LargeAllocation::heap):
(JSC::LargeAllocation::vm):
(JSC::LargeAllocation::weakSet):
(JSC::LargeAllocation::clearNewlyAllocated):
(JSC::LargeAllocation::isNewlyAllocated):
(JSC::LargeAllocation::isMarked):
(JSC::LargeAllocation::isMarkedOrNewlyAllocated):
(JSC::LargeAllocation::isLive):
(JSC::LargeAllocation::hasValidCell):
(JSC::LargeAllocation::cellSize):
(JSC::LargeAllocation::aboveLowerBound):
(JSC::LargeAllocation::belowUpperBound):
(JSC::LargeAllocation::contains):
(JSC::LargeAllocation::attributes):
(JSC::LargeAllocation::flipIfNecessary):
(JSC::LargeAllocation::flipIfNecessaryConcurrently):
(JSC::LargeAllocation::testAndSetMarked):
(JSC::LargeAllocation::setMarked):
(JSC::LargeAllocation::clearMarked):
(JSC::LargeAllocation::noteMarked):
(JSC::LargeAllocation::headerSize):

  • heap/MarkedAllocator.cpp:

(JSC::MarkedAllocator::MarkedAllocator):
(JSC::MarkedAllocator::isPagedOut):
(JSC::MarkedAllocator::retire):
(JSC::MarkedAllocator::filterNextBlock):
(JSC::MarkedAllocator::setNextBlockToSweep):
(JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl):
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::allocateSlowCase):
(JSC::MarkedAllocator::tryAllocateSlowCase):
(JSC::MarkedAllocator::allocateSlowCaseImpl):
(JSC::blockHeaderSize):
(JSC::MarkedAllocator::blockSizeForBytes):
(JSC::MarkedAllocator::tryAllocateBlock):
(JSC::MarkedAllocator::addBlock):
(JSC::MarkedAllocator::removeBlock):
(JSC::MarkedAllocator::stopAllocating):
(JSC::MarkedAllocator::reset):
(JSC::MarkedAllocator::lastChanceToFinalize):
(JSC::MarkedAllocator::setFreeList):
(JSC::isListPagedOut): Deleted.
(JSC::MarkedAllocator::tryAllocateHelper): Deleted.
(JSC::MarkedAllocator::tryPopFreeList): Deleted.
(JSC::MarkedAllocator::tryAllocate): Deleted.
(JSC::MarkedAllocator::allocateBlock): Deleted.

  • heap/MarkedAllocator.h:

(JSC::MarkedAllocator::takeLastActiveBlock):
(JSC::MarkedAllocator::offsetOfFreeList):
(JSC::MarkedAllocator::offsetOfCellSize):
(JSC::MarkedAllocator::tryAllocate):
(JSC::MarkedAllocator::allocate):
(JSC::MarkedAllocator::forEachBlock):
(JSC::MarkedAllocator::offsetOfFreeListHead): Deleted.
(JSC::MarkedAllocator::MarkedAllocator): Deleted.
(JSC::MarkedAllocator::init): Deleted.
(JSC::MarkedAllocator::stopAllocating): Deleted.

  • heap/MarkedBlock.cpp:

(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::MarkedBlock):
(JSC::MarkedBlock::Handle::specializedSweep):
(JSC::MarkedBlock::Handle::sweep):
(JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode):
(JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode):
(JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated):
(JSC::SetNewlyAllocatedFunctor::SetNewlyAllocatedFunctor):
(JSC::SetNewlyAllocatedFunctor::operator()):
(JSC::MarkedBlock::Handle::stopAllocating):
(JSC::MarkedBlock::Handle::lastChanceToFinalize):
(JSC::MarkedBlock::Handle::resumeAllocating):
(JSC::MarkedBlock::Handle::zap):
(JSC::MarkedBlock::Handle::forEachFreeCell):
(JSC::MarkedBlock::flipIfNecessary):
(JSC::MarkedBlock::Handle::flipIfNecessary):
(JSC::MarkedBlock::flipIfNecessarySlow):
(JSC::MarkedBlock::flipIfNecessaryConcurrentlySlow):
(JSC::MarkedBlock::clearMarks):
(JSC::MarkedBlock::assertFlipped):
(JSC::MarkedBlock::needsFlip):
(JSC::MarkedBlock::Handle::needsFlip):
(JSC::MarkedBlock::Handle::willRemoveBlock):
(JSC::MarkedBlock::Handle::didConsumeFreeList):
(JSC::MarkedBlock::markCount):
(JSC::MarkedBlock::Handle::isEmpty):
(JSC::MarkedBlock::clearHasAnyMarked):
(JSC::MarkedBlock::noteMarkedSlow):
(WTF::printInternal):
(JSC::MarkedBlock::create): Deleted.
(JSC::MarkedBlock::destroy): Deleted.
(JSC::MarkedBlock::callDestructor): Deleted.
(JSC::MarkedBlock::specializedSweep): Deleted.
(JSC::MarkedBlock::sweep): Deleted.
(JSC::MarkedBlock::sweepHelper): Deleted.
(JSC::MarkedBlock::stopAllocating): Deleted.
(JSC::MarkedBlock::clearMarksWithCollectionType): Deleted.
(JSC::MarkedBlock::lastChanceToFinalize): Deleted.
(JSC::MarkedBlock::resumeAllocating): Deleted.
(JSC::MarkedBlock::didRetireBlock): Deleted.

  • heap/MarkedBlock.h:

(JSC::MarkedBlock::VoidFunctor::returnValue):
(JSC::MarkedBlock::CountFunctor::CountFunctor):
(JSC::MarkedBlock::CountFunctor::count):
(JSC::MarkedBlock::CountFunctor::returnValue):
(JSC::MarkedBlock::Handle::hasAnyNewlyAllocated):
(JSC::MarkedBlock::Handle::isOnBlocksToSweep):
(JSC::MarkedBlock::Handle::setIsOnBlocksToSweep):
(JSC::MarkedBlock::Handle::state):
(JSC::MarkedBlock::needsDestruction):
(JSC::MarkedBlock::handle):
(JSC::MarkedBlock::Handle::block):
(JSC::MarkedBlock::firstAtom):
(JSC::MarkedBlock::atoms):
(JSC::MarkedBlock::isAtomAligned):
(JSC::MarkedBlock::Handle::cellAlign):
(JSC::MarkedBlock::blockFor):
(JSC::MarkedBlock::Handle::allocator):
(JSC::MarkedBlock::Handle::heap):
(JSC::MarkedBlock::Handle::vm):
(JSC::MarkedBlock::vm):
(JSC::MarkedBlock::Handle::weakSet):
(JSC::MarkedBlock::weakSet):
(JSC::MarkedBlock::Handle::shrink):
(JSC::MarkedBlock::Handle::visitWeakSet):
(JSC::MarkedBlock::Handle::reapWeakSet):
(JSC::MarkedBlock::Handle::cellSize):
(JSC::MarkedBlock::cellSize):
(JSC::MarkedBlock::Handle::attributes):
(JSC::MarkedBlock::attributes):
(JSC::MarkedBlock::Handle::needsDestruction):
(JSC::MarkedBlock::Handle::destruction):
(JSC::MarkedBlock::Handle::cellKind):
(JSC::MarkedBlock::Handle::markCount):
(JSC::MarkedBlock::Handle::size):
(JSC::MarkedBlock::atomNumber):
(JSC::MarkedBlock::flipIfNecessary):
(JSC::MarkedBlock::flipIfNecessaryConcurrently):
(JSC::MarkedBlock::Handle::flipIfNecessary):
(JSC::MarkedBlock::Handle::flipIfNecessaryConcurrently):
(JSC::MarkedBlock::Handle::flipForEdenCollection):
(JSC::MarkedBlock::assertFlipped):
(JSC::MarkedBlock::Handle::assertFlipped):
(JSC::MarkedBlock::isMarked):
(JSC::MarkedBlock::testAndSetMarked):
(JSC::MarkedBlock::Handle::isNewlyAllocated):
(JSC::MarkedBlock::Handle::setNewlyAllocated):
(JSC::MarkedBlock::Handle::clearNewlyAllocated):
(JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::Handle::isLive):
(JSC::MarkedBlock::isAtom):
(JSC::MarkedBlock::Handle::isLiveCell):
(JSC::MarkedBlock::Handle::forEachCell):
(JSC::MarkedBlock::Handle::forEachLiveCell):
(JSC::MarkedBlock::Handle::forEachDeadCell):
(JSC::MarkedBlock::Handle::needsSweeping):
(JSC::MarkedBlock::Handle::isAllocated):
(JSC::MarkedBlock::Handle::isMarked):
(JSC::MarkedBlock::Handle::isFreeListed):
(JSC::MarkedBlock::hasAnyMarked):
(JSC::MarkedBlock::noteMarked):
(WTF::MarkedBlockHash::hash):
(JSC::MarkedBlock::FreeList::FreeList): Deleted.
(JSC::MarkedBlock::allocator): Deleted.
(JSC::MarkedBlock::heap): Deleted.
(JSC::MarkedBlock::shrink): Deleted.
(JSC::MarkedBlock::visitWeakSet): Deleted.
(JSC::MarkedBlock::reapWeakSet): Deleted.
(JSC::MarkedBlock::willRemoveBlock): Deleted.
(JSC::MarkedBlock::didConsumeFreeList): Deleted.
(JSC::MarkedBlock::markCount): Deleted.
(JSC::MarkedBlock::isEmpty): Deleted.
(JSC::MarkedBlock::destruction): Deleted.
(JSC::MarkedBlock::cellKind): Deleted.
(JSC::MarkedBlock::size): Deleted.
(JSC::MarkedBlock::capacity): Deleted.
(JSC::MarkedBlock::setMarked): Deleted.
(JSC::MarkedBlock::clearMarked): Deleted.
(JSC::MarkedBlock::isNewlyAllocated): Deleted.
(JSC::MarkedBlock::setNewlyAllocated): Deleted.
(JSC::MarkedBlock::clearNewlyAllocated): Deleted.
(JSC::MarkedBlock::isLive): Deleted.
(JSC::MarkedBlock::isLiveCell): Deleted.
(JSC::MarkedBlock::forEachCell): Deleted.
(JSC::MarkedBlock::forEachLiveCell): Deleted.
(JSC::MarkedBlock::forEachDeadCell): Deleted.
(JSC::MarkedBlock::needsSweeping): Deleted.
(JSC::MarkedBlock::isAllocated): Deleted.
(JSC::MarkedBlock::isMarkedOrRetired): Deleted.

  • heap/MarkedSpace.cpp:

(JSC::MarkedSpace::initializeSizeClassForStepSize):
(JSC::MarkedSpace::MarkedSpace):
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::lastChanceToFinalize):
(JSC::MarkedSpace::allocate):
(JSC::MarkedSpace::tryAllocate):
(JSC::MarkedSpace::allocateLarge):
(JSC::MarkedSpace::tryAllocateLarge):
(JSC::MarkedSpace::sweep):
(JSC::MarkedSpace::sweepLargeAllocations):
(JSC::MarkedSpace::zombifySweep):
(JSC::MarkedSpace::resetAllocators):
(JSC::MarkedSpace::visitWeakSets):
(JSC::MarkedSpace::reapWeakSets):
(JSC::MarkedSpace::stopAllocating):
(JSC::MarkedSpace::prepareForMarking):
(JSC::MarkedSpace::resumeAllocating):
(JSC::MarkedSpace::isPagedOut):
(JSC::MarkedSpace::freeBlock):
(JSC::MarkedSpace::freeOrShrinkBlock):
(JSC::MarkedSpace::shrink):
(JSC::MarkedSpace::clearNewlyAllocated):
(JSC::VerifyMarked::operator()):
(JSC::MarkedSpace::flip):
(JSC::MarkedSpace::objectCount):
(JSC::MarkedSpace::size):
(JSC::MarkedSpace::capacity):
(JSC::MarkedSpace::addActiveWeakSet):
(JSC::MarkedSpace::didAddBlock):
(JSC::MarkedSpace::didAllocateInBlock):
(JSC::MarkedSpace::forEachAllocator): Deleted.
(JSC::VerifyMarkedOrRetired::operator()): Deleted.
(JSC::MarkedSpace::clearMarks): Deleted.

  • heap/MarkedSpace.h:

(JSC::MarkedSpace::sizeClassToIndex):
(JSC::MarkedSpace::indexToSizeClass):
(JSC::MarkedSpace::version):
(JSC::MarkedSpace::blocksWithNewObjects):
(JSC::MarkedSpace::largeAllocations):
(JSC::MarkedSpace::largeAllocationsNurseryOffset):
(JSC::MarkedSpace::largeAllocationsOffsetForThisCollection):
(JSC::MarkedSpace::largeAllocationsForThisCollectionBegin):
(JSC::MarkedSpace::largeAllocationsForThisCollectionEnd):
(JSC::MarkedSpace::largeAllocationsForThisCollectionSize):
(JSC::MarkedSpace::forEachLiveCell):
(JSC::MarkedSpace::forEachDeadCell):
(JSC::MarkedSpace::allocatorFor):
(JSC::MarkedSpace::destructorAllocatorFor):
(JSC::MarkedSpace::auxiliaryAllocatorFor):
(JSC::MarkedSpace::allocateWithoutDestructor):
(JSC::MarkedSpace::allocateWithDestructor):
(JSC::MarkedSpace::allocateAuxiliary):
(JSC::MarkedSpace::tryAllocateAuxiliary):
(JSC::MarkedSpace::forEachBlock):
(JSC::MarkedSpace::forEachAllocator):
(JSC::MarkedSpace::optimalSizeFor):
(JSC::MarkedSpace::didAddBlock): Deleted.
(JSC::MarkedSpace::didAllocateInBlock): Deleted.
(JSC::MarkedSpace::objectCount): Deleted.
(JSC::MarkedSpace::size): Deleted.
(JSC::MarkedSpace::capacity): Deleted.

  • heap/SlotVisitor.cpp:

(JSC::SlotVisitor::SlotVisitor):
(JSC::SlotVisitor::didStartMarking):
(JSC::SlotVisitor::reset):
(JSC::SlotVisitor::append):
(JSC::SlotVisitor::appendJSCellOrAuxiliary):
(JSC::SlotVisitor::setMarkedAndAppendToMarkStack):
(JSC::SlotVisitor::appendToMarkStack):
(JSC::SlotVisitor::markAuxiliary):
(JSC::SlotVisitor::noteLiveAuxiliaryCell):
(JSC::SlotVisitor::visitChildren):

  • heap/SlotVisitor.h:
  • heap/WeakBlock.cpp:

(JSC::WeakBlock::create):
(JSC::WeakBlock::WeakBlock):
(JSC::WeakBlock::visit):
(JSC::WeakBlock::reap):

  • heap/WeakBlock.h:

(JSC::WeakBlock::disconnectContainer):
(JSC::WeakBlock::disconnectMarkedBlock): Deleted.

  • heap/WeakSet.cpp:

(JSC::WeakSet::~WeakSet):
(JSC::WeakSet::sweep):
(JSC::WeakSet::shrink):
(JSC::WeakSet::addAllocator):

  • heap/WeakSet.h:

(JSC::WeakSet::container):
(JSC::WeakSet::setContainer):
(JSC::WeakSet::WeakSet):
(JSC::WeakSet::visit):
(JSC::WeakSet::shrink): Deleted.

  • heap/WeakSetInlines.h:

(JSC::WeakSet::allocate):

  • inspector/InjectedScriptManager.cpp:
  • inspector/JSGlobalObjectInspectorController.cpp:
  • inspector/JSJavaScriptCallFrame.cpp:
  • inspector/ScriptDebugServer.cpp:
  • inspector/agents/InspectorDebuggerAgent.cpp:
  • interpreter/CachedCall.h:

(JSC::CachedCall::CachedCall):

  • interpreter/Interpreter.cpp:

(JSC::loadVarargs):
(JSC::StackFrame::sourceID): Deleted.
(JSC::StackFrame::sourceURL): Deleted.
(JSC::StackFrame::functionName): Deleted.
(JSC::StackFrame::computeLineAndColumn): Deleted.
(JSC::StackFrame::toString): Deleted.

  • interpreter/Interpreter.h:

(JSC::StackFrame::isNative): Deleted.

  • jit/AssemblyHelpers.h:

(JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator):
(JSC::AssemblyHelpers::emitAllocate):
(JSC::AssemblyHelpers::emitAllocateJSCell):
(JSC::AssemblyHelpers::emitAllocateJSObject):
(JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize):
(JSC::AssemblyHelpers::emitAllocateVariableSized):

  • jit/GCAwareJITStubRoutine.cpp:

(JSC::GCAwareJITStubRoutine::GCAwareJITStubRoutine):

  • jit/JIT.cpp:

(JSC::JIT::compileCTINativeCall):
(JSC::JIT::link):

  • jit/JIT.h:

(JSC::JIT::compileCTINativeCall): Deleted.

  • jit/JITExceptions.cpp:

(JSC::genericUnwind):

  • jit/JITExceptions.h:
  • jit/JITOpcodes.cpp:

(JSC::JIT::emit_op_new_object):
(JSC::JIT::emitSlow_op_new_object):
(JSC::JIT::emit_op_create_this):
(JSC::JIT::emitSlow_op_create_this):

  • jit/JITOpcodes32_64.cpp:

(JSC::JIT::emit_op_new_object):
(JSC::JIT::emitSlow_op_new_object):
(JSC::JIT::emit_op_create_this):
(JSC::JIT::emitSlow_op_create_this):

  • jit/JITOperations.cpp:
  • jit/JITOperations.h:
  • jit/JITPropertyAccess.cpp:

(JSC::JIT::emitWriteBarrier):

  • jit/JITThunks.cpp:
  • jit/JITThunks.h:
  • jsc.cpp:

(functionDescribeArray):
(main):

  • llint/LLIntData.cpp:

(JSC::LLInt::Data::performAssertions):

  • llint/LLIntExceptions.cpp:
  • llint/LLIntThunks.cpp:
  • llint/LLIntThunks.h:
  • llint/LowLevelInterpreter.asm:
  • llint/LowLevelInterpreter.cpp:
  • llint/LowLevelInterpreter32_64.asm:
  • llint/LowLevelInterpreter64.asm:
  • parser/ModuleAnalyzer.cpp:
  • parser/NodeConstructors.h:
  • parser/Nodes.h:
  • profiler/ProfilerBytecode.cpp:
  • profiler/ProfilerBytecode.h:
  • profiler/ProfilerBytecodeSequence.cpp:
  • runtime/ArrayConventions.h:

(JSC::indexingHeaderForArrayStorage):
(JSC::baseIndexingHeaderForArrayStorage):
(JSC::indexingHeaderForArray): Deleted.
(JSC::baseIndexingHeaderForArray): Deleted.

  • runtime/ArrayPrototype.cpp:

(JSC::arrayProtoFuncSplice):
(JSC::concatAppendOne):
(JSC::arrayProtoPrivateFuncConcatMemcpy):

  • runtime/ArrayStorage.h:

(JSC::ArrayStorage::vectorLength):
(JSC::ArrayStorage::totalSizeFor):
(JSC::ArrayStorage::totalSize):
(JSC::ArrayStorage::availableVectorLength):
(JSC::ArrayStorage::optimalVectorLength):
(JSC::ArrayStorage::sizeFor): Deleted.

  • runtime/AuxiliaryBarrier.h: Added.

(JSC::AuxiliaryBarrier::AuxiliaryBarrier):
(JSC::AuxiliaryBarrier::clear):
(JSC::AuxiliaryBarrier::get):
(JSC::AuxiliaryBarrier::slot):
(JSC::AuxiliaryBarrier::operator bool):
(JSC::AuxiliaryBarrier::setWithoutBarrier):

  • runtime/AuxiliaryBarrierInlines.h: Added.

(JSC::AuxiliaryBarrier<T>::AuxiliaryBarrier):
(JSC::AuxiliaryBarrier<T>::set):

  • runtime/Butterfly.h:
  • runtime/ButterflyInlines.h:

(JSC::Butterfly::availableContiguousVectorLength):
(JSC::Butterfly::optimalContiguousVectorLength):
(JSC::Butterfly::createUninitialized):
(JSC::Butterfly::growArrayRight):

  • runtime/ClonedArguments.cpp:

(JSC::ClonedArguments::createEmpty):

  • runtime/CommonSlowPathsExceptions.cpp:
  • runtime/CommonSlowPathsExceptions.h:
  • runtime/DataView.cpp:
  • runtime/DirectArguments.h:
  • runtime/ECMAScriptSpecInternalFunctions.cpp:
  • runtime/Error.cpp:
  • runtime/Error.h:
  • runtime/ErrorInstance.cpp:
  • runtime/ErrorInstance.h:
  • runtime/Exception.cpp:
  • runtime/Exception.h:
  • runtime/GeneratorFrame.cpp:
  • runtime/GeneratorPrototype.cpp:
  • runtime/InternalFunction.cpp:

(JSC::InternalFunction::InternalFunction):

  • runtime/IntlCollator.cpp:
  • runtime/IntlCollatorConstructor.cpp:
  • runtime/IntlCollatorPrototype.cpp:
  • runtime/IntlDateTimeFormat.cpp:
  • runtime/IntlDateTimeFormatConstructor.cpp:
  • runtime/IntlDateTimeFormatPrototype.cpp:
  • runtime/IntlNumberFormat.cpp:
  • runtime/IntlNumberFormatConstructor.cpp:
  • runtime/IntlNumberFormatPrototype.cpp:
  • runtime/IntlObject.cpp:
  • runtime/IteratorPrototype.cpp:
  • runtime/JSArray.cpp:

(JSC::JSArray::tryCreateUninitialized):
(JSC::JSArray::setLengthWritable):
(JSC::JSArray::unshiftCountSlowCase):
(JSC::JSArray::setLengthWithArrayStorage):
(JSC::JSArray::appendMemcpy):
(JSC::JSArray::setLength):
(JSC::JSArray::pop):
(JSC::JSArray::push):
(JSC::JSArray::fastSlice):
(JSC::JSArray::shiftCountWithArrayStorage):
(JSC::JSArray::shiftCountWithAnyIndexingType):
(JSC::JSArray::unshiftCountWithArrayStorage):
(JSC::JSArray::fillArgList):
(JSC::JSArray::copyToArguments):

  • runtime/JSArray.h:

(JSC::createContiguousArrayButterfly):
(JSC::createArrayButterfly):
(JSC::JSArray::create):
(JSC::JSArray::tryCreateUninitialized): Deleted.

  • runtime/JSArrayBufferView.h:
  • runtime/JSCInlines.h:
  • runtime/JSCJSValue.cpp:

(JSC::JSValue::dumpInContextAssumingStructure):

  • runtime/JSCallee.cpp:

(JSC::JSCallee::JSCallee):

  • runtime/JSCell.cpp:

(JSC::JSCell::estimatedSize):

  • runtime/JSCell.h:

(JSC::JSCell::cellStateOffset): Deleted.

  • runtime/JSCellInlines.h:

(JSC::ExecState::vm):
(JSC::JSCell::classInfo):
(JSC::JSCell::callDestructor):
(JSC::JSCell::vm): Deleted.

  • runtime/JSFunction.cpp:

(JSC::JSFunction::create):
(JSC::JSFunction::allocateAndInitializeRareData):
(JSC::JSFunction::initializeRareData):
(JSC::JSFunction::getOwnPropertySlot):
(JSC::JSFunction::put):
(JSC::JSFunction::deleteProperty):
(JSC::JSFunction::defineOwnProperty):
(JSC::JSFunction::setFunctionName):
(JSC::JSFunction::reifyLength):
(JSC::JSFunction::reifyName):
(JSC::JSFunction::reifyLazyPropertyIfNeeded):
(JSC::JSFunction::reifyBoundNameIfNeeded):

  • runtime/JSFunction.h:
  • runtime/JSFunctionInlines.h:

(JSC::JSFunction::createWithInvalidatedReallocationWatchpoint):
(JSC::JSFunction::JSFunction):

  • runtime/JSGenericTypedArrayViewInlines.h:

(JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory):

  • runtime/JSInternalPromise.cpp:
  • runtime/JSInternalPromiseConstructor.cpp:
  • runtime/JSInternalPromiseDeferred.cpp:
  • runtime/JSInternalPromisePrototype.cpp:
  • runtime/JSJob.cpp:
  • runtime/JSMapIterator.cpp:
  • runtime/JSModuleNamespaceObject.cpp:
  • runtime/JSModuleRecord.cpp:
  • runtime/JSObject.cpp:

(JSC::JSObject::visitButterfly):
(JSC::JSObject::notifyPresenceOfIndexedAccessors):
(JSC::JSObject::createInitialIndexedStorage):
(JSC::JSObject::createInitialUndecided):
(JSC::JSObject::createInitialInt32):
(JSC::JSObject::createInitialDouble):
(JSC::JSObject::createInitialContiguous):
(JSC::JSObject::createArrayStorage):
(JSC::JSObject::createInitialArrayStorage):
(JSC::JSObject::convertUndecidedToInt32):
(JSC::JSObject::convertUndecidedToContiguous):
(JSC::JSObject::convertUndecidedToArrayStorage):
(JSC::JSObject::convertInt32ToDouble):
(JSC::JSObject::convertInt32ToArrayStorage):
(JSC::JSObject::convertDoubleToArrayStorage):
(JSC::JSObject::convertContiguousToArrayStorage):
(JSC::JSObject::putByIndexBeyondVectorLength):
(JSC::JSObject::putDirectIndexBeyondVectorLength):
(JSC::JSObject::getNewVectorLength):
(JSC::JSObject::increaseVectorLength):
(JSC::JSObject::ensureLengthSlow):
(JSC::JSObject::growOutOfLineStorage):
(JSC::JSObject::copyButterfly): Deleted.
(JSC::JSObject::copyBackingStore): Deleted.

  • runtime/JSObject.h:

(JSC::JSObject::globalObject):
(JSC::JSObject::putDirectInternal):
(JSC::JSObject::setStructureAndReallocateStorageIfNecessary): Deleted.

  • runtime/JSObjectInlines.h:
  • runtime/JSPromise.cpp:
  • runtime/JSPromiseConstructor.cpp:
  • runtime/JSPromiseDeferred.cpp:
  • runtime/JSPromisePrototype.cpp:
  • runtime/JSPropertyNameIterator.cpp:
  • runtime/JSScope.cpp:

(JSC::JSScope::resolve):

  • runtime/JSScope.h:

(JSC::JSScope::globalObject):
(JSC::JSScope::vm): Deleted.

  • runtime/JSSetIterator.cpp:
  • runtime/JSStringIterator.cpp:
  • runtime/JSTemplateRegistryKey.cpp:
  • runtime/JSTypedArrayViewConstructor.cpp:
  • runtime/JSTypedArrayViewPrototype.cpp:
  • runtime/JSWeakMap.cpp:
  • runtime/JSWeakSet.cpp:
  • runtime/MapConstructor.cpp:
  • runtime/MapIteratorPrototype.cpp:
  • runtime/MapPrototype.cpp:
  • runtime/NativeErrorConstructor.cpp:
  • runtime/NativeStdFunctionCell.cpp:
  • runtime/Operations.h:

(JSC::scribbleFreeCells):
(JSC::scribble):

  • runtime/Options.h:
  • runtime/PropertyTable.cpp:
  • runtime/ProxyConstructor.cpp:
  • runtime/ProxyObject.cpp:
  • runtime/ProxyRevoke.cpp:
  • runtime/RegExp.cpp:

(JSC::RegExp::match):
(JSC::RegExp::matchConcurrently):
(JSC::RegExp::matchCompareWithInterpreter):

  • runtime/RegExp.h:
  • runtime/RegExpConstructor.h:
  • runtime/RegExpInlines.h:

(JSC::RegExp::matchInline):

  • runtime/RegExpMatchesArray.h:

(JSC::tryCreateUninitializedRegExpMatchesArray):
(JSC::createRegExpMatchesArray):

  • runtime/RegExpPrototype.cpp:

(JSC::genericSplit):

  • runtime/RuntimeType.cpp:
  • runtime/SamplingProfiler.cpp:

(JSC::SamplingProfiler::processUnverifiedStackTraces):

  • runtime/SetConstructor.cpp:
  • runtime/SetIteratorPrototype.cpp:
  • runtime/SetPrototype.cpp:
  • runtime/StackFrame.cpp: Added.

(JSC::StackFrame::sourceID):
(JSC::StackFrame::sourceURL):
(JSC::StackFrame::functionName):
(JSC::StackFrame::computeLineAndColumn):
(JSC::StackFrame::toString):

  • runtime/StackFrame.h: Added.

(JSC::StackFrame::isNative):

  • runtime/StringConstructor.cpp:
  • runtime/StringIteratorPrototype.cpp:
  • runtime/StructureInlines.h:

(JSC::Structure::propertyTable):

  • runtime/TemplateRegistry.cpp:
  • runtime/TestRunnerUtils.cpp:

(JSC::finalizeStatsAtEndOfTesting):

  • runtime/TestRunnerUtils.h:
  • runtime/TypeProfilerLog.cpp:
  • runtime/TypeSet.cpp:
  • runtime/VM.cpp:

(JSC::VM::VM):
(JSC::VM::ensureStackCapacityForCLoop):
(JSC::VM::isSafeToRecurseSoftCLoop):

  • runtime/VM.h:
  • runtime/VMEntryScope.h:
  • runtime/VMInlines.h:

(JSC::VM::ensureStackCapacityFor):
(JSC::VM::isSafeToRecurseSoft):

  • runtime/WeakMapConstructor.cpp:
  • runtime/WeakMapData.cpp:
  • runtime/WeakMapPrototype.cpp:
  • runtime/WeakSetConstructor.cpp:
  • runtime/WeakSetPrototype.cpp:
  • testRegExp.cpp:

(testOneRegExp):

  • tools/JSDollarVM.cpp:
  • tools/JSDollarVMPrototype.cpp:

(JSC::JSDollarVMPrototype::isInObjectSpace):

Source/WebCore:

No new tests because no new WebCore behavior.

Just rewiring #includes.

  • ForwardingHeaders/heap/HeapInlines.h: Added.
  • ForwardingHeaders/interpreter/Interpreter.h: Removed.
  • ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h: Added.
  • Modules/indexeddb/IDBCursorWithValue.cpp:
  • Modules/indexeddb/client/TransactionOperation.cpp:
  • Modules/indexeddb/server/SQLiteIDBBackingStore.cpp:
  • Modules/indexeddb/server/UniqueIDBDatabase.cpp:
  • bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp:
  • bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp:
  • bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp:
  • bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp:
  • bindings/js/JSClientRectCustom.cpp:
  • bindings/js/JSDOMBinding.cpp:
  • bindings/js/JSDOMBinding.h:
  • bindings/js/JSDeviceMotionEventCustom.cpp:
  • bindings/js/JSDeviceOrientationEventCustom.cpp:
  • bindings/js/JSErrorEventCustom.cpp:
  • bindings/js/JSIDBCursorWithValueCustom.cpp:
  • bindings/js/JSIDBIndexCustom.cpp:
  • bindings/js/JSPopStateEventCustom.cpp:
  • bindings/js/JSWebGL2RenderingContextCustom.cpp:
  • bindings/js/JSWorkerGlobalScopeCustom.cpp:
  • bindings/js/WorkerScriptController.cpp:
  • contentextensions/ContentExtensionParser.cpp:
  • dom/ErrorEvent.cpp:
  • html/HTMLCanvasElement.cpp:
  • html/MediaDocument.cpp:
  • inspector/CommandLineAPIModule.cpp:
  • loader/EmptyClients.cpp:
  • page/CaptionUserPreferences.cpp:
  • page/Frame.cpp:
  • page/PageGroup.cpp:
  • page/UserContentController.cpp:
  • platform/mock/mediasource/MockBox.cpp:
  • testing/GCObservation.cpp:

Source/WebKit2:


Just rewiring some #includes.

  • UIProcess/ViewGestureController.cpp:
  • UIProcess/WebPageProxy.cpp:
  • UIProcess/WebProcessPool.cpp:
  • UIProcess/WebProcessProxy.cpp:
  • WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp:
  • WebProcess/Plugins/Netscape/JSNPObject.cpp:

Source/WTF:


I needed tryFastAlignedMalloc() so I added it.

  • wtf/FastMalloc.cpp:

(WTF::tryFastAlignedMalloc):

  • wtf/FastMalloc.h:
  • wtf/ParkingLot.cpp:

(WTF::ParkingLot::forEachImpl):
(WTF::ParkingLot::forEach): Deleted.

  • wtf/ParkingLot.h:

(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::forEach):

  • wtf/ScopedLambda.h:

(WTF::scopedLambdaRef):

  • wtf/SentinelLinkedList.h:

(WTF::SentinelLinkedList::forEach):
(WTF::RawNode>::takeFrom):

  • wtf/SimpleStats.h:

(WTF::SimpleStats::operator bool):
(WTF::SimpleStats::operator!): Deleted.

Tools:

  • DumpRenderTree/TestRunner.cpp:
  • DumpRenderTree/mac/DumpRenderTree.mm:

(DumpRenderTreeMain):

  • Scripts/run-jsc-stress-tests:
  • TestWebKitAPI/Tests/WTF/Vector.cpp:

(TestWebKitAPI::TEST):

Location:
trunk
Files:
23 added
1 deleted
291 edited

Legend:

Unmodified
Added
Removed

  • trunk/JSTests/ChangeLog

    r205389 r205462  
     12016-08-31  Filip Pizlo  <fpizlo@apple.com>
     2
     3        Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
     4        https://bugs.webkit.org/show_bug.cgi?id=160125
     5
     6        Reviewed by Geoffrey Garen and Keith Miller.
     7       
     8        Most of the things I did properly covered by existing tests, but I found some simple cases of
     9        unshifting that had sketchy coverage.
     10
     11        * stress/array-storage-array-unshift.js: Added.
     12        * stress/contiguous-array-unshift.js: Added.
     13        * stress/double-array-unshift.js: Added.
     14        * stress/int32-array-unshift.js: Added.
     15
    1162016-09-02  Michael Saboff  <msaboff@apple.com>
    217

  • trunk/Source/JavaScriptCore/API/JSManagedValue.mm

    r197563 r205462  
    214214
    215215    JSC::JSValue jsValue = toJS(exec, [value JSValueRef]);
     216    dataLog("Creating managed value with value ", jsValue, "\n");
    216217    if (jsValue.isObject())
    217218        m_weakValue.setObject(JSC::jsCast<JSC::JSObject*>(jsValue.asCell()), self);

  • trunk/Source/JavaScriptCore/API/JSTypedArray.cpp

    r205198 r205462  
    3333#include "Error.h"
    3434#include "JSArrayBufferViewInlines.h"
    35 #include "JSCJSValueInlines.h"
     35#include "JSCInlines.h"
    3636#include "JSDataView.h"
    3737#include "JSGenericTypedArrayViewInlines.h"

  • trunk/Source/JavaScriptCore/API/ObjCCallbackFunction.mm

    r204912 r205462  
    3232#import "APICast.h"
    3333#import "Error.h"
    34 #import "JSCJSValueInlines.h"
    3534#import "JSCell.h"
    36 #import "JSCellInlines.h"
     35#import "JSCInlines.h"
    3736#import "JSContextInternal.h"
    3837#import "JSWrapperMap.h"

  • trunk/Source/JavaScriptCore/API/tests/testapi.mm

    r202846 r205462  
    511511}
    512512
    513 // This test is flaky. Since GC marks C stack and registers as roots conservatively,
    514 // objects not referenced logically can be accidentally marked and alive.
    515 // To avoid this situation as possible as we can,
    516 // 1. run this test first before stack is polluted,
    517 // 2. extract this test as a function to suppress stack height.
    518 static void testWeakValue()
    519 {
    520     @autoreleasepool {
    521         JSVirtualMachine *vm = [[JSVirtualMachine alloc] init];
    522         TestObject *testObject = [TestObject testObject];
    523         JSManagedValue *weakValue;
    524         @autoreleasepool {
    525             JSContext *context = [[JSContext alloc] initWithVirtualMachine:vm];
    526             context[@"testObject"] = testObject;
    527             weakValue = [[JSManagedValue alloc] initWithValue:context[@"testObject"]];
    528         }
    529 
    530         @autoreleasepool {
    531             JSContext *context = [[JSContext alloc] initWithVirtualMachine:vm];
    532             context[@"testObject"] = testObject;
    533             JSSynchronousGarbageCollectForDebugging([context JSGlobalContextRef]);
    534             checkResult(@"weak value == nil", ![weakValue value]);
    535             checkResult(@"root is still alive", !context[@"testObject"].isUndefined);
    536         }
    537     }
    538 }
    539 
    540513static void testObjectiveCAPIMain()
    541514{
    … …  
    15141487    NSLog(@"Testing Objective-C API");
    15151488    checkNegativeNSIntegers();
    1516     testWeakValue();
    15171489    testObjectiveCAPIMain();
    15181490}

  • trunk/Source/JavaScriptCore/CMakeLists.txt

    r205418 r205462  
    6767    assembler/MacroAssemblerARM.cpp
    6868    assembler/MacroAssemblerARMv7.cpp
     69    assembler/MacroAssemblerCodeRef.cpp
    6970    assembler/MacroAssemblerPrinter.cpp
    7071    assembler/MacroAssemblerX86Common.cpp
    … …  
    448449    heap/EdenGCActivityCallback.cpp
    449450    heap/FullGCActivityCallback.cpp
     451    heap/FreeList.cpp
    450452    heap/GCActivityCallback.cpp
    451453    heap/GCLogging.cpp
    … …  
    455457    heap/HeapCell.cpp
    456458    heap/HeapHelperPool.cpp
     459    heap/HeapOperation.cpp
    457460    heap/HeapProfiler.cpp
    458461    heap/HeapSnapshot.cpp
    … …  
    463466    heap/IncrementalSweeper.cpp
    464467    heap/JITStubRoutineSet.cpp
     468    heap/LargeAllocation.cpp
    465469    heap/LiveObjectList.cpp
    466470    heap/MachineStackMarker.cpp
    … …  
    620624    runtime/ArrayBufferView.cpp
    621625    runtime/ArrayConstructor.cpp
     626    runtime/ArrayConventions.cpp
    622627    runtime/ArrayIteratorPrototype.cpp
    623628    runtime/ArrayPrototype.cpp
    … …  
    800805    runtime/SmallStrings.cpp
    801806    runtime/SparseArrayValueMap.cpp
     807    runtime/StackFrame.cpp
    802808    runtime/StrictEvalActivation.cpp
    803809    runtime/StringConstructor.cpp

  • trunk/Source/JavaScriptCore/ChangeLog

    r205418 r205462  
     12016-08-31  Filip Pizlo  <fpizlo@apple.com>
     2
     3        Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
     4        https://bugs.webkit.org/show_bug.cgi?id=160125
     5
     6        Reviewed by Geoffrey Garen and Keith Miller.
     7
     8        In order to make the GC concurrent (bug 149432), we would either need to enable concurrent
     9        copying or we would need to not copy. Concurrent copying carries a 1-2% throughput overhead
     10        from the barriers alone. Considering that MarkedSpace does a decent job of avoiding
     11        fragmentation, it's unlikely that it's worth paying 1-2% throughput for copying. So, we want
     12        to get rid of copied space. This change moves copied space's biggest client over to marked
     13        space.
     14       
     15        Moving butterflies to marked space means having them use the new Auxiliary HeapCell
     16        allocation path. This is a fairly mechanical change, but it caused performance regressions
     17        everywhere, so this change also fixes MarkedSpace's performance issues.
     18       
     19        At a high level the mechanical changes are:
     20       
     21        - We use AuxiliaryBarrier instead of CopyBarrier.
     22       
     23        - We use tryAllocateAuxiliary instead of tryAllocateStorage. I got rid of the silly
     24          CheckedBoolean stuff, since it's so much more trouble than it's worth.
     25       
     26        - The JITs have to emit inlined marked space allocations instead of inline copy space
     27          allocations.
     28       
     29        - Everyone has to get used to zeroing their butterflies after allocation instead of relying
     30          on them being pre-zeroed by the GC. Copied space would zero things for you, while marked
     31          space doesn't.
     32       
     33        That's about 1/3 of this change. But this led to performance problems, which I fixed with
     34        optimizations that amounted to a major MarkedSpace rewrite:
     35       
     36        - MarkedSpace always causes internal fragmentation for array allocations because the vector
     37          length we choose when we resize usually leads to a cell size that doesn't correspond to any
     38          size class. I got around this by making array allocations usually round up vectorLength to
     39          the maximum allowed by the size class that we would have allocated in. Also,
     40          ensureLengthSlow() and friends first make sure that the requested length can't just be
     41          fulfilled with the current allocation size. This safeguard means that not every array
     42          allocation has to do size class queries. For example, the fast path of new Array(length)
     43          never does any size class queries, under the assumption that (1) the speed gained from
     44          avoiding an ensureLengthSlow() call, which then just changes the vectorLength by doing the
     45          size class query, is too small to offset the speed lost by doing the query on every
     46          allocation and (2) new Array(length) is a pretty good hint that resizing is not very
     47          likely.
     48       
     49        - Size classes in MarkedSpace were way too precise, which led to external fragmentation. This
     50          changes MarkedSpace size classes to use a linear progression for very small sizes followed
     51          by a geometric progression that naturally transitions to a hyperbolic progression. We want
     52          hyperbolic sizes when we get close to blockSize: for example the largest size we want is
     53          payloadSize / 2 rounded down, to ensure we get exactly two cells with minimal slop. The
     54          next size down should be payloadSize / 3 rounded down, and so on. After the last precise
     55          size (80 bytes), we proceed using a geometric progression, but round up each size to
     56          minimize slop at the end of the block. This naturally causes the geometric progression to
     57          turn hyperbolic for large sizes. The size class configuration happens at VM start-up, so
     58          it can be controlled with runtime options. I found that a base of 1.4 works pretty well.
     59       
     60        - Large allocations caused massive internal fragmentation, since the smallest large
     61          allocation had to use exactly blockSize, and the largest small allocation used
     62          blockSize / 2. The next size up - the first large allocation size to require two blocks -
     63          also had 50% internal fragmentation. This is because we required large allocations to be
     64          blockSize aligned, so that MarkedBlock::blockFor() would work. I decided to rewrite all of
     65          that. Cells no longer have to be owned by a MarkedBlock. They can now alternatively be
     66          owned by a LargeAllocation. These two things are abstracted as CellContainer. You know that
     67          a cell is owned by a LargeAllocation if the MarkedBlock::atomSize / 2 bit is set.
     68          Basically, large allocations are deliberately misaligned by 8 bytes. This actually works
     69          out great since (1) typed arrays won't use large allocations anyway since they have their
     70          own malloc fallback and (2) large array butterflies already have a 8 byte header, which
     71          means that the 8 byte base misalignment aligns the large array payload on a 16 byte
     72          boundary. I took extreme care to make sure that the isLargeAllocation bit checks are as
     73          rare as possible; for example, ExecState::vm() skips the check because we know that callees
     74          must be small allocations. It's also possible to use template tricks to do one check for
     75          cell container kind, and then invoke a function specialized for MarkedBlock or a function
     76          specialized for LargeAllocation. LargeAllocation includes stubs for all MarkedBlock methods
     77          that get used from functions that are template-specialized like this. That's mostly to
     78          speed up the GC marking code. Most other code can use CellContainer API or HeapCell API
     79          directly. That's another thing: HeapCell, the common base of JSCell and auxiliary
     80          allocations, is now smart enough to do a lot of things for you, like HeapCell::vm(),
     81          HeapCell::heap(), HeapCell::isLargeAllocation(), and HeapCell::cellContainer(). The size
     82          cutoff for large allocations is runtime-configurable, so long as you don't choose something
     83          so small that callees end up large. I found that 400 bytes is roughly optimal. This means
     84          that the MarkedBlock size classes end up being:
     85         
     86          16, 32, 48, 64, 80, 112, 160, 224, 320
     87         
     88          The next size class would have been 432, but that's above the 400 byte cutoff. All of this
     89          is configurable with --sizeClassProgression and --largeAllocationCutoff. You can see what
     90          size classes you end up with by doing --dumpSizeClasses=true.
     91       
     92        - Copied space uses 64KB blocks, while marked space used to use 16KB blocks. Allocating a lot
     93          of stuff in 16KB blocks was slower than allocating it in 64KB blocks because the GC had a
     94          lot of per-block overhead. I removed this overhead: It's now 2x faster to scan all
     95          MarkedBlocks because the list that contains the interesting meta-data is allocated on the
     96          side, for better locality during a sequential walk. It's no longer necessary to scan
     97          MarkedBlocks to find WeakSets, since the sets of WeakSets for eden scan and full scan are
     98          maintained on-the-fly. It's no longer necessary to scan all MarkedBlocks to clear mark
     99          bits because we now use versioned mark bits: to clear then, just increment the 64-bit
     100          heap version. It's no longer necessary to scan retired MarkedBlocks while allocating
     101          because marking retires them on-the-fly. It's no longer necessary to sort all blocks in
     102          the IncrementalSweeper's snapshot because blocks now know if they are in the snapshot. Put
     103          together, these optimizations allowed me to reduce block size to 16KB without losing much
     104          performance. There is some small perf loss on JetStream/splay, but not enough to hurt
     105          JetStream overall. I tried reducing block sizes further, to 4KB, since that is a
     106          progression on membuster. That's not possible yet, since there is still enough per-block
     107          overhead yet that such a reduction hurts JetStream too much. I filed a bug about improving
     108          this further: https://bugs.webkit.org/show_bug.cgi?id=161581.
     109       
     110        - Even after all of that, copying butterflies was still faster because it allowed us to skip
     111          sweeping dead space. A good GC allocates over dead bytes without explicitly freeing them,
     112          so the GC pause is O(size of live), not O(size of live + dead). O(dead) is usually much
     113          larger than O(live), especially in an eden collection. Copying satisfies this premise while
     114          mark+sweep does not. So, I invented a new kind of allocator: bump'n'pop. Previously, our
     115          MarkedSpace allocator was a freelist pop. That's simple and easy to inline but requires
     116          that we walk the block to build a free list. This means walking dead space. The new
     117          allocator allows totally free MarkedBlocks to simply set up a bump-pointer arena instead.
     118          The allocator is a hybrid of bump-pointer and freelist pop. It tries bump first. The bump
     119          pointer always bumps by cellSize, so the result of filling a block with bumping looks as if
     120          we had used freelist popping to fill it. Additionally, each MarkedBlock now has a bit to
     121          quickly tell if the block is entirely free. This makes sweeping O(1) whenever a MarkedBlock
     122          is completely empty, which is the common case because of the generational hypothesis: the
     123          number of objects that survive an eden collection is a tiny fraction of the number of
     124          objects that had been allocated, and this fraction is so small that there are typically
     125          fewer than one survivors per MarkedBlock. This change was enough to make this change a net
     126          win over tip-of-tree.
     127       
     128        - FTL now shares the same allocation fast paths as everything else, which is great, because
     129          bump'n'pop has gnarly control flow. We don't really want B3 to have to think about that
     130          control flow, since it won't be able to improve the machine code we write ourselves. GC
     131          fast paths are best written in assembly. So, I've empowered B3 to have even better support
     132          for Patchpoint terminals. It's now totally fine for a Patchpoint terminal to be non-Void.
     133          So, the new FTL allocation fast paths are just Patchpoint terminals that call through to
     134          AssemblyHelpers::emitAllocate(). B3 still reasons about things like constant-folding the
     135          size class calculation and constant-hoisting the allocator. Also, I gave the FTL the
     136          ability to constant-fold some allocator logic (in case we first assume that we're doing a
     137          variable-length allocation but then realize that the length is known). I think it makes
     138          sense to have constant folding rules in FTL::Output, or whatever the B3 IR builder is,
     139          since this makes lowering easier (you can constant fold during lowering more easily) and it
     140          reduces the amount of malloc traffic. In the future, we could teach B3 how to better
     141          constant-fold this code. That would require allowing loads to be constant-folded, which is
     142          doable but hella tricky.
     143       
     144        - It used to be that if a logical object allocation required two physical allocations (first
     145          the butterfly and then the cell), then the JIT would emit the code in such a way that a
     146          failure in the second fast path would cause us to forget the successful first physical
     147          allocation. This was pointlessly wasteful. It turns out that it's very cheap to devote a
     148          register to storing either the butterfly or null, because the butterfly register is anyway
     149          going to be free inside the first allocation. The only overhead here is zeroing the
     150          butterfly register. With that in place, we can just pass the butterfly-or-null to the slow
     151          path, which can then either allocate a butterfly or not. So now we never waste a successful
     152          allocation. This patch implements such a solution both in DFG (where it's easy to do this
     153          since we control registers already) and in FTL (where it's annoying, because mutable
     154          "butterfly-or-null" variables are hard to say in SSA; also I realized that we had code
     155          duplicated the JSArray allocation utility, so I deduplicated it). This came up because in
     156          one version of this patch, this wastage would resonate with some Kraken benchmark: the
     157          benchmark would always allocate N small things followed by one bigger thing. The problem
     158          was I accidentally adjusted the various fixed overheads in MarkedBlock in such a way that
     159          the JSObject size class, which both the small and big thing shared for their cell, could
     160          hold exactly N cells per MarkedBlock. Then the benchmark would always call slow path when
     161          it allocated the big thing. So, it would end up having to allocate the big thing's large
     162          butterfly twice, every single time! Ouch!
     163       
     164        - It used to be that we zeroed CopiedBlocks using memset, and so array allocations enjoyed
     165          amortization of the cost of zeroing. This doesn't work anymore - it's now up to the client
     166          of the allocator to initialize the object to whatever state they need. It used to be that
     167          we would just use a dumb loop. I initially changed this so that we would end up in memset
     168          for large allocations, but this didn't actually help performance that much. I got a much
     169          better result by playing with different memsets written in assembly. First I wrote one
     170          using non-temporal stores. That was a small speed-up over memset. Then I tried the classic
     171          "rep stos" approach, and holy cow that version was fast. It's a ~20% speed-up on array
     172          allocation microbenchmarks. So, this patch adds code paths to do "rep stos" on x86_64, or
     173          memset, or use a loop, as appropriate, for both "contiguous" arrays (holes are zero) and
     174          double arrays (holes are PNaN). Note that the JIT always emits either a loop or a flat slab
     175          of stores (if the size is known), but those paths in the JIT won't trigger for
     176          NewArrayWithSize() if the size is large, since that takes us to the
     177          operationNewArrayWithSize() slow path, which calls into JSArray::create(). That's why the
     178          optimizations here are all in JSArray::create() - that's the hot place for large arrays
     179          that need to be filled with holes.
     180       
     181        All of this put together gives us neutral perf on JetStream,  membuster, and PLT3, a ~1%
     182        regression on Speedometer, and up to a 4% regression Kraken. The Kraken regression is
     183        because Kraken was allocating exactly 1024 element arrays at a rate of 400MB/sec. This is a
     184        best-case scenario for bump allocation. I think that we should fix bmalloc to make up the
     185        difference, but take the hit for now because it's a crazy corner case. By comparison, the
     186        alternative approach of using a copy barrier would have cost us 1-2%. That's the real
     187        apples-to-apples comparison if your premise is that we should have a concurrent GC. After we
     188        finish removing copied space, we will be barrier-ready for concurrent GC: we already have a
     189        marking barrier and we simply won't need a copying barrier. This change gets us there for
     190        the purposes of our benchmarks, since the remaining clients of copied space are not very
     191        important. On the other hand, if we keep copying, then getting barrier-ready would mean
     192        adding back the copy barrier, which costs more perf.
     193       
     194        We might get bigger speed-ups once we remove CopiedSpace altogether. That requires moving
     195        typed arrays and a few other weird things over to Aux MarkedSpace.
     196       
     197        This also includes some header sanitization. The introduction of AuxiliaryBarrier, HeapCell,
     198        and CellContainer meant that I had to include those files from everywhere. Fortunately,
     199        just including JSCInlines.h (instead of manually including the files that includes) is
     200        usually enough. So, I made most of JSC's cpp files include JSCInlines.h, which is something
     201        that we were already basically doing. In places where JSCInlines.h would be too much, I just
     202        included HeapInlines.h. This got weird, because we previously included HeapInlines.h from
     203        JSObject.h. That's bad because it led to some circular dependencies, so I fixed it - but that
     204        meant having to manually include HeapInlines.h from the places that previously got it
     205        implicitly via JSObject.h. But that led to more problems for some reason: I started getting
     206        build errors because non-JSC files were having trouble including Opcode.h. That's just silly,
     207        since Opcode.h is meant to be an internal JSC header. So, I made it an internal header and
     208        made it impossible to include it from outside JSC. This was a lot of work, but it was
     209        necessary to get the patch to build on all ports. It's also a net win. There were many places
     210        in WebCore that were transitively including a *ton* of JSC headers just because of the
     211        JSObject.h->HeapInlines.h edge and a bunch of dependency edges that arose from some public
     212        (for WebCore) JSC headers needing Interpreter.h or Opcode.h for bad reasons.
     213
     214        * API/JSManagedValue.mm:
     215        (-[JSManagedValue initWithValue:]):
     216        * API/JSTypedArray.cpp:
     217        * API/ObjCCallbackFunction.mm:
     218        * API/tests/testapi.mm:
     219        (testObjectiveCAPI):
     220        (testWeakValue): Deleted.
     221        * CMakeLists.txt:
     222        * JavaScriptCore.xcodeproj/project.pbxproj:
     223        * Scripts/builtins/builtins_generate_combined_implementation.py:
     224        (BuiltinsCombinedImplementationGenerator.generate_secondary_header_includes):
     225        * Scripts/builtins/builtins_generate_internals_wrapper_implementation.py:
     226        (BuiltinsInternalsWrapperImplementationGenerator.generate_secondary_header_includes):
     227        * Scripts/builtins/builtins_generate_separate_implementation.py:
     228        (BuiltinsSeparateImplementationGenerator.generate_secondary_header_includes):
     229        * assembler/AbstractMacroAssembler.h:
     230        (JSC::AbstractMacroAssembler::JumpList::link):
     231        (JSC::AbstractMacroAssembler::JumpList::linkTo):
     232        * assembler/MacroAssembler.h:
     233        * assembler/MacroAssemblerARM64.h:
     234        (JSC::MacroAssemblerARM64::add32):
     235        * assembler/MacroAssemblerCodeRef.cpp: Added.
     236        (JSC::MacroAssemblerCodePtr::createLLIntCodePtr):
     237        (JSC::MacroAssemblerCodePtr::dumpWithName):
     238        (JSC::MacroAssemblerCodePtr::dump):
     239        (JSC::MacroAssemblerCodeRef::createLLIntCodeRef):
     240        (JSC::MacroAssemblerCodeRef::dump):
     241        * assembler/MacroAssemblerCodeRef.h:
     242        (JSC::MacroAssemblerCodePtr::createLLIntCodePtr): Deleted.
     243        (JSC::MacroAssemblerCodePtr::dumpWithName): Deleted.
     244        (JSC::MacroAssemblerCodePtr::dump): Deleted.
     245        (JSC::MacroAssemblerCodeRef::createLLIntCodeRef): Deleted.
     246        (JSC::MacroAssemblerCodeRef::dump): Deleted.
     247        * b3/B3BasicBlock.cpp:
     248        (JSC::B3::BasicBlock::appendBoolConstant):
     249        * b3/B3BasicBlock.h:
     250        * b3/B3DuplicateTails.cpp:
     251        * b3/B3StackmapGenerationParams.h:
     252        * b3/testb3.cpp:
     253        (JSC::B3::testPatchpointTerminalReturnValue):
     254        (JSC::B3::run):
     255        * bindings/ScriptValue.cpp:
     256        * bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp:
     257        * bytecode/BytecodeBasicBlock.cpp:
     258        * bytecode/BytecodeLivenessAnalysis.cpp:
     259        * bytecode/BytecodeUseDef.h:
     260        * bytecode/CallLinkInfo.cpp:
     261        (JSC::CallLinkInfo::callTypeFor):
     262        * bytecode/CallLinkInfo.h:
     263        (JSC::CallLinkInfo::callTypeFor): Deleted.
     264        * bytecode/CallLinkStatus.cpp:
     265        * bytecode/CodeBlock.cpp:
     266        (JSC::CodeBlock::finishCreation):
     267        (JSC::CodeBlock::clearLLIntGetByIdCache):
     268        (JSC::CodeBlock::predictedMachineCodeSize):
     269        * bytecode/CodeBlock.h:
     270        (JSC::CodeBlock::jitCodeMap): Deleted.
     271        (JSC::clearLLIntGetByIdCache): Deleted.
     272        * bytecode/ExecutionCounter.h:
     273        * bytecode/Instruction.h:
     274        * bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp:
     275        (JSC::LLIntPrototypeLoadAdaptiveStructureWatchpoint::fireInternal):
     276        * bytecode/ObjectAllocationProfile.h:
     277        (JSC::ObjectAllocationProfile::isNull):
     278        (JSC::ObjectAllocationProfile::initialize):
     279        * bytecode/Opcode.h:
     280        (JSC::padOpcodeName):
     281        * bytecode/PolymorphicAccess.cpp:
     282        (JSC::AccessCase::generateImpl):
     283        (JSC::PolymorphicAccess::regenerate):
     284        * bytecode/PolymorphicAccess.h:
     285        * bytecode/PreciseJumpTargets.cpp:
     286        * bytecode/StructureStubInfo.cpp:
     287        * bytecode/StructureStubInfo.h:
     288        * bytecode/UnlinkedCodeBlock.cpp:
     289        (JSC::UnlinkedCodeBlock::vm): Deleted.
     290        * bytecode/UnlinkedCodeBlock.h:
     291        * bytecode/UnlinkedInstructionStream.cpp:
     292        * bytecode/UnlinkedInstructionStream.h:
     293        * dfg/DFGOperations.cpp:
     294        * dfg/DFGSpeculativeJIT.cpp:
     295        (JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
     296        (JSC::DFG::SpeculativeJIT::compileMakeRope):
     297        (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
     298        (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
     299        * dfg/DFGSpeculativeJIT.h:
     300        (JSC::DFG::SpeculativeJIT::emitAllocateJSCell):
     301        (JSC::DFG::SpeculativeJIT::emitAllocateJSObject):
     302        * dfg/DFGSpeculativeJIT32_64.cpp:
     303        (JSC::DFG::SpeculativeJIT::compile):
     304        (JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
     305        * dfg/DFGSpeculativeJIT64.cpp:
     306        (JSC::DFG::SpeculativeJIT::compile):
     307        (JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
     308        * dfg/DFGStrengthReductionPhase.cpp:
     309        (JSC::DFG::StrengthReductionPhase::handleNode):
     310        * ftl/FTLAbstractHeapRepository.h:
     311        * ftl/FTLCompile.cpp:
     312        * ftl/FTLJITFinalizer.cpp:
     313        * ftl/FTLLowerDFGToB3.cpp:
     314        (JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments):
     315        (JSC::FTL::DFG::LowerDFGToB3::compileCreateRest):
     316        (JSC::FTL::DFG::LowerDFGToB3::allocateArrayWithSize):
     317        (JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize):
     318        (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
     319        (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
     320        (JSC::FTL::DFG::LowerDFGToB3::initializeArrayElements):
     321        (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
     322        (JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell):
     323        (JSC::FTL::DFG::LowerDFGToB3::allocateCell):
     324        (JSC::FTL::DFG::LowerDFGToB3::allocateObject):
     325        (JSC::FTL::DFG::LowerDFGToB3::allocatorForSize):
     326        (JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedObject):
     327        (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray):
     328        (JSC::FTL::DFG::LowerDFGToB3::compileAllocateArrayWithSize): Deleted.
     329        * ftl/FTLOutput.cpp:
     330        (JSC::FTL::Output::constBool):
     331        (JSC::FTL::Output::add):
     332        (JSC::FTL::Output::shl):
     333        (JSC::FTL::Output::aShr):
     334        (JSC::FTL::Output::lShr):
     335        (JSC::FTL::Output::zeroExt):
     336        (JSC::FTL::Output::equal):
     337        (JSC::FTL::Output::notEqual):
     338        (JSC::FTL::Output::above):
     339        (JSC::FTL::Output::aboveOrEqual):
     340        (JSC::FTL::Output::below):
     341        (JSC::FTL::Output::belowOrEqual):
     342        (JSC::FTL::Output::greaterThan):
     343        (JSC::FTL::Output::greaterThanOrEqual):
     344        (JSC::FTL::Output::lessThan):
     345        (JSC::FTL::Output::lessThanOrEqual):
     346        (JSC::FTL::Output::select):
     347        (JSC::FTL::Output::appendSuccessor):
     348        (JSC::FTL::Output::addIncomingToPhi):
     349        * ftl/FTLOutput.h:
     350        * ftl/FTLValueFromBlock.h:
     351        (JSC::FTL::ValueFromBlock::operator bool):
     352        (JSC::FTL::ValueFromBlock::ValueFromBlock): Deleted.
     353        * ftl/FTLWeightedTarget.h:
     354        (JSC::FTL::WeightedTarget::frequentedBlock):
     355        * heap/CellContainer.h: Added.
     356        (JSC::CellContainer::CellContainer):
     357        (JSC::CellContainer::operator bool):
     358        (JSC::CellContainer::isMarkedBlock):
     359        (JSC::CellContainer::isLargeAllocation):
     360        (JSC::CellContainer::markedBlock):
     361        (JSC::CellContainer::largeAllocation):
     362        * heap/CellContainerInlines.h: Added.
     363        (JSC::CellContainer::isMarked):
     364        (JSC::CellContainer::isMarkedOrNewlyAllocated):
     365        (JSC::CellContainer::noteMarked):
     366        (JSC::CellContainer::cellSize):
     367        (JSC::CellContainer::weakSet):
     368        (JSC::CellContainer::flipIfNecessary):
     369        * heap/ConservativeRoots.cpp:
     370        (JSC::ConservativeRoots::ConservativeRoots):
     371        (JSC::ConservativeRoots::~ConservativeRoots):
     372        (JSC::ConservativeRoots::grow):
     373        (JSC::ConservativeRoots::genericAddPointer):
     374        (JSC::ConservativeRoots::genericAddSpan):
     375        * heap/ConservativeRoots.h:
     376        (JSC::ConservativeRoots::roots):
     377        * heap/CopyToken.h:
     378        * heap/FreeList.cpp: Added.
     379        (JSC::FreeList::dump):
     380        * heap/FreeList.h: Added.
     381        (JSC::FreeList::FreeList):
     382        (JSC::FreeList::list):
     383        (JSC::FreeList::bump):
     384        (JSC::FreeList::operator==):
     385        (JSC::FreeList::operator!=):
     386        (JSC::FreeList::operator bool):
     387        (JSC::FreeList::allocationWillFail):
     388        (JSC::FreeList::allocationWillSucceed):
     389        * heap/GCTypeMap.h: Added.
     390        (JSC::GCTypeMap::operator[]):
     391        * heap/Heap.cpp:
     392        (JSC::Heap::Heap):
     393        (JSC::Heap::lastChanceToFinalize):
     394        (JSC::Heap::finalizeUnconditionalFinalizers):
     395        (JSC::Heap::markRoots):
     396        (JSC::Heap::copyBackingStores):
     397        (JSC::Heap::gatherStackRoots):
     398        (JSC::Heap::gatherJSStackRoots):
     399        (JSC::Heap::gatherScratchBufferRoots):
     400        (JSC::Heap::clearLivenessData):
     401        (JSC::Heap::visitSmallStrings):
     402        (JSC::Heap::visitConservativeRoots):
     403        (JSC::Heap::removeDeadCompilerWorklistEntries):
     404        (JSC::Heap::gatherExtraHeapSnapshotData):
     405        (JSC::Heap::removeDeadHeapSnapshotNodes):
     406        (JSC::Heap::visitProtectedObjects):
     407        (JSC::Heap::visitArgumentBuffers):
     408        (JSC::Heap::visitException):
     409        (JSC::Heap::visitStrongHandles):
     410        (JSC::Heap::visitHandleStack):
     411        (JSC::Heap::visitSamplingProfiler):
     412        (JSC::Heap::traceCodeBlocksAndJITStubRoutines):
     413        (JSC::Heap::converge):
     414        (JSC::Heap::visitWeakHandles):
     415        (JSC::Heap::updateObjectCounts):
     416        (JSC::Heap::clearUnmarkedExecutables):
     417        (JSC::Heap::deleteUnmarkedCompiledCode):
     418        (JSC::Heap::collectAllGarbage):
     419        (JSC::Heap::collect):
     420        (JSC::Heap::collectWithoutAnySweep):
     421        (JSC::Heap::collectImpl):
     422        (JSC::Heap::suspendCompilerThreads):
     423        (JSC::Heap::willStartCollection):
     424        (JSC::Heap::flushOldStructureIDTables):
     425        (JSC::Heap::flushWriteBarrierBuffer):
     426        (JSC::Heap::stopAllocation):
     427        (JSC::Heap::prepareForMarking):
     428        (JSC::Heap::reapWeakHandles):
     429        (JSC::Heap::pruneStaleEntriesFromWeakGCMaps):
     430        (JSC::Heap::sweepArrayBuffers):
     431        (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor):
     432        (JSC::MarkedBlockSnapshotFunctor::operator()):
     433        (JSC::Heap::snapshotMarkedSpace):
     434        (JSC::Heap::deleteSourceProviderCaches):
     435        (JSC::Heap::notifyIncrementalSweeper):
     436        (JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks):
     437        (JSC::Heap::resetAllocators):
     438        (JSC::Heap::updateAllocationLimits):
     439        (JSC::Heap::didFinishCollection):
     440        (JSC::Heap::resumeCompilerThreads):
     441        (JSC::Zombify::visit):
     442        (JSC::Heap::forEachCodeBlockImpl):
     443        * heap/Heap.h:
     444        (JSC::Heap::allocatorForObjectWithoutDestructor):
     445        (JSC::Heap::allocatorForObjectWithDestructor):
     446        (JSC::Heap::allocatorForAuxiliaryData):
     447        (JSC::Heap::jitStubRoutines):
     448        (JSC::Heap::codeBlockSet):
     449        (JSC::Heap::storageAllocator): Deleted.
     450        * heap/HeapCell.h:
     451        (JSC::HeapCell::isZapped): Deleted.
     452        * heap/HeapCellInlines.h: Added.
     453        (JSC::HeapCell::isLargeAllocation):
     454        (JSC::HeapCell::cellContainer):
     455        (JSC::HeapCell::markedBlock):
     456        (JSC::HeapCell::largeAllocation):
     457        (JSC::HeapCell::heap):
     458        (JSC::HeapCell::vm):
     459        (JSC::HeapCell::cellSize):
     460        (JSC::HeapCell::allocatorAttributes):
     461        (JSC::HeapCell::destructionMode):
     462        (JSC::HeapCell::cellKind):
     463        * heap/HeapInlines.h:
     464        (JSC::Heap::heap):
     465        (JSC::Heap::isLive):
     466        (JSC::Heap::isMarked):
     467        (JSC::Heap::testAndSetMarked):
     468        (JSC::Heap::setMarked):
     469        (JSC::Heap::cellSize):
     470        (JSC::Heap::forEachCodeBlock):
     471        (JSC::Heap::allocateObjectOfType):
     472        (JSC::Heap::subspaceForObjectOfType):
     473        (JSC::Heap::allocatorForObjectOfType):
     474        (JSC::Heap::allocateAuxiliary):
     475        (JSC::Heap::tryAllocateAuxiliary):
     476        (JSC::Heap::tryReallocateAuxiliary):
     477        (JSC::Heap::isPointerGCObject): Deleted.
     478        (JSC::Heap::isValueGCObject): Deleted.
     479        * heap/HeapOperation.cpp: Added.
     480        (WTF::printInternal):
     481        * heap/HeapOperation.h:
     482        * heap/HeapUtil.h: Added.
     483        (JSC::HeapUtil::findGCObjectPointersForMarking):
     484        (JSC::HeapUtil::isPointerGCObjectJSCell):
     485        (JSC::HeapUtil::isValueGCObject):
     486        * heap/IncrementalSweeper.cpp:
     487        (JSC::IncrementalSweeper::sweepNextBlock):
     488        * heap/IncrementalSweeper.h:
     489        * heap/LargeAllocation.cpp: Added.
     490        (JSC::LargeAllocation::tryCreate):
     491        (JSC::LargeAllocation::LargeAllocation):
     492        (JSC::LargeAllocation::lastChanceToFinalize):
     493        (JSC::LargeAllocation::shrink):
     494        (JSC::LargeAllocation::visitWeakSet):
     495        (JSC::LargeAllocation::reapWeakSet):
     496        (JSC::LargeAllocation::flip):
     497        (JSC::LargeAllocation::isEmpty):
     498        (JSC::LargeAllocation::sweep):
     499        (JSC::LargeAllocation::destroy):
     500        (JSC::LargeAllocation::dump):
     501        * heap/LargeAllocation.h: Added.
     502        (JSC::LargeAllocation::fromCell):
     503        (JSC::LargeAllocation::cell):
     504        (JSC::LargeAllocation::isLargeAllocation):
     505        (JSC::LargeAllocation::heap):
     506        (JSC::LargeAllocation::vm):
     507        (JSC::LargeAllocation::weakSet):
     508        (JSC::LargeAllocation::clearNewlyAllocated):
     509        (JSC::LargeAllocation::isNewlyAllocated):
     510        (JSC::LargeAllocation::isMarked):
     511        (JSC::LargeAllocation::isMarkedOrNewlyAllocated):
     512        (JSC::LargeAllocation::isLive):
     513        (JSC::LargeAllocation::hasValidCell):
     514        (JSC::LargeAllocation::cellSize):
     515        (JSC::LargeAllocation::aboveLowerBound):
     516        (JSC::LargeAllocation::belowUpperBound):
     517        (JSC::LargeAllocation::contains):
     518        (JSC::LargeAllocation::attributes):
     519        (JSC::LargeAllocation::flipIfNecessary):
     520        (JSC::LargeAllocation::flipIfNecessaryConcurrently):
     521        (JSC::LargeAllocation::testAndSetMarked):
     522        (JSC::LargeAllocation::setMarked):
     523        (JSC::LargeAllocation::clearMarked):
     524        (JSC::LargeAllocation::noteMarked):
     525        (JSC::LargeAllocation::headerSize):
     526        * heap/MarkedAllocator.cpp:
     527        (JSC::MarkedAllocator::MarkedAllocator):
     528        (JSC::MarkedAllocator::isPagedOut):
     529        (JSC::MarkedAllocator::retire):
     530        (JSC::MarkedAllocator::filterNextBlock):
     531        (JSC::MarkedAllocator::setNextBlockToSweep):
     532        (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl):
     533        (JSC::MarkedAllocator::tryAllocateWithoutCollecting):
     534        (JSC::MarkedAllocator::allocateSlowCase):
     535        (JSC::MarkedAllocator::tryAllocateSlowCase):
     536        (JSC::MarkedAllocator::allocateSlowCaseImpl):
     537        (JSC::blockHeaderSize):
     538        (JSC::MarkedAllocator::blockSizeForBytes):
     539        (JSC::MarkedAllocator::tryAllocateBlock):
     540        (JSC::MarkedAllocator::addBlock):
     541        (JSC::MarkedAllocator::removeBlock):
     542        (JSC::MarkedAllocator::stopAllocating):
     543        (JSC::MarkedAllocator::reset):
     544        (JSC::MarkedAllocator::lastChanceToFinalize):
     545        (JSC::MarkedAllocator::setFreeList):
     546        (JSC::isListPagedOut): Deleted.
     547        (JSC::MarkedAllocator::tryAllocateHelper): Deleted.
     548        (JSC::MarkedAllocator::tryPopFreeList): Deleted.
     549        (JSC::MarkedAllocator::tryAllocate): Deleted.
     550        (JSC::MarkedAllocator::allocateBlock): Deleted.
     551        * heap/MarkedAllocator.h:
     552        (JSC::MarkedAllocator::takeLastActiveBlock):
     553        (JSC::MarkedAllocator::offsetOfFreeList):
     554        (JSC::MarkedAllocator::offsetOfCellSize):
     555        (JSC::MarkedAllocator::tryAllocate):
     556        (JSC::MarkedAllocator::allocate):
     557        (JSC::MarkedAllocator::forEachBlock):
     558        (JSC::MarkedAllocator::offsetOfFreeListHead): Deleted.
     559        (JSC::MarkedAllocator::MarkedAllocator): Deleted.
     560        (JSC::MarkedAllocator::init): Deleted.
     561        (JSC::MarkedAllocator::stopAllocating): Deleted.
     562        * heap/MarkedBlock.cpp:
     563        (JSC::MarkedBlock::tryCreate):
     564        (JSC::MarkedBlock::Handle::Handle):
     565        (JSC::MarkedBlock::Handle::~Handle):
     566        (JSC::MarkedBlock::MarkedBlock):
     567        (JSC::MarkedBlock::Handle::specializedSweep):
     568        (JSC::MarkedBlock::Handle::sweep):
     569        (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode):
     570        (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode):
     571        (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated):
     572        (JSC::SetNewlyAllocatedFunctor::SetNewlyAllocatedFunctor):
     573        (JSC::SetNewlyAllocatedFunctor::operator()):
     574        (JSC::MarkedBlock::Handle::stopAllocating):
     575        (JSC::MarkedBlock::Handle::lastChanceToFinalize):
     576        (JSC::MarkedBlock::Handle::resumeAllocating):
     577        (JSC::MarkedBlock::Handle::zap):
     578        (JSC::MarkedBlock::Handle::forEachFreeCell):
     579        (JSC::MarkedBlock::flipIfNecessary):
     580        (JSC::MarkedBlock::Handle::flipIfNecessary):
     581        (JSC::MarkedBlock::flipIfNecessarySlow):
     582        (JSC::MarkedBlock::flipIfNecessaryConcurrentlySlow):
     583        (JSC::MarkedBlock::clearMarks):
     584        (JSC::MarkedBlock::assertFlipped):
     585        (JSC::MarkedBlock::needsFlip):
     586        (JSC::MarkedBlock::Handle::needsFlip):
     587        (JSC::MarkedBlock::Handle::willRemoveBlock):
     588        (JSC::MarkedBlock::Handle::didConsumeFreeList):
     589        (JSC::MarkedBlock::markCount):
     590        (JSC::MarkedBlock::Handle::isEmpty):
     591        (JSC::MarkedBlock::clearHasAnyMarked):
     592        (JSC::MarkedBlock::noteMarkedSlow):
     593        (WTF::printInternal):
     594        (JSC::MarkedBlock::create): Deleted.
     595        (JSC::MarkedBlock::destroy): Deleted.
     596        (JSC::MarkedBlock::callDestructor): Deleted.
     597        (JSC::MarkedBlock::specializedSweep): Deleted.
     598        (JSC::MarkedBlock::sweep): Deleted.
     599        (JSC::MarkedBlock::sweepHelper): Deleted.
     600        (JSC::MarkedBlock::stopAllocating): Deleted.
     601        (JSC::MarkedBlock::clearMarksWithCollectionType): Deleted.
     602        (JSC::MarkedBlock::lastChanceToFinalize): Deleted.
     603        (JSC::MarkedBlock::resumeAllocating): Deleted.
     604        (JSC::MarkedBlock::didRetireBlock): Deleted.
     605        * heap/MarkedBlock.h:
     606        (JSC::MarkedBlock::VoidFunctor::returnValue):
     607        (JSC::MarkedBlock::CountFunctor::CountFunctor):
     608        (JSC::MarkedBlock::CountFunctor::count):
     609        (JSC::MarkedBlock::CountFunctor::returnValue):
     610        (JSC::MarkedBlock::Handle::hasAnyNewlyAllocated):
     611        (JSC::MarkedBlock::Handle::isOnBlocksToSweep):
     612        (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep):
     613        (JSC::MarkedBlock::Handle::state):
     614        (JSC::MarkedBlock::needsDestruction):
     615        (JSC::MarkedBlock::handle):
     616        (JSC::MarkedBlock::Handle::block):
     617        (JSC::MarkedBlock::firstAtom):
     618        (JSC::MarkedBlock::atoms):
     619        (JSC::MarkedBlock::isAtomAligned):
     620        (JSC::MarkedBlock::Handle::cellAlign):
     621        (JSC::MarkedBlock::blockFor):
     622        (JSC::MarkedBlock::Handle::allocator):
     623        (JSC::MarkedBlock::Handle::heap):
     624        (JSC::MarkedBlock::Handle::vm):
     625        (JSC::MarkedBlock::vm):
     626        (JSC::MarkedBlock::Handle::weakSet):
     627        (JSC::MarkedBlock::weakSet):
     628        (JSC::MarkedBlock::Handle::shrink):
     629        (JSC::MarkedBlock::Handle::visitWeakSet):
     630        (JSC::MarkedBlock::Handle::reapWeakSet):
     631        (JSC::MarkedBlock::Handle::cellSize):
     632        (JSC::MarkedBlock::cellSize):
     633        (JSC::MarkedBlock::Handle::attributes):
     634        (JSC::MarkedBlock::attributes):
     635        (JSC::MarkedBlock::Handle::needsDestruction):
     636        (JSC::MarkedBlock::Handle::destruction):
     637        (JSC::MarkedBlock::Handle::cellKind):
     638        (JSC::MarkedBlock::Handle::markCount):
     639        (JSC::MarkedBlock::Handle::size):
     640        (JSC::MarkedBlock::atomNumber):
     641        (JSC::MarkedBlock::flipIfNecessary):
     642        (JSC::MarkedBlock::flipIfNecessaryConcurrently):
     643        (JSC::MarkedBlock::Handle::flipIfNecessary):
     644        (JSC::MarkedBlock::Handle::flipIfNecessaryConcurrently):
     645        (JSC::MarkedBlock::Handle::flipForEdenCollection):
     646        (JSC::MarkedBlock::assertFlipped):
     647        (JSC::MarkedBlock::Handle::assertFlipped):
     648        (JSC::MarkedBlock::isMarked):
     649        (JSC::MarkedBlock::testAndSetMarked):
     650        (JSC::MarkedBlock::Handle::isNewlyAllocated):
     651        (JSC::MarkedBlock::Handle::setNewlyAllocated):
     652        (JSC::MarkedBlock::Handle::clearNewlyAllocated):
     653        (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated):
     654        (JSC::MarkedBlock::isMarkedOrNewlyAllocated):
     655        (JSC::MarkedBlock::Handle::isLive):
     656        (JSC::MarkedBlock::isAtom):
     657        (JSC::MarkedBlock::Handle::isLiveCell):
     658        (JSC::MarkedBlock::Handle::forEachCell):
     659        (JSC::MarkedBlock::Handle::forEachLiveCell):
     660        (JSC::MarkedBlock::Handle::forEachDeadCell):
     661        (JSC::MarkedBlock::Handle::needsSweeping):
     662        (JSC::MarkedBlock::Handle::isAllocated):
     663        (JSC::MarkedBlock::Handle::isMarked):
     664        (JSC::MarkedBlock::Handle::isFreeListed):
     665        (JSC::MarkedBlock::hasAnyMarked):
     666        (JSC::MarkedBlock::noteMarked):
     667        (WTF::MarkedBlockHash::hash):
     668        (JSC::MarkedBlock::FreeList::FreeList): Deleted.
     669        (JSC::MarkedBlock::allocator): Deleted.
     670        (JSC::MarkedBlock::heap): Deleted.
     671        (JSC::MarkedBlock::shrink): Deleted.
     672        (JSC::MarkedBlock::visitWeakSet): Deleted.
     673        (JSC::MarkedBlock::reapWeakSet): Deleted.
     674        (JSC::MarkedBlock::willRemoveBlock): Deleted.
     675        (JSC::MarkedBlock::didConsumeFreeList): Deleted.
     676        (JSC::MarkedBlock::markCount): Deleted.
     677        (JSC::MarkedBlock::isEmpty): Deleted.
     678        (JSC::MarkedBlock::destruction): Deleted.
     679        (JSC::MarkedBlock::cellKind): Deleted.
     680        (JSC::MarkedBlock::size): Deleted.
     681        (JSC::MarkedBlock::capacity): Deleted.
     682        (JSC::MarkedBlock::setMarked): Deleted.
     683        (JSC::MarkedBlock::clearMarked): Deleted.
     684        (JSC::MarkedBlock::isNewlyAllocated): Deleted.
     685        (JSC::MarkedBlock::setNewlyAllocated): Deleted.
     686        (JSC::MarkedBlock::clearNewlyAllocated): Deleted.
     687        (JSC::MarkedBlock::isLive): Deleted.
     688        (JSC::MarkedBlock::isLiveCell): Deleted.
     689        (JSC::MarkedBlock::forEachCell): Deleted.
     690        (JSC::MarkedBlock::forEachLiveCell): Deleted.
     691        (JSC::MarkedBlock::forEachDeadCell): Deleted.
     692        (JSC::MarkedBlock::needsSweeping): Deleted.
     693        (JSC::MarkedBlock::isAllocated): Deleted.
     694        (JSC::MarkedBlock::isMarkedOrRetired): Deleted.
     695        * heap/MarkedSpace.cpp:
     696        (JSC::MarkedSpace::initializeSizeClassForStepSize):
     697        (JSC::MarkedSpace::MarkedSpace):
     698        (JSC::MarkedSpace::~MarkedSpace):
     699        (JSC::MarkedSpace::lastChanceToFinalize):
     700        (JSC::MarkedSpace::allocate):
     701        (JSC::MarkedSpace::tryAllocate):
     702        (JSC::MarkedSpace::allocateLarge):
     703        (JSC::MarkedSpace::tryAllocateLarge):
     704        (JSC::MarkedSpace::sweep):
     705        (JSC::MarkedSpace::sweepLargeAllocations):
     706        (JSC::MarkedSpace::zombifySweep):
     707        (JSC::MarkedSpace::resetAllocators):
     708        (JSC::MarkedSpace::visitWeakSets):
     709        (JSC::MarkedSpace::reapWeakSets):
     710        (JSC::MarkedSpace::stopAllocating):
     711        (JSC::MarkedSpace::prepareForMarking):
     712        (JSC::MarkedSpace::resumeAllocating):
     713        (JSC::MarkedSpace::isPagedOut):
     714        (JSC::MarkedSpace::freeBlock):
     715        (JSC::MarkedSpace::freeOrShrinkBlock):
     716        (JSC::MarkedSpace::shrink):
     717        (JSC::MarkedSpace::clearNewlyAllocated):
     718        (JSC::VerifyMarked::operator()):
     719        (JSC::MarkedSpace::flip):
     720        (JSC::MarkedSpace::objectCount):
     721        (JSC::MarkedSpace::size):
     722        (JSC::MarkedSpace::capacity):
     723        (JSC::MarkedSpace::addActiveWeakSet):
     724        (JSC::MarkedSpace::didAddBlock):
     725        (JSC::MarkedSpace::didAllocateInBlock):
     726        (JSC::MarkedSpace::forEachAllocator): Deleted.
     727        (JSC::VerifyMarkedOrRetired::operator()): Deleted.
     728        (JSC::MarkedSpace::clearMarks): Deleted.
     729        * heap/MarkedSpace.h:
     730        (JSC::MarkedSpace::sizeClassToIndex):
     731        (JSC::MarkedSpace::indexToSizeClass):
     732        (JSC::MarkedSpace::version):
     733        (JSC::MarkedSpace::blocksWithNewObjects):
     734        (JSC::MarkedSpace::largeAllocations):
     735        (JSC::MarkedSpace::largeAllocationsNurseryOffset):
     736        (JSC::MarkedSpace::largeAllocationsOffsetForThisCollection):
     737        (JSC::MarkedSpace::largeAllocationsForThisCollectionBegin):
     738        (JSC::MarkedSpace::largeAllocationsForThisCollectionEnd):
     739        (JSC::MarkedSpace::largeAllocationsForThisCollectionSize):
     740        (JSC::MarkedSpace::forEachLiveCell):
     741        (JSC::MarkedSpace::forEachDeadCell):
     742        (JSC::MarkedSpace::allocatorFor):
     743        (JSC::MarkedSpace::destructorAllocatorFor):
     744        (JSC::MarkedSpace::auxiliaryAllocatorFor):
     745        (JSC::MarkedSpace::allocateWithoutDestructor):
     746        (JSC::MarkedSpace::allocateWithDestructor):
     747        (JSC::MarkedSpace::allocateAuxiliary):
     748        (JSC::MarkedSpace::tryAllocateAuxiliary):
     749        (JSC::MarkedSpace::forEachBlock):
     750        (JSC::MarkedSpace::forEachAllocator):
     751        (JSC::MarkedSpace::optimalSizeFor):
     752        (JSC::MarkedSpace::didAddBlock): Deleted.
     753        (JSC::MarkedSpace::didAllocateInBlock): Deleted.
     754        (JSC::MarkedSpace::objectCount): Deleted.
     755        (JSC::MarkedSpace::size): Deleted.
     756        (JSC::MarkedSpace::capacity): Deleted.
     757        * heap/SlotVisitor.cpp:
     758        (JSC::SlotVisitor::SlotVisitor):
     759        (JSC::SlotVisitor::didStartMarking):
     760        (JSC::SlotVisitor::reset):
     761        (JSC::SlotVisitor::append):
     762        (JSC::SlotVisitor::appendJSCellOrAuxiliary):
     763        (JSC::SlotVisitor::setMarkedAndAppendToMarkStack):
     764        (JSC::SlotVisitor::appendToMarkStack):
     765        (JSC::SlotVisitor::markAuxiliary):
     766        (JSC::SlotVisitor::noteLiveAuxiliaryCell):
     767        (JSC::SlotVisitor::visitChildren):
     768        * heap/SlotVisitor.h:
     769        * heap/WeakBlock.cpp:
     770        (JSC::WeakBlock::create):
     771        (JSC::WeakBlock::WeakBlock):
     772        (JSC::WeakBlock::visit):
     773        (JSC::WeakBlock::reap):
     774        * heap/WeakBlock.h:
     775        (JSC::WeakBlock::disconnectContainer):
     776        (JSC::WeakBlock::disconnectMarkedBlock): Deleted.
     777        * heap/WeakSet.cpp:
     778        (JSC::WeakSet::~WeakSet):
     779        (JSC::WeakSet::sweep):
     780        (JSC::WeakSet::shrink):
     781        (JSC::WeakSet::addAllocator):
     782        * heap/WeakSet.h:
     783        (JSC::WeakSet::container):
     784        (JSC::WeakSet::setContainer):
     785        (JSC::WeakSet::WeakSet):
     786        (JSC::WeakSet::visit):
     787        (JSC::WeakSet::shrink): Deleted.
     788        * heap/WeakSetInlines.h:
     789        (JSC::WeakSet::allocate):
     790        * inspector/InjectedScriptManager.cpp:
     791        * inspector/JSGlobalObjectInspectorController.cpp:
     792        * inspector/JSJavaScriptCallFrame.cpp:
     793        * inspector/ScriptDebugServer.cpp:
     794        * inspector/agents/InspectorDebuggerAgent.cpp:
     795        * interpreter/CachedCall.h:
     796        (JSC::CachedCall::CachedCall):
     797        * interpreter/Interpreter.cpp:
     798        (JSC::loadVarargs):
     799        (JSC::StackFrame::sourceID): Deleted.
     800        (JSC::StackFrame::sourceURL): Deleted.
     801        (JSC::StackFrame::functionName): Deleted.
     802        (JSC::StackFrame::computeLineAndColumn): Deleted.
     803        (JSC::StackFrame::toString): Deleted.
     804        * interpreter/Interpreter.h:
     805        (JSC::StackFrame::isNative): Deleted.
     806        * jit/AssemblyHelpers.h:
     807        (JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator):
     808        (JSC::AssemblyHelpers::emitAllocate):
     809        (JSC::AssemblyHelpers::emitAllocateJSCell):
     810        (JSC::AssemblyHelpers::emitAllocateJSObject):
     811        (JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize):
     812        (JSC::AssemblyHelpers::emitAllocateVariableSized):
     813        * jit/GCAwareJITStubRoutine.cpp:
     814        (JSC::GCAwareJITStubRoutine::GCAwareJITStubRoutine):
     815        * jit/JIT.cpp:
     816        (JSC::JIT::compileCTINativeCall):
     817        (JSC::JIT::link):
     818        * jit/JIT.h:
     819        (JSC::JIT::compileCTINativeCall): Deleted.
     820        * jit/JITExceptions.cpp:
     821        (JSC::genericUnwind):
     822        * jit/JITExceptions.h:
     823        * jit/JITOpcodes.cpp:
     824        (JSC::JIT::emit_op_new_object):
     825        (JSC::JIT::emitSlow_op_new_object):
     826        (JSC::JIT::emit_op_create_this):
     827        (JSC::JIT::emitSlow_op_create_this):
     828        * jit/JITOpcodes32_64.cpp:
     829        (JSC::JIT::emit_op_new_object):
     830        (JSC::JIT::emitSlow_op_new_object):
     831        (JSC::JIT::emit_op_create_this):
     832        (JSC::JIT::emitSlow_op_create_this):
     833        * jit/JITOperations.cpp:
     834        * jit/JITOperations.h:
     835        * jit/JITPropertyAccess.cpp:
     836        (JSC::JIT::emitWriteBarrier):
     837        * jit/JITThunks.cpp:
     838        * jit/JITThunks.h:
     839        * jsc.cpp:
     840        (functionDescribeArray):
     841        (main):
     842        * llint/LLIntData.cpp:
     843        (JSC::LLInt::Data::performAssertions):
     844        * llint/LLIntExceptions.cpp:
     845        * llint/LLIntThunks.cpp:
     846        * llint/LLIntThunks.h:
     847        * llint/LowLevelInterpreter.asm:
     848        * llint/LowLevelInterpreter.cpp:
     849        * llint/LowLevelInterpreter32_64.asm:
     850        * llint/LowLevelInterpreter64.asm:
     851        * parser/ModuleAnalyzer.cpp:
     852        * parser/NodeConstructors.h:
     853        * parser/Nodes.h:
     854        * profiler/ProfilerBytecode.cpp:
     855        * profiler/ProfilerBytecode.h:
     856        * profiler/ProfilerBytecodeSequence.cpp:
     857        * runtime/ArrayConventions.h:
     858        (JSC::indexingHeaderForArrayStorage):
     859        (JSC::baseIndexingHeaderForArrayStorage):
     860        (JSC::indexingHeaderForArray): Deleted.
     861        (JSC::baseIndexingHeaderForArray): Deleted.
     862        * runtime/ArrayPrototype.cpp:
     863        (JSC::arrayProtoFuncSplice):
     864        (JSC::concatAppendOne):
     865        (JSC::arrayProtoPrivateFuncConcatMemcpy):
     866        * runtime/ArrayStorage.h:
     867        (JSC::ArrayStorage::vectorLength):
     868        (JSC::ArrayStorage::totalSizeFor):
     869        (JSC::ArrayStorage::totalSize):
     870        (JSC::ArrayStorage::availableVectorLength):
     871        (JSC::ArrayStorage::optimalVectorLength):
     872        (JSC::ArrayStorage::sizeFor): Deleted.
     873        * runtime/AuxiliaryBarrier.h: Added.
     874        (JSC::AuxiliaryBarrier::AuxiliaryBarrier):
     875        (JSC::AuxiliaryBarrier::clear):
     876        (JSC::AuxiliaryBarrier::get):
     877        (JSC::AuxiliaryBarrier::slot):
     878        (JSC::AuxiliaryBarrier::operator bool):
     879        (JSC::AuxiliaryBarrier::setWithoutBarrier):
     880        * runtime/AuxiliaryBarrierInlines.h: Added.
     881        (JSC::AuxiliaryBarrier<T>::AuxiliaryBarrier):
     882        (JSC::AuxiliaryBarrier<T>::set):
     883        * runtime/Butterfly.h:
     884        * runtime/ButterflyInlines.h:
     885        (JSC::Butterfly::availableContiguousVectorLength):
     886        (JSC::Butterfly::optimalContiguousVectorLength):
     887        (JSC::Butterfly::createUninitialized):
     888        (JSC::Butterfly::growArrayRight):
     889        * runtime/ClonedArguments.cpp:
     890        (JSC::ClonedArguments::createEmpty):
     891        * runtime/CommonSlowPathsExceptions.cpp:
     892        * runtime/CommonSlowPathsExceptions.h:
     893        * runtime/DataView.cpp:
     894        * runtime/DirectArguments.h:
     895        * runtime/ECMAScriptSpecInternalFunctions.cpp:
     896        * runtime/Error.cpp:
     897        * runtime/Error.h:
     898        * runtime/ErrorInstance.cpp:
     899        * runtime/ErrorInstance.h:
     900        * runtime/Exception.cpp:
     901        * runtime/Exception.h:
     902        * runtime/GeneratorFrame.cpp:
     903        * runtime/GeneratorPrototype.cpp:
     904        * runtime/InternalFunction.cpp:
     905        (JSC::InternalFunction::InternalFunction):
     906        * runtime/IntlCollator.cpp:
     907        * runtime/IntlCollatorConstructor.cpp:
     908        * runtime/IntlCollatorPrototype.cpp:
     909        * runtime/IntlDateTimeFormat.cpp:
     910        * runtime/IntlDateTimeFormatConstructor.cpp:
     911        * runtime/IntlDateTimeFormatPrototype.cpp:
     912        * runtime/IntlNumberFormat.cpp:
     913        * runtime/IntlNumberFormatConstructor.cpp:
     914        * runtime/IntlNumberFormatPrototype.cpp:
     915        * runtime/IntlObject.cpp:
     916        * runtime/IteratorPrototype.cpp:
     917        * runtime/JSArray.cpp:
     918        (JSC::JSArray::tryCreateUninitialized):
     919        (JSC::JSArray::setLengthWritable):
     920        (JSC::JSArray::unshiftCountSlowCase):
     921        (JSC::JSArray::setLengthWithArrayStorage):
     922        (JSC::JSArray::appendMemcpy):
     923        (JSC::JSArray::setLength):
     924        (JSC::JSArray::pop):
     925        (JSC::JSArray::push):
     926        (JSC::JSArray::fastSlice):
     927        (JSC::JSArray::shiftCountWithArrayStorage):
     928        (JSC::JSArray::shiftCountWithAnyIndexingType):
     929        (JSC::JSArray::unshiftCountWithArrayStorage):
     930        (JSC::JSArray::fillArgList):
     931        (JSC::JSArray::copyToArguments):
     932        * runtime/JSArray.h:
     933        (JSC::createContiguousArrayButterfly):
     934        (JSC::createArrayButterfly):
     935        (JSC::JSArray::create):
     936        (JSC::JSArray::tryCreateUninitialized): Deleted.
     937        * runtime/JSArrayBufferView.h:
     938        * runtime/JSCInlines.h:
     939        * runtime/JSCJSValue.cpp:
     940        (JSC::JSValue::dumpInContextAssumingStructure):
     941        * runtime/JSCallee.cpp:
     942        (JSC::JSCallee::JSCallee):
     943        * runtime/JSCell.cpp:
     944        (JSC::JSCell::estimatedSize):
     945        * runtime/JSCell.h:
     946        (JSC::JSCell::cellStateOffset): Deleted.
     947        * runtime/JSCellInlines.h:
     948        (JSC::ExecState::vm):
     949        (JSC::JSCell::classInfo):
     950        (JSC::JSCell::callDestructor):
     951        (JSC::JSCell::vm): Deleted.
     952        * runtime/JSFunction.cpp:
     953        (JSC::JSFunction::create):
     954        (JSC::JSFunction::allocateAndInitializeRareData):
     955        (JSC::JSFunction::initializeRareData):
     956        (JSC::JSFunction::getOwnPropertySlot):
     957        (JSC::JSFunction::put):
     958        (JSC::JSFunction::deleteProperty):
     959        (JSC::JSFunction::defineOwnProperty):
     960        (JSC::JSFunction::setFunctionName):
     961        (JSC::JSFunction::reifyLength):
     962        (JSC::JSFunction::reifyName):
     963        (JSC::JSFunction::reifyLazyPropertyIfNeeded):
     964        (JSC::JSFunction::reifyBoundNameIfNeeded):
     965        * runtime/JSFunction.h:
     966        * runtime/JSFunctionInlines.h:
     967        (JSC::JSFunction::createWithInvalidatedReallocationWatchpoint):
     968        (JSC::JSFunction::JSFunction):
     969        * runtime/JSGenericTypedArrayViewInlines.h:
     970        (JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory):
     971        * runtime/JSInternalPromise.cpp:
     972        * runtime/JSInternalPromiseConstructor.cpp:
     973        * runtime/JSInternalPromiseDeferred.cpp:
     974        * runtime/JSInternalPromisePrototype.cpp:
     975        * runtime/JSJob.cpp:
     976        * runtime/JSMapIterator.cpp:
     977        * runtime/JSModuleNamespaceObject.cpp:
     978        * runtime/JSModuleRecord.cpp:
     979        * runtime/JSObject.cpp:
     980        (JSC::JSObject::visitButterfly):
     981        (JSC::JSObject::notifyPresenceOfIndexedAccessors):
     982        (JSC::JSObject::createInitialIndexedStorage):
     983        (JSC::JSObject::createInitialUndecided):
     984        (JSC::JSObject::createInitialInt32):
     985        (JSC::JSObject::createInitialDouble):
     986        (JSC::JSObject::createInitialContiguous):
     987        (JSC::JSObject::createArrayStorage):
     988        (JSC::JSObject::createInitialArrayStorage):
     989        (JSC::JSObject::convertUndecidedToInt32):
     990        (JSC::JSObject::convertUndecidedToContiguous):
     991        (JSC::JSObject::convertUndecidedToArrayStorage):
     992        (JSC::JSObject::convertInt32ToDouble):
     993        (JSC::JSObject::convertInt32ToArrayStorage):
     994        (JSC::JSObject::convertDoubleToArrayStorage):
     995        (JSC::JSObject::convertContiguousToArrayStorage):
     996        (JSC::JSObject::putByIndexBeyondVectorLength):
     997        (JSC::JSObject::putDirectIndexBeyondVectorLength):
     998        (JSC::JSObject::getNewVectorLength):
     999        (JSC::JSObject::increaseVectorLength):
     1000        (JSC::JSObject::ensureLengthSlow):
     1001        (JSC::JSObject::growOutOfLineStorage):
     1002        (JSC::JSObject::copyButterfly): Deleted.
     1003        (JSC::JSObject::copyBackingStore): Deleted.
     1004        * runtime/JSObject.h:
     1005        (JSC::JSObject::globalObject):
     1006        (JSC::JSObject::putDirectInternal):
     1007        (JSC::JSObject::setStructureAndReallocateStorageIfNecessary): Deleted.
     1008        * runtime/JSObjectInlines.h:
     1009        * runtime/JSPromise.cpp:
     1010        * runtime/JSPromiseConstructor.cpp:
     1011        * runtime/JSPromiseDeferred.cpp:
     1012        * runtime/JSPromisePrototype.cpp:
     1013        * runtime/JSPropertyNameIterator.cpp:
     1014        * runtime/JSScope.cpp:
     1015        (JSC::JSScope::resolve):
     1016        * runtime/JSScope.h:
     1017        (JSC::JSScope::globalObject):
     1018        (JSC::JSScope::vm): Deleted.
     1019        * runtime/JSSetIterator.cpp:
     1020        * runtime/JSStringIterator.cpp:
     1021        * runtime/JSTemplateRegistryKey.cpp:
     1022        * runtime/JSTypedArrayViewConstructor.cpp:
     1023        * runtime/JSTypedArrayViewPrototype.cpp:
     1024        * runtime/JSWeakMap.cpp:
     1025        * runtime/JSWeakSet.cpp:
     1026        * runtime/MapConstructor.cpp:
     1027        * runtime/MapIteratorPrototype.cpp:
     1028        * runtime/MapPrototype.cpp:
     1029        * runtime/NativeErrorConstructor.cpp:
     1030        * runtime/NativeStdFunctionCell.cpp:
     1031        * runtime/Operations.h:
     1032        (JSC::scribbleFreeCells):
     1033        (JSC::scribble):
     1034        * runtime/Options.h:
     1035        * runtime/PropertyTable.cpp:
     1036        * runtime/ProxyConstructor.cpp:
     1037        * runtime/ProxyObject.cpp:
     1038        * runtime/ProxyRevoke.cpp:
     1039        * runtime/RegExp.cpp:
     1040        (JSC::RegExp::match):
     1041        (JSC::RegExp::matchConcurrently):
     1042        (JSC::RegExp::matchCompareWithInterpreter):
     1043        * runtime/RegExp.h:
     1044        * runtime/RegExpConstructor.h:
     1045        * runtime/RegExpInlines.h:
     1046        (JSC::RegExp::matchInline):
     1047        * runtime/RegExpMatchesArray.h:
     1048        (JSC::tryCreateUninitializedRegExpMatchesArray):
     1049        (JSC::createRegExpMatchesArray):
     1050        * runtime/RegExpPrototype.cpp:
     1051        (JSC::genericSplit):
     1052        * runtime/RuntimeType.cpp:
     1053        * runtime/SamplingProfiler.cpp:
     1054        (JSC::SamplingProfiler::processUnverifiedStackTraces):
     1055        * runtime/SetConstructor.cpp:
     1056        * runtime/SetIteratorPrototype.cpp:
     1057        * runtime/SetPrototype.cpp:
     1058        * runtime/StackFrame.cpp: Added.
     1059        (JSC::StackFrame::sourceID):
     1060        (JSC::StackFrame::sourceURL):
     1061        (JSC::StackFrame::functionName):
     1062        (JSC::StackFrame::computeLineAndColumn):
     1063        (JSC::StackFrame::toString):
     1064        * runtime/StackFrame.h: Added.
     1065        (JSC::StackFrame::isNative):
     1066        * runtime/StringConstructor.cpp:
     1067        * runtime/StringIteratorPrototype.cpp:
     1068        * runtime/StructureInlines.h:
     1069        (JSC::Structure::propertyTable):
     1070        * runtime/TemplateRegistry.cpp:
     1071        * runtime/TestRunnerUtils.cpp:
     1072        (JSC::finalizeStatsAtEndOfTesting):
     1073        * runtime/TestRunnerUtils.h:
     1074        * runtime/TypeProfilerLog.cpp:
     1075        * runtime/TypeSet.cpp:
     1076        * runtime/VM.cpp:
     1077        (JSC::VM::VM):
     1078        (JSC::VM::ensureStackCapacityForCLoop):
     1079        (JSC::VM::isSafeToRecurseSoftCLoop):
     1080        * runtime/VM.h:
     1081        * runtime/VMEntryScope.h:
     1082        * runtime/VMInlines.h:
     1083        (JSC::VM::ensureStackCapacityFor):
     1084        (JSC::VM::isSafeToRecurseSoft):
     1085        * runtime/WeakMapConstructor.cpp:
     1086        * runtime/WeakMapData.cpp:
     1087        * runtime/WeakMapPrototype.cpp:
     1088        * runtime/WeakSetConstructor.cpp:
     1089        * runtime/WeakSetPrototype.cpp:
     1090        * testRegExp.cpp:
     1091        (testOneRegExp):
     1092        * tools/JSDollarVM.cpp:
     1093        * tools/JSDollarVMPrototype.cpp:
     1094        (JSC::JSDollarVMPrototype::isInObjectSpace):
     1095
    110962016-09-04  Commit Queue  <commit-queue@webkit.org>
    21097

  • trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj

    r205330 r205462  
    8989                0F04396E1B03DC0B009598B7 /* DFGCombinedLiveness.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F04396C1B03DC0B009598B7 /* DFGCombinedLiveness.h */; };
    9090                0F05C3B41683CF9200BAF45B /* DFGArrayifySlowPathGenerator.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F05C3B21683CF8F00BAF45B /* DFGArrayifySlowPathGenerator.h */; };
     91                0F070A471D543A8B006E7232 /* CellContainer.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F070A421D543A89006E7232 /* CellContainer.h */; settings = {ATTRIBUTES = (Private, ); }; };
     92                0F070A481D543A90006E7232 /* CellContainerInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F070A431D543A89006E7232 /* CellContainerInlines.h */; settings = {ATTRIBUTES = (Private, ); }; };
     93                0F070A491D543A93006E7232 /* HeapCellInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F070A441D543A89006E7232 /* HeapCellInlines.h */; settings = {ATTRIBUTES = (Private, ); }; };
     94                0F070A4A1D543A95006E7232 /* LargeAllocation.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F070A451D543A89006E7232 /* LargeAllocation.cpp */; };
     95                0F070A4B1D543A98006E7232 /* LargeAllocation.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F070A461D543A89006E7232 /* LargeAllocation.h */; settings = {ATTRIBUTES = (Private, ); }; };
    9196                0F0776BF14FF002B00102332 /* JITCompilationEffort.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F0776BD14FF002800102332 /* JITCompilationEffort.h */; settings = {ATTRIBUTES = (Private, ); }; };
    9297                0F0A75221B94BFA900110660 /* InferredType.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F0A75201B94BFA900110660 /* InferredType.cpp */; };
    … …  
    324329                0F38B01917CFE75500B144D3 /* DFGCompilationMode.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F38B01517CFE75500B144D3 /* DFGCompilationMode.cpp */; };
    325330                0F38B01A17CFE75500B144D3 /* DFGCompilationMode.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F38B01617CFE75500B144D3 /* DFGCompilationMode.h */; settings = {ATTRIBUTES = (Private, ); }; };
     331                0F38D2A21D44196800680499 /* AuxiliaryBarrier.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F38D2A01D44196600680499 /* AuxiliaryBarrier.h */; settings = {ATTRIBUTES = (Private, ); }; };
     332                0F38D2A31D44196D00680499 /* AuxiliaryBarrierInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F38D2A11D44196600680499 /* AuxiliaryBarrierInlines.h */; settings = {ATTRIBUTES = (Private, ); }; };
    326333                0F392C891B46188400844728 /* DFGOSRExitFuzz.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F392C871B46188400844728 /* DFGOSRExitFuzz.cpp */; };
    327334                0F392C8A1B46188400844728 /* DFGOSRExitFuzz.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F392C881B46188400844728 /* DFGOSRExitFuzz.h */; };
    … …  
    365372                0F4680CB14BBB17200BFE272 /* LLIntOfflineAsmConfig.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680C614BBB16900BFE272 /* LLIntOfflineAsmConfig.h */; settings = {ATTRIBUTES = (Private, ); }; };
    366373                0F4680CC14BBB17A00BFE272 /* LowLevelInterpreter.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F4680C714BBB16900BFE272 /* LowLevelInterpreter.cpp */; };
    367                 0F4680CD14BBB17D00BFE272 /* LowLevelInterpreter.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680C814BBB16900BFE272 /* LowLevelInterpreter.h */; settings = {ATTRIBUTES = (Private, ); }; };
     374                0F4680CD14BBB17D00BFE272 /* LowLevelInterpreter.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680C814BBB16900BFE272 /* LowLevelInterpreter.h */; };
    368375                0F4680D214BBD16500BFE272 /* LLIntData.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F4680CE14BBB3D100BFE272 /* LLIntData.cpp */; };
    369                 0F4680D314BBD16700BFE272 /* LLIntData.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680CF14BBB3D100BFE272 /* LLIntData.h */; settings = {ATTRIBUTES = (Private, ); }; };
     376                0F4680D314BBD16700BFE272 /* LLIntData.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680CF14BBB3D100BFE272 /* LLIntData.h */; };
    370377                0F4680D414BBD24900BFE272 /* HostCallReturnValue.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F4680D014BBC5F800BFE272 /* HostCallReturnValue.cpp */; };
    371378                0F4680D514BBD24B00BFE272 /* HostCallReturnValue.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680D114BBC5F800BFE272 /* HostCallReturnValue.h */; settings = {ATTRIBUTES = (Private, ); }; };
    … …  
    387394                0F4F29E018B6AD1C0057BC15 /* DFGStaticExecutionCountEstimationPhase.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4F29DE18B6AD1C0057BC15 /* DFGStaticExecutionCountEstimationPhase.h */; };
    388395                0F50AF3C193E8B3900674EE8 /* DFGStructureClobberState.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F50AF3B193E8B3900674EE8 /* DFGStructureClobberState.h */; };
     396                0F5513A61D5A682C00C32BD8 /* FreeList.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F5513A51D5A682A00C32BD8 /* FreeList.h */; settings = {ATTRIBUTES = (Private, ); }; };
     397                0F5513A81D5A68CD00C32BD8 /* FreeList.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F5513A71D5A68CB00C32BD8 /* FreeList.cpp */; };
    389398                0F5541B11613C1FB00CE3E25 /* SpecialPointer.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F5541AF1613C1FB00CE3E25 /* SpecialPointer.cpp */; };
    390399                0F5541B21613C1FB00CE3E25 /* SpecialPointer.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F5541B01613C1FB00CE3E25 /* SpecialPointer.h */; settings = {ATTRIBUTES = (Private, ); }; };
    … …  
    459468                0F6C73501AC9F99F00BE1682 /* VariableWriteFireDetail.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F6C734E1AC9F99F00BE1682 /* VariableWriteFireDetail.cpp */; };
    460469                0F6C73511AC9F99F00BE1682 /* VariableWriteFireDetail.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F6C734F1AC9F99F00BE1682 /* VariableWriteFireDetail.h */; settings = {ATTRIBUTES = (Private, ); }; };
     470                0F6DB7E91D6124B500CDBF8E /* StackFrame.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F6DB7E81D6124B200CDBF8E /* StackFrame.h */; settings = {ATTRIBUTES = (Private, ); }; };
     471                0F6DB7EA1D6124B800CDBF8E /* StackFrame.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F6DB7E71D6124B200CDBF8E /* StackFrame.cpp */; };
     472                0F6DB7EC1D617D1100CDBF8E /* MacroAssemblerCodeRef.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F6DB7EB1D617D0F00CDBF8E /* MacroAssemblerCodeRef.cpp */; };
    461473                0F6E845A19030BEF00562741 /* DFGVariableAccessData.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F6E845919030BEF00562741 /* DFGVariableAccessData.cpp */; };
    462474                0F6FC750196110A800E1D02D /* ComplexGetStatus.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F6FC74E196110A800E1D02D /* ComplexGetStatus.cpp */; };
    … …  
    494506                0F8335B81639C1EA001443B5 /* ArrayAllocationProfile.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F8335B51639C1E3001443B5 /* ArrayAllocationProfile.h */; settings = {ATTRIBUTES = (Private, ); }; };
    495507                0F8364B7164B0C110053329A /* DFGBranchDirection.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F8364B5164B0C0E0053329A /* DFGBranchDirection.h */; };
     508                0F86A26D1D6F796500CB0C92 /* HeapOperation.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F86A26C1D6F796200CB0C92 /* HeapOperation.cpp */; };
     509                0F86A26F1D6F7B3300CB0C92 /* GCTypeMap.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F86A26E1D6F7B3100CB0C92 /* GCTypeMap.h */; };
    496510                0F86AE201C5311C5006BE8EC /* B3ComputeDivisionMagic.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F86AE1F1C5311C5006BE8EC /* B3ComputeDivisionMagic.h */; };
    497511                0F885E111849A3BE00F1E3FA /* BytecodeUseDef.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F885E101849A3BE00F1E3FA /* BytecodeUseDef.h */; settings = {ATTRIBUTES = (Private, ); }; };
    … …  
    574588                0FA7A8EC18B413C80052371D /* Reg.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FA7A8EA18B413C80052371D /* Reg.h */; settings = {ATTRIBUTES = (Private, ); }; };
    575589                0FA7A8EE18CE4FD80052371D /* ScratchRegisterAllocator.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FA7A8ED18CE4FD80052371D /* ScratchRegisterAllocator.cpp */; };
     590                0FADE6731D4D23BE00768457 /* HeapUtil.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FADE6721D4D23BC00768457 /* HeapUtil.h */; };
    576591                0FAF7EFD165BA91B000C8455 /* JITDisassembler.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FAF7EFA165BA919000C8455 /* JITDisassembler.cpp */; };
    577592                0FAF7EFE165BA91F000C8455 /* JITDisassembler.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FAF7EFB165BA919000C8455 /* JITDisassembler.h */; settings = {ATTRIBUTES = (Private, ); }; };
    … …  
    594609                0FB387901BFBC44D00E3AB1E /* AirOptimizeBlockOrder.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FB3878D1BFBC44D00E3AB1E /* AirOptimizeBlockOrder.h */; };
    595610                0FB387921BFD31A100E3AB1E /* FTLCompile.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FB387911BFD31A100E3AB1E /* FTLCompile.cpp */; };
     611                0FB415841D78FB4C00DF8D09 /* ArrayConventions.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FB415831D78F98200DF8D09 /* ArrayConventions.cpp */; };
    596612                0FB438A319270B1D00E1FBC9 /* StructureSet.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FB438A219270B1D00E1FBC9 /* StructureSet.cpp */; };
    597613                0FB4FB731BC843140025CA5A /* FTLLazySlowPath.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FB4FB701BC843140025CA5A /* FTLLazySlowPath.cpp */; };
    … …  
    9901006                14280870107EC1340013E7B2 /* JSWrapperObject.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 65C7A1710A8EAACB00FA37EA /* JSWrapperObject.cpp */; };
    9911007                14280875107EC13E0013E7B2 /* JSLock.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 65EA4C99092AF9E20093D800 /* JSLock.cpp */; };
    992                 1429D77C0ED20D7300B89619 /* Interpreter.h in Headers */ = {isa = PBXBuildFile; fileRef = 1429D77B0ED20D7300B89619 /* Interpreter.h */; settings = {ATTRIBUTES = (Private, ); }; };
     1008                1429D77C0ED20D7300B89619 /* Interpreter.h in Headers */ = {isa = PBXBuildFile; fileRef = 1429D77B0ED20D7300B89619 /* Interpreter.h */; };
    9931009                1429D7D40ED2128200B89619 /* Interpreter.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 1429D7D30ED2128200B89619 /* Interpreter.cpp */; };
    9941010                1429D8780ED21ACD00B89619 /* ExceptionHelpers.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 1429D8770ED21ACD00B89619 /* ExceptionHelpers.cpp */; };
    … …  
    14501466                969A07990ED1D3AE00F1F681 /* Instruction.h in Headers */ = {isa = PBXBuildFile; fileRef = 969A07930ED1D3AE00F1F681 /* Instruction.h */; settings = {ATTRIBUTES = (Private, ); }; };
    14511467                969A079A0ED1D3AE00F1F681 /* Opcode.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 969A07940ED1D3AE00F1F681 /* Opcode.cpp */; };
    1452                 969A079B0ED1D3AE00F1F681 /* Opcode.h in Headers */ = {isa = PBXBuildFile; fileRef = 969A07950ED1D3AE00F1F681 /* Opcode.h */; settings = {ATTRIBUTES = (Private, ); }; };
     1468                969A079B0ED1D3AE00F1F681 /* Opcode.h in Headers */ = {isa = PBXBuildFile; fileRef = 969A07950ED1D3AE00F1F681 /* Opcode.h */; };
    14531469                978801401471AD920041B016 /* JSDateMath.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 9788FC221471AD0C0068CE2D /* JSDateMath.cpp */; };
    14541470                978801411471AD920041B016 /* JSDateMath.h in Headers */ = {isa = PBXBuildFile; fileRef = 9788FC231471AD0C0068CE2D /* JSDateMath.h */; settings = {ATTRIBUTES = (Private, ); }; };
    … …  
    23142330                0F04396C1B03DC0B009598B7 /* DFGCombinedLiveness.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGCombinedLiveness.h; path = dfg/DFGCombinedLiveness.h; sourceTree = "<group>"; };
    23152331                0F05C3B21683CF8F00BAF45B /* DFGArrayifySlowPathGenerator.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGArrayifySlowPathGenerator.h; path = dfg/DFGArrayifySlowPathGenerator.h; sourceTree = "<group>"; };
     2332                0F070A421D543A89006E7232 /* CellContainer.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CellContainer.h; sourceTree = "<group>"; };
     2333                0F070A431D543A89006E7232 /* CellContainerInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CellContainerInlines.h; sourceTree = "<group>"; };
     2334                0F070A441D543A89006E7232 /* HeapCellInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = HeapCellInlines.h; sourceTree = "<group>"; };
     2335                0F070A451D543A89006E7232 /* LargeAllocation.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = LargeAllocation.cpp; sourceTree = "<group>"; };
     2336                0F070A461D543A89006E7232 /* LargeAllocation.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = LargeAllocation.h; sourceTree = "<group>"; };
    23162337                0F0776BD14FF002800102332 /* JITCompilationEffort.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JITCompilationEffort.h; sourceTree = "<group>"; };
    23172338                0F0A75201B94BFA900110660 /* InferredType.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = InferredType.cpp; sourceTree = "<group>"; };
    … …  
    25482569                0F38B01517CFE75500B144D3 /* DFGCompilationMode.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGCompilationMode.cpp; path = dfg/DFGCompilationMode.cpp; sourceTree = "<group>"; };
    25492570                0F38B01617CFE75500B144D3 /* DFGCompilationMode.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGCompilationMode.h; path = dfg/DFGCompilationMode.h; sourceTree = "<group>"; };
     2571                0F38D2A01D44196600680499 /* AuxiliaryBarrier.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AuxiliaryBarrier.h; sourceTree = "<group>"; };
     2572                0F38D2A11D44196600680499 /* AuxiliaryBarrierInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AuxiliaryBarrierInlines.h; sourceTree = "<group>"; };
    25502573                0F392C871B46188400844728 /* DFGOSRExitFuzz.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGOSRExitFuzz.cpp; path = dfg/DFGOSRExitFuzz.cpp; sourceTree = "<group>"; };
    25512574                0F392C881B46188400844728 /* DFGOSRExitFuzz.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGOSRExitFuzz.h; path = dfg/DFGOSRExitFuzz.h; sourceTree = "<group>"; };
    … …  
    26092632                0F4F29DE18B6AD1C0057BC15 /* DFGStaticExecutionCountEstimationPhase.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGStaticExecutionCountEstimationPhase.h; path = dfg/DFGStaticExecutionCountEstimationPhase.h; sourceTree = "<group>"; };
    26102633                0F50AF3B193E8B3900674EE8 /* DFGStructureClobberState.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGStructureClobberState.h; path = dfg/DFGStructureClobberState.h; sourceTree = "<group>"; };
     2634                0F5513A51D5A682A00C32BD8 /* FreeList.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = FreeList.h; sourceTree = "<group>"; };
     2635                0F5513A71D5A68CB00C32BD8 /* FreeList.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = FreeList.cpp; sourceTree = "<group>"; };
    26112636                0F5541AF1613C1FB00CE3E25 /* SpecialPointer.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = SpecialPointer.cpp; sourceTree = "<group>"; };
    26122637                0F5541B01613C1FB00CE3E25 /* SpecialPointer.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = SpecialPointer.h; sourceTree = "<group>"; };
    … …  
    26832708                0F6C734E1AC9F99F00BE1682 /* VariableWriteFireDetail.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = VariableWriteFireDetail.cpp; sourceTree = "<group>"; };
    26842709                0F6C734F1AC9F99F00BE1682 /* VariableWriteFireDetail.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = VariableWriteFireDetail.h; sourceTree = "<group>"; };
     2710                0F6DB7E71D6124B200CDBF8E /* StackFrame.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = StackFrame.cpp; sourceTree = "<group>"; };
     2711                0F6DB7E81D6124B200CDBF8E /* StackFrame.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = StackFrame.h; sourceTree = "<group>"; };
     2712                0F6DB7EB1D617D0F00CDBF8E /* MacroAssemblerCodeRef.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = MacroAssemblerCodeRef.cpp; sourceTree = "<group>"; };
    26852713                0F6E845919030BEF00562741 /* DFGVariableAccessData.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGVariableAccessData.cpp; path = dfg/DFGVariableAccessData.cpp; sourceTree = "<group>"; };
    26862714                0F6FC74E196110A800E1D02D /* ComplexGetStatus.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ComplexGetStatus.cpp; sourceTree = "<group>"; };
    … …  
    27162744                0F8335B51639C1E3001443B5 /* ArrayAllocationProfile.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ArrayAllocationProfile.h; sourceTree = "<group>"; };
    27172745                0F8364B5164B0C0E0053329A /* DFGBranchDirection.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGBranchDirection.h; path = dfg/DFGBranchDirection.h; sourceTree = "<group>"; };
     2746                0F86A26C1D6F796200CB0C92 /* HeapOperation.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = HeapOperation.cpp; sourceTree = "<group>"; };
     2747                0F86A26E1D6F7B3100CB0C92 /* GCTypeMap.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = GCTypeMap.h; sourceTree = "<group>"; };
    27182748                0F86AE1F1C5311C5006BE8EC /* B3ComputeDivisionMagic.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = B3ComputeDivisionMagic.h; path = b3/B3ComputeDivisionMagic.h; sourceTree = "<group>"; };
    27192749                0F885E101849A3BE00F1E3FA /* BytecodeUseDef.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = BytecodeUseDef.h; sourceTree = "<group>"; };
    … …  
    27952825                0FA7A8EA18B413C80052371D /* Reg.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = Reg.h; sourceTree = "<group>"; };
    27962826                0FA7A8ED18CE4FD80052371D /* ScratchRegisterAllocator.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ScratchRegisterAllocator.cpp; sourceTree = "<group>"; };
     2827                0FADE6721D4D23BC00768457 /* HeapUtil.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = HeapUtil.h; sourceTree = "<group>"; };
    27972828                0FAF7EFA165BA919000C8455 /* JITDisassembler.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = JITDisassembler.cpp; sourceTree = "<group>"; };
    27982829                0FAF7EFB165BA919000C8455 /* JITDisassembler.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JITDisassembler.h; sourceTree = "<group>"; };
    … …  
    28152846                0FB3878D1BFBC44D00E3AB1E /* AirOptimizeBlockOrder.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = AirOptimizeBlockOrder.h; path = b3/air/AirOptimizeBlockOrder.h; sourceTree = "<group>"; };
    28162847                0FB387911BFD31A100E3AB1E /* FTLCompile.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = FTLCompile.cpp; path = ftl/FTLCompile.cpp; sourceTree = "<group>"; };
     2848                0FB415831D78F98200DF8D09 /* ArrayConventions.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ArrayConventions.cpp; sourceTree = "<group>"; };
    28172849                0FB438A219270B1D00E1FBC9 /* StructureSet.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = StructureSet.cpp; sourceTree = "<group>"; };
    28182850                0FB4B51016B3A964003F696B /* DFGMinifiedID.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGMinifiedID.h; path = dfg/DFGMinifiedID.h; sourceTree = "<group>"; };
    … …  
    52405272                                0F9630351D4192C3005609D9 /* AllocatorAttributes.cpp */,
    52415273                                0F9630361D4192C3005609D9 /* AllocatorAttributes.h */,
     5274                                0F070A421D543A89006E7232 /* CellContainer.h */,
     5275                                0F070A431D543A89006E7232 /* CellContainerInlines.h */,
    52425276                                0F1C3DD91BBCE09E00E523E4 /* CellState.h */,
    52435277                                0FD8A31117D4326C00CA2C40 /* CodeBlockSet.cpp */,
    … …  
    52645298                                2A83638318D7D0EE0000EBCC /* EdenGCActivityCallback.cpp */,
    52655299                                2A83638418D7D0EE0000EBCC /* EdenGCActivityCallback.h */,
     5300                                0F5513A71D5A68CB00C32BD8 /* FreeList.cpp */,
     5301                                0F5513A51D5A682A00C32BD8 /* FreeList.h */,
    52665302                                2A83638718D7D0FE0000EBCC /* FullGCActivityCallback.cpp */,
    52675303                                2A83638818D7D0FE0000EBCC /* FullGCActivityCallback.h */,
    … …  
    52775313                                2A343F7418A1748B0039B085 /* GCSegmentedArray.h */,
    52785314                                2A343F7718A1749D0039B085 /* GCSegmentedArrayInlines.h */,
     5315                                0F86A26E1D6F7B3100CB0C92 /* GCTypeMap.h */,
    52795316                                142E312B134FF0A600AFADB5 /* Handle.h */,
    52805317                                C28318FF16FE4B7D00157BFD /* HandleBlock.h */,
    … …  
    52895326                                DC3D2B0B1D34376E00BA918C /* HeapCell.cpp */,
    52905327                                DC3D2B091D34316100BA918C /* HeapCell.h */,
     5328                                0F070A441D543A89006E7232 /* HeapCellInlines.h */,
    52915329                                0F32BD0E1BB34F190093A57F /* HeapHelperPool.cpp */,
    52925330                                0F32BD0F1BB34F190093A57F /* HeapHelperPool.h */,
    … …  
    52945332                                2AD8932917E3868F00668276 /* HeapIterationScope.h */,
    52955333                                A5339EC81BB4B4510054F005 /* HeapObserver.h */,
     5334                                0F86A26C1D6F796200CB0C92 /* HeapOperation.cpp */,
    52965335                                2A6F462517E959CE00C45C98 /* HeapOperation.h */,
    52975336                                A5398FA91C750D950060A963 /* HeapProfiler.cpp */,
    … …  
    53065345                                C2E526BB1590EF000054E48D /* HeapTimer.cpp */,
    53075346                                C2E526BC1590EF000054E48D /* HeapTimer.h */,
     5347                                0FADE6721D4D23BC00768457 /* HeapUtil.h */,
    53085348                                FE7BA60D1A1A7CEC00F1F7B4 /* HeapVerifier.cpp */,
    53095349                                FE7BA60E1A1A7CEC00F1F7B4 /* HeapVerifier.h */,
    … …  
    53125352                                0F766D2915A8CC34008F363E /* JITStubRoutineSet.cpp */,
    53135353                                0F766D2A15A8CC34008F363E /* JITStubRoutineSet.h */,
     5354                                0F070A451D543A89006E7232 /* LargeAllocation.cpp */,
     5355                                0F070A461D543A89006E7232 /* LargeAllocation.h */,
    53145356                                0F431736146BAC65007E3890 /* ListableHandler.h */,
    53155357                                FE3913511B794AC900EDAF71 /* LiveObjectData.h */,
    … …  
    56635705                                BC7952060E15E8A800A898AB /* ArrayConstructor.cpp */,
    56645706                                BC7952070E15E8A800A898AB /* ArrayConstructor.h */,
     5707                                0FB415831D78F98200DF8D09 /* ArrayConventions.cpp */,
    56655708                                0FB7F38915ED8E3800F167B2 /* ArrayConventions.h */,
    56665709                                A7BDAEC217F4EA1400F6140C /* ArrayIteratorPrototype.cpp */,
    … …  
    56695712                                F692A84E0255597D01FF60F7 /* ArrayPrototype.h */,
    56705713                                0FB7F38A15ED8E3800F167B2 /* ArrayStorage.h */,
     5714                                0F38D2A01D44196600680499 /* AuxiliaryBarrier.h */,
     5715                                0F38D2A11D44196600680499 /* AuxiliaryBarrierInlines.h */,
    56715716                                52678F8C1A031009006A306D /* BasicBlockLocation.cpp */,
    56725717                                52678F8D1A031009006A306D /* BasicBlockLocation.h */,
    … …  
    58335878                                93ADFCE60CCBD7AC00D30B08 /* JSArray.cpp */,
    58345879                                938772E5038BFE19008635CE /* JSArray.h */,
    5835                                 539FB8B91C99DA7C00940FA1 /* JSArrayInlines.h */,
    58365880                                0F2B66B417B6B5AB00A7AE3F /* JSArrayBuffer.cpp */,
    58375881                                0F2B66B517B6B5AB00A7AE3F /* JSArrayBuffer.h */,
    … …  
    58435887                                0F2B66BB17B6B5AB00A7AE3F /* JSArrayBufferView.h */,
    58445888                                0F2B66BC17B6B5AB00A7AE3F /* JSArrayBufferViewInlines.h */,
     5889                                539FB8B91C99DA7C00940FA1 /* JSArrayInlines.h */,
    58455890                                86FA9E8F142BBB2D001773B7 /* JSBoundFunction.cpp */,
    58465891                                86FA9E90142BBB2E001773B7 /* JSBoundFunction.h */,
    … …  
    59115956                                E3D239C61B829C1C00BBEF67 /* JSModuleEnvironment.cpp */,
    59125957                                E3D239C71B829C1C00BBEF67 /* JSModuleEnvironment.h */,
     5958                                1879510614C540FFB561C124 /* JSModuleLoader.cpp */,
     5959                                77B25CB2C3094A92A38E1DB3 /* JSModuleLoader.h */,
    59135960                                E318CBBE1B8AEF5100A2929D /* JSModuleNamespaceObject.cpp */,
    59145961                                E318CBBF1B8AEF5100A2929D /* JSModuleNamespaceObject.h */,
    … …  
    61036150                                0FB7F39215ED8E3800F167B2 /* SparseArrayValueMap.h */,
    61046151                                0F3AC751183EA1040032029F /* StackAlignment.h */,
     6152                                0F6DB7E71D6124B200CDBF8E /* StackFrame.cpp */,
     6153                                0F6DB7E81D6124B200CDBF8E /* StackFrame.h */,
    61056154                                A730B6111250068F009D25B1 /* StrictEvalActivation.cpp */,
    61066155                                A730B6101250068F009D25B1 /* StrictEvalActivation.h */,
    … …  
    61916240                                A7DCB77912E3D90500911940 /* WriteBarrier.h */,
    61926241                                C2B6D75218A33793004A9301 /* WriteBarrierInlines.h */,
    6193                                 77B25CB2C3094A92A38E1DB3 /* JSModuleLoader.h */,
    6194                                 1879510614C540FFB561C124 /* JSModuleLoader.cpp */,
    61956242                        );
    61966243                        path = runtime;
    … …  
    66356682                                A729009B17976C6000317298 /* MacroAssemblerARMv7.cpp */,
    66366683                                86ADD1440FDDEA980006EEC2 /* MacroAssemblerARMv7.h */,
     6684                                0F6DB7EB1D617D0F00CDBF8E /* MacroAssemblerCodeRef.cpp */,
    66376685                                863B23DF0FC60E6200703AA4 /* MacroAssemblerCodeRef.h */,
    66386686                                86C568DE11A213EE0007F7F0 /* MacroAssemblerMIPS.h */,
    … …  
    72467294                                0F338E111BF0276C0013C88F /* B3OpaqueByproduct.h in Headers */,
    72477295                                FEA0C4031CDD7D1D00481991 /* FunctionWhitelist.h in Headers */,
     7296                                0F6DB7E91D6124B500CDBF8E /* StackFrame.h in Headers */,
    72487297                                E3A421431D6F58930007C617 /* PreciseJumpTargetsInlines.h in Headers */,
    72497298                                99DA00AA1BD5993100F4575C /* builtins_generate_separate_implementation.py in Headers */,
    … …  
    73387387                                0F9495881C57F47500413A48 /* B3StackSlot.h in Headers */,
    73397388                                C4F4B6F31A05C944005CAB76 /* cpp_generator_templates.py in Headers */,
     7389                                0F38D2A21D44196800680499 /* AuxiliaryBarrier.h in Headers */,
    73407390                                5DE6E5B30E1728EC00180407 /* create_hash_table in Headers */,
    73417391                                9959E92B1BD17FA4001AA413 /* cssmin.py in Headers */,
    … …  
    74677517                                99D6A1161BEAD34D00E25C37 /* RemoteAutomationTarget.h in Headers */,
    74687518                                79C4B15E1BA2158F00FD592E /* DFGLiveCatchVariablePreservationPhase.h in Headers */,
     7519                                0F86A26F1D6F7B3300CB0C92 /* GCTypeMap.h in Headers */,
    74697520                                A7D89CFC17A0B8CC00773AD8 /* DFGLivenessAnalysisPhase.h in Headers */,
    74707521                                0FF0F19B16B729FA005DF95B /* DFGLongLivedState.h in Headers */,
    … …  
    75637614                                0FC97F4218202119002C9B26 /* DFGWatchpointCollectionPhase.h in Headers */,
    75647615                                0FDB2CE8174830A2007B3C1B /* DFGWorklist.h in Headers */,
     7616                                0F070A491D543A93006E7232 /* HeapCellInlines.h in Headers */,
    75657617                                0FE050181AA9091100D33B33 /* DirectArguments.h in Headers */,
    75667618                                0FE050161AA9091100D33B33 /* DirectArgumentsOffset.h in Headers */,
    … …  
    76937745                                FE3A06C01C11041A00390FDD /* JITRightShiftGenerator.h in Headers */,
    76947746                                708EBE241CE8F35800453146 /* IntlObjectInlines.h in Headers */,
     7747                                0F070A481D543A90006E7232 /* CellContainerInlines.h in Headers */,
    76957748                                FE6029D91D6E1E4F0030204D /* ThrowScopeLocation.h in Headers */,
    76967749                                0FE0501B1AA9091100D33B33 /* GenericOffset.h in Headers */,
    … …  
    77807833                                FE187A0F1C030D6C0038BBCA /* SnippetOperand.h in Headers */,
    77817834                                A1587D701B4DC14100D69849 /* IntlDateTimeFormatConstructor.h in Headers */,
     7835                                0FADE6731D4D23BE00768457 /* HeapUtil.h in Headers */,
    77827836                                A1587D751B4DC1C600D69849 /* IntlDateTimeFormatConstructor.lut.h in Headers */,
    77837837                                A5398FAB1C750DA40060A963 /* HeapProfiler.h in Headers */,
    … …  
    79047958                                2A4BB7F318A41179008A0FCD /* JSManagedValueInternal.h in Headers */,
    79057959                                A700874217CBE8EB00C3E643 /* JSMap.h in Headers */,
     7960                                0F38D2A31D44196D00680499 /* AuxiliaryBarrierInlines.h in Headers */,
    79067961                                A74DEF96182D991400522C22 /* JSMapIterator.h in Headers */,
    79077962                                9959E92D1BD17FA4001AA413 /* jsmin.py in Headers */,
    … …  
    79417996                                86E85539111B9968001AF51E /* JSStringBuilder.h in Headers */,
    79427997                                70EC0EC31AA0D7DA00B6AAFA /* JSStringIterator.h in Headers */,
     7998                                0F070A471D543A8B006E7232 /* CellContainer.h in Headers */,
    79437999                                2600B5A7152BAAA70091EE5F /* JSStringJoiner.h in Headers */,
    79448000                                BC18C4280E16F5CD00B34460 /* JSStringRef.h in Headers */,
    … …  
    80118067                                86C36EEA0EE1289D00B3DF59 /* MacroAssembler.h in Headers */,
    80128068                                43422A671C16267800E2EB98 /* B3ReduceDoubleToFloat.h in Headers */,
     8069                                0F070A4B1D543A98006E7232 /* LargeAllocation.h in Headers */,
    80138070                                86D3B2C610156BDE002865E7 /* MacroAssemblerARM.h in Headers */,
    80148071                                A1A009C01831A22D00CF8711 /* MacroAssemblerARM64.h in Headers */,
    … …  
    80588115                                996B73211BDA08EF00331B84 /* NumberPrototype.lut.h in Headers */,
    80598116                                142D3939103E4560007DCB52 /* NumericStrings.h in Headers */,
     8117                                0F5513A61D5A682C00C32BD8 /* FreeList.h in Headers */,
    80608118                                A5EA710C19F6DE820098F5EC /* objc_generator.py in Headers */,
    80618119                                C4F4B6F61A05C984005CAB76 /* objc_generator_templates.py in Headers */,
    … …  
    88408898                                0F4DE1CE1C4C1B54004D6C11 /* AirFixObviousSpills.cpp in Sources */,
    88418899                                0FEC85711BDACDC70080FF74 /* AirBasicBlock.cpp in Sources */,
     8900                                0F070A4A1D543A95006E7232 /* LargeAllocation.cpp in Sources */,
    88428901                                0FEC85731BDACDC70080FF74 /* AirCCallSpecial.cpp in Sources */,
    88438902                                0FEC85751BDACDC70080FF74 /* AirCode.cpp in Sources */,
    … …  
    91019160                                0FBE0F7416C1DB090082C5E8 /* DFGPredictionInjectionPhase.cpp in Sources */,
    91029161                                0FFFC95D14EF90B300C72532 /* DFGPredictionPropagationPhase.cpp in Sources */,
     9162                                0F86A26D1D6F796500CB0C92 /* HeapOperation.cpp in Sources */,
    91039163                                0F3E01AA19D353A500F61B7F /* DFGPrePostNumbering.cpp in Sources */,
    91049164                                0F2B9CEC19D0BA7D00B1D1B5 /* DFGPromotedHeapLocation.cpp in Sources */,
    … …  
    92629322                                A1B9E2391B4E0D6700BC7FED /* IntlCollator.cpp in Sources */,
    92639323                                A1B9E23B1B4E0D6700BC7FED /* IntlCollatorConstructor.cpp in Sources */,
     9324                                0F6DB7EA1D6124B800CDBF8E /* StackFrame.cpp in Sources */,
    92649325                                A1B9E23D1B4E0D6700BC7FED /* IntlCollatorPrototype.cpp in Sources */,
    92659326                                A1587D6D1B4DC14100D69849 /* IntlDateTimeFormat.cpp in Sources */,
    … …  
    92859346                                0F8F94441667635400D61971 /* JITCode.cpp in Sources */,
    92869347                                0FAF7EFD165BA91B000C8455 /* JITDisassembler.cpp in Sources */,
     9348                                0F6DB7EC1D617D1100CDBF8E /* MacroAssemblerCodeRef.cpp in Sources */,
    92879349                                0F46808314BA573100BFE272 /* JITExceptions.cpp in Sources */,
    92889350                                0FB14E1E18124ACE009B6B4D /* JITInlineCacheGenerator.cpp in Sources */,
    … …  
    92909352                                A71236E51195F33C00BD2174 /* JITOpcodes32_64.cpp in Sources */,
    92919353                                0F24E54C17EE274900ABB217 /* JITOperations.cpp in Sources */,
     9354                                0F5513A81D5A68CD00C32BD8 /* FreeList.cpp in Sources */,
    92929355                                FE99B24A1C24C3D700C82159 /* JITNegGenerator.cpp in Sources */,
    92939356                                86CC85C40EE7A89400288682 /* JITPropertyAccess.cpp in Sources */,
    92949357                                A7C1E8E4112E72EF00A37F98 /* JITPropertyAccess32_64.cpp in Sources */,
     9358                                0FB415841D78FB4C00DF8D09 /* ArrayConventions.cpp in Sources */,
    92959359                                0F766D2815A8CC1E008F363E /* JITStubRoutine.cpp in Sources */,
    92969360                                0F766D2B15A8CC38008F363E /* JITStubRoutineSet.cpp in Sources */,

  • trunk/Source/JavaScriptCore/Scripts/builtins/builtins_generate_combined_implementation.py

    r204912 r205462  
    7373            ),
    7474            (["JavaScriptCore", "WebCore"],
     75                ("JavaScriptCore", "heap/HeapInlines.h"),
     76            ),
     77            (["JavaScriptCore", "WebCore"],
    7578                ("JavaScriptCore", "runtime/Executable.h"),
    7679            ),

  • trunk/Source/JavaScriptCore/Scripts/builtins/builtins_generate_internals_wrapper_implementation.py

    r204912 r205462  
    6767            (["WebCore"],
    6868                ("WebCore", "WebCoreJSClientData.h"),
     69            ),
     70            (["WebCore"],
     71                ("JavaScriptCore", "heap/HeapInlines.h"),
    6972            ),
    7073            (["WebCore"],

  • trunk/Source/JavaScriptCore/Scripts/builtins/builtins_generate_separate_implementation.py

    r204912 r205462  
    8585            ),
    8686            (["JavaScriptCore", "WebCore"],
     87                ("JavaScriptCore", "heap/HeapInlines.h"),
     88            ),
     89            (["JavaScriptCore", "WebCore"],
    8790                ("JavaScriptCore", "runtime/Executable.h"),
    8891            ),

  • trunk/Source/JavaScriptCore/assembler/AbstractMacroAssembler.h

    r204912 r205462  
    726726        }
    727727
    728         void link(AbstractMacroAssemblerType* masm)
     728        void link(AbstractMacroAssemblerType* masm) const
    729729        {
    730730            size_t size = m_jumps.size();
    731731            for (size_t i = 0; i < size; ++i)
    732732                m_jumps[i].link(masm);
    733             m_jumps.clear();
    734         }
    735        
    736         void linkTo(Label label, AbstractMacroAssemblerType* masm)
     733        }
     734       
     735        void linkTo(Label label, AbstractMacroAssemblerType* masm) const
    737736        {
    738737            size_t size = m_jumps.size();
    739738            for (size_t i = 0; i < size; ++i)
    740739                m_jumps[i].linkTo(label, masm);
    741             m_jumps.clear();
    742740        }
    743741       

  • trunk/Source/JavaScriptCore/assembler/MacroAssembler.h

    r204912 r205462  
    2828
    2929#if ENABLE(ASSEMBLER)
     30
     31#include "JSCJSValue.h"
    3032
    3133#if CPU(ARM_THUMB2)

  • trunk/Source/JavaScriptCore/assembler/MacroAssemblerARM64.h

    r205283 r205462  
    167167        else if (isUInt12(-imm.m_value))
    168168            m_assembler.sub<32>(dest, src, UInt12(-imm.m_value));
    169         else {
     169        else if (src != dest) {
     170            move(imm, dest);
     171            add32(src, dest);
     172        } else {
    170173            move(imm, getCachedDataTempRegisterIDAndInvalidate());
    171174            m_assembler.add<32>(dest, src, dataTempRegister);

  • trunk/Source/JavaScriptCore/assembler/MacroAssemblerCodeRef.h

    r204912 r205462  
    2929#include "Disassembler.h"
    3030#include "ExecutableAllocator.h"
    31 #include "LLIntData.h"
    3231#include <wtf/DataLog.h>
    3332#include <wtf/PassRefPtr.h>
    … …  
    5453namespace JSC {
    5554
     55enum OpcodeID : unsigned;
     56
    5657// FunctionPtr:
    5758//
    … …  
    274275    }
    275276
    276     static MacroAssemblerCodePtr createLLIntCodePtr(OpcodeID codeId)
    277     {
    278         return createFromExecutableAddress(LLInt::getCodePtr(codeId));
    279     }
     277    static MacroAssemblerCodePtr createLLIntCodePtr(OpcodeID codeId);
    280278
    281279    explicit MacroAssemblerCodePtr(ReturnAddressPtr ra)
    … …  
    300298    }
    301299
    302     void dumpWithName(const char* name, PrintStream& out) const
    303     {
    304         if (!m_value) {
    305             out.print(name, "(null)");
    306             return;
    307         }
    308         if (executableAddress() == dataLocation()) {
    309             out.print(name, "(", RawPointer(executableAddress()), ")");
    310             return;
    311         }
    312         out.print(name, "(executable = ", RawPointer(executableAddress()), ", dataLocation = ", RawPointer(dataLocation()), ")");
    313     }
    314    
    315     void dump(PrintStream& out) const
    316     {
    317         dumpWithName("CodePtr", out);
    318     }
     300    void dumpWithName(const char* name, PrintStream& out) const;
     301   
     302    void dump(PrintStream& out) const;
    319303   
    320304    enum EmptyValueTag { EmptyValue };
    … …  
    390374   
    391375    // Helper for creating self-managed code refs from LLInt.
    392     static MacroAssemblerCodeRef createLLIntCodeRef(OpcodeID codeId)
    393     {
    394         return createSelfManagedCodeRef(MacroAssemblerCodePtr::createFromExecutableAddress(LLInt::getCodePtr(codeId)));
    395     }
     376    static MacroAssemblerCodeRef createLLIntCodeRef(OpcodeID codeId);
    396377
    397378    ExecutableMemoryHandle* executableMemory() const
    … …  
    419400    explicit operator bool() const { return !!m_codePtr; }
    420401   
    421     void dump(PrintStream& out) const
    422     {
    423         m_codePtr.dumpWithName("CodeRef", out);
    424     }
     402    void dump(PrintStream& out) const;
    425403
    426404private:

  • trunk/Source/JavaScriptCore/b3/B3BasicBlock.cpp

    r204912 r205462  
    8686}
    8787
     88Value* BasicBlock::appendBoolConstant(Procedure& proc, Origin origin, bool value)
     89{
     90    return appendIntConstant(proc, origin, Int32, value ? 1 : 0);
     91}
     92
    8893void BasicBlock::clearSuccessors()
    8994{

  • trunk/Source/JavaScriptCore/b3/B3BasicBlock.h

    r204912 r205462  
    8383    JS_EXPORT_PRIVATE Value* appendIntConstant(Procedure&, Origin, Type, int64_t value);
    8484    Value* appendIntConstant(Procedure&, Value* likeValue, int64_t value);
     85    Value* appendBoolConstant(Procedure&, Origin, bool);
    8586
    8687    void removeLast(Procedure&);

  • trunk/Source/JavaScriptCore/b3/B3DuplicateTails.cpp

    r204920 r205462  
    7272
    7373        for (BasicBlock* block : m_proc) {
    74             if (block->size() > m_maxSize || block->numSuccessors() > m_maxSuccessors)
     74            if (block->size() > m_maxSize)
     75                continue;
     76            if (block->numSuccessors() > m_maxSuccessors)
     77                continue;
     78            if (block->last()->type() != Void) // Demoting doesn't handle terminals with values.
    7579                continue;
    7680

  • trunk/Source/JavaScriptCore/b3/B3StackmapGenerationParams.h

    r204912 r205462  
    9393    // Returns true if the successor at the given index is going to be emitted right after the
    9494    // patchpoint.
    95     bool fallsThroughToSuccessor(unsigned successorIndex) const;
     95    JS_EXPORT_PRIVATE bool fallsThroughToSuccessor(unsigned successorIndex) const;
    9696
    9797    // This is provided for convenience; it means that you don't have to capture it if you don't want to.

  • trunk/Source/JavaScriptCore/b3/testb3.cpp

    r204912 r205462  
    1291912919    CHECK_EQ(terminal.args[1].kind(), firstKind);
    1292012920    CHECK(terminal.args[2].kind() == Air::Arg::BitImm || terminal.args[2].kind() == Air::Arg::BitImm64);
     12921}
     12922
     12923void testPatchpointTerminalReturnValue(bool successIsRare)
     12924{
     12925    // This is a unit test for how FTL's heap allocation fast paths behave.
     12926    Procedure proc;
     12927   
     12928    BasicBlock* root = proc.addBlock();
     12929    BasicBlock* success = proc.addBlock();
     12930    BasicBlock* slowPath = proc.addBlock();
     12931    BasicBlock* continuation = proc.addBlock();
     12932   
     12933    Value* arg = root->appendNew<Value>(
     12934        proc, Trunc, Origin(),
     12935        root->appendNew<ArgumentRegValue>(proc, Origin(), GPRInfo::argumentGPR0));
     12936   
     12937    PatchpointValue* patchpoint = root->appendNew<PatchpointValue>(proc, Int32, Origin());
     12938    patchpoint->effects.terminal = true;
     12939    patchpoint->clobber(RegisterSet::macroScratchRegisters());
     12940   
     12941    if (successIsRare) {
     12942        root->appendSuccessor(FrequentedBlock(success, FrequencyClass::Rare));
     12943        root->appendSuccessor(slowPath);
     12944    } else {
     12945        root->appendSuccessor(success);
     12946        root->appendSuccessor(FrequentedBlock(slowPath, FrequencyClass::Rare));
     12947    }
     12948   
     12949    patchpoint->appendSomeRegister(arg);
     12950   
     12951    patchpoint->setGenerator(
     12952        [&] (CCallHelpers& jit, const StackmapGenerationParams& params) {
     12953            AllowMacroScratchRegisterUsage allowScratch(jit);
     12954           
     12955            CCallHelpers::Jump jumpToSlow =
     12956                jit.branch32(CCallHelpers::Above, params[1].gpr(), CCallHelpers::TrustedImm32(42));
     12957           
     12958            jit.add32(CCallHelpers::TrustedImm32(31), params[1].gpr(), params[0].gpr());
     12959           
     12960            CCallHelpers::Jump jumpToSuccess;
     12961            if (!params.fallsThroughToSuccessor(0))
     12962                jumpToSuccess = jit.jump();
     12963           
     12964            Vector<Box<CCallHelpers::Label>> labels = params.successorLabels();
     12965           
     12966            params.addLatePath(
     12967                [=] (CCallHelpers& jit) {
     12968                    jumpToSlow.linkTo(*labels[1], &jit);
     12969                    if (jumpToSuccess.isSet())
     12970                        jumpToSuccess.linkTo(*labels[0], &jit);
     12971                });
     12972        });
     12973   
     12974    UpsilonValue* successUpsilon = success->appendNew<UpsilonValue>(proc, Origin(), patchpoint);
     12975    success->appendNew<Value>(proc, Jump, Origin());
     12976    success->setSuccessors(continuation);
     12977   
     12978    UpsilonValue* slowPathUpsilon = slowPath->appendNew<UpsilonValue>(
     12979        proc, Origin(), slowPath->appendNew<Const32Value>(proc, Origin(), 666));
     12980    slowPath->appendNew<Value>(proc, Jump, Origin());
     12981    slowPath->setSuccessors(continuation);
     12982   
     12983    Value* phi = continuation->appendNew<Value>(proc, Phi, Int32, Origin());
     12984    successUpsilon->setPhi(phi);
     12985    slowPathUpsilon->setPhi(phi);
     12986    continuation->appendNew<Value>(proc, Return, Origin(), phi);
     12987   
     12988    auto code = compile(proc);
     12989    CHECK_EQ(invoke<int>(*code, 0), 31);
     12990    CHECK_EQ(invoke<int>(*code, 1), 32);
     12991    CHECK_EQ(invoke<int>(*code, 41), 72);
     12992    CHECK_EQ(invoke<int>(*code, 42), 73);
     12993    CHECK_EQ(invoke<int>(*code, 43), 666);
     12994    CHECK_EQ(invoke<int>(*code, -1), 666);
    1292112995}
    1292212996
    … …  
    1433814412
    1433914413    RUN(testSomeEarlyRegister());
     14414    RUN(testPatchpointTerminalReturnValue(true));
     14415    RUN(testPatchpointTerminalReturnValue(false));
    1434014416   
    1434114417    if (isX86()) {

  • trunk/Source/JavaScriptCore/bindings/ScriptValue.cpp

    r205324 r205462  
    3333#include "APICast.h"
    3434#include "InspectorValues.h"
     35#include "JSCInlines.h"
    3536#include "JSLock.h"
    36 #include "JSObjectInlines.h"
    37 #include "StructureInlines.h"
    3837
    3938using namespace JSC;

  • trunk/Source/JavaScriptCore/bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp

    r204912 r205462  
    2727#include "AdaptiveInferredPropertyValueWatchpointBase.h"
    2828
    29 #include "JSCellInlines.h"
    30 #include "StructureInlines.h"
     29#include "JSCInlines.h"
    3130
    3231namespace JSC {

  • trunk/Source/JavaScriptCore/bytecode/BytecodeLivenessAnalysis.cpp

    r204994 r205462  
    3333#include "FullBytecodeLiveness.h"
    3434#include "InterpreterInlines.h"
     35#include "PreciseJumpTargets.h"
    3536
    3637namespace JSC {

  • trunk/Source/JavaScriptCore/bytecode/BytecodeRewriter.cpp

    r204994 r205462  
    2828#include "BytecodeRewriter.h"
    2929
     30#include "HeapInlines.h"
    3031#include "PreciseJumpTargetsInlines.h"
    3132#include <wtf/BubbleSort.h>

  • trunk/Source/JavaScriptCore/bytecode/BytecodeUseDef.h

    r204994 r205462  
    2828
    2929#include "CodeBlock.h"
     30#include "Interpreter.h"
    3031
    3132namespace JSC {

  • trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.cpp

    r204912 r205462  
    3131#include "DFGThunks.h"
    3232#include "JSCInlines.h"
     33#include "Opcode.h"
    3334#include "Repatch.h"
    3435#include <wtf/ListDump.h>
    … …  
    3637#if ENABLE(JIT)
    3738namespace JSC {
     39
     40CallLinkInfo::CallType CallLinkInfo::callTypeFor(OpcodeID opcodeID)
     41{
     42    if (opcodeID == op_call || opcodeID == op_call_eval)
     43        return Call;
     44    if (opcodeID == op_call_varargs)
     45        return CallVarargs;
     46    if (opcodeID == op_construct)
     47        return Construct;
     48    if (opcodeID == op_construct_varargs)
     49        return ConstructVarargs;
     50    if (opcodeID == op_tail_call)
     51        return TailCall;
     52    ASSERT(opcodeID == op_tail_call_varargs || op_tail_call_forward_arguments);
     53    return TailCallVarargs;
     54}
    3855
    3956CallLinkInfo::CallLinkInfo()

  • trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.h

    r204912 r205462  
    3232#include "JITWriteBarrier.h"
    3333#include "JSFunction.h"
    34 #include "Opcode.h"
    3534#include "PolymorphicCallStubRoutine.h"
    3635#include "WriteBarrier.h"
    … …  
    4140#if ENABLE(JIT)
    4241
     42enum OpcodeID : unsigned;
    4343struct CallFrameShuffleData;
    4444
    … …  
    4646public:
    4747    enum CallType { None, Call, CallVarargs, Construct, ConstructVarargs, TailCall, TailCallVarargs };
    48     static CallType callTypeFor(OpcodeID opcodeID)
    49     {
    50         if (opcodeID == op_call || opcodeID == op_call_eval)
    51             return Call;
    52         if (opcodeID == op_call_varargs)
    53             return CallVarargs;
    54         if (opcodeID == op_construct)
    55             return Construct;
    56         if (opcodeID == op_construct_varargs)
    57             return ConstructVarargs;
    58         if (opcodeID == op_tail_call)
    59             return TailCall;
    60         ASSERT(opcodeID == op_tail_call_varargs || op_tail_call_forward_arguments);
    61         return TailCallVarargs;
    62     }
     48    static CallType callTypeFor(OpcodeID opcodeID);
    6349
    6450    static bool isVarargsCallType(CallType callType)

  • trunk/Source/JavaScriptCore/bytecode/CallLinkStatus.cpp

    r204912 r205462  
    3131#include "DFGJITCode.h"
    3232#include "InlineCallFrame.h"
     33#include "Interpreter.h"
    3334#include "LLIntCallLinkInfo.h"
    3435#include "JSCInlines.h"

  • trunk/Source/JavaScriptCore/bytecode/CodeBlock.cpp

    r205321 r205462  
    5353#include "JSLexicalEnvironment.h"
    5454#include "JSModuleEnvironment.h"
     55#include "LLIntData.h"
    5556#include "LLIntEntrypoint.h"
    5657#include "LLIntPrototypeLoadAdaptiveStructureWatchpoint.h"
    … …  
    7172#include <wtf/BagToHashMap.h>
    7273#include <wtf/CommaPrinter.h>
     74#include <wtf/SimpleStats.h>
    7375#include <wtf/StringExtras.h>
    7476#include <wtf/StringPrintStream.h>
    … …  
    18771879    }
    18781880   
    1879     heap()->m_codeBlocks.add(this);
     1881    heap()->m_codeBlocks->add(this);
    18801882}
    18811883
    … …  
    23432345        dumpBytecode();
    23442346   
    2345     heap()->m_codeBlocks.add(this);
     2347    heap()->m_codeBlocks->add(this);
    23462348    heap()->reportExtraMemoryAllocated(m_instructions.size() * sizeof(Instruction));
    23472349}
    … …  
    23802382    Base::finishCreation(vm);
    23812383
    2382     heap()->m_codeBlocks.add(this);
     2384    heap()->m_codeBlocks->add(this);
    23832385}
    23842386#endif
    … …  
    27822784    codeBlock->propagateTransitions(visitor);
    27832785    codeBlock->determineLiveness(visitor);
     2786}
     2787
     2788void CodeBlock::clearLLIntGetByIdCache(Instruction* instruction)
     2789{
     2790    instruction[0].u.opcode = LLInt::getOpcode(op_get_by_id);
     2791    instruction[4].u.pointer = nullptr;
     2792    instruction[5].u.pointer = nullptr;
     2793    instruction[6].u.pointer = nullptr;
    27842794}
    27852795
    … …  
    41854195        return 0;
    41864196   
    4187     if (!m_vm->machineCodeBytesPerBytecodeWordForBaselineJIT)
     4197    if (!*m_vm->machineCodeBytesPerBytecodeWordForBaselineJIT)
    41884198        return 0; // It's as good of a prediction as we'll get.
    41894199   
    41904200    // Be conservative: return a size that will be an overestimation 84% of the time.
    4191     double multiplier = m_vm->machineCodeBytesPerBytecodeWordForBaselineJIT.mean() +
    4192         m_vm->machineCodeBytesPerBytecodeWordForBaselineJIT.standardDeviation();
     4201    double multiplier = m_vm->machineCodeBytesPerBytecodeWordForBaselineJIT->mean() +
     4202        m_vm->machineCodeBytesPerBytecodeWordForBaselineJIT->standardDeviation();
    41934203   
    41944204    // Be paranoid: silently reject bogus multipiers. Silently doing the "wrong" thing

  • trunk/Source/JavaScriptCore/bytecode/CodeBlock.h

    r204994 r205462  
    294294        return m_jitCodeMap.get();
    295295    }
     296   
     297    static void clearLLIntGetByIdCache(Instruction*);
    296298
    297299    unsigned bytecodeOffset(Instruction* returnAddress)
    … …  
    12841286#endif
    12851287
    1286 inline void clearLLIntGetByIdCache(Instruction* instruction)
    1287 {
    1288     instruction[0].u.opcode = LLInt::getOpcode(op_get_by_id);
    1289     instruction[4].u.pointer = nullptr;
    1290     instruction[5].u.pointer = nullptr;
    1291     instruction[6].u.pointer = nullptr;
    1292 }
    1293 
    12941288inline Register& ExecState::r(int index)
    12951289{

  • trunk/Source/JavaScriptCore/bytecode/ExecutionCounter.h

    r183506 r205462  
    3030#include "Options.h"
    3131#include <wtf/PrintStream.h>
    32 #include <wtf/SimpleStats.h>
    3332
    3433namespace JSC {

  • trunk/Source/JavaScriptCore/bytecode/Instruction.h

    r204912 r205462  
    3232#include "BasicBlockLocation.h"
    3333#include "MacroAssembler.h"
    34 #include "Opcode.h"
    3534#include "PutByIdFlags.h"
    3635#include "SymbolTable.h"
    … …  
    5251struct LLIntCallLinkInfo;
    5352struct ValueProfile;
     53
     54#if ENABLE(COMPUTED_GOTO_OPCODES)
     55typedef void* Opcode;
     56#else
     57typedef OpcodeID Opcode;
     58#endif
    5459
    5560struct Instruction {

  • trunk/Source/JavaScriptCore/bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp

    r204912 r205462  
    2929#include "CodeBlock.h"
    3030#include "Instruction.h"
    31 #include "StructureInlines.h"
     31#include "JSCInlines.h"
    3232
    3333namespace JSC {
    … …  
    6060    StringFireDetail stringDetail(out.toCString().data());
    6161
    62     clearLLIntGetByIdCache(m_getByIdInstruction);
     62    CodeBlock::clearLLIntGetByIdCache(m_getByIdInstruction);
    6363}
    6464

  • trunk/Source/JavaScriptCore/bytecode/ObjectAllocationProfile.h

    r204912 r205462  
    4646    }
    4747
    48     bool isNull() { return !m_allocator; }
     48    bool isNull() { return !m_structure; }
    4949
    5050    void initialize(VM& vm, JSCell* owner, JSObject* prototype, unsigned inferredInlineCapacity)
    … …  
    8181
    8282        size_t allocationSize = JSFinalObject::allocationSize(inlineCapacity);
    83         MarkedAllocator* allocator = &vm.heap.allocatorForObjectWithoutDestructor(allocationSize);
    84         ASSERT(allocator->cellSize());
    85 
     83        MarkedAllocator* allocator = vm.heap.allocatorForObjectWithoutDestructor(allocationSize);
     84       
    8685        // Take advantage of extra inline capacity available in the size class.
    87         size_t slop = (allocator->cellSize() - allocationSize) / sizeof(WriteBarrier<Unknown>);
    88         inlineCapacity += slop;
    89         if (inlineCapacity > JSFinalObject::maxInlineCapacity())
    90             inlineCapacity = JSFinalObject::maxInlineCapacity();
     86        if (allocator) {
     87            size_t slop = (allocator->cellSize() - allocationSize) / sizeof(WriteBarrier<Unknown>);
     88            inlineCapacity += slop;
     89            if (inlineCapacity > JSFinalObject::maxInlineCapacity())
     90                inlineCapacity = JSFinalObject::maxInlineCapacity();
     91        }
    9192
    9293        Structure* structure = vm.prototypeMap.emptyObjectStructureForPrototype(prototype, inlineCapacity);

  • trunk/Source/JavaScriptCore/bytecode/Opcode.h

    r204994 r205462  
    5656
    5757#define OPCODE_ID_ENUM(opcode, length) opcode,
    58     typedef enum { FOR_EACH_OPCODE_ID(OPCODE_ID_ENUM) } OpcodeID;
     58    enum OpcodeID : unsigned { FOR_EACH_OPCODE_ID(OPCODE_ID_ENUM) };
    5959#undef OPCODE_ID_ENUM
    6060

  • trunk/Source/JavaScriptCore/bytecode/PolymorphicAccess.cpp

    r204912 r205462  
    12071207           
    12081208            if (allocatingInline) {
    1209                 CopiedAllocator* copiedAllocator = &vm.heap.storageAllocator();
    1210 
    1211                 if (!reallocating) {
    1212                     jit.loadPtr(&copiedAllocator->m_currentRemaining, scratchGPR);
    1213                     slowPath.append(
    1214                         jit.branchSubPtr(
    1215                             CCallHelpers::Signed, CCallHelpers::TrustedImm32(newSize), scratchGPR));
    1216                     jit.storePtr(scratchGPR, &copiedAllocator->m_currentRemaining);
    1217                     jit.negPtr(scratchGPR);
    1218                     jit.addPtr(
    1219                         CCallHelpers::AbsoluteAddress(&copiedAllocator->m_currentPayloadEnd), scratchGPR);
    1220                     jit.addPtr(CCallHelpers::TrustedImm32(sizeof(JSValue)), scratchGPR);
    1221                 } else {
     1209                MarkedAllocator* allocator = vm.heap.allocatorForAuxiliaryData(newSize);
     1210               
     1211                if (!allocator) {
     1212                    // Yuck, this case would suck!
     1213                    slowPath.append(jit.jump());
     1214                }
     1215               
     1216                jit.move(CCallHelpers::TrustedImmPtr(allocator), scratchGPR2);
     1217                jit.emitAllocate(scratchGPR, allocator, scratchGPR2, scratchGPR3, slowPath);
     1218                jit.addPtr(CCallHelpers::TrustedImm32(newSize + sizeof(IndexingHeader)), scratchGPR);
     1219               
     1220                if (reallocating) {
    12221221                    // Handle the case where we are reallocating (i.e. the old structure/butterfly
    12231222                    // already had out-of-line property storage).
    … …  
    12261225           
    12271226                    jit.loadPtr(CCallHelpers::Address(baseGPR, JSObject::butterflyOffset()), scratchGPR3);
    1228                     jit.loadPtr(&copiedAllocator->m_currentRemaining, scratchGPR);
    1229                     slowPath.append(
    1230                         jit.branchSubPtr(
    1231                             CCallHelpers::Signed, CCallHelpers::TrustedImm32(newSize), scratchGPR));
    1232                     jit.storePtr(scratchGPR, &copiedAllocator->m_currentRemaining);
    1233                     jit.negPtr(scratchGPR);
    1234                     jit.addPtr(
    1235                         CCallHelpers::AbsoluteAddress(&copiedAllocator->m_currentPayloadEnd), scratchGPR);
    1236                     jit.addPtr(CCallHelpers::TrustedImm32(sizeof(JSValue)), scratchGPR);
     1227                   
    12371228                    // We have scratchGPR = new storage, scratchGPR3 = old storage,
    12381229                    // scratchGPR2 = available
    … …  
    16601651        for (unsigned i = cases.size(); i--;) {
    16611652            fallThrough.link(&jit);
     1653            fallThrough.clear();
    16621654            cases[i]->generateWithGuard(state, fallThrough);
    16631655        }

  • trunk/Source/JavaScriptCore/bytecode/PolymorphicAccess.h

    r204912 r205462  
    3030
    3131#include "CodeOrigin.h"
     32#include "JITStubRoutine.h"
    3233#include "JSFunctionInlines.h"
    3334#include "MacroAssembler.h"
    3435#include "ObjectPropertyConditionSet.h"
    35 #include "Opcode.h"
    3636#include "ScratchRegisterAllocator.h"
    3737#include "Structure.h"

  • trunk/Source/JavaScriptCore/bytecode/StructureStubInfo.cpp

    r204912 r205462  
    2828
    2929#include "JSObject.h"
     30#include "JSCInlines.h"
    3031#include "PolymorphicAccess.h"
    3132#include "Repatch.h"

  • trunk/Source/JavaScriptCore/bytecode/StructureStubInfo.h

    r204912 r205462  
    3232#include "MacroAssembler.h"
    3333#include "ObjectPropertyConditionSet.h"
    34 #include "Opcode.h"
    3534#include "Options.h"
    3635#include "RegisterSet.h"

  • trunk/Source/JavaScriptCore/bytecode/SuperSampler.cpp

    r198396 r205462  
    3737volatile uint32_t g_superSamplerCount;
    3838
     39static StaticLock lock;
    3940static double in;
    4041static double out;
    … …  
    5253            for (;;) {
    5354                for (int ms = 0; ms < printingPeriod; ms += sleepQuantum) {
    54                     if (g_superSamplerCount)
    55                         in++;
    56                     else
    57                         out++;
     55                    {
     56                        LockHolder locker(lock);
     57                        if (g_superSamplerCount)
     58                            in++;
     59                        else
     60                            out++;
     61                    }
    5862                    sleepMS(sleepQuantum);
    5963                }
    … …  
    6569}
    6670
     71void resetSuperSamplerState()
     72{
     73    LockHolder locker(lock);
     74    in = 0;
     75    out = 0;
     76}
     77
    6778void printSuperSamplerState()
    6879{
    … …  
    7081        return;
    7182
     83    LockHolder locker(lock);
    7284    double percentage = 100.0 * in / (in + out);
    7385    if (percentage != percentage)

  • trunk/Source/JavaScriptCore/bytecode/SuperSampler.h

    r198364 r205462  
    5454};
    5555
     56JS_EXPORT_PRIVATE void resetSuperSamplerState();
    5657JS_EXPORT_PRIVATE void printSuperSamplerState();
    5758

  • trunk/Source/JavaScriptCore/bytecode/UnlinkedCodeBlock.cpp

    r204994 r205462  
    9191}
    9292
    93 VM* UnlinkedCodeBlock::vm() const
    94 {
    95     return MarkedBlock::blockFor(this)->vm();
    96 }
    97 
    9893void UnlinkedCodeBlock::visitChildren(JSCell* cell, SlotVisitor& visitor)
    9994{

  • trunk/Source/JavaScriptCore/bytecode/UnlinkedCodeBlock.h

    r204994 r205462  
    284284    void addExceptionHandler(const UnlinkedHandlerInfo& handler) { createRareDataIfNecessary(); return m_rareData->m_exceptionHandlers.append(handler); }
    285285    UnlinkedHandlerInfo& exceptionHandler(int index) { ASSERT(m_rareData); return m_rareData->m_exceptionHandlers[index]; }
    286 
    287     VM* vm() const;
    288286
    289287    UnlinkedArrayProfile addArrayProfile() { return m_arrayProfileCount++; }

  • trunk/Source/JavaScriptCore/bytecode/UnlinkedInstructionStream.cpp

    r204912 r205462  
    2626#include "config.h"
    2727#include "UnlinkedInstructionStream.h"
     28
     29#include "Opcode.h"
    2830
    2931namespace JSC {

  • trunk/Source/JavaScriptCore/bytecode/UnlinkedInstructionStream.h

    r204912 r205462  
    2828#define UnlinkedInstructionStream_h
    2929
     30#include "Opcode.h"
    3031#include "UnlinkedCodeBlock.h"
    3132#include <wtf/RefCountedArray.h>

  • trunk/Source/JavaScriptCore/dfg/DFGCallArrayAllocatorSlowPathGenerator.h

    r204439 r205462  
    3939public:
    4040    CallArrayAllocatorSlowPathGenerator(
    41         MacroAssembler::JumpList from, SpeculativeJIT* jit, P_JITOperation_EStZ function,
     41        MacroAssembler::JumpList from, SpeculativeJIT* jit, P_JITOperation_EStZB function,
    4242        GPRReg resultGPR, GPRReg storageGPR, Structure* structure, size_t size)
    4343        : JumpingSlowPathGenerator<MacroAssembler::JumpList>(from, jit)
    … …  
    5858        for (unsigned i = 0; i < m_plans.size(); ++i)
    5959            jit->silentSpill(m_plans[i]);
    60         jit->callOperation(m_function, m_resultGPR, m_structure, m_size);
     60        jit->callOperation(m_function, m_resultGPR, m_structure, m_size, m_storageGPR);
    6161        GPRReg canTrample = SpeculativeJIT::pickCanTrample(m_resultGPR);
    6262        for (unsigned i = m_plans.size(); i--;)
    … …  
    6868   
    6969private:
    70     P_JITOperation_EStZ m_function;
     70    P_JITOperation_EStZB m_function;
    7171    GPRReg m_resultGPR;
    7272    GPRReg m_storageGPR;
    … …  
    7979public:
    8080    CallArrayAllocatorWithVariableSizeSlowPathGenerator(
    81         MacroAssembler::JumpList from, SpeculativeJIT* jit, P_JITOperation_EStZ function,
    82         GPRReg resultGPR, Structure* contiguousStructure, Structure* arrayStorageStructure, GPRReg sizeGPR)
     81        MacroAssembler::JumpList from, SpeculativeJIT* jit, P_JITOperation_EStZB function,
     82        GPRReg resultGPR, Structure* contiguousStructure, Structure* arrayStorageStructure, GPRReg sizeGPR, GPRReg storageGPR)
    8383        : JumpingSlowPathGenerator<MacroAssembler::JumpList>(from, jit)
    8484        , m_function(function)
    … …  
    8787        , m_arrayStorageOrContiguousStructure(arrayStorageStructure)
    8888        , m_sizeGPR(sizeGPR)
     89        , m_storageGPR(storageGPR)
    8990    {
    9091        jit->silentSpillAllRegistersImpl(false, m_plans, resultGPR);
    … …  
    9798        for (unsigned i = 0; i < m_plans.size(); ++i)
    9899            jit->silentSpill(m_plans[i]);
    99         GPRReg scratchGPR = AssemblyHelpers::selectScratchGPR(m_sizeGPR);
     100        GPRReg scratchGPR = AssemblyHelpers::selectScratchGPR(m_sizeGPR, m_storageGPR);
    100101        if (m_contiguousStructure != m_arrayStorageOrContiguousStructure) {
    101102            MacroAssembler::Jump bigLength = jit->m_jit.branch32(MacroAssembler::AboveOrEqual, m_sizeGPR, MacroAssembler::TrustedImm32(MIN_ARRAY_STORAGE_CONSTRUCTION_LENGTH));
    … …  
    107108        } else
    108109            jit->m_jit.move(MacroAssembler::TrustedImmPtr(m_contiguousStructure), scratchGPR);
    109         jit->callOperation(m_function, m_resultGPR, scratchGPR, m_sizeGPR);
     110        jit->callOperation(m_function, m_resultGPR, scratchGPR, m_sizeGPR, m_storageGPR);
    110111        GPRReg canTrample = SpeculativeJIT::pickCanTrample(m_resultGPR);
    111112        for (unsigned i = m_plans.size(); i--;)
    … …  
    116117   
    117118private:
    118     P_JITOperation_EStZ m_function;
     119    P_JITOperation_EStZB m_function;
    119120    GPRReg m_resultGPR;
    120121    Structure* m_contiguousStructure;
    121122    Structure* m_arrayStorageOrContiguousStructure;
    122123    GPRReg m_sizeGPR;
     124    GPRReg m_storageGPR;
    123125    Vector<SilentRegisterSavePlan, 2> m_plans;
    124126};

  • trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp

    r205198 r205462  
    934934}
    935935
    936 char* JIT_OPERATION operationNewArrayWithSize(ExecState* exec, Structure* arrayStructure, int32_t size)
    937 {
    938     VM* vm = &exec->vm();
    939     NativeCallFrameTracer tracer(vm, exec);
    940     auto scope = DECLARE_THROW_SCOPE(*vm);
     936char* JIT_OPERATION operationNewArrayWithSize(ExecState* exec, Structure* arrayStructure, int32_t size, Butterfly* butterfly)
     937{
     938    VM& vm = exec->vm();
     939    NativeCallFrameTracer tracer(&vm, exec);
     940    auto scope = DECLARE_THROW_SCOPE(vm);
    941941
    942942    if (UNLIKELY(size < 0))
    943943        return bitwise_cast<char*>(throwException(exec, scope, createRangeError(exec, ASCIILiteral("Array size is not a small enough positive integer."))));
    944944
    945     JSArray* result = JSArray::create(*vm, arrayStructure, size);
    946     result->butterfly(); // Ensure that the backing store is in to-space.
     945    JSArray* result;
     946    if (butterfly)
     947        result = JSArray::createWithButterfly(vm, arrayStructure, butterfly);
     948    else
     949        result = JSArray::create(vm, arrayStructure, size);
    947950    return bitwise_cast<char*>(result);
    948951}
    … …  
    16301633}
    16311634
    1632 char* JIT_OPERATION operationNewRawObject(ExecState* exec, Structure* structure, int32_t length)
    1633 {
    1634     VM& vm = exec->vm();
    1635     NativeCallFrameTracer tracer(&vm, exec);
    1636 
    1637     Butterfly* butterfly;
    1638     if (structure->outOfLineCapacity() || hasIndexedProperties(structure->indexingType())) {
     1635char* JIT_OPERATION operationNewRawObject(ExecState* exec, Structure* structure, int32_t length, Butterfly* butterfly)
     1636{
     1637    VM& vm = exec->vm();
     1638    NativeCallFrameTracer tracer(&vm, exec);
     1639
     1640    if (!butterfly
     1641        && (structure->outOfLineCapacity() || hasIndexedProperties(structure->indexingType()))) {
    16391642        IndexingHeader header;
    16401643        header.setVectorLength(length);
    … …  
    16451648            hasIndexedProperties(structure->indexingType()), header,
    16461649            length * sizeof(EncodedJSValue));
    1647     } else
    1648         butterfly = nullptr;
     1650    }
    16491651
    16501652    JSObject* result = JSObject::createRawObject(exec, structure, butterfly);
    … …  
    16531655}
    16541656
    1655 JSCell* JIT_OPERATION operationNewObjectWithButterfly(ExecState* exec, Structure* structure)
    1656 {
    1657     VM& vm = exec->vm();
    1658     NativeCallFrameTracer tracer(&vm, exec);
    1659    
    1660     Butterfly* butterfly = Butterfly::create(
    1661         vm, nullptr, 0, structure->outOfLineCapacity(), false, IndexingHeader(), 0);
     1657JSCell* JIT_OPERATION operationNewObjectWithButterfly(ExecState* exec, Structure* structure, Butterfly* butterfly)
     1658{
     1659    VM& vm = exec->vm();
     1660    NativeCallFrameTracer tracer(&vm, exec);
     1661   
     1662    if (!butterfly) {
     1663        butterfly = Butterfly::create(
     1664            vm, nullptr, 0, structure->outOfLineCapacity(), false, IndexingHeader(), 0);
     1665    }
    16621666   
    16631667    JSObject* result = JSObject::createRawObject(exec, structure, butterfly);
    … …  
    16661670}
    16671671
    1668 JSCell* JIT_OPERATION operationNewObjectWithButterflyWithIndexingHeaderAndVectorLength(ExecState* exec, Structure* structure, unsigned length)
     1672JSCell* JIT_OPERATION operationNewObjectWithButterflyWithIndexingHeaderAndVectorLength(ExecState* exec, Structure* structure, unsigned length, Butterfly* butterfly)
    16691673{
    16701674    VM& vm = exec->vm();
    … …  
    16741678    header.setVectorLength(length);
    16751679    header.setPublicLength(0);
    1676     Butterfly* butterfly = Butterfly::create(
    1677         vm, nullptr, 0, structure->outOfLineCapacity(), true, header,
    1678         sizeof(EncodedJSValue) * length);
    1679 
     1680    if (butterfly)
     1681        *butterfly->indexingHeader() = header;
     1682    else {
     1683        butterfly = Butterfly::create(
     1684            vm, nullptr, 0, structure->outOfLineCapacity(), true, header,
     1685            sizeof(EncodedJSValue) * length);
     1686    }
     1687   
    16801688    // Paradoxically this may allocate a JSArray. That's totally cool.
    16811689    JSObject* result = JSObject::createRawObject(exec, structure, butterfly);

  • trunk/Source/JavaScriptCore/dfg/DFGOperations.h

    r205112 r205462  
    7171char* JIT_OPERATION operationNewArrayBuffer(ExecState*, Structure*, size_t, size_t) WTF_INTERNAL;
    7272char* JIT_OPERATION operationNewEmptyArray(ExecState*, Structure*) WTF_INTERNAL;
    73 char* JIT_OPERATION operationNewArrayWithSize(ExecState*, Structure*, int32_t) WTF_INTERNAL;
     73char* JIT_OPERATION operationNewArrayWithSize(ExecState*, Structure*, int32_t, Butterfly*) WTF_INTERNAL;
    7474char* JIT_OPERATION operationNewInt8ArrayWithSize(ExecState*, Structure*, int32_t) WTF_INTERNAL;
    7575char* JIT_OPERATION operationNewInt8ArrayWithOneArgument(ExecState*, Structure*, EncodedJSValue) WTF_INTERNAL;
    … …  
    177177size_t JIT_OPERATION operationDefaultHasInstance(ExecState*, JSCell* value, JSCell* proto);
    178178
    179 char* JIT_OPERATION operationNewRawObject(ExecState*, Structure*, int32_t) WTF_INTERNAL;
    180 JSCell* JIT_OPERATION operationNewObjectWithButterfly(ExecState*, Structure*) WTF_INTERNAL;
    181 JSCell* JIT_OPERATION operationNewObjectWithButterflyWithIndexingHeaderAndVectorLength(ExecState*, Structure*, unsigned length) WTF_INTERNAL;
     179char* JIT_OPERATION operationNewRawObject(ExecState*, Structure*, int32_t, Butterfly*) WTF_INTERNAL;
     180JSCell* JIT_OPERATION operationNewObjectWithButterfly(ExecState*, Structure*, Butterfly*) WTF_INTERNAL;
     181JSCell* JIT_OPERATION operationNewObjectWithButterflyWithIndexingHeaderAndVectorLength(ExecState*, Structure*, unsigned length, Butterfly*) WTF_INTERNAL;
    182182
    183183void JIT_OPERATION operationProcessTypeProfilerLogDFG(ExecState*) WTF_INTERNAL;

  • trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp

    r205364 r205462  
    9595
    9696    ASSERT(vectorLength >= numElements);
    97     vectorLength = std::max(BASE_VECTOR_LEN, vectorLength);
     97    vectorLength = Butterfly::optimalContiguousVectorLength(structure, vectorLength);
    9898   
    9999    JITCompiler::JumpList slowCases;
    … …  
    104104    size += outOfLineCapacity * sizeof(JSValue);
    105105
     106    m_jit.move(TrustedImmPtr(0), storageGPR);
     107   
    106108    if (size) {
    107         slowCases.append(
    108             emitAllocateBasicStorage(TrustedImm32(size), storageGPR));
    109         if (hasIndexingHeader)
    110             m_jit.subPtr(TrustedImm32(vectorLength * sizeof(JSValue)), storageGPR);
    111         else
    112             m_jit.addPtr(TrustedImm32(sizeof(IndexingHeader)), storageGPR);
     109        if (MarkedAllocator* allocator = m_jit.vm()->heap.allocatorForAuxiliaryData(size)) {
     110            m_jit.move(TrustedImmPtr(allocator), scratchGPR);
     111            m_jit.emitAllocate(storageGPR, allocator, scratchGPR, scratch2GPR, slowCases);
     112           
     113            m_jit.addPtr(
     114                TrustedImm32(outOfLineCapacity * sizeof(JSValue) + sizeof(IndexingHeader)),
     115                storageGPR);
     116           
     117            if (hasIndexingHeader)
     118                m_jit.store32(TrustedImm32(vectorLength), MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
     119        } else
     120            slowCases.append(m_jit.jump());
     121    }
     122
     123    size_t allocationSize = JSFinalObject::allocationSize(inlineCapacity);
     124    MarkedAllocator* allocatorPtr = m_jit.vm()->heap.allocatorForObjectWithoutDestructor(allocationSize);
     125    if (allocatorPtr) {
     126        m_jit.move(TrustedImmPtr(allocatorPtr), scratchGPR);
     127        emitAllocateJSObject(resultGPR, allocatorPtr, scratchGPR, TrustedImmPtr(structure), storageGPR, scratch2GPR, slowCases);
    113128    } else
    114         m_jit.move(TrustedImmPtr(0), storageGPR);
    115 
    116     size_t allocationSize = JSFinalObject::allocationSize(inlineCapacity);
    117     MarkedAllocator* allocatorPtr = &m_jit.vm()->heap.allocatorForObjectWithoutDestructor(allocationSize);
    118     m_jit.move(TrustedImmPtr(allocatorPtr), scratchGPR);
    119     emitAllocateJSObject(resultGPR, scratchGPR, TrustedImmPtr(structure), storageGPR, scratch2GPR, slowCases);
    120 
    121     if (hasIndexingHeader)
    122         m_jit.store32(TrustedImm32(vectorLength), MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
     129        slowCases.append(m_jit.jump());
    123130
    124131    // I want a slow path that also loads out the storage pointer, and that's
    … …  
    129136        structure, vectorLength));
    130137
    131     if (hasDouble(structure->indexingType()) && numElements < vectorLength) {
     138    if (numElements < vectorLength) {
    132139#if USE(JSVALUE64)
    133         m_jit.move(TrustedImm64(bitwise_cast<int64_t>(PNaN)), scratchGPR);
     140        if (hasDouble(structure->indexingType()))
     141            m_jit.move(TrustedImm64(bitwise_cast<int64_t>(PNaN)), scratchGPR);
     142        else
     143            m_jit.move(TrustedImm64(JSValue::encode(JSValue())), scratchGPR);
    134144        for (unsigned i = numElements; i < vectorLength; ++i)
    135145            m_jit.store64(scratchGPR, MacroAssembler::Address(storageGPR, sizeof(double) * i));
    136146#else
    137147        EncodedValueDescriptor value;
    138         value.asInt64 = JSValue::encode(JSValue(JSValue::EncodeAsDouble, PNaN));
     148        if (hasDouble(structure->indexingType()))
     149            value.asInt64 = JSValue::encode(JSValue(JSValue::EncodeAsDouble, PNaN));
     150        else
     151            value.asInt64 = JSValue::encode(JSValue());
    139152        for (unsigned i = numElements; i < vectorLength; ++i) {
    140153            m_jit.store32(TrustedImm32(value.asBits.tag), MacroAssembler::Address(storageGPR, sizeof(double) * i + OBJECT_OFFSETOF(JSValue, u.asBits.tag)));
    … …  
    38243837   
    38253838    JITCompiler::JumpList slowPath;
    3826     MarkedAllocator& markedAllocator = m_jit.vm()->heap.allocatorForObjectWithDestructor(sizeof(JSRopeString));
    3827     m_jit.move(TrustedImmPtr(&markedAllocator), allocatorGPR);
    3828     emitAllocateJSCell(resultGPR, allocatorGPR, TrustedImmPtr(m_jit.vm()->stringStructure.get()), scratchGPR, slowPath);
     3839    MarkedAllocator* markedAllocator = m_jit.vm()->heap.allocatorForObjectWithDestructor(sizeof(JSRopeString));
     3840    RELEASE_ASSERT(markedAllocator);
     3841    m_jit.move(TrustedImmPtr(markedAllocator), allocatorGPR);
     3842    emitAllocateJSCell(resultGPR, markedAllocator, allocatorGPR, TrustedImmPtr(m_jit.vm()->stringStructure.get()), scratchGPR, slowPath);
    38293843       
    38303844    m_jit.storePtr(TrustedImmPtr(0), JITCompiler::Address(resultGPR, JSString::offsetOfValue()));
    … …  
    69096923void SpeculativeJIT::compileAllocatePropertyStorage(Node* node)
    69106924{
    6911     if (node->transition()->previous->couldHaveIndexingHeader()) {
     6925    ASSERT(!node->transition()->previous->outOfLineCapacity());
     6926    ASSERT(initialOutOfLineCapacity == node->transition()->next->outOfLineCapacity());
     6927   
     6928    size_t size = initialOutOfLineCapacity * sizeof(JSValue);
     6929
     6930    MarkedAllocator* allocator = m_jit.vm()->heap.allocatorForAuxiliaryData(size);
     6931
     6932    if (!allocator || node->transition()->previous->couldHaveIndexingHeader()) {
    69126933        SpeculateCellOperand base(this, node->child1());
    69136934       
    … …  
    69266947    SpeculateCellOperand base(this, node->child1());
    69276948    GPRTemporary scratch1(this);
     6949    GPRTemporary scratch2(this);
     6950    GPRTemporary scratch3(this);
    69286951       
    69296952    GPRReg baseGPR = base.gpr();
    69306953    GPRReg scratchGPR1 = scratch1.gpr();
    6931        
    6932     ASSERT(!node->transition()->previous->outOfLineCapacity());
    6933     ASSERT(initialOutOfLineCapacity == node->transition()->next->outOfLineCapacity());
    6934    
    6935     JITCompiler::Jump slowPath =
    6936         emitAllocateBasicStorage(
    6937             TrustedImm32(initialOutOfLineCapacity * sizeof(JSValue)), scratchGPR1);
    6938 
    6939     m_jit.addPtr(JITCompiler::TrustedImm32(sizeof(IndexingHeader)), scratchGPR1);
     6954    GPRReg scratchGPR2 = scratch2.gpr();
     6955    GPRReg scratchGPR3 = scratch3.gpr();
     6956       
     6957    m_jit.move(JITCompiler::TrustedImmPtr(allocator), scratchGPR2);
     6958    JITCompiler::JumpList slowPath;
     6959    m_jit.emitAllocate(scratchGPR1, allocator, scratchGPR2, scratchGPR3, slowPath);
     6960    m_jit.addPtr(JITCompiler::TrustedImm32(size + sizeof(IndexingHeader)), scratchGPR1);
    69406961       
    69416962    addSlowPathGenerator(
    … …  
    69526973    size_t newSize = oldSize * outOfLineGrowthFactor;
    69536974    ASSERT(newSize == node->transition()->next->outOfLineCapacity() * sizeof(JSValue));
    6954 
    6955     if (node->transition()->previous->couldHaveIndexingHeader()) {
     6975   
     6976    MarkedAllocator* allocator = m_jit.vm()->heap.allocatorForAuxiliaryData(newSize);
     6977
     6978    if (!allocator || node->transition()->previous->couldHaveIndexingHeader()) {
    69566979        SpeculateCellOperand base(this, node->child1());
    69576980       
    … …  
    69726995    GPRTemporary scratch1(this);
    69736996    GPRTemporary scratch2(this);
     6997    GPRTemporary scratch3(this);
    69746998       
    69756999    GPRReg baseGPR = base.gpr();
    … …  
    69777001    GPRReg scratchGPR1 = scratch1.gpr();
    69787002    GPRReg scratchGPR2 = scratch2.gpr();
    6979        
    6980     JITCompiler::Jump slowPath =
    6981         emitAllocateBasicStorage(TrustedImm32(newSize), scratchGPR1);
    6982 
    6983     m_jit.addPtr(JITCompiler::TrustedImm32(sizeof(IndexingHeader)), scratchGPR1);
     7003    GPRReg scratchGPR3 = scratch3.gpr();
     7004   
     7005    JITCompiler::JumpList slowPath;
     7006    m_jit.move(JITCompiler::TrustedImmPtr(allocator), scratchGPR2);
     7007    m_jit.emitAllocate(scratchGPR1, allocator, scratchGPR2, scratchGPR3, slowPath);
     7008   
     7009    m_jit.addPtr(JITCompiler::TrustedImm32(newSize + sizeof(IndexingHeader)), scratchGPR1);
    69847010       
    69857011    addSlowPathGenerator(

  • trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h

    r205112 r205462  
    10021002    {
    10031003        m_jit.setupArgumentsWithExecState(arg1, arg2);
     1004        return appendCallSetResult(operation, result);
     1005    }
     1006    JITCompiler::Call callOperation(P_JITOperation_EStZB operation, GPRReg result, Structure* structure, GPRReg arg2, GPRReg butterfly)
     1007    {
     1008        m_jit.setupArgumentsWithExecState(TrustedImmPtr(structure), arg2, butterfly);
     1009        return appendCallSetResult(operation, result);
     1010    }
     1011    JITCompiler::Call callOperation(P_JITOperation_EStZB operation, GPRReg result, Structure* structure, size_t arg2, GPRReg butterfly)
     1012    {
     1013        m_jit.setupArgumentsWithExecState(TrustedImmPtr(structure), TrustedImm32(arg2), butterfly);
     1014        return appendCallSetResult(operation, result);
     1015    }
     1016    JITCompiler::Call callOperation(P_JITOperation_EStZB operation, GPRReg result, GPRReg arg1, GPRReg arg2, GPRReg butterfly)
     1017    {
     1018        m_jit.setupArgumentsWithExecState(arg1, arg2, butterfly);
     1019        return appendCallSetResult(operation, result);
     1020    }
     1021    JITCompiler::Call callOperation(P_JITOperation_EStZB operation, GPRReg result, GPRReg arg1, GPRReg arg2, Butterfly* butterfly)
     1022    {
     1023        m_jit.setupArgumentsWithExecState(arg1, arg2, TrustedImmPtr(butterfly));
    10041024        return appendCallSetResult(operation, result);
    10051025    }
    … …  
    25582578    // Allocator for a cell of a specific size.
    25592579    template <typename StructureType> // StructureType can be GPR or ImmPtr.
    2560     void emitAllocateJSCell(GPRReg resultGPR, GPRReg allocatorGPR, StructureType structure,
     2580    void emitAllocateJSCell(
     2581        GPRReg resultGPR, MarkedAllocator* allocator, GPRReg allocatorGPR, StructureType structure,
    25612582        GPRReg scratchGPR, MacroAssembler::JumpList& slowPath)
    25622583    {
    2563         m_jit.emitAllocateJSCell(resultGPR, allocatorGPR, structure, scratchGPR, slowPath);
     2584        m_jit.emitAllocateJSCell(resultGPR, allocator, allocatorGPR, structure, scratchGPR, slowPath);
    25642585    }
    25652586
    25662587    // Allocator for an object of a specific size.
    25672588    template <typename StructureType, typename StorageType> // StructureType and StorageType can be GPR or ImmPtr.
    2568     void emitAllocateJSObject(GPRReg resultGPR, GPRReg allocatorGPR, StructureType structure,
     2589    void emitAllocateJSObject(
     2590        GPRReg resultGPR, MarkedAllocator* allocator, GPRReg allocatorGPR, StructureType structure,
    25692591        StorageType storage, GPRReg scratchGPR, MacroAssembler::JumpList& slowPath)
    25702592    {
    2571         m_jit.emitAllocateJSObject(resultGPR, allocatorGPR, structure, storage, scratchGPR, slowPath);
     2593        m_jit.emitAllocateJSObject(
     2594            resultGPR, allocator, allocatorGPR, structure, storage, scratchGPR, slowPath);
    25722595    }
    25732596

  • trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp

    r205112 r205462  
    38673867        m_jit.move(TrustedImmPtr(globalObject->arrayStructureForIndexingTypeDuringAllocation(ArrayWithArrayStorage)), structureGPR);
    38683868        done.link(&m_jit);
    3869         callOperation(
    3870             operationNewArrayWithSize, resultGPR, structureGPR, sizeGPR);
     3869        callOperation(operationNewArrayWithSize, resultGPR, structureGPR, sizeGPR, nullptr);
    38713870        m_jit.exceptionCheck();
    38723871        cellResult(resultGPR, node);
    … …  
    40344033        m_jit.loadPtr(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfStructure()), structureGPR);
    40354034        slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, allocatorGPR));
    4036         emitAllocateJSObject(resultGPR, allocatorGPR, structureGPR, TrustedImmPtr(0), scratchGPR, slowPath);
     4035        emitAllocateJSObject(resultGPR, nullptr, allocatorGPR, structureGPR, TrustedImmPtr(0), scratchGPR, slowPath);
    40374036
    40384037        addSlowPathGenerator(slowPathCall(slowPath, this, operationCreateThis, resultGPR, calleeGPR, node->inlineCapacity()));
    … …  
    40554054        Structure* structure = node->structure();
    40564055        size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
    4057         MarkedAllocator* allocatorPtr = &m_jit.vm()->heap.allocatorForObjectWithoutDestructor(allocationSize);
     4056        MarkedAllocator* allocatorPtr = m_jit.vm()->heap.allocatorForObjectWithoutDestructor(allocationSize);
    40584057
    40594058        m_jit.move(TrustedImmPtr(allocatorPtr), allocatorGPR);
    4060         emitAllocateJSObject(resultGPR, allocatorGPR, TrustedImmPtr(structure), TrustedImmPtr(0), scratchGPR, slowPath);
     4059        emitAllocateJSObject(resultGPR, allocatorPtr, allocatorGPR, TrustedImmPtr(structure), TrustedImmPtr(0), scratchGPR, slowPath);
    40614060
    40624061        addSlowPathGenerator(slowPathCall(slowPath, this, operationNewObject, resultGPR, structure));
    … …  
    53655364    GPRReg scratch2GPR = scratch2.gpr();
    53665365   
     5366    m_jit.move(TrustedImmPtr(0), storageGPR);
     5367           
    53675368    MacroAssembler::JumpList slowCases;
    53685369    if (shouldConvertLargeSizeToArrayStorage)
    53695370        slowCases.append(m_jit.branch32(MacroAssembler::AboveOrEqual, sizeGPR, TrustedImm32(MIN_ARRAY_STORAGE_CONSTRUCTION_LENGTH)));
    5370    
     5371           
    53715372    ASSERT((1 << 3) == sizeof(JSValue));
    53725373    m_jit.move(sizeGPR, scratchGPR);
    53735374    m_jit.lshift32(TrustedImm32(3), scratchGPR);
    53745375    m_jit.add32(TrustedImm32(sizeof(IndexingHeader)), scratchGPR, resultGPR);
    5375     slowCases.append(
    5376         emitAllocateBasicStorage(resultGPR, storageGPR));
    5377     m_jit.subPtr(scratchGPR, storageGPR);
     5376    m_jit.emitAllocateVariableSized(
     5377        storageGPR, m_jit.vm()->heap.subspaceForAuxiliaryData(), resultGPR, scratchGPR,
     5378        scratch2GPR, slowCases);
     5379    m_jit.addPtr(TrustedImm32(sizeof(IndexingHeader)), storageGPR);
     5380
     5381    m_jit.store32(sizeGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
     5382    m_jit.store32(sizeGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
     5383           
     5384    JSValue hole;
     5385    if (hasDouble(indexingType))
     5386        hole = JSValue(JSValue::EncodeAsDouble, PNaN);
     5387    else
     5388        hole = JSValue();
     5389           
     5390    m_jit.move(sizeGPR, scratchGPR);
     5391    MacroAssembler::Jump done = m_jit.branchTest32(MacroAssembler::Zero, scratchGPR);
     5392    MacroAssembler::Label loop = m_jit.label();
     5393    m_jit.sub32(TrustedImm32(1), scratchGPR);
     5394    m_jit.store32(TrustedImm32(hole.u.asBits.tag), MacroAssembler::BaseIndex(storageGPR, scratchGPR, MacroAssembler::TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.tag)));
     5395    m_jit.store32(TrustedImm32(hole.u.asBits.payload), MacroAssembler::BaseIndex(storageGPR, scratchGPR, MacroAssembler::TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.payload)));
     5396    m_jit.branchTest32(MacroAssembler::NonZero, scratchGPR).linkTo(loop, &m_jit);
     5397    done.link(&m_jit);
     5398   
    53785399    Structure* structure = globalObject->arrayStructureForIndexingTypeDuringAllocation(indexingType);
    53795400    emitAllocateJSObject<JSArray>(resultGPR, TrustedImmPtr(structure), storageGPR, scratchGPR, scratch2GPR, slowCases);
    5380    
    5381     m_jit.store32(sizeGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
    5382     m_jit.store32(sizeGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
    5383    
    5384     if (hasDouble(indexingType)) {
    5385         JSValue nan = JSValue(JSValue::EncodeAsDouble, PNaN);
    5386        
    5387         m_jit.move(sizeGPR, scratchGPR);
    5388         MacroAssembler::Jump done = m_jit.branchTest32(MacroAssembler::Zero, scratchGPR);
    5389         MacroAssembler::Label loop = m_jit.label();
    5390         m_jit.sub32(TrustedImm32(1), scratchGPR);
    5391         m_jit.store32(TrustedImm32(nan.u.asBits.tag), MacroAssembler::BaseIndex(storageGPR, scratchGPR, MacroAssembler::TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.tag)));
    5392         m_jit.store32(TrustedImm32(nan.u.asBits.payload), MacroAssembler::BaseIndex(storageGPR, scratchGPR, MacroAssembler::TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.payload)));
    5393         m_jit.branchTest32(MacroAssembler::NonZero, scratchGPR).linkTo(loop, &m_jit);
    5394         done.link(&m_jit);
    5395     }
    5396    
     5401           
    53975402    addSlowPathGenerator(std::make_unique<CallArrayAllocatorWithVariableSizeSlowPathGenerator>(
    53985403        slowCases, this, operationNewArrayWithSize, resultGPR,
    53995404        structure,
    54005405        shouldConvertLargeSizeToArrayStorage ? globalObject->arrayStructureForIndexingTypeDuringAllocation(ArrayWithArrayStorage) : structure,
    5401         sizeGPR));
     5406        sizeGPR, storageGPR));
    54025407}
    54035408

  • trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp

    r205112 r205462  
    38193819        m_jit.move(TrustedImmPtr(globalObject->arrayStructureForIndexingTypeDuringAllocation(ArrayWithArrayStorage)), structureGPR);
    38203820        done.link(&m_jit);
    3821         callOperation(operationNewArrayWithSize, resultGPR, structureGPR, sizeGPR);
     3821        callOperation(operationNewArrayWithSize, resultGPR, structureGPR, sizeGPR, nullptr);
    38223822        m_jit.exceptionCheck();
    38233823        cellResult(resultGPR, node);
    … …  
    39763976        m_jit.loadPtr(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfStructure()), structureGPR);
    39773977        slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, allocatorGPR));
    3978         emitAllocateJSObject(resultGPR, allocatorGPR, structureGPR, TrustedImmPtr(0), scratchGPR, slowPath);
     3978        emitAllocateJSObject(resultGPR, nullptr, allocatorGPR, structureGPR, TrustedImmPtr(0), scratchGPR, slowPath);
    39793979
    39803980        addSlowPathGenerator(slowPathCall(slowPath, this, operationCreateThis, resultGPR, calleeGPR, node->inlineCapacity()));
    … …  
    39973997        Structure* structure = node->structure();
    39983998        size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
    3999         MarkedAllocator* allocatorPtr = &m_jit.vm()->heap.allocatorForObjectWithoutDestructor(allocationSize);
     3999        MarkedAllocator* allocatorPtr = m_jit.vm()->heap.allocatorForObjectWithoutDestructor(allocationSize);
    40004000
    40014001        m_jit.move(TrustedImmPtr(allocatorPtr), allocatorGPR);
    4002         emitAllocateJSObject(resultGPR, allocatorGPR, TrustedImmPtr(structure), TrustedImmPtr(0), scratchGPR, slowPath);
     4002        emitAllocateJSObject(resultGPR, allocatorPtr, allocatorGPR, TrustedImmPtr(structure), TrustedImmPtr(0), scratchGPR, slowPath);
    40034003
    40044004        addSlowPathGenerator(slowPathCall(slowPath, this, operationNewObject, resultGPR, structure));
    … …  
    52375237        unsigned bytecodeIndex = node->origin.semantic.bytecodeIndex;
    52385238        auto triggerIterator = m_jit.jitCode()->tierUpEntryTriggers.find(bytecodeIndex);
    5239         RELEASE_ASSERT(triggerIterator != m_jit.jitCode()->tierUpEntryTriggers.end());
     5239        DFG_ASSERT(m_jit.graph(), node, triggerIterator != m_jit.jitCode()->tierUpEntryTriggers.end());
    52405240        uint8_t* forceEntryTrigger = &(m_jit.jitCode()->tierUpEntryTriggers.find(bytecodeIndex)->value);
    52415241
    … …  
    54215421    GPRReg scratch2GPR = scratch2.gpr();
    54225422   
     5423    m_jit.move(TrustedImmPtr(0), storageGPR);
     5424   
    54235425    MacroAssembler::JumpList slowCases;
    54245426    if (shouldConvertLargeSizeToArrayStorage)
    54255427        slowCases.append(m_jit.branch32(MacroAssembler::AboveOrEqual, sizeGPR, TrustedImm32(MIN_ARRAY_STORAGE_CONSTRUCTION_LENGTH)));
    5426    
     5428           
    54275429    ASSERT((1 << 3) == sizeof(JSValue));
    54285430    m_jit.move(sizeGPR, scratchGPR);
    54295431    m_jit.lshift32(TrustedImm32(3), scratchGPR);
    54305432    m_jit.add32(TrustedImm32(sizeof(IndexingHeader)), scratchGPR, resultGPR);
    5431     slowCases.append(
    5432         emitAllocateBasicStorage(resultGPR, storageGPR));
    5433     m_jit.subPtr(scratchGPR, storageGPR);
    5434     Structure* structure = globalObject->arrayStructureForIndexingTypeDuringAllocation(indexingType);
    5435     emitAllocateJSObject<JSArray>(resultGPR, TrustedImmPtr(structure), storageGPR, scratchGPR, scratch2GPR, slowCases);
    5436    
     5433    m_jit.emitAllocateVariableSized(
     5434        storageGPR, m_jit.vm()->heap.subspaceForAuxiliaryData(), resultGPR, scratchGPR,
     5435        scratch2GPR, slowCases);
     5436    m_jit.addPtr(TrustedImm32(sizeof(IndexingHeader)), storageGPR);
     5437
    54375438    m_jit.store32(sizeGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
    54385439    m_jit.store32(sizeGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
    5439    
    5440     if (hasDouble(indexingType)) {
     5440           
     5441    if (hasDouble(indexingType))
    54415442        m_jit.move(TrustedImm64(bitwise_cast<int64_t>(PNaN)), scratchGPR);
    5442         m_jit.move(sizeGPR, scratch2GPR);
    5443         MacroAssembler::Jump done = m_jit.branchTest32(MacroAssembler::Zero, scratch2GPR);
    5444         MacroAssembler::Label loop = m_jit.label();
    5445         m_jit.sub32(TrustedImm32(1), scratch2GPR);
    5446         m_jit.store64(scratchGPR, MacroAssembler::BaseIndex(storageGPR, scratch2GPR, MacroAssembler::TimesEight));
    5447         m_jit.branchTest32(MacroAssembler::NonZero, scratch2GPR).linkTo(loop, &m_jit);
    5448         done.link(&m_jit);
    5449     }
     5443    else
     5444        m_jit.move(TrustedImm64(JSValue::encode(JSValue())), scratchGPR);
     5445    m_jit.move(sizeGPR, scratch2GPR);
     5446    MacroAssembler::Jump done = m_jit.branchTest32(MacroAssembler::Zero, scratch2GPR);
     5447    MacroAssembler::Label loop = m_jit.label();
     5448    m_jit.sub32(TrustedImm32(1), scratch2GPR);
     5449    m_jit.store64(scratchGPR, MacroAssembler::BaseIndex(storageGPR, scratch2GPR, MacroAssembler::TimesEight));
     5450    m_jit.branchTest32(MacroAssembler::NonZero, scratch2GPR).linkTo(loop, &m_jit);
     5451    done.link(&m_jit);
     5452           
     5453    Structure* structure = globalObject->arrayStructureForIndexingTypeDuringAllocation(indexingType);
     5454           
     5455    emitAllocateJSObject<JSArray>(resultGPR, TrustedImmPtr(structure), storageGPR, scratchGPR, scratch2GPR, slowCases);
    54505456   
    54515457    addSlowPathGenerator(std::make_unique<CallArrayAllocatorWithVariableSizeSlowPathGenerator>(
    … …  
    54535459        structure,
    54545460        shouldConvertLargeSizeToArrayStorage ? globalObject->arrayStructureForIndexingTypeDuringAllocation(ArrayWithArrayStorage) : structure,
    5455         sizeGPR));
     5461        sizeGPR, storageGPR));
    54565462}
    54575463

  • trunk/Source/JavaScriptCore/dfg/DFGStrengthReductionPhase.cpp

    r204958 r205462  
    4141#include "StringPrototype.h"
    4242#include <cstdlib>
     43#include <wtf/text/StringBuilder.h>
    4344
    4445namespace JSC { namespace DFG {
    … …  
    421422                break;
    422423            }
    423 
     424           
    424425            unsigned lastIndex;
    425426            if (regExp->globalOrSticky()) {
    … …  
    469470
    470471            MatchResult result;
    471             Vector<int, 32> ovector;
     472            Vector<int> ovector;
    472473            // We have to call the kind of match function that the main thread would have called.
    473474            // Otherwise, we might not have the desired Yarr code compiled, and the match will fail.
    … …  
    515516
    516517                    unsigned publicLength = resultArray.size();
    517                     unsigned vectorLength = std::max(BASE_VECTOR_LEN, publicLength);
     518                    unsigned vectorLength =
     519                        Butterfly::optimalContiguousVectorLength(structure, publicLength);
    518520
    519521                    UniquedStringImpl* indexUID = vm().propertyNames->index.impl();
    … …  
    650652            do {
    651653                MatchResult result;
    652                 Vector<int, 32> ovector;
     654                Vector<int> ovector;
    653655                // Model which version of match() is called by the main thread.
    654656                if (replace.isEmpty() && regExp->global()) {

  • trunk/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h

    r204912 r205462  
    7777    macro(JSSymbolTableObject_symbolTable, JSSymbolTableObject::offsetOfSymbolTable()) \
    7878    macro(JSWrapperObject_internalValue, JSWrapperObject::internalValueOffset()) \
    79     macro(MarkedAllocator_freeListHead, MarkedAllocator::offsetOfFreeListHead()) \
    8079    macro(RegExpConstructor_cachedResult_lastRegExp, RegExpConstructor::offsetOfCachedResult() + RegExpCachedResult::offsetOfLastRegExp()) \
    8180    macro(RegExpConstructor_cachedResult_lastInput, RegExpConstructor::offsetOfCachedResult() + RegExpCachedResult::offsetOfLastInput()) \
    … …  
    110109    macro(JSPropertyNameEnumerator_cachedPropertyNamesVectorContents, 0, sizeof(WriteBarrier<JSString>)) \
    111110    macro(JSRopeString_fibers, JSRopeString::offsetOfFibers(), sizeof(WriteBarrier<JSString>)) \
    112     macro(MarkedSpace_Subspace_impreciseAllocators, OBJECT_OFFSETOF(MarkedSpace::Subspace, impreciseAllocators), sizeof(MarkedAllocator)) \
    113     macro(MarkedSpace_Subspace_preciseAllocators, OBJECT_OFFSETOF(MarkedSpace::Subspace, preciseAllocators), sizeof(MarkedAllocator)) \
     111    macro(MarkedSpace_Subspace_allocatorForSizeStep, OBJECT_OFFSETOF(MarkedSpace::Subspace, allocatorForSizeStep), sizeof(MarkedAllocator*)) \
    114112    macro(ScopedArguments_overflowStorage, ScopedArguments::overflowStorageOffset(), sizeof(EncodedJSValue)) \
    115113    macro(WriteBarrierBuffer_bufferContents, 0, sizeof(JSCell*)) \

  • trunk/Source/JavaScriptCore/ftl/FTLCompile.cpp

    r204912 r205462  
    4343#include "FTLThunks.h"
    4444#include "JITSubGenerator.h"
     45#include "JSCInlines.h"
    4546#include "LinkBuffer.h"
    4647#include "PCToCodeOriginMap.h"

  • trunk/Source/JavaScriptCore/ftl/FTLJITFinalizer.cpp

    r204912 r205462  
    3333#include "FTLState.h"
    3434#include "FTLThunks.h"
     35#include "JSCInlines.h"
    3536#include "ProfilerDatabase.h"
    3637

  • trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp

    r205380 r205462  
    38163816           
    38173817            fastObject = allocateVariableSizedObject<DirectArguments>(
    3818                 size, structure, m_out.intPtrZero, slowPath);
     3818                m_out.zeroExtPtr(size), structure, m_out.intPtrZero, slowPath);
    38193819        }
    38203820       
    … …  
    39163916            LValue arrayLength = lowInt32(m_node->child1());
    39173917            LBasicBlock loopStart = m_out.newBlock();
    3918             bool shouldLargeArraySizeCreateArrayStorage = false;
    3919             LValue array = compileAllocateArrayWithSize(arrayLength, ArrayWithContiguous, shouldLargeArraySizeCreateArrayStorage);
    3920 
    3921             LValue butterfly = m_out.loadPtr(array, m_heaps.JSObject_butterfly);
     3918            JSGlobalObject* globalObject = m_graph.globalObjectFor(m_node->origin.semantic);
     3919            Structure* structure = globalObject->arrayStructureForIndexingTypeDuringAllocation(ArrayWithContiguous);
     3920            ArrayValues arrayValues = allocateUninitializedContiguousJSArray(arrayLength, structure);
     3921            LValue array = arrayValues.array;
     3922            LValue butterfly = arrayValues.butterfly;
    39223923            ValueFromBlock startLength = m_out.anchor(arrayLength);
    39233924            LValue argumentRegion = m_out.add(getArgumentsStart(), m_out.constInt64(sizeof(Register) * m_node->numberOfArgumentsToSkip()));
    … …  
    39313932            LValue loadedValue = m_out.load64(m_out.baseIndex(m_heaps.variables, argumentRegion, m_out.zeroExtPtr(currentOffset)));
    39323933            IndexedAbstractHeap& heap = m_heaps.indexedContiguousProperties;
    3933             m_out.store(loadedValue, m_out.baseIndex(heap, butterfly, m_out.zeroExtPtr(currentOffset)), Output::Store64);
     3934            m_out.store64(loadedValue, m_out.baseIndex(heap, butterfly, m_out.zeroExtPtr(currentOffset)));
    39343935            m_out.branch(m_out.equal(currentOffset, m_out.constInt32(0)), unsure(continuation), unsure(loopStart));
    39353936
    … …  
    39883989            unsigned numElements = m_node->numChildren();
    39893990           
    3990             ArrayValues arrayValues = allocateJSArray(structure, numElements);
     3991            ArrayValues arrayValues =
     3992                allocateUninitializedContiguousJSArray(m_out.constInt32(numElements), structure);
    39913993           
    39923994            for (unsigned operandIndex = 0; operandIndex < m_node->numChildren(); ++operandIndex) {
    … …  
    40644066            unsigned numElements = m_node->numConstants();
    40654067           
    4066             ArrayValues arrayValues = allocateJSArray(structure, numElements);
     4068            ArrayValues arrayValues =
     4069                allocateUninitializedContiguousJSArray(m_out.constInt32(numElements), structure);
    40674070           
    40684071            JSValue* data = codeBlock()->constantBuffer(m_node->startConstant());
    … …  
    40904093    }
    40914094
    4092     LValue compileAllocateArrayWithSize(LValue publicLength, IndexingType indexingType, bool shouldLargeArraySizeCreateArrayStorage = true)
    4093     {
    4094         JSGlobalObject* globalObject = m_graph.globalObjectFor(m_node->origin.semantic);
    4095         Structure* structure = globalObject->arrayStructureForIndexingTypeDuringAllocation(indexingType);
    4096         ASSERT(
    4097             hasUndecided(structure->indexingType())
    4098             || hasInt32(structure->indexingType())
    4099             || hasDouble(structure->indexingType())
    4100             || hasContiguous(structure->indexingType()));
    4101 
    4102         LBasicBlock fastCase = m_out.newBlock();
    4103         LBasicBlock largeCase = shouldLargeArraySizeCreateArrayStorage ? m_out.newBlock() : nullptr;
    4104         LBasicBlock failCase = m_out.newBlock();
    4105         LBasicBlock continuation = m_out.newBlock();
    4106         LBasicBlock lastNext = nullptr;
    4107         if (shouldLargeArraySizeCreateArrayStorage) {
    4108             m_out.branch(
    4109                 m_out.aboveOrEqual(publicLength, m_out.constInt32(MIN_ARRAY_STORAGE_CONSTRUCTION_LENGTH)),
    4110                 rarely(largeCase), usually(fastCase));
    4111             lastNext = m_out.appendTo(fastCase, largeCase);
    4112         }
    4113 
    4114        
    4115         // We don't round up to BASE_VECTOR_LEN for new Array(blah).
    4116         LValue vectorLength = publicLength;
    4117        
    4118         LValue payloadSize =
    4119             m_out.shl(m_out.zeroExt(vectorLength, pointerType()), m_out.constIntPtr(3));
    4120        
    4121         LValue butterflySize = m_out.add(
    4122             payloadSize, m_out.constIntPtr(sizeof(IndexingHeader)));
    4123        
    4124         LValue endOfStorage = allocateBasicStorageAndGetEnd(butterflySize, failCase);
    4125        
    4126         LValue butterfly = m_out.sub(endOfStorage, payloadSize);
    4127        
    4128         LValue object = allocateObject<JSArray>(structure, butterfly, failCase);
    4129        
    4130         m_out.store32(publicLength, butterfly, m_heaps.Butterfly_publicLength);
    4131         m_out.store32(vectorLength, butterfly, m_heaps.Butterfly_vectorLength);
    4132 
    4133         initializeArrayElements(indexingType, vectorLength, butterfly);
    4134        
    4135         ValueFromBlock fastResult = m_out.anchor(object);
    4136         m_out.jump(continuation);
    4137        
    4138         LValue structureValue;
    4139         if (shouldLargeArraySizeCreateArrayStorage) {
    4140             LBasicBlock slowCase = m_out.newBlock();
    4141 
    4142             m_out.appendTo(largeCase, failCase);
    4143             ValueFromBlock largeStructure = m_out.anchor(m_out.constIntPtr(
    4144                 globalObject->arrayStructureForIndexingTypeDuringAllocation(ArrayWithArrayStorage)));
    4145             m_out.jump(slowCase);
    4146 
    4147             m_out.appendTo(failCase, slowCase);
    4148             ValueFromBlock failStructure = m_out.anchor(m_out.constIntPtr(structure));
    4149             m_out.jump(slowCase);
    4150 
    4151             m_out.appendTo(slowCase, continuation);
    4152             structureValue = m_out.phi(
    4153                 pointerType(), largeStructure, failStructure);
    4154         } else {
    4155             ASSERT(!lastNext);
    4156             lastNext = m_out.appendTo(failCase, continuation);
    4157             structureValue = m_out.constIntPtr(structure);
    4158         }
    4159 
    4160         LValue slowResultValue = lazySlowPath(
    4161             [=] (const Vector<Location>& locations) -> RefPtr<LazySlowPath::Generator> {
    4162                 return createLazyCallGenerator(
    4163                     operationNewArrayWithSize, locations[0].directGPR(),
    4164                     locations[1].directGPR(), locations[2].directGPR());
    4165             },
    4166             structureValue, publicLength);
    4167         ValueFromBlock slowResult = m_out.anchor(slowResultValue);
    4168         m_out.jump(continuation);
    4169        
    4170         m_out.appendTo(continuation, lastNext);
    4171         return m_out.phi(pointerType(), fastResult, slowResult);
    4172     }
    4173    
    41744095    void compileNewArrayWithSize()
    41754096    {
    … …  
    41814102       
    41824103        if (!globalObject->isHavingABadTime() && !hasAnyArrayStorage(m_node->indexingType())) {
    4183             setJSValue(compileAllocateArrayWithSize(publicLength, m_node->indexingType()));
     4104            setJSValue(
     4105                allocateJSArray(
     4106                    publicLength,
     4107                    globalObject->arrayStructureForIndexingTypeDuringAllocation(
     4108                        m_node->indexingType())).array);
    41844109            return;
    41854110        }
    … …  
    41904115                globalObject->arrayStructureForIndexingTypeDuringAllocation(ArrayWithArrayStorage)),
    41914116            m_out.constIntPtr(structure));
    4192         setJSValue(vmCall(Int64, m_out.operation(operationNewArrayWithSize), m_callFrame, structureValue, publicLength));
     4117        setJSValue(vmCall(Int64, m_out.operation(operationNewArrayWithSize), m_callFrame, structureValue, publicLength, m_out.intPtrZero));
    41934118    }
    41944119
    … …  
    44504375        LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath);
    44514376       
    4452         MarkedAllocator& allocator =
     4377        MarkedAllocator* allocator =
    44534378            vm().heap.allocatorForObjectWithDestructor(sizeof(JSRopeString));
     4379        DFG_ASSERT(m_graph, m_node, allocator);
    44544380       
    44554381        LValue result = allocateCell(
    4456             m_out.constIntPtr(&allocator),
    4457             vm().stringStructure.get(),
    4458             slowPath);
     4382            m_out.constIntPtr(allocator), vm().stringStructure.get(), slowPath);
    44594383       
    44604384        m_out.storePtr(m_out.intPtrZero, result, m_heaps.JSString_value);
    … …  
    70056929            if (structure->outOfLineCapacity() || hasIndexedProperties(structure->indexingType())) {
    70066930                size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
    7007                 MarkedAllocator* allocator = &vm().heap.allocatorForObjectWithoutDestructor(allocationSize);
     6931                MarkedAllocator* cellAllocator = vm().heap.allocatorForObjectWithoutDestructor(allocationSize);
     6932                DFG_ASSERT(m_graph, m_node, cellAllocator);
    70086933
    70096934                bool hasIndexingHeader = hasIndexedProperties(structure->indexingType());
    … …  
    70306955                        m_out.mul(m_out.zeroExtPtr(vectorLength), m_out.intPtrEight);
    70316956                }
    7032 
     6957               
    70336958                LValue butterflySize = m_out.add(
    70346959                    m_out.constIntPtr(
    … …  
    70416966                LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath);
    70426967               
    7043                 LValue endOfStorage = allocateBasicStorageAndGetEnd(butterflySize, slowPath);
     6968                ValueFromBlock noButterfly = m_out.anchor(m_out.intPtrZero);
     6969               
     6970                LValue startOfStorage = allocateHeapCell(
     6971                    allocatorForSize(vm().heap.subspaceForAuxiliaryData(), butterflySize, slowPath),
     6972                    slowPath);
    70446973
    70456974                LValue fastButterflyValue = m_out.add(
    7046                     m_out.sub(endOfStorage, indexingPayloadSizeInBytes),
    7047                     m_out.constIntPtr(sizeof(IndexingHeader) - indexingHeaderSize));
     6975                    startOfStorage,
     6976                    m_out.constIntPtr(
     6977                        structure->outOfLineCapacity() * sizeof(JSValue) + sizeof(IndexingHeader)));
     6978               
     6979                ValueFromBlock haveButterfly = m_out.anchor(fastButterflyValue);
    70486980
    70496981                m_out.store32(vectorLength, fastButterflyValue, m_heaps.Butterfly_vectorLength);
    70506982               
    70516983                LValue fastObjectValue = allocateObject(
    7052                     m_out.constIntPtr(allocator), structure, fastButterflyValue, slowPath);
     6984                    m_out.constIntPtr(cellAllocator), structure, fastButterflyValue, slowPath);
    70536985
    70546986                ValueFromBlock fastObject = m_out.anchor(fastObjectValue);
    … …  
    70576989               
    70586990                m_out.appendTo(slowPath, continuation);
     6991               
     6992                LValue butterflyValue = m_out.phi(pointerType(), noButterfly, haveButterfly);
    70596993
    70606994                LValue slowObjectValue;
    … …  
    70656999                                operationNewObjectWithButterflyWithIndexingHeaderAndVectorLength,
    70667000                                locations[0].directGPR(), CCallHelpers::TrustedImmPtr(structure),
    7067                                 locations[1].directGPR());
     7001                                locations[1].directGPR(), locations[2].directGPR());
    70687002                        },
    7069                         vectorLength);
     7003                        vectorLength, butterflyValue);
    70707004                } else {
    70717005                    slowObjectValue = lazySlowPath(
    … …  
    70737007                            return createLazyCallGenerator(
    70747008                                operationNewObjectWithButterfly, locations[0].directGPR(),
    7075                                 CCallHelpers::TrustedImmPtr(structure));
    7076                         });
     7009                                CCallHelpers::TrustedImmPtr(structure), locations[1].directGPR());
     7010                        },
     7011                        butterflyValue);
    70777012                }
    70787013                ValueFromBlock slowObject = m_out.anchor(slowObjectValue);
    … …  
    70897024                m_out.store32(publicLength, butterfly, m_heaps.Butterfly_publicLength);
    70907025
    7091                 initializeArrayElements(structure->indexingType(), vectorLength, butterfly);
     7026                initializeArrayElements(structure->indexingType(), m_out.int32Zero, vectorLength, butterfly);
    70927027
    70937028                HashMap<int32_t, LValue, DefaultHash<int32_t>::Hash, WTF::UnsignedWithZeroKeyHashTraits<int32_t>> indexMap;
    … …  
    78347769    }
    78357770
    7836     void initializeArrayElements(IndexingType indexingType, LValue vectorLength, LValue butterfly)
    7837     {
    7838         if (!hasDouble(indexingType)) {
    7839             // The GC already initialized everything to JSValue() for us.
    7840             return;
     7771    void initializeArrayElements(IndexingType indexingType, LValue begin, LValue end, LValue butterfly)
     7772    {
     7773        if (hasUndecided(indexingType))
     7774            return;
     7775       
     7776        if (begin == end)
     7777            return;
     7778       
     7779        IndexedAbstractHeap* heap = m_heaps.forIndexingType(indexingType);
     7780        DFG_ASSERT(m_graph, m_node, heap);
     7781       
     7782        LValue hole;
     7783        if (hasDouble(indexingType))
     7784            hole = m_out.constInt64(bitwise_cast<int64_t>(PNaN));
     7785        else
     7786            hole = m_out.constInt64(JSValue::encode(JSValue()));
     7787       
     7788        const uint64_t unrollingLimit = 10;
     7789        if (begin->hasInt() && end->hasInt()) {
     7790            uint64_t beginConst = static_cast<uint64_t>(begin->asInt());
     7791            uint64_t endConst = static_cast<uint64_t>(end->asInt());
     7792           
     7793            if (endConst - beginConst <= unrollingLimit) {
     7794                for (uint64_t i = beginConst; i < endConst; ++i)
     7795                    m_out.store64(hole, butterfly, heap->at(i));
     7796                return;
     7797            }
    78417798        }
    78427799
    … …  
    78457802        LBasicBlock initDone = m_out.newBlock();
    78467803       
    7847         ValueFromBlock originalIndex = m_out.anchor(vectorLength);
     7804        ValueFromBlock originalIndex = m_out.anchor(end);
    78487805        ValueFromBlock originalPointer = m_out.anchor(butterfly);
    7849         m_out.branch(
    7850             m_out.notZero32(vectorLength), unsure(initLoop), unsure(initDone));
     7806        m_out.branch(m_out.notEqual(end, begin), unsure(initLoop), unsure(initDone));
    78517807       
    78527808        LBasicBlock initLastNext = m_out.appendTo(initLoop, initDone);
    … …  
    78547810        LValue pointer = m_out.phi(pointerType(), originalPointer);
    78557811       
    7856         m_out.store64(
    7857             m_out.constInt64(bitwise_cast<int64_t>(PNaN)),
    7858             TypedPointer(m_heaps.indexedDoubleProperties.atAnyIndex(), pointer));
     7812        m_out.store64(hole, TypedPointer(heap->atAnyIndex(), pointer));
    78597813       
    78607814        LValue nextIndex = m_out.sub(index, m_out.int32One);
    … …  
    78627816        m_out.addIncomingToPhi(pointer, m_out.anchor(m_out.add(pointer, m_out.intPtrEight)));
    78637817        m_out.branch(
    7864             m_out.notZero32(nextIndex), unsure(initLoop), unsure(initDone));
     7818            m_out.notEqual(nextIndex, begin), unsure(initLoop), unsure(initDone));
    78657819       
    78667820        m_out.appendTo(initDone, initLastNext);
    … …  
    79177871       
    79187872        LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath);
    7919        
    7920         LValue endOfStorage = allocateBasicStorageAndGetEnd(
    7921             m_out.constIntPtr(sizeInValues * sizeof(JSValue)), slowPath);
    7922        
     7873
     7874        size_t sizeInBytes = sizeInValues * sizeof(JSValue);
     7875        MarkedAllocator* allocator = vm().heap.allocatorForAuxiliaryData(sizeInBytes);
     7876        LValue startOfStorage = allocateHeapCell(m_out.constIntPtr(allocator), slowPath);
    79237877        ValueFromBlock fastButterfly = m_out.anchor(
    7924             m_out.add(m_out.constIntPtr(sizeof(IndexingHeader)), endOfStorage));
    7925        
     7878            m_out.add(m_out.constIntPtr(sizeInBytes + sizeof(IndexingHeader)), startOfStorage));
    79267879        m_out.jump(continuation);
    79277880       
    … …  
    84888441    }
    84898442
    8490     LValue allocateCell(LValue allocator, LBasicBlock slowPath)
    8491     {
    8492         LBasicBlock success = m_out.newBlock();
    8493    
    8494         LValue result;
    8495         LValue condition;
    8496         if (Options::forceGCSlowPaths()) {
    8497             result = m_out.intPtrZero;
    8498             condition = m_out.booleanFalse;
    8499         } else {
    8500             result = m_out.loadPtr(
    8501                 allocator, m_heaps.MarkedAllocator_freeListHead);
    8502             condition = m_out.notNull(result);
    8503         }
    8504         m_out.branch(condition, usually(success), rarely(slowPath));
    8505        
    8506         m_out.appendTo(success);
    8507        
    8508         m_out.storePtr(
    8509             m_out.loadPtr(result, m_heaps.JSCell_freeListNext),
    8510             allocator, m_heaps.MarkedAllocator_freeListHead);
    8511 
    8512         return result;
     8443    LValue allocateHeapCell(LValue allocator, LBasicBlock slowPath)
     8444    {
     8445        MarkedAllocator* actualAllocator = nullptr;
     8446        if (allocator->hasIntPtr())
     8447            actualAllocator = bitwise_cast<MarkedAllocator*>(allocator->asIntPtr());
     8448       
     8449        if (!actualAllocator) {
     8450            // This means that either we know that the allocator is null or we don't know what the
     8451            // allocator is. In either case, we need the null check.
     8452            LBasicBlock haveAllocator = m_out.newBlock();
     8453            LBasicBlock lastNext = m_out.insertNewBlocksBefore(haveAllocator);
     8454            m_out.branch(allocator, usually(haveAllocator), rarely(slowPath));
     8455            m_out.appendTo(haveAllocator, lastNext);
     8456        }
     8457       
     8458        LBasicBlock continuation = m_out.newBlock();
     8459       
     8460        LBasicBlock lastNext = m_out.insertNewBlocksBefore(continuation);
     8461       
     8462        PatchpointValue* patchpoint = m_out.patchpoint(pointerType());
     8463        patchpoint->effects.terminal = true;
     8464        patchpoint->appendSomeRegister(allocator);
     8465        patchpoint->numGPScratchRegisters++;
     8466        patchpoint->resultConstraint = ValueRep::SomeEarlyRegister;
     8467       
     8468        m_out.appendSuccessor(usually(continuation));
     8469        m_out.appendSuccessor(rarely(slowPath));
     8470       
     8471        patchpoint->setGenerator(
     8472            [=] (CCallHelpers& jit, const StackmapGenerationParams& params) {
     8473                CCallHelpers::JumpList jumpToSlowPath;
     8474               
     8475                // We use a patchpoint to emit the allocation path because whenever we mess with
     8476                // allocation paths, we already reason about them at the machine code level. We know
     8477                // exactly what instruction sequence we want. We're confident that no compiler
     8478                // optimization could make this code better. So, it's best to have the code in
     8479                // AssemblyHelpers::emitAllocate(). That way, the same optimized path is shared by
     8480                // all of the compiler tiers.
     8481                jit.emitAllocateWithNonNullAllocator(
     8482                    params[0].gpr(), actualAllocator, params[1].gpr(), params.gpScratch(0),
     8483                    jumpToSlowPath);
     8484               
     8485                CCallHelpers::Jump jumpToSuccess;
     8486                if (!params.fallsThroughToSuccessor(0))
     8487                    jumpToSuccess = jit.jump();
     8488               
     8489                Vector<Box<CCallHelpers::Label>> labels = params.successorLabels();
     8490               
     8491                params.addLatePath(
     8492                    [=] (CCallHelpers& jit) {
     8493                        jumpToSlowPath.linkTo(*labels[1], &jit);
     8494                        if (jumpToSuccess.isSet())
     8495                            jumpToSuccess.linkTo(*labels[0], &jit);
     8496                    });
     8497            });
     8498       
     8499        m_out.appendTo(continuation, lastNext);
     8500        return patchpoint;
    85138501    }
    85148502   
    … …  
    85238511    LValue allocateCell(LValue allocator, Structure* structure, LBasicBlock slowPath)
    85248512    {
    8525         LValue result = allocateCell(allocator, slowPath);
     8513        LValue result = allocateHeapCell(allocator, slowPath);
    85268514        storeStructure(result, structure);
    85278515        return result;
    … …  
    85408528        size_t size, Structure* structure, LValue butterfly, LBasicBlock slowPath)
    85418529    {
    8542         MarkedAllocator* allocator = &vm().heap.allocatorForObjectOfType<ClassType>(size);
     8530        MarkedAllocator* allocator = vm().heap.allocatorForObjectOfType<ClassType>(size);
    85438531        return allocateObject(m_out.constIntPtr(allocator), structure, butterfly, slowPath);
    85448532    }
    … …  
    85498537        return allocateObject<ClassType>(
    85508538            ClassType::allocationSize(0), structure, butterfly, slowPath);
     8539    }
     8540   
     8541    LValue allocatorForSize(LValue subspace, LValue size, LBasicBlock slowPath)
     8542    {
     8543        static_assert(!(MarkedSpace::sizeStep & (MarkedSpace::sizeStep - 1)), "MarkedSpace::sizeStep must be a power of two.");
     8544       
     8545        // Try to do some constant-folding here.
     8546        if (subspace->hasIntPtr() && size->hasIntPtr()) {
     8547            MarkedSpace::Subspace* actualSubspace = bitwise_cast<MarkedSpace::Subspace*>(subspace->asIntPtr());
     8548            size_t actualSize = size->asIntPtr();
     8549           
     8550            MarkedAllocator* actualAllocator = MarkedSpace::allocatorFor(*actualSubspace, actualSize);
     8551            if (!actualAllocator) {
     8552                LBasicBlock continuation = m_out.newBlock();
     8553                LBasicBlock lastNext = m_out.insertNewBlocksBefore(continuation);
     8554                m_out.jump(slowPath);
     8555                m_out.appendTo(continuation, lastNext);
     8556                return m_out.intPtrZero;
     8557            }
     8558           
     8559            return m_out.constIntPtr(actualAllocator);
     8560        }
     8561       
     8562        unsigned stepShift = getLSBSet(MarkedSpace::sizeStep);
     8563       
     8564        LBasicBlock continuation = m_out.newBlock();
     8565       
     8566        LBasicBlock lastNext = m_out.insertNewBlocksBefore(continuation);
     8567       
     8568        LValue sizeClassIndex = m_out.lShr(
     8569            m_out.add(size, m_out.constIntPtr(MarkedSpace::sizeStep - 1)),
     8570            m_out.constInt32(stepShift));
     8571       
     8572        m_out.branch(
     8573            m_out.above(sizeClassIndex, m_out.constIntPtr(MarkedSpace::largeCutoff >> stepShift)),
     8574            rarely(slowPath), usually(continuation));
     8575       
     8576        m_out.appendTo(continuation, lastNext);
     8577       
     8578        return m_out.loadPtr(
     8579            m_out.baseIndex(
     8580                m_heaps.MarkedSpace_Subspace_allocatorForSizeStep,
     8581                subspace, m_out.sub(sizeClassIndex, m_out.intPtrOne)));
     8582    }
     8583   
     8584    LValue allocatorForSize(MarkedSpace::Subspace& subspace, LValue size, LBasicBlock slowPath)
     8585    {
     8586        return allocatorForSize(m_out.constIntPtr(&subspace), size, slowPath);
    85518587    }
    85528588   
    … …  
    85558591        LValue size, Structure* structure, LValue butterfly, LBasicBlock slowPath)
    85568592    {
    8557         static_assert(!(MarkedSpace::preciseStep & (MarkedSpace::preciseStep - 1)), "MarkedSpace::preciseStep must be a power of two.");
    8558         static_assert(!(MarkedSpace::impreciseStep & (MarkedSpace::impreciseStep - 1)), "MarkedSpace::impreciseStep must be a power of two.");
    8559 
    8560         LValue subspace = m_out.constIntPtr(&vm().heap.subspaceForObjectOfType<ClassType>());
    8561        
    8562         LBasicBlock smallCaseBlock = m_out.newBlock();
    8563         LBasicBlock largeOrOversizeCaseBlock = m_out.newBlock();
    8564         LBasicBlock largeCaseBlock = m_out.newBlock();
    8565         LBasicBlock continuation = m_out.newBlock();
    8566        
    8567         LValue uproundedSize = m_out.add(size, m_out.constInt32(MarkedSpace::preciseStep - 1));
    8568         LValue isSmall = m_out.below(uproundedSize, m_out.constInt32(MarkedSpace::preciseCutoff));
    8569         m_out.branch(isSmall, unsure(smallCaseBlock), unsure(largeOrOversizeCaseBlock));
    8570        
    8571         LBasicBlock lastNext = m_out.appendTo(smallCaseBlock, largeOrOversizeCaseBlock);
    8572         TypedPointer address = m_out.baseIndex(
    8573             m_heaps.MarkedSpace_Subspace_preciseAllocators, subspace,
    8574             m_out.zeroExtPtr(m_out.lShr(uproundedSize, m_out.constInt32(getLSBSet(MarkedSpace::preciseStep)))));
    8575         ValueFromBlock smallAllocator = m_out.anchor(address.value());
    8576         m_out.jump(continuation);
    8577        
    8578         m_out.appendTo(largeOrOversizeCaseBlock, largeCaseBlock);
    8579         m_out.branch(
    8580             m_out.below(uproundedSize, m_out.constInt32(MarkedSpace::impreciseCutoff)),
    8581             usually(largeCaseBlock), rarely(slowPath));
    8582        
    8583         m_out.appendTo(largeCaseBlock, continuation);
    8584         address = m_out.baseIndex(
    8585             m_heaps.MarkedSpace_Subspace_impreciseAllocators, subspace,
    8586             m_out.zeroExtPtr(m_out.lShr(uproundedSize, m_out.constInt32(getLSBSet(MarkedSpace::impreciseStep)))));
    8587         ValueFromBlock largeAllocator = m_out.anchor(address.value());
    8588         m_out.jump(continuation);
    8589        
    8590         m_out.appendTo(continuation, lastNext);
    8591         LValue allocator = m_out.phi(pointerType(), smallAllocator, largeAllocator);
    8592        
     8593        LValue allocator = allocatorForSize(
     8594            vm().heap.subspaceForObjectOfType<ClassType>(), size, slowPath);
    85938595        return allocateObject(allocator, structure, butterfly, slowPath);
    85948596    }
    … …  
    86238625    {
    86248626        size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
    8625         MarkedAllocator* allocator = &vm().heap.allocatorForObjectWithoutDestructor(allocationSize);
     8627        MarkedAllocator* allocator = vm().heap.allocatorForObjectWithoutDestructor(allocationSize);
     8628       
     8629        // FIXME: If the allocator is null, we could simply emit a normal C call to the allocator
     8630        // instead of putting it on the slow path.
     8631        // https://bugs.webkit.org/show_bug.cgi?id=161062
    86268632       
    86278633        LBasicBlock slowPath = m_out.newBlock();
    … …  
    86668672        LValue butterfly;
    86678673    };
    8668     ArrayValues allocateJSArray(
    8669         Structure* structure, unsigned numElements, LBasicBlock slowPath)
    8670     {
     8674
     8675    ArrayValues allocateJSArray(LValue publicLength, Structure* structure, bool shouldInitializeElements = true, bool shouldLargeArraySizeCreateArrayStorage = true)
     8676    {
     8677        JSGlobalObject* globalObject = m_graph.globalObjectFor(m_node->origin.semantic);
     8678        IndexingType indexingType = structure->indexingType();
    86718679        ASSERT(
    8672             hasUndecided(structure->indexingType())
    8673             || hasInt32(structure->indexingType())
    8674             || hasDouble(structure->indexingType())
    8675             || hasContiguous(structure->indexingType()));
    8676        
    8677         unsigned vectorLength = std::max(BASE_VECTOR_LEN, numElements);
    8678        
    8679         LValue endOfStorage = allocateBasicStorageAndGetEnd(
    8680             m_out.constIntPtr(sizeof(JSValue) * vectorLength + sizeof(IndexingHeader)),
    8681             slowPath);
    8682        
    8683         LValue butterfly = m_out.sub(
    8684             endOfStorage, m_out.constIntPtr(sizeof(JSValue) * vectorLength));
    8685        
    8686         LValue object = allocateObject<JSArray>(
    8687             structure, butterfly, slowPath);
    8688        
    8689         m_out.store32(m_out.constInt32(numElements), butterfly, m_heaps.Butterfly_publicLength);
    8690         m_out.store32(m_out.constInt32(vectorLength), butterfly, m_heaps.Butterfly_vectorLength);
    8691        
    8692         if (hasDouble(structure->indexingType())) {
    8693             for (unsigned i = numElements; i < vectorLength; ++i) {
    8694                 m_out.store64(
    8695                     m_out.constInt64(bitwise_cast<int64_t>(PNaN)),
    8696                     butterfly, m_heaps.indexedDoubleProperties[i]);
     8680            hasUndecided(indexingType)
     8681            || hasInt32(indexingType)
     8682            || hasDouble(indexingType)
     8683            || hasContiguous(indexingType));
     8684
     8685        LBasicBlock fastCase = m_out.newBlock();
     8686        LBasicBlock largeCase = m_out.newBlock();
     8687        LBasicBlock failCase = m_out.newBlock();
     8688        LBasicBlock continuation = m_out.newBlock();
     8689        LBasicBlock slowCase = m_out.newBlock();
     8690       
     8691        LBasicBlock lastNext = m_out.insertNewBlocksBefore(fastCase);
     8692       
     8693        ValueFromBlock noButterfly = m_out.anchor(m_out.intPtrZero);
     8694       
     8695        LValue predicate;
     8696        if (shouldLargeArraySizeCreateArrayStorage)
     8697            predicate = m_out.aboveOrEqual(publicLength, m_out.constInt32(MIN_ARRAY_STORAGE_CONSTRUCTION_LENGTH));
     8698        else
     8699            predicate = m_out.booleanFalse;
     8700       
     8701        m_out.branch(predicate, rarely(largeCase), usually(fastCase));
     8702       
     8703        m_out.appendTo(fastCase, largeCase);
     8704
     8705        LValue vectorLength = nullptr;
     8706        if (publicLength->hasInt32()) {
     8707            unsigned publicLengthConst = static_cast<unsigned>(publicLength->asInt32());
     8708            if (publicLengthConst <= MAX_STORAGE_VECTOR_LENGTH) {
     8709                vectorLength = m_out.constInt32(
     8710                    Butterfly::optimalContiguousVectorLength(
     8711                        structure->outOfLineCapacity(), publicLengthConst));
    86978712            }
    86988713        }
    86998714       
    8700         return ArrayValues(object, butterfly);
    8701     }
    8702    
    8703     ArrayValues allocateJSArray(Structure* structure, unsigned numElements)
    8704     {
    8705         LBasicBlock slowPath = m_out.newBlock();
    8706         LBasicBlock continuation = m_out.newBlock();
    8707        
    8708         LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath);
    8709        
    8710         ArrayValues fastValues = allocateJSArray(structure, numElements, slowPath);
    8711         ValueFromBlock fastArray = m_out.anchor(fastValues.array);
    8712         ValueFromBlock fastButterfly = m_out.anchor(fastValues.butterfly);
    8713        
     8715        if (!vectorLength) {
     8716            // We don't compute the optimal vector length for new Array(blah) where blah is not
     8717            // statically known, since the compute effort of doing it here is probably not worth it.
     8718            vectorLength = publicLength;
     8719        }
     8720           
     8721        LValue payloadSize =
     8722            m_out.shl(m_out.zeroExt(vectorLength, pointerType()), m_out.constIntPtr(3));
     8723           
     8724        LValue butterflySize = m_out.add(
     8725            payloadSize, m_out.constIntPtr(sizeof(IndexingHeader)));
     8726           
     8727        LValue allocator = allocatorForSize(
     8728            vm().heap.subspaceForAuxiliaryData(), butterflySize, failCase);
     8729        LValue startOfStorage = allocateHeapCell(allocator, failCase);
     8730           
     8731        LValue butterfly = m_out.add(startOfStorage, m_out.constIntPtr(sizeof(IndexingHeader)));
     8732       
     8733        m_out.store32(publicLength, butterfly, m_heaps.Butterfly_publicLength);
     8734        m_out.store32(vectorLength, butterfly, m_heaps.Butterfly_vectorLength);
     8735   
     8736        initializeArrayElements(
     8737            indexingType,
     8738            shouldInitializeElements ? m_out.int32Zero : publicLength, vectorLength,
     8739            butterfly);
     8740       
     8741        ValueFromBlock haveButterfly = m_out.anchor(butterfly);
     8742       
     8743        LValue object = allocateObject<JSArray>(structure, butterfly, failCase);
     8744           
     8745        ValueFromBlock fastResult = m_out.anchor(object);
     8746        ValueFromBlock fastButterfly = m_out.anchor(butterfly);
    87148747        m_out.jump(continuation);
    87158748       
    8716         m_out.appendTo(slowPath, continuation);
    8717 
    8718         LValue slowArrayValue = lazySlowPath(
     8749        m_out.appendTo(largeCase, failCase);
     8750        ValueFromBlock largeStructure = m_out.anchor(
     8751            m_out.constIntPtr(
     8752                globalObject->arrayStructureForIndexingTypeDuringAllocation(ArrayWithArrayStorage)));
     8753        m_out.jump(slowCase);
     8754       
     8755        m_out.appendTo(failCase, slowCase);
     8756        ValueFromBlock failStructure = m_out.anchor(m_out.constIntPtr(structure));
     8757        m_out.jump(slowCase);
     8758       
     8759        m_out.appendTo(slowCase, continuation);
     8760        LValue structureValue = m_out.phi(pointerType(), largeStructure, failStructure);
     8761        LValue butterflyValue = m_out.phi(pointerType(), noButterfly, haveButterfly);
     8762
     8763        LValue slowResultValue = lazySlowPath(
    87198764            [=] (const Vector<Location>& locations) -> RefPtr<LazySlowPath::Generator> {
    87208765                return createLazyCallGenerator(
    87218766                    operationNewArrayWithSize, locations[0].directGPR(),
    8722                     CCallHelpers::TrustedImmPtr(structure), CCallHelpers::TrustedImm32(numElements));
    8723             });
    8724         ValueFromBlock slowArray = m_out.anchor(slowArrayValue);
     8767                    locations[1].directGPR(), locations[2].directGPR(), locations[3].directGPR());
     8768            },
     8769            structureValue, publicLength, butterflyValue);
     8770        ValueFromBlock slowResult = m_out.anchor(slowResultValue);
    87258771        ValueFromBlock slowButterfly = m_out.anchor(
    8726             m_out.loadPtr(slowArrayValue, m_heaps.JSObject_butterfly));
    8727 
     8772            m_out.loadPtr(slowResultValue, m_heaps.JSObject_butterfly));
    87288773        m_out.jump(continuation);
    87298774       
    87308775        m_out.appendTo(continuation, lastNext);
    8731        
    87328776        return ArrayValues(
    8733             m_out.phi(pointerType(), fastArray, slowArray),
     8777            m_out.phi(pointerType(), fastResult, slowResult),
    87348778            m_out.phi(pointerType(), fastButterfly, slowButterfly));
     8779    }
     8780   
     8781    ArrayValues allocateUninitializedContiguousJSArray(LValue publicLength, Structure* structure)
     8782    {
     8783        bool shouldInitializeElements = false;
     8784        bool shouldLargeArraySizeCreateArrayStorage = false;
     8785        return allocateJSArray(
     8786            publicLength, structure, shouldInitializeElements,
     8787            shouldLargeArraySizeCreateArrayStorage);
    87358788    }
    87368789   

  • trunk/Source/JavaScriptCore/ftl/FTLOutput.cpp

    r204912 r205462  
    101101LValue Output::constBool(bool value)
    102102{
     103    if (value)
     104        return booleanTrue;
     105    return booleanFalse;
     106}
     107
     108LValue Output::constInt32(int32_t value)
     109{
    103110    return m_block->appendNew<B3::Const32Value>(m_proc, origin(), value);
    104111}
    105112
    106 LValue Output::constInt32(int32_t value)
    107 {
    108     return m_block->appendNew<B3::Const32Value>(m_proc, origin(), value);
    109 }
    110 
    111113LValue Output::constInt64(int64_t value)
    112114{
    … …  
    126128LValue Output::add(LValue left, LValue right)
    127129{
     130    if (Value* result = left->addConstant(m_proc, right)) {
     131        m_block->append(result);
     132        return result;
     133    }
    128134    return m_block->appendNew<B3::Value>(m_proc, B3::Add, origin(), left, right);
    129135}
    … …  
    206212LValue Output::shl(LValue left, LValue right)
    207213{
    208     return m_block->appendNew<B3::Value>(m_proc, B3::Shl, origin(), left, castToInt32(right));
     214    right = castToInt32(right);
     215    if (Value* result = left->shlConstant(m_proc, right)) {
     216        m_block->append(result);
     217        return result;
     218    }
     219    return m_block->appendNew<B3::Value>(m_proc, B3::Shl, origin(), left, right);
    209220}
    210221
    211222LValue Output::aShr(LValue left, LValue right)
    212223{
    213     return m_block->appendNew<B3::Value>(m_proc, B3::SShr, origin(), left, castToInt32(right));
     224    right = castToInt32(right);
     225    if (Value* result = left->sShrConstant(m_proc, right)) {
     226        m_block->append(result);
     227        return result;
     228    }
     229    return m_block->appendNew<B3::Value>(m_proc, B3::SShr, origin(), left, right);
    214230}
    215231
    216232LValue Output::lShr(LValue left, LValue right)
    217233{
    218     return m_block->appendNew<B3::Value>(m_proc, B3::ZShr, origin(), left, castToInt32(right));
     234    right = castToInt32(right);
     235    if (Value* result = left->zShrConstant(m_proc, right)) {
     236        m_block->append(result);
     237        return result;
     238    }
     239    return m_block->appendNew<B3::Value>(m_proc, B3::ZShr, origin(), left, right);
    219240}
    220241
    … …  
    344365    if (value->type() == type)
    345366        return value;
     367    if (value->hasInt32())
     368        return m_block->appendIntConstant(m_proc, origin(), Int64, static_cast<uint64_t>(static_cast<uint32_t>(value->asInt32())));
    346369    return m_block->appendNew<B3::Value>(m_proc, B3::ZExt32, origin(), value);
    347370}
    … …  
    359382LValue Output::castToInt32(LValue value)
    360383{
    361     return value->type() == B3::Int32 ? value :
    362         m_block->appendNew<B3::Value>(m_proc, B3::Trunc, origin(), value);
     384    if (value->type() == Int32)
     385        return value;
     386    if (value->hasInt64())
     387        return constInt32(static_cast<int32_t>(value->asInt64()));
     388    return m_block->appendNew<B3::Value>(m_proc, B3::Trunc, origin(), value);
    363389}
    364390
    … …  
    454480LValue Output::equal(LValue left, LValue right)
    455481{
     482    TriState result = left->equalConstant(right);
     483    if (result != MixedTriState)
     484        return constBool(result == TrueTriState);
    456485    return m_block->appendNew<B3::Value>(m_proc, B3::Equal, origin(), left, right);
    457486}
    … …  
    459488LValue Output::notEqual(LValue left, LValue right)
    460489{
     490    TriState result = left->notEqualConstant(right);
     491    if (result != MixedTriState)
     492        return constBool(result == TrueTriState);
    461493    return m_block->appendNew<B3::Value>(m_proc, B3::NotEqual, origin(), left, right);
    462494}
    … …  
    464496LValue Output::above(LValue left, LValue right)
    465497{
     498    TriState result = left->aboveConstant(right);
     499    if (result != MixedTriState)
     500        return constBool(result == TrueTriState);
    466501    return m_block->appendNew<B3::Value>(m_proc, B3::Above, origin(), left, right);
    467502}
    … …  
    469504LValue Output::aboveOrEqual(LValue left, LValue right)
    470505{
     506    TriState result = left->aboveEqualConstant(right);
     507    if (result != MixedTriState)
     508        return constBool(result == TrueTriState);
    471509    return m_block->appendNew<B3::Value>(m_proc, B3::AboveEqual, origin(), left, right);
    472510}
    … …  
    474512LValue Output::below(LValue left, LValue right)
    475513{
     514    TriState result = left->belowConstant(right);
     515    if (result != MixedTriState)
     516        return constBool(result == TrueTriState);
    476517    return m_block->appendNew<B3::Value>(m_proc, B3::Below, origin(), left, right);
    477518}
    … …  
    479520LValue Output::belowOrEqual(LValue left, LValue right)
    480521{
     522    TriState result = left->belowEqualConstant(right);
     523    if (result != MixedTriState)
     524        return constBool(result == TrueTriState);
    481525    return m_block->appendNew<B3::Value>(m_proc, B3::BelowEqual, origin(), left, right);
    482526}
    … …  
    484528LValue Output::greaterThan(LValue left, LValue right)
    485529{
     530    TriState result = left->greaterThanConstant(right);
     531    if (result != MixedTriState)
     532        return constBool(result == TrueTriState);
    486533    return m_block->appendNew<B3::Value>(m_proc, B3::GreaterThan, origin(), left, right);
    487534}
    … …  
    489536LValue Output::greaterThanOrEqual(LValue left, LValue right)
    490537{
     538    TriState result = left->greaterEqualConstant(right);
     539    if (result != MixedTriState)
     540        return constBool(result == TrueTriState);
    491541    return m_block->appendNew<B3::Value>(m_proc, B3::GreaterEqual, origin(), left, right);
    492542}
    … …  
    494544LValue Output::lessThan(LValue left, LValue right)
    495545{
     546    TriState result = left->lessThanConstant(right);
     547    if (result != MixedTriState)
     548        return constBool(result == TrueTriState);
    496549    return m_block->appendNew<B3::Value>(m_proc, B3::LessThan, origin(), left, right);
    497550}
    … …  
    499552LValue Output::lessThanOrEqual(LValue left, LValue right)
    500553{
     554    TriState result = left->lessEqualConstant(right);
     555    if (result != MixedTriState)
     556        return constBool(result == TrueTriState);
    501557    return m_block->appendNew<B3::Value>(m_proc, B3::LessEqual, origin(), left, right);
    502558}
    … …  
    584640LValue Output::select(LValue value, LValue taken, LValue notTaken)
    585641{
     642    if (value->hasInt32()) {
     643        if (value->asInt32())
     644            return taken;
     645        else
     646            return notTaken;
     647    }
    586648    return m_block->appendNew<B3::Value>(m_proc, B3::Select, origin(), value, taken, notTaken);
    587649}
    … …  
    620682{
    621683    m_block->appendNewControlValue(m_proc, B3::Oops, origin());
     684}
     685
     686void Output::appendSuccessor(WeightedTarget target)
     687{
     688    m_block->appendSuccessor(target.frequentedBlock());
    622689}
    623690
    … …  
    742809void Output::addIncomingToPhi(LValue phi, ValueFromBlock value)
    743810{
    744     value.value()->as<B3::UpsilonValue>()->setPhi(phi);
     811    if (value)
     812        value.value()->as<B3::UpsilonValue>()->setPhi(phi);
    745813}
    746814

  • trunk/Source/JavaScriptCore/ftl/FTLOutput.h

    r204912 r205462  
    399399
    400400    void unreachable();
     401   
     402    void appendSuccessor(WeightedTarget);
    401403
    402404    B3::CheckValue* speculate(LValue);

  • trunk/Source/JavaScriptCore/ftl/FTLValueFromBlock.h

    r204912 r205462  
    4646    {
    4747    }
     48   
     49    explicit operator bool() const { return m_value || m_block; }
    4850
    4951    LValue value() const { return m_value; }

  • trunk/Source/JavaScriptCore/ftl/FTLWeightedTarget.h

    r204912 r205462  
    11/*
    2  * Copyright (C) 2014 Apple Inc. All rights reserved.
     2 * Copyright (C) 2014, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    5656    Weight weight() const { return m_weight; }
    5757   
     58    B3::FrequentedBlock frequentedBlock() const
     59    {
     60        return B3::FrequentedBlock(target(), weight().frequencyClass());
     61    }
     62   
    5863private:
    5964    LBasicBlock m_target;

  • trunk/Source/JavaScriptCore/heap/ConservativeRoots.cpp

    r204912 r205462  
    11/*
    2  * Copyright (C) 2011 Apple Inc. All rights reserved.
     2 * Copyright (C) 2011, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3232#include "CopiedSpaceInlines.h"
    3333#include "HeapInlines.h"
     34#include "HeapUtil.h"
     35#include "JITStubRoutineSet.h"
    3436#include "JSCell.h"
    3537#include "JSObject.h"
    … …  
    4042namespace JSC {
    4143
    42 ConservativeRoots::ConservativeRoots(MarkedBlockSet* blocks, CopiedSpace* copiedSpace)
     44ConservativeRoots::ConservativeRoots(Heap& heap)
    4345    : m_roots(m_inlineRoots)
    4446    , m_size(0)
    4547    , m_capacity(inlineCapacity)
    46     , m_blocks(blocks)
    47     , m_copiedSpace(copiedSpace)
     48    , m_heap(heap)
    4849{
    4950}
    … …  
    5253{
    5354    if (m_roots != m_inlineRoots)
    54         OSAllocator::decommitAndRelease(m_roots, m_capacity * sizeof(JSCell*));
     55        OSAllocator::decommitAndRelease(m_roots, m_capacity * sizeof(HeapCell*));
    5556}
    5657
    … …  
    5859{
    5960    size_t newCapacity = m_capacity == inlineCapacity ? nonInlineCapacity : m_capacity * 2;
    60     JSCell** newRoots = static_cast<JSCell**>(OSAllocator::reserveAndCommit(newCapacity * sizeof(JSCell*)));
    61     memcpy(newRoots, m_roots, m_size * sizeof(JSCell*));
     61    HeapCell** newRoots = static_cast<HeapCell**>(OSAllocator::reserveAndCommit(newCapacity * sizeof(HeapCell*)));
     62    memcpy(newRoots, m_roots, m_size * sizeof(HeapCell*));
    6263    if (m_roots != m_inlineRoots)
    63         OSAllocator::decommitAndRelease(m_roots, m_capacity * sizeof(JSCell*));
     64        OSAllocator::decommitAndRelease(m_roots, m_capacity * sizeof(HeapCell*));
    6465    m_capacity = newCapacity;
    6566    m_roots = newRoots;
    … …  
    6768
    6869template<typename MarkHook>
    69 inline void ConservativeRoots::genericAddPointer(void* p, TinyBloomFilter filter, MarkHook& markHook)
     70inline void ConservativeRoots::genericAddPointer(void* p, int64_t version, TinyBloomFilter filter, MarkHook& markHook)
    7071{
    7172    markHook.mark(p);
    7273
    73     m_copiedSpace->pinIfNecessary(p);
     74    m_heap.storageSpace().pinIfNecessary(p);
    7475
    75     if (!Heap::isPointerGCObject(filter, *m_blocks, p))
    76         return;
    77 
    78     if (m_size == m_capacity)
    79         grow();
    80 
    81     m_roots[m_size++] = static_cast<JSCell*>(p);
     76    HeapUtil::findGCObjectPointersForMarking(
     77        m_heap, version, filter, p,
     78        [&] (void* p) {
     79            if (m_size == m_capacity)
     80                grow();
     81           
     82            m_roots[m_size++] = bitwise_cast<HeapCell*>(p);
     83        });
    8284}
    8385
    … …  
    9597    RELEASE_ASSERT(isPointerAligned(end));
    9698
    97     TinyBloomFilter filter = m_blocks->filter(); // Make a local copy of filter to show the compiler it won't alias, and can be register-allocated.
     99    TinyBloomFilter filter = m_heap.objectSpace().blocks().filter(); // Make a local copy of filter to show the compiler it won't alias, and can be register-allocated.
     100    int64_t version = m_heap.objectSpace().version();
    98101    for (char** it = static_cast<char**>(begin); it != static_cast<char**>(end); ++it)
    99         genericAddPointer(*it, filter, markHook);
     102        genericAddPointer(*it, version, filter, markHook);
    100103}
    101104

  • trunk/Source/JavaScriptCore/heap/ConservativeRoots.h

    r204912 r205462  
    11/*
    2  * Copyright (C) 2009 Apple Inc. All rights reserved.
     2 * Copyright (C) 2009, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3232
    3333class CodeBlockSet;
     34class HeapCell;
    3435class JITStubRoutineSet;
    35 class JSCell;
    3636
    3737class ConservativeRoots {
    3838public:
    39     ConservativeRoots(MarkedBlockSet*, CopiedSpace*);
     39    ConservativeRoots(Heap&);
    4040    ~ConservativeRoots();
    4141
    … …  
    4545   
    4646    size_t size();
    47     JSCell** roots();
     47    HeapCell** roots();
    4848
    4949private:
    5050    static const size_t inlineCapacity = 128;
    51     static const size_t nonInlineCapacity = 8192 / sizeof(JSCell*);
     51    static const size_t nonInlineCapacity = 8192 / sizeof(HeapCell*);
    5252   
    5353    template<typename MarkHook>
    54     void genericAddPointer(void*, TinyBloomFilter, MarkHook&);
     54    void genericAddPointer(void*, int64_t heapVersion, TinyBloomFilter, MarkHook&);
    5555
    5656    template<typename MarkHook>
    … …  
    5959    void grow();
    6060
    61     JSCell** m_roots;
     61    HeapCell** m_roots;
    6262    size_t m_size;
    6363    size_t m_capacity;
    64     MarkedBlockSet* m_blocks;
    65     CopiedSpace* m_copiedSpace;
    66     JSCell* m_inlineRoots[inlineCapacity];
     64    Heap& m_heap;
     65    HeapCell* m_inlineRoots[inlineCapacity];
    6766};
    6867
    … …  
    7271}
    7372
    74 inline JSCell** ConservativeRoots::roots()
     73inline HeapCell** ConservativeRoots::roots()
    7574{
    7675    return m_roots;

  • trunk/Source/JavaScriptCore/heap/CopyToken.h

    r204912 r205462  
    11/*
    2  * Copyright (C) 2013, 2015 Apple Inc. All rights reserved.
     2 * Copyright (C) 2013, 2015-2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3030
    3131enum CopyToken {
    32     ButterflyCopyToken,
    3332    TypedArrayVectorCopyToken,
    3433    MapBackingStoreCopyToken,

  • trunk/Source/JavaScriptCore/heap/Heap.cpp

    r204912 r205462  
    3232#include "GCActivityCallback.h"
    3333#include "GCIncomingRefCountedSetInlines.h"
     34#include "GCTypeMap.h"
    3435#include "HeapHelperPool.h"
    3536#include "HeapIterationScope.h"
    … …  
    4142#include "IncrementalSweeper.h"
    4243#include "Interpreter.h"
     44#include "JITStubRoutineSet.h"
    4345#include "JITWorklist.h"
    4446#include "JSCInlines.h"
    … …  
    4850#include "SamplingProfiler.h"
    4951#include "ShadowChicken.h"
     52#include "SuperSampler.h"
    5053#include "TypeProfilerLog.h"
    5154#include "UnlinkedCodeBlock.h"
    … …  
    5861#include <wtf/ProcessID.h>
    5962#include <wtf/RAMSize.h>
     63#include <wtf/SimpleStats.h>
    6064
    6165#if USE(FOUNDATION)
    … …  
    7579
    7680static const size_t largeHeapSize = 32 * MB; // About 1.5X the average webpage.
    77 static const size_t smallHeapSize = 1 * MB; // Matches the FastMalloc per-thread cache.
    78 
    79 #define ENABLE_GC_LOGGING 0
    80 
    81 #if ENABLE(GC_LOGGING)
    82 #if COMPILER(CLANG)
    83 #define DEFINE_GC_LOGGING_GLOBAL(type, name, arguments) \
    84 _Pragma("clang diagnostic push") \
    85 _Pragma("clang diagnostic ignored \"-Wglobal-constructors\"") \
    86 _Pragma("clang diagnostic ignored \"-Wexit-time-destructors\"") \
    87 static type name arguments; \
    88 _Pragma("clang diagnostic pop")
    89 #else
    90 #define DEFINE_GC_LOGGING_GLOBAL(type, name, arguments) \
    91 static type name arguments;
    92 #endif // COMPILER(CLANG)
    93 
    94 struct GCTimer {
    95     GCTimer(const char* name)
    96         : name(name)
    97     {
    98     }
    99     ~GCTimer()
    100     {
    101         logData(allCollectionData, "(All)");
    102         logData(edenCollectionData, "(Eden)");
    103         logData(fullCollectionData, "(Full)");
    104     }
    105 
    106     struct TimeRecord {
    107         TimeRecord()
    108             : time(0)
    109             , min(std::numeric_limits<double>::infinity())
    110             , max(0)
    111             , count(0)
    112         {
    113         }
    114 
    115         double time;
    116         double min;
    117         double max;
    118         size_t count;
    119     };
    120 
    121     void logData(const TimeRecord& data, const char* extra)
    122     {
    123         dataLogF("[%d] %s (Parent: %s) %s: %.2lfms (avg. %.2lf, min. %.2lf, max. %.2lf, count %lu)\n",
    124             getCurrentProcessID(),
    125             name,
    126             parent ? parent->name : "nullptr",
    127             extra,
    128             data.time * 1000,
    129             data.time * 1000 / data.count,
    130             data.min * 1000,
    131             data.max * 1000,
    132             data.count);
    133     }
    134 
    135     void updateData(TimeRecord& data, double duration)
    136     {
    137         if (duration < data.min)
    138             data.min = duration;
    139         if (duration > data.max)
    140             data.max = duration;
    141         data.count++;
    142         data.time += duration;
    143     }
    144 
    145     void didFinishPhase(HeapOperation collectionType, double duration)
    146     {
    147         TimeRecord& data = collectionType == EdenCollection ? edenCollectionData : fullCollectionData;
    148         updateData(data, duration);
    149         updateData(allCollectionData, duration);
    150     }
    151 
    152     static GCTimer* s_currentGlobalTimer;
    153 
    154     TimeRecord allCollectionData;
    155     TimeRecord fullCollectionData;
    156     TimeRecord edenCollectionData;
    157     const char* name;
    158     GCTimer* parent { nullptr };
    159 };
    160 
    161 GCTimer* GCTimer::s_currentGlobalTimer = nullptr;
    162 
    163 struct GCTimerScope {
    164     GCTimerScope(GCTimer& timer, HeapOperation collectionType)
    165         : timer(timer)
    166         , start(WTF::monotonicallyIncreasingTime())
    167         , collectionType(collectionType)
    168     {
    169         timer.parent = GCTimer::s_currentGlobalTimer;
    170         GCTimer::s_currentGlobalTimer = &timer;
    171     }
    172     ~GCTimerScope()
    173     {
    174         double delta = WTF::monotonicallyIncreasingTime() - start;
    175         timer.didFinishPhase(collectionType, delta);
    176         GCTimer::s_currentGlobalTimer = timer.parent;
    177     }
    178     GCTimer& timer;
    179     double start;
    180     HeapOperation collectionType;
    181 };
    182 
    183 struct GCCounter {
    184     GCCounter(const char* name)
    185         : name(name)
    186         , count(0)
    187         , total(0)
    188         , min(10000000)
    189         , max(0)
    190     {
    191     }
    192    
    193     void add(size_t amount)
    194     {
    195         count++;
    196         total += amount;
    197         if (amount < min)
    198             min = amount;
    199         if (amount > max)
    200             max = amount;
    201     }
    202     ~GCCounter()
    203     {
    204         dataLogF("[%d] %s: %zu values (avg. %zu, min. %zu, max. %zu)\n", getCurrentProcessID(), name, total, total / count, min, max);
    205     }
    206     const char* name;
    207     size_t count;
    208     size_t total;
    209     size_t min;
    210     size_t max;
    211 };
    212 
    213 #define GCPHASE(name) DEFINE_GC_LOGGING_GLOBAL(GCTimer, name##Timer, (#name)); GCTimerScope name##TimerScope(name##Timer, m_operationInProgress)
    214 #define GCCOUNTER(name, value) do { DEFINE_GC_LOGGING_GLOBAL(GCCounter, name##Counter, (#name)); name##Counter.add(value); } while (false)
    215    
    216 #else
    217 
    218 #define GCPHASE(name) do { } while (false)
    219 #define GCCOUNTER(name, value) do { } while (false)
    220 #endif
    221 
    222 static inline size_t minHeapSize(HeapType heapType, size_t ramSize)
     81const size_t smallHeapSize = 1 * MB; // Matches the FastMalloc per-thread cache.
     82
     83size_t minHeapSize(HeapType heapType, size_t ramSize)
    22384{
    22485    if (heapType == LargeHeap)
    … …  
    22788}
    22889
    229 static inline size_t proportionalHeapSize(size_t heapSize, size_t ramSize)
     90size_t proportionalHeapSize(size_t heapSize, size_t ramSize)
    23091{
    23192    // Try to stay under 1/2 RAM size to leave room for the DOM, rendering, networking, etc.
    … …  
    23798}
    23899
    239 static inline bool isValidSharedInstanceThreadState(VM* vm)
     100bool isValidSharedInstanceThreadState(VM* vm)
    240101{
    241102    return vm->currentThreadIsHoldingAPILock();
    242103}
    243104
    244 static inline bool isValidThreadState(VM* vm)
     105bool isValidThreadState(VM* vm)
    245106{
    246107    if (vm->atomicStringTable() != wtfThreadData().atomicStringTable())
    … …  
    253114}
    254115
    255 static inline void recordType(TypeCountSet& set, JSCell* cell)
     116void recordType(TypeCountSet& set, JSCell* cell)
    256117{
    257118    const char* typeName = "[unknown]";
    … …  
    261122    set.add(typeName);
    262123}
     124
     125bool measurePhaseTiming()
     126{
     127    return false;
     128}
     129
     130HashMap<const char*, GCTypeMap<SimpleStats>>& timingStats()
     131{
     132    static HashMap<const char*, GCTypeMap<SimpleStats>>* result;
     133    static std::once_flag once;
     134    std::call_once(
     135        once,
     136        [] {
     137            result = new HashMap<const char*, GCTypeMap<SimpleStats>>();
     138        });
     139    return *result;
     140}
     141
     142SimpleStats& timingStats(const char* name, HeapOperation operation)
     143{
     144    return timingStats().add(name, GCTypeMap<SimpleStats>()).iterator->value[operation];
     145}
     146
     147class TimingScope {
     148public:
     149    TimingScope(HeapOperation operation, const char* name)
     150        : m_operation(operation)
     151        , m_name(name)
     152    {
     153        if (measurePhaseTiming())
     154            m_before = monotonicallyIncreasingTimeMS();
     155    }
     156   
     157    TimingScope(Heap& heap, const char* name)
     158        : TimingScope(heap.operationInProgress(), name)
     159    {
     160    }
     161   
     162    void setOperation(HeapOperation operation)
     163    {
     164        m_operation = operation;
     165    }
     166   
     167    void setOperation(Heap& heap)
     168    {
     169        setOperation(heap.operationInProgress());
     170    }
     171   
     172    ~TimingScope()
     173    {
     174        if (measurePhaseTiming()) {
     175            double after = monotonicallyIncreasingTimeMS();
     176            double timing = after - m_before;
     177            SimpleStats& stats = timingStats(m_name, m_operation);
     178            stats.add(timing);
     179            dataLog("[GC:", m_operation, "] ", m_name, " took: ", timing, " ms (average ", stats.mean(), " ms).\n");
     180        }
     181    }
     182private:
     183    HeapOperation m_operation;
     184    double m_before;
     185    const char* m_name;
     186};
    263187
    264188} // anonymous namespace
    … …  
    288212    , m_slotVisitor(*this)
    289213    , m_handleSet(vm)
     214    , m_codeBlocks(std::make_unique<CodeBlockSet>())
     215    , m_jitStubRoutines(std::make_unique<JITStubRoutineSet>())
    290216    , m_isSafeToCollect(false)
    291217    , m_writeBarrierBuffer(256)
    … …  
    332258
    333259    m_arrayBuffers.lastChanceToFinalize();
    334     m_codeBlocks.lastChanceToFinalize();
     260    m_codeBlocks->lastChanceToFinalize();
    335261    m_objectSpace.lastChanceToFinalize();
    336262    releaseDelayedReleasedObjects();
    … …  
    435361void Heap::finalizeUnconditionalFinalizers()
    436362{
    437     GCPHASE(FinalizeUnconditionalFinalizers);
    438363    m_slotVisitor.finalizeUnconditionalFinalizers();
    439364}
    … …  
    461386void Heap::markRoots(double gcStartTime, void* stackOrigin, void* stackTop, MachineThreads::RegisterState& calleeSavedRegisters)
    462387{
    463     GCPHASE(MarkRoots);
     388    TimingScope markRootsTimingScope(*this, "Heap::markRoots");
     389   
    464390    ASSERT(isValidThreadState(m_vm));
    465391
    466     // We gather conservative roots before clearing mark bits because conservative
    467     // gathering uses the mark bits to determine whether a reference is valid.
    468     ConservativeRoots conservativeRoots(&m_objectSpace.blocks(), &m_storageSpace);
    469     gatherStackRoots(conservativeRoots, stackOrigin, stackTop, calleeSavedRegisters);
    470     gatherJSStackRoots(conservativeRoots);
    471     gatherScratchBufferRoots(conservativeRoots);
     392    HeapRootVisitor heapRootVisitor(m_slotVisitor);
     393   
     394    ConservativeRoots conservativeRoots(*this);
     395    {
     396        TimingScope preConvergenceTimingScope(*this, "Heap::markRoots before convergence");
     397        // We gather conservative roots before clearing mark bits because conservative
     398        // gathering uses the mark bits to determine whether a reference is valid.
     399        {
     400            TimingScope preConvergenceTimingScope(*this, "Heap::markRoots conservative scan");
     401            SuperSamplerScope superSamplerScope(false);
     402            gatherStackRoots(conservativeRoots, stackOrigin, stackTop, calleeSavedRegisters);
     403            gatherJSStackRoots(conservativeRoots);
     404            gatherScratchBufferRoots(conservativeRoots);
     405        }
    472406
    473407#if ENABLE(DFG_JIT)
    474     DFG::rememberCodeBlocks(*m_vm);
     408        DFG::rememberCodeBlocks(*m_vm);
    475409#endif
    476410
    477411#if ENABLE(SAMPLING_PROFILER)
    478     if (SamplingProfiler* samplingProfiler = m_vm->samplingProfiler()) {
    479         // Note that we need to own the lock from now until we're done
    480         // marking the SamplingProfiler's data because once we verify the
    481         // SamplingProfiler's stack traces, we don't want it to accumulate
    482         // more stack traces before we get the chance to mark it.
    483         // This lock is released inside visitSamplingProfiler().
    484         samplingProfiler->getLock().lock();
    485         samplingProfiler->processUnverifiedStackTraces();
    486     }
     412        if (SamplingProfiler* samplingProfiler = m_vm->samplingProfiler()) {
     413            // Note that we need to own the lock from now until we're done
     414            // marking the SamplingProfiler's data because once we verify the
     415            // SamplingProfiler's stack traces, we don't want it to accumulate
     416            // more stack traces before we get the chance to mark it.
     417            // This lock is released inside visitSamplingProfiler().
     418            samplingProfiler->getLock().lock();
     419            samplingProfiler->processUnverifiedStackTraces();
     420        }
    487421#endif // ENABLE(SAMPLING_PROFILER)
    488422
    489     if (m_operationInProgress == FullCollection) {
    490         m_opaqueRoots.clear();
    491         m_slotVisitor.clearMarkStack();
    492     }
    493 
    494     clearLivenessData();
    495 
    496     m_parallelMarkersShouldExit = false;
    497 
    498     m_helperClient.setFunction(
    499         [this] () {
    500             SlotVisitor* slotVisitor;
    501             {
    502                 LockHolder locker(m_parallelSlotVisitorLock);
    503                 if (m_availableParallelSlotVisitors.isEmpty()) {
    504                     std::unique_ptr<SlotVisitor> newVisitor =
    505                         std::make_unique<SlotVisitor>(*this);
    506                     slotVisitor = newVisitor.get();
    507                     m_parallelSlotVisitors.append(WTFMove(newVisitor));
    508                 } else
    509                     slotVisitor = m_availableParallelSlotVisitors.takeLast();
    510             }
    511 
    512             WTF::registerGCThread();
    513 
    514             {
    515                 ParallelModeEnabler parallelModeEnabler(*slotVisitor);
    516                 slotVisitor->didStartMarking();
    517                 slotVisitor->drainFromShared(SlotVisitor::SlaveDrain);
    518             }
    519 
    520             {
    521                 LockHolder locker(m_parallelSlotVisitorLock);
    522                 m_availableParallelSlotVisitors.append(slotVisitor);
    523             }
    524         });
    525 
    526     m_slotVisitor.didStartMarking();
    527    
    528     HeapRootVisitor heapRootVisitor(m_slotVisitor);
    529 
    530     {
     423        if (m_operationInProgress == FullCollection) {
     424            m_opaqueRoots.clear();
     425            m_slotVisitor.clearMarkStack();
     426        }
     427
     428        clearLivenessData();
     429
     430        m_parallelMarkersShouldExit = false;
     431
     432        m_helperClient.setFunction(
     433            [this] () {
     434                SlotVisitor* slotVisitor;
     435                {
     436                    LockHolder locker(m_parallelSlotVisitorLock);
     437                    if (m_availableParallelSlotVisitors.isEmpty()) {
     438                        std::unique_ptr<SlotVisitor> newVisitor =
     439                            std::make_unique<SlotVisitor>(*this);
     440                        slotVisitor = newVisitor.get();
     441                        m_parallelSlotVisitors.append(WTFMove(newVisitor));
     442                    } else
     443                        slotVisitor = m_availableParallelSlotVisitors.takeLast();
     444                }
     445
     446                WTF::registerGCThread();
     447
     448                {
     449                    ParallelModeEnabler parallelModeEnabler(*slotVisitor);
     450                    slotVisitor->didStartMarking();
     451                    slotVisitor->drainFromShared(SlotVisitor::SlaveDrain);
     452                }
     453
     454                {
     455                    LockHolder locker(m_parallelSlotVisitorLock);
     456                    m_availableParallelSlotVisitors.append(slotVisitor);
     457                }
     458            });
     459
     460        m_slotVisitor.didStartMarking();
     461    }
     462   
     463    {
     464        SuperSamplerScope superSamplerScope(false);
     465        TimingScope convergenceTimingScope(*this, "Heap::markRoots convergence");
    531466        ParallelModeEnabler enabler(m_slotVisitor);
    532 
     467       
    533468        m_slotVisitor.donateAndDrain();
    534469        visitExternalRememberedSet();
    … …  
    545480        converge();
    546481    }
     482   
     483    TimingScope postConvergenceTimingScope(*this, "Heap::markRoots after convergence");
    547484
    548485    // Weak references must be marked last because their liveness depends on
    … …  
    562499void Heap::copyBackingStores()
    563500{
    564     GCPHASE(CopyBackingStores);
     501    SuperSamplerScope superSamplerScope(false);
    565502    if (m_operationInProgress == EdenCollection)
    566503        m_storageSpace.startedCopying<EdenCollection>();
    … …  
    599536                        CopyWorkList& workList = block->workList();
    600537                        for (CopyWorklistItem item : workList) {
    601                             if (item.token() == ButterflyCopyToken) {
    602                                 JSObject::copyBackingStore(
    603                                     item.cell(), copyVisitor, ButterflyCopyToken);
    604                                 continue;
    605                             }
    606                            
    607538                            item.cell()->methodTable()->copyBackingStore(
    608539                                item.cell(), copyVisitor, item.token());
    … …  
    620551void Heap::gatherStackRoots(ConservativeRoots& roots, void* stackOrigin, void* stackTop, MachineThreads::RegisterState& calleeSavedRegisters)
    621552{
    622     GCPHASE(GatherStackRoots);
    623     m_jitStubRoutines.clearMarks();
    624     m_machineThreads.gatherConservativeRoots(roots, m_jitStubRoutines, m_codeBlocks, stackOrigin, stackTop, calleeSavedRegisters);
     553    m_jitStubRoutines->clearMarks();
     554    m_machineThreads.gatherConservativeRoots(roots, *m_jitStubRoutines, *m_codeBlocks, stackOrigin, stackTop, calleeSavedRegisters);
    625555}
    626556
    … …  
    628558{
    629559#if !ENABLE(JIT)
    630     GCPHASE(GatherJSStackRoots);
    631     m_vm->interpreter->cloopStack().gatherConservativeRoots(roots, m_jitStubRoutines, m_codeBlocks);
     560    m_vm->interpreter->cloopStack().gatherConservativeRoots(roots, *m_jitStubRoutines, *m_codeBlocks);
    632561#else
    633562    UNUSED_PARAM(roots);
    … …  
    638567{
    639568#if ENABLE(DFG_JIT)
    640     GCPHASE(GatherScratchBufferRoots);
    641569    m_vm->gatherConservativeRoots(roots);
    642570#else
    … …  
    647575void Heap::clearLivenessData()
    648576{
    649     GCPHASE(ClearLivenessData);
     577    TimingScope timingScope(*this, "Heap::clearLivenessData");
    650578    if (m_operationInProgress == FullCollection)
    651         m_codeBlocks.clearMarksForFullCollection();
    652 
    653     m_objectSpace.clearNewlyAllocated();
    654     m_objectSpace.clearMarks();
     579        m_codeBlocks->clearMarksForFullCollection();
     580   
     581    {
     582        TimingScope clearNewlyAllocatedTimingScope(*this, "m_objectSpace.clearNewlyAllocated");
     583        m_objectSpace.clearNewlyAllocated();
     584    }
     585   
     586    {
     587        TimingScope clearMarksTimingScope(*this, "m_objectSpace.clearMarks");
     588        m_objectSpace.flip();
     589    }
    655590}
    656591
    … …  
    664599void Heap::visitSmallStrings()
    665600{
    666     GCPHASE(VisitSmallStrings);
    667601    if (!m_vm->smallStrings.needsToBeVisited(m_operationInProgress))
    668602        return;
    … …  
    676610void Heap::visitConservativeRoots(ConservativeRoots& roots)
    677611{
    678     GCPHASE(VisitConservativeRoots);
    679612    m_slotVisitor.append(roots);
    680613
    … …  
    699632{
    700633#if ENABLE(DFG_JIT)
    701     GCPHASE(FinalizeDFGWorklists);
    702634    for (auto worklist : m_suspendedCompilerWorklists)
    703635        worklist->removeDeadPlans(*m_vm);
    … …  
    733665void Heap::gatherExtraHeapSnapshotData(HeapProfiler& heapProfiler)
    734666{
    735     GCPHASE(GatherExtraHeapSnapshotData);
    736667    if (HeapSnapshotBuilder* builder = heapProfiler.activeSnapshotBuilder()) {
    737668        HeapIterationScope heapIterationScope(*this);
    … …  
    759690void Heap::removeDeadHeapSnapshotNodes(HeapProfiler& heapProfiler)
    760691{
    761     GCPHASE(RemoveDeadHeapSnapshotNodes);
    762692    if (HeapSnapshot* snapshot = heapProfiler.mostRecentSnapshot()) {
    763693        HeapIterationScope heapIterationScope(*this);
    … …  
    770700void Heap::visitProtectedObjects(HeapRootVisitor& heapRootVisitor)
    771701{
    772     GCPHASE(VisitProtectedObjects);
    773 
    774702    for (auto& pair : m_protectedValues)
    775703        heapRootVisitor.visit(&pair.key);
    … …  
    783711void Heap::visitArgumentBuffers(HeapRootVisitor& visitor)
    784712{
    785     GCPHASE(MarkingArgumentBuffers);
    786713    if (!m_markListSet || !m_markListSet->size())
    787714        return;
    … …  
    797724void Heap::visitException(HeapRootVisitor& visitor)
    798725{
    799     GCPHASE(MarkingException);
    800726    if (!m_vm->exception() && !m_vm->lastException())
    801727        return;
    … …  
    812738void Heap::visitStrongHandles(HeapRootVisitor& visitor)
    813739{
    814     GCPHASE(VisitStrongHandles);
    815740    m_handleSet.visitStrongHandles(visitor);
    816741
    … …  
    823748void Heap::visitHandleStack(HeapRootVisitor& visitor)
    824749{
    825     GCPHASE(VisitHandleStack);
    826750    m_handleStack.visit(visitor);
    827751
    … …  
    837761    if (SamplingProfiler* samplingProfiler = m_vm->samplingProfiler()) {
    838762        ASSERT(samplingProfiler->getLock().isLocked());
    839         GCPHASE(VisitSamplingProfiler);
    840763        samplingProfiler->visit(m_slotVisitor);
    841764        if (Options::logGC() == GCLogging::Verbose)
    … …  
    855778void Heap::traceCodeBlocksAndJITStubRoutines()
    856779{
    857     GCPHASE(TraceCodeBlocksAndJITStubRoutines);
    858     m_jitStubRoutines.traceMarkedStubRoutines(m_slotVisitor);
     780    m_jitStubRoutines->traceMarkedStubRoutines(m_slotVisitor);
    859781
    860782    if (Options::logGC() == GCLogging::Verbose)
    … …  
    866788void Heap::converge()
    867789{
    868     GCPHASE(Convergence);
    869790    m_slotVisitor.drainFromShared(SlotVisitor::MasterDrain);
    870791}
    … …  
    872793void Heap::visitWeakHandles(HeapRootVisitor& visitor)
    873794{
    874     GCPHASE(VisitingLiveWeakHandles);
     795    TimingScope timingScope(*this, "Heap::visitWeakHandles");
    875796    while (true) {
    876         m_objectSpace.visitWeakSets(visitor);
     797        {
     798            TimingScope timingScope(*this, "m_objectSpace.visitWeakSets");
     799            m_objectSpace.visitWeakSets(visitor);
     800        }
    877801        harvestWeakReferences();
    878802        visitCompilerWorklistWeakReferences();
    … …  
    893817void Heap::updateObjectCounts(double gcStartTime)
    894818{
    895     GCCOUNTER(VisitedValueCount, m_slotVisitor.visitCount() + threadVisitCount());
    896 
    897819    if (Options::logGC() == GCLogging::Verbose) {
    898820        size_t visitCount = m_slotVisitor.visitCount();
    … …  
    1034956void Heap::clearUnmarkedExecutables()
    1035957{
    1036     GCPHASE(ClearUnmarkedExecutables);
    1037958    for (unsigned i = m_executables.size(); i--;) {
    1038959        ExecutableBase* current = m_executables[i];
    … …  
    1052973void Heap::deleteUnmarkedCompiledCode()
    1053974{
    1054     GCPHASE(DeleteCodeBlocks);
    1055975    clearUnmarkedExecutables();
    1056     m_codeBlocks.deleteUnmarkedAndUnreferenced(m_operationInProgress);
    1057     m_jitStubRoutines.deleteUnmarkedJettisonedStubRoutines();
     976    m_codeBlocks->deleteUnmarkedAndUnreferenced(m_operationInProgress);
     977    m_jitStubRoutines->deleteUnmarkedJettisonedStubRoutines();
    1058978}
    1059979
    … …  
    1074994void Heap::collectAllGarbage()
    1075995{
     996    SuperSamplerScope superSamplerScope(false);
    1076997    if (!m_isSafeToCollect)
    1077998        return;
    1078999
    1079     collect(FullCollection);
     1000    collectWithoutAnySweep(FullCollection);
    10801001
    10811002    DeferGCForAWhile deferGC(*this);
    … …  
    10911012}
    10921013
    1093 NEVER_INLINE void Heap::collect(HeapOperation collectionType)
     1014void Heap::collect(HeapOperation collectionType)
     1015{
     1016    SuperSamplerScope superSamplerScope(false);
     1017    if (!m_isSafeToCollect)
     1018        return;
     1019
     1020    collectWithoutAnySweep(collectionType);
     1021}
     1022
     1023NEVER_INLINE void Heap::collectWithoutAnySweep(HeapOperation collectionType)
    10941024{
    10951025    void* stackTop;
    … …  
    11031033NEVER_INLINE void Heap::collectImpl(HeapOperation collectionType, void* stackOrigin, void* stackTop, MachineThreads::RegisterState& calleeSavedRegisters)
    11041034{
     1035    SuperSamplerScope superSamplerScope(false);
     1036    TimingScope collectImplTimingScope(collectionType, "Heap::collectImpl");
     1037   
    11051038#if ENABLE(ALLOCATION_LOGGING)
    11061039    dataLogF("JSC GC starting collection.\n");
    … …  
    11131046    }
    11141047   
    1115     if (vm()->typeProfiler()) {
    1116         DeferGCForAWhile awhile(*this);
    1117         vm()->typeProfilerLog()->processLogEntries(ASCIILiteral("GC"));
    1118     }
     1048    double gcStartTime;
     1049    {
     1050        TimingScope earlyTimingScope(collectionType, "Heap::collectImpl before markRoots");
     1051
     1052        if (vm()->typeProfiler()) {
     1053            DeferGCForAWhile awhile(*this);
     1054            vm()->typeProfilerLog()->processLogEntries(ASCIILiteral("GC"));
     1055        }
    11191056
    11201057#if ENABLE(JIT)
    1121     {
    1122         DeferGCForAWhile awhile(*this);
    1123         JITWorklist::instance()->completeAllForVM(*m_vm);
    1124     }
     1058        {
     1059            DeferGCForAWhile awhile(*this);
     1060            JITWorklist::instance()->completeAllForVM(*m_vm);
     1061        }
    11251062#endif // ENABLE(JIT)
    11261063
    1127     vm()->shadowChicken().update(*vm(), vm()->topCallFrame);
    1128 
    1129     RELEASE_ASSERT(!m_deferralDepth);
    1130     ASSERT(vm()->currentThreadIsHoldingAPILock());
    1131     RELEASE_ASSERT(vm()->atomicStringTable() == wtfThreadData().atomicStringTable());
    1132     ASSERT(m_isSafeToCollect);
    1133     RELEASE_ASSERT(m_operationInProgress == NoOperation);
    1134 
    1135     suspendCompilerThreads();
    1136     willStartCollection(collectionType);
    1137     GCPHASE(Collect);
    1138 
    1139     double gcStartTime = WTF::monotonicallyIncreasingTime();
    1140     if (m_verifier) {
    1141         // Verify that live objects from the last GC cycle haven't been corrupted by
    1142         // mutators before we begin this new GC cycle.
    1143         m_verifier->verify(HeapVerifier::Phase::BeforeGC);
    1144 
    1145         m_verifier->initializeGCCycle();
    1146         m_verifier->gatherLiveObjects(HeapVerifier::Phase::BeforeMarking);
    1147     }
    1148 
    1149     flushOldStructureIDTables();
    1150     stopAllocation();
    1151     flushWriteBarrierBuffer();
     1064        vm()->shadowChicken().update(*vm(), vm()->topCallFrame);
     1065
     1066        RELEASE_ASSERT(!m_deferralDepth);
     1067        ASSERT(vm()->currentThreadIsHoldingAPILock());
     1068        RELEASE_ASSERT(vm()->atomicStringTable() == wtfThreadData().atomicStringTable());
     1069        ASSERT(m_isSafeToCollect);
     1070        RELEASE_ASSERT(m_operationInProgress == NoOperation);
     1071
     1072        suspendCompilerThreads();
     1073        willStartCollection(collectionType);
     1074       
     1075        collectImplTimingScope.setOperation(*this);
     1076        earlyTimingScope.setOperation(*this);
     1077
     1078        gcStartTime = WTF::monotonicallyIncreasingTime();
     1079        if (m_verifier) {
     1080            // Verify that live objects from the last GC cycle haven't been corrupted by
     1081            // mutators before we begin this new GC cycle.
     1082            m_verifier->verify(HeapVerifier::Phase::BeforeGC);
     1083
     1084            m_verifier->initializeGCCycle();
     1085            m_verifier->gatherLiveObjects(HeapVerifier::Phase::BeforeMarking);
     1086        }
     1087
     1088        flushOldStructureIDTables();
     1089        stopAllocation();
     1090        prepareForMarking();
     1091        flushWriteBarrierBuffer();
     1092    }
    11521093
    11531094    markRoots(gcStartTime, stackOrigin, stackTop, calleeSavedRegisters);
     1095   
     1096    TimingScope lateTimingScope(*this, "Heap::collectImpl after markRoots");
    11541097
    11551098    if (m_verifier) {
    … …  
    11651108    sweepArrayBuffers();
    11661109    snapshotMarkedSpace();
    1167 
    11681110    copyBackingStores();
    1169 
    11701111    finalizeUnconditionalFinalizers();
    11711112    removeDeadCompilerWorklistEntries();
    … …  
    11801121    didFinishCollection(gcStartTime);
    11811122    resumeCompilerThreads();
    1182 
     1123    sweepLargeAllocations();
     1124   
    11831125    if (m_verifier) {
    11841126        m_verifier->trimDeadObjects();
    … …  
    11921134}
    11931135
     1136void Heap::sweepLargeAllocations()
     1137{
     1138    m_objectSpace.sweepLargeAllocations();
     1139}
     1140
    11941141void Heap::suspendCompilerThreads()
    11951142{
    11961143#if ENABLE(DFG_JIT)
    1197     GCPHASE(SuspendCompilerThreads);
    11981144    ASSERT(m_suspendedCompilerWorklists.isEmpty());
    11991145    for (unsigned i = DFG::numberOfWorklists(); i--;) {
    … …  
    12081154void Heap::willStartCollection(HeapOperation collectionType)
    12091155{
    1210     GCPHASE(StartingCollection);
    1211    
    12121156    if (Options::logGC())
    12131157        dataLog("=> ");
    … …  
    12471191void Heap::flushOldStructureIDTables()
    12481192{
    1249     GCPHASE(FlushOldStructureIDTables);
    12501193    m_structureIDTable.flushOldTables();
    12511194}
    … …  
    12531196void Heap::flushWriteBarrierBuffer()
    12541197{
    1255     GCPHASE(FlushWriteBarrierBuffer);
    12561198    if (m_operationInProgress == EdenCollection) {
    12571199        m_writeBarrierBuffer.flush(*this);
    … …  
    12631205void Heap::stopAllocation()
    12641206{
    1265     GCPHASE(StopAllocation);
    12661207    m_objectSpace.stopAllocating();
    12671208    if (m_operationInProgress == FullCollection)
    … …  
    12691210}
    12701211
     1212void Heap::prepareForMarking()
     1213{
     1214    m_objectSpace.prepareForMarking();
     1215}
     1216
    12711217void Heap::reapWeakHandles()
    12721218{
    1273     GCPHASE(ReapingWeakHandles);
    12741219    m_objectSpace.reapWeakSets();
    12751220}
    … …  
    12771222void Heap::pruneStaleEntriesFromWeakGCMaps()
    12781223{
    1279     GCPHASE(PruningStaleEntriesFromWeakGCMaps);
    12801224    if (m_operationInProgress != FullCollection)
    12811225        return;
    … …  
    12861230void Heap::sweepArrayBuffers()
    12871231{
    1288     GCPHASE(SweepingArrayBuffers);
    12891232    m_arrayBuffers.sweep();
    12901233}
    12911234
    12921235struct MarkedBlockSnapshotFunctor : public MarkedBlock::VoidFunctor {
    1293     MarkedBlockSnapshotFunctor(Vector<MarkedBlock*>& blocks)
     1236    MarkedBlockSnapshotFunctor(Vector<MarkedBlock::Handle*>& blocks)
    12941237        : m_index(0)
    12951238        , m_blocks(blocks)
    … …  
    12971240    }
    12981241
    1299     void operator()(MarkedBlock* block) const { m_blocks[m_index++] = block; }
     1242    void operator()(MarkedBlock::Handle* block) const
     1243    {
     1244        block->setIsOnBlocksToSweep(true);
     1245        m_blocks[m_index++] = block;
     1246    }
    13001247
    13011248    // FIXME: This is a mutable field becaue this isn't a C++ lambda.
    13021249    // https://bugs.webkit.org/show_bug.cgi?id=159644
    13031250    mutable size_t m_index;
    1304     Vector<MarkedBlock*>& m_blocks;
     1251    Vector<MarkedBlock::Handle*>& m_blocks;
    13051252};
    13061253
    13071254void Heap::snapshotMarkedSpace()
    13081255{
    1309     GCPHASE(SnapshotMarkedSpace);
    1310 
     1256    TimingScope timingScope(*this, "Heap::snapshotMarkedSpace");
     1257    // FIXME: This should probably be renamed. It's not actually snapshotting all of MarkedSpace.
     1258    // This is used by IncrementalSweeper, so it only needs to snapshot blocks. However, if we ever
     1259    // wanted to add other snapshotting login, we'd probably put it here.
     1260   
    13111261    if (m_operationInProgress == EdenCollection) {
    1312         m_blockSnapshot.appendVector(m_objectSpace.blocksWithNewObjects());
    1313         // Sort and deduplicate the block snapshot since we might be appending to an unfinished work list.
    1314         std::sort(m_blockSnapshot.begin(), m_blockSnapshot.end());
    1315         m_blockSnapshot.shrink(std::unique(m_blockSnapshot.begin(), m_blockSnapshot.end()) - m_blockSnapshot.begin());
     1262        for (MarkedBlock::Handle* handle : m_objectSpace.blocksWithNewObjects()) {
     1263            if (handle->isOnBlocksToSweep())
     1264                continue;
     1265            m_blockSnapshot.append(handle);
     1266            handle->setIsOnBlocksToSweep(true);
     1267        }
    13161268    } else {
    13171269        m_blockSnapshot.resizeToFit(m_objectSpace.blocks().set().size());
    … …  
    13231275void Heap::deleteSourceProviderCaches()
    13241276{
    1325     GCPHASE(DeleteSourceProviderCaches);
    13261277    m_vm->clearSourceProviderCaches();
    13271278}
    … …  
    13291280void Heap::notifyIncrementalSweeper()
    13301281{
    1331     GCPHASE(NotifyIncrementalSweeper);
    1332 
    13331282    if (m_operationInProgress == FullCollection) {
    13341283        if (!m_logicallyEmptyWeakBlocks.isEmpty())
    … …  
    13411290void Heap::writeBarrierCurrentlyExecutingCodeBlocks()
    13421291{
    1343     GCPHASE(WriteBarrierCurrentlyExecutingCodeBlocks);
    1344     m_codeBlocks.writeBarrierCurrentlyExecutingCodeBlocks(this);
     1292    m_codeBlocks->writeBarrierCurrentlyExecutingCodeBlocks(this);
    13451293}
    13461294
    13471295void Heap::resetAllocators()
    13481296{
    1349     GCPHASE(ResetAllocators);
    13501297    m_objectSpace.resetAllocators();
    13511298}
    … …  
    13531300void Heap::updateAllocationLimits()
    13541301{
    1355     GCPHASE(UpdateAllocationLimits);
     1302    static const bool verbose = false;
     1303   
     1304    if (verbose) {
     1305        dataLog("\n");
     1306        dataLog("bytesAllocatedThisCycle = ", m_bytesAllocatedThisCycle, "\n");
     1307    }
    13561308   
    13571309    // Calculate our current heap size threshold for the purpose of figuring out when we should
    … …  
    13701322    // cells usually have a narrow range of sizes. So, the underestimation is probably OK.
    13711323    currentHeapSize += m_totalBytesVisited;
     1324    if (verbose)
     1325        dataLog("totalBytesVisited = ", m_totalBytesVisited, ", currentHeapSize = ", currentHeapSize, "\n");
    13721326
    13731327    // For copied space, we use the capacity of storage space. This is because copied space may get
    … …  
    13851339    ASSERT(m_totalBytesCopied <= m_storageSpace.size());
    13861340    currentHeapSize += m_storageSpace.capacity();
     1341    if (verbose)
     1342        dataLog("storageSpace.capacity() = ", m_storageSpace.capacity(), ", currentHeapSize = ", currentHeapSize, "\n");
    13871343
    13881344    // It's up to the user to ensure that extraMemorySize() ends up corresponding to allocation-time
    13891345    // extra memory reporting.
    13901346    currentHeapSize += extraMemorySize();
     1347
     1348    if (verbose)
     1349        dataLog("extraMemorySize() = ", extraMemorySize(), ", currentHeapSize = ", currentHeapSize, "\n");
    13911350   
    13921351    if (Options::gcMaxHeapSize() && currentHeapSize > Options::gcMaxHeapSize())
    … …  
    13981357        // fixed minimum.
    13991358        m_maxHeapSize = max(minHeapSize(m_heapType, m_ramSize), proportionalHeapSize(currentHeapSize, m_ramSize));
     1359        if (verbose)
     1360            dataLog("Full: maxHeapSize = ", m_maxHeapSize, "\n");
    14001361        m_maxEdenSize = m_maxHeapSize - currentHeapSize;
     1362        if (verbose)
     1363            dataLog("Full: maxEdenSize = ", m_maxEdenSize, "\n");
    14011364        m_sizeAfterLastFullCollect = currentHeapSize;
     1365        if (verbose)
     1366            dataLog("Full: sizeAfterLastFullCollect = ", currentHeapSize, "\n");
    14021367        m_bytesAbandonedSinceLastFullCollect = 0;
     1368        if (verbose)
     1369            dataLog("Full: bytesAbandonedSinceLastFullCollect = ", 0, "\n");
    14031370    } else {
    1404         static const bool verbose = false;
    1405        
    14061371        ASSERT(currentHeapSize >= m_sizeAfterLastCollect);
    1407         m_maxEdenSize = m_maxHeapSize - currentHeapSize;
     1372        // Theoretically, we shouldn't ever scan more memory than the heap size we planned to have.
     1373        // But we are sloppy, so we have to defend against the overflow.
     1374        m_maxEdenSize = currentHeapSize > m_maxHeapSize ? 0 : m_maxHeapSize - currentHeapSize;
     1375        if (verbose)
     1376            dataLog("Eden: maxEdenSize = ", m_maxEdenSize, "\n");
    14081377        m_sizeAfterLastEdenCollect = currentHeapSize;
    1409         if (verbose) {
    1410             dataLog("Max heap size: ", m_maxHeapSize, "\n");
    1411             dataLog("Current heap size: ", currentHeapSize, "\n");
    1412             dataLog("Size after last eden collection: ", m_sizeAfterLastEdenCollect, "\n");
    1413         }
     1378        if (verbose)
     1379            dataLog("Eden: sizeAfterLastEdenCollect = ", currentHeapSize, "\n");
    14141380        double edenToOldGenerationRatio = (double)m_maxEdenSize / (double)m_maxHeapSize;
    1415         if (verbose)
    1416             dataLog("Eden to old generation ratio: ", edenToOldGenerationRatio, "\n");
    14171381        double minEdenToOldGenerationRatio = 1.0 / 3.0;
    14181382        if (edenToOldGenerationRatio < minEdenToOldGenerationRatio)
    … …  
    14201384        // This seems suspect at first, but what it does is ensure that the nursery size is fixed.
    14211385        m_maxHeapSize += currentHeapSize - m_sizeAfterLastCollect;
     1386        if (verbose)
     1387            dataLog("Eden: maxHeapSize = ", m_maxHeapSize, "\n");
    14221388        m_maxEdenSize = m_maxHeapSize - currentHeapSize;
     1389        if (verbose)
     1390            dataLog("Eden: maxEdenSize = ", m_maxEdenSize, "\n");
    14231391        if (m_fullActivityCallback) {
    14241392            ASSERT(currentHeapSize >= m_sizeAfterLastFullCollect);
    … …  
    14281396
    14291397    m_sizeAfterLastCollect = currentHeapSize;
     1398    if (verbose)
     1399        dataLog("sizeAfterLastCollect = ", m_sizeAfterLastCollect, "\n");
    14301400    m_bytesAllocatedThisCycle = 0;
    14311401
    … …  
    14361406void Heap::didFinishCollection(double gcStartTime)
    14371407{
    1438     GCPHASE(FinishingCollection);
    14391408    double gcEndTime = WTF::monotonicallyIncreasingTime();
    14401409    HeapOperation operation = m_operationInProgress;
    … …  
    14721441{
    14731442#if ENABLE(DFG_JIT)
    1474     GCPHASE(ResumeCompilerThreads);
    14751443    for (auto worklist : m_suspendedCompilerWorklists)
    14761444        worklist->resumeAllThreads();
    … …  
    15811549            current++;
    15821550
    1583         void* limit = static_cast<void*>(reinterpret_cast<char*>(cell) + MarkedBlock::blockFor(cell)->cellSize());
     1551        void* limit = static_cast<void*>(reinterpret_cast<char*>(cell) + cell->cellSize());
    15841552        for (; current < limit; current++)
    15851553            *current = zombifiedBits;
    … …  
    16871655}
    16881656
     1657void Heap::forEachCodeBlockImpl(const ScopedLambda<bool(CodeBlock*)>& func)
     1658{
     1659    // We don't know the full set of CodeBlocks until compilation has terminated.
     1660    completeAllJITPlans();
     1661
     1662    return m_codeBlocks->iterate(func);
     1663}
     1664
    16891665} // namespace JSC

  • trunk/Source/JavaScriptCore/heap/Heap.h

    r204912 r205462  
    2424
    2525#include "ArrayBuffer.h"
    26 #include "CodeBlockSet.h"
    2726#include "CopyVisitor.h"
    2827#include "GCIncomingRefCountedSet.h"
    … …  
    3130#include "HeapObserver.h"
    3231#include "HeapOperation.h"
    33 #include "JITStubRoutineSet.h"
    3432#include "ListableHandler.h"
    3533#include "MachineStackMarker.h"
    … …  
    5452
    5553class CodeBlock;
     54class CodeBlockSet;
    5655class CopiedSpace;
    5756class EdenGCActivityCallback;
    … …  
    6665class IncrementalSweeper;
    6766class JITStubRoutine;
     67class JITStubRoutineSet;
    6868class JSCell;
    6969class JSValue;
    … …  
    8383
    8484enum HeapType { SmallHeap, LargeHeap };
     85
     86class HeapUtil;
    8587
    8688class Heap {
    … …  
    9092    friend class DFG::SpeculativeJIT;
    9193    static Heap* heap(const JSValue); // 0 for immediate values
    92     static Heap* heap(const JSCell*);
     94    static Heap* heap(const HeapCell*);
    9395
    9496    // This constant determines how many blocks we iterate between checks of our
    … …  
    100102    static bool isLive(const void*);
    101103    static bool isMarked(const void*);
    102     static bool testAndSetMarked(const void*);
     104    static bool testAndSetMarked(int64_t, const void*);
    103105    static void setMarked(const void*);
    104 
    105     // This function must be run after stopAllocation() is called and
    106     // before liveness data is cleared to be accurate.
    107     static bool isPointerGCObject(TinyBloomFilter, MarkedBlockSet&, void* pointer);
    108     static bool isValueGCObject(TinyBloomFilter, MarkedBlockSet&, JSValue);
     106   
     107    static size_t cellSize(const void*);
    109108
    110109    void writeBarrier(const JSCell*);
    … …  
    148147    MarkedSpace::Subspace& subspaceForAuxiliaryData() { return m_objectSpace.subspaceForAuxiliaryData(); }
    149148    template<typename ClassType> MarkedSpace::Subspace& subspaceForObjectOfType();
    150     MarkedAllocator& allocatorForObjectWithoutDestructor(size_t bytes) { return m_objectSpace.allocatorFor(bytes); }
    151     MarkedAllocator& allocatorForObjectWithDestructor(size_t bytes) { return m_objectSpace.destructorAllocatorFor(bytes); }
    152     template<typename ClassType> MarkedAllocator& allocatorForObjectOfType(size_t bytes);
     149    MarkedAllocator* allocatorForObjectWithoutDestructor(size_t bytes) { return m_objectSpace.allocatorFor(bytes); }
     150    MarkedAllocator* allocatorForObjectWithDestructor(size_t bytes) { return m_objectSpace.destructorAllocatorFor(bytes); }
     151    template<typename ClassType> MarkedAllocator* allocatorForObjectOfType(size_t bytes);
     152    MarkedAllocator* allocatorForAuxiliaryData(size_t bytes) { return m_objectSpace.auxiliaryAllocatorFor(bytes); }
    153153    CopiedAllocator& storageAllocator() { return m_storageSpace.allocator(); }
     154    void* allocateAuxiliary(JSCell* intendedOwner, size_t);
     155    void* tryAllocateAuxiliary(JSCell* intendedOwner, size_t);
     156    void* tryReallocateAuxiliary(JSCell* intendedOwner, void* oldBase, size_t oldSize, size_t newSize);
    154157    CheckedBoolean tryAllocateStorage(JSCell* intendedOwner, size_t, void**);
    155158    CheckedBoolean tryReallocateStorage(JSCell* intendedOwner, void**, size_t, size_t);
    … …  
    231234    bool isPagedOut(double deadline);
    232235   
    233     const JITStubRoutineSet& jitStubRoutines() { return m_jitStubRoutines; }
     236    const JITStubRoutineSet& jitStubRoutines() { return *m_jitStubRoutines; }
    234237   
    235238    void addReference(JSCell*, ArrayBuffer*);
    … …  
    239242    StructureIDTable& structureIDTable() { return m_structureIDTable; }
    240243
    241     CodeBlockSet& codeBlockSet() { return m_codeBlocks; }
     244    CodeBlockSet& codeBlockSet() { return *m_codeBlocks; }
    242245
    243246#if USE(FOUNDATION)
    … …  
    268271    friend class GCThread;
    269272    friend class HandleSet;
     273    friend class HeapUtil;
    270274    friend class HeapVerifier;
    271275    friend class JITStubRoutine;
    … …  
    284288    template<typename T> friend void* allocateCell(Heap&, size_t);
    285289
     290    void collectWithoutAnySweep(HeapOperation collectionType = AnyCollection);
     291
    286292    void* allocateWithDestructor(size_t); // For use with objects with destructors.
    287293    void* allocateWithoutDestructor(size_t); // For use with objects without destructors.
    … …  
    305311    void flushWriteBarrierBuffer();
    306312    void stopAllocation();
     313    void prepareForMarking();
    307314   
    308315    void markRoots(double gcStartTime, void* stackOrigin, void* stackTop, MachineThreads::RegisterState&);
    … …  
    349356    void gatherExtraHeapSnapshotData(HeapProfiler&);
    350357    void removeDeadHeapSnapshotNodes(HeapProfiler&);
    351 
     358    void sweepLargeAllocations();
     359   
    352360    void sweepAllLogicallyEmptyWeakBlocks();
    353361    bool sweepNextLogicallyEmptyWeakBlock();
    … …  
    362370    size_t threadBytesVisited();
    363371    size_t threadBytesCopied();
     372   
     373    void forEachCodeBlockImpl(const ScopedLambda<bool(CodeBlock*)>&);
    364374
    365375    const HeapType m_heapType;
    … …  
    409419    HandleSet m_handleSet;
    410420    HandleStack m_handleStack;
    411     CodeBlockSet m_codeBlocks;
    412     JITStubRoutineSet m_jitStubRoutines;
     421    std::unique_ptr<CodeBlockSet> m_codeBlocks;
     422    std::unique_ptr<JITStubRoutineSet> m_jitStubRoutines;
    413423    FinalizerOwner m_finalizerOwner;
    414424   
    … …  
    429439    RefPtr<GCActivityCallback> m_edenActivityCallback;
    430440    std::unique_ptr<IncrementalSweeper> m_sweeper;
    431     Vector<MarkedBlock*> m_blockSnapshot;
     441    Vector<MarkedBlock::Handle*> m_blockSnapshot;
    432442
    433443    Vector<HeapObserver*> m_observers;

  • trunk/Source/JavaScriptCore/heap/HeapCell.h

    r204912 r205462  
    2626#pragma once
    2727
     28#include "DestructionMode.h"
     29
    2830namespace JSC {
     31
     32class CellContainer;
     33class Heap;
     34class LargeAllocation;
     35class MarkedBlock;
     36class VM;
     37struct AllocatorAttributes;
    2938
    3039class HeapCell {
    … …  
    3948    void zap() { *reinterpret_cast<uintptr_t**>(this) = 0; }
    4049    bool isZapped() const { return !*reinterpret_cast<uintptr_t* const*>(this); }
     50   
     51    bool isLargeAllocation() const;
     52    CellContainer cellContainer() const;
     53    MarkedBlock& markedBlock() const;
     54    LargeAllocation& largeAllocation() const;
     55
     56    // If you want performance and you know that your cell is small, you can do this instead:
     57    // ASSERT(!cell->isLargeAllocation());
     58    // cell->markedBlock().vm()
     59    // We currently only use this hack for callees to make ExecState::vm() fast. It's not
     60    // recommended to use it for too many other things, since the large allocation cutoff is
     61    // a runtime option and its default value is small (400 bytes).
     62    Heap* heap() const;
     63    VM* vm() const;
     64   
     65    size_t cellSize() const;
     66    AllocatorAttributes allocatorAttributes() const;
     67    DestructionMode destructionMode() const;
     68    Kind cellKind() const;
    4169};
    4270

  • trunk/Source/JavaScriptCore/heap/HeapInlines.h

    r204912 r205462  
    2929#include "CopyBarrier.h"
    3030#include "Heap.h"
     31#include "HeapCellInlines.h"
     32#include "IndexingHeader.h"
     33#include "JSCallee.h"
    3134#include "JSCell.h"
    3235#include "Structure.h"
    … …  
    6063}
    6164
    62 inline Heap* Heap::heap(const JSCell* cell)
    63 {
    64     return MarkedBlock::blockFor(cell)->heap();
     65ALWAYS_INLINE Heap* Heap::heap(const HeapCell* cell)
     66{
     67    return cell->heap();
    6568}
    6669
    … …  
    7275}
    7376
    74 inline bool Heap::isLive(const void* cell)
    75 {
    76     return MarkedBlock::blockFor(cell)->isLiveCell(cell);
    77 }
    78 
    79 inline bool Heap::isMarked(const void* cell)
    80 {
    81     return MarkedBlock::blockFor(cell)->isMarked(cell);
    82 }
    83 
    84 inline bool Heap::testAndSetMarked(const void* cell)
    85 {
    86     return MarkedBlock::blockFor(cell)->testAndSetMarked(cell);
    87 }
    88 
    89 inline void Heap::setMarked(const void* cell)
    90 {
    91     MarkedBlock::blockFor(cell)->setMarked(cell);
     77inline bool Heap::isLive(const void* rawCell)
     78{
     79    HeapCell* cell = bitwise_cast<HeapCell*>(rawCell);
     80    if (cell->isLargeAllocation())
     81        return cell->largeAllocation().isLive();
     82    MarkedBlock& block = cell->markedBlock();
     83    block.flipIfNecessary(block.vm()->heap.objectSpace().version());
     84    return block.handle().isLiveCell(cell);
     85}
     86
     87ALWAYS_INLINE bool Heap::isMarked(const void* rawCell)
     88{
     89    HeapCell* cell = bitwise_cast<HeapCell*>(rawCell);
     90    if (cell->isLargeAllocation())
     91        return cell->largeAllocation().isMarked();
     92    MarkedBlock& block = cell->markedBlock();
     93    block.flipIfNecessary(block.vm()->heap.objectSpace().version());
     94    return block.isMarked(cell);
     95}
     96
     97ALWAYS_INLINE bool Heap::testAndSetMarked(int64_t version, const void* rawCell)
     98{
     99    HeapCell* cell = bitwise_cast<HeapCell*>(rawCell);
     100    if (cell->isLargeAllocation())
     101        return cell->largeAllocation().testAndSetMarked();
     102    MarkedBlock& block = cell->markedBlock();
     103    block.flipIfNecessaryConcurrently(version);
     104    return block.testAndSetMarked(cell);
     105}
     106
     107inline void Heap::setMarked(const void* rawCell)
     108{
     109    HeapCell* cell = bitwise_cast<HeapCell*>(rawCell);
     110    if (cell->isLargeAllocation()) {
     111        cell->largeAllocation().setMarked();
     112        return;
     113    }
     114    MarkedBlock& block = cell->markedBlock();
     115    block.flipIfNecessary(block.vm()->heap.objectSpace().version());
     116    block.setMarked(cell);
     117}
     118
     119ALWAYS_INLINE size_t Heap::cellSize(const void* rawCell)
     120{
     121    return bitwise_cast<HeapCell*>(rawCell)->cellSize();
    92122}
    93123
    … …  
    166196}
    167197
    168 template<typename Functor> inline void Heap::forEachCodeBlock(const Functor& functor)
    169 {
    170     // We don't know the full set of CodeBlocks until compilation has terminated.
    171     completeAllJITPlans();
    172 
    173     return m_codeBlocks.iterate<Functor>(functor);
     198template<typename Functor> inline void Heap::forEachCodeBlock(const Functor& func)
     199{
     200    forEachCodeBlockImpl(scopedLambdaRef<bool(CodeBlock*)>(func));
    174201}
    175202
    … …  
    200227
    201228template<typename ClassType>
    202 void* Heap::allocateObjectOfType(size_t bytes)
     229inline void* Heap::allocateObjectOfType(size_t bytes)
    203230{
    204231    // JSCell::classInfo() expects objects allocated with normal destructor to derive from JSDestructibleObject.
    … …  
    211238
    212239template<typename ClassType>
    213 MarkedSpace::Subspace& Heap::subspaceForObjectOfType()
     240inline MarkedSpace::Subspace& Heap::subspaceForObjectOfType()
    214241{
    215242    // JSCell::classInfo() expects objects allocated with normal destructor to derive from JSDestructibleObject.
    … …  
    222249
    223250template<typename ClassType>
    224 MarkedAllocator& Heap::allocatorForObjectOfType(size_t bytes)
     251inline MarkedAllocator* Heap::allocatorForObjectOfType(size_t bytes)
    225252{
    226253    // JSCell::classInfo() expects objects allocated with normal destructor to derive from JSDestructibleObject.
    227254    ASSERT((!ClassType::needsDestruction || (ClassType::StructureFlags & StructureIsImmortal) || std::is_convertible<ClassType, JSDestructibleObject>::value));
     255
     256    MarkedAllocator* result;
     257    if (ClassType::needsDestruction)
     258        result = allocatorForObjectWithDestructor(bytes);
     259    else
     260        result = allocatorForObjectWithoutDestructor(bytes);
    228261   
    229     if (ClassType::needsDestruction)
    230         return allocatorForObjectWithDestructor(bytes);
    231     return allocatorForObjectWithoutDestructor(bytes);
     262    ASSERT(result || !ClassType::info()->isSubClassOf(JSCallee::info()));
     263    return result;
     264}
     265
     266inline void* Heap::allocateAuxiliary(JSCell* intendedOwner, size_t bytes)
     267{
     268    void* result = m_objectSpace.allocateAuxiliary(bytes);
     269#if ENABLE(ALLOCATION_LOGGING)
     270    dataLogF("JSC GC allocating %lu bytes of auxiliary for %p: %p.\n", bytes, intendedOwner, result);
     271#else
     272    UNUSED_PARAM(intendedOwner);
     273#endif
     274    return result;
     275}
     276
     277inline void* Heap::tryAllocateAuxiliary(JSCell* intendedOwner, size_t bytes)
     278{
     279    void* result = m_objectSpace.tryAllocateAuxiliary(bytes);
     280#if ENABLE(ALLOCATION_LOGGING)
     281    dataLogF("JSC GC allocating %lu bytes of auxiliary for %p: %p.\n", bytes, intendedOwner, result);
     282#else
     283    UNUSED_PARAM(intendedOwner);
     284#endif
     285    return result;
     286}
     287
     288inline void* Heap::tryReallocateAuxiliary(JSCell* intendedOwner, void* oldBase, size_t oldSize, size_t newSize)
     289{
     290    void* newBase = tryAllocateAuxiliary(intendedOwner, newSize);
     291    if (!newBase)
     292        return nullptr;
     293    memcpy(newBase, oldBase, oldSize);
     294    return newBase;
    232295}
    233296
    … …  
    355418}
    356419
    357 inline bool Heap::isPointerGCObject(TinyBloomFilter filter, MarkedBlockSet& markedBlockSet, void* pointer)
    358 {
    359     MarkedBlock* candidate = MarkedBlock::blockFor(pointer);
    360     if (filter.ruleOut(bitwise_cast<Bits>(candidate))) {
    361         ASSERT(!candidate || !markedBlockSet.set().contains(candidate));
    362         return false;
    363     }
    364 
    365     if (!MarkedBlock::isAtomAligned(pointer))
    366         return false;
    367 
    368     if (!markedBlockSet.set().contains(candidate))
    369         return false;
    370 
    371     if (!candidate->isLiveCell(pointer))
    372         return false;
    373 
    374     return true;
    375 }
    376 
    377 inline bool Heap::isValueGCObject(TinyBloomFilter filter, MarkedBlockSet& markedBlockSet, JSValue value)
    378 {
    379     if (!value.isCell())
    380         return false;
    381     return isPointerGCObject(filter, markedBlockSet, static_cast<void*>(value.asCell()));
    382 }
    383 
    384420} // namespace JSC
    385421

  • trunk/Source/JavaScriptCore/heap/HeapOperation.h

    r165940 r205462  
    3333} // namespace JSC
    3434
     35namespace WTF {
     36
     37class PrintStream;
     38
     39void printInternal(PrintStream& out, JSC::HeapOperation);
     40
     41} // namespace WTF
     42
    3543#endif // HeapOperation_h

  • trunk/Source/JavaScriptCore/heap/IncrementalSweeper.cpp

    r204466 r205462  
    11/*
    2  * Copyright (C) 2012 Apple Inc. All rights reserved.
     2 * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    129129{
    130130    while (!m_blocksToSweep.isEmpty()) {
    131         MarkedBlock* block = m_blocksToSweep.takeLast();
     131        MarkedBlock::Handle* block = m_blocksToSweep.takeLast();
     132        block->setIsOnBlocksToSweep(false);
    132133
    133134        if (!block->needsSweeping())

  • trunk/Source/JavaScriptCore/heap/IncrementalSweeper.h

    r197563 r205462  
    11/*
    2  * Copyright (C) 2012 Apple Inc. All rights reserved.
     2 * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2828
    2929#include "HeapTimer.h"
     30#include "MarkedBlock.h"
    3031#include <wtf/Vector.h>
    3132
    … …  
    5657    void cancelTimer();
    5758   
    58     Vector<MarkedBlock*>& m_blocksToSweep;
     59    Vector<MarkedBlock::Handle*>& m_blocksToSweep;
    5960#endif
    6061};

  • trunk/Source/JavaScriptCore/heap/MarkedAllocator.cpp

    r204912 r205462  
    3131#include "IncrementalSweeper.h"
    3232#include "JSCInlines.h"
     33#include "SuperSampler.h"
    3334#include "VM.h"
    3435#include <wtf/CurrentTime.h>
    … …  
    3637namespace JSC {
    3738
    38 static bool isListPagedOut(double deadline, DoublyLinkedList<MarkedBlock>& list)
     39MarkedAllocator::MarkedAllocator(Heap* heap, MarkedSpace* markedSpace, size_t cellSize, const AllocatorAttributes& attributes)
     40    : m_currentBlock(0)
     41    , m_lastActiveBlock(0)
     42    , m_nextBlockToSweep(nullptr)
     43    , m_cellSize(static_cast<unsigned>(cellSize))
     44    , m_attributes(attributes)
     45    , m_heap(heap)
     46    , m_markedSpace(markedSpace)
     47{
     48}
     49
     50bool MarkedAllocator::isPagedOut(double deadline)
    3951{
    4052    unsigned itersSinceLastTimeCheck = 0;
    41     MarkedBlock* block = list.head();
     53    MarkedBlock::Handle* block = m_blockList.begin();
    4254    while (block) {
    43         block = block->next();
     55        block = filterNextBlock(block->next());
     56        if (block)
     57            block->flipIfNecessary(); // Forces us to touch the memory of the block, but has no semantic effect.
    4458        ++itersSinceLastTimeCheck;
    4559        if (itersSinceLastTimeCheck >= Heap::s_timeCheckResolution) {
    … …  
    5367}
    5468
    55 bool MarkedAllocator::isPagedOut(double deadline)
    56 {
    57     if (isListPagedOut(deadline, m_blockList))
    58         return true;
    59     return false;
    60 }
    61 
    62 void MarkedAllocator::retire(MarkedBlock* block, MarkedBlock::FreeList& freeList)
    63 {
    64     m_blockList.remove(block);
     69void MarkedAllocator::retire(MarkedBlock::Handle* block)
     70{
     71    LockHolder locker(m_lock); // This will be called in parallel during GC.
     72    if (block == m_currentBlock) {
     73        // This happens when the mutator is running. We finished a full GC and marked too few things
     74        // to retire. Then we started allocating in this block. Then a barrier ran, which marked an
     75        // object in this block, which put it over the retirement threshold. It's OK to simply do
     76        // nothing in that case.
     77        return;
     78    }
     79    if (block == m_lastActiveBlock) {
     80        // This can easily happen during marking. It would be easy to handle this case, but it's
     81        // just as easy to ignore it.
     82        return;
     83    }
     84    RELEASE_ASSERT(block->isOnList());
     85    if (block == m_nextBlockToSweep)
     86        m_nextBlockToSweep = filterNextBlock(block->next());
     87    block->remove();
    6588    m_retiredBlocks.push(block);
    66     block->didRetireBlock(freeList);
    67 }
    68 
    69 inline void* MarkedAllocator::tryAllocateHelper(size_t bytes)
    70 {
     89}
     90
     91MarkedBlock::Handle* MarkedAllocator::filterNextBlock(MarkedBlock::Handle* block)
     92{
     93    if (block == m_blockList.end())
     94        return nullptr;
     95    return block;
     96}
     97
     98void MarkedAllocator::setNextBlockToSweep(MarkedBlock::Handle* block)
     99{
     100    m_nextBlockToSweep = filterNextBlock(block);
     101}
     102
     103void* MarkedAllocator::tryAllocateWithoutCollectingImpl()
     104{
     105    SuperSamplerScope superSamplerScope(false);
     106   
    71107    if (m_currentBlock) {
    72108        ASSERT(m_currentBlock == m_nextBlockToSweep);
    73109        m_currentBlock->didConsumeFreeList();
    74         m_nextBlockToSweep = m_currentBlock->next();
    75     }
    76 
    77     MarkedBlock* next;
    78     for (MarkedBlock*& block = m_nextBlockToSweep; block; block = next) {
    79         next = block->next();
    80 
    81         MarkedBlock::FreeList freeList = block->sweep(MarkedBlock::SweepToFreeList);
     110        setNextBlockToSweep(m_currentBlock->next());
     111    }
     112   
     113    setFreeList(FreeList());
     114
     115    RELEASE_ASSERT(m_nextBlockToSweep != m_blockList.end());
     116
     117    MarkedBlock::Handle* next;
     118    for (MarkedBlock::Handle*& block = m_nextBlockToSweep; block; block = next) {
     119        next = filterNextBlock(block->next());
     120
     121        // It would be super weird if the blocks we are sweeping have anything allocated during this
     122        // cycle.
     123        ASSERT(!block->hasAnyNewlyAllocated());
    82124       
    83         double utilization = ((double)MarkedBlock::blockSize - (double)freeList.bytes) / (double)MarkedBlock::blockSize;
    84         if (utilization >= Options::minMarkedBlockUtilization()) {
    85             ASSERT(freeList.bytes || !freeList.head);
    86             retire(block, freeList);
     125        FreeList freeList = block->sweep(MarkedBlock::Handle::SweepToFreeList);
     126       
     127        // It's possible to stumble on a complete-full block. Marking tries to retire these, but
     128        // that algorithm is racy and may forget to do it sometimes.
     129        if (freeList.allocationWillFail()) {
     130            ASSERT(block->isFreeListed());
     131            block->unsweepWithNoNewlyAllocated();
     132            ASSERT(block->isMarked());
     133            retire(block);
    87134            continue;
    88135        }
    89136
    90         if (bytes > block->cellSize()) {
    91             block->stopAllocating(freeList);
    92             continue;
    93         }
    94 
    95137        m_currentBlock = block;
    96         m_freeList = freeList;
     138        setFreeList(freeList);
    97139        break;
    98140    }
    99141   
    100     if (!m_freeList.head) {
     142    if (!m_freeList) {
    101143        m_currentBlock = 0;
    102144        return 0;
    103145    }
    104146
    105     ASSERT(m_freeList.head);
    106     void* head = tryPopFreeList(bytes);
    107     ASSERT(head);
     147    void* result;
     148    if (m_freeList.remaining) {
     149        unsigned cellSize = m_cellSize;
     150        m_freeList.remaining -= cellSize;
     151        result = m_freeList.payloadEnd - m_freeList.remaining - cellSize;
     152    } else {
     153        FreeCell* head = m_freeList.head;
     154        m_freeList.head = head->next;
     155        result = head;
     156    }
     157    RELEASE_ASSERT(result);
    108158    m_markedSpace->didAllocateInBlock(m_currentBlock);
    109     return head;
    110 }
    111 
    112 inline void* MarkedAllocator::tryPopFreeList(size_t bytes)
    113 {
    114     ASSERT(m_currentBlock);
    115     if (bytes > m_currentBlock->cellSize())
    116         return 0;
    117 
    118     MarkedBlock::FreeCell* head = m_freeList.head;
    119     m_freeList.head = head->next;
    120     return head;
    121 }
    122 
    123 inline void* MarkedAllocator::tryAllocate(size_t bytes)
     159    return result;
     160}
     161
     162inline void* MarkedAllocator::tryAllocateWithoutCollecting()
    124163{
    125164    ASSERT(!m_heap->isBusy());
    126165    m_heap->m_operationInProgress = Allocation;
    127     void* result = tryAllocateHelper(bytes);
     166    void* result = tryAllocateWithoutCollectingImpl();
    128167
    129168    m_heap->m_operationInProgress = NoOperation;
    … …  
    147186}
    148187
    149 void* MarkedAllocator::allocateSlowCase(size_t bytes)
    150 {
     188void* MarkedAllocator::allocateSlowCase()
     189{
     190    bool crashOnFailure = true;
     191    return allocateSlowCaseImpl(crashOnFailure);
     192}
     193
     194void* MarkedAllocator::tryAllocateSlowCase()
     195{
     196    bool crashOnFailure = false;
     197    return allocateSlowCaseImpl(crashOnFailure);
     198}
     199
     200void* MarkedAllocator::allocateSlowCaseImpl(bool crashOnFailure)
     201{
     202    SuperSamplerScope superSamplerScope(false);
    151203    ASSERT(m_heap->vm()->currentThreadIsHoldingAPILock());
    152204    doTestCollectionsIfNeeded();
    153205
    154206    ASSERT(!m_markedSpace->isIterating());
    155     ASSERT(!m_freeList.head);
    156     m_heap->didAllocate(m_freeList.bytes);
    157    
    158     void* result = tryAllocate(bytes);
     207    m_heap->didAllocate(m_freeList.originalSize);
     208   
     209    void* result = tryAllocateWithoutCollecting();
    159210   
    160211    if (LIKELY(result != 0))
    … …  
    162213   
    163214    if (m_heap->collectIfNecessaryOrDefer()) {
    164         result = tryAllocate(bytes);
     215        result = tryAllocateWithoutCollecting();
    165216        if (result)
    166217            return result;
    … …  
    169220    ASSERT(!m_heap->shouldCollect());
    170221   
    171     MarkedBlock* block = allocateBlock(bytes);
    172     ASSERT(block);
     222    MarkedBlock::Handle* block = tryAllocateBlock();
     223    if (!block) {
     224        if (crashOnFailure)
     225            RELEASE_ASSERT_NOT_REACHED();
     226        else
     227            return nullptr;
     228    }
    173229    addBlock(block);
    174230       
    175     result = tryAllocate(bytes);
     231    result = tryAllocateWithoutCollecting();
    176232    ASSERT(result);
    177233    return result;
    178234}
    179235
    180 MarkedBlock* MarkedAllocator::allocateBlock(size_t bytes)
     236static size_t blockHeaderSize()
     237{
     238    return WTF::roundUpToMultipleOf<MarkedBlock::atomSize>(sizeof(MarkedBlock));
     239}
     240
     241size_t MarkedAllocator::blockSizeForBytes(size_t bytes)
    181242{
    182243    size_t minBlockSize = MarkedBlock::blockSize;
    183     size_t minAllocationSize = WTF::roundUpToMultipleOf<MarkedBlock::atomSize>(sizeof(MarkedBlock)) + WTF::roundUpToMultipleOf<MarkedBlock::atomSize>(bytes);
     244    size_t minAllocationSize = blockHeaderSize() + WTF::roundUpToMultipleOf<MarkedBlock::atomSize>(bytes);
    184245    minAllocationSize = WTF::roundUpToMultipleOf(WTF::pageSize(), minAllocationSize);
    185     size_t blockSize = std::max(minBlockSize, minAllocationSize);
    186 
    187     size_t cellSize = m_cellSize ? m_cellSize : WTF::roundUpToMultipleOf<MarkedBlock::atomSize>(bytes);
    188 
    189     return MarkedBlock::create(*m_heap, this, blockSize, cellSize, m_attributes);
    190 }
    191 
    192 void MarkedAllocator::addBlock(MarkedBlock* block)
     246    return std::max(minBlockSize, minAllocationSize);
     247}
     248
     249MarkedBlock::Handle* MarkedAllocator::tryAllocateBlock()
     250{
     251    SuperSamplerScope superSamplerScope(false);
     252    return MarkedBlock::tryCreate(*m_heap, this, m_cellSize, m_attributes);
     253}
     254
     255void MarkedAllocator::addBlock(MarkedBlock::Handle* block)
    193256{
    194257    ASSERT(!m_currentBlock);
    195     ASSERT(!m_freeList.head);
     258    ASSERT(!m_freeList);
    196259   
    197260    m_blockList.append(block);
    198     m_nextBlockToSweep = block;
     261    setNextBlockToSweep(block);
    199262    m_markedSpace->didAddBlock(block);
    200263}
    201264
    202 void MarkedAllocator::removeBlock(MarkedBlock* block)
     265void MarkedAllocator::removeBlock(MarkedBlock::Handle* block)
    203266{
    204267    if (m_currentBlock == block) {
    205         m_currentBlock = m_currentBlock->next();
    206         m_freeList = MarkedBlock::FreeList();
     268        m_currentBlock = filterNextBlock(m_currentBlock->next());
     269        setFreeList(FreeList());
    207270    }
    208271    if (m_nextBlockToSweep == block)
    209         m_nextBlockToSweep = m_nextBlockToSweep->next();
     272        setNextBlockToSweep(m_nextBlockToSweep->next());
    210273
    211274    block->willRemoveBlock();
    … …  
    213276}
    214277
     278void MarkedAllocator::stopAllocating()
     279{
     280    if (m_heap->operationInProgress() == FullCollection)
     281        m_blockList.takeFrom(m_retiredBlocks);
     282
     283    ASSERT(!m_lastActiveBlock);
     284    if (!m_currentBlock) {
     285        ASSERT(!m_freeList);
     286        return;
     287    }
     288   
     289    m_currentBlock->stopAllocating(m_freeList);
     290    m_lastActiveBlock = m_currentBlock;
     291    m_currentBlock = 0;
     292    m_freeList = FreeList();
     293}
     294
    215295void MarkedAllocator::reset()
    216296{
    217297    m_lastActiveBlock = 0;
    218298    m_currentBlock = 0;
    219     m_freeList = MarkedBlock::FreeList();
    220     if (m_heap->operationInProgress() == FullCollection)
    221         m_blockList.append(m_retiredBlocks);
    222 
    223     m_nextBlockToSweep = m_blockList.head();
     299    setFreeList(FreeList());
     300
     301    setNextBlockToSweep(m_blockList.begin());
    224302
    225303    if (UNLIKELY(Options::useImmortalObjects())) {
    226         MarkedBlock* next;
    227         for (MarkedBlock*& block = m_nextBlockToSweep; block; block = next) {
    228             next = block->next();
    229 
    230             MarkedBlock::FreeList freeList = block->sweep(MarkedBlock::SweepToFreeList);
    231             retire(block, freeList);
     304        MarkedBlock::Handle* next;
     305        for (MarkedBlock::Handle*& block = m_nextBlockToSweep; block; block = next) {
     306            next = filterNextBlock(block->next());
     307
     308            FreeList freeList = block->sweep(MarkedBlock::Handle::SweepToFreeList);
     309            block->zap(freeList);
     310            retire(block);
    232311        }
    233312    }
    … …  
    236315void MarkedAllocator::lastChanceToFinalize()
    237316{
    238     m_blockList.append(m_retiredBlocks);
     317    m_blockList.takeFrom(m_retiredBlocks);
    239318    forEachBlock(
    240         [&] (MarkedBlock* block) {
     319        [&] (MarkedBlock::Handle* block) {
    241320            block->lastChanceToFinalize();
    242321        });
    243322}
    244323
     324void MarkedAllocator::setFreeList(const FreeList& freeList)
     325{
     326    m_freeList = freeList;
     327}
     328
    245329} // namespace JSC

  • trunk/Source/JavaScriptCore/heap/MarkedAllocator.h

    r204912 r205462  
    2828
    2929#include "AllocatorAttributes.h"
     30#include "FreeList.h"
    3031#include "MarkedBlock.h"
    31 #include <wtf/DoublyLinkedList.h>
     32#include <wtf/SentinelLinkedList.h>
    3233
    3334namespace JSC {
    … …  
    4142
    4243public:
    43     static ptrdiff_t offsetOfFreeListHead();
     44    static ptrdiff_t offsetOfFreeList();
     45    static ptrdiff_t offsetOfCellSize();
    4446
    45     MarkedAllocator();
     47    MarkedAllocator(Heap*, MarkedSpace*, size_t cellSize, const AllocatorAttributes&);
    4648    void lastChanceToFinalize();
    4749    void reset();
    … …  
    5355    DestructionMode destruction() const { return m_attributes.destruction; }
    5456    HeapCell::Kind cellKind() const { return m_attributes.cellKind; }
    55     void* allocate(size_t);
     57    void* allocate();
     58    void* tryAllocate();
    5659    Heap* heap() { return m_heap; }
    57     MarkedBlock* takeLastActiveBlock()
     60    MarkedBlock::Handle* takeLastActiveBlock()
    5861    {
    59         MarkedBlock* block = m_lastActiveBlock;
     62        MarkedBlock::Handle* block = m_lastActiveBlock;
    6063        m_lastActiveBlock = 0;
    6164        return block;
    … …  
    6467    template<typename Functor> void forEachBlock(const Functor&);
    6568   
    66     void addBlock(MarkedBlock*);
    67     void removeBlock(MarkedBlock*);
    68     void init(Heap*, MarkedSpace*, size_t cellSize, const AllocatorAttributes&);
     69    void addBlock(MarkedBlock::Handle*);
     70    void removeBlock(MarkedBlock::Handle*);
    6971
    7072    bool isPagedOut(double deadline);
     73   
     74    static size_t blockSizeForBytes(size_t);
    7175   
    7276private:
    73     JS_EXPORT_PRIVATE void* allocateSlowCase(size_t);
    74     void* tryAllocate(size_t);
    75     void* tryAllocateHelper(size_t);
    76     void* tryPopFreeList(size_t);
    77     MarkedBlock* allocateBlock(size_t);
     77    friend class MarkedBlock;
     78   
     79    JS_EXPORT_PRIVATE void* allocateSlowCase();
     80    JS_EXPORT_PRIVATE void* tryAllocateSlowCase();
     81    void* allocateSlowCaseImpl(bool crashOnFailure);
     82    void* tryAllocateWithoutCollecting();
     83    void* tryAllocateWithoutCollectingImpl();
     84    MarkedBlock::Handle* tryAllocateBlock();
    7885    ALWAYS_INLINE void doTestCollectionsIfNeeded();
    79     void retire(MarkedBlock*, MarkedBlock::FreeList&);
     86    void retire(MarkedBlock::Handle*);
    8087   
    81     MarkedBlock::FreeList m_freeList;
    82     MarkedBlock* m_currentBlock;
    83     MarkedBlock* m_lastActiveBlock;
    84     MarkedBlock* m_nextBlockToSweep;
    85     DoublyLinkedList<MarkedBlock> m_blockList;
    86     DoublyLinkedList<MarkedBlock> m_retiredBlocks;
    87     size_t m_cellSize;
     88    void setFreeList(const FreeList&);
     89   
     90    MarkedBlock::Handle* filterNextBlock(MarkedBlock::Handle*);
     91    void setNextBlockToSweep(MarkedBlock::Handle*);
     92   
     93    FreeList m_freeList;
     94    MarkedBlock::Handle* m_currentBlock;
     95    MarkedBlock::Handle* m_lastActiveBlock;
     96    MarkedBlock::Handle* m_nextBlockToSweep;
     97    SentinelLinkedList<MarkedBlock::Handle, BasicRawSentinelNode<MarkedBlock::Handle>> m_blockList;
     98    SentinelLinkedList<MarkedBlock::Handle, BasicRawSentinelNode<MarkedBlock::Handle>> m_retiredBlocks;
     99    Lock m_lock;
     100    unsigned m_cellSize;
    88101    AllocatorAttributes m_attributes;
    89102    Heap* m_heap;
    … …  
    91104};
    92105
    93 inline ptrdiff_t MarkedAllocator::offsetOfFreeListHead()
     106inline ptrdiff_t MarkedAllocator::offsetOfFreeList()
    94107{
    95     return OBJECT_OFFSETOF(MarkedAllocator, m_freeList) + OBJECT_OFFSETOF(MarkedBlock::FreeList, head);
     108    return OBJECT_OFFSETOF(MarkedAllocator, m_freeList);
    96109}
    97110
    98 inline MarkedAllocator::MarkedAllocator()
    99     : m_currentBlock(0)
    100     , m_lastActiveBlock(0)
    101     , m_nextBlockToSweep(0)
    102     , m_cellSize(0)
    103     , m_heap(0)
    104     , m_markedSpace(0)
     111inline ptrdiff_t MarkedAllocator::offsetOfCellSize()
    105112{
     113    return OBJECT_OFFSETOF(MarkedAllocator, m_cellSize);
    106114}
    107115
    108 inline void MarkedAllocator::init(Heap* heap, MarkedSpace* markedSpace, size_t cellSize, const AllocatorAttributes& attributes)
     116ALWAYS_INLINE void* MarkedAllocator::tryAllocate()
    109117{
    110     m_heap = heap;
    111     m_markedSpace = markedSpace;
    112     m_cellSize = cellSize;
    113     m_attributes = attributes;
    114 }
    115 
    116 inline void* MarkedAllocator::allocate(size_t bytes)
    117 {
    118     MarkedBlock::FreeCell* head = m_freeList.head;
    119     if (UNLIKELY(!head)) {
    120         void* result = allocateSlowCase(bytes);
    121 #ifndef NDEBUG
    122         memset(result, 0xCD, bytes);
    123 #endif
    124         return result;
     118    unsigned remaining = m_freeList.remaining;
     119    if (remaining) {
     120        unsigned cellSize = m_cellSize;
     121        remaining -= cellSize;
     122        m_freeList.remaining = remaining;
     123        return m_freeList.payloadEnd - remaining - cellSize;
    125124    }
    126125   
     126    FreeCell* head = m_freeList.head;
     127    if (UNLIKELY(!head))
     128        return tryAllocateSlowCase();
     129   
    127130    m_freeList.head = head->next;
    128 #ifndef NDEBUG
    129     memset(head, 0xCD, bytes);
    130 #endif
    131131    return head;
    132132}
    133133
    134 inline void MarkedAllocator::stopAllocating()
     134ALWAYS_INLINE void* MarkedAllocator::allocate()
    135135{
    136     ASSERT(!m_lastActiveBlock);
    137     if (!m_currentBlock) {
    138         ASSERT(!m_freeList.head);
    139         return;
     136    unsigned remaining = m_freeList.remaining;
     137    if (remaining) {
     138        unsigned cellSize = m_cellSize;
     139        remaining -= cellSize;
     140        m_freeList.remaining = remaining;
     141        return m_freeList.payloadEnd - remaining - cellSize;
    140142    }
    141143   
    142     m_currentBlock->stopAllocating(m_freeList);
    143     m_lastActiveBlock = m_currentBlock;
    144     m_currentBlock = 0;
    145     m_freeList = MarkedBlock::FreeList();
     144    FreeCell* head = m_freeList.head;
     145    if (UNLIKELY(!head))
     146        return allocateSlowCase();
     147   
     148    m_freeList.head = head->next;
     149    return head;
    146150}
    147151
    … …  
    158162template <typename Functor> inline void MarkedAllocator::forEachBlock(const Functor& functor)
    159163{
    160     MarkedBlock* next;
    161     for (MarkedBlock* block = m_blockList.head(); block; block = next) {
    162         next = block->next();
    163         functor(block);
    164     }
    165 
    166     for (MarkedBlock* block = m_retiredBlocks.head(); block; block = next) {
    167         next = block->next();
    168         functor(block);
    169     }
     164    m_blockList.forEach(functor);
     165    m_retiredBlocks.forEach(functor);
    170166}
    171167

  • trunk/Source/JavaScriptCore/heap/MarkedBlock.cpp

    r204912 r205462  
    3030#include "JSDestructibleObject.h"
    3131#include "JSCInlines.h"
     32#include "SuperSampler.h"
    3233
    3334namespace JSC {
    … …  
    3637static size_t balance;
    3738
    38 MarkedBlock* MarkedBlock::create(Heap& heap, MarkedAllocator* allocator, size_t capacity, size_t cellSize, const AllocatorAttributes& attributes)
     39MarkedBlock::Handle* MarkedBlock::tryCreate(Heap& heap, MarkedAllocator* allocator, size_t cellSize, const AllocatorAttributes& attributes)
    3940{
    4041    if (computeBalance) {
    … …  
    4344            dataLog("MarkedBlock Balance: ", balance, "\n");
    4445    }
    45     MarkedBlock* block = new (NotNull, fastAlignedMalloc(blockSize, capacity)) MarkedBlock(allocator, capacity, cellSize, attributes);
    46     heap.didAllocateBlock(capacity);
    47     return block;
    48 }
    49 
    50 void MarkedBlock::destroy(Heap& heap, MarkedBlock* block)
    51 {
     46    void* blockSpace = tryFastAlignedMalloc(blockSize, blockSize);
     47    if (!blockSpace)
     48        return nullptr;
     49    if (scribbleFreeCells())
     50        scribble(blockSpace, blockSize);
     51    return new Handle(heap, allocator, cellSize, attributes, blockSpace);
     52}
     53
     54MarkedBlock::Handle::Handle(Heap& heap, MarkedAllocator* allocator, size_t cellSize, const AllocatorAttributes& attributes, void* blockSpace)
     55    : m_atomsPerCell((cellSize + atomSize - 1) / atomSize)
     56    , m_endAtom(atomsPerBlock - m_atomsPerCell + 1)
     57    , m_attributes(attributes)
     58    , m_state(New) // All cells start out unmarked.
     59    , m_allocator(allocator)
     60    , m_weakSet(allocator->heap()->vm(), CellContainer())
     61{
     62    m_block = new (NotNull, blockSpace) MarkedBlock(*heap.vm(), *this);
     63   
     64    m_weakSet.setContainer(*m_block);
     65   
     66    heap.didAllocateBlock(blockSize);
     67    HEAP_LOG_BLOCK_STATE_TRANSITION(this);
     68    ASSERT(allocator);
     69    if (m_attributes.cellKind != HeapCell::JSCell)
     70        RELEASE_ASSERT(m_attributes.destruction == DoesNotNeedDestruction);
     71}
     72
     73MarkedBlock::Handle::~Handle()
     74{
     75    Heap& heap = *this->heap();
    5276    if (computeBalance) {
    5377        balance--;
    … …  
    5579            dataLog("MarkedBlock Balance: ", balance, "\n");
    5680    }
    57     size_t capacity = block->capacity();
    58     block->~MarkedBlock();
    59     fastAlignedFree(block);
    60     heap.didFreeBlock(capacity);
    61 }
    62 
    63 MarkedBlock::MarkedBlock(MarkedAllocator* allocator, size_t capacity, size_t cellSize, const AllocatorAttributes& attributes)
    64     : DoublyLinkedListNode<MarkedBlock>()
    65     , m_atomsPerCell((cellSize + atomSize - 1) / atomSize)
    66     , m_endAtom((allocator->cellSize() ? atomsPerBlock - m_atomsPerCell : firstAtom()) + 1)
    67     , m_capacity(capacity)
    68     , m_attributes(attributes)
    69     , m_allocator(allocator)
    70     , m_state(New) // All cells start out unmarked.
    71     , m_weakSet(allocator->heap()->vm(), *this)
    72 {
    73     ASSERT(allocator);
    74     HEAP_LOG_BLOCK_STATE_TRANSITION(this);
    75     if (m_attributes.cellKind != HeapCell::JSCell)
    76         RELEASE_ASSERT(m_attributes.destruction == DoesNotNeedDestruction);
    77 }
    78 
    79 inline void MarkedBlock::callDestructor(HeapCell* cell)
    80 {
    81     // A previous eager sweep may already have run cell's destructor.
    82     if (cell->isZapped())
    83         return;
    84    
    85     JSCell* jsCell = static_cast<JSCell*>(cell);
    86 
    87     ASSERT(jsCell->structureID());
    88     if (jsCell->inlineTypeFlags() & StructureIsImmortal)
    89         jsCell->structure(*vm())->classInfo()->methodTable.destroy(jsCell);
    90     else
    91         jsCast<JSDestructibleObject*>(jsCell)->classInfo()->methodTable.destroy(jsCell);
    92     cell->zap();
    93 }
    94 
    95 template<MarkedBlock::BlockState blockState, MarkedBlock::SweepMode sweepMode, bool callDestructors>
    96 MarkedBlock::FreeList MarkedBlock::specializedSweep()
    97 {
    98     ASSERT(blockState != Allocated && blockState != FreeListed);
    99     ASSERT(!(!callDestructors && sweepMode == SweepOnly));
     81    m_block->~MarkedBlock();
     82    fastAlignedFree(m_block);
     83    heap.didFreeBlock(blockSize);
     84}
     85
     86MarkedBlock::MarkedBlock(VM& vm, Handle& handle)
     87    : m_needsDestruction(handle.needsDestruction())
     88    , m_handle(handle)
     89    , m_vm(&vm)
     90    , m_version(vm.heap.objectSpace().version())
     91{
     92    unsigned cellsPerBlock = MarkedSpace::blockPayload / handle.cellSize();
     93    double markCountBias = -(Options::minMarkedBlockUtilization() * cellsPerBlock);
     94   
     95    // The mark count bias should be comfortably within this range.
     96    RELEASE_ASSERT(markCountBias > static_cast<double>(std::numeric_limits<int16_t>::min()));
     97    RELEASE_ASSERT(markCountBias < 0);
     98   
     99    m_markCountBias = static_cast<int16_t>(markCountBias);
     100   
     101    m_biasedMarkCount = m_markCountBias; // This means we haven't marked anything yet.
     102}
     103
     104template<MarkedBlock::BlockState blockState, MarkedBlock::Handle::SweepMode sweepMode, DestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode, MarkedBlock::Handle::NewlyAllocatedMode newlyAllocatedMode>
     105FreeList MarkedBlock::Handle::specializedSweep()
     106{
     107    SuperSamplerScope superSamplerScope(false);
     108    ASSERT(blockState == New || blockState == Marked);
     109    ASSERT(!(destructionMode == DoesNotNeedDestruction && sweepMode == SweepOnly));
     110   
     111    assertFlipped();
     112    MarkedBlock& block = this->block();
     113   
     114    bool isNewBlock = blockState == New;
     115    bool isEmptyBlock = !block.hasAnyMarked()
     116        && newlyAllocatedMode == DoesNotHaveNewlyAllocated
     117        && destructionMode == DoesNotNeedDestruction;
     118    if (Options::useBumpAllocator() && (isNewBlock || isEmptyBlock)) {
     119        ASSERT(block.m_marks.isEmpty());
     120       
     121        char* startOfLastCell = static_cast<char*>(cellAlign(block.atoms() + m_endAtom - 1));
     122        char* payloadEnd = startOfLastCell + cellSize();
     123        RELEASE_ASSERT(payloadEnd - MarkedBlock::blockSize <= bitwise_cast<char*>(&block));
     124        char* payloadBegin = bitwise_cast<char*>(block.atoms() + firstAtom());
     125        if (scribbleMode == Scribble)
     126            scribble(payloadBegin, payloadEnd - payloadBegin);
     127        m_state = ((sweepMode == SweepToFreeList) ? FreeListed : Marked);
     128        FreeList result = FreeList::bump(payloadEnd, payloadEnd - payloadBegin);
     129        if (false)
     130            dataLog("Quickly swept block ", RawPointer(this), " with cell size ", cellSize(), " and attributes ", m_attributes, ": ", result, "\n");
     131        return result;
     132    }
    100133
    101134    // This produces a free list that is ordered in reverse through the block.
    … …  
    105138    size_t count = 0;
    106139    for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell) {
    107         if (blockState == Marked && (m_marks.get(i) || (m_newlyAllocated && m_newlyAllocated->get(i))))
     140        if (blockState == Marked
     141            && (block.m_marks.get(i)
     142                || (newlyAllocatedMode == HasNewlyAllocated && m_newlyAllocated->get(i))))
    108143            continue;
    109144
    110         HeapCell* cell = reinterpret_cast_ptr<HeapCell*>(&atoms()[i]);
    111 
    112         if (callDestructors && blockState != New)
    113             callDestructor(cell);
     145        HeapCell* cell = reinterpret_cast_ptr<HeapCell*>(&block.atoms()[i]);
     146
     147        if (destructionMode == NeedsDestruction && blockState != New)
     148            static_cast<JSCell*>(cell)->callDestructor(*vm());
    114149
    115150        if (sweepMode == SweepToFreeList) {
    116151            FreeCell* freeCell = reinterpret_cast<FreeCell*>(cell);
     152            if (scribbleMode == Scribble)
     153                scribble(freeCell, cellSize());
    117154            freeCell->next = head;
    118155            head = freeCell;
    … …  
    123160    // We only want to discard the newlyAllocated bits if we're creating a FreeList,
    124161    // otherwise we would lose information on what's currently alive.
    125     if (sweepMode == SweepToFreeList && m_newlyAllocated)
     162    if (sweepMode == SweepToFreeList && newlyAllocatedMode == HasNewlyAllocated)
    126163        m_newlyAllocated = nullptr;
    127164
    128     m_state = ((sweepMode == SweepToFreeList) ? FreeListed : Marked);
    129     return FreeList(head, count * cellSize());
    130 }
    131 
    132 MarkedBlock::FreeList MarkedBlock::sweep(SweepMode sweepMode)
    133 {
     165    FreeList result = FreeList::list(head, count * cellSize());
     166    m_state = (sweepMode == SweepToFreeList ? FreeListed : Marked);
     167    if (false)
     168        dataLog("Slowly swept block ", RawPointer(&block), " with cell size ", cellSize(), " and attributes ", m_attributes, ": ", result, "\n");
     169    return result;
     170}
     171
     172FreeList MarkedBlock::Handle::sweep(SweepMode sweepMode)
     173{
     174    flipIfNecessary();
     175   
    134176    HEAP_LOG_BLOCK_STATE_TRANSITION(this);
    135177
    … …  
    140182
    141183    if (m_attributes.destruction == NeedsDestruction)
    142         return sweepHelper<true>(sweepMode);
    143     return sweepHelper<false>(sweepMode);
    144 }
    145 
    146 template<bool callDestructors>
    147 MarkedBlock::FreeList MarkedBlock::sweepHelper(SweepMode sweepMode)
     184        return sweepHelperSelectScribbleMode<NeedsDestruction>(sweepMode);
     185    return sweepHelperSelectScribbleMode<DoesNotNeedDestruction>(sweepMode);
     186}
     187
     188template<DestructionMode destructionMode>
     189FreeList MarkedBlock::Handle::sweepHelperSelectScribbleMode(SweepMode sweepMode)
     190{
     191    if (scribbleFreeCells())
     192        return sweepHelperSelectStateAndSweepMode<destructionMode, Scribble>(sweepMode);
     193    return sweepHelperSelectStateAndSweepMode<destructionMode, DontScribble>(sweepMode);
     194}
     195
     196template<DestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode>
     197FreeList MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode(SweepMode sweepMode)
    148198{
    149199    switch (m_state) {
    150200    case New:
    151201        ASSERT(sweepMode == SweepToFreeList);
    152         return specializedSweep<New, SweepToFreeList, callDestructors>();
     202        return specializedSweep<New, SweepToFreeList, destructionMode, scribbleMode, DoesNotHaveNewlyAllocated>();
    153203    case FreeListed:
    154204        // Happens when a block transitions to fully allocated.
    155205        ASSERT(sweepMode == SweepToFreeList);
    156206        return FreeList();
    157     case Retired:
    158207    case Allocated:
    159208        RELEASE_ASSERT_NOT_REACHED();
    160209        return FreeList();
    161210    case Marked:
    162         return sweepMode == SweepToFreeList
    163             ? specializedSweep<Marked, SweepToFreeList, callDestructors>()
    164             : specializedSweep<Marked, SweepOnly, callDestructors>();
     211        if (m_newlyAllocated) {
     212            return sweepMode == SweepToFreeList
     213                ? specializedSweep<Marked, SweepToFreeList, destructionMode, scribbleMode, HasNewlyAllocated>()
     214                : specializedSweep<Marked, SweepOnly, destructionMode, scribbleMode, HasNewlyAllocated>();
     215        } else {
     216            return sweepMode == SweepToFreeList
     217                ? specializedSweep<Marked, SweepToFreeList, destructionMode, scribbleMode, DoesNotHaveNewlyAllocated>()
     218                : specializedSweep<Marked, SweepOnly, destructionMode, scribbleMode, DoesNotHaveNewlyAllocated>();
     219        }
    165220    }
    166221    RELEASE_ASSERT_NOT_REACHED();
    … …  
    168223}
    169224
     225void MarkedBlock::Handle::unsweepWithNoNewlyAllocated()
     226{
     227    flipIfNecessary();
     228   
     229    HEAP_LOG_BLOCK_STATE_TRANSITION(this);
     230   
     231    RELEASE_ASSERT(m_state == FreeListed);
     232    m_state = Marked;
     233}
     234
    170235class SetNewlyAllocatedFunctor : public MarkedBlock::VoidFunctor {
    171236public:
    172     SetNewlyAllocatedFunctor(MarkedBlock* block)
     237    SetNewlyAllocatedFunctor(MarkedBlock::Handle* block)
    173238        : m_block(block)
    174239    {
    … …  
    177242    IterationStatus operator()(HeapCell* cell, HeapCell::Kind) const
    178243    {
    179         ASSERT(MarkedBlock::blockFor(cell) == m_block);
     244        ASSERT(MarkedBlock::blockFor(cell) == &m_block->block());
    180245        m_block->setNewlyAllocated(cell);
    181246        return IterationStatus::Continue;
    … …  
    183248
    184249private:
    185     MarkedBlock* m_block;
     250    MarkedBlock::Handle* m_block;
    186251};
    187252
    188 void MarkedBlock::stopAllocating(const FreeList& freeList)
    189 {
    190     HEAP_LOG_BLOCK_STATE_TRANSITION(this);
    191     FreeCell* head = freeList.head;
     253void MarkedBlock::Handle::stopAllocating(const FreeList& freeList)
     254{
     255    flipIfNecessary();
     256    HEAP_LOG_BLOCK_STATE_TRANSITION(this);
    192257
    193258    if (m_state == Marked) {
    194         // If the block is in the Marked state then we know that:
    195         // 1) It was not used for allocation during the previous allocation cycle.
    196         // 2) It may have dead objects, and we only know them to be dead by the
    197         //    fact that their mark bits are unset.
     259        // If the block is in the Marked state then we know that one of these
     260        // conditions holds:
     261        //
     262        // - It was not used for allocation during the previous allocation cycle.
     263        //   It may have dead objects, and we only know them to be dead by the
     264        //   fact that their mark bits are unset.
     265        //
     266        // - Someone had already done stopAllocating(), for example because of
     267        //   heap iteration, and they had already
    198268        // Hence if the block is Marked we need to leave it Marked.
    199        
    200         ASSERT(!head);
     269        ASSERT(freeList.allocationWillFail());
    201270        return;
    202271    }
    203    
     272    
    204273    ASSERT(m_state == FreeListed);
    205274   
    … …  
    214283    forEachCell(functor);
    215284
    216     FreeCell* next;
    217     for (FreeCell* current = head; current; current = next) {
    218         next = current->next;
    219         if (m_attributes.destruction == NeedsDestruction)
    220             reinterpret_cast<HeapCell*>(current)->zap();
    221         clearNewlyAllocated(current);
    222     }
     285    forEachFreeCell(
     286        freeList,
     287        [&] (HeapCell* cell) {
     288            if (m_attributes.destruction == NeedsDestruction)
     289                cell->zap();
     290            clearNewlyAllocated(cell);
     291        });
    223292   
    224293    m_state = Marked;
    225294}
    226295
    227 void MarkedBlock::clearMarks()
    228 {
    229     if (heap()->operationInProgress() == JSC::EdenCollection)
    230         this->clearMarksWithCollectionType<EdenCollection>();
    231     else
    232         this->clearMarksWithCollectionType<FullCollection>();
    233 }
    234 
    235 template <HeapOperation collectionType>
    236 void MarkedBlock::clearMarksWithCollectionType()
    237 {
    238     ASSERT(collectionType == FullCollection || collectionType == EdenCollection);
    239     HEAP_LOG_BLOCK_STATE_TRANSITION(this);
    240 
    241     ASSERT(m_state != New && m_state != FreeListed);
    242     if (collectionType == FullCollection) {
    243         m_marks.clearAll();
    244         // This will become true at the end of the mark phase. We set it now to
    245         // avoid an extra pass to do so later.
    246         m_state = Marked;
    247         return;
    248     }
    249 
    250     ASSERT(collectionType == EdenCollection);
    251     // If a block was retired then there's no way an EdenCollection can un-retire it.
    252     if (m_state != Retired)
    253         m_state = Marked;
    254 }
    255 
    256 void MarkedBlock::lastChanceToFinalize()
    257 {
     296void MarkedBlock::Handle::lastChanceToFinalize()
     297{
     298    m_block->clearMarks();
    258299    m_weakSet.lastChanceToFinalize();
    259300
    260301    clearNewlyAllocated();
    261     clearMarksWithCollectionType<FullCollection>();
    262302    sweep();
    263303}
    264304
    265 MarkedBlock::FreeList MarkedBlock::resumeAllocating()
    266 {
     305FreeList MarkedBlock::Handle::resumeAllocating()
     306{
     307    flipIfNecessary();
    267308    HEAP_LOG_BLOCK_STATE_TRANSITION(this);
    268309
    … …  
    275316    }
    276317
    277     // Re-create our free list from before stopping allocation.
     318    // Re-create our free list from before stopping allocation. Note that this may return an empty
     319    // freelist, in which case the block will still be Marked!
    278320    return sweep(SweepToFreeList);
    279321}
    280322
    281 void MarkedBlock::didRetireBlock(const FreeList& freeList)
    282 {
    283     HEAP_LOG_BLOCK_STATE_TRANSITION(this);
    284     FreeCell* head = freeList.head;
    285 
    286     // Currently we don't notify the Heap that we're giving up on this block.
    287     // The Heap might be able to make a better decision about how many bytes should
    288     // be allocated before the next collection if it knew about this retired block.
    289     // On the other hand we'll waste at most 10% of our Heap space between FullCollections
    290     // and only under heavy fragmentation.
    291 
    292     // We need to zap the free list when retiring a block so that we don't try to destroy
    293     // previously destroyed objects when we re-sweep the block in the future.
    294     FreeCell* next;
    295     for (FreeCell* current = head; current; current = next) {
    296         next = current->next;
    297         if (m_attributes.destruction == NeedsDestruction)
    298             reinterpret_cast<HeapCell*>(current)->zap();
    299     }
    300 
     323void MarkedBlock::Handle::zap(const FreeList& freeList)
     324{
     325    forEachFreeCell(
     326        freeList,
     327        [&] (HeapCell* cell) {
     328            if (m_attributes.destruction == NeedsDestruction)
     329                cell->zap();
     330        });
     331}
     332
     333template<typename Func>
     334void MarkedBlock::Handle::forEachFreeCell(const FreeList& freeList, const Func& func)
     335{
     336    if (freeList.remaining) {
     337        for (unsigned remaining = freeList.remaining; remaining; remaining -= cellSize())
     338            func(bitwise_cast<HeapCell*>(freeList.payloadEnd - remaining));
     339    } else {
     340        for (FreeCell* current = freeList.head; current;) {
     341            FreeCell* next = current->next;
     342            func(bitwise_cast<HeapCell*>(current));
     343            current = next;
     344        }
     345    }
     346}
     347
     348void MarkedBlock::flipIfNecessary()
     349{
     350    flipIfNecessary(vm()->heap.objectSpace().version());
     351}
     352
     353void MarkedBlock::Handle::flipIfNecessary()
     354{
     355    block().flipIfNecessary();
     356}
     357
     358void MarkedBlock::flipIfNecessarySlow()
     359{
     360    ASSERT(m_version != vm()->heap.objectSpace().version());
     361    clearMarks();
     362}
     363
     364void MarkedBlock::flipIfNecessaryConcurrentlySlow()
     365{
     366    LockHolder locker(m_lock);
     367    if (m_version != vm()->heap.objectSpace().version())
     368        clearMarks();
     369}
     370
     371void MarkedBlock::clearMarks()
     372{
     373    m_marks.clearAll();
     374    clearHasAnyMarked();
     375    // This will become true at the end of the mark phase. We set it now to
     376    // avoid an extra pass to do so later.
     377    handle().m_state = Marked;
     378    WTF::storeStoreFence();
     379    m_version = vm()->heap.objectSpace().version();
     380}
     381
     382#if !ASSERT_DISABLED
     383void MarkedBlock::assertFlipped()
     384{
     385    ASSERT(m_version == vm()->heap.objectSpace().version());
     386}
     387#endif // !ASSERT_DISABLED
     388
     389bool MarkedBlock::needsFlip()
     390{
     391    return vm()->heap.objectSpace().version() != m_version;
     392}
     393
     394bool MarkedBlock::Handle::needsFlip()
     395{
     396    return m_block->needsFlip();
     397}
     398
     399void MarkedBlock::Handle::willRemoveBlock()
     400{
     401    flipIfNecessary();
     402}
     403
     404void MarkedBlock::Handle::didConsumeFreeList()
     405{
     406    flipIfNecessary();
     407    HEAP_LOG_BLOCK_STATE_TRANSITION(this);
     408   
    301409    ASSERT(m_state == FreeListed);
    302     m_state = Retired;
     410
     411    m_state = Allocated;
     412}
     413
     414size_t MarkedBlock::markCount()
     415{
     416    flipIfNecessary();
     417    return m_marks.count();
     418}
     419
     420bool MarkedBlock::Handle::isEmpty()
     421{
     422    flipIfNecessary();
     423    return m_state == Marked && !block().hasAnyMarked() && m_weakSet.isEmpty() && (!m_newlyAllocated || m_newlyAllocated->isEmpty());
     424}
     425
     426void MarkedBlock::clearHasAnyMarked()
     427{
     428    m_biasedMarkCount = m_markCountBias;
     429}
     430
     431void MarkedBlock::noteMarkedSlow()
     432{
     433    handle().m_allocator->retire(&handle());
    303434}
    304435
    305436} // namespace JSC
     437
     438namespace WTF {
     439
     440using namespace JSC;
     441
     442void printInternal(PrintStream& out, MarkedBlock::BlockState blockState)
     443{
     444    switch (blockState) {
     445    case MarkedBlock::New:
     446        out.print("New");
     447        return;
     448    case MarkedBlock::FreeListed:
     449        out.print("FreeListed");
     450        return;
     451    case MarkedBlock::Allocated:
     452        out.print("Allocated");
     453        return;
     454    case MarkedBlock::Marked:
     455        out.print("Marked");
     456        return;
     457    }
     458    RELEASE_ASSERT_NOT_REACHED();
     459}
     460
     461} // namespace WTF
     462

  • trunk/Source/JavaScriptCore/heap/MarkedBlock.h

    r204912 r205462  
    2525#include "AllocatorAttributes.h"
    2626#include "DestructionMode.h"
     27#include "FreeList.h"
    2728#include "HeapCell.h"
    2829#include "HeapOperation.h"
    … …  
    3536#include <wtf/StdLibExtras.h>
    3637
     38namespace JSC {
     39   
     40class Heap;
     41class JSCell;
     42class MarkedAllocator;
     43
     44typedef uintptr_t Bits;
     45
    3746// Set to log state transitions of blocks.
    3847#define HEAP_LOG_BLOCK_STATE_TRANSITIONS 0
    3948
    4049#if HEAP_LOG_BLOCK_STATE_TRANSITIONS
    41 #define HEAP_LOG_BLOCK_STATE_TRANSITION(block) do {                     \
    42         dataLogF(                                                    \
    43             "%s:%d %s: block %s = %p, %d\n",                            \
    44             __FILE__, __LINE__, __FUNCTION__,                           \
    45             #block, (block), (block)->m_state);                         \
     50#define HEAP_LOG_BLOCK_STATE_TRANSITION(handle) do {            \
     51        dataLogF(                                               \
     52            "%s:%d %s: block %s = %p, %d\n",                    \
     53            __FILE__, __LINE__, __FUNCTION__,                   \
     54            #handle, &(handle)->block(), (handle)->m_state);    \
    4655    } while (false)
    4756#else
    48 #define HEAP_LOG_BLOCK_STATE_TRANSITION(block) ((void)0)
     57#define HEAP_LOG_BLOCK_STATE_TRANSITION(handle) ((void)0)
    4958#endif
    5059
    51 namespace JSC {
    52    
    53     class Heap;
    54     class JSCell;
    55     class MarkedAllocator;
    56 
    57     typedef uintptr_t Bits;
    58 
    59     // A marked block is a page-aligned container for heap-allocated objects.
    60     // Objects are allocated within cells of the marked block. For a given
    61     // marked block, all cells have the same size. Objects smaller than the
    62     // cell size may be allocated in the marked block, in which case the
    63     // allocation suffers from internal fragmentation: wasted space whose
    64     // size is equal to the difference between the cell size and the object
    65     // size.
    66 
    67     class MarkedBlock : public DoublyLinkedListNode<MarkedBlock> {
    68         friend class WTF::DoublyLinkedListNode<MarkedBlock>;
     60// A marked block is a page-aligned container for heap-allocated objects.
     61// Objects are allocated within cells of the marked block. For a given
     62// marked block, all cells have the same size. Objects smaller than the
     63// cell size may be allocated in the marked block, in which case the
     64// allocation suffers from internal fragmentation: wasted space whose
     65// size is equal to the difference between the cell size and the object
     66// size.
     67
     68class MarkedBlock {
     69    WTF_MAKE_NONCOPYABLE(MarkedBlock);
     70    friend class LLIntOffsetsExtractor;
     71    friend struct VerifyMarked;
     72
     73public:
     74    class Handle;
     75private:
     76    friend class Handle;
     77public:
     78    enum BlockState : uint8_t { New, FreeListed, Allocated, Marked };
     79       
     80    static const size_t atomSize = 16; // bytes
     81    static const size_t blockSize = 16 * KB;
     82    static const size_t blockMask = ~(blockSize - 1); // blockSize must be a power of two.
     83
     84    static const size_t atomsPerBlock = blockSize / atomSize;
     85
     86    static_assert(!(MarkedBlock::atomSize & (MarkedBlock::atomSize - 1)), "MarkedBlock::atomSize must be a power of two.");
     87    static_assert(!(MarkedBlock::blockSize & (MarkedBlock::blockSize - 1)), "MarkedBlock::blockSize must be a power of two.");
     88
     89    struct VoidFunctor {
     90        typedef void ReturnType;
     91        void returnValue() { }
     92    };
     93
     94    class CountFunctor {
     95    public:
     96        typedef size_t ReturnType;
     97
     98        CountFunctor() : m_count(0) { }
     99        void count(size_t count) const { m_count += count; }
     100        ReturnType returnValue() const { return m_count; }
     101
     102    private:
     103        // FIXME: This is mutable because we're using a functor rather than C++ lambdas.
     104        // https://bugs.webkit.org/show_bug.cgi?id=159644
     105        mutable ReturnType m_count;
     106    };
     107       
     108    class Handle : public BasicRawSentinelNode<Handle> {
     109        WTF_MAKE_NONCOPYABLE(Handle);
     110        WTF_MAKE_FAST_ALLOCATED;
     111        friend class DoublyLinkedListNode<Handle>;
    69112        friend class LLIntOffsetsExtractor;
    70         friend struct VerifyMarkedOrRetired;
     113        friend class MarkedBlock;
     114        friend struct VerifyMarked;
    71115    public:
    72         static const size_t atomSize = 16; // bytes
    73         static const size_t blockSize = 16 * KB;
    74         static const size_t blockMask = ~(blockSize - 1); // blockSize must be a power of two.
    75 
    76         static const size_t atomsPerBlock = blockSize / atomSize;
    77 
    78         static_assert(!(MarkedBlock::atomSize & (MarkedBlock::atomSize - 1)), "MarkedBlock::atomSize must be a power of two.");
    79         static_assert(!(MarkedBlock::blockSize & (MarkedBlock::blockSize - 1)), "MarkedBlock::blockSize must be a power of two.");
    80 
    81         struct FreeCell {
    82             FreeCell* next;
    83         };
    84        
    85         struct FreeList {
    86             FreeCell* head;
    87             size_t bytes;
    88 
    89             FreeList();
    90             FreeList(FreeCell*, size_t);
    91         };
    92 
    93         struct VoidFunctor {
    94             typedef void ReturnType;
    95             void returnValue() { }
    96         };
    97 
    98         class CountFunctor {
    99         public:
    100             typedef size_t ReturnType;
    101 
    102             CountFunctor() : m_count(0) { }
    103             void count(size_t count) const { m_count += count; }
    104             ReturnType returnValue() const { return m_count; }
    105 
    106         private:
    107             // FIXME: This is mutable because we're using a functor rather than C++ lambdas.
    108             // https://bugs.webkit.org/show_bug.cgi?id=159644
    109             mutable ReturnType m_count;
    110         };
    111 
    112         static MarkedBlock* create(Heap&, MarkedAllocator*, size_t capacity, size_t cellSize, const AllocatorAttributes&);
    113         static void destroy(Heap&, MarkedBlock*);
    114 
    115         static bool isAtomAligned(const void*);
    116         static MarkedBlock* blockFor(const void*);
    117         static size_t firstAtom();
    118        
     116           
     117        ~Handle();
     118           
     119        MarkedBlock& block();
     120           
     121        void* cellAlign(void*);
     122           
     123        bool isEmpty();
     124
    119125        void lastChanceToFinalize();
    120126
    … …  
    123129        VM* vm() const;
    124130        WeakSet& weakSet();
    125        
     131           
    126132        enum SweepMode { SweepOnly, SweepToFreeList };
    127133        FreeList sweep(SweepMode = SweepOnly);
    128 
     134       
     135        void unsweepWithNoNewlyAllocated();
     136       
     137        void zap(const FreeList&);
     138       
    129139        void shrink();
    130 
    131         void visitWeakSet(HeapRootVisitor&);
     140           
     141        unsigned visitWeakSet(HeapRootVisitor&);
    132142        void reapWeakSet();
    133 
     143           
    134144        // While allocating from a free list, MarkedBlock temporarily has bogus
    135145        // cell liveness data. To restore accurate cell liveness data, call one
    … …  
    138148        void stopAllocating(const FreeList&);
    139149        FreeList resumeAllocating(); // Call this if you canonicalized a block for some non-collection related purpose.
    140 
     150           
    141151        // Returns true if the "newly allocated" bitmap was non-null
    142152        // and was successfully cleared and false otherwise.
    143153        bool clearNewlyAllocated();
    144         void clearMarks();
    145         template <HeapOperation collectionType>
    146         void clearMarksWithCollectionType();
    147 
    148         size_t markCount();
    149         bool isEmpty();
    150 
     154           
     155        void flipForEdenCollection();
     156           
    151157        size_t cellSize();
    152158        const AllocatorAttributes& attributes() const;
    … …  
    154160        bool needsDestruction() const;
    155161        HeapCell::Kind cellKind() const;
    156 
     162           
     163        size_t markCount();
    157164        size_t size();
    158         size_t capacity();
    159 
    160         bool isMarked(const void*);
    161         bool testAndSetMarked(const void*);
     165           
    162166        bool isLive(const HeapCell*);
    163167        bool isLiveCell(const void*);
    164         bool isAtom(const void*);
    165168        bool isMarkedOrNewlyAllocated(const HeapCell*);
    166         void setMarked(const void*);
    167         void clearMarked(const void*);
    168 
     169           
    169170        bool isNewlyAllocated(const void*);
    170171        void setNewlyAllocated(const void*);
    171172        void clearNewlyAllocated(const void*);
    172 
     173       
     174        bool hasAnyNewlyAllocated() const { return !!m_newlyAllocated; }
     175           
    173176        bool isAllocated() const;
    174         bool isMarkedOrRetired() const;
     177        bool isMarked() const;
     178        bool isFreeListed() const;
    175179        bool needsSweeping() const;
    176         void didRetireBlock(const FreeList&);
    177180        void willRemoveBlock();
    178181
    … …  
    180183        template <typename Functor> IterationStatus forEachLiveCell(const Functor&);
    181184        template <typename Functor> IterationStatus forEachDeadCell(const Functor&);
    182 
     185           
     186        bool needsFlip();
     187           
     188        void flipIfNecessaryConcurrently(uint64_t heapVersion);
     189        void flipIfNecessary(uint64_t heapVersion);
     190        void flipIfNecessary();
     191           
     192        void assertFlipped();
     193           
     194        bool isOnBlocksToSweep() const { return m_isOnBlocksToSweep; }
     195        void setIsOnBlocksToSweep(bool value) { m_isOnBlocksToSweep = value; }
     196       
     197        BlockState state() const { return m_state; }
     198           
    183199    private:
    184         static const size_t atomAlignmentMask = atomSize - 1;
    185 
    186         // During allocation, we look for available space in free lists in blocks.
    187         // If a block's utilization is sufficiently high (i.e. it's almost full),
    188         // we want to remove that block as a candidate for allocating to reduce
    189         // the likelihood of allocation having to take a slow path. When the
    190         // block is in this state, we say that it is "Retired".
    191         //
    192         // A full GC can take a Retired blocks out of retirement. An eden GC
    193         // will simply ignore Retired blocks (i.e. they will not be swept even
    194         // if they no longer have live objects).
    195 
    196         enum BlockState { New, FreeListed, Allocated, Marked, Retired };
    197         template<bool callDestructors> FreeList sweepHelper(SweepMode = SweepOnly);
    198 
    199         typedef char Atom[atomSize];
    200 
    201         MarkedBlock(MarkedAllocator*, size_t capacity, size_t cellSize, const AllocatorAttributes&);
    202         Atom* atoms();
    203         size_t atomNumber(const void*);
    204         void callDestructor(HeapCell*);
    205         template<BlockState, SweepMode, bool callDestructors> FreeList specializedSweep();
    206        
    207         MarkedBlock* m_prev;
    208         MarkedBlock* m_next;
    209 
     200        Handle(Heap&, MarkedAllocator*, size_t cellSize, const AllocatorAttributes&, void*);
     201           
     202        template<DestructionMode>
     203        FreeList sweepHelperSelectScribbleMode(SweepMode = SweepOnly);
     204           
     205        enum ScribbleMode { DontScribble, Scribble };
     206           
     207        template<DestructionMode, ScribbleMode>
     208        FreeList sweepHelperSelectStateAndSweepMode(SweepMode = SweepOnly);
     209           
     210        enum NewlyAllocatedMode { HasNewlyAllocated, DoesNotHaveNewlyAllocated };
     211           
     212        template<BlockState, SweepMode, DestructionMode, ScribbleMode, NewlyAllocatedMode>
     213        FreeList specializedSweep();
     214           
     215        template<typename Func>
     216        void forEachFreeCell(const FreeList&, const Func&);
     217           
     218        MarkedBlock::Handle* m_prev;
     219        MarkedBlock::Handle* m_next;
     220           
    210221        size_t m_atomsPerCell;
    211222        size_t m_endAtom; // This is a fuzzy end. Always test for < m_endAtom.
    212         WTF::Bitmap<atomsPerBlock, WTF::BitmapAtomic, uint8_t> m_marks;
     223           
    213224        std::unique_ptr<WTF::Bitmap<atomsPerBlock>> m_newlyAllocated;
    214 
    215         size_t m_capacity;
     225           
    216226        AllocatorAttributes m_attributes;
     227        BlockState m_state;
     228        bool m_isOnBlocksToSweep { false };
     229           
    217230        MarkedAllocator* m_allocator;
    218         BlockState m_state;
    219231        WeakSet m_weakSet;
     232           
     233        MarkedBlock* m_block;
    220234    };
    221 
    222     inline MarkedBlock::FreeList::FreeList()
    223         : head(0)
    224         , bytes(0)
    225     {
     235       
     236    static MarkedBlock::Handle* tryCreate(Heap&, MarkedAllocator*, size_t cellSize, const AllocatorAttributes&);
     237       
     238    Handle& handle();
     239       
     240    VM* vm() const;
     241
     242    static bool isAtomAligned(const void*);
     243    static MarkedBlock* blockFor(const void*);
     244    static size_t firstAtom();
     245    size_t atomNumber(const void*);
     246       
     247    size_t markCount();
     248
     249    bool isMarked(const void*);
     250    bool testAndSetMarked(const void*);
     251       
     252    bool isMarkedOrNewlyAllocated(const HeapCell*);
     253
     254    bool isAtom(const void*);
     255    void setMarked(const void*);
     256    void clearMarked(const void*);
     257       
     258    size_t cellSize();
     259    const AllocatorAttributes& attributes() const;
     260
     261    bool hasAnyMarked() const;
     262    void noteMarked();
     263       
     264    WeakSet& weakSet();
     265
     266    bool needsFlip();
     267       
     268    void flipIfNecessaryConcurrently(uint64_t heapVersion);
     269    void flipIfNecessary(uint64_t heapVersion);
     270    void flipIfNecessary();
     271       
     272    void assertFlipped();
     273       
     274    bool needsDestruction() const { return m_needsDestruction; }
     275       
     276private:
     277    static const size_t atomAlignmentMask = atomSize - 1;
     278
     279    typedef char Atom[atomSize];
     280
     281    MarkedBlock(VM&, Handle&);
     282    Atom* atoms();
     283       
     284    void flipIfNecessaryConcurrentlySlow();
     285    void flipIfNecessarySlow();
     286    void clearMarks();
     287    void clearHasAnyMarked();
     288   
     289    void noteMarkedSlow();
     290       
     291    WTF::Bitmap<atomsPerBlock, WTF::BitmapAtomic, uint8_t> m_marks;
     292
     293    bool m_needsDestruction;
     294    Lock m_lock;
     295   
     296    // The actual mark count can be computed by doing: m_biasedMarkCount - m_markCountBias. Note
     297    // that this count is racy. It will accurately detect whether or not exactly zero things were
     298    // marked, but if N things got marked, then this may report anything in the range [1, N] (or
     299    // before unbiased, it would be [1 + m_markCountBias, N + m_markCountBias].)
     300    int16_t m_biasedMarkCount;
     301   
     302    // We bias the mark count so that if m_biasedMarkCount >= 0 then the block should be retired.
     303    // We go to all this trouble to make marking a bit faster: this way, marking knows when to
     304    // retire a block using a js/jns on m_biasedMarkCount.
     305    //
     306    // For example, if a block has room for 100 objects and retirement happens whenever 90% are
     307    // live, then m_markCountBias will be -90. This way, when marking begins, this will cause us to
     308    // set m_biasedMarkCount to -90 as well, since:
     309    //
     310    //     m_biasedMarkCount = actualMarkCount + m_markCountBias.
     311    //
     312    // Marking an object will increment m_biasedMarkCount. Once 90 objects get marked, we will have
     313    // m_biasedMarkCount = 0, which will trigger retirement. In other words, we want to set
     314    // m_markCountBias like so:
     315    //
     316    //     m_markCountBias = -(minMarkedBlockUtilization * cellsPerBlock)
     317    //
     318    // All of this also means that you can detect if any objects are marked by doing:
     319    //
     320    //     m_biasedMarkCount != m_markCountBias
     321    int16_t m_markCountBias;
     322   
     323    Handle& m_handle;
     324    VM* m_vm;
     325       
     326    uint64_t m_version;
     327};
     328
     329inline MarkedBlock::Handle& MarkedBlock::handle()
     330{
     331    return m_handle;
     332}
     333
     334inline MarkedBlock& MarkedBlock::Handle::block()
     335{
     336    return *m_block;
     337}
     338
     339inline size_t MarkedBlock::firstAtom()
     340{
     341    return WTF::roundUpToMultipleOf<atomSize>(sizeof(MarkedBlock)) / atomSize;
     342}
     343
     344inline MarkedBlock::Atom* MarkedBlock::atoms()
     345{
     346    return reinterpret_cast<Atom*>(this);
     347}
     348
     349inline bool MarkedBlock::isAtomAligned(const void* p)
     350{
     351    return !(reinterpret_cast<Bits>(p) & atomAlignmentMask);
     352}
     353
     354inline void* MarkedBlock::Handle::cellAlign(void* p)
     355{
     356    Bits base = reinterpret_cast<Bits>(block().atoms() + firstAtom());
     357    Bits bits = reinterpret_cast<Bits>(p);
     358    bits -= base;
     359    bits -= bits % cellSize();
     360    bits += base;
     361    return reinterpret_cast<void*>(bits);
     362}
     363
     364inline MarkedBlock* MarkedBlock::blockFor(const void* p)
     365{
     366    return reinterpret_cast<MarkedBlock*>(reinterpret_cast<Bits>(p) & blockMask);
     367}
     368
     369inline MarkedAllocator* MarkedBlock::Handle::allocator() const
     370{
     371    return m_allocator;
     372}
     373
     374inline Heap* MarkedBlock::Handle::heap() const
     375{
     376    return m_weakSet.heap();
     377}
     378
     379inline VM* MarkedBlock::Handle::vm() const
     380{
     381    return m_weakSet.vm();
     382}
     383
     384inline VM* MarkedBlock::vm() const
     385{
     386    return m_vm;
     387}
     388
     389inline WeakSet& MarkedBlock::Handle::weakSet()
     390{
     391    return m_weakSet;
     392}
     393
     394inline WeakSet& MarkedBlock::weakSet()
     395{
     396    return m_handle.weakSet();
     397}
     398
     399inline void MarkedBlock::Handle::shrink()
     400{
     401    m_weakSet.shrink();
     402}
     403
     404inline unsigned MarkedBlock::Handle::visitWeakSet(HeapRootVisitor& heapRootVisitor)
     405{
     406    return m_weakSet.visit(heapRootVisitor);
     407}
     408
     409inline void MarkedBlock::Handle::reapWeakSet()
     410{
     411    m_weakSet.reap();
     412}
     413
     414inline size_t MarkedBlock::Handle::cellSize()
     415{
     416    return m_atomsPerCell * atomSize;
     417}
     418
     419inline size_t MarkedBlock::cellSize()
     420{
     421    return m_handle.cellSize();
     422}
     423
     424inline const AllocatorAttributes& MarkedBlock::Handle::attributes() const
     425{
     426    return m_attributes;
     427}
     428
     429inline const AllocatorAttributes& MarkedBlock::attributes() const
     430{
     431    return m_handle.attributes();
     432}
     433
     434inline bool MarkedBlock::Handle::needsDestruction() const
     435{
     436    return m_attributes.destruction == NeedsDestruction;
     437}
     438
     439inline DestructionMode MarkedBlock::Handle::destruction() const
     440{
     441    return m_attributes.destruction;
     442}
     443
     444inline HeapCell::Kind MarkedBlock::Handle::cellKind() const
     445{
     446    return m_attributes.cellKind;
     447}
     448
     449inline size_t MarkedBlock::Handle::markCount()
     450{
     451    return m_block->markCount();
     452}
     453
     454inline size_t MarkedBlock::Handle::size()
     455{
     456    return markCount() * cellSize();
     457}
     458
     459inline size_t MarkedBlock::atomNumber(const void* p)
     460{
     461    return (reinterpret_cast<Bits>(p) - reinterpret_cast<Bits>(this)) / atomSize;
     462}
     463
     464inline void MarkedBlock::flipIfNecessary(uint64_t heapVersion)
     465{
     466    if (UNLIKELY(heapVersion != m_version))
     467        flipIfNecessarySlow();
     468}
     469
     470inline void MarkedBlock::flipIfNecessaryConcurrently(uint64_t heapVersion)
     471{
     472    if (UNLIKELY(heapVersion != m_version))
     473        flipIfNecessaryConcurrentlySlow();
     474    WTF::loadLoadFence();
     475}
     476
     477inline void MarkedBlock::Handle::flipIfNecessary(uint64_t heapVersion)
     478{
     479    block().flipIfNecessary(heapVersion);
     480}
     481
     482inline void MarkedBlock::Handle::flipIfNecessaryConcurrently(uint64_t heapVersion)
     483{
     484    block().flipIfNecessaryConcurrently(heapVersion);
     485}
     486
     487inline void MarkedBlock::Handle::flipForEdenCollection()
     488{
     489    assertFlipped();
     490       
     491    HEAP_LOG_BLOCK_STATE_TRANSITION(this);
     492   
     493    ASSERT(m_state != New && m_state != FreeListed);
     494   
     495    m_state = Marked;
     496}
     497
     498#if ASSERT_DISABLED
     499inline void MarkedBlock::assertFlipped()
     500{
     501}
     502#endif // ASSERT_DISABLED
     503
     504inline void MarkedBlock::Handle::assertFlipped()
     505{
     506    block().assertFlipped();
     507}
     508
     509inline bool MarkedBlock::isMarked(const void* p)
     510{
     511    assertFlipped();
     512    return m_marks.get(atomNumber(p));
     513}
     514
     515inline bool MarkedBlock::testAndSetMarked(const void* p)
     516{
     517    assertFlipped();
     518    return m_marks.concurrentTestAndSet(atomNumber(p));
     519}
     520
     521inline bool MarkedBlock::Handle::isNewlyAllocated(const void* p)
     522{
     523    return m_newlyAllocated->get(m_block->atomNumber(p));
     524}
     525
     526inline void MarkedBlock::Handle::setNewlyAllocated(const void* p)
     527{
     528    m_newlyAllocated->set(m_block->atomNumber(p));
     529}
     530
     531inline void MarkedBlock::Handle::clearNewlyAllocated(const void* p)
     532{
     533    m_newlyAllocated->clear(m_block->atomNumber(p));
     534}
     535
     536inline bool MarkedBlock::Handle::clearNewlyAllocated()
     537{
     538    if (m_newlyAllocated) {
     539        m_newlyAllocated = nullptr;
     540        return true;
    226541    }
    227 
    228     inline MarkedBlock::FreeList::FreeList(FreeCell* head, size_t bytes)
    229         : head(head)
    230         , bytes(bytes)
    231     {
    232     }
    233 
    234     inline size_t MarkedBlock::firstAtom()
    235     {
    236         return WTF::roundUpToMultipleOf<atomSize>(sizeof(MarkedBlock)) / atomSize;
    237     }
    238 
    239     inline MarkedBlock::Atom* MarkedBlock::atoms()
    240     {
    241         return reinterpret_cast<Atom*>(this);
    242     }
    243 
    244     inline bool MarkedBlock::isAtomAligned(const void* p)
    245     {
    246         return !(reinterpret_cast<Bits>(p) & atomAlignmentMask);
    247     }
    248 
    249     inline MarkedBlock* MarkedBlock::blockFor(const void* p)
    250     {
    251         return reinterpret_cast<MarkedBlock*>(reinterpret_cast<Bits>(p) & blockMask);
    252     }
    253 
    254     inline MarkedAllocator* MarkedBlock::allocator() const
    255     {
    256         return m_allocator;
    257     }
    258 
    259     inline Heap* MarkedBlock::heap() const
    260     {
    261         return m_weakSet.heap();
    262     }
    263 
    264     inline VM* MarkedBlock::vm() const
    265     {
    266         return m_weakSet.vm();
    267     }
    268 
    269     inline WeakSet& MarkedBlock::weakSet()
    270     {
    271         return m_weakSet;
    272     }
    273 
    274     inline void MarkedBlock::shrink()
    275     {
    276         m_weakSet.shrink();
    277     }
    278 
    279     inline void MarkedBlock::visitWeakSet(HeapRootVisitor& heapRootVisitor)
    280     {
    281         m_weakSet.visit(heapRootVisitor);
    282     }
    283 
    284     inline void MarkedBlock::reapWeakSet()
    285     {
    286         m_weakSet.reap();
    287     }
    288 
    289     inline void MarkedBlock::willRemoveBlock()
    290     {
    291         ASSERT(m_state != Retired);
    292     }
    293 
    294     inline void MarkedBlock::didConsumeFreeList()
    295     {
    296         HEAP_LOG_BLOCK_STATE_TRANSITION(this);
    297 
    298         ASSERT(m_state == FreeListed);
    299         m_state = Allocated;
    300     }
    301 
    302     inline size_t MarkedBlock::markCount()
    303     {
    304         return m_marks.count();
    305     }
    306 
    307     inline bool MarkedBlock::isEmpty()
    308     {
    309         return m_marks.isEmpty() && m_weakSet.isEmpty() && (!m_newlyAllocated || m_newlyAllocated->isEmpty());
    310     }
    311 
    312     inline size_t MarkedBlock::cellSize()
    313     {
    314         return m_atomsPerCell * atomSize;
    315     }
    316 
    317     inline const AllocatorAttributes& MarkedBlock::attributes() const
    318     {
    319         return m_attributes;
    320     }
    321 
    322     inline bool MarkedBlock::needsDestruction() const
    323     {
    324         return m_attributes.destruction == NeedsDestruction;
    325     }
    326 
    327     inline DestructionMode MarkedBlock::destruction() const
    328     {
    329         return m_attributes.destruction;
    330     }
    331 
    332     inline HeapCell::Kind MarkedBlock::cellKind() const
    333     {
    334         return m_attributes.cellKind;
    335     }
    336 
    337     inline size_t MarkedBlock::size()
    338     {
    339         return markCount() * cellSize();
    340     }
    341 
    342     inline size_t MarkedBlock::capacity()
    343     {
    344         return m_capacity;
    345     }
    346 
    347     inline size_t MarkedBlock::atomNumber(const void* p)
    348     {
    349         return (reinterpret_cast<Bits>(p) - reinterpret_cast<Bits>(this)) / atomSize;
    350     }
    351 
    352     inline bool MarkedBlock::isMarked(const void* p)
    353     {
    354         return m_marks.get(atomNumber(p));
    355     }
    356 
    357     inline bool MarkedBlock::testAndSetMarked(const void* p)
    358     {
    359         return m_marks.concurrentTestAndSet(atomNumber(p));
    360     }
    361 
    362     inline void MarkedBlock::setMarked(const void* p)
    363     {
    364         m_marks.set(atomNumber(p));
    365     }
    366 
    367     inline void MarkedBlock::clearMarked(const void* p)
    368     {
    369         ASSERT(m_marks.get(atomNumber(p)));
    370         m_marks.clear(atomNumber(p));
    371     }
    372 
    373     inline bool MarkedBlock::isNewlyAllocated(const void* p)
    374     {
    375         return m_newlyAllocated->get(atomNumber(p));
    376     }
    377 
    378     inline void MarkedBlock::setNewlyAllocated(const void* p)
    379     {
    380         m_newlyAllocated->set(atomNumber(p));
    381     }
    382 
    383     inline void MarkedBlock::clearNewlyAllocated(const void* p)
    384     {
    385         m_newlyAllocated->clear(atomNumber(p));
    386     }
    387 
    388     inline bool MarkedBlock::clearNewlyAllocated()
    389     {
    390         if (m_newlyAllocated) {
    391             m_newlyAllocated = nullptr;
    392             return true;
    393         }
    394         return false;
    395     }
    396 
    397     inline bool MarkedBlock::isMarkedOrNewlyAllocated(const HeapCell* cell)
    398     {
    399         ASSERT(m_state == Retired || m_state == Marked);
    400         return m_marks.get(atomNumber(cell)) || (m_newlyAllocated && isNewlyAllocated(cell));
    401     }
    402 
    403     inline bool MarkedBlock::isLive(const HeapCell* cell)
    404     {
    405         switch (m_state) {
    406         case Allocated:
    407             return true;
    408 
    409         case Retired:
    410         case Marked:
    411             return isMarkedOrNewlyAllocated(cell);
    412 
    413         case New:
    414         case FreeListed:
    415             RELEASE_ASSERT_NOT_REACHED();
    416             return false;
    417         }
    418 
     542    return false;
     543}
     544
     545inline bool MarkedBlock::Handle::isMarkedOrNewlyAllocated(const HeapCell* cell)
     546{
     547    ASSERT(m_state == Marked);
     548    return m_block->isMarked(cell) || (m_newlyAllocated && isNewlyAllocated(cell));
     549}
     550
     551inline bool MarkedBlock::isMarkedOrNewlyAllocated(const HeapCell* cell)
     552{
     553    ASSERT(m_handle.m_state == Marked);
     554    return isMarked(cell) || (m_handle.m_newlyAllocated && m_handle.isNewlyAllocated(cell));
     555}
     556
     557inline bool MarkedBlock::Handle::isLive(const HeapCell* cell)
     558{
     559    assertFlipped();
     560    switch (m_state) {
     561    case Allocated:
     562        return true;
     563
     564    case Marked:
     565        return isMarkedOrNewlyAllocated(cell);
     566
     567    case New:
     568    case FreeListed:
    419569        RELEASE_ASSERT_NOT_REACHED();
    420570        return false;
    421571    }
    422572
    423     inline bool MarkedBlock::isAtom(const void* p)
     573    RELEASE_ASSERT_NOT_REACHED();
     574    return false;
     575}
     576
     577inline bool MarkedBlock::isAtom(const void* p)
     578{
     579    ASSERT(MarkedBlock::isAtomAligned(p));
     580    size_t atomNumber = this->atomNumber(p);
     581    size_t firstAtom = MarkedBlock::firstAtom();
     582    if (atomNumber < firstAtom) // Filters pointers into MarkedBlock metadata.
     583        return false;
     584    if ((atomNumber - firstAtom) % m_handle.m_atomsPerCell) // Filters pointers into cell middles.
     585        return false;
     586    if (atomNumber >= m_handle.m_endAtom) // Filters pointers into invalid cells out of the range.
     587        return false;
     588    return true;
     589}
     590
     591inline bool MarkedBlock::Handle::isLiveCell(const void* p)
     592{
     593    if (!m_block->isAtom(p))
     594        return false;
     595    return isLive(static_cast<const HeapCell*>(p));
     596}
     597
     598template <typename Functor>
     599inline IterationStatus MarkedBlock::Handle::forEachCell(const Functor& functor)
     600{
     601    HeapCell::Kind kind = m_attributes.cellKind;
     602    for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell) {
     603        HeapCell* cell = reinterpret_cast_ptr<HeapCell*>(&m_block->atoms()[i]);
     604        if (functor(cell, kind) == IterationStatus::Done)
     605            return IterationStatus::Done;
     606    }
     607    return IterationStatus::Continue;
     608}
     609
     610template <typename Functor>
     611inline IterationStatus MarkedBlock::Handle::forEachLiveCell(const Functor& functor)
     612{
     613    flipIfNecessary();
     614    HeapCell::Kind kind = m_attributes.cellKind;
     615    for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell) {
     616        HeapCell* cell = reinterpret_cast_ptr<HeapCell*>(&m_block->atoms()[i]);
     617        if (!isLive(cell))
     618            continue;
     619
     620        if (functor(cell, kind) == IterationStatus::Done)
     621            return IterationStatus::Done;
     622    }
     623    return IterationStatus::Continue;
     624}
     625
     626template <typename Functor>
     627inline IterationStatus MarkedBlock::Handle::forEachDeadCell(const Functor& functor)
     628{
     629    flipIfNecessary();
     630    HeapCell::Kind kind = m_attributes.cellKind;
     631    for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell) {
     632        HeapCell* cell = reinterpret_cast_ptr<HeapCell*>(&m_block->atoms()[i]);
     633        if (isLive(cell))
     634            continue;
     635
     636        if (functor(cell, kind) == IterationStatus::Done)
     637            return IterationStatus::Done;
     638    }
     639    return IterationStatus::Continue;
     640}
     641
     642inline bool MarkedBlock::Handle::needsSweeping() const
     643{
     644    const_cast<MarkedBlock::Handle*>(this)->flipIfNecessary();
     645    return m_state == Marked;
     646}
     647
     648inline bool MarkedBlock::Handle::isAllocated() const
     649{
     650    const_cast<MarkedBlock::Handle*>(this)->flipIfNecessary();
     651    return m_state == Allocated;
     652}
     653
     654inline bool MarkedBlock::Handle::isMarked() const
     655{
     656    const_cast<MarkedBlock::Handle*>(this)->flipIfNecessary();
     657    return m_state == Marked;
     658}
     659
     660inline bool MarkedBlock::Handle::isFreeListed() const
     661{
     662    const_cast<MarkedBlock::Handle*>(this)->flipIfNecessary();
     663    return m_state == FreeListed;
     664}
     665
     666inline bool MarkedBlock::hasAnyMarked() const
     667{
     668    return m_biasedMarkCount != m_markCountBias;
     669}
     670
     671inline void MarkedBlock::noteMarked()
     672{
     673    // This is racy by design. We don't want to pay the price of an atomic increment!
     674    int16_t biasedMarkCount = m_biasedMarkCount;
     675    ++biasedMarkCount;
     676    m_biasedMarkCount = biasedMarkCount;
     677    if (UNLIKELY(!biasedMarkCount))
     678        noteMarkedSlow();
     679}
     680
     681} // namespace JSC
     682
     683namespace WTF {
     684
     685struct MarkedBlockHash : PtrHash<JSC::MarkedBlock*> {
     686    static unsigned hash(JSC::MarkedBlock* const& key)
    424687    {
    425         ASSERT(MarkedBlock::isAtomAligned(p));
    426         size_t atomNumber = this->atomNumber(p);
    427         size_t firstAtom = this->firstAtom();
    428         if (atomNumber < firstAtom) // Filters pointers into MarkedBlock metadata.
    429             return false;
    430         if ((atomNumber - firstAtom) % m_atomsPerCell) // Filters pointers into cell middles.
    431             return false;
    432         if (atomNumber >= m_endAtom) // Filters pointers into invalid cells out of the range.
    433             return false;
    434         return true;
     688        // Aligned VM regions tend to be monotonically increasing integers,
     689        // which is a great hash function, but we have to remove the low bits,
     690        // since they're always zero, which is a terrible hash function!
     691        return reinterpret_cast<JSC::Bits>(key) / JSC::MarkedBlock::blockSize;
    435692    }
    436 
    437     inline bool MarkedBlock::isLiveCell(const void* p)
    438     {
    439         if (!isAtom(p))
    440             return false;
    441         return isLive(static_cast<const HeapCell*>(p));
    442     }
    443 
    444     template <typename Functor> inline IterationStatus MarkedBlock::forEachCell(const Functor& functor)
    445     {
    446         HeapCell::Kind kind = m_attributes.cellKind;
    447         for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell) {
    448             HeapCell* cell = reinterpret_cast_ptr<HeapCell*>(&atoms()[i]);
    449             if (functor(cell, kind) == IterationStatus::Done)
    450                 return IterationStatus::Done;
    451         }
    452         return IterationStatus::Continue;
    453     }
    454 
    455     template <typename Functor> inline IterationStatus MarkedBlock::forEachLiveCell(const Functor& functor)
    456     {
    457         HeapCell::Kind kind = m_attributes.cellKind;
    458         for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell) {
    459             HeapCell* cell = reinterpret_cast_ptr<HeapCell*>(&atoms()[i]);
    460             if (!isLive(cell))
    461                 continue;
    462 
    463             if (functor(cell, kind) == IterationStatus::Done)
    464                 return IterationStatus::Done;
    465         }
    466         return IterationStatus::Continue;
    467     }
    468 
    469     template <typename Functor> inline IterationStatus MarkedBlock::forEachDeadCell(const Functor& functor)
    470     {
    471         HeapCell::Kind kind = m_attributes.cellKind;
    472         for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell) {
    473             HeapCell* cell = reinterpret_cast_ptr<HeapCell*>(&atoms()[i]);
    474             if (isLive(cell))
    475                 continue;
    476 
    477             if (functor(cell, kind) == IterationStatus::Done)
    478                 return IterationStatus::Done;
    479         }
    480         return IterationStatus::Continue;
    481     }
    482 
    483     inline bool MarkedBlock::needsSweeping() const
    484     {
    485         return m_state == Marked;
    486     }
    487 
    488     inline bool MarkedBlock::isAllocated() const
    489     {
    490         return m_state == Allocated;
    491     }
    492 
    493     inline bool MarkedBlock::isMarkedOrRetired() const
    494     {
    495         return m_state == Marked || m_state == Retired;
    496     }
    497 
    498 } // namespace JSC
    499 
    500 namespace WTF {
    501 
    502     struct MarkedBlockHash : PtrHash<JSC::MarkedBlock*> {
    503         static unsigned hash(JSC::MarkedBlock* const& key)
    504         {
    505             // Aligned VM regions tend to be monotonically increasing integers,
    506             // which is a great hash function, but we have to remove the low bits,
    507             // since they're always zero, which is a terrible hash function!
    508             return reinterpret_cast<JSC::Bits>(key) / JSC::MarkedBlock::blockSize;
    509         }
    510     };
    511 
    512     template<> struct DefaultHash<JSC::MarkedBlock*> {
    513         typedef MarkedBlockHash Hash;
    514     };
     693};
     694
     695template<> struct DefaultHash<JSC::MarkedBlock*> {
     696    typedef MarkedBlockHash Hash;
     697};
     698
     699void printInternal(PrintStream& out, JSC::MarkedBlock::BlockState);
    515700
    516701} // namespace WTF

  • trunk/Source/JavaScriptCore/heap/MarkedSpace.cpp

    r204912 r205462  
    2525#include "JSObject.h"
    2626#include "JSCInlines.h"
     27#include "SuperSampler.h"
     28#include <wtf/ListDump.h>
    2729
    2830namespace JSC {
     31
     32std::array<size_t, MarkedSpace::numSizeClasses> MarkedSpace::s_sizeClassForSizeStep;
     33
     34namespace {
     35
     36const Vector<size_t>& sizeClasses()
     37{
     38    static Vector<size_t>* result;
     39    static std::once_flag once;
     40    std::call_once(
     41        once,
     42        [] {
     43            result = new Vector<size_t>();
     44           
     45            auto add = [&] (size_t sizeClass) {
     46                if (Options::dumpSizeClasses())
     47                    dataLog("Adding JSC MarkedSpace size class: ", sizeClass, "\n");
     48                // Perform some validation as we go.
     49                RELEASE_ASSERT(!(sizeClass % MarkedSpace::sizeStep));
     50                if (result->isEmpty())
     51                    RELEASE_ASSERT(sizeClass == MarkedSpace::sizeStep);
     52                else
     53                    RELEASE_ASSERT(sizeClass > result->last());
     54                result->append(sizeClass);
     55            };
     56           
     57            // This is a definition of the size classes in our GC. It must define all of the
     58            // size classes from sizeStep up to largeCutoff.
     59   
     60            // Have very precise size classes for the small stuff. This is a loop to make it easy to reduce
     61            // atomSize.
     62            for (size_t size = MarkedSpace::sizeStep; size < MarkedSpace::preciseCutoff; size += MarkedSpace::sizeStep)
     63                add(size);
     64           
     65            // We want to make sure that the remaining size classes minimize internal fragmentation (i.e.
     66            // the wasted space at the tail end of a MarkedBlock) while proceeding roughly in an exponential
     67            // way starting at just above the precise size classes to four cells per block.
     68           
     69            if (Options::dumpSizeClasses())
     70                dataLog("    Marked block payload size: ", static_cast<size_t>(MarkedSpace::blockPayload), "\n");
     71           
     72            for (unsigned i = 0; ; ++i) {
     73                double approximateSize = MarkedSpace::preciseCutoff * pow(Options::sizeClassProgression(), i);
     74               
     75                if (Options::dumpSizeClasses())
     76                    dataLog("    Next size class as a double: ", approximateSize, "\n");
     77       
     78                size_t approximateSizeInBytes = static_cast<size_t>(approximateSize);
     79       
     80                if (Options::dumpSizeClasses())
     81                    dataLog("    Next size class as bytes: ", approximateSizeInBytes, "\n");
     82       
     83                // Make sure that the computer did the math correctly.
     84                RELEASE_ASSERT(approximateSizeInBytes >= MarkedSpace::preciseCutoff);
     85               
     86                if (approximateSizeInBytes > MarkedSpace::largeCutoff)
     87                    break;
     88               
     89                size_t sizeClass =
     90                    WTF::roundUpToMultipleOf<MarkedSpace::sizeStep>(approximateSizeInBytes);
     91               
     92                if (Options::dumpSizeClasses())
     93                    dataLog("    Size class: ", sizeClass, "\n");
     94               
     95                // Optimize the size class so that there isn't any slop at the end of the block's
     96                // payload.
     97                unsigned cellsPerBlock = MarkedSpace::blockPayload / sizeClass;
     98                size_t possiblyBetterSizeClass = (MarkedSpace::blockPayload / cellsPerBlock) & ~(MarkedSpace::sizeStep - 1);
     99               
     100                if (Options::dumpSizeClasses())
     101                    dataLog("    Possibly better size class: ", possiblyBetterSizeClass, "\n");
     102
     103                // The size class we just came up with is better than the other one if it reduces
     104                // total wastage assuming we only allocate cells of that size.
     105                size_t originalWastage = MarkedSpace::blockPayload - cellsPerBlock * sizeClass;
     106                size_t newWastage = (possiblyBetterSizeClass - sizeClass) * cellsPerBlock;
     107               
     108                if (Options::dumpSizeClasses())
     109                    dataLog("    Original wastage: ", originalWastage, ", new wastage: ", newWastage, "\n");
     110               
     111                size_t betterSizeClass;
     112                if (newWastage > originalWastage)
     113                    betterSizeClass = sizeClass;
     114                else
     115                    betterSizeClass = possiblyBetterSizeClass;
     116               
     117                if (Options::dumpSizeClasses())
     118                    dataLog("    Choosing size class: ", betterSizeClass, "\n");
     119               
     120                if (betterSizeClass == result->last()) {
     121                    // Defense for when expStep is small.
     122                    continue;
     123                }
     124               
     125                // This is usually how we get out of the loop.
     126                if (betterSizeClass > MarkedSpace::largeCutoff
     127                    || betterSizeClass > Options::largeAllocationCutoff())
     128                    break;
     129               
     130                add(betterSizeClass);
     131            }
     132           
     133            if (Options::dumpSizeClasses())
     134                dataLog("JSC Heap MarkedSpace size class dump: ", listDump(*result), "\n");
     135
     136            // We have an optimiation in MarkedSpace::optimalSizeFor() that assumes things about
     137            // the size class table. This checks our results against that function's assumptions.
     138            for (size_t size = MarkedSpace::sizeStep, i = 0; size <= MarkedSpace::preciseCutoff; size += MarkedSpace::sizeStep, i++)
     139                RELEASE_ASSERT(result->at(i) == size);
     140        });
     141    return *result;
     142}
     143
     144template<typename TableType, typename SizeClassCons, typename DefaultCons>
     145void buildSizeClassTable(TableType& table, const SizeClassCons& cons, const DefaultCons& defaultCons)
     146{
     147    size_t nextIndex = 0;
     148    for (size_t sizeClass : sizeClasses()) {
     149        auto entry = cons(sizeClass);
     150        size_t index = MarkedSpace::sizeClassToIndex(sizeClass);
     151        for (size_t i = nextIndex; i <= index; ++i)
     152            table[i] = entry;
     153        nextIndex = index + 1;
     154    }
     155    for (size_t i = nextIndex; i < MarkedSpace::numSizeClasses; ++i)
     156        table[i] = defaultCons(MarkedSpace::indexToSizeClass(i));
     157}
     158
     159} // anonymous namespace
     160
     161void MarkedSpace::initializeSizeClassForStepSize()
     162{
     163    // We call this multiple times and we may call it simultaneously from multiple threads. That's
     164    // OK, since it always stores the same values into the table.
     165   
     166    buildSizeClassTable(
     167        s_sizeClassForSizeStep,
     168        [&] (size_t sizeClass) -> size_t {
     169            return sizeClass;
     170        },
     171        [&] (size_t sizeClass) -> size_t {
     172            return sizeClass;
     173        });
     174}
    29175
    30176MarkedSpace::MarkedSpace(Heap* heap)
    … …  
    33179    , m_isIterating(false)
    34180{
    35     forEachAllocator(
    36         [&] (MarkedAllocator& allocator, size_t cellSize, AllocatorAttributes attributes) -> IterationStatus {
    37             allocator.init(heap, this, cellSize, attributes);
     181    initializeSizeClassForStepSize();
     182   
     183    forEachSubspace(
     184        [&] (Subspace& subspace, AllocatorAttributes attributes) -> IterationStatus {
     185            subspace.attributes = attributes;
     186           
     187            buildSizeClassTable(
     188                subspace.allocatorForSizeStep,
     189                [&] (size_t sizeClass) -> MarkedAllocator* {
     190                    return subspace.bagOfAllocators.add(heap, this, sizeClass, attributes);
     191                },
     192                [&] (size_t) -> MarkedAllocator* {
     193                    return nullptr;
     194                });
     195           
    38196            return IterationStatus::Continue;
    39197        });
    … …  
    43201{
    44202    forEachBlock(
    45         [&] (MarkedBlock* block) {
     203        [&] (MarkedBlock::Handle* block) {
    46204            freeBlock(block);
    47205        });
    … …  
    53211    stopAllocating();
    54212    forEachAllocator(
    55         [&] (MarkedAllocator& allocator, size_t, AllocatorAttributes) -> IterationStatus {
     213        [&] (MarkedAllocator& allocator) -> IterationStatus {
    56214            allocator.lastChanceToFinalize();
    57215            return IterationStatus::Continue;
    58216        });
     217    for (LargeAllocation* allocation : m_largeAllocations)
     218        allocation->lastChanceToFinalize();
     219}
     220
     221void* MarkedSpace::allocate(Subspace& subspace, size_t bytes)
     222{
     223    if (MarkedAllocator* allocator = allocatorFor(subspace, bytes))
     224        return allocator->allocate();
     225    return allocateLarge(subspace, bytes);
     226}
     227
     228void* MarkedSpace::tryAllocate(Subspace& subspace, size_t bytes)
     229{
     230    if (MarkedAllocator* allocator = allocatorFor(subspace, bytes))
     231        return allocator->tryAllocate();
     232    return tryAllocateLarge(subspace, bytes);
     233}
     234
     235void* MarkedSpace::allocateLarge(Subspace& subspace, size_t size)
     236{
     237    void* result = tryAllocateLarge(subspace, size);
     238    RELEASE_ASSERT(result);
     239    return result;
     240}
     241
     242void* MarkedSpace::tryAllocateLarge(Subspace& subspace, size_t size)
     243{
     244    m_heap->collectIfNecessaryOrDefer();
     245   
     246    size = WTF::roundUpToMultipleOf<sizeStep>(size);
     247    LargeAllocation* allocation = LargeAllocation::tryCreate(*m_heap, size, subspace.attributes);
     248    if (!allocation)
     249        return nullptr;
     250   
     251    m_largeAllocations.append(allocation);
     252    m_heap->didAllocate(size);
     253    m_capacity += size;
     254    return allocation->cell();
    59255}
    60256
    … …  
    63259    m_heap->sweeper()->willFinishSweeping();
    64260    forEachBlock(
    65         [&] (MarkedBlock* block) {
     261        [&] (MarkedBlock::Handle* block) {
    66262            block->sweep();
    67263        });
     264}
     265
     266void MarkedSpace::sweepLargeAllocations()
     267{
     268    RELEASE_ASSERT(m_largeAllocationsNurseryOffset == m_largeAllocations.size());
     269    unsigned srcIndex = m_largeAllocationsNurseryOffsetForSweep;
     270    unsigned dstIndex = srcIndex;
     271    while (srcIndex < m_largeAllocations.size()) {
     272        LargeAllocation* allocation = m_largeAllocations[srcIndex++];
     273        allocation->sweep();
     274        if (allocation->isEmpty()) {
     275            m_capacity -= allocation->cellSize();
     276            allocation->destroy();
     277            continue;
     278        }
     279        m_largeAllocations[dstIndex++] = allocation;
     280    }
     281    m_largeAllocations.resize(dstIndex);
     282    m_largeAllocationsNurseryOffset = m_largeAllocations.size();
    68283}
    69284
    … …  
    74289    m_heap->sweeper()->willFinishSweeping();
    75290    forEachBlock(
    76         [&] (MarkedBlock* block) {
     291        [&] (MarkedBlock::Handle* block) {
    77292            if (block->needsSweeping())
    78293                block->sweep();
    … …  
    83298{
    84299    forEachAllocator(
    85         [&] (MarkedAllocator& allocator, size_t, AllocatorAttributes) -> IterationStatus {
     300        [&] (MarkedAllocator& allocator) -> IterationStatus {
    86301            allocator.reset();
    87302            return IterationStatus::Continue;
    … …  
    89304
    90305    m_blocksWithNewObjects.clear();
     306    m_activeWeakSets.takeFrom(m_newActiveWeakSets);
     307    if (m_heap->operationInProgress() == EdenCollection)
     308        m_largeAllocationsNurseryOffsetForSweep = m_largeAllocationsNurseryOffset;
     309    else
     310        m_largeAllocationsNurseryOffsetForSweep = 0;
     311    m_largeAllocationsNurseryOffset = m_largeAllocations.size();
    91312}
    92313
    93314void MarkedSpace::visitWeakSets(HeapRootVisitor& heapRootVisitor)
    94315{
    95     if (m_heap->operationInProgress() == EdenCollection) {
    96         for (unsigned i = 0; i < m_blocksWithNewObjects.size(); ++i)
    97             m_blocksWithNewObjects[i]->visitWeakSet(heapRootVisitor);
    98     } else {
    99         forEachBlock(
    100             [&] (MarkedBlock* block) {
    101                 block->visitWeakSet(heapRootVisitor);
    102             });
    103     }
     316    auto visit = [&] (WeakSet* weakSet) {
     317        weakSet->visit(heapRootVisitor);
     318    };
     319   
     320    m_newActiveWeakSets.forEach(visit);
     321   
     322    if (m_heap->operationInProgress() == FullCollection)
     323        m_activeWeakSets.forEach(visit);
    104324}
    105325
    106326void MarkedSpace::reapWeakSets()
    107327{
    108     if (m_heap->operationInProgress() == EdenCollection) {
    109         for (unsigned i = 0; i < m_blocksWithNewObjects.size(); ++i)
    110             m_blocksWithNewObjects[i]->reapWeakSet();
    111     } else {
    112         forEachBlock(
    113             [&] (MarkedBlock* block) {
    114                 block->reapWeakSet();
    115             });
    116     }
    117 }
    118 
    119 template <typename Functor>
    120 void MarkedSpace::forEachAllocator(const Functor& functor)
    121 {
    122     forEachSubspace(
    123         [&] (Subspace& subspace, AllocatorAttributes attributes) -> IterationStatus {
    124             for (size_t cellSize = preciseStep; cellSize <= preciseCutoff; cellSize += preciseStep) {
    125                 if (functor(allocatorFor(subspace, cellSize), cellSize, attributes) == IterationStatus::Done)
    126                     return IterationStatus::Done;
    127             }
    128             for (size_t cellSize = impreciseStart; cellSize <= impreciseCutoff; cellSize += impreciseStep) {
    129                 if (functor(allocatorFor(subspace, cellSize), cellSize, attributes) == IterationStatus::Done)
    130                     return IterationStatus::Done;
    131             }
    132             if (functor(subspace.largeAllocator, 0, attributes) == IterationStatus::Done)
    133                 return IterationStatus::Done;
    134            
    135             return IterationStatus::Continue;
    136         });
     328    auto visit = [&] (WeakSet* weakSet) {
     329        weakSet->reap();
     330    };
     331   
     332    m_newActiveWeakSets.forEach(visit);
     333   
     334    if (m_heap->operationInProgress() == FullCollection)
     335        m_activeWeakSets.forEach(visit);
    137336}
    138337
    … …  
    141340    ASSERT(!isIterating());
    142341    forEachAllocator(
    143         [&] (MarkedAllocator& allocator, size_t, AllocatorAttributes) -> IterationStatus {
     342        [&] (MarkedAllocator& allocator) -> IterationStatus {
    144343            allocator.stopAllocating();
    145344            return IterationStatus::Continue;
     345        });
     346}
     347
     348void MarkedSpace::prepareForMarking()
     349{
     350    if (m_heap->operationInProgress() == EdenCollection)
     351        m_largeAllocationsOffsetForThisCollection = m_largeAllocationsNurseryOffset;
     352    else
     353        m_largeAllocationsOffsetForThisCollection = 0;
     354    m_largeAllocationsForThisCollectionBegin = m_largeAllocations.begin() + m_largeAllocationsOffsetForThisCollection;
     355    m_largeAllocationsForThisCollectionSize = m_largeAllocations.size() - m_largeAllocationsOffsetForThisCollection;
     356    m_largeAllocationsForThisCollectionEnd = m_largeAllocations.end();
     357    RELEASE_ASSERT(m_largeAllocationsForThisCollectionEnd == m_largeAllocationsForThisCollectionBegin + m_largeAllocationsForThisCollectionSize);
     358    std::sort(
     359        m_largeAllocationsForThisCollectionBegin, m_largeAllocationsForThisCollectionEnd,
     360        [&] (LargeAllocation* a, LargeAllocation* b) {
     361            return a < b;
    146362        });
    147363}
    … …  
    151367    ASSERT(isIterating());
    152368    forEachAllocator(
    153         [&] (MarkedAllocator& allocator, size_t, AllocatorAttributes) -> IterationStatus {
     369        [&] (MarkedAllocator& allocator) -> IterationStatus {
    154370            allocator.resumeAllocating();
    155371            return IterationStatus::Continue;
    156372        });
     373    // Nothing to do for LargeAllocations.
    157374}
    158375
    … …  
    161378    bool result = false;
    162379    forEachAllocator(
    163         [&] (MarkedAllocator& allocator, size_t, AllocatorAttributes) -> IterationStatus {
     380        [&] (MarkedAllocator& allocator) -> IterationStatus {
    164381            if (allocator.isPagedOut(deadline)) {
    165382                result = true;
    … …  
    168385            return IterationStatus::Continue;
    169386        });
     387    // FIXME: Consider taking LargeAllocations into account here.
    170388    return result;
    171389}
    172390
    173 void MarkedSpace::freeBlock(MarkedBlock* block)
     391void MarkedSpace::freeBlock(MarkedBlock::Handle* block)
    174392{
    175393    block->allocator()->removeBlock(block);
    176     m_capacity -= block->capacity();
    177     m_blocks.remove(block);
    178     MarkedBlock::destroy(*m_heap, block);
    179 }
    180 
    181 void MarkedSpace::freeOrShrinkBlock(MarkedBlock* block)
     394    m_capacity -= MarkedBlock::blockSize;
     395    m_blocks.remove(&block->block());
     396    delete block;
     397}
     398
     399void MarkedSpace::freeOrShrinkBlock(MarkedBlock::Handle* block)
    182400{
    183401    if (!block->isEmpty()) {
    … …  
    192410{
    193411    forEachBlock(
    194         [&] (MarkedBlock* block) {
     412        [&] (MarkedBlock::Handle* block) {
    195413            freeOrShrinkBlock(block);
    196414        });
     415    // For LargeAllocations, we do the moral equivalent in sweepLargeAllocations().
    197416}
    198417
    … …  
    200419{
    201420    forEachAllocator(
    202         [&] (MarkedAllocator& allocator, size_t size, AllocatorAttributes) -> IterationStatus {
    203             if (!size) {
    204                 // This means it's a largeAllocator.
    205                 allocator.forEachBlock(
    206                     [&] (MarkedBlock* block) {
    207                         block->clearNewlyAllocated();
    208                     });
    209                 return IterationStatus::Continue;
    210             }
    211            
    212             if (MarkedBlock* block = allocator.takeLastActiveBlock())
     421        [&] (MarkedAllocator& allocator) -> IterationStatus {
     422            if (MarkedBlock::Handle* block = allocator.takeLastActiveBlock())
    213423                block->clearNewlyAllocated();
    214424            return IterationStatus::Continue;
    215425        });
     426   
     427    for (unsigned i = m_largeAllocationsOffsetForThisCollection; i < m_largeAllocations.size(); ++i)
     428        m_largeAllocations[i]->clearNewlyAllocated();
    216429
    217430#if !ASSERT_DISABLED
    218431    forEachBlock(
    219         [&] (MarkedBlock* block) {
     432        [&] (MarkedBlock::Handle* block) {
    220433            ASSERT(!block->clearNewlyAllocated());
    221434        });
     435
     436    for (LargeAllocation* allocation : m_largeAllocations)
     437        ASSERT(!allocation->isNewlyAllocated());
    222438#endif // !ASSERT_DISABLED
    223439}
    224440
    225441#ifndef NDEBUG
    226 struct VerifyMarkedOrRetired : MarkedBlock::VoidFunctor {
    227     void operator()(MarkedBlock* block) const
     442struct VerifyMarked : MarkedBlock::VoidFunctor {
     443    void operator()(MarkedBlock::Handle* block) const
    228444    {
     445        if (block->needsFlip())
     446            return;
    229447        switch (block->m_state) {
    230448        case MarkedBlock::Marked:
    231         case MarkedBlock::Retired:
    232449            return;
    233450        default:
    … …  
    238455#endif
    239456
    240 void MarkedSpace::clearMarks()
     457void MarkedSpace::flip()
    241458{
    242459    if (m_heap->operationInProgress() == EdenCollection) {
    243460        for (unsigned i = 0; i < m_blocksWithNewObjects.size(); ++i)
    244             m_blocksWithNewObjects[i]->clearMarks();
     461            m_blocksWithNewObjects[i]->flipForEdenCollection();
    245462    } else {
    246         forEachBlock(
    247             [&] (MarkedBlock* block) {
    248                 block->clearMarks();
    249             });
     463        m_version++; // Henceforth, flipIfNecessary() will trigger on all blocks.
     464        for (LargeAllocation* allocation : m_largeAllocations)
     465            allocation->flip();
    250466    }
    251467
    252468#ifndef NDEBUG
    253     VerifyMarkedOrRetired verifyFunctor;
     469    VerifyMarked verifyFunctor;
    254470    forEachBlock(verifyFunctor);
    255471#endif
    … …  
    270486}
    271487
     488size_t MarkedSpace::objectCount()
     489{
     490    size_t result = 0;
     491    forEachBlock(
     492        [&] (MarkedBlock::Handle* block) {
     493            result += block->markCount();
     494        });
     495    for (LargeAllocation* allocation : m_largeAllocations) {
     496        if (allocation->isMarked())
     497            result++;
     498    }
     499    return result;
     500}
     501
     502size_t MarkedSpace::size()
     503{
     504    size_t result = 0;
     505    forEachBlock(
     506        [&] (MarkedBlock::Handle* block) {
     507            result += block->markCount() * block->cellSize();
     508        });
     509    for (LargeAllocation* allocation : m_largeAllocations) {
     510        if (allocation->isMarked())
     511            result += allocation->cellSize();
     512    }
     513    return result;
     514}
     515
     516size_t MarkedSpace::capacity()
     517{
     518    return m_capacity;
     519}
     520
     521void MarkedSpace::addActiveWeakSet(WeakSet* weakSet)
     522{
     523    // We conservatively assume that the WeakSet should belong in the new set. In fact, some weak
     524    // sets might contain new weak handles even though they are tied to old objects. This slightly
     525    // increases the amount of scanning that an eden collection would have to do, but the effect
     526    // ought to be small.
     527    m_newActiveWeakSets.append(weakSet);
     528}
     529
     530void MarkedSpace::didAddBlock(MarkedBlock::Handle* block)
     531{
     532    m_capacity += MarkedBlock::blockSize;
     533    m_blocks.add(&block->block());
     534}
     535
     536void MarkedSpace::didAllocateInBlock(MarkedBlock::Handle* block)
     537{
     538    block->assertFlipped();
     539    m_blocksWithNewObjects.append(block);
     540   
     541    if (block->weakSet().isOnList()) {
     542        block->weakSet().remove();
     543        m_newActiveWeakSets.append(&block->weakSet());
     544    }
     545}
     546
    272547} // namespace JSC

  • trunk/Source/JavaScriptCore/heap/MarkedSpace.h

    r204912 r205462  
    2424
    2525#include "IterationStatus.h"
     26#include "LargeAllocation.h"
    2627#include "MarkedAllocator.h"
    2728#include "MarkedBlock.h"
    2829#include "MarkedBlockSet.h"
    2930#include <array>
     31#include <wtf/Bag.h>
    3032#include <wtf/HashSet.h>
    3133#include <wtf/Noncopyable.h>
    3234#include <wtf/RetainPtr.h>
     35#include <wtf/SentinelLinkedList.h>
    3336#include <wtf/Vector.h>
    3437
    … …  
    3841class HeapIterationScope;
    3942class LLIntOffsetsExtractor;
     43class WeakSet;
    4044
    4145class MarkedSpace {
    4246    WTF_MAKE_NONCOPYABLE(MarkedSpace);
    4347public:
    44     // [ 16 ... 768 ]
    45     static const size_t preciseStep = MarkedBlock::atomSize;
    46     static const size_t preciseCutoff = 768;
    47     static const size_t preciseCount = preciseCutoff / preciseStep;
    48 
    49     // [ 1024 ... blockSize/2 ]
    50     static const size_t impreciseStart = 1024;
    51     static const size_t impreciseStep = 256;
    52     static const size_t impreciseCutoff = MarkedBlock::blockSize / 2;
    53     static const size_t impreciseCount = impreciseCutoff / impreciseStep;
    54 
     48    // sizeStep is really a synonym for atomSize; it's no accident that they are the same.
     49    static const size_t sizeStep = MarkedBlock::atomSize;
     50   
     51    // Sizes up to this amount get a size class for each size step.
     52    static const size_t preciseCutoff = 80;
     53   
     54    // The amount of available payload in a block is the block's size minus the header. But the
     55    // header size might not be atom size aligned, so we round down the result accordingly.
     56    static const size_t blockPayload = (MarkedBlock::blockSize - sizeof(MarkedBlock)) & ~(MarkedBlock::atomSize - 1);
     57   
     58    // The largest cell we're willing to allocate in a MarkedBlock the "normal way" (i.e. using size
     59    // classes, rather than a large allocation) is half the size of the payload, rounded down. This
     60    // ensures that we only use the size class approach if it means being able to pack two things
     61    // into one block.
     62    static const size_t largeCutoff = (blockPayload / 2) & ~(sizeStep - 1);
     63
     64    static const size_t numSizeClasses = largeCutoff / sizeStep;
     65   
     66    static size_t sizeClassToIndex(size_t size)
     67    {
     68        ASSERT(size);
     69        return (size + sizeStep - 1) / sizeStep - 1;
     70    }
     71   
     72    static size_t indexToSizeClass(size_t index)
     73    {
     74        return (index + 1) * sizeStep;
     75    }
     76   
     77    // Each Subspace corresponds to all of the blocks for all of the sizes for some "class" of
     78    // objects. There are three classes: non-destructor JSCells, destructor JSCells, and auxiliary.
     79    // MarkedSpace is set up to make it relatively easy to add new Subspaces.
    5580    struct Subspace {
    56         std::array<MarkedAllocator, preciseCount> preciseAllocators;
    57         std::array<MarkedAllocator, impreciseCount> impreciseAllocators;
    58         MarkedAllocator largeAllocator;
     81        std::array<MarkedAllocator*, numSizeClasses> allocatorForSizeStep;
     82       
     83        // Each MarkedAllocator is a size class.
     84        Bag<MarkedAllocator> bagOfAllocators;
     85       
     86        AllocatorAttributes attributes;
    5987    };
    60 
     88   
    6189    MarkedSpace(Heap*);
    6290    ~MarkedSpace();
    6391    void lastChanceToFinalize();
    6492
    65     MarkedAllocator& allocatorFor(size_t);
    66     MarkedAllocator& destructorAllocatorFor(size_t);
    67     MarkedAllocator& auxiliaryAllocatorFor(size_t);
     93    static size_t optimalSizeFor(size_t);
     94   
     95    static MarkedAllocator* allocatorFor(Subspace&, size_t);
     96
     97    MarkedAllocator* allocatorFor(size_t);
     98    MarkedAllocator* destructorAllocatorFor(size_t);
     99    MarkedAllocator* auxiliaryAllocatorFor(size_t);
     100
     101    JS_EXPORT_PRIVATE void* allocate(Subspace&, size_t);
     102    JS_EXPORT_PRIVATE void* tryAllocate(Subspace&, size_t);
     103   
    68104    void* allocateWithDestructor(size_t);
    69105    void* allocateWithoutDestructor(size_t);
    70106    void* allocateAuxiliary(size_t);
    71 
     107    void* tryAllocateAuxiliary(size_t);
     108   
    72109    Subspace& subspaceForObjectsWithDestructor() { return m_destructorSpace; }
    73110    Subspace& subspaceForObjectsWithoutDestructor() { return m_normalSpace; }
    74111    Subspace& subspaceForAuxiliaryData() { return m_auxiliarySpace; }
    75 
     112   
    76113    void resetAllocators();
    77114
    … …  
    87124    void stopAllocating();
    88125    void resumeAllocating(); // If we just stopped allocation but we didn't do a collection, we need to resume allocation.
     126   
     127    void prepareForMarking();
    89128
    90129    typedef HashSet<MarkedBlock*>::iterator BlockIterator;
    … …  
    95134
    96135    void shrink();
    97     void freeBlock(MarkedBlock*);
    98     void freeOrShrinkBlock(MarkedBlock*);
    99 
    100     void didAddBlock(MarkedBlock*);
    101     void didConsumeFreeList(MarkedBlock*);
    102     void didAllocateInBlock(MarkedBlock*);
    103 
    104     void clearMarks();
     136    void freeBlock(MarkedBlock::Handle*);
     137    void freeOrShrinkBlock(MarkedBlock::Handle*);
     138
     139    void didAddBlock(MarkedBlock::Handle*);
     140    void didConsumeFreeList(MarkedBlock::Handle*);
     141    void didAllocateInBlock(MarkedBlock::Handle*);
     142
     143    void flip();
    105144    void clearNewlyAllocated();
    106145    void sweep();
     146    void sweepLargeAllocations();
    107147    void zombifySweep();
    108148    size_t objectCount();
    … …  
    111151
    112152    bool isPagedOut(double deadline);
    113 
    114     const Vector<MarkedBlock*>& blocksWithNewObjects() const { return m_blocksWithNewObjects; }
     153   
     154    uint64_t version() const { return m_version; }
     155
     156    const Vector<MarkedBlock::Handle*>& blocksWithNewObjects() const { return m_blocksWithNewObjects; }
     157   
     158    const Vector<LargeAllocation*>& largeAllocations() const { return m_largeAllocations; }
     159    unsigned largeAllocationsNurseryOffset() const { return m_largeAllocationsNurseryOffset; }
     160    unsigned largeAllocationsOffsetForThisCollection() const { return m_largeAllocationsOffsetForThisCollection; }
     161   
     162    // These are cached pointers and offsets for quickly searching the large allocations that are
     163    // relevant to this collection.
     164    LargeAllocation** largeAllocationsForThisCollectionBegin() const { return m_largeAllocationsForThisCollectionBegin; }
     165    LargeAllocation** largeAllocationsForThisCollectionEnd() const { return m_largeAllocationsForThisCollectionEnd; }
     166    unsigned largeAllocationsForThisCollectionSize() const { return m_largeAllocationsForThisCollectionSize; }
    115167
    116168private:
    117169    friend class LLIntOffsetsExtractor;
    118170    friend class JIT;
     171    friend class WeakSet;
     172   
     173    JS_EXPORT_PRIVATE static std::array<size_t, numSizeClasses> s_sizeClassForSizeStep;
     174   
     175    JS_EXPORT_PRIVATE void* allocateLarge(Subspace&, size_t);
     176    JS_EXPORT_PRIVATE void* tryAllocateLarge(Subspace&, size_t);
     177
     178    static void initializeSizeClassForStepSize();
     179   
     180    void initializeSubspace(Subspace&);
    119181
    120182    template<typename Functor> void forEachAllocator(const Functor&);
    121183    template<typename Functor> void forEachSubspace(const Functor&);
    122     MarkedAllocator& allocatorFor(Subspace&, size_t);
     184   
     185    void addActiveWeakSet(WeakSet*);
    123186
    124187    Subspace m_destructorSpace;
    … …  
    127190
    128191    Heap* m_heap;
     192    uint64_t m_version { 42 }; // This can start at any value, including random garbage values.
    129193    size_t m_capacity;
    130194    bool m_isIterating;
    131195    MarkedBlockSet m_blocks;
    132     Vector<MarkedBlock*> m_blocksWithNewObjects;
     196    Vector<MarkedBlock::Handle*> m_blocksWithNewObjects;
     197    Vector<LargeAllocation*> m_largeAllocations;
     198    unsigned m_largeAllocationsNurseryOffset { 0 };
     199    unsigned m_largeAllocationsOffsetForThisCollection { 0 };
     200    unsigned m_largeAllocationsNurseryOffsetForSweep { 0 };
     201    LargeAllocation** m_largeAllocationsForThisCollectionBegin { nullptr };
     202    LargeAllocation** m_largeAllocationsForThisCollectionEnd { nullptr };
     203    unsigned m_largeAllocationsForThisCollectionSize { 0 };
     204    SentinelLinkedList<WeakSet, BasicRawSentinelNode<WeakSet>> m_activeWeakSets;
     205    SentinelLinkedList<WeakSet, BasicRawSentinelNode<WeakSet>> m_newActiveWeakSets;
    133206};
    134207
    … …  
    138211    BlockIterator end = m_blocks.set().end();
    139212    for (BlockIterator it = m_blocks.set().begin(); it != end; ++it) {
    140         if ((*it)->forEachLiveCell(functor) == IterationStatus::Done)
    141             break;
     213        if ((*it)->handle().forEachLiveCell(functor) == IterationStatus::Done)
     214            return;
     215    }
     216    for (LargeAllocation* allocation : m_largeAllocations) {
     217        if (allocation->isLive()) {
     218            if (functor(allocation->cell(), allocation->attributes().cellKind) == IterationStatus::Done)
     219                return;
     220        }
    142221    }
    143222}
    … …  
    148227    BlockIterator end = m_blocks.set().end();
    149228    for (BlockIterator it = m_blocks.set().begin(); it != end; ++it) {
    150         if ((*it)->forEachDeadCell(functor) == IterationStatus::Done)
    151             break;
    152     }
    153 }
    154 
    155 inline MarkedAllocator& MarkedSpace::allocatorFor(size_t bytes)
     229        if ((*it)->handle().forEachDeadCell(functor) == IterationStatus::Done)
     230            return;
     231    }
     232    for (LargeAllocation* allocation : m_largeAllocations) {
     233        if (!allocation->isLive()) {
     234            if (functor(allocation->cell(), allocation->attributes().cellKind) == IterationStatus::Done)
     235                return;
     236        }
     237    }
     238}
     239
     240inline MarkedAllocator* MarkedSpace::allocatorFor(Subspace& space, size_t bytes)
     241{
     242    ASSERT(bytes);
     243    if (bytes <= largeCutoff)
     244        return space.allocatorForSizeStep[sizeClassToIndex(bytes)];
     245    return nullptr;
     246}
     247
     248inline MarkedAllocator* MarkedSpace::allocatorFor(size_t bytes)
    156249{
    157250    return allocatorFor(m_normalSpace, bytes);
    158251}
    159252
    160 inline MarkedAllocator& MarkedSpace::destructorAllocatorFor(size_t bytes)
     253inline MarkedAllocator* MarkedSpace::destructorAllocatorFor(size_t bytes)
    161254{
    162255    return allocatorFor(m_destructorSpace, bytes);
    163256}
    164257
    165 inline MarkedAllocator& MarkedSpace::auxiliaryAllocatorFor(size_t bytes)
     258inline MarkedAllocator* MarkedSpace::auxiliaryAllocatorFor(size_t bytes)
    166259{
    167260    return allocatorFor(m_auxiliarySpace, bytes);
    … …  
    170263inline void* MarkedSpace::allocateWithoutDestructor(size_t bytes)
    171264{
    172     return allocatorFor(bytes).allocate(bytes);
     265    return allocate(m_normalSpace, bytes);
    173266}
    174267
    175268inline void* MarkedSpace::allocateWithDestructor(size_t bytes)
    176269{
    177     return destructorAllocatorFor(bytes).allocate(bytes);
     270    return allocate(m_destructorSpace, bytes);
    178271}
    179272
    180273inline void* MarkedSpace::allocateAuxiliary(size_t bytes)
    181274{
    182     return auxiliaryAllocatorFor(bytes).allocate(bytes);
     275    return allocate(m_auxiliarySpace, bytes);
     276}
     277
     278inline void* MarkedSpace::tryAllocateAuxiliary(size_t bytes)
     279{
     280    return tryAllocate(m_auxiliarySpace, bytes);
    183281}
    184282
    185283template <typename Functor> inline void MarkedSpace::forEachBlock(const Functor& functor)
     284{
     285    forEachAllocator(
     286        [&] (MarkedAllocator& allocator) -> IterationStatus {
     287            allocator.forEachBlock(functor);
     288            return IterationStatus::Continue;
     289        });
     290}
     291
     292template <typename Functor>
     293void MarkedSpace::forEachAllocator(const Functor& functor)
    186294{
    187295    forEachSubspace(
    188296        [&] (Subspace& subspace, AllocatorAttributes) -> IterationStatus {
    189             for (size_t i = 0; i < preciseCount; ++i)
    190                 subspace.preciseAllocators[i].forEachBlock(functor);
    191             for (size_t i = 0; i < impreciseCount; ++i)
    192                 subspace.impreciseAllocators[i].forEachBlock(functor);
    193             subspace.largeAllocator.forEachBlock(functor);
     297            for (MarkedAllocator* allocator : subspace.bagOfAllocators) {
     298                if (functor(*allocator) == IterationStatus::Done)
     299                    return IterationStatus::Done;
     300            }
     301           
    194302            return IterationStatus::Continue;
    195303        });
    196 }
    197 
    198 inline void MarkedSpace::didAddBlock(MarkedBlock* block)
    199 {
    200     m_capacity += block->capacity();
    201     m_blocks.add(block);
    202 }
    203 
    204 inline void MarkedSpace::didAllocateInBlock(MarkedBlock* block)
    205 {
    206     m_blocksWithNewObjects.append(block);
    207 }
    208 
    209 inline size_t MarkedSpace::objectCount()
    210 {
    211     size_t result = 0;
    212     forEachBlock(
    213         [&] (MarkedBlock* block) {
    214             result += block->markCount();
    215         });
    216     return result;
    217 }
    218 
    219 inline size_t MarkedSpace::size()
    220 {
    221     size_t result = 0;
    222     forEachBlock(
    223         [&] (MarkedBlock* block) {
    224             result += block->markCount() * block->cellSize();
    225         });
    226     return result;
    227 }
    228 
    229 inline size_t MarkedSpace::capacity()
    230 {
    231     return m_capacity;
    232304}
    233305
    … …  
    252324}
    253325
    254 inline MarkedAllocator& MarkedSpace::allocatorFor(Subspace& space, size_t bytes)
     326ALWAYS_INLINE size_t MarkedSpace::optimalSizeFor(size_t bytes)
    255327{
    256328    ASSERT(bytes);
    257329    if (bytes <= preciseCutoff)
    258         return space.preciseAllocators[(bytes - 1) / preciseStep];
    259     if (bytes <= impreciseCutoff)
    260         return space.impreciseAllocators[(bytes - 1) / impreciseStep];
    261     return space.largeAllocator;
     330        return WTF::roundUpToMultipleOf<sizeStep>(bytes);
     331    if (bytes <= largeCutoff)
     332        return s_sizeClassForSizeStep[sizeClassToIndex(bytes)];
     333    return bytes;
    262334}
    263335

  • trunk/Source/JavaScriptCore/heap/SlotVisitor.cpp

    r204912 r205462  
    11/*
    2  * Copyright (C) 2012, 2015 Apple Inc. All rights reserved.
     2 * Copyright (C) 2012, 2015-2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3232#include "CopiedSpace.h"
    3333#include "CopiedSpaceInlines.h"
     34#include "HeapCellInlines.h"
    3435#include "HeapProfiler.h"
    3536#include "HeapSnapshotBuilder.h"
    3637#include "JSArray.h"
    3738#include "JSDestructibleObject.h"
    38 #include "VM.h"
    3939#include "JSObject.h"
    4040#include "JSString.h"
    4141#include "JSCInlines.h"
     42#include "SuperSampler.h"
     43#include "VM.h"
    4244#include <wtf/Lock.h>
    4345
    … …  
    8082    , m_visitCount(0)
    8183    , m_isInParallelMode(false)
     84    , m_version(42)
    8285    , m_heap(heap)
    8386#if !ASSERT_DISABLED
    … …  
    97100    if (heap()->operationInProgress() == FullCollection)
    98101        ASSERT(m_opaqueRoots.isEmpty()); // Should have merged by now.
     102    else
     103        reset();
    99104
    100105    if (HeapProfiler* heapProfiler = vm().heapProfiler())
    101106        m_heapSnapshotBuilder = heapProfiler->activeSnapshotBuilder();
     107   
     108    m_version = heap()->objectSpace().version();
    102109}
    103110
    … …  
    109116    m_heapSnapshotBuilder = nullptr;
    110117    ASSERT(!m_currentCell);
    111     ASSERT(m_stack.isEmpty());
    112118}
    113119
    … …  
    119125void SlotVisitor::append(ConservativeRoots& conservativeRoots)
    120126{
    121     JSCell** roots = conservativeRoots.roots();
     127    HeapCell** roots = conservativeRoots.roots();
    122128    size_t size = conservativeRoots.size();
    123129    for (size_t i = 0; i < size; ++i)
    124         append(roots[i]);
     130        appendJSCellOrAuxiliary(roots[i]);
     131}
     132
     133void SlotVisitor::appendJSCellOrAuxiliary(HeapCell* heapCell)
     134{
     135    if (!heapCell)
     136        return;
     137   
     138    ASSERT(!m_isCheckingForDefaultMarkViolation);
     139   
     140    if (Heap::testAndSetMarked(m_version, heapCell))
     141        return;
     142   
     143    switch (heapCell->cellKind()) {
     144    case HeapCell::JSCell: {
     145        JSCell* jsCell = static_cast<JSCell*>(heapCell);
     146       
     147        if (!jsCell->structure()) {
     148            ASSERT_NOT_REACHED();
     149            return;
     150        }
     151       
     152        jsCell->setCellState(CellState::NewGrey);
     153
     154        appendToMarkStack(jsCell);
     155        return;
     156    }
     157       
     158    case HeapCell::Auxiliary: {
     159        noteLiveAuxiliaryCell(heapCell);
     160        return;
     161    } }
    125162}
    126163
    … …  
    146183void SlotVisitor::setMarkedAndAppendToMarkStack(JSCell* cell)
    147184{
     185    SuperSamplerScope superSamplerScope(false);
     186   
    148187    ASSERT(!m_isCheckingForDefaultMarkViolation);
    149188    if (!cell)
    … …  
    153192    validate(cell);
    154193#endif
    155 
    156     if (Heap::testAndSetMarked(cell) || !cell->structure()) {
    157         ASSERT(cell->structure());
    158         return;
    159     }
    160 
     194   
     195    if (cell->isLargeAllocation())
     196        setMarkedAndAppendToMarkStack(cell->largeAllocation(), cell);
     197    else
     198        setMarkedAndAppendToMarkStack(cell->markedBlock(), cell);
     199}
     200
     201template<typename ContainerType>
     202ALWAYS_INLINE void SlotVisitor::setMarkedAndAppendToMarkStack(ContainerType& container, JSCell* cell)
     203{
     204    container.flipIfNecessaryConcurrently(m_version);
     205   
     206    if (container.testAndSetMarked(cell))
     207        return;
     208   
     209    ASSERT(cell->structure());
     210   
    161211    // Indicate that the object is grey and that:
    162212    // In case of concurrent GC: it's the first time it is grey in this GC cycle.
    163213    // In case of eden collection: it's a new object that became grey rather than an old remembered object.
    164214    cell->setCellState(CellState::NewGrey);
    165 
    166     appendToMarkStack(cell);
     215   
     216    appendToMarkStack(container, cell);
    167217}
    168218
    169219void SlotVisitor::appendToMarkStack(JSCell* cell)
     220{
     221    if (cell->isLargeAllocation())
     222        appendToMarkStack(cell->largeAllocation(), cell);
     223    else
     224        appendToMarkStack(cell->markedBlock(), cell);
     225}
     226
     227template<typename ContainerType>
     228ALWAYS_INLINE void SlotVisitor::appendToMarkStack(ContainerType& container, JSCell* cell)
    170229{
    171230    ASSERT(Heap::isMarked(cell));
    172231    ASSERT(!cell->isZapped());
    173 
     232   
     233    container.noteMarked();
     234   
     235    // FIXME: These "just work" because the GC resets these fields before doing anything else. But
     236    // that won't be the case when we do concurrent GC.
    174237    m_visitCount++;
    175     m_bytesVisited += MarkedBlock::blockFor(cell)->cellSize();
     238    m_bytesVisited += container.cellSize();
     239   
    176240    m_stack.append(cell);
    177241
    178242    if (UNLIKELY(m_heapSnapshotBuilder))
    179243        m_heapSnapshotBuilder->appendNode(cell);
     244}
     245
     246void SlotVisitor::markAuxiliary(const void* base)
     247{
     248    HeapCell* cell = bitwise_cast<HeapCell*>(base);
     249   
     250    if (Heap::testAndSetMarked(m_version, cell)) {
     251        RELEASE_ASSERT(Heap::isMarked(cell));
     252        return;
     253    }
     254   
     255    noteLiveAuxiliaryCell(cell);
     256}
     257
     258void SlotVisitor::noteLiveAuxiliaryCell(HeapCell* cell)
     259{
     260    // We get here once per GC under these circumstances:
     261    //
     262    // Eden collection: if the cell was allocated since the last collection and is live somehow.
     263    //
     264    // Full collection: if the cell is live somehow.
     265   
     266    CellContainer container = cell->cellContainer();
     267   
     268    container.noteMarked();
     269   
     270    m_visitCount++;
     271    m_bytesVisited += container.cellSize();
    180272}
    181273
    … …  
    203295{
    204296    ASSERT(Heap::isMarked(cell));
    205 
     297   
    206298    SetCurrentCellScope currentCellScope(*this, cell);
    207 
     299   
    208300    m_currentObjectCellStateBeforeVisiting = cell->cellState();
    209301    cell->setCellState(CellState::OldBlack);

  • trunk/Source/JavaScriptCore/heap/SlotVisitor.h

    r204912 r205462  
    3838class GCThreadSharedData;
    3939class Heap;
     40class HeapCell;
    4041class HeapSnapshotBuilder;
    4142template<typename T> class JITWriteBarrier;
     43class MarkedBlock;
    4244class UnconditionalFinalizer;
    4345template<typename T> class Weak;
    … …  
    105107    void harvestWeakReferences();
    106108    void finalizeUnconditionalFinalizers();
     109   
     110    // This informs the GC about auxiliary of some size that we are keeping alive. If you don't do
     111    // this then the space will be freed at end of GC.
     112    void markAuxiliary(const void* base);
    107113
    108114    void copyLater(JSCell*, CopyToken, void*, size_t);
    … …  
    124130   
    125131    JS_EXPORT_PRIVATE void append(JSValue); // This is private to encourage clients to use WriteBarrier<T>.
     132    void appendJSCellOrAuxiliary(HeapCell*);
    126133    void appendHidden(JSValue);
    127134
    128135    JS_EXPORT_PRIVATE void setMarkedAndAppendToMarkStack(JSCell*);
     136   
     137    template<typename ContainerType>
     138    void setMarkedAndAppendToMarkStack(ContainerType&, JSCell*);
     139   
    129140    void appendToMarkStack(JSCell*);
     141   
     142    template<typename ContainerType>
     143    void appendToMarkStack(ContainerType&, JSCell*);
     144   
     145    void noteLiveAuxiliaryCell(HeapCell*);
    130146   
    131147    JS_EXPORT_PRIVATE void mergeOpaqueRoots();
    … …  
    144160    size_t m_visitCount;
    145161    bool m_isInParallelMode;
     162   
     163    uint64_t m_version;
    146164   
    147165    Heap& m_heap;

  • trunk/Source/JavaScriptCore/heap/WeakBlock.cpp

    r204912 r205462  
    11/*
    2  * Copyright (C) 2012 Apple Inc. All rights reserved.
     2 * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2727#include "WeakBlock.h"
    2828
     29#include "CellContainerInlines.h"
    2930#include "Heap.h"
    3031#include "HeapRootVisitor.h"
    … …  
    3536namespace JSC {
    3637
    37 WeakBlock* WeakBlock::create(Heap& heap, MarkedBlock& markedBlock)
     38WeakBlock* WeakBlock::create(Heap& heap, CellContainer container)
    3839{
    3940    heap.didAllocateBlock(WeakBlock::blockSize);
    40     return new (NotNull, fastMalloc(blockSize)) WeakBlock(markedBlock);
     41    return new (NotNull, fastMalloc(blockSize)) WeakBlock(container);
    4142}
    4243
    … …  
    4849}
    4950
    50 WeakBlock::WeakBlock(MarkedBlock& markedBlock)
     51WeakBlock::WeakBlock(CellContainer container)
    5152    : DoublyLinkedListNode<WeakBlock>()
    52     , m_markedBlock(&markedBlock)
     53    , m_container(container)
    5354{
    5455    for (size_t i = 0; i < weakImplCount(); ++i) {
    … …  
    102103        return;
    103104
    104     // If this WeakBlock doesn't belong to a MarkedBlock, we won't even be here.
    105     ASSERT(m_markedBlock);
     105    // If this WeakBlock doesn't belong to a CellContainer, we won't even be here.
     106    ASSERT(m_container);
     107   
     108    m_container.flipIfNecessary();
    106109
    107110    // We only visit after marking.
    108     ASSERT(m_markedBlock->isMarkedOrRetired());
     111    ASSERT(m_container.isMarked());
    109112
    110113    SlotVisitor& visitor = heapRootVisitor.visitor();
    … …  
    120123
    121124        const JSValue& jsValue = weakImpl->jsValue();
    122         if (m_markedBlock->isMarkedOrNewlyAllocated(jsValue.asCell()))
     125        if (m_container.isMarkedOrNewlyAllocated(jsValue.asCell()))
    123126            continue;
    124 
     127       
    125128        if (!weakHandleOwner->isReachableFromOpaqueRoots(Handle<Unknown>::wrapSlot(&const_cast<JSValue&>(jsValue)), weakImpl->context(), visitor))
    126129            continue;
    … …  
    136139        return;
    137140
    138     // If this WeakBlock doesn't belong to a MarkedBlock, we won't even be here.
    139     ASSERT(m_markedBlock);
     141    // If this WeakBlock doesn't belong to a CellContainer, we won't even be here.
     142    ASSERT(m_container);
     143   
     144    m_container.flipIfNecessary();
    140145
    141146    // We only reap after marking.
    142     ASSERT(m_markedBlock->isMarkedOrRetired());
     147    ASSERT(m_container.isMarked());
    143148
    144149    for (size_t i = 0; i < weakImplCount(); ++i) {
    … …  
    147152            continue;
    148153
    149         if (m_markedBlock->isMarkedOrNewlyAllocated(weakImpl->jsValue().asCell())) {
     154        if (m_container.isMarkedOrNewlyAllocated(weakImpl->jsValue().asCell())) {
    150155            ASSERT(weakImpl->state() == WeakImpl::Live);
    151156            continue;

  • trunk/Source/JavaScriptCore/heap/WeakBlock.h

    r204912 r205462  
    11/*
    2  * Copyright (C) 2012 Apple Inc. All rights reserved.
     2 * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2727#define WeakBlock_h
    2828
     29#include "CellContainer.h"
    2930#include "WeakImpl.h"
    3031#include <wtf/DoublyLinkedList.h>
    … …  
    3536class Heap;
    3637class HeapRootVisitor;
    37 class MarkedBlock;
    3838
    3939class WeakBlock : public DoublyLinkedListNode<WeakBlock> {
    4040public:
    4141    friend class WTF::DoublyLinkedListNode<WeakBlock>;
    42     static const size_t blockSize = 1 * KB; // 1/16 of MarkedBlock size
     42    static const size_t blockSize = 256; // 1/16 of MarkedBlock size
    4343
    4444    struct FreeCell {
    … …  
    5454    };
    5555
    56     static WeakBlock* create(Heap&, MarkedBlock&);
     56    static WeakBlock* create(Heap&, CellContainer);
    5757    static void destroy(Heap&, WeakBlock*);
    5858
    … …  
    6969
    7070    void lastChanceToFinalize();
    71     void disconnectMarkedBlock() { m_markedBlock = nullptr; }
     71    void disconnectContainer() { m_container = CellContainer(); }
    7272
    7373private:
    7474    static FreeCell* asFreeCell(WeakImpl*);
    7575
    76     explicit WeakBlock(MarkedBlock&);
     76    explicit WeakBlock(CellContainer);
    7777    void finalize(WeakImpl*);
    7878    WeakImpl* weakImpls();
    … …  
    8080    void addToFreeList(FreeCell**, WeakImpl*);
    8181
    82     MarkedBlock* m_markedBlock;
     82    CellContainer m_container;
    8383    WeakBlock* m_prev;
    8484    WeakBlock* m_next;

  • trunk/Source/JavaScriptCore/heap/WeakSet.cpp

    r204912 r205462  
    11/*
    2  * Copyright (C) 2012 Apple Inc. All rights reserved.
     2 * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3535WeakSet::~WeakSet()
    3636{
     37    if (isOnList())
     38        remove();
     39   
    3740    Heap& heap = *this->heap();
    3841    WeakBlock* next = 0;
    … …  
    5457            // If this WeakBlock is logically empty, but still has Weaks pointing into it,
    5558            // we can't destroy it just yet. Detach it from the WeakSet and hand ownership
    56             // to the Heap so we don't pin down the entire 64kB MarkedBlock.
     59            // to the Heap so we don't pin down the entire MarkedBlock or LargeAllocation.
    5760            m_blocks.remove(block);
    5861            heap()->addLogicallyEmptyWeakBlock(block);
    59             block->disconnectMarkedBlock();
     62            block->disconnectContainer();
    6063        }
    6164        block = nextBlock;
    … …  
    6366
    6467    resetAllocator();
     68}
     69
     70void WeakSet::shrink()
     71{
     72    WeakBlock* next;
     73    for (WeakBlock* block = m_blocks.head(); block; block = next) {
     74        next = block->next();
     75
     76        if (block->isEmpty())
     77            removeAllocator(block);
     78    }
     79
     80    resetAllocator();
     81   
     82    if (m_blocks.isEmpty() && isOnList())
     83        remove();
    6584}
    6685
    … …  
    89108WeakBlock::FreeCell* WeakSet::addAllocator()
    90109{
    91     WeakBlock* block = WeakBlock::create(*heap(), m_markedBlock);
     110    if (m_blocks.isEmpty() && !isOnList())
     111        heap()->objectSpace().addActiveWeakSet(this);
     112   
     113    WeakBlock* block = WeakBlock::create(*heap(), m_container);
    92114    heap()->didAllocate(WeakBlock::blockSize);
    93115    m_blocks.append(block);

  • trunk/Source/JavaScriptCore/heap/WeakSet.h

    r204912 r205462  
    11/*
    2  * Copyright (C) 2012 Apple Inc. All rights reserved.
     2 * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2727#define WeakSet_h
    2828
     29#include "CellContainer.h"
    2930#include "WeakBlock.h"
     31#include <wtf/SentinelLinkedList.h>
    3032
    3133namespace JSC {
    3234
    3335class Heap;
    34 class MarkedBlock;
    3536class WeakImpl;
    3637
    37 class WeakSet {
     38class WeakSet : public BasicRawSentinelNode<WeakSet> {
    3839    friend class LLIntOffsetsExtractor;
    3940
    … …  
    4243    static void deallocate(WeakImpl*);
    4344
    44     WeakSet(VM*, MarkedBlock&);
     45    WeakSet(VM*, CellContainer);
    4546    ~WeakSet();
    4647    void lastChanceToFinalize();
     48   
     49    CellContainer container() const { return m_container; }
     50    void setContainer(CellContainer container) { m_container = container; }
    4751
    4852    Heap* heap() const;
    … …  
    5155    bool isEmpty() const;
    5256
    53     void visit(HeapRootVisitor&);
     57    unsigned visit(HeapRootVisitor&);
    5458    void reap();
    5559    void sweep();
    … …  
    6771    DoublyLinkedList<WeakBlock> m_blocks;
    6872    VM* m_vm;
    69     MarkedBlock& m_markedBlock;
     73    CellContainer m_container;
    7074};
    7175
    72 inline WeakSet::WeakSet(VM* vm, MarkedBlock& markedBlock)
     76inline WeakSet::WeakSet(VM* vm, CellContainer container)
    7377    : m_allocator(0)
    7478    , m_nextAllocator(0)
    7579    , m_vm(vm)
    76     , m_markedBlock(markedBlock)
     80    , m_container(container)
    7781{
    7882}
    … …  
    104108}
    105109
    106 inline void WeakSet::visit(HeapRootVisitor& visitor)
     110inline unsigned WeakSet::visit(HeapRootVisitor& visitor)
    107111{
    108     for (WeakBlock* block = m_blocks.head(); block; block = block->next())
     112    unsigned count = 0;
     113    for (WeakBlock* block = m_blocks.head(); block; block = block->next()) {
     114        count++;
    109115        block->visit(visitor);
     116    }
     117    return count;
    110118}
    111119
    … …  
    114122    for (WeakBlock* block = m_blocks.head(); block; block = block->next())
    115123        block->reap();
    116 }
    117 
    118 inline void WeakSet::shrink()
    119 {
    120     WeakBlock* next;
    121     for (WeakBlock* block = m_blocks.head(); block; block = next) {
    122         next = block->next();
    123 
    124         if (block->isEmpty())
    125             removeAllocator(block);
    126     }
    127 
    128     resetAllocator();
    129124}
    130125

  • trunk/Source/JavaScriptCore/heap/WeakSetInlines.h

    r204912 r205462  
    11/*
    2  * Copyright (C) 2012 Apple Inc. All rights reserved.
     2 * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2727#define WeakSetInlines_h
    2828
     29#include "CellContainerInlines.h"
    2930#include "MarkedBlock.h"
    3031
    … …  
    3334inline WeakImpl* WeakSet::allocate(JSValue jsValue, WeakHandleOwner* weakHandleOwner, void* context)
    3435{
    35     WeakSet& weakSet = MarkedBlock::blockFor(jsValue.asCell())->weakSet();
     36    WeakSet& weakSet = jsValue.asCell()->cellContainer().weakSet();
    3637    WeakBlock::FreeCell* allocator = weakSet.m_allocator;
    3738    if (UNLIKELY(!allocator))

  • trunk/Source/JavaScriptCore/inspector/InjectedScriptManager.cpp

    r204912 r205462  
    3636#include "InjectedScriptSource.h"
    3737#include "InspectorValues.h"
     38#include "JSCInlines.h"
    3839#include "JSInjectedScriptHost.h"
    3940#include "JSLock.h"

  • trunk/Source/JavaScriptCore/inspector/JSGlobalObjectInspectorController.cpp

    r205091 r205462  
    3939#include "InspectorHeapAgent.h"
    4040#include "InspectorScriptProfilerAgent.h"
     41#include "JSCInlines.h"
    4142#include "JSGlobalObject.h"
    4243#include "JSGlobalObjectConsoleAgent.h"

  • trunk/Source/JavaScriptCore/inspector/JSJavaScriptCallFrame.cpp

    r205198 r205462  
    3030#include "Error.h"
    3131#include "IdentifierInlines.h"
    32 #include "JSCJSValue.h"
    33 #include "JSCellInlines.h"
     32#include "JSCInlines.h"
    3433#include "JSJavaScriptCallFramePrototype.h"
    3534#include "ObjectConstructor.h"
    36 #include "StructureInlines.h"
    3735
    3836using namespace JSC;

  • trunk/Source/JavaScriptCore/inspector/ScriptDebugServer.cpp

    r204912 r205462  
    3535#include "DebuggerScope.h"
    3636#include "Exception.h"
     37#include "JSCInlines.h"
    3738#include "JSJavaScriptCallFrame.h"
    3839#include "JSLock.h"

  • trunk/Source/JavaScriptCore/inspector/agents/InspectorDebuggerAgent.cpp

    r204912 r205462  
    3636#include "InspectorFrontendRouter.h"
    3737#include "InspectorValues.h"
     38#include "JSCInlines.h"
    3839#include "RegularExpression.h"
    3940#include "ScriptDebugServer.h"

  • trunk/Source/JavaScriptCore/interpreter/CachedCall.h

    r205198 r205462  
    4343            : m_valid(false)
    4444            , m_interpreter(callFrame->interpreter())
    45             , m_entryScope(callFrame->vm(), function->scope()->globalObject())
     45            , m_vm(callFrame->vm())
     46            , m_entryScope(m_vm, function->scope()->globalObject(m_vm))
    4647        {
    4748            VM& vm = m_entryScope.vm();
    … …  
    6869        bool m_valid;
    6970        Interpreter* m_interpreter;
     71        VM& m_vm;
    7072        VMEntryScope m_entryScope;
    7173        ProtoCallFrame m_protoCallFrame;

  • trunk/Source/JavaScriptCore/interpreter/Interpreter.cpp

    r205324 r205462  
    5252#include "JSWithScope.h"
    5353#include "LLIntCLoop.h"
     54#include "LLIntData.h"
    5455#include "LLIntThunks.h"
    5556#include "LiteralParser.h"
    … …  
    8586
    8687namespace JSC {
    87 
    88 intptr_t StackFrame::sourceID() const
    89 {
    90     if (!codeBlock)
    91         return noSourceID;
    92     return codeBlock->ownerScriptExecutable()->sourceID();
    93 }
    94 
    95 String StackFrame::sourceURL() const
    96 {
    97     if (!codeBlock)
    98         return ASCIILiteral("[native code]");
    99 
    100     String sourceURL = codeBlock->ownerScriptExecutable()->sourceURL();
    101     if (!sourceURL.isNull())
    102         return sourceURL;
    103     return emptyString();
    104 }
    105 
    106 String StackFrame::functionName(VM& vm) const
    107 {
    108     if (codeBlock) {
    109         switch (codeBlock->codeType()) {
    110         case EvalCode:
    111             return ASCIILiteral("eval code");
    112         case ModuleCode:
    113             return ASCIILiteral("module code");
    114         case FunctionCode:
    115             break;
    116         case GlobalCode:
    117             return ASCIILiteral("global code");
    118         default:
    119             ASSERT_NOT_REACHED();
    120         }
    121     }
    122     String name;
    123     if (callee)
    124         name = getCalculatedDisplayName(vm, callee.get()).impl();
    125     return name.isNull() ? emptyString() : name;
    126 }
    12788
    12889JSValue eval(CallFrame* callFrame)
    … …  
    275236   
    276237    JSCell* cell = arguments.asCell();
     238
    277239    switch (cell->type()) {
    278240    case DirectArgumentsType:
    … …  
    481443    return opcode >= 0 && opcode <= op_end;
    482444#endif
    483 }
    484 
    485 void StackFrame::computeLineAndColumn(unsigned& line, unsigned& column) const
    486 {
    487     if (!codeBlock) {
    488         line = 0;
    489         column = 0;
    490         return;
    491     }
    492 
    493     int divot = 0;
    494     int unusedStartOffset = 0;
    495     int unusedEndOffset = 0;
    496     codeBlock->expressionRangeForBytecodeOffset(bytecodeOffset, divot, unusedStartOffset, unusedEndOffset, line, column);
    497 
    498     ScriptExecutable* executable = codeBlock->ownerScriptExecutable();
    499     if (executable->hasOverrideLineNumber())
    500         line = executable->overrideLineNumber();
    501 }
    502 
    503 String StackFrame::toString(VM& vm) const
    504 {
    505     StringBuilder traceBuild;
    506     String functionName = this->functionName(vm);
    507     String sourceURL = this->sourceURL();
    508     traceBuild.append(functionName);
    509     if (!sourceURL.isEmpty()) {
    510         if (!functionName.isEmpty())
    511             traceBuild.append('@');
    512         traceBuild.append(sourceURL);
    513         if (codeBlock) {
    514             unsigned line;
    515             unsigned column;
    516             computeLineAndColumn(line, column);
    517 
    518             traceBuild.append(':');
    519             traceBuild.appendNumber(line);
    520             traceBuild.append(':');
    521             traceBuild.appendNumber(column);
    522         }
    523     }
    524     return traceBuild.toString().impl();
    525445}
    526446

  • trunk/Source/JavaScriptCore/interpreter/Interpreter.h

    r204994 r205462  
    3838#include "SourceProvider.h"
    3939#include "StackAlignment.h"
     40#include "StackFrame.h"
    4041#include <wtf/HashMap.h>
    4142#include <wtf/text/StringBuilder.h>
    … …  
    6869    struct UnlinkedInstruction;
    6970
    70     enum UnwindStart { UnwindFromCurrentFrame, UnwindFromCallerFrame };
     71    enum UnwindStart : uint8_t { UnwindFromCurrentFrame, UnwindFromCallerFrame };
    7172
    7273    enum DebugHookID {
    … …  
    8586        StackFrameFunctionCode,
    8687        StackFrameNativeCode
    87     };
    88 
    89     struct StackFrame {
    90         Strong<JSObject> callee;
    91         Strong<CodeBlock> codeBlock;
    92         unsigned bytecodeOffset;
    93 
    94         bool isNative() const { return !codeBlock; }
    95 
    96         void computeLineAndColumn(unsigned& line, unsigned& column) const;
    97         String functionName(VM&) const;
    98         intptr_t sourceID() const;
    99         String sourceURL() const;
    100         String toString(VM&) const;
    10188    };
    10289

  • trunk/Source/JavaScriptCore/jit/AssemblyHelpers.h

    r204912 r205462  
    14061406    void emitRandomThunk(GPRReg scratch0, GPRReg scratch1, GPRReg scratch2, GPRReg scratch3, FPRReg result);
    14071407#endif
    1408    
    1409     void emitAllocate(GPRReg resultGPR, GPRReg allocatorGPR, GPRReg scratchGPR, JumpList& slowPath)
    1410     {
    1411         if (Options::forceGCSlowPaths())
     1408
     1409    // Call this if you know that the value held in allocatorGPR is non-null. This DOES NOT mean
     1410    // that allocator is non-null; allocator can be null as a signal that we don't know what the
     1411    // value of allocatorGPR is.
     1412    void emitAllocateWithNonNullAllocator(GPRReg resultGPR, MarkedAllocator* allocator, GPRReg allocatorGPR, GPRReg scratchGPR, JumpList& slowPath)
     1413    {
     1414        // NOTE: This is carefully written so that we can call it while we disallow scratch
     1415        // register usage.
     1416       
     1417        if (Options::forceGCSlowPaths()) {
    14121418            slowPath.append(jump());
     1419            return;
     1420        }
     1421       
     1422        Jump popPath;
     1423        Jump done;
     1424       
     1425        load32(Address(allocatorGPR, MarkedAllocator::offsetOfFreeList() + OBJECT_OFFSETOF(FreeList, remaining)), resultGPR);
     1426        popPath = branchTest32(Zero, resultGPR);
     1427        if (allocator)
     1428            add32(TrustedImm32(-allocator->cellSize()), resultGPR, scratchGPR);
    14131429        else {
    1414             loadPtr(Address(allocatorGPR, MarkedAllocator::offsetOfFreeListHead()), resultGPR);
    1415             slowPath.append(branchTestPtr(Zero, resultGPR));
    1416         }
     1430            move(resultGPR, scratchGPR);
     1431            sub32(Address(allocatorGPR, MarkedAllocator::offsetOfCellSize()), scratchGPR);
     1432        }
     1433        negPtr(resultGPR);
     1434        store32(scratchGPR, Address(allocatorGPR, MarkedAllocator::offsetOfFreeList() + OBJECT_OFFSETOF(FreeList, remaining)));
     1435        Address payloadEndAddr = Address(allocatorGPR, MarkedAllocator::offsetOfFreeList() + OBJECT_OFFSETOF(FreeList, payloadEnd));
     1436        if (isX86())
     1437            addPtr(payloadEndAddr, resultGPR);
     1438        else {
     1439            loadPtr(payloadEndAddr, scratchGPR);
     1440            addPtr(scratchGPR, resultGPR);
     1441        }
     1442       
     1443        done = jump();
     1444       
     1445        popPath.link(this);
     1446       
     1447        loadPtr(Address(allocatorGPR, MarkedAllocator::offsetOfFreeList() + OBJECT_OFFSETOF(FreeList, head)), resultGPR);
     1448        slowPath.append(branchTestPtr(Zero, resultGPR));
    14171449       
    14181450        // The object is half-allocated: we have what we know is a fresh object, but
    14191451        // it's still on the GC's free list.
    14201452        loadPtr(Address(resultGPR), scratchGPR);
    1421         storePtr(scratchGPR, Address(allocatorGPR, MarkedAllocator::offsetOfFreeListHead()));
     1453        storePtr(scratchGPR, Address(allocatorGPR, MarkedAllocator::offsetOfFreeList() + OBJECT_OFFSETOF(FreeList, head)));
     1454       
     1455        done.link(this);
     1456    }
     1457   
     1458    void emitAllocate(GPRReg resultGPR, MarkedAllocator* allocator, GPRReg allocatorGPR, GPRReg scratchGPR, JumpList& slowPath)
     1459    {
     1460        if (!allocator)
     1461            slowPath.append(branchTestPtr(Zero, allocatorGPR));
     1462        emitAllocateWithNonNullAllocator(resultGPR, allocator, allocatorGPR, scratchGPR, slowPath);
    14221463    }
    14231464   
    14241465    template<typename StructureType>
    1425     void emitAllocateJSCell(GPRReg resultGPR, GPRReg allocatorGPR, StructureType structure, GPRReg scratchGPR, JumpList& slowPath)
    1426     {
    1427         emitAllocate(resultGPR, allocatorGPR, scratchGPR, slowPath);
     1466    void emitAllocateJSCell(GPRReg resultGPR, MarkedAllocator* allocator, GPRReg allocatorGPR, StructureType structure, GPRReg scratchGPR, JumpList& slowPath)
     1467    {
     1468        emitAllocate(resultGPR, allocator, allocatorGPR, scratchGPR, slowPath);
    14281469        emitStoreStructureWithTypeInfo(structure, resultGPR, scratchGPR);
    14291470    }
    14301471   
    14311472    template<typename StructureType, typename StorageType>
    1432     void emitAllocateJSObject(GPRReg resultGPR, GPRReg allocatorGPR, StructureType structure, StorageType storage, GPRReg scratchGPR, JumpList& slowPath)
    1433     {
    1434         emitAllocateJSCell(resultGPR, allocatorGPR, structure, scratchGPR, slowPath);
     1473    void emitAllocateJSObject(GPRReg resultGPR, MarkedAllocator* allocator, GPRReg allocatorGPR, StructureType structure, StorageType storage, GPRReg scratchGPR, JumpList& slowPath)
     1474    {
     1475        emitAllocateJSCell(resultGPR, allocator, allocatorGPR, structure, scratchGPR, slowPath);
    14351476        storePtr(storage, Address(resultGPR, JSObject::butterflyOffset()));
    14361477    }
    … …  
    14411482        GPRReg scratchGPR2, JumpList& slowPath, size_t size)
    14421483    {
    1443         MarkedAllocator* allocator = &vm()->heap.allocatorForObjectOfType<ClassType>(size);
     1484        MarkedAllocator* allocator = vm()->heap.allocatorForObjectOfType<ClassType>(size);
     1485        if (!allocator) {
     1486            slowPath.append(jump());
     1487            return;
     1488        }
    14441489        move(TrustedImmPtr(allocator), scratchGPR1);
    1445         emitAllocateJSObject(resultGPR, scratchGPR1, structure, storage, scratchGPR2, slowPath);
     1490        emitAllocateJSObject(resultGPR, allocator, scratchGPR1, structure, storage, scratchGPR2, slowPath);
    14461491    }
    14471492   
    … …  
    14521497    }
    14531498   
     1499    // allocationSize can be aliased with any of the other input GPRs. If it's not aliased then it
     1500    // won't be clobbered.
    14541501    void emitAllocateVariableSized(GPRReg resultGPR, MarkedSpace::Subspace& subspace, GPRReg allocationSize, GPRReg scratchGPR1, GPRReg scratchGPR2, JumpList& slowPath)
    14551502    {
    1456         static_assert(!(MarkedSpace::preciseStep & (MarkedSpace::preciseStep - 1)), "MarkedSpace::preciseStep must be a power of two.");
    1457         static_assert(!(MarkedSpace::impreciseStep & (MarkedSpace::impreciseStep - 1)), "MarkedSpace::impreciseStep must be a power of two.");
    1458        
    1459         add32(TrustedImm32(MarkedSpace::preciseStep - 1), allocationSize);
    1460         Jump notSmall = branch32(AboveOrEqual, allocationSize, TrustedImm32(MarkedSpace::preciseCutoff));
    1461         rshift32(allocationSize, TrustedImm32(getLSBSet(MarkedSpace::preciseStep)), scratchGPR1);
    1462         mul32(TrustedImm32(sizeof(MarkedAllocator)), scratchGPR1, scratchGPR1);
    1463         addPtr(TrustedImmPtr(&subspace.preciseAllocators[0]), scratchGPR1);
    1464 
    1465         Jump selectedSmallSpace = jump();
    1466         notSmall.link(this);
    1467         slowPath.append(branch32(AboveOrEqual, allocationSize, TrustedImm32(MarkedSpace::impreciseCutoff)));
    1468         rshift32(allocationSize, TrustedImm32(getLSBSet(MarkedSpace::impreciseStep)), scratchGPR1);
    1469         mul32(TrustedImm32(sizeof(MarkedAllocator)), scratchGPR1, scratchGPR1);
    1470         addPtr(TrustedImmPtr(&subspace.impreciseAllocators[0]), scratchGPR1);
    1471 
    1472         selectedSmallSpace.link(this);
    1473        
    1474         emitAllocate(resultGPR, scratchGPR1, scratchGPR2, slowPath);
     1503        static_assert(!(MarkedSpace::sizeStep & (MarkedSpace::sizeStep - 1)), "MarkedSpace::sizeStep must be a power of two.");
     1504       
     1505        unsigned stepShift = getLSBSet(MarkedSpace::sizeStep);
     1506       
     1507        add32(TrustedImm32(MarkedSpace::sizeStep - 1), allocationSize, scratchGPR1);
     1508        urshift32(TrustedImm32(stepShift), scratchGPR1);
     1509        slowPath.append(branch32(Above, scratchGPR1, TrustedImm32(MarkedSpace::largeCutoff >> stepShift)));
     1510        move(TrustedImmPtr(&subspace.allocatorForSizeStep[0] - 1), scratchGPR2);
     1511        loadPtr(BaseIndex(scratchGPR2, scratchGPR1, timesPtr()), scratchGPR1);
     1512       
     1513        emitAllocate(resultGPR, nullptr, scratchGPR1, scratchGPR2, slowPath);
    14751514    }
    14761515   

  • trunk/Source/JavaScriptCore/jit/CCallHelpers.h

    r203600 r205462  
    289289
    290290    ALWAYS_INLINE void setupArgumentsWithExecState(TrustedImmPtr arg1, TrustedImm32 arg2, TrustedImm32 arg3)
     291    {
     292        resetCallArguments();
     293        addCallArgument(GPRInfo::callFrameRegister);
     294        addCallArgument(arg1);
     295        addCallArgument(arg2);
     296        addCallArgument(arg3);
     297    }
     298
     299    ALWAYS_INLINE void setupArgumentsWithExecState(TrustedImmPtr arg1, TrustedImm32 arg2, GPRReg arg3)
    291300    {
    292301        resetCallArguments();
    … …  
    14091418    }
    14101419
     1420    ALWAYS_INLINE void setupArgumentsWithExecState(TrustedImmPtr arg1, TrustedImm32 arg2, GPRReg arg3)
     1421    {
     1422        move(arg3, GPRInfo::argumentGPR3);
     1423        move(arg1, GPRInfo::argumentGPR1);
     1424        move(arg2, GPRInfo::argumentGPR2);
     1425        move(GPRInfo::callFrameRegister, GPRInfo::argumentGPR0);
     1426    }
     1427
    14111428    ALWAYS_INLINE void setupArgumentsWithExecState(TrustedImmPtr arg1, TrustedImm32 arg2, TrustedImm32 arg3)
    14121429    {

  • trunk/Source/JavaScriptCore/jit/GCAwareJITStubRoutine.cpp

    r204912 r205462  
    3333#include "Heap.h"
    3434#include "VM.h"
     35#include "JITStubRoutineSet.h"
    3536#include "JSCInlines.h"
    3637#include "SlotVisitor.h"
    … …  
    4647    , m_isJettisoned(false)
    4748{
    48     vm.heap.m_jitStubRoutines.add(this);
     49    vm.heap.m_jitStubRoutines->add(this);
    4950}
    5051

  • trunk/Source/JavaScriptCore/jit/JIT.cpp

    r204994 r205462  
    4949#include "TypeProfilerLog.h"
    5050#include <wtf/CryptographicallyRandomNumber.h>
     51#include <wtf/SimpleStats.h>
    5152
    5253using namespace std;
    … …  
    6566        CodeLocationCall(MacroAssemblerCodePtr(returnAddress)),
    6667        newCalleeFunction);
     68}
     69
     70JIT::CodeRef JIT::compileCTINativeCall(VM* vm, NativeFunction func)
     71{
     72    if (!vm->canUseJIT())
     73        return CodeRef::createLLIntCodeRef(llint_native_call_trampoline);
     74    JIT jit(vm, 0);
     75    return jit.privateCompileCTINativeCall(vm, func);
    6776}
    6877
    … …  
    787796        ("Baseline JIT code for %s", toCString(CodeBlockWithJITType(m_codeBlock, JITCode::BaselineJIT)).data()));
    788797   
    789     m_vm->machineCodeBytesPerBytecodeWordForBaselineJIT.add(
     798    m_vm->machineCodeBytesPerBytecodeWordForBaselineJIT->add(
    790799        static_cast<double>(result.size()) /
    791800        static_cast<double>(m_instructions.size()));

  • trunk/Source/JavaScriptCore/jit/JIT.h

    r204994 r205462  
    4141#include "CodeBlock.h"
    4242#include "CompactJITCodeMap.h"
    43 #include "Interpreter.h"
    4443#include "JITDisassembler.h"
    4544#include "JITInlineCacheGenerator.h"
    4645#include "JITMathIC.h"
    4746#include "JSInterfaceJIT.h"
    48 #include "Opcode.h"
    4947#include "PCToCodeOriginMap.h"
    5048#include "UnusedPointer.h"
    5149
    5250namespace JSC {
     51
     52    enum OpcodeID : unsigned;
    5353
    5454    class ArrayAllocationProfile;
    … …  
    249249        }
    250250
    251         static CodeRef compileCTINativeCall(VM* vm, NativeFunction func)
    252         {
    253             if (!vm->canUseJIT()) {
    254                 return CodeRef::createLLIntCodeRef(llint_native_call_trampoline);
    255             }
    256             JIT jit(vm, 0);
    257             return jit.privateCompileCTINativeCall(vm, func);
    258         }
     251        static CodeRef compileCTINativeCall(VM*, NativeFunction);
    259252
    260253        static unsigned frameRegisterCountFor(CodeBlock*);

  • trunk/Source/JavaScriptCore/jit/JITExceptions.cpp

    r204912 r205462  
    9191}
    9292
     93void genericUnwind(VM* vm, ExecState* callFrame)
     94{
     95    genericUnwind(vm, callFrame, UnwindFromCurrentFrame);
     96}
     97
    9398} // namespace JSC

  • trunk/Source/JavaScriptCore/jit/JITExceptions.h

    r204912 r205462  
    2727#define JITExceptions_h
    2828
    29 #include "Interpreter.h"
    30 #include "JSCJSValue.h"
     29namespace JSC {
    3130
    32 namespace JSC {
     31enum UnwindStart : uint8_t;
    3332
    3433class ExecState;
    3534class VM;
    3635
    37 void genericUnwind(VM*, ExecState*, UnwindStart = UnwindFromCurrentFrame);
     36void genericUnwind(VM*, ExecState*, UnwindStart);
     37void genericUnwind(VM*, ExecState*);
    3838
    3939} // namespace JSC

  • trunk/Source/JavaScriptCore/jit/JITOpcodes.cpp

    r204994 r205462  
    3333#include "Exception.h"
    3434#include "Heap.h"
     35#include "Interpreter.h"
    3536#include "JITInlines.h"
    3637#include "JSArray.h"
    … …  
    8485    Structure* structure = currentInstruction[3].u.objectAllocationProfile->structure();
    8586    size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
    86     MarkedAllocator* allocator = &m_vm->heap.allocatorForObjectWithoutDestructor(allocationSize);
     87    MarkedAllocator* allocator = m_vm->heap.allocatorForObjectWithoutDestructor(allocationSize);
    8788
    8889    RegisterID resultReg = regT0;
    … …  
    9192
    9293    move(TrustedImmPtr(allocator), allocatorReg);
     94    if (allocator)
     95        addSlowCase(Jump());
    9396    JumpList slowCases;
    94     emitAllocateJSObject(resultReg, allocatorReg, TrustedImmPtr(structure), TrustedImmPtr(0), scratchReg, slowCases);
     97    emitAllocateJSObject(resultReg, allocator, allocatorReg, TrustedImmPtr(structure), TrustedImmPtr(0), scratchReg, slowCases);
    9598    addSlowCase(slowCases);
    9699    emitPutVirtualRegister(currentInstruction[1].u.operand);
    … …  
    99102void JIT::emitSlow_op_new_object(Instruction* currentInstruction, Vector<SlowCaseEntry>::iterator& iter)
    100103{
     104    linkSlowCase(iter);
    101105    linkSlowCase(iter);
    102106    int dst = currentInstruction[1].u.operand;
    … …  
    773777
    774778    JumpList slowCases;
    775     emitAllocateJSObject(resultReg, allocatorReg, structureReg, TrustedImmPtr(0), scratchReg, slowCases);
     779    emitAllocateJSObject(resultReg, nullptr, allocatorReg, structureReg, TrustedImmPtr(0), scratchReg, slowCases);
    776780    addSlowCase(slowCases);
    777781    emitPutVirtualRegister(currentInstruction[1].u.operand);
    … …  
    783787    linkSlowCase(iter); // doesn't have rare data
    784788    linkSlowCase(iter); // doesn't have an allocation profile
    785     linkSlowCase(iter); // allocation failed
     789    linkSlowCase(iter); // allocation failed (no allocator)
     790    linkSlowCase(iter); // allocation failed (allocator empty)
    786791    linkSlowCase(iter); // cached function didn't match
    787792

  • trunk/Source/JavaScriptCore/jit/JITOpcodes32_64.cpp

    r204912 r205462  
    4040#include "LinkBuffer.h"
    4141#include "MaxFrameExtentForSlowPathCall.h"
     42#include "Opcode.h"
    4243#include "SlowPathCall.h"
    4344#include "TypeProfilerLog.h"
    … …  
    164165    Structure* structure = currentInstruction[3].u.objectAllocationProfile->structure();
    165166    size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
    166     MarkedAllocator* allocator = &m_vm->heap.allocatorForObjectWithoutDestructor(allocationSize);
     167    MarkedAllocator* allocator = m_vm->heap.allocatorForObjectWithoutDestructor(allocationSize);
    167168
    168169    RegisterID resultReg = returnValueGPR;
    … …  
    171172
    172173    move(TrustedImmPtr(allocator), allocatorReg);
     174    if (allocator)
     175        addSlowCase(Jump());
    173176    JumpList slowCases;
    174     emitAllocateJSObject(resultReg, allocatorReg, TrustedImmPtr(structure), TrustedImmPtr(0), scratchReg, slowCases);
     177    emitAllocateJSObject(resultReg, allocator, allocatorReg, TrustedImmPtr(structure), TrustedImmPtr(0), scratchReg, slowCases);
    175178    addSlowCase(slowCases);
    176179    emitStoreCell(currentInstruction[1].u.operand, resultReg);
    … …  
    179182void JIT::emitSlow_op_new_object(Instruction* currentInstruction, Vector<SlowCaseEntry>::iterator& iter)
    180183{
     184    linkSlowCase(iter);
    181185    linkSlowCase(iter);
    182186    int dst = currentInstruction[1].u.operand;
    … …  
    10331037
    10341038    JumpList slowCases;
    1035     emitAllocateJSObject(resultReg, allocatorReg, structureReg, TrustedImmPtr(0), scratchReg, slowCases);
     1039    emitAllocateJSObject(resultReg, nullptr, allocatorReg, structureReg, TrustedImmPtr(0), scratchReg, slowCases);
    10361040    addSlowCase(slowCases);
    10371041    emitStoreCell(currentInstruction[1].u.operand, resultReg);
    … …  
    10431047    linkSlowCase(iter); // doesn't have rare data
    10441048    linkSlowCase(iter); // doesn't have an allocation profile
    1045     linkSlowCase(iter); // allocation failed
     1049    linkSlowCase(iter); // allocation failed (no allocator)
     1050    linkSlowCase(iter); // allocation failed (allocator empty)
    10461051    linkSlowCase(iter); // cached function didn't match
    10471052

  • trunk/Source/JavaScriptCore/jit/JITOperations.cpp

    r205198 r205462  
    4545#include "HostCallReturnValue.h"
    4646#include "ICStats.h"
     47#include "Interpreter.h"
    4748#include "JIT.h"
    4849#include "JITExceptions.h"
    … …  
    479480    if (stubInfo->considerCaching(structure))
    480481        repatchPutByID(exec, baseObject, structure, ident, slot, *stubInfo, Direct);
    481 }
    482 
    483 void JIT_OPERATION operationReallocateStorageAndFinishPut(ExecState* exec, JSObject* base, Structure* structure, PropertyOffset offset, EncodedJSValue value)
    484 {
    485     VM& vm = exec->vm();
    486     NativeCallFrameTracer tracer(&vm, exec);
    487 
    488     ASSERT(structure->outOfLineCapacity() > base->structure(vm)->outOfLineCapacity());
    489     ASSERT(!vm.heap.storageAllocator().fastPathShouldSucceed(structure->outOfLineCapacity() * sizeof(JSValue)));
    490     base->setStructureAndReallocateStorageIfNecessary(vm, structure);
    491     base->putDirect(vm, offset, JSValue::decode(value));
    492482}
    493483
    … …  
    21582148
    21592149    ASSERT(!object->structure()->outOfLineCapacity());
    2160     DeferGC deferGC(vm.heap);
    21612150    Butterfly* result = object->growOutOfLineStorage(vm, 0, initialOutOfLineCapacity);
    21622151    object->setButterflyWithoutChangingStructure(vm, result);
    … …  
    21692158    NativeCallFrameTracer tracer(&vm, exec);
    21702159
    2171     DeferGC deferGC(vm.heap);
    21722160    Butterfly* result = object->growOutOfLineStorage(vm, object->structure()->outOfLineCapacity(), newSize);
    21732161    object->setButterflyWithoutChangingStructure(vm, result);

  • trunk/Source/JavaScriptCore/jit/JITOperations.h

    r204912 r205462  
    3939namespace JSC {
    4040
     41typedef int64_t EncodedJSValue;
     42   
    4143class ArrayAllocationProfile;
    4244class ArrayProfile;
     45class Butterfly;
    4346class CallLinkInfo;
    4447class CodeBlock;
    … …  
    4649class JITAddGenerator;
    4750class JSArray;
     51class JSCell;
    4852class JSFunction;
     53class JSGlobalObject;
    4954class JSLexicalEnvironment;
     55class JSObject;
    5056class JSScope;
     57class JSString;
     58class JSValue;
    5159class RegExpObject;
    5260class Register;
     61class Structure;
    5362class StructureStubInfo;
    5463class SymbolTable;
    … …  
    5766struct ByValInfo;
    5867struct InlineCallFrame;
     68struct Instruction;
    5969struct ArithProfile;
    6070
    … …  
    7383    Ap: ArrayProfile*
    7484    Arp: ArithProfile*
     85    B: Butterfly*
    7586    By: ByValInfo*
    7687    C: JSCell*
    … …  
    280291typedef char* (JIT_OPERATION *P_JITOperation_EStSS)(ExecState*, Structure*, size_t, size_t);
    281292typedef char* (JIT_OPERATION *P_JITOperation_EStZ)(ExecState*, Structure*, int32_t);
     293typedef char* (JIT_OPERATION *P_JITOperation_EStZB)(ExecState*, Structure*, int32_t, Butterfly*);
    282294typedef char* (JIT_OPERATION *P_JITOperation_EZZ)(ExecState*, int32_t, int32_t);
    283295typedef SlowPathReturnType (JIT_OPERATION *Sprt_JITOperation_ECli)(ExecState*, CallLinkInfo*);
    … …  
    321333void JIT_OPERATION operationPutByIdDirectStrictBuildList(ExecState*, StructureStubInfo*, EncodedJSValue encodedValue, EncodedJSValue encodedBase, UniquedStringImpl*) WTF_INTERNAL;
    322334void JIT_OPERATION operationPutByIdDirectNonStrictBuildList(ExecState*, StructureStubInfo*, EncodedJSValue encodedValue, EncodedJSValue encodedBase, UniquedStringImpl*) WTF_INTERNAL;
    323 void JIT_OPERATION operationReallocateStorageAndFinishPut(ExecState*, JSObject*, Structure*, PropertyOffset, EncodedJSValue) WTF_INTERNAL;
    324335void JIT_OPERATION operationPutByValOptimize(ExecState*, EncodedJSValue, EncodedJSValue, EncodedJSValue, ByValInfo*) WTF_INTERNAL;
    325336void JIT_OPERATION operationDirectPutByValOptimize(ExecState*, EncodedJSValue, EncodedJSValue, EncodedJSValue, ByValInfo*) WTF_INTERNAL;

  • trunk/Source/JavaScriptCore/jit/JITPropertyAccess.cpp

    r204992 r205462  
    12471247void JIT::emitWriteBarrier(JSCell* owner)
    12481248{
    1249     if (!MarkedBlock::blockFor(owner)->isMarked(owner)) {
     1249    if (!owner->cellContainer().isMarked(owner)) {
    12501250        Jump ownerIsRememberedOrInEden = jumpIfIsRememberedOrInEden(owner);
    12511251        callOperation(operationUnconditionalWriteBarrier, owner);

  • trunk/Source/JavaScriptCore/jit/JITThunks.cpp

    r204912 r205462  
    3131#include "Executable.h"
    3232#include "JIT.h"
     33#include "JSCInlines.h"
     34#include "LLIntData.h"
    3335#include "VM.h"
    34 #include "JSCInlines.h"
    3536
    3637namespace JSC {

  • trunk/Source/JavaScriptCore/jit/JITThunks.h

    r204912 r205462  
    3131#include "CallData.h"
    3232#include "Intrinsic.h"
    33 #include "LowLevelInterpreter.h"
    3433#include "MacroAssemblerCodeRef.h"
    3534#include "ThunkGenerator.h"

  • trunk/Source/JavaScriptCore/jsc.cpp

    r205387 r205462  
    637637static EncodedJSValue JSC_HOST_CALL functionPlatformSupportsSamplingProfiler(ExecState*);
    638638static EncodedJSValue JSC_HOST_CALL functionGenerateHeapSnapshot(ExecState*);
     639static EncodedJSValue JSC_HOST_CALL functionResetSuperSamplerState(ExecState*);
     640static EncodedJSValue JSC_HOST_CALL functionEnsureArrayStorage(ExecState*);
    639641#if ENABLE(SAMPLING_PROFILER)
    640642static EncodedJSValue JSC_HOST_CALL functionStartSamplingProfiler(ExecState*);
    … …  
    873875        addFunction(vm, "platformSupportsSamplingProfiler", functionPlatformSupportsSamplingProfiler, 0);
    874876        addFunction(vm, "generateHeapSnapshot", functionGenerateHeapSnapshot, 0);
     877        addFunction(vm, "resetSuperSamplerState", functionResetSuperSamplerState, 0);
     878        addFunction(vm, "ensureArrayStorage", functionEnsureArrayStorage, 0);
    875879#if ENABLE(SAMPLING_PROFILER)
    876880        addFunction(vm, "startSamplingProfiler", functionStartSamplingProfiler, 0);
    … …  
    12141218    if (!object)
    12151219        return JSValue::encode(jsNontrivialString(exec, ASCIILiteral("<not object>")));
    1216     return JSValue::encode(jsNontrivialString(exec, toString("<Public length: ", object->getArrayLength(), "; vector length: ", object->getVectorLength(), ">")));
     1220    return JSValue::encode(jsNontrivialString(exec, toString("<Butterfly: ", RawPointer(object->butterfly()), "; public length: ", object->getArrayLength(), "; vector length: ", object->getVectorLength(), ">")));
    12171221}
    12181222
    … …  
    19611965}
    19621966
     1967EncodedJSValue JSC_HOST_CALL functionResetSuperSamplerState(ExecState*)
     1968{
     1969    resetSuperSamplerState();
     1970    return JSValue::encode(jsUndefined());
     1971}
     1972
     1973EncodedJSValue JSC_HOST_CALL functionEnsureArrayStorage(ExecState* exec)
     1974{
     1975    for (unsigned i = 0; i < exec->argumentCount(); ++i) {
     1976        if (JSObject* object = jsDynamicCast<JSObject*>(exec->argument(0)))
     1977            object->ensureArrayStorage(exec->vm());
     1978    }
     1979    return JSValue::encode(jsUndefined());
     1980}
     1981
    19631982#if ENABLE(SAMPLING_PROFILER)
    19641983EncodedJSValue JSC_HOST_CALL functionStartSamplingProfiler(ExecState* exec)
    … …  
    20722091        res = jscmain(argc, argv);
    20732092    EXCEPT(res = 3)
    2074     if (Options::logHeapStatisticsAtExit())
    2075         HeapStatistics::reportSuccess();
    2076     if (Options::reportLLIntStats())
    2077         LLInt::Data::finalizeStats();
     2093    finalizeStatsAtEndOfTesting();
    20782094
    20792095#if PLATFORM(EFL)

  • trunk/Source/JavaScriptCore/llint/LLIntData.cpp

    r204912 r205462  
    212212    STATIC_ASSERT(GetPutInfo::initializationBits == 0xffc00);
    213213
    214     STATIC_ASSERT(MarkedBlock::blockMask == ~static_cast<decltype(MarkedBlock::blockMask)>(0x3fff));
     214    STATIC_ASSERT(MarkedBlock::blockSize == 16 * 1024);
    215215
    216216    ASSERT(bitwise_cast<uintptr_t>(ShadowChicken::Packet::tailMarker()) == static_cast<uintptr_t>(0x7a11));

  • trunk/Source/JavaScriptCore/llint/LLIntExceptions.cpp

    r204912 r205462  
    3030#include "Instruction.h"
    3131#include "LLIntCommon.h"
     32#include "LLIntData.h"
    3233#include "LowLevelInterpreter.h"
    3334#include "JSCInlines.h"

  • trunk/Source/JavaScriptCore/llint/LLIntThunks.cpp

    r205330 r205462  
    3434#include "JSObject.h"
    3535#include "LLIntCLoop.h"
     36#include "LLIntData.h"
    3637#include "LinkBuffer.h"
    3738#include "LowLevelInterpreter.h"

  • trunk/Source/JavaScriptCore/llint/LLIntThunks.h

    r205330 r205462  
    3333class VM;
    3434struct ProtoCallFrame;
     35typedef int64_t EncodedJSValue;
    3536
    3637extern "C" {

  • trunk/Source/JavaScriptCore/llint/LowLevelInterpreter.asm

    r205321 r205462  
    10691069end
    10701070
    1071 macro allocateJSObject(allocator, structure, result, scratch1, slowCase)
    1072     const offsetOfFirstFreeCell =
    1073         MarkedAllocator::m_freeList +
    1074         MarkedBlock::FreeList::head
    1075 
    1076     # Get the object from the free list.   
    1077     loadp offsetOfFirstFreeCell[allocator], result
    1078     btpz result, slowCase
    1079    
    1080     # Remove the object from the free list.
    1081     loadp [result], scratch1
    1082     storep scratch1, offsetOfFirstFreeCell[allocator]
    1083 
    1084     # Initialize the object.
    1085     storep 0, JSObject::m_butterfly[result]
    1086     storeStructureWithTypeInfo(result, structure, scratch1)
    1087 end
    1088 
    10891071macro doReturn()
    10901072    restoreCalleeSavesUsedByLLInt()
    … …  
    13061288    callOpcodeSlowPath(_slow_path_create_cloned_arguments)
    13071289    dispatch(2)
     1290
     1291
     1292_llint_op_create_this:
     1293    traceExecution()
     1294    callOpcodeSlowPath(_slow_path_create_this)
     1295    dispatch(5)
     1296
     1297
     1298_llint_op_new_object:
     1299    traceExecution()
     1300    callOpcodeSlowPath(_llint_slow_path_new_object)
     1301    dispatch(4)
    13081302
    13091303

  • trunk/Source/JavaScriptCore/llint/LowLevelInterpreter.cpp

    r204912 r205462  
    2626#include "config.h"
    2727#include "LowLevelInterpreter.h"
     28
    2829#include "LLIntOfflineAsmConfig.h"
    2930#include <wtf/InlineASM.h>
    3031
    3132#if !ENABLE(JIT)
     33#include "CLoopStackInlines.h"
    3234#include "CodeBlock.h"
    3335#include "CommonSlowPaths.h"
     36#include "Interpreter.h"
    3437#include "LLIntCLoop.h"
     38#include "LLIntData.h"
    3539#include "LLIntSlowPaths.h"
    3640#include "JSCInlines.h"

  • trunk/Source/JavaScriptCore/llint/LowLevelInterpreter32_64.asm

    r204912 r205462  
    306306    loadp Callee + PayloadOffset[cfr], t3
    307307    andp MarkedBlockMask, t3
    308     loadp MarkedBlock::m_weakSet + WeakSet::m_vm[t3], t3
     308    loadp MarkedBlock::m_vm[t3], t3
    309309    restoreCalleeSavesFromVMEntryFrameCalleeSavesBuffer(t3, t0)
    310310    loadp VM::callFrameForCatch[t3], cfr
    … …  
    654654    loadp Callee + PayloadOffset[cfr], t3
    655655    andp MarkedBlockMask, t3
    656     loadp MarkedBlock::m_weakSet + WeakSet::m_vm[t3], t3
     656    loadp MarkedBlock::m_vm[t3], t3
    657657    btiz VM::m_exception[t3], .noException
    658658    jmp label
    … …  
    703703
    704704
    705 _llint_op_create_this:
    706     traceExecution()
    707     loadi 8[PC], t0
    708     loadp PayloadOffset[cfr, t0, 8], t0
    709     bbneq JSCell::m_type[t0], JSFunctionType, .opCreateThisSlow
    710     loadp JSFunction::m_rareData[t0], t5
    711     btpz t5, .opCreateThisSlow
    712     loadp FunctionRareData::m_objectAllocationProfile + ObjectAllocationProfile::m_allocator[t5], t1
    713     loadp FunctionRareData::m_objectAllocationProfile + ObjectAllocationProfile::m_structure[t5], t2
    714     btpz t1, .opCreateThisSlow
    715     loadpFromInstruction(4, t5)
    716     bpeq t5, 1, .hasSeenMultipleCallee
    717     bpneq t5, t0, .opCreateThisSlow
    718 .hasSeenMultipleCallee:
    719     allocateJSObject(t1, t2, t0, t3, .opCreateThisSlow)
    720     loadi 4[PC], t1
    721     storei CellTag, TagOffset[cfr, t1, 8]
    722     storei t0, PayloadOffset[cfr, t1, 8]
    723     dispatch(5)
    724 
    725 .opCreateThisSlow:
    726     callOpcodeSlowPath(_slow_path_create_this)
    727     dispatch(5)
    728 
    729 
    730705_llint_op_to_this:
    731706    traceExecution()
    … …  
    740715.opToThisSlow:
    741716    callOpcodeSlowPath(_slow_path_to_this)
    742     dispatch(4)
    743 
    744 
    745 _llint_op_new_object:
    746     traceExecution()
    747     loadpFromInstruction(3, t0)
    748     loadp ObjectAllocationProfile::m_allocator[t0], t1
    749     loadp ObjectAllocationProfile::m_structure[t0], t2
    750     allocateJSObject(t1, t2, t0, t3, .opNewObjectSlow)
    751     loadi 4[PC], t1
    752     storei CellTag, TagOffset[cfr, t1, 8]
    753     storei t0, PayloadOffset[cfr, t1, 8]
    754     dispatch(4)
    755 
    756 .opNewObjectSlow:
    757     callOpcodeSlowPath(_llint_slow_path_new_object)
    758717    dispatch(4)
    759718
    … …  
    19981957    loadp Callee + PayloadOffset[cfr], t3
    19991958    andp MarkedBlockMask, t3
    2000     loadp MarkedBlock::m_weakSet + WeakSet::m_vm[t3], t3
     1959    loadp MarkedBlock::m_vm[t3], t3
    20011960    restoreCalleeSavesFromVMEntryFrameCalleeSavesBuffer(t3, t0)
    20021961    loadp VM::callFrameForCatch[t3], cfr
    … …  
    20131972    loadp Callee + PayloadOffset[cfr], t3
    20141973    andp MarkedBlockMask, t3
    2015     loadp MarkedBlock::m_weakSet + WeakSet::m_vm[t3], t3
     1974    loadp MarkedBlock::m_vm[t3], t3
    20161975
    20171976    loadi VM::m_exception[t3], t0
    … …  
    20482007    loadp Callee[cfr], t1
    20492008    andp MarkedBlockMask, t1
    2050     loadp MarkedBlock::m_weakSet + WeakSet::m_vm[t1], t1
     2009    loadp MarkedBlock::m_vm[t1], t1
    20512010    copyCalleeSavesToVMEntryFrameCalleeSavesBuffer(t1, t2)
    20522011    jmp VM::targetMachinePCForThrow[t1]
    … …  
    20672026        subp 8, sp # align stack pointer
    20682027        andp MarkedBlockMask, t1
    2069         loadp MarkedBlock::m_weakSet + WeakSet::m_vm[t1], t3
     2028        loadp MarkedBlock::m_vm[t1], t3
    20702029        storep cfr, VM::topCallFrame[t3]
    20712030        move cfr, a0  # a0 = ecx
    … …  
    20772036        loadp Callee + PayloadOffset[cfr], t3
    20782037        andp MarkedBlockMask, t3
    2079         loadp MarkedBlock::m_weakSet + WeakSet::m_vm[t3], t3
     2038        loadp MarkedBlock::m_vm[t3], t3
    20802039        addp 8, sp
    20812040    elsif ARM or ARMv7 or ARMv7_TRADITIONAL or C_LOOP or MIPS or SH4
    … …  
    20832042        # t1 already contains the Callee.
    20842043        andp MarkedBlockMask, t1
    2085         loadp MarkedBlock::m_weakSet + WeakSet::m_vm[t1], t1
     2044        loadp MarkedBlock::m_vm[t1], t1
    20862045        storep cfr, VM::topCallFrame[t1]
    20872046        move cfr, a0
    … …  
    20962055        loadp Callee + PayloadOffset[cfr], t3
    20972056        andp MarkedBlockMask, t3
    2098         loadp MarkedBlock::m_weakSet + WeakSet::m_vm[t3], t3
     2057        loadp MarkedBlock::m_vm[t3], t3
    20992058        addp 8, sp
    21002059    else

  • trunk/Source/JavaScriptCore/llint/LowLevelInterpreter64.asm

    r204912 r205462  
    278278    loadp Callee[cfr], t3
    279279    andp MarkedBlockMask, t3
    280     loadp MarkedBlock::m_weakSet + WeakSet::m_vm[t3], t3
     280    loadp MarkedBlock::m_vm[t3], t3
    281281    restoreCalleeSavesFromVMEntryFrameCalleeSavesBuffer(t3, t0)
    282282    loadp VM::callFrameForCatch[t3], cfr
    … …  
    560560    loadp Callee[cfr], t3
    561561    andp MarkedBlockMask, t3
    562     loadp MarkedBlock::m_weakSet + WeakSet::m_vm[t3], t3
     562    loadp MarkedBlock::m_vm[t3], t3
    563563    btqz VM::m_exception[t3], .noException
    564564    jmp label
    … …  
    608608
    609609
    610 _llint_op_create_this:
    611     traceExecution()
    612     loadisFromInstruction(2, t0)
    613     loadp [cfr, t0, 8], t0
    614     bbneq JSCell::m_type[t0], JSFunctionType, .opCreateThisSlow
    615     loadp JSFunction::m_rareData[t0], t3
    616     btpz t3, .opCreateThisSlow
    617     loadp FunctionRareData::m_objectAllocationProfile + ObjectAllocationProfile::m_allocator[t3], t1
    618     loadp FunctionRareData::m_objectAllocationProfile + ObjectAllocationProfile::m_structure[t3], t2
    619     btpz t1, .opCreateThisSlow
    620     loadpFromInstruction(4, t3)
    621     bpeq t3, 1, .hasSeenMultipleCallee
    622     bpneq t3, t0, .opCreateThisSlow
    623 .hasSeenMultipleCallee:
    624     allocateJSObject(t1, t2, t0, t3, .opCreateThisSlow)
    625     loadisFromInstruction(1, t1)
    626     storeq t0, [cfr, t1, 8]
    627     dispatch(5)
    628 
    629 .opCreateThisSlow:
    630     callOpcodeSlowPath(_slow_path_create_this)
    631     dispatch(5)
    632 
    633 
    634610_llint_op_to_this:
    635611    traceExecution()
    … …  
    645621.opToThisSlow:
    646622    callOpcodeSlowPath(_slow_path_to_this)
    647     dispatch(4)
    648 
    649 
    650 _llint_op_new_object:
    651     traceExecution()
    652     loadpFromInstruction(3, t0)
    653     loadp ObjectAllocationProfile::m_allocator[t0], t1
    654     loadp ObjectAllocationProfile::m_structure[t0], t2
    655     allocateJSObject(t1, t2, t0, t3, .opNewObjectSlow)
    656     loadisFromInstruction(1, t1)
    657     storeq t0, [cfr, t1, 8]
    658     dispatch(4)
    659 
    660 .opNewObjectSlow:
    661     callOpcodeSlowPath(_llint_slow_path_new_object)
    662623    dispatch(4)
    663624
    … …  
    19591920    loadp Callee[cfr], t3
    19601921    andp MarkedBlockMask, t3
    1961     loadp MarkedBlock::m_weakSet + WeakSet::m_vm[t3], t3
     1922    loadp MarkedBlock::m_vm[t3], t3
    19621923    restoreCalleeSavesFromVMEntryFrameCalleeSavesBuffer(t3, t0)
    19631924    loadp VM::callFrameForCatch[t3], cfr
    … …  
    19781939    loadp Callee[cfr], t3
    19791940    andp MarkedBlockMask, t3
    1980     loadp MarkedBlock::m_weakSet + WeakSet::m_vm[t3], t3
     1941    loadp MarkedBlock::m_vm[t3], t3
    19811942
    19821943    loadq VM::m_exception[t3], t0
    … …  
    20051966    loadp Callee[cfr], t1
    20061967    andp MarkedBlockMask, t1
    2007     loadp MarkedBlock::m_weakSet + WeakSet::m_vm[t1], t1
     1968    loadp MarkedBlock::m_vm[t1], t1
    20081969    copyCalleeSavesToVMEntryFrameCalleeSavesBuffer(t1, t2)
    20091970
    … …  
    20151976    loadp Callee[cfr], t1
    20161977    andp MarkedBlockMask, t1
    2017     loadp MarkedBlock::m_weakSet + WeakSet::m_vm[t1], t1
     1978    loadp MarkedBlock::m_vm[t1], t1
    20181979    jmp VM::targetMachinePCForThrow[t1]
    20191980
    … …  
    20301991    loadp Callee[cfr], t0
    20311992    andp MarkedBlockMask, t0, t1
    2032     loadp MarkedBlock::m_weakSet + WeakSet::m_vm[t1], t1
     1993    loadp MarkedBlock::m_vm[t1], t1
    20331994    storep cfr, VM::topCallFrame[t1]
    20341995    if ARM64 or C_LOOP
    … …  
    20522013    loadp Callee[cfr], t3
    20532014    andp MarkedBlockMask, t3
    2054     loadp MarkedBlock::m_weakSet + WeakSet::m_vm[t3], t3
     2015    loadp MarkedBlock::m_vm[t3], t3
    20552016
    20562017    functionEpilogue()

  • trunk/Source/JavaScriptCore/parser/ModuleAnalyzer.cpp

    r204912 r205462  
    11/*
    2  * Copyright (C) 2015 Apple Inc. All rights reserved.
     2 * Copyright (C) 2015-2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2727#include "ModuleAnalyzer.h"
    2828
    29 #include "IdentifierInlines.h"
    30 #include "JSCJSValueInlines.h"
    31 #include "JSCellInlines.h"
     29#include "JSCInlines.h"
    3230#include "JSGlobalObject.h"
    3331#include "JSModuleRecord.h"

  • trunk/Source/JavaScriptCore/parser/NodeConstructors.h

    r204912 r205462  
    2424#include "Nodes.h"
    2525#include "Lexer.h"
     26#include "Opcode.h"
    2627#include "Parser.h"
    2728

  • trunk/Source/JavaScriptCore/parser/Nodes.h

    r204912 r205462  
    3030#include "Error.h"
    3131#include "JITCode.h"
    32 #include "Opcode.h"
    3332#include "ParserArena.h"
    3433#include "ParserTokens.h"
    … …  
    4140
    4241namespace JSC {
     42
     43    enum OpcodeID : unsigned;
    4344
    4445    class ArgumentListNode;

  • trunk/Source/JavaScriptCore/profiler/ProfilerBytecode.cpp

    r204912 r205462  
    2929#include "JSGlobalObject.h"
    3030#include "ObjectConstructor.h"
     31#include "Opcode.h"
    3132#include "JSCInlines.h"
    3233

  • trunk/Source/JavaScriptCore/profiler/ProfilerBytecode.h

    r204912 r205462  
    2828
    2929#include "JSCJSValue.h"
    30 #include "Opcode.h"
    3130#include <wtf/text/CString.h>
    3231
    33 namespace JSC { namespace Profiler {
     32namespace JSC {
     33
     34enum OpcodeID : unsigned;
     35
     36namespace Profiler {
    3437
    3538class Bytecode {

  • trunk/Source/JavaScriptCore/profiler/ProfilerBytecodeSequence.cpp

    r204912 r205462  
    2828
    2929#include "CodeBlock.h"
     30#include "Interpreter.h"
     31#include "JSCInlines.h"
    3032#include "JSGlobalObject.h"
    3133#include "Operands.h"
    32 #include "JSCInlines.h"
    3334#include <wtf/StringPrintStream.h>
    3435

  • trunk/Source/JavaScriptCore/runtime/ArrayConventions.h

    r204912 r205462  
    11/*
    22 *  Copyright (C) 1999-2000 Harri Porten (porten@kde.org)
    3  *  Copyright (C) 2003, 2007, 2008, 2009, 2012 Apple Inc. All rights reserved.
     3 *  Copyright (C) 2003, 2007, 2008, 2009, 2012, 2016 Apple Inc. All rights reserved.
    44 *
    55 *  This library is free software; you can redistribute it and/or
    … …  
    7171#define MAX_ARRAY_INDEX 0xFFFFFFFEU
    7272
    73 // The value BASE_VECTOR_LEN is the maximum number of vector elements we'll allocate
     73// The value BASE_XXX_VECTOR_LEN is the maximum number of vector elements we'll allocate
    7474// for an array that was created with a sepcified length (e.g. a = new Array(123))
    75 #define BASE_VECTOR_LEN 4U
    76    
     75#define BASE_CONTIGUOUS_VECTOR_LEN 3U
     76#define BASE_CONTIGUOUS_VECTOR_LEN_EMPTY 5U
     77#define BASE_ARRAY_STORAGE_VECTOR_LEN 4U
     78
    7779// The upper bound to the size we'll grow a zero length array when the first element
    7880// is added.
    79 #define FIRST_VECTOR_GROW 4U
     81#define FIRST_ARRAY_STORAGE_VECTOR_GROW 4U
    8082
    8183#define MIN_BEYOND_LENGTH_SPARSE_INDEX 1000
    … …  
    9799}
    98100
    99 inline IndexingHeader indexingHeaderForArray(unsigned length, unsigned vectorLength)
     101inline IndexingHeader indexingHeaderForArrayStorage(unsigned length, unsigned vectorLength)
    100102{
    101103    IndexingHeader result;
    … …  
    105107}
    106108
    107 inline IndexingHeader baseIndexingHeaderForArray(unsigned length)
     109inline IndexingHeader baseIndexingHeaderForArrayStorage(unsigned length)
    108110{
    109     return indexingHeaderForArray(length, BASE_VECTOR_LEN);
     111    return indexingHeaderForArrayStorage(length, BASE_ARRAY_STORAGE_VECTOR_LEN);
     112}
     113
     114#if USE(JSVALUE64)
     115JS_EXPORT_PRIVATE void clearArrayMemset(WriteBarrier<Unknown>* base, unsigned count);
     116JS_EXPORT_PRIVATE void clearArrayMemset(double* base, unsigned count);
     117#endif // USE(JSVALUE64)
     118
     119ALWAYS_INLINE void clearArray(WriteBarrier<Unknown>* base, unsigned count)
     120{
     121#if USE(JSVALUE64)
     122    const unsigned minCountForMemset = 100;
     123    if (count >= minCountForMemset) {
     124        clearArrayMemset(base, count);
     125        return;
     126    }
     127#endif
     128   
     129    for (unsigned i = count; i--;)
     130        base[i].clear();
     131}
     132
     133ALWAYS_INLINE void clearArray(double* base, unsigned count)
     134{
     135#if USE(JSVALUE64)
     136    const unsigned minCountForMemset = 100;
     137    if (count >= minCountForMemset) {
     138        clearArrayMemset(base, count);
     139        return;
     140    }
     141#endif
     142   
     143    for (unsigned i = count; i--;)
     144        base[i] = PNaN;
    110145}
    111146

  • trunk/Source/JavaScriptCore/runtime/ArrayPrototype.cpp

    r205198 r205462  
    10071007            return JSValue::encode(jsUndefined());
    10081008    }
    1009 
     1009   
    10101010    setLength(exec, thisObj, length - deleteCount + additionalArgs);
    10111011    return JSValue::encode(result);
    … …  
    11441144
    11451145    IndexingType type = first->mergeIndexingTypeForCopying(indexingTypeForValue(second) | IsArray);
     1146   
    11461147    if (type == NonArray)
    11471148        type = first->indexingType();
    … …  
    11721173
    11731174    JSArray* firstArray = jsCast<JSArray*>(exec->uncheckedArgument(0));
    1174 
     1175   
    11751176    // This code assumes that neither array has set Symbol.isConcatSpreadable. If the first array
    11761177    // has indexed accessors then one of those accessors might change the value of Symbol.isConcatSpreadable
    … …  
    11881189
    11891190    JSArray* secondArray = jsCast<JSArray*>(second);
    1190 
     1191   
    11911192    Butterfly* firstButterfly = firstArray->butterfly();
    11921193    Butterfly* secondButterfly = secondArray->butterfly();
    … …  
    11951196    unsigned secondArraySize = secondButterfly->publicLength();
    11961197
    1197     IndexingType type = firstArray->mergeIndexingTypeForCopying(secondArray->indexingType());
     1198    IndexingType secondType = secondArray->indexingType();
     1199    IndexingType type = firstArray->mergeIndexingTypeForCopying(secondType);
    11981200    if (type == NonArray || !firstArray->canFastCopy(vm, secondArray) || firstArraySize + secondArraySize >= MIN_SPARSE_ARRAY_INDEX) {
    11991201        JSArray* result = constructEmptyArray(exec, nullptr, firstArraySize + secondArraySize);
    … …  
    12141216    if (!result)
    12151217        return JSValue::encode(throwOutOfMemoryError(exec, scope));
    1216 
     1218   
    12171219    if (type == ArrayWithDouble) {
    12181220        double* buffer = result->butterfly()->contiguousDouble().data();
    … …  
    12221224        WriteBarrier<Unknown>* buffer = result->butterfly()->contiguous().data();
    12231225        memcpy(buffer, firstButterfly->contiguous().data(), sizeof(JSValue) * firstArraySize);
    1224         memcpy(buffer + firstArraySize, secondButterfly->contiguous().data(), sizeof(JSValue) * secondArraySize);
     1226        if (secondType != ArrayWithUndecided)
     1227            memcpy(buffer + firstArraySize, secondButterfly->contiguous().data(), sizeof(JSValue) * secondArraySize);
     1228        else {
     1229            for (unsigned i = secondArraySize; i--;)
     1230                buffer[i + firstArraySize].clear();
     1231        }
    12251232    }
    12261233

  • trunk/Source/JavaScriptCore/runtime/ArrayStorage.h

    r204912 r205462  
    11/*
    2  * Copyright (C) 2012 Apple Inc. All rights reserved.
     2 * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3030#include "Butterfly.h"
    3131#include "IndexingHeader.h"
     32#include "MarkedSpace.h"
    3233#include "SparseArrayValueMap.h"
     34#include "Structure.h"
    3335#include "WriteBarrier.h"
    3436#include <wtf/Noncopyable.h>
    … …  
    5961    unsigned length() const { return indexingHeader()->publicLength(); }
    6062    void setLength(unsigned length) { indexingHeader()->setPublicLength(length); }
    61     unsigned vectorLength() { return indexingHeader()->vectorLength(); }
     63    unsigned vectorLength() const { return indexingHeader()->vectorLength(); }
    6264    void setVectorLength(unsigned length) { indexingHeader()->setVectorLength(length); }
    6365   
    … …  
    100102        return ArrayStorage::vectorOffset() + vectorLength * sizeof(WriteBarrier<Unknown>);
    101103    }
     104   
     105    static size_t totalSizeFor(unsigned indexBias, size_t propertyCapacity, unsigned vectorLength)
     106    {
     107        return Butterfly::totalSize(indexBias, propertyCapacity, true, sizeFor(vectorLength));
     108    }
     109   
     110    size_t totalSize(size_t propertyCapacity) const
     111    {
     112        return totalSizeFor(m_indexBias, propertyCapacity, vectorLength());
     113    }
     114   
     115    size_t totalSize(Structure* structure) const
     116    {
     117        return totalSize(structure->outOfLineCapacity());
     118    }
     119   
     120    static unsigned availableVectorLength(unsigned indexBias, size_t propertyCapacity, unsigned vectorLength)
     121    {
     122        size_t cellSize = MarkedSpace::optimalSizeFor(totalSizeFor(indexBias, propertyCapacity, vectorLength));
     123       
     124        vectorLength = (cellSize - totalSizeFor(indexBias, propertyCapacity, 0)) / sizeof(WriteBarrier<Unknown>);
     125
     126        return vectorLength;
     127    }
     128   
     129    static unsigned availableVectorLength(unsigned indexBias, Structure* structure, unsigned vectorLength)
     130    {
     131        return availableVectorLength(indexBias, structure->outOfLineCapacity(), vectorLength);
     132    }
     133   
     134    unsigned availableVectorLength(size_t propertyCapacity, unsigned vectorLength)
     135    {
     136        return availableVectorLength(m_indexBias, propertyCapacity, vectorLength);
     137    }
     138   
     139    unsigned availableVectorLength(Structure* structure, unsigned vectorLength)
     140    {
     141        return availableVectorLength(structure->outOfLineCapacity(), vectorLength);
     142    }
     143
     144    static unsigned optimalVectorLength(unsigned indexBias, size_t propertyCapacity, unsigned vectorLength)
     145    {
     146        vectorLength = std::max(BASE_ARRAY_STORAGE_VECTOR_LEN, vectorLength);
     147        return availableVectorLength(indexBias, propertyCapacity, vectorLength);
     148    }
     149   
     150    static unsigned optimalVectorLength(unsigned indexBias, Structure* structure, unsigned vectorLength)
     151    {
     152        return optimalVectorLength(indexBias, structure->outOfLineCapacity(), vectorLength);
     153    }
     154   
     155    unsigned optimalVectorLength(size_t propertyCapacity, unsigned vectorLength)
     156    {
     157        return optimalVectorLength(m_indexBias, propertyCapacity, vectorLength);
     158    }
     159   
     160    unsigned optimalVectorLength(Structure* structure, unsigned vectorLength)
     161    {
     162        return optimalVectorLength(structure->outOfLineCapacity(), vectorLength);
     163    }
    102164};
    103165

  • trunk/Source/JavaScriptCore/runtime/Butterfly.h

    r204912 r205462  
    11/*
    2  * Copyright (C) 2012 Apple Inc. All rights reserved.
     2 * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    9191    }
    9292   
     93    ALWAYS_INLINE static unsigned availableContiguousVectorLength(size_t propertyCapacity, unsigned vectorLength);
     94    static unsigned availableContiguousVectorLength(Structure*, unsigned vectorLength);
     95   
     96    ALWAYS_INLINE static unsigned optimalContiguousVectorLength(size_t propertyCapacity, unsigned vectorLength);
     97    static unsigned optimalContiguousVectorLength(Structure*, unsigned vectorLength);
     98   
    9399    // This method is here not just because it's handy, but to remind you that
    94100    // the whole point of butterflies is to do evil pointer arithmetic.

  • trunk/Source/JavaScriptCore/runtime/ButterflyInlines.h

    r204912 r205462  
    11/*
    2  * Copyright (C) 2012 Apple Inc. All rights reserved.
     2 * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3636namespace JSC {
    3737
     38ALWAYS_INLINE unsigned Butterfly::availableContiguousVectorLength(size_t propertyCapacity, unsigned vectorLength)
     39{
     40    size_t cellSize = totalSize(0, propertyCapacity, true, sizeof(EncodedJSValue) * vectorLength);
     41    cellSize = MarkedSpace::optimalSizeFor(cellSize);
     42    vectorLength = (cellSize - totalSize(0, propertyCapacity, true, 0)) / sizeof(EncodedJSValue);
     43    return vectorLength;
     44}
     45
     46ALWAYS_INLINE unsigned Butterfly::availableContiguousVectorLength(Structure* structure, unsigned vectorLength)
     47{
     48    return availableContiguousVectorLength(structure ? structure->outOfLineCapacity() : 0, vectorLength);
     49}
     50
     51ALWAYS_INLINE unsigned Butterfly::optimalContiguousVectorLength(size_t propertyCapacity, unsigned vectorLength)
     52{
     53    if (!vectorLength)
     54        vectorLength = BASE_CONTIGUOUS_VECTOR_LEN_EMPTY;
     55    else
     56        vectorLength = std::max(BASE_CONTIGUOUS_VECTOR_LEN, vectorLength);
     57    return availableContiguousVectorLength(propertyCapacity, vectorLength);
     58}
     59
     60ALWAYS_INLINE unsigned Butterfly::optimalContiguousVectorLength(Structure* structure, unsigned vectorLength)
     61{
     62    return optimalContiguousVectorLength(structure ? structure->outOfLineCapacity() : 0, vectorLength);
     63}
     64
    3865inline Butterfly* Butterfly::createUninitialized(VM& vm, JSCell* intendedOwner, size_t preCapacity, size_t propertyCapacity, bool hasIndexingHeader, size_t indexingPayloadSizeInBytes)
    3966{
    40     void* temp;
    4167    size_t size = totalSize(preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes);
    42     RELEASE_ASSERT(vm.heap.tryAllocateStorage(intendedOwner, size, &temp));
    43     Butterfly* result = fromBase(temp, preCapacity, propertyCapacity);
     68    void* base = vm.heap.allocateAuxiliary(intendedOwner, size);
     69    Butterfly* result = fromBase(base, preCapacity, propertyCapacity);
    4470    return result;
    4571}
    … …  
    120146    size_t oldSize = totalSize(0, propertyCapacity, hadIndexingHeader, oldIndexingPayloadSizeInBytes);
    121147    size_t newSize = totalSize(0, propertyCapacity, true, newIndexingPayloadSizeInBytes);
    122     if (!vm.heap.tryReallocateStorage(intendedOwner, &theBase, oldSize, newSize))
     148    theBase = vm.heap.tryReallocateAuxiliary(intendedOwner, theBase, oldSize, newSize);
     149    if (!theBase)
    123150        return 0;
    124151    return fromBase(theBase, 0, propertyCapacity);

  • trunk/Source/JavaScriptCore/runtime/ClonedArguments.cpp

    r204912 r205462  
    4545    VM& vm, Structure* structure, JSFunction* callee, unsigned length)
    4646{
    47     unsigned vectorLength = std::max(BASE_VECTOR_LEN, length);
     47    unsigned vectorLength = length;
    4848    if (vectorLength > MAX_STORAGE_VECTOR_LENGTH)
    4949        return 0;
    5050
    51     void* temp;
    52     if (!vm.heap.tryAllocateStorage(0, Butterfly::totalSize(0, structure->outOfLineCapacity(), true, vectorLength * sizeof(EncodedJSValue)), &temp))
     51    void* temp = vm.heap.tryAllocateAuxiliary(nullptr, Butterfly::totalSize(0, structure->outOfLineCapacity(), true, vectorLength * sizeof(EncodedJSValue)));
     52    if (!temp)
    5353        return 0;
    5454    Butterfly* butterfly = Butterfly::fromBase(temp, 0, structure->outOfLineCapacity());
    5555    butterfly->setVectorLength(vectorLength);
    5656    butterfly->setPublicLength(length);
     57   
     58    for (unsigned i = length; i < vectorLength; ++i)
     59        butterfly->contiguous()[i].clear();
    5760
    5861    ClonedArguments* result =

  • trunk/Source/JavaScriptCore/runtime/CommonSlowPathsExceptions.cpp

    r205198 r205462  
    2929#include "CallFrame.h"
    3030#include "CodeBlock.h"
     31#include "Interpreter.h"
    3132#include "JITExceptions.h"
    3233#include "LLIntCommon.h"

  • trunk/Source/JavaScriptCore/runtime/CommonSlowPathsExceptions.h

    r204912 r205462  
    2727#define CommonSlowPathExceptions_h
    2828
    29 #include "MacroAssemblerCodeRef.h"
    30 
    3129namespace JSC {
    3230
    3331class ExecState;
     32class JSObject;
    3433
    3534namespace CommonSlowPaths {

  • trunk/Source/JavaScriptCore/runtime/DataView.cpp

    r204912 r205462  
    2727#include "DataView.h"
    2828
     29#include "JSCInlines.h"
    2930#include "JSDataView.h"
    3031#include "JSGlobalObject.h"

  • trunk/Source/JavaScriptCore/runtime/DirectArguments.h

    r204912 r205462  
    2727#define DirectArguments_h
    2828
     29#include "CopyBarrier.h"
    2930#include "DirectArgumentsOffset.h"
    3031#include "GenericArguments.h"

  • trunk/Source/JavaScriptCore/runtime/ECMAScriptSpecInternalFunctions.cpp

    r204912 r205462  
    2929#include "CallFrame.h"
    3030#include "ConstructData.h"
    31 #include "JSCJSValueInlines.h"
     31#include "JSCInlines.h"
    3232#include "RegExpObject.h"
    3333

  • trunk/Source/JavaScriptCore/runtime/Error.cpp

    r205198 r205462  
    2929#include "ExceptionHelpers.h"
    3030#include "FunctionPrototype.h"
     31#include "Interpreter.h"
    3132#include "JSArray.h"
    3233#include "JSFunction.h"
    … …  
    3435#include "JSObject.h"
    3536#include "JSString.h"
     37#include "JSCInlines.h"
    3638#include "NativeErrorConstructor.h"
    37 #include "JSCInlines.h"
    3839#include "SourceCode.h"
     40#include "StackFrame.h"
    3941
    4042namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/Error.h

    r205198 r205462  
    2626#include "ErrorInstance.h"
    2727#include "InternalFunction.h"
    28 #include "Interpreter.h"
    2928#include "JSObject.h"
    3029#include "ThrowScope.h"

  • trunk/Source/JavaScriptCore/runtime/ErrorInstance.cpp

    r204912 r205462  
    2727#include "JSCInlines.h"
    2828#include "JSGlobalObjectFunctions.h"
     29#include <wtf/text/StringBuilder.h>
    2930
    3031namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/ErrorInstance.h

    r204912 r205462  
    2222#define ErrorInstance_h
    2323
    24 #include "Interpreter.h"
     24#include "JSObject.h"
    2525#include "RuntimeType.h"
    2626#include "SourceProvider.h"

  • trunk/Source/JavaScriptCore/runtime/Exception.cpp

    r204912 r205462  
    2727#include "Exception.h"
    2828
     29#include "Interpreter.h"
    2930#include "JSCInlines.h"
    3031

  • trunk/Source/JavaScriptCore/runtime/Exception.h

    r204912 r205462  
    2727#define Exception_h
    2828
    29 #include "Interpreter.h"
     29#include "JSObject.h"
     30#include "StackFrame.h"
    3031#include <wtf/Vector.h>
    3132

  • trunk/Source/JavaScriptCore/runtime/GeneratorPrototype.cpp

    r204912 r205462  
    2828
    2929#include "JSCBuiltins.h"
    30 #include "JSCJSValueInlines.h"
    31 #include "JSCellInlines.h"
     30#include "JSCInlines.h"
    3231#include "JSGlobalObject.h"
    33 #include "StructureInlines.h"
    3432
    3533#include "GeneratorPrototype.lut.h"

  • trunk/Source/JavaScriptCore/runtime/InternalFunction.cpp

    r204912 r205462  
    3838    : JSDestructibleObject(vm, structure)
    3939{
     40    // exec->vm() wants callees to not be large allocations.
     41    RELEASE_ASSERT(!isLargeAllocation());
    4042}
    4143

  • trunk/Source/JavaScriptCore/runtime/IntlCollator.cpp

    r205198 r205462  
    22 * Copyright (C) 2015 Andy VanWagoner (thetalecrafter@gmail.com)
    33 * Copyright (C) 2015 Sukolsak Sakshuwong (sukolsak@gmail.com)
     4 * Copyright (C) 2016 Apple Inc. All Rights Reserved.
    45 *
    56 * Redistribution and use in source and binary forms, with or without
    … …  
    3435#include "IntlObject.h"
    3536#include "JSBoundFunction.h"
    36 #include "JSCJSValueInlines.h"
    37 #include "JSCellInlines.h"
     37#include "JSCInlines.h"
    3838#include "ObjectConstructor.h"
    3939#include "SlotVisitorInlines.h"

  • trunk/Source/JavaScriptCore/runtime/IntlCollatorConstructor.cpp

    r204912 r205462  
    3434#include "IntlCollatorPrototype.h"
    3535#include "IntlObject.h"
    36 #include "JSCJSValueInlines.h"
    37 #include "JSCellInlines.h"
     36#include "JSCInlines.h"
    3837#include "Lookup.h"
    39 #include "SlotVisitorInlines.h"
    40 #include "StructureInlines.h"
    4138
    4239namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/IntlCollatorPrototype.cpp

    r205198 r205462  
    3333#include "IntlCollator.h"
    3434#include "JSBoundFunction.h"
    35 #include "JSCJSValueInlines.h"
    36 #include "JSCellInlines.h"
    37 #include "JSObject.h"
    38 #include "StructureInlines.h"
     35#include "JSCInlines.h"
    3936
    4037namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/IntlDateTimeFormat.cpp

    r205332 r205462  
    3535#include "IntlObject.h"
    3636#include "JSBoundFunction.h"
    37 #include "JSCellInlines.h"
    3837#include "JSCInlines.h"
    3938#include "ObjectConstructor.h"
    … …  
    4140#include <unicode/udatpg.h>
    4241#include <unicode/uenum.h>
     42#include <wtf/text/StringBuilder.h>
    4343
    4444namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/IntlDateTimeFormatConstructor.cpp

    r204912 r205462  
    3434#include "IntlObject.h"
    3535#include "IntlObjectInlines.h"
    36 #include "JSCJSValueInlines.h"
    37 #include "JSCellInlines.h"
     36#include "JSCInlines.h"
    3837#include "Lookup.h"
    39 #include "SlotVisitorInlines.h"
    40 #include "StructureInlines.h"
    4138
    4239namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/IntlDateTimeFormatPrototype.cpp

    r205324 r205462  
    3636#include "IntlObject.h"
    3737#include "JSBoundFunction.h"
    38 #include "JSCJSValueInlines.h"
    39 #include "JSCellInlines.h"
     38#include "JSCInlines.h"
    4039#include "JSObjectInlines.h"
    41 #include "StructureInlines.h"
    4240
    4341namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/IntlNumberFormat.cpp

    r205332 r205462  
    3535#include "IntlObject.h"
    3636#include "JSBoundFunction.h"
    37 #include "JSCellInlines.h"
    3837#include "JSCInlines.h"
    3938#include "ObjectConstructor.h"

  • trunk/Source/JavaScriptCore/runtime/IntlNumberFormatConstructor.cpp

    r204912 r205462  
    3434#include "IntlObject.h"
    3535#include "IntlObjectInlines.h"
    36 #include "JSCJSValueInlines.h"
    37 #include "JSCellInlines.h"
     36#include "JSCInlines.h"
    3837#include "Lookup.h"
    39 #include "SlotVisitorInlines.h"
    40 #include "StructureInlines.h"
    4138
    4239namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/IntlNumberFormatPrototype.cpp

    r205324 r205462  
    3434#include "IntlNumberFormat.h"
    3535#include "JSBoundFunction.h"
    36 #include "JSCJSValueInlines.h"
    37 #include "JSCellInlines.h"
     36#include "JSCInlines.h"
    3837#include "JSObjectInlines.h"
    39 #include "StructureInlines.h"
    4038
    4139namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/IntlObject.cpp

    r205198 r205462  
    5151#include <wtf/NeverDestroyed.h>
    5252#include <wtf/PlatformUserPreferredLanguages.h>
     53#include <wtf/text/StringBuilder.h>
    5354
    5455namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/IteratorPrototype.cpp

    r204912 r205462  
    2828
    2929#include "JSCBuiltins.h"
    30 #include "JSCJSValueInlines.h"
    31 #include "JSCellInlines.h"
     30#include "JSCInlines.h"
    3231#include "JSGlobalObject.h"
    3332#include "ObjectConstructor.h"
    34 #include "StructureInlines.h"
    3533
    3634namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/JSArray.cpp

    r205324 r205462  
    6161}
    6262
     63JSArray* JSArray::tryCreateUninitialized(VM& vm, Structure* structure, unsigned initialLength)
     64{
     65    if (initialLength > MAX_STORAGE_VECTOR_LENGTH)
     66        return 0;
     67
     68    unsigned outOfLineStorage = structure->outOfLineCapacity();
     69
     70    Butterfly* butterfly;
     71    IndexingType indexingType = structure->indexingType();
     72    if (LIKELY(!hasAnyArrayStorage(indexingType))) {
     73        ASSERT(
     74            hasUndecided(indexingType)
     75            || hasInt32(indexingType)
     76            || hasDouble(indexingType)
     77            || hasContiguous(indexingType));
     78
     79        unsigned vectorLength = Butterfly::optimalContiguousVectorLength(structure, initialLength);
     80        void* temp = vm.heap.tryAllocateAuxiliary(nullptr, Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)));
     81        if (!temp)
     82            return nullptr;
     83        butterfly = Butterfly::fromBase(temp, 0, outOfLineStorage);
     84        butterfly->setVectorLength(vectorLength);
     85        butterfly->setPublicLength(initialLength);
     86        if (hasDouble(indexingType)) {
     87            for (unsigned i = initialLength; i < vectorLength; ++i)
     88                butterfly->contiguousDouble()[i] = PNaN;
     89        } else {
     90            for (unsigned i = initialLength; i < vectorLength; ++i)
     91                butterfly->contiguous()[i].clear();
     92        }
     93    } else {
     94        unsigned vectorLength = ArrayStorage::optimalVectorLength(0, structure, initialLength);
     95        void* temp = vm.heap.tryAllocateAuxiliary(nullptr, Butterfly::totalSize(0, outOfLineStorage, true, ArrayStorage::sizeFor(vectorLength)));
     96        if (!temp)
     97            return nullptr;
     98        butterfly = Butterfly::fromBase(temp, 0, outOfLineStorage);
     99        *butterfly->indexingHeader() = indexingHeaderForArrayStorage(initialLength, vectorLength);
     100        ArrayStorage* storage = butterfly->arrayStorage();
     101        storage->m_indexBias = 0;
     102        storage->m_sparseMap.clear();
     103        storage->m_numValuesInVector = initialLength;
     104        for (unsigned i = initialLength; i < vectorLength; ++i)
     105            storage->m_vector[i].clear();
     106    }
     107
     108    return createWithButterfly(vm, structure, butterfly);
     109}
     110
    63111void JSArray::setLengthWritable(ExecState* exec, bool writable)
    64112{
    … …  
    244292
    245293// This method makes room in the vector, but leaves the new space for count slots uncleared.
    246 bool JSArray::unshiftCountSlowCase(VM& vm, bool addToFront, unsigned count)
     294bool JSArray::unshiftCountSlowCase(VM& vm, DeferGC&, bool addToFront, unsigned count)
    247295{
    248296    ArrayStorage* storage = ensureArrayStorage(vm);
    249297    Butterfly* butterfly = storage->butterfly();
    250     unsigned propertyCapacity = structure(vm)->outOfLineCapacity();
    251     unsigned propertySize = structure(vm)->outOfLineSize();
    252 
     298    Structure* structure = this->structure(vm);
     299    unsigned propertyCapacity = structure->outOfLineCapacity();
     300    unsigned propertySize = structure->outOfLineSize();
     301   
    253302    // If not, we should have handled this on the fast path.
    254303    ASSERT(!addToFront || count > storage->m_indexBias);
    … …  
    262311
    263312    unsigned length = storage->length();
    264     unsigned usedVectorLength = min(storage->vectorLength(), length);
     313    unsigned oldVectorLength = storage->vectorLength();
     314    unsigned usedVectorLength = min(oldVectorLength, length);
    265315    ASSERT(usedVectorLength <= MAX_STORAGE_VECTOR_LENGTH);
    266316    // Check that required vector length is possible, in an overflow-safe fashion.
    … …  
    273323    unsigned currentCapacity = storage->vectorLength() + storage->m_indexBias;
    274324    // The calculation of desiredCapacity won't overflow, due to the range of MAX_STORAGE_VECTOR_LENGTH.
    275     unsigned desiredCapacity = min(MAX_STORAGE_VECTOR_LENGTH, max(BASE_VECTOR_LEN, requiredVectorLength) << 1);
     325    // FIXME: This code should be fixed to avoid internal fragmentation. It's not super high
     326    // priority since increaseVectorLength() will "fix" any mistakes we make, but it would be cool
     327    // to get this right eventually.
     328    unsigned desiredCapacity = min(MAX_STORAGE_VECTOR_LENGTH, max(BASE_ARRAY_STORAGE_VECTOR_LEN, requiredVectorLength) << 1);
    276329
    277330    // Step 2:
    278331    // We're either going to choose to allocate a new ArrayStorage, or we're going to reuse the existing one.
    279332
    280     DeferGC deferGC(vm.heap);
    281333    void* newAllocBase = 0;
    282334    unsigned newStorageCapacity;
     335    bool allocatedNewStorage;
    283336    // If the current storage array is sufficiently large (but not too large!) then just keep using it.
    284337    if (currentCapacity > desiredCapacity && isDenseEnoughForVector(currentCapacity, requiredVectorLength)) {
    285         newAllocBase = butterfly->base(structure(vm));
     338        newAllocBase = butterfly->base(structure);
    286339        newStorageCapacity = currentCapacity;
     340        allocatedNewStorage = false;
    287341    } else {
    288342        size_t newSize = Butterfly::totalSize(0, propertyCapacity, true, ArrayStorage::sizeFor(desiredCapacity));
    289         if (!vm.heap.tryAllocateStorage(this, newSize, &newAllocBase))
     343        newAllocBase = vm.heap.tryAllocateAuxiliary(this, newSize);
     344        if (!newAllocBase)
    290345            return false;
    291346        newStorageCapacity = desiredCapacity;
     347        allocatedNewStorage = true;
    292348    }
    293349
    … …  
    307363        postCapacity = min((storage->vectorLength() - length) >> 1, newStorageCapacity - requiredVectorLength);
    308364        // If we're moving contents within the same allocation, the post-capacity is being reduced.
    309         ASSERT(newAllocBase != butterfly->base(structure(vm)) || postCapacity < storage->vectorLength() - length);
     365        ASSERT(newAllocBase != butterfly->base(structure) || postCapacity < storage->vectorLength() - length);
    310366    }
    311367
    … …  
    319375        memmove(newButterfly->arrayStorage()->m_vector + count, storage->m_vector, sizeof(JSValue) * usedVectorLength);
    320376        memmove(newButterfly->propertyStorage() - propertySize, butterfly->propertyStorage() - propertySize, sizeof(JSValue) * propertySize + sizeof(IndexingHeader) + ArrayStorage::sizeFor(0));
    321     } else if ((newAllocBase != butterfly->base(structure(vm))) || (newIndexBias != storage->m_indexBias)) {
     377       
     378        if (allocatedNewStorage) {
     379            // We will set the vectorLength to newVectorLength. We populated requiredVectorLength
     380            // (usedVectorLength + count), which is less. Clear the difference.
     381            for (unsigned i = requiredVectorLength; i < newVectorLength; ++i)
     382                newButterfly->arrayStorage()->m_vector[i].clear();
     383        }
     384    } else if ((newAllocBase != butterfly->base(structure)) || (newIndexBias != storage->m_indexBias)) {
    322385        memmove(newButterfly->propertyStorage() - propertySize, butterfly->propertyStorage() - propertySize, sizeof(JSValue) * propertySize + sizeof(IndexingHeader) + ArrayStorage::sizeFor(0));
    323386        memmove(newButterfly->arrayStorage()->m_vector, storage->m_vector, sizeof(JSValue) * usedVectorLength);
    324 
    325         WriteBarrier<Unknown>* newVector = newButterfly->arrayStorage()->m_vector;
     387       
    326388        for (unsigned i = requiredVectorLength; i < newVectorLength; i++)
    327             newVector[i].clear();
     389            newButterfly->arrayStorage()->m_vector[i].clear();
    328390    }
    329391
    330392    newButterfly->arrayStorage()->setVectorLength(newVectorLength);
    331393    newButterfly->arrayStorage()->m_indexBias = newIndexBias;
     394   
    332395    setButterflyWithoutChangingStructure(vm, newButterfly);
    333396
    … …  
    338401{
    339402    unsigned length = storage->length();
    340 
     403   
    341404    // If the length is read only then we enter sparse mode, so should enter the following 'if'.
    342405    ASSERT(isLengthWritable() || storage->m_sparseMap);
    … …  
    9981061    unsigned vectorLength = storage->vectorLength();
    9991062
     1063    // Need to have GC deferred around the unshiftCountSlowCase(), since that leaves the butterfly in
     1064    // a weird state: some parts of it will be left uninitialized, which we will fill in here.
     1065    DeferGC deferGC(vm.heap);
     1066   
    10001067    if (moveFront && storage->m_indexBias >= count) {
    10011068        Butterfly* newButterfly = storage->butterfly()->unshift(structure(), count);
    … …  
    10061073    } else if (!moveFront && vectorLength - length >= count)
    10071074        storage = storage->butterfly()->arrayStorage();
    1008     else if (unshiftCountSlowCase(vm, moveFront, count))
     1075    else if (unshiftCountSlowCase(vm, deferGC, moveFront, count))
    10091076        storage = arrayStorage();
    10101077    else {
    … …  
    12001267
    12011268    Butterfly* butterfly = m_butterfly.get();
    1202    
    12031269    switch (indexingType()) {
    12041270    case ArrayClass:

  • trunk/Source/JavaScriptCore/runtime/JSArray.h

    r205324 r205462  
    11/*
    22 *  Copyright (C) 1999-2000 Harri Porten (porten@kde.org)
    3  *  Copyright (C) 2003, 2007, 2008, 2009, 2012, 2015 Apple Inc. All rights reserved.
     3 *  Copyright (C) 2003, 2007, 2008, 2009, 2012, 2015-2016 Apple Inc. All rights reserved.
    44 *
    55 *  This library is free software; you can redistribute it and/or
    … …  
    6161    //   - null-check the result (indicating out of memory, or otherwise unable to allocate vector).
    6262    //   - call 'initializeIndex' for all properties in sequence, for 0 <= i < initialLength.
    63     static JSArray* tryCreateUninitialized(VM&, Structure*, unsigned initialLength);
     63    JS_EXPORT_PRIVATE static JSArray* tryCreateUninitialized(VM&, Structure*, unsigned initialLength);
    6464
    6565    JS_EXPORT_PRIVATE static bool defineOwnProperty(JSObject*, ExecState*, PropertyName, const PropertyDescriptor&, bool throwException);
    … …  
    169169    bool unshiftCountWithAnyIndexingType(ExecState*, unsigned startIndex, unsigned count);
    170170    bool unshiftCountWithArrayStorage(ExecState*, unsigned startIndex, unsigned count, ArrayStorage*);
    171     bool unshiftCountSlowCase(VM&, bool, unsigned);
     171    bool unshiftCountSlowCase(VM&, DeferGC&, bool, unsigned);
    172172
    173173    bool setLengthWithArrayStorage(ExecState*, unsigned newLength, bool throwException, ArrayStorage*);
    … …  
    178178{
    179179    IndexingHeader header;
    180     vectorLength = std::max(length, BASE_VECTOR_LEN);
     180    vectorLength = Butterfly::optimalContiguousVectorLength(
     181        intendedOwner ? intendedOwner->structure(vm) : 0, length);
    181182    header.setVectorLength(vectorLength);
    182183    header.setPublicLength(length);
    … …  
    189190{
    190191    Butterfly* butterfly = Butterfly::create(
    191         vm, intendedOwner, 0, 0, true, baseIndexingHeaderForArray(initialLength),
    192         ArrayStorage::sizeFor(BASE_VECTOR_LEN));
     192        vm, intendedOwner, 0, 0, true, baseIndexingHeaderForArrayStorage(initialLength),
     193        ArrayStorage::sizeFor(BASE_ARRAY_STORAGE_VECTOR_LEN));
    193194    ArrayStorage* storage = butterfly->arrayStorage();
     195    storage->m_sparseMap.clear();
    194196    storage->m_indexBias = 0;
    195     storage->m_sparseMap.clear();
    196197    storage->m_numValuesInVector = 0;
    197198    return butterfly;
    … …  
    212213        unsigned vectorLength;
    213214        butterfly = createContiguousArrayButterfly(vm, 0, initialLength, vectorLength);
    214         ASSERT(initialLength < MIN_ARRAY_STORAGE_CONSTRUCTION_LENGTH);
    215         if (hasDouble(structure->indexingType())) {
    216             for (unsigned i = 0; i < vectorLength; ++i)
    217                 butterfly->contiguousDouble()[i] = PNaN;
    218         }
     215        if (hasDouble(structure->indexingType()))
     216            clearArray(butterfly->contiguousDouble().data(), vectorLength);
     217        else
     218            clearArray(butterfly->contiguous().data(), vectorLength);
    219219    } else {
    220220        ASSERT(
    … …  
    222222            || structure->indexingType() == ArrayWithArrayStorage);
    223223        butterfly = createArrayButterfly(vm, 0, initialLength);
    224     }
    225 
    226     return createWithButterfly(vm, structure, butterfly);
    227 }
    228 
    229 inline JSArray* JSArray::tryCreateUninitialized(VM& vm, Structure* structure, unsigned initialLength)
    230 {
    231     unsigned vectorLength = std::max(BASE_VECTOR_LEN, initialLength);
    232     if (vectorLength > MAX_STORAGE_VECTOR_LENGTH)
    233         return 0;
    234 
    235     unsigned outOfLineStorage = structure->outOfLineCapacity();
    236 
    237     Butterfly* butterfly;
    238     if (LIKELY(!hasAnyArrayStorage(structure->indexingType()))) {
    239         ASSERT(
    240             hasUndecided(structure->indexingType())
    241             || hasInt32(structure->indexingType())
    242             || hasDouble(structure->indexingType())
    243             || hasContiguous(structure->indexingType()));
    244 
    245         void* temp;
    246         if (!vm.heap.tryAllocateStorage(0, Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)), &temp))
    247             return 0;
    248         butterfly = Butterfly::fromBase(temp, 0, outOfLineStorage);
    249         butterfly->setVectorLength(vectorLength);
    250         butterfly->setPublicLength(initialLength);
    251         if (hasDouble(structure->indexingType())) {
    252             for (unsigned i = initialLength; i < vectorLength; ++i)
    253                 butterfly->contiguousDouble()[i] = PNaN;
    254         }
    255     } else {
    256         void* temp;
    257         if (!vm.heap.tryAllocateStorage(0, Butterfly::totalSize(0, outOfLineStorage, true, ArrayStorage::sizeFor(vectorLength)), &temp))
    258             return 0;
    259         butterfly = Butterfly::fromBase(temp, 0, outOfLineStorage);
    260         *butterfly->indexingHeader() = indexingHeaderForArray(initialLength, vectorLength);
    261         ArrayStorage* storage = butterfly->arrayStorage();
    262         storage->m_indexBias = 0;
    263         storage->m_sparseMap.clear();
    264         storage->m_numValuesInVector = initialLength;
     224        for (unsigned i = 0; i < BASE_ARRAY_STORAGE_VECTOR_LEN; ++i)
     225            butterfly->arrayStorage()->m_vector[i].clear();
    265226    }
    266227

  • trunk/Source/JavaScriptCore/runtime/JSArrayBufferView.h

    r205131 r205462  
    2727#define JSArrayBufferView_h
    2828
     29#include "CopyBarrier.h"
    2930#include "JSObject.h"
    3031

  • trunk/Source/JavaScriptCore/runtime/JSCInlines.h

    r205198 r205462  
    11/*
    2  * Copyright (C) 2014 Apple Inc. All rights reserved.
     2 * Copyright (C) 2014, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    4242#include "HeapInlines.h"
    4343#include "IdentifierInlines.h"
    44 #include "Interpreter.h"
    4544#include "JSArrayBufferViewInlines.h"
    4645#include "JSCJSValueInlines.h"

  • trunk/Source/JavaScriptCore/runtime/JSCJSValue.cpp

    r205198 r205462  
    3030#include "ExceptionHelpers.h"
    3131#include "GetterSetter.h"
    32 #include "JSCJSValueInlines.h"
     32#include "JSCInlines.h"
    3333#include "JSFunction.h"
    3434#include "JSGlobalObject.h"
    3535#include "NumberObject.h"
    36 #include "StructureInlines.h"
    3736#include <wtf/MathExtras.h>
    3837#include <wtf/StringExtras.h>
    … …  
    279278        else if (structure->classInfo()->isSubClassOf(Structure::info()))
    280279            out.print("Structure: ", inContext(*jsCast<Structure*>(asCell()), context));
    281         else {
     280        else if (structure->classInfo()->isSubClassOf(JSObject::info())) {
     281            out.print("Object: ", RawPointer(asCell()));
     282            out.print(" with butterfly ", RawPointer(asObject(asCell())->butterfly()));
     283            out.print(" (", inContext(*structure, context), ")");
     284        } else {
    282285            out.print("Cell: ", RawPointer(asCell()));
    283286            out.print(" (", inContext(*structure, context), ")");

  • trunk/Source/JavaScriptCore/runtime/JSCallee.cpp

    r204912 r205462  
    11/*
    2  * Copyright (C) 2014 Apple Inc. All rights reserved.
     2 * Copyright (C) 2014, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2828
    2929#include "GetterSetter.h"
    30 #include "JSCJSValueInlines.h"
    31 #include "JSCell.h"
    32 #include "JSCellInlines.h"
     30#include "JSCInlines.h"
    3331#include "JSGlobalObject.h"
    34 #include "SlotVisitorInlines.h"
    3532#include "StackVisitor.h"
    36 #include "StructureInlines.h"
    3733
    3834namespace JSC {
    … …  
    4440    , m_scope(vm, this, globalObject)
    4541{
     42    RELEASE_ASSERT(!isLargeAllocation());
    4643}
    4744

  • trunk/Source/JavaScriptCore/runtime/JSCell.cpp

    r205131 r205462  
    5959size_t JSCell::estimatedSize(JSCell* cell)
    6060{
    61     return MarkedBlock::blockFor(cell)->cellSize();
     61    return cell->cellSize();
    6262}
    6363

  • trunk/Source/JavaScriptCore/runtime/JSCell.h

    r205131 r205462  
    8181    enum CreatingEarlyCellTag { CreatingEarlyCell };
    8282    JSCell(CreatingEarlyCellTag);
    83 
     83   
    8484protected:
    8585    JSCell(VM&, Structure*);
    … …  
    108108
    109109    const char* className() const;
    110 
    111     VM* vm() const;
    112110
    113111    // Extracting the value.
    … …  
    191189        return OBJECT_OFFSETOF(JSCell, m_cellState);
    192190    }
     191   
     192    void callDestructor(VM&);
    193193
    194194    static const TypedArrayType TypedArrayStorageType = NotTypedArray;

  • trunk/Source/JavaScriptCore/runtime/JSCellInlines.h

    r204912 r205462  
    114114}
    115115
    116 inline VM* JSCell::vm() const
    117 {
    118     return MarkedBlock::blockFor(this)->vm();
    119 }
    120 
    121116ALWAYS_INLINE VM& ExecState::vm() const
    122117{
    123118    ASSERT(callee());
    124119    ASSERT(callee()->vm());
    125     return *calleeAsValue().asCell()->vm();
     120    ASSERT(!callee()->isLargeAllocation());
     121    // This is an important optimization since we access this so often.
     122    return *calleeAsValue().asCell()->markedBlock().vm();
    126123}
    127124
    … …  
    234231}
    235232
    236 inline const ClassInfo* JSCell::classInfo() const
    237 {
    238     MarkedBlock* block = MarkedBlock::blockFor(this);
    239     if (block->needsDestruction() && !(inlineTypeFlags() & StructureIsImmortal))
     233ALWAYS_INLINE const ClassInfo* JSCell::classInfo() const
     234{
     235    if (isLargeAllocation()) {
     236        LargeAllocation& allocation = largeAllocation();
     237        if (allocation.attributes().destruction == NeedsDestruction
     238            && !(inlineTypeFlags() & StructureIsImmortal))
     239            return static_cast<const JSDestructibleObject*>(this)->classInfo();
     240        return structure(*allocation.vm())->classInfo();
     241    }
     242    MarkedBlock& block = markedBlock();
     243    if (block.needsDestruction() && !(inlineTypeFlags() & StructureIsImmortal))
    240244        return static_cast<const JSDestructibleObject*>(this)->classInfo();
    241     return structure(*block->vm())->classInfo();
     245    return structure(*block.vm())->classInfo();
    242246}
    243247
    … …  
    258262}
    259263
     264inline void JSCell::callDestructor(VM& vm)
     265{
     266    if (isZapped())
     267        return;
     268    ASSERT(structureID());
     269    if (inlineTypeFlags() & StructureIsImmortal)
     270        structure(vm)->classInfo()->methodTable.destroy(this);
     271    else
     272        jsCast<JSDestructibleObject*>(this)->classInfo()->methodTable.destroy(this);
     273    zap();
     274}
     275
    260276} // namespace JSC
    261277

  • trunk/Source/JavaScriptCore/runtime/JSFunction.cpp

    r205335 r205462  
    6565JSFunction* JSFunction::create(VM& vm, FunctionExecutable* executable, JSScope* scope)
    6666{
    67     return create(vm, executable, scope, scope->globalObject()->functionStructure());
     67    return create(vm, executable, scope, scope->globalObject(vm)->functionStructure());
    6868}
    6969
    … …  
    7979{
    8080    JSFunction* function = new (NotNull, allocateCell<JSFunction>(vm.heap)) JSFunction(vm, executable, scope);
    81     ASSERT(function->structure()->globalObject());
     81    ASSERT(function->structure(vm)->globalObject());
    8282    function->finishCreation(vm);
    8383    return function;
    … …  
    146146    JSObject* prototype = jsDynamicCast<JSObject*>(get(exec, vm.propertyNames->prototype));
    147147    if (!prototype)
    148         prototype = globalObject()->objectPrototype();
     148        prototype = globalObject(vm)->objectPrototype();
    149149    FunctionRareData* rareData = FunctionRareData::create(vm);
    150     rareData->initializeObjectAllocationProfile(globalObject()->vm(), prototype, inlineCapacity);
     150    rareData->initializeObjectAllocationProfile(vm, prototype, inlineCapacity);
    151151
    152152    // A DFG compilation thread may be trying to read the rare data
    … …  
    164164    JSObject* prototype = jsDynamicCast<JSObject*>(get(exec, vm.propertyNames->prototype));
    165165    if (!prototype)
    166         prototype = globalObject()->objectPrototype();
    167     m_rareData->initializeObjectAllocationProfile(globalObject()->vm(), prototype, inlineCapacity);
     166        prototype = globalObject(vm)->objectPrototype();
     167    m_rareData->initializeObjectAllocationProfile(vm, prototype, inlineCapacity);
    168168    return m_rareData.get();
    169169}
    … …  
    346346bool JSFunction::getOwnPropertySlot(JSObject* object, ExecState* exec, PropertyName propertyName, PropertySlot& slot)
    347347{
     348    VM& vm = exec->vm();
    348349    JSFunction* thisObject = jsCast<JSFunction*>(object);
    349350    if (thisObject->isHostOrBuiltinFunction()) {
    350         thisObject->reifyBoundNameIfNeeded(exec, propertyName);
     351        thisObject->reifyBoundNameIfNeeded(vm, exec, propertyName);
    351352        return Base::getOwnPropertySlot(thisObject, exec, propertyName, slot);
    352353    }
    353354
    354     if (propertyName == exec->propertyNames().prototype && !thisObject->jsExecutable()->isArrowFunction()) {
    355         VM& vm = exec->vm();
     355    if (propertyName == vm.propertyNames->prototype && !thisObject->jsExecutable()->isArrowFunction()) {
    356356        unsigned attributes;
    357357        PropertyOffset offset = thisObject->getDirectOffset(vm, propertyName, attributes);
    … …  
    359359            JSObject* prototype = nullptr;
    360360            if (thisObject->jsExecutable()->parseMode() == SourceParseMode::GeneratorWrapperFunctionMode)
    361                 prototype = constructEmptyObject(exec, thisObject->globalObject()->generatorPrototype());
     361                prototype = constructEmptyObject(exec, thisObject->globalObject(vm)->generatorPrototype());
    362362            else
    363363                prototype = constructEmptyObject(exec);
    364364
    365             prototype->putDirect(vm, exec->propertyNames().constructor, thisObject, DontEnum);
    366             thisObject->putDirect(vm, exec->propertyNames().prototype, prototype, DontDelete | DontEnum);
    367             offset = thisObject->getDirectOffset(vm, exec->propertyNames().prototype, attributes);
     365            prototype->putDirect(vm, vm.propertyNames->constructor, thisObject, DontEnum);
     366            thisObject->putDirect(vm, vm.propertyNames->prototype, prototype, DontDelete | DontEnum);
     367            offset = thisObject->getDirectOffset(vm, vm.propertyNames->prototype, attributes);
    368368            ASSERT(isValidOffset(offset));
    369369        }
    … …  
    372372    }
    373373
    374     if (propertyName == exec->propertyNames().arguments) {
     374    if (propertyName == vm.propertyNames->arguments) {
    375375        if (thisObject->jsExecutable()->isStrictMode() || thisObject->jsExecutable()->isClassConstructorFunction()) {
    376376            bool result = Base::getOwnPropertySlot(thisObject, exec, propertyName, slot);
    377377            if (!result) {
    378                 GetterSetter* errorGetterSetter = thisObject->globalObject()->throwTypeErrorArgumentsCalleeAndCallerGetterSetter();
     378                GetterSetter* errorGetterSetter = thisObject->globalObject(vm)->throwTypeErrorArgumentsCalleeAndCallerGetterSetter();
    379379                thisObject->putDirectAccessor(exec, propertyName, errorGetterSetter, DontDelete | DontEnum | Accessor);
    380380                result = Base::getOwnPropertySlot(thisObject, exec, propertyName, slot);
    … …  
    387387    }
    388388
    389     if (propertyName == exec->propertyNames().caller) {
     389    if (propertyName == vm.propertyNames->caller) {
    390390        if (thisObject->jsExecutable()->isStrictMode() || thisObject->jsExecutable()->isClassConstructorFunction()) {
    391391            bool result = Base::getOwnPropertySlot(thisObject, exec, propertyName, slot);
    392392            if (!result) {
    393                 GetterSetter* errorGetterSetter = thisObject->globalObject()->throwTypeErrorArgumentsCalleeAndCallerGetterSetter();
     393                GetterSetter* errorGetterSetter = thisObject->globalObject(vm)->throwTypeErrorArgumentsCalleeAndCallerGetterSetter();
    394394                thisObject->putDirectAccessor(exec, propertyName, errorGetterSetter, DontDelete | DontEnum | Accessor);
    395395                result = Base::getOwnPropertySlot(thisObject, exec, propertyName, slot);
    … …  
    402402    }
    403403
    404     thisObject->reifyLazyPropertyIfNeeded(exec, propertyName);
     404    thisObject->reifyLazyPropertyIfNeeded(vm, exec, propertyName);
    405405
    406406    return Base::getOwnPropertySlot(thisObject, exec, propertyName, slot);
    … …  
    438438
    439439    if (thisObject->isHostOrBuiltinFunction()) {
    440         thisObject->reifyBoundNameIfNeeded(exec, propertyName);
     440        thisObject->reifyBoundNameIfNeeded(vm, exec, propertyName);
    441441        return Base::put(thisObject, exec, propertyName, value, slot);
    442442    }
    443443
    444     if (propertyName == exec->propertyNames().prototype) {
     444    if (propertyName == vm.propertyNames->prototype) {
    445445        // Make sure prototype has been reified, such that it can only be overwritten
    446446        // following the rules set out in ECMA-262 8.12.9.
    … …  
    454454        return Base::put(thisObject, exec, propertyName, value, dontCache);
    455455    }
    456     if (thisObject->jsExecutable()->isStrictMode() && (propertyName == exec->propertyNames().arguments || propertyName == exec->propertyNames().caller)) {
     456    if (thisObject->jsExecutable()->isStrictMode() && (propertyName == vm.propertyNames->arguments || propertyName == vm.propertyNames->caller)) {
    457457        // This will trigger the property to be reified, if this is not already the case!
    458458        bool okay = thisObject->hasProperty(exec, propertyName);
    … …  
    461461        return Base::put(thisObject, exec, propertyName, value, slot);
    462462    }
    463     if (propertyName == exec->propertyNames().arguments || propertyName == exec->propertyNames().caller) {
     463    if (propertyName == vm.propertyNames->arguments || propertyName == vm.propertyNames->caller) {
    464464        if (slot.isStrictMode())
    465465            throwTypeError(exec, scope, StrictModeReadonlyPropertyWriteError);
    466466        return false;
    467467    }
    468     thisObject->reifyLazyPropertyIfNeeded(exec, propertyName);
     468    thisObject->reifyLazyPropertyIfNeeded(vm, exec, propertyName);
    469469    scope.release();
    470470    return Base::put(thisObject, exec, propertyName, value, slot);
    … …  
    475475    JSFunction* thisObject = jsCast<JSFunction*>(cell);
    476476    if (thisObject->isHostOrBuiltinFunction())
    477         thisObject->reifyBoundNameIfNeeded(exec, propertyName);
     477        thisObject->reifyBoundNameIfNeeded(exec->vm(), exec, propertyName);
    478478    else if (exec->vm().deletePropertyMode() != VM::DeletePropertyMode::IgnoreConfigurable) {
    479479        // For non-host functions, don't let these properties by deleted - except by DefineOwnProperty.
     480        VM& vm = exec->vm();
    480481        FunctionExecutable* executable = thisObject->jsExecutable();
    481         if (propertyName == exec->propertyNames().arguments
    482             || (propertyName == exec->propertyNames().prototype && !executable->isArrowFunction())
    483             || propertyName == exec->propertyNames().caller)
     482        if (propertyName == vm.propertyNames->arguments
     483            || (propertyName == vm.propertyNames->prototype && !executable->isArrowFunction())
     484            || propertyName == vm.propertyNames->caller)
    484485            return false;
    485486
    486         thisObject->reifyLazyPropertyIfNeeded(exec, propertyName);
     487        thisObject->reifyLazyPropertyIfNeeded(vm, exec, propertyName);
    487488    }
    488489   
    … …  
    497498    JSFunction* thisObject = jsCast<JSFunction*>(object);
    498499    if (thisObject->isHostOrBuiltinFunction()) {
    499         thisObject->reifyBoundNameIfNeeded(exec, propertyName);
     500        thisObject->reifyBoundNameIfNeeded(vm, exec, propertyName);
    500501        return Base::defineOwnProperty(object, exec, propertyName, descriptor, throwException);
    501502    }
    502503
    503     if (propertyName == exec->propertyNames().prototype) {
     504    if (propertyName == vm.propertyNames->prototype) {
    504505        // Make sure prototype has been reified, such that it can only be overwritten
    505506        // following the rules set out in ECMA-262 8.12.9.
    … …  
    512513
    513514    bool valueCheck;
    514     if (propertyName == exec->propertyNames().arguments) {
     515    if (propertyName == vm.propertyNames->arguments) {
    515516        if (thisObject->jsExecutable()->isStrictMode()) {
    516517            PropertySlot slot(thisObject, PropertySlot::InternalMethodType::VMInquiry);
    517518            if (!Base::getOwnPropertySlot(thisObject, exec, propertyName, slot))
    518                 thisObject->putDirectAccessor(exec, propertyName, thisObject->globalObject()->throwTypeErrorArgumentsCalleeAndCallerGetterSetter(), DontDelete | DontEnum | Accessor);
     519                thisObject->putDirectAccessor(exec, propertyName, thisObject->globalObject(vm)->throwTypeErrorArgumentsCalleeAndCallerGetterSetter(), DontDelete | DontEnum | Accessor);
    519520            return Base::defineOwnProperty(object, exec, propertyName, descriptor, throwException);
    520521        }
    521522        valueCheck = !descriptor.value() || sameValue(exec, descriptor.value(), retrieveArguments(exec, thisObject));
    522     } else if (propertyName == exec->propertyNames().caller) {
     523    } else if (propertyName == vm.propertyNames->caller) {
    523524        if (thisObject->jsExecutable()->isStrictMode()) {
    524525            PropertySlot slot(thisObject, PropertySlot::InternalMethodType::VMInquiry);
    525526            if (!Base::getOwnPropertySlot(thisObject, exec, propertyName, slot))
    526                 thisObject->putDirectAccessor(exec, propertyName, thisObject->globalObject()->throwTypeErrorArgumentsCalleeAndCallerGetterSetter(), DontDelete | DontEnum | Accessor);
     527                thisObject->putDirectAccessor(exec, propertyName, thisObject->globalObject(vm)->throwTypeErrorArgumentsCalleeAndCallerGetterSetter(), DontDelete | DontEnum | Accessor);
    527528            return Base::defineOwnProperty(object, exec, propertyName, descriptor, throwException);
    528529        }
    529530        valueCheck = !descriptor.value() || sameValue(exec, descriptor.value(), retrieveCallerFunction(exec, thisObject));
    530531    } else {
    531         thisObject->reifyLazyPropertyIfNeeded(exec, propertyName);
     532        thisObject->reifyLazyPropertyIfNeeded(vm, exec, propertyName);
    532533        return Base::defineOwnProperty(object, exec, propertyName, descriptor, throwException);
    533534    }
    … …  
    591592void JSFunction::setFunctionName(ExecState* exec, JSValue value)
    592593{
     594    VM& vm = exec->vm();
    593595    // The "name" property may have been already been defined as part of a property list in an
    594596    // object literal (and therefore reified).
    … …  
    606608            name = makeString('[', String(&uid), ']');
    607609    } else {
    608         VM& vm = exec->vm();
    609610        JSString* jsStr = value.toString(exec);
    610611        if (vm.exception())
    … …  
    614615            return;
    615616    }
    616     reifyName(exec, name);
    617 }
    618 
    619 void JSFunction::reifyLength(ExecState* exec)
    620 {
    621     VM& vm = exec->vm();
     617    reifyName(vm, exec, name);
     618}
     619
     620void JSFunction::reifyLength(VM& vm)
     621{
    622622    FunctionRareData* rareData = this->rareData(vm);
    623623
    … …  
    626626    JSValue initialValue = jsNumber(jsExecutable()->parameterCount());
    627627    unsigned initialAttributes = DontEnum | ReadOnly;
    628     const Identifier& identifier = exec->propertyNames().length;
     628    const Identifier& identifier = vm.propertyNames->length;
    629629    putDirect(vm, identifier, initialValue, initialAttributes);
    630630
    … …  
    632632}
    633633
    634 void JSFunction::reifyName(ExecState* exec)
     634void JSFunction::reifyName(VM& vm, ExecState* exec)
    635635{
    636636    const Identifier& ecmaName = jsExecutable()->ecmaName();
    … …  
    643643    else
    644644        name = ecmaName.string();
    645     reifyName(exec, name);
    646 }
    647 
    648 void JSFunction::reifyName(ExecState* exec, String name)
    649 {
    650     VM& vm = exec->vm();
     645    reifyName(vm, exec, name);
     646}
     647
     648void JSFunction::reifyName(VM& vm, ExecState* exec, String name)
     649{
    651650    FunctionRareData* rareData = this->rareData(vm);
    652651
    … …  
    654653    ASSERT(!isHostFunction());
    655654    unsigned initialAttributes = DontEnum | ReadOnly;
    656     const Identifier& propID = exec->propertyNames().name;
     655    const Identifier& propID = vm.propertyNames->name;
    657656
    658657    if (exec->lexicalGlobalObject()->needsSiteSpecificQuirks()) {
    … …  
    673672}
    674673
    675 void JSFunction::reifyLazyPropertyIfNeeded(ExecState* exec, PropertyName propertyName)
    676 {
    677     if (propertyName == exec->propertyNames().length) {
     674void JSFunction::reifyLazyPropertyIfNeeded(VM& vm, ExecState* exec, PropertyName propertyName)
     675{
     676    if (propertyName == vm.propertyNames->length) {
    678677        if (!hasReifiedLength())
    679             reifyLength(exec);
    680     } else if (propertyName == exec->propertyNames().name) {
     678            reifyLength(vm);
     679    } else if (propertyName == vm.propertyNames->name) {
    681680        if (!hasReifiedName())
    682             reifyName(exec);
    683     }
    684 }
    685 
    686 void JSFunction::reifyBoundNameIfNeeded(ExecState* exec, PropertyName propertyName)
    687 {
    688     const Identifier& nameIdent = exec->propertyNames().name;
     681            reifyName(vm, exec);
     682    }
     683}
     684
     685void JSFunction::reifyBoundNameIfNeeded(VM& vm, ExecState* exec, PropertyName propertyName)
     686{
     687    const Identifier& nameIdent = vm.propertyNames->name;
    689688    if (propertyName != nameIdent)
    690689        return;
    … …  
    694693
    695694    if (this->inherits(JSBoundFunction::info())) {
    696         VM& vm = exec->vm();
    697695        FunctionRareData* rareData = this->rareData(vm);
    698696        String name = makeString("bound ", static_cast<NativeExecutable*>(m_executable.get())->name());

  • trunk/Source/JavaScriptCore/runtime/JSFunction.h

    r204912 r205462  
    190190    bool hasReifiedLength() const;
    191191    bool hasReifiedName() const;
    192     void reifyLength(ExecState*);
    193     void reifyName(ExecState*);
    194     void reifyBoundNameIfNeeded(ExecState*, PropertyName);
    195     void reifyName(ExecState*, String name);
    196     void reifyLazyPropertyIfNeeded(ExecState*, PropertyName propertyName);
     192    void reifyLength(VM&);
     193    void reifyName(VM&, ExecState*);
     194    void reifyBoundNameIfNeeded(VM&, ExecState*, PropertyName);
     195    void reifyName(VM&, ExecState*, String name);
     196    void reifyLazyPropertyIfNeeded(VM&, ExecState*, PropertyName propertyName);
    197197
    198198    friend class LLIntOffsetsExtractor;

  • trunk/Source/JavaScriptCore/runtime/JSFunctionInlines.h

    r204912 r205462  
    3636{
    3737    ASSERT(executable->singletonFunction()->hasBeenInvalidated());
    38     return createImpl(vm, executable, scope, scope->globalObject()->functionStructure());
     38    return createImpl(vm, executable, scope, scope->globalObject(vm)->functionStructure());
    3939}
    4040
    … …  
    4848#if ENABLE(WEBASSEMBLY)
    4949inline JSFunction::JSFunction(VM& vm, WebAssemblyExecutable* executable, JSScope* scope)
    50     : Base(vm, scope, scope->globalObject()->functionStructure())
     50    : Base(vm, scope, scope->globalObject(vm)->functionStructure())
    5151    , m_executable(vm, this, executable)
    5252    , m_rareData()

  • trunk/Source/JavaScriptCore/runtime/JSGenericTypedArrayViewInlines.h

    r205198 r205462  
    521521    // that you *had* done those allocations and it will GC appropriately.
    522522    Heap* heap = Heap::heap(thisObject);
     523    VM& vm = *heap->vm();
    523524    DeferGCForAWhile deferGC(*heap);
    524525   
    525526    ASSERT(!thisObject->hasIndexingHeader());
    526527
    527     size_t size = thisObject->byteSize();
    528    
    529     if (thisObject->m_mode == FastTypedArray
    530         && !thisObject->butterfly() && size >= sizeof(IndexingHeader)) {
    531         ASSERT(thisObject->m_vector);
    532         // Reuse already allocated memory if at all possible.
    533         thisObject->m_butterfly.setWithoutBarrier(
    534             bitwise_cast<IndexingHeader*>(thisObject->vector())->butterfly());
    535     } else {
    536         RELEASE_ASSERT(!thisObject->hasIndexingHeader());
    537         VM& vm = *heap->vm();
    538         thisObject->m_butterfly.set(vm, thisObject, Butterfly::createOrGrowArrayRight(
    539             thisObject->butterfly(), vm, thisObject, thisObject->structure(),
    540             thisObject->structure()->outOfLineCapacity(), false, 0, 0));
    541     }
     528    RELEASE_ASSERT(!thisObject->hasIndexingHeader());
     529    thisObject->m_butterfly.set(vm, thisObject, Butterfly::createOrGrowArrayRight(
     530        thisObject->butterfly(), vm, thisObject, thisObject->structure(),
     531        thisObject->structure()->outOfLineCapacity(), false, 0, 0));
    542532
    543533    RefPtr<ArrayBuffer> buffer;

  • trunk/Source/JavaScriptCore/runtime/JSInternalPromise.cpp

    r205324 r205462  
    2828
    2929#include "BuiltinNames.h"
    30 #include "JSCJSValueInlines.h"
    31 #include "JSCellInlines.h"
    32 #include "JSObjectInlines.h"
    33 #include "StructureInlines.h"
     30#include "JSCInlines.h"
    3431
    3532namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/JSInternalPromiseConstructor.cpp

    r204912 r205462  
    2828
    2929#include "JSCBuiltins.h"
    30 #include "JSCJSValueInlines.h"
    31 #include "JSCellInlines.h"
     30#include "JSCInlines.h"
    3231#include "JSInternalPromise.h"
    3332#include "JSInternalPromisePrototype.h"
    34 #include "StructureInlines.h"
    3533
    3634#include "JSInternalPromiseConstructor.lut.h"

  • trunk/Source/JavaScriptCore/runtime/JSInternalPromiseDeferred.cpp

    r205324 r205462  
    3030#include "Error.h"
    3131#include "Exception.h"
    32 #include "JSCJSValueInlines.h"
    33 #include "JSCellInlines.h"
     32#include "JSCInlines.h"
    3433#include "JSInternalPromise.h"
    3534#include "JSInternalPromiseConstructor.h"
    36 #include "JSObjectInlines.h"
    37 #include "SlotVisitorInlines.h"
    38 #include "StructureInlines.h"
    3935
    4036namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/JSInternalPromisePrototype.cpp

    r204912 r205462  
    2929#include "Error.h"
    3030#include "JSCBuiltins.h"
    31 #include "JSCJSValueInlines.h"
    32 #include "JSCellInlines.h"
     31#include "JSCInlines.h"
    3332#include "JSGlobalObject.h"
    3433#include "JSInternalPromise.h"
    3534#include "Microtask.h"
    36 #include "StructureInlines.h"
    3735
    3836namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/JSJob.cpp

    r205324 r205462  
    2929#include "Error.h"
    3030#include "Exception.h"
    31 #include "JSCJSValueInlines.h"
    32 #include "JSCellInlines.h"
     31#include "JSCInlines.h"
    3332#include "JSGlobalObject.h"
    3433#include "JSObjectInlines.h"
    3534#include "Microtask.h"
    36 #include "SlotVisitorInlines.h"
    3735#include "StrongInlines.h"
    3836

  • trunk/Source/JavaScriptCore/runtime/JSMapIterator.cpp

    r204912 r205462  
    11/*
    2  * Copyright (C) 2013 Apple, Inc. All rights reserved.
     2 * Copyright (C) 2013, 2016 Apple, Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2727#include "JSMapIterator.h"
    2828
    29 #include "JSCJSValueInlines.h"
    30 #include "JSCellInlines.h"
     29#include "JSCInlines.h"
    3130#include "JSMap.h"
    3231#include "MapDataInlines.h"
    33 #include "SlotVisitorInlines.h"
    34 #include "StructureInlines.h"
    3532
    3633namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/JSModuleNamespaceObject.cpp

    r205198 r205462  
    11/*
    2  * Copyright (C) 2015 Apple Inc. All rights reserved.
     2 * Copyright (C) 2015-2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2828
    2929#include "Error.h"
    30 #include "IdentifierInlines.h"
    31 #include "JSCJSValueInlines.h"
    32 #include "JSCellInlines.h"
     30#include "JSCInlines.h"
    3331#include "JSModuleEnvironment.h"
    3432#include "JSModuleRecord.h"
    3533#include "JSPropertyNameIterator.h"
    36 #include "SlotVisitorInlines.h"
    37 #include "StructureInlines.h"
    3834
    3935namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/JSModuleRecord.cpp

    r205324 r205462  
    11/*
    2  * Copyright (C) 2015 Apple Inc. All rights reserved.
     2 * Copyright (C) 2015-2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2929#include "Error.h"
    3030#include "Executable.h"
    31 #include "IdentifierInlines.h"
    32 #include "JSCJSValueInlines.h"
    33 #include "JSCellInlines.h"
     31#include "Interpreter.h"
     32#include "JSCInlines.h"
    3433#include "JSMap.h"
    3534#include "JSModuleEnvironment.h"
    3635#include "JSModuleNamespaceObject.h"
    37 #include "JSObjectInlines.h"
    38 #include "SlotVisitorInlines.h"
    39 #include "StructureInlines.h"
    4036
    4137namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/JSObject.cpp

    r205372 r205462  
    8989}
    9090
    91 ALWAYS_INLINE void JSObject::copyButterfly(CopyVisitor& visitor, Butterfly* butterfly, size_t storageSize)
    92 {
    93     ASSERT(butterfly);
    94    
    95     Structure* structure = this->structure();
    96    
    97     size_t propertyCapacity = structure->outOfLineCapacity();
    98     size_t preCapacity;
    99     size_t indexingPayloadSizeInBytes;
    100     bool hasIndexingHeader = this->hasIndexingHeader();
    101     if (UNLIKELY(hasIndexingHeader)) {
    102         preCapacity = butterfly->indexingHeader()->preCapacity(structure);
    103         indexingPayloadSizeInBytes = butterfly->indexingHeader()->indexingPayloadSizeInBytes(structure);
    104     } else {
    105         preCapacity = 0;
    106         indexingPayloadSizeInBytes = 0;
    107     }
    108     size_t capacityInBytes = Butterfly::totalSize(preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes);
    109     if (visitor.checkIfShouldCopy(butterfly->base(preCapacity, propertyCapacity))) {
    110         Butterfly* newButterfly = Butterfly::createUninitializedDuringCollection(visitor, preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes);
    111 
    112         // Copy the properties.
    113         PropertyStorage currentTarget = newButterfly->propertyStorage();
    114         PropertyStorage currentSource = butterfly->propertyStorage();
    115         for (size_t count = storageSize; count--;)
    116             (--currentTarget)->setWithoutWriteBarrier((--currentSource)->get());
    117        
    118         if (UNLIKELY(hasIndexingHeader)) {
    119             *newButterfly->indexingHeader() = *butterfly->indexingHeader();
    120            
    121             // Copy the array if appropriate.
    122            
    123             WriteBarrier<Unknown>* currentTarget;
    124             WriteBarrier<Unknown>* currentSource;
    125             size_t count;
    126            
    127             switch (this->indexingType()) {
    128             case ALL_UNDECIDED_INDEXING_TYPES:
    129             case ALL_CONTIGUOUS_INDEXING_TYPES:
    130             case ALL_INT32_INDEXING_TYPES:
    131             case ALL_DOUBLE_INDEXING_TYPES: {
    132                 currentTarget = newButterfly->contiguous().data();
    133                 currentSource = butterfly->contiguous().data();
    134                 RELEASE_ASSERT(newButterfly->publicLength() <= newButterfly->vectorLength());
    135                 count = newButterfly->vectorLength();
    136                 break;
    137             }
    138                
    139             case ALL_ARRAY_STORAGE_INDEXING_TYPES: {
    140                 newButterfly->arrayStorage()->copyHeaderFromDuringGC(*butterfly->arrayStorage());
    141                 currentTarget = newButterfly->arrayStorage()->m_vector;
    142                 currentSource = butterfly->arrayStorage()->m_vector;
    143                 count = newButterfly->arrayStorage()->vectorLength();
    144                 break;
    145             }
    146                
    147             default:
    148                 currentTarget = 0;
    149                 currentSource = 0;
    150                 count = 0;
    151                 break;
    152             }
    153 
    154             memcpy(currentTarget, currentSource, count * sizeof(EncodedJSValue));
    155         }
    156        
    157         m_butterfly.setWithoutBarrier(newButterfly);
    158         visitor.didCopy(butterfly->base(preCapacity, propertyCapacity), capacityInBytes);
    159     }
    160 }
    161 
    16291ALWAYS_INLINE void JSObject::visitButterfly(SlotVisitor& visitor, Butterfly* butterfly, Structure* structure)
    16392{
    … …  
    16796    size_t propertyCapacity = structure->outOfLineCapacity();
    16897    size_t preCapacity;
    169     size_t indexingPayloadSizeInBytes;
    17098    bool hasIndexingHeader = this->hasIndexingHeader();
    171     if (UNLIKELY(hasIndexingHeader)) {
     99    if (UNLIKELY(hasIndexingHeader))
    172100        preCapacity = butterfly->indexingHeader()->preCapacity(structure);
    173         indexingPayloadSizeInBytes = butterfly->indexingHeader()->indexingPayloadSizeInBytes(structure);
    174     } else {
     101    else
    175102        preCapacity = 0;
    176         indexingPayloadSizeInBytes = 0;
    177     }
    178     size_t capacityInBytes = Butterfly::totalSize(preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes);
    179 
     103   
     104    HeapCell* base = bitwise_cast<HeapCell*>(butterfly->base(preCapacity, propertyCapacity));
     105   
     106    ASSERT(Heap::heap(base) == visitor.heap());
     107
     108    // Keep the butterfly alive.
     109    visitor.markAuxiliary(base);
     110   
    180111    // Mark the properties.
    181112    visitor.appendValuesHidden(butterfly->propertyStorage() - storageSize, storageSize);
    182     visitor.copyLater(
    183         this, ButterflyCopyToken,
    184         butterfly->base(preCapacity, propertyCapacity), capacityInBytes);
    185113   
    186114    // Mark the array if appropriate.
    … …  
    224152    visitor.m_isCheckingForDefaultMarkViolation = wasCheckingForDefaultMarkViolation;
    225153#endif
    226 }
    227 
    228 void JSObject::copyBackingStore(JSCell* cell, CopyVisitor& visitor, CopyToken token)
    229 {
    230     JSObject* thisObject = jsCast<JSObject*>(cell);
    231     ASSERT_GC_OBJECT_INHERITS(thisObject, info());
    232 
    233     if (token != ButterflyCopyToken)
    234         return;
    235    
    236     Butterfly* butterfly = thisObject->m_butterfly.get();
    237     if (butterfly)
    238         thisObject->copyButterfly(visitor, butterfly, thisObject->structure()->outOfLineSize());
    239154}
    240155
    … …  
    784699        return;
    785700   
    786     globalObject()->haveABadTime(vm);
    787 }
    788 
    789 Butterfly* JSObject::createInitialIndexedStorage(VM& vm, unsigned length, size_t elementSize)
     701    globalObject(vm)->haveABadTime(vm);
     702}
     703
     704Butterfly* JSObject::createInitialIndexedStorage(VM& vm, unsigned length)
    790705{
    791706    ASSERT(length < MAX_ARRAY_INDEX);
    … …  
    794709    ASSERT(!structure()->needsSlowPutIndexing());
    795710    ASSERT(!indexingShouldBeSparse());
    796     unsigned vectorLength = std::max(length, BASE_VECTOR_LEN);
     711    Structure* structure = this->structure(vm);
     712    unsigned propertyCapacity = structure->outOfLineCapacity();
     713    unsigned vectorLength = Butterfly::optimalContiguousVectorLength(propertyCapacity, length);
    797714    Butterfly* newButterfly = Butterfly::createOrGrowArrayRight(
    798         m_butterfly.get(), vm, this, structure(), structure()->outOfLineCapacity(), false, 0,
    799         elementSize * vectorLength);
     715        m_butterfly.get(), vm, this, structure, propertyCapacity, false, 0,
     716        sizeof(EncodedJSValue) * vectorLength);
    800717    newButterfly->setPublicLength(length);
    801718    newButterfly->setVectorLength(vectorLength);
    … …  
    806723{
    807724    DeferGC deferGC(vm.heap);
    808     Butterfly* newButterfly = createInitialIndexedStorage(vm, length, sizeof(EncodedJSValue));
     725    Butterfly* newButterfly = createInitialIndexedStorage(vm, length);
    809726    Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateUndecided);
    810727    setStructureAndButterfly(vm, newStructure, newButterfly);
    … …  
    815732{
    816733    DeferGC deferGC(vm.heap);
    817     Butterfly* newButterfly = createInitialIndexedStorage(vm, length, sizeof(EncodedJSValue));
     734    Butterfly* newButterfly = createInitialIndexedStorage(vm, length);
     735    for (unsigned i = newButterfly->vectorLength(); i--;)
     736        newButterfly->contiguousInt32()[i].setWithoutWriteBarrier(JSValue());
    818737    Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateInt32);
    819738    setStructureAndButterfly(vm, newStructure, newButterfly);
    … …  
    824743{
    825744    DeferGC deferGC(vm.heap);
    826     Butterfly* newButterfly = createInitialIndexedStorage(vm, length, sizeof(double));
     745    Butterfly* newButterfly = createInitialIndexedStorage(vm, length);
    827746    for (unsigned i = newButterfly->vectorLength(); i--;)
    828747        newButterfly->contiguousDouble()[i] = PNaN;
    … …  
    835754{
    836755    DeferGC deferGC(vm.heap);
    837     Butterfly* newButterfly = createInitialIndexedStorage(vm, length, sizeof(EncodedJSValue));
     756    Butterfly* newButterfly = createInitialIndexedStorage(vm, length);
     757    for (unsigned i = newButterfly->vectorLength(); i--;)
     758        newButterfly->contiguous()[i].setWithoutWriteBarrier(JSValue());
    838759    Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateContiguous);
    839760    setStructureAndButterfly(vm, newStructure, newButterfly);
    … …  
    858779    result->m_numValuesInVector = 0;
    859780    result->m_indexBias = 0;
     781    for (size_t i = vectorLength; i--;)
     782        result->m_vector[i].setWithoutWriteBarrier(JSValue());
    860783    Structure* newStructure = Structure::nonPropertyTransition(vm, structure, structure->suggestedArrayStorageTransition());
    861784    setStructureAndButterfly(vm, newStructure, newButterfly);
    … …  
    865788ArrayStorage* JSObject::createInitialArrayStorage(VM& vm)
    866789{
    867     return createArrayStorage(vm, 0, BASE_VECTOR_LEN);
     790    return createArrayStorage(
     791        vm, 0, ArrayStorage::optimalVectorLength(0, structure(vm)->outOfLineCapacity(), 0));
    868792}
    869793
    … …  
    871795{
    872796    ASSERT(hasUndecided(indexingType()));
     797
     798    Butterfly* butterfly = m_butterfly.get();
     799    for (unsigned i = butterfly->vectorLength(); i--;)
     800        butterfly->contiguousInt32()[i].setWithoutWriteBarrier(JSValue());
     801
    873802    setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateInt32));
    874803    return m_butterfly.get()->contiguousInt32();
    … …  
    890819{
    891820    ASSERT(hasUndecided(indexingType()));
     821
     822    Butterfly* butterfly = m_butterfly.get();
     823    for (unsigned i = butterfly->vectorLength(); i--;)
     824        butterfly->contiguous()[i].setWithoutWriteBarrier(JSValue());
     825
    892826    setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateContiguous));
    893827    return m_butterfly.get()->contiguous();
    … …  
    926860    unsigned vectorLength = m_butterfly.get()->vectorLength();
    927861    ArrayStorage* storage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
    928     // No need to copy elements.
     862   
     863    for (unsigned i = vectorLength; i--;)
     864        storage->m_vector[i].setWithoutWriteBarrier(JSValue());
    929865   
    930866    Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), transition);
    … …  
    947883        double* currentAsDouble = bitwise_cast<double*>(current);
    948884        JSValue v = current->get();
    949         if (!v) {
     885        // NOTE: Since this may be used during initialization, v could be garbage. If it's garbage,
     886        // that means it will be overwritten later.
     887        if (!v.isInt32()) {
    950888            *currentAsDouble = PNaN;
    951889            continue;
    952890        }
    953         ASSERT(v.isInt32());
    954891        *currentAsDouble = v.asInt32();
    955892    }
    … …  
    975912    ArrayStorage* newStorage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
    976913    Butterfly* butterfly = m_butterfly.get();
    977     for (unsigned i = 0; i < butterfly->publicLength(); i++) {
     914    for (unsigned i = 0; i < vectorLength; i++) {
    978915        JSValue v = butterfly->contiguous()[i].get();
    979         if (v) {
    980             newStorage->m_vector[i].setWithoutWriteBarrier(v);
     916        newStorage->m_vector[i].setWithoutWriteBarrier(v);
     917        if (v)
    981918            newStorage->m_numValuesInVector++;
    982         } else
    983             ASSERT(newStorage->m_vector[i].get().isEmpty());
    984919    }
    985920   
    … …  
    1023958    ArrayStorage* newStorage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
    1024959    Butterfly* butterfly = m_butterfly.get();
    1025     for (unsigned i = 0; i < butterfly->publicLength(); i++) {
     960    for (unsigned i = 0; i < vectorLength; i++) {
    1026961        double value = butterfly->contiguousDouble()[i];
    1027         if (value == value) {
    1028             newStorage->m_vector[i].setWithoutWriteBarrier(JSValue(JSValue::EncodeAsDouble, value));
     962        newStorage->m_vector[i].setWithoutWriteBarrier(JSValue(JSValue::EncodeAsDouble, value));
     963        if (value == value)
    1029964            newStorage->m_numValuesInVector++;
    1030         } else
    1031             ASSERT(newStorage->m_vector[i].get().isEmpty());
    1032965    }
    1033966   
    … …  
    1050983    ArrayStorage* newStorage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
    1051984    Butterfly* butterfly = m_butterfly.get();
    1052     for (unsigned i = 0; i < butterfly->publicLength(); i++) {
     985    for (unsigned i = 0; i < vectorLength; i++) {
    1053986        JSValue v = butterfly->contiguous()[i].get();
    1054         if (v) {
    1055             newStorage->m_vector[i].setWithoutWriteBarrier(v);
     987        newStorage->m_vector[i].setWithoutWriteBarrier(v);
     988        if (v)
    1056989            newStorage->m_numValuesInVector++;
    1057         } else
    1058             ASSERT(newStorage->m_vector[i].get().isEmpty());
    1059990    }
    1060991   
    … …  
    24072338        if (structure(vm)->needsSlowPutIndexing()) {
    24082339            // Convert the indexing type to the SlowPutArrayStorage and retry.
    2409             createArrayStorage(vm, i + 1, getNewVectorLength(0, 0, i + 1));
     2340            createArrayStorage(vm, i + 1, getNewVectorLength(0, 0, 0, i + 1));
    24102341            return putByIndex(this, exec, i, value, shouldThrow);
    24112342        }
    … …  
    25482479        }
    25492480        if (structure(vm)->needsSlowPutIndexing()) {
    2550             ArrayStorage* storage = createArrayStorage(vm, i + 1, getNewVectorLength(0, 0, i + 1));
     2481            ArrayStorage* storage = createArrayStorage(vm, i + 1, getNewVectorLength(0, 0, 0, i + 1));
    25512482            storage->m_vector[i].set(vm, this, value);
    25522483            storage->m_numValuesInVector++;
    … …  
    26672598}
    26682599
    2669 ALWAYS_INLINE unsigned JSObject::getNewVectorLength(unsigned currentVectorLength, unsigned currentLength, unsigned desiredLength)
     2600// NOTE: This method is for ArrayStorage vectors.
     2601ALWAYS_INLINE unsigned JSObject::getNewVectorLength(unsigned indexBias, unsigned currentVectorLength, unsigned currentLength, unsigned desiredLength)
    26702602{
    26712603    ASSERT(desiredLength <= MAX_STORAGE_VECTOR_LENGTH);
    … …  
    26842616    ASSERT(increasedLength >= desiredLength);
    26852617
    2686     lastArraySize = std::min(increasedLength, FIRST_VECTOR_GROW);
    2687 
    2688     return std::min(increasedLength, MAX_STORAGE_VECTOR_LENGTH);
     2618    lastArraySize = std::min(increasedLength, FIRST_ARRAY_STORAGE_VECTOR_GROW);
     2619
     2620    return ArrayStorage::optimalVectorLength(
     2621        indexBias, structure()->outOfLineCapacity(),
     2622        std::min(increasedLength, MAX_STORAGE_VECTOR_LENGTH));
    26892623}
    26902624
    26912625ALWAYS_INLINE unsigned JSObject::getNewVectorLength(unsigned desiredLength)
    26922626{
    2693     unsigned vectorLength;
    2694     unsigned length;
     2627    unsigned indexBias = 0;
     2628    unsigned vectorLength = 0;
     2629    unsigned length = 0;
    26952630   
    26962631    if (hasIndexedProperties(indexingType())) {
     2632        if (ArrayStorage* storage = arrayStorageOrNull())
     2633            indexBias = storage->m_indexBias;
    26972634        vectorLength = m_butterfly.get()->vectorLength();
    26982635        length = m_butterfly.get()->publicLength();
    2699     } else {
    2700         vectorLength = 0;
    2701         length = 0;
    2702     }
    2703 
    2704     return getNewVectorLength(vectorLength, length, desiredLength);
     2636    }
     2637
     2638    return getNewVectorLength(indexBias, vectorLength, length, desiredLength);
    27052639}
    27062640
    … …  
    27552689bool JSObject::increaseVectorLength(VM& vm, unsigned newLength)
    27562690{
     2691    ArrayStorage* storage = arrayStorage();
     2692   
     2693    unsigned vectorLength = storage->vectorLength();
     2694    unsigned availableVectorLength = storage->availableVectorLength(structure(vm), vectorLength);
     2695    if (availableVectorLength >= newLength) {
     2696        // The cell was already big enough for the desired length!
     2697        for (unsigned i = vectorLength; i < availableVectorLength; ++i)
     2698            storage->m_vector[i].clear();
     2699        storage->setVectorLength(availableVectorLength);
     2700        return true;
     2701    }
     2702   
    27572703    // This function leaves the array in an internally inconsistent state, because it does not move any values from sparse value map
    27582704    // to the vector. Callers have to account for that, because they can do it more efficiently.
    … …  
    27602706        return false;
    27612707
    2762     ArrayStorage* storage = arrayStorage();
    2763    
    27642708    if (newLength >= MIN_SPARSE_ARRAY_INDEX
    27652709        && !isDenseEnoughForVector(newLength, storage->m_numValuesInVector))
    … …  
    27672711
    27682712    unsigned indexBias = storage->m_indexBias;
    2769     unsigned vectorLength = storage->vectorLength();
    27702713    ASSERT(newLength > vectorLength);
    27712714    unsigned newVectorLength = getNewVectorLength(newLength);
    … …  
    27802723        if (!newButterfly)
    27812724            return false;
     2725        for (unsigned i = vectorLength; i < newVectorLength; ++i)
     2726            newButterfly->arrayStorage()->m_vector[i].clear();
    27822727        newButterfly->arrayStorage()->setVectorLength(newVectorLength);
    27832728        setButterflyWithoutChangingStructure(vm, newButterfly);
    … …  
    27942739    if (!newButterfly)
    27952740        return false;
     2741    for (unsigned i = vectorLength; i < newVectorLength; ++i)
     2742        newButterfly->arrayStorage()->m_vector[i].clear();
    27962743    newButterfly->arrayStorage()->setVectorLength(newVectorLength);
    27972744    newButterfly->arrayStorage()->m_indexBias = newIndexBias;
    … …  
    28082755    ASSERT(length > butterfly->vectorLength());
    28092756   
    2810     unsigned newVectorLength = std::min(
    2811         length << 1,
    2812         MAX_STORAGE_VECTOR_LENGTH);
    28132757    unsigned oldVectorLength = butterfly->vectorLength();
    2814     DeferGC deferGC(vm.heap);
    2815     butterfly = butterfly->growArrayRight(
    2816         vm, this, structure(), structure()->outOfLineCapacity(), true,
    2817         oldVectorLength * sizeof(EncodedJSValue),
    2818         newVectorLength * sizeof(EncodedJSValue));
    2819     if (!butterfly)
    2820         return false;
    2821     m_butterfly.set(vm, this, butterfly);
     2758    unsigned newVectorLength;
     2759   
     2760    Structure* structure = this->structure(vm);
     2761    unsigned propertyCapacity = structure->outOfLineCapacity();
     2762   
     2763    unsigned availableOldLength =
     2764        Butterfly::availableContiguousVectorLength(propertyCapacity, oldVectorLength);
     2765    if (availableOldLength >= length) {
     2766        // This is the case where someone else selected a vector length that caused internal
     2767        // fragmentation. If we did our jobs right, this would never happen. But I bet we will mess
     2768        // this up, so this defense should stay.
     2769        newVectorLength = availableOldLength;
     2770    } else {
     2771        newVectorLength = Butterfly::optimalContiguousVectorLength(
     2772            propertyCapacity, std::min(length << 1, MAX_STORAGE_VECTOR_LENGTH));
     2773        butterfly = butterfly->growArrayRight(
     2774            vm, this, structure, propertyCapacity, true,
     2775            oldVectorLength * sizeof(EncodedJSValue),
     2776            newVectorLength * sizeof(EncodedJSValue));
     2777        if (!butterfly)
     2778            return false;
     2779        m_butterfly.set(vm, this, butterfly);
     2780    }
    28222781
    28232782    butterfly->setVectorLength(newVectorLength);
    … …  
    28252784    if (hasDouble(indexingType())) {
    28262785        for (unsigned i = oldVectorLength; i < newVectorLength; ++i)
    2827             butterfly->contiguousDouble().data()[i] = PNaN;
    2828     }
     2786            butterfly->contiguousDouble()[i] = PNaN;
     2787    } else {
     2788        for (unsigned i = oldVectorLength; i < newVectorLength; ++i)
     2789            butterfly->contiguous()[i].clear();
     2790    }
     2791
    28292792    return true;
    28302793}

  • trunk/Source/JavaScriptCore/runtime/JSObject.h

    r205372 r205462  
    2727#include "ArrayConventions.h"
    2828#include "ArrayStorage.h"
     29#include "AuxiliaryBarrier.h"
    2930#include "Butterfly.h"
    3031#include "CallFrame.h"
    3132#include "ClassInfo.h"
    3233#include "CommonIdentifiers.h"
    33 #include "CopyBarrier.h"
    3434#include "CustomGetterSetter.h"
    3535#include "DeferGC.h"
    3636#include "Heap.h"
    37 #include "HeapInlines.h"
    3837#include "IndexingHeaderInlines.h"
    3938#include "JSCell.h"
    … …  
    104103    JS_EXPORT_PRIVATE static size_t estimatedSize(JSCell*);
    105104    JS_EXPORT_PRIVATE static void visitChildren(JSCell*, SlotVisitor&);
    106     JS_EXPORT_PRIVATE static void copyBackingStore(JSCell*, CopyVisitor&, CopyToken);
    107105    JS_EXPORT_PRIVATE static void heapSnapshot(JSCell*, HeapSnapshotBuilder&);
    108106
    … …  
    421419    }
    422420
     421    // NOTE: Clients of this method may call it more than once for any index, and this is supposed
     422    // to work.
    423423    void initializeIndex(VM& vm, unsigned i, JSValue v, IndexingType indexingType)
    424424    {
    … …  
    693693    void setStructure(VM&, Structure*);
    694694    void setStructureAndButterfly(VM&, Structure*, Butterfly*);
    695     void setStructureAndReallocateStorageIfNecessary(VM&, unsigned oldCapacity, Structure*);
    696     void setStructureAndReallocateStorageIfNecessary(VM&, Structure*);
    697695
    698696    JS_EXPORT_PRIVATE void convertToDictionary(VM&);
    … …  
    709707        ASSERT(!isGlobalObject() || ((JSObject*)structure()->globalObject()) == this);
    710708        return structure()->globalObject();
     709    }
     710       
     711    JSGlobalObject* globalObject(VM& vm) const
     712    {
     713        ASSERT(structure(vm)->globalObject());
     714        ASSERT(!isGlobalObject() || ((JSObject*)structure()->globalObject()) == this);
     715        return structure(vm)->globalObject();
    711716    }
    712717       
    … …  
    804809       
    805810    void visitButterfly(SlotVisitor&, Butterfly*, Structure*);
    806     void copyButterfly(CopyVisitor&, Butterfly*, size_t storageSize);
    807811
    808812    // Call this if you know that the object is in a mode where it has array
    … …  
    915919    void isString();
    916920       
    917     Butterfly* createInitialIndexedStorage(VM&, unsigned length, size_t elementSize);
     921    Butterfly* createInitialIndexedStorage(VM&, unsigned length);
    918922       
    919923    ArrayStorage* enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(VM&, ArrayStorage*);
    … …  
    939943    JS_EXPORT_PRIVATE bool putDirectIndexBeyondVectorLength(ExecState*, unsigned propertyName, JSValue, unsigned attributes, PutDirectIndexMode);
    940944       
    941     unsigned getNewVectorLength(unsigned currentVectorLength, unsigned currentLength, unsigned desiredLength);
     945    unsigned getNewVectorLength(unsigned indexBias, unsigned currentVectorLength, unsigned currentLength, unsigned desiredLength);
    942946    unsigned getNewVectorLength(unsigned desiredLength);
    943947
    … …  
    956960
    957961protected:
    958     CopyBarrier<Butterfly> m_butterfly;
     962    AuxiliaryBarrier<Butterfly*> m_butterfly;
    959963#if USE(JSVALUE32_64)
    960964private:
    … …  
    14181422    validateOffset(offset);
    14191423    ASSERT(newStructure->isValidOffset(offset));
    1420     setStructureAndReallocateStorageIfNecessary(vm, newStructure);
    1421 
     1424    DeferGC deferGC(vm.heap);
     1425    size_t oldCapacity = structure->outOfLineCapacity();
     1426    size_t newCapacity = newStructure->outOfLineCapacity();
     1427    ASSERT(oldCapacity <= newCapacity);
     1428    if (oldCapacity == newCapacity)
     1429        setStructure(vm, newStructure);
     1430    else {
     1431        Butterfly* newButterfly = growOutOfLineStorage(vm, oldCapacity, newCapacity);
     1432        setStructureAndButterfly(vm, newStructure, newButterfly);
     1433    }
    14221434    putDirect(vm, offset, value);
    14231435    slot.setNewProperty(this, offset);
    … …  
    14251437        newStructure->setContainsReadOnlyProperties();
    14261438    return true;
    1427 }
    1428 
    1429 inline void JSObject::setStructureAndReallocateStorageIfNecessary(VM& vm, unsigned oldCapacity, Structure* newStructure)
    1430 {
    1431     ASSERT(oldCapacity <= newStructure->outOfLineCapacity());
    1432    
    1433     if (oldCapacity == newStructure->outOfLineCapacity()) {
    1434         setStructure(vm, newStructure);
    1435         return;
    1436     }
    1437 
    1438     DeferGC deferGC(vm.heap);
    1439     Butterfly* newButterfly = growOutOfLineStorage(
    1440         vm, oldCapacity, newStructure->outOfLineCapacity());
    1441     setStructureAndButterfly(vm, newStructure, newButterfly);
    1442 }
    1443 
    1444 inline void JSObject::setStructureAndReallocateStorageIfNecessary(VM& vm, Structure* newStructure)
    1445 {
    1446     setStructureAndReallocateStorageIfNecessary(
    1447         vm, structure(vm)->outOfLineCapacity(), newStructure);
    14481439}
    14491440

  • trunk/Source/JavaScriptCore/runtime/JSObjectInlines.h

    r205324 r205462  
    22 *  Copyright (C) 1999-2001 Harri Porten (porten@kde.org)
    33 *  Copyright (C) 2001 Peter Kelly (pmk@post.com)
    4  *  Copyright (C) 2003-2006, 2008, 2009, 2012-2015 Apple Inc. All rights reserved.
     4 *  Copyright (C) 2003-2006, 2008, 2009, 2012-2016 Apple Inc. All rights reserved.
    55 *  Copyright (C) 2007 Eric Seidel (eric@webkit.org)
    66 *
    … …  
    2525#define JSObjectInlines_h
    2626
     27#include "AuxiliaryBarrierInlines.h"
    2728#include "Error.h"
    2829#include "JSObject.h"

  • trunk/Source/JavaScriptCore/runtime/JSPromise.cpp

    r204912 r205462  
    2929#include "BuiltinNames.h"
    3030#include "Error.h"
    31 #include "JSCJSValueInlines.h"
    32 #include "JSCellInlines.h"
     31#include "JSCInlines.h"
    3332#include "JSPromiseConstructor.h"
    3433#include "Microtask.h"
    35 #include "SlotVisitorInlines.h"
    36 #include "StructureInlines.h"
    3734
    3835namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/JSPromiseConstructor.cpp

    r205198 r205462  
    3333#include "IteratorOperations.h"
    3434#include "JSCBuiltins.h"
    35 #include "JSCJSValueInlines.h"
    36 #include "JSCellInlines.h"
     35#include "JSCInlines.h"
    3736#include "JSFunction.h"
    3837#include "JSPromise.h"
    … …  
    4039#include "Lookup.h"
    4140#include "NumberObject.h"
    42 #include "StructureInlines.h"
    4341
    4442namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/JSPromiseDeferred.cpp

    r205324 r205462  
    3030#include "Error.h"
    3131#include "Exception.h"
    32 #include "JSCJSValueInlines.h"
    33 #include "JSCellInlines.h"
     32#include "JSCInlines.h"
    3433#include "JSObjectInlines.h"
    3534#include "JSPromise.h"
    3635#include "JSPromiseConstructor.h"
    37 #include "SlotVisitorInlines.h"
    38 #include "StructureInlines.h"
    3936
    4037namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/JSPromisePrototype.cpp

    r204912 r205462  
    3030#include "Error.h"
    3131#include "JSCBuiltins.h"
    32 #include "JSCJSValueInlines.h"
    33 #include "JSCellInlines.h"
     32#include "JSCInlines.h"
    3433#include "JSFunction.h"
    3534#include "JSGlobalObject.h"
    3635#include "JSPromise.h"
    3736#include "Microtask.h"
    38 #include "StructureInlines.h"
    3937
    4038namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/JSPropertyNameIterator.cpp

    r205198 r205462  
    11/*
    2  * Copyright (C) 2015 Apple, Inc. All rights reserved.
     2 * Copyright (C) 2015-2016 Apple, Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2727#include "JSPropertyNameIterator.h"
    2828
    29 #include "IdentifierInlines.h"
    3029#include "IteratorOperations.h"
    31 #include "JSCJSValueInlines.h"
    32 #include "JSCellInlines.h"
     30#include "JSCInlines.h"
    3331#include "JSPropertyNameEnumerator.h"
    34 #include "SlotVisitorInlines.h"
    35 #include "StructureInlines.h"
    3632
    3733namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/JSScope.cpp

    r204912 r205462  
    218218JSObject* JSScope::resolve(ExecState* exec, JSScope* scope, const Identifier& ident)
    219219{
     220    VM& vm = exec->vm();
    220221    ScopeChainIterator end = scope->end();
    221222    ScopeChainIterator it = scope->begin();
    … …  
    226227        // Global scope.
    227228        if (++it == end) {
    228             JSScope* globalScopeExtension = scope->globalObject()->globalScopeExtension();
     229            JSScope* globalScopeExtension = scope->globalObject(vm)->globalScopeExtension();
    229230            if (UNLIKELY(globalScopeExtension)) {
    230231                if (object->hasProperty(exec, ident))

  • trunk/Source/JavaScriptCore/runtime/JSScope.h

    r204912 r205462  
    7070
    7171    JSGlobalObject* globalObject();
    72     VM* vm();
     72    JSGlobalObject* globalObject(VM&);
    7373    JSObject* globalThis();
    7474
    … …  
    130130}
    131131
    132 inline VM* JSScope::vm()
     132inline JSGlobalObject* JSScope::globalObject(VM& vm)
    133133{
    134     return MarkedBlock::blockFor(this)->vm();
     134    return structure(vm)->globalObject();
    135135}
    136136

  • trunk/Source/JavaScriptCore/runtime/JSSetIterator.cpp

    r204912 r205462  
    11/*
    2  * Copyright (C) 2013 Apple, Inc. All rights reserved.
     2 * Copyright (C) 2013, 2016 Apple, Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2727#include "JSSetIterator.h"
    2828
    29 #include "JSCJSValueInlines.h"
    30 #include "JSCellInlines.h"
     29#include "JSCInlines.h"
    3130#include "JSSet.h"
    3231#include "MapDataInlines.h"
    33 #include "SlotVisitorInlines.h"
    34 #include "StructureInlines.h"
    3532
    3633namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/JSStringIterator.cpp

    r204912 r205462  
    2929
    3030#include "BuiltinNames.h"
    31 #include "JSCJSValueInlines.h"
    32 #include "JSCellInlines.h"
    33 #include "StructureInlines.h"
     31#include "JSCInlines.h"
    3432
    3533namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/JSTemplateRegistryKey.cpp

    r204912 r205462  
    11/*
    22 * Copyright (C) 2015 Yusuke Suzuki <utatane.tea@gmail.com>.
     3 * Copyright (C) 2016 Apple Inc. All Rights Reserved.
    34 *
    45 * Redistribution and use in source and binary forms, with or without
    … …  
    2728#include "JSTemplateRegistryKey.h"
    2829
    29 #include "JSCJSValueInlines.h"
    30 #include "JSCellInlines.h"
    31 #include "StructureInlines.h"
     30#include "JSCInlines.h"
    3231#include "VM.h"
    3332

  • trunk/Source/JavaScriptCore/runtime/JSTypedArrayViewConstructor.cpp

    r205198 r205462  
    3131#include "GetterSetter.h"
    3232#include "JSCBuiltins.h"
    33 #include "JSCellInlines.h"
     33#include "JSCInlines.h"
    3434#include "JSGenericTypedArrayViewConstructorInlines.h"
    35 #include "JSObject.h"
    3635#include "JSTypedArrayViewPrototype.h"
    3736#include "JSTypedArrays.h"

  • trunk/Source/JavaScriptCore/runtime/JSTypedArrayViewPrototype.cpp

    r205324 r205462  
    3030#include "CallFrame.h"
    3131#include "GetterSetter.h"
    32 #include "JSCellInlines.h"
     32#include "JSCInlines.h"
    3333#include "JSFunction.h"
    3434#include "JSGenericTypedArrayViewPrototypeFunctions.h"

  • trunk/Source/JavaScriptCore/runtime/JSWeakMap.cpp

    r205131 r205462  
    2727#include "JSWeakMap.h"
    2828
    29 #include "JSCJSValueInlines.h"
    30 #include "SlotVisitorInlines.h"
    31 #include "StructureInlines.h"
     29#include "JSCInlines.h"
    3230#include "WeakMapData.h"
    33 #include "WriteBarrierInlines.h"
    3431
    3532namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/JSWeakSet.cpp

    r205131 r205462  
    2727#include "JSWeakSet.h"
    2828
    29 #include "JSCJSValueInlines.h"
    30 #include "SlotVisitorInlines.h"
    31 #include "StructureInlines.h"
     29#include "JSCInlines.h"
    3230#include "WeakMapData.h"
    33 #include "WriteBarrierInlines.h"
    3431
    3532namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/MapConstructor.cpp

    r205324 r205462  
    3030#include "GetterSetter.h"
    3131#include "IteratorOperations.h"
    32 #include "JSCJSValueInlines.h"
    33 #include "JSCellInlines.h"
     32#include "JSCInlines.h"
    3433#include "JSGlobalObject.h"
    3534#include "JSMap.h"
    3635#include "JSObjectInlines.h"
    3736#include "MapPrototype.h"
    38 #include "StructureInlines.h"
    3937
    4038namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/MapIteratorPrototype.cpp

    r205198 r205462  
    2828
    2929#include "IteratorOperations.h"
    30 #include "JSCJSValueInlines.h"
    31 #include "JSCellInlines.h"
     30#include "JSCInlines.h"
    3231#include "JSMapIterator.h"
    33 #include "StructureInlines.h"
    3432
    3533namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/MapPrototype.cpp

    r205198 r205462  
    3232#include "GetterSetter.h"
    3333#include "IteratorOperations.h"
    34 #include "JSCJSValueInlines.h"
    35 #include "JSFunctionInlines.h"
     34#include "JSCInlines.h"
    3635#include "JSMap.h"
    3736#include "JSMapIterator.h"
    3837#include "Lookup.h"
    3938#include "MapDataInlines.h"
    40 #include "StructureInlines.h"
    4139
    4240#include "MapPrototype.lut.h"

  • trunk/Source/JavaScriptCore/runtime/NativeErrorConstructor.cpp

    r204912 r205462  
    2323
    2424#include "ErrorInstance.h"
     25#include "Interpreter.h"
    2526#include "JSFunction.h"
    2627#include "JSString.h"

  • trunk/Source/JavaScriptCore/runtime/NativeStdFunctionCell.cpp

    r204912 r205462  
    2727#include "NativeStdFunctionCell.h"
    2828
    29 #include "JSCJSValueInlines.h"
    30 #include "JSCellInlines.h"
    31 #include "JSFunctionInlines.h"
    32 #include "SlotVisitorInlines.h"
     29#include "JSCInlines.h"
    3330
    3431namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/Operations.h

    r205198 r205462  
    200200}
    201201
     202inline bool scribbleFreeCells()
     203{
     204    return !ASSERT_DISABLED || Options::scribbleFreeCells();
     205}
     206
     207inline void scribble(void* base, size_t size)
     208{
     209    for (size_t i = size / sizeof(EncodedJSValue); i--;) {
     210        // Use a 16-byte aligned value to ensure that it passes the cell check.
     211        static_cast<EncodedJSValue*>(base)[i] = JSValue::encode(
     212            bitwise_cast<JSCell*>(static_cast<intptr_t>(0xbadbeef0)));
     213    }
     214}
     215
    202216} // namespace JSC
    203217

  • trunk/Source/JavaScriptCore/runtime/Options.cpp

    r204394 r205462  
    371371        Options::useOSREntryToFTL() = false;
    372372    }
    373 
     373   
    374374#if PLATFORM(IOS) && !PLATFORM(IOS_SIMULATOR) && __IPHONE_OS_VERSION_MIN_REQUIRED >= 100000
    375375    // Override globally for now. Longer term we'll just make the default

  • trunk/Source/JavaScriptCore/runtime/Options.h

    r204912 r205462  
    183183    v(bool, verboseSanitizeStack, false, Normal, nullptr) \
    184184    v(bool, useGenerationalGC, true, Normal, nullptr) \
     185    v(bool, scribbleFreeCells, false, Normal, nullptr) \
     186    v(double, sizeClassProgression, 1.4, Normal, nullptr) \
     187    v(unsigned, largeAllocationCutoff, 100000, Normal, nullptr) \
     188    v(bool, dumpSizeClasses, false, Normal, nullptr) \
     189    v(bool, useBumpAllocator, true, Normal, nullptr) \
    185190    v(bool, eagerlyUpdateTopCallFrame, false, Normal, nullptr) \
    186191    \

  • trunk/Source/JavaScriptCore/runtime/PropertyTable.cpp

    r204912 r205462  
    2727#include "PropertyMapHashTable.h"
    2828
    29 #include "JSCJSValueInlines.h"
    30 #include "JSCellInlines.h"
    31 #include "SlotVisitorInlines.h"
    32 #include "StructureInlines.h"
     29#include "JSCInlines.h"
    3330
    3431namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/ProxyConstructor.cpp

    r205198 r205462  
    2929#include "Error.h"
    3030#include "IdentifierInlines.h"
    31 #include "JSCJSValueInlines.h"
    32 #include "JSCellInlines.h"
     31#include "JSCInlines.h"
    3332#include "ObjectConstructor.h"
    3433#include "ObjectPrototype.h"

  • trunk/Source/JavaScriptCore/runtime/ProxyObject.cpp

    r205198 r205462  
    3030#include "Error.h"
    3131#include "IdentifierInlines.h"
    32 #include "JSCJSValueInlines.h"
    33 #include "JSCellInlines.h"
     32#include "JSCInlines.h"
    3433#include "JSObjectInlines.h"
    3534#include "ObjectConstructor.h"

  • trunk/Source/JavaScriptCore/runtime/ProxyRevoke.cpp

    r204912 r205462  
    2727#include "ProxyRevoke.h"
    2828
    29 #include "JSCJSValueInlines.h"
     29#include "JSCInlines.h"
    3030#include "ProxyObject.h"
    31 #include "SlotVisitorInlines.h"
    32 #include "StructureInlines.h"
    3331
    3432namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/RegExp.cpp

    r204912 r205462  
    297297}
    298298
    299 int RegExp::match(VM& vm, const String& s, unsigned startOffset, Vector<int, 32>& ovector)
     299int RegExp::match(VM& vm, const String& s, unsigned startOffset, Vector<int>& ovector)
    300300{
    301301    return matchInline(vm, s, startOffset, ovector);
    … …  
    303303
    304304bool RegExp::matchConcurrently(
    305     VM& vm, const String& s, unsigned startOffset, int& position, Vector<int, 32>& ovector)
     305    VM& vm, const String& s, unsigned startOffset, int& position, Vector<int>& ovector)
    306306{
    307307    ConcurrentJITLocker locker(m_lock);
    … …  
    383383{
    384384    int offsetVectorSize = (m_numSubpatterns + 1) * 2;
    385     Vector<int, 32> interpreterOvector;
     385    Vector<int> interpreterOvector;
    386386    interpreterOvector.resize(offsetVectorSize);
    387387    int* interpreterOffsetVector = interpreterOvector.data();

  • trunk/Source/JavaScriptCore/runtime/RegExp.h

    r204912 r205462  
    6565    const char* errorMessage() const { return m_constructionError; }
    6666
    67     JS_EXPORT_PRIVATE int match(VM&, const String&, unsigned startOffset, Vector<int, 32>& ovector);
     67    JS_EXPORT_PRIVATE int match(VM&, const String&, unsigned startOffset, Vector<int>& ovector);
    6868
    6969    // Returns false if we couldn't run the regular expression for any reason.
    70     bool matchConcurrently(VM&, const String&, unsigned startOffset, int& position, Vector<int, 32>& ovector);
     70    bool matchConcurrently(VM&, const String&, unsigned startOffset, int& position, Vector<int>& ovector);
    7171   
    7272    JS_EXPORT_PRIVATE MatchResult match(VM&, const String&, unsigned startOffset);
    … …  
    7575
    7676    // Call these versions of the match functions if you're desperate for performance.
    77     int matchInline(VM&, const String&, unsigned startOffset, Vector<int, 32>& ovector);
     77    template<typename VectorType>
     78    int matchInline(VM&, const String&, unsigned startOffset, VectorType& ovector);
    7879    MatchResult matchInline(VM&, const String&, unsigned startOffset);
    7980   

  • trunk/Source/JavaScriptCore/runtime/RegExpConstructor.h

    r204912 r205462  
    8181    RegExpCachedResult m_cachedResult;
    8282    bool m_multiline;
    83     Vector<int, 32> m_ovector;
     83    Vector<int> m_ovector;
    8484};
    8585

  • trunk/Source/JavaScriptCore/runtime/RegExpInlines.h

    r204912 r205462  
    9595}
    9696
    97 ALWAYS_INLINE int RegExp::matchInline(VM& vm, const String& s, unsigned startOffset, Vector<int, 32>& ovector)
     97template<typename VectorType>
     98ALWAYS_INLINE int RegExp::matchInline(VM& vm, const String& s, unsigned startOffset, VectorType& ovector)
    9899{
    99100#if ENABLE(REGEXP_TRACING)

  • trunk/Source/JavaScriptCore/runtime/RegExpMatchesArray.h

    r204912 r205462  
    3535ALWAYS_INLINE JSArray* tryCreateUninitializedRegExpMatchesArray(VM& vm, Structure* structure, unsigned initialLength)
    3636{
    37     unsigned vectorLength = std::max(BASE_VECTOR_LEN, initialLength);
     37    unsigned vectorLength = initialLength;
    3838    if (vectorLength > MAX_STORAGE_VECTOR_LENGTH)
    3939        return 0;
    4040
    41     void* temp;
    42     if (!vm.heap.tryAllocateStorage(0, Butterfly::totalSize(0, structure->outOfLineCapacity(), true, vectorLength * sizeof(EncodedJSValue)), &temp))
    43         return 0;
     41    void* temp = vm.heap.tryAllocateAuxiliary(nullptr, Butterfly::totalSize(0, structure->outOfLineCapacity(), true, vectorLength * sizeof(EncodedJSValue)));
     42    if (!temp)
     43        return nullptr;
    4444    Butterfly* butterfly = Butterfly::fromBase(temp, 0, structure->outOfLineCapacity());
    4545    butterfly->setVectorLength(vectorLength);
    4646    butterfly->setPublicLength(initialLength);
    47 
     47   
     48    for (unsigned i = initialLength; i < vectorLength; ++i)
     49        butterfly->contiguous()[i].clear();
     50   
    4851    return JSArray::createWithButterfly(vm, structure, butterfly);
    4952}
    … …  
    6871    // https://bugs.webkit.org/show_bug.cgi?id=155144
    6972   
     73    auto setProperties = [&] () {
     74        array->putDirect(vm, RegExpMatchesArrayIndexPropertyOffset, jsNumber(result.start));
     75        array->putDirect(vm, RegExpMatchesArrayInputPropertyOffset, input);
     76    };
     77   
     78    unsigned numSubpatterns = regExp->numSubpatterns();
     79   
    7080    if (UNLIKELY(globalObject->isHavingABadTime())) {
    71         array = JSArray::tryCreateUninitialized(vm, globalObject->regExpMatchesArrayStructure(), regExp->numSubpatterns() + 1);
     81        array = JSArray::tryCreateUninitialized(vm, globalObject->regExpMatchesArrayStructure(), numSubpatterns + 1);
     82       
     83        setProperties();
     84       
     85        array->initializeIndex(vm, 0, jsUndefined());
     86       
     87        for (unsigned i = 1; i <= numSubpatterns; ++i)
     88            array->initializeIndex(vm, i, jsUndefined());
     89       
     90        // Now the object is safe to scan by GC.
    7291       
    7392        array->initializeIndex(vm, 0, jsSubstringOfResolved(vm, input, result.start, result.end - result.start));
    7493       
    75         if (unsigned numSubpatterns = regExp->numSubpatterns()) {
    76             for (unsigned i = 1; i <= numSubpatterns; ++i) {
    77                 int start = subpatternResults[2 * i];
    78                 if (start >= 0)
    79                     array->initializeIndex(vm, i, JSRopeString::createSubstringOfResolved(vm, input, start, subpatternResults[2 * i + 1] - start));
    80                 else
    81                     array->initializeIndex(vm, i, jsUndefined());
    82             }
     94        for (unsigned i = 1; i <= numSubpatterns; ++i) {
     95            int start = subpatternResults[2 * i];
     96            if (start >= 0)
     97                array->initializeIndex(vm, i, JSRopeString::createSubstringOfResolved(vm, input, start, subpatternResults[2 * i + 1] - start));
    8398        }
    8499    } else {
    85         array = tryCreateUninitializedRegExpMatchesArray(vm, globalObject->regExpMatchesArrayStructure(), regExp->numSubpatterns() + 1);
     100        array = tryCreateUninitializedRegExpMatchesArray(vm, globalObject->regExpMatchesArrayStructure(), numSubpatterns + 1);
    86101        RELEASE_ASSERT(array);
    87102       
     103        setProperties();
     104       
     105        array->initializeIndex(vm, 0, jsUndefined(), ArrayWithContiguous);
     106       
     107        for (unsigned i = 1; i <= numSubpatterns; ++i)
     108            array->initializeIndex(vm, i, jsUndefined(), ArrayWithContiguous);
     109       
     110        // Now the object is safe to scan by GC.
     111
    88112        array->initializeIndex(vm, 0, jsSubstringOfResolved(vm, input, result.start, result.end - result.start), ArrayWithContiguous);
    89113       
    90         if (unsigned numSubpatterns = regExp->numSubpatterns()) {
    91             for (unsigned i = 1; i <= numSubpatterns; ++i) {
    92                 int start = subpatternResults[2 * i];
    93                 if (start >= 0)
    94                     array->initializeIndex(vm, i, JSRopeString::createSubstringOfResolved(vm, input, start, subpatternResults[2 * i + 1] - start), ArrayWithContiguous);
    95                 else
    96                     array->initializeIndex(vm, i, jsUndefined(), ArrayWithContiguous);
    97             }
     114        for (unsigned i = 1; i <= numSubpatterns; ++i) {
     115            int start = subpatternResults[2 * i];
     116            if (start >= 0)
     117                array->initializeIndex(vm, i, JSRopeString::createSubstringOfResolved(vm, input, start, subpatternResults[2 * i + 1] - start), ArrayWithContiguous);
    98118        }
    99119    }
    100 
    101     array->putDirect(vm, RegExpMatchesArrayIndexPropertyOffset, jsNumber(result.start));
    102     array->putDirect(vm, RegExpMatchesArrayInputPropertyOffset, input);
    103120
    104121    return array;

  • trunk/Source/JavaScriptCore/runtime/RegExpPrototype.cpp

    r205198 r205462  
    504504    const ControlFunc& control, const PushFunc& push)
    505505{
     506    Vector<int> ovector;
     507       
    506508    while (matchPosition < inputSize) {
    507509        if (control() == AbortSplit)
    508510            return;
    509511       
    510         Vector<int, 32> ovector;
    511 
     512        ovector.resize(0);
     513       
    512514        // a. Perform ? Set(splitter, "lastIndex", q, true).
    513515        // b. Let z be ? RegExpExec(splitter, S).

  • trunk/Source/JavaScriptCore/runtime/RuntimeType.cpp

    r204912 r205462  
    2929#include "RuntimeType.h"
    3030
    31 #include "JSCJSValue.h"
    32 #include "JSCJSValueInlines.h"
     31#include "JSCInlines.h"
    3332
    3433namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/SamplingProfiler.cpp

    r205324 r205462  
    3434#include "HeapInlines.h"
    3535#include "HeapIterationScope.h"
     36#include "HeapUtil.h"
    3637#include "InlineCallFrame.h"
    3738#include "Interpreter.h"
    … …  
    358359
    359360    TinyBloomFilter filter = m_vm.heap.objectSpace().blocks().filter();
    360     MarkedBlockSet& markedBlockSet = m_vm.heap.objectSpace().blocks();
    361361
    362362    for (UnprocessedStackTrace& unprocessedStackTrace : m_unprocessedStackTraces) {
    … …  
    392392            StackFrame& stackFrame = stackTrace.frames.last();
    393393            bool alreadyHasExecutable = !!stackFrame.executable;
    394             if (!Heap::isValueGCObject(filter, markedBlockSet, callee)) {
     394            if (!HeapUtil::isValueGCObject(m_vm.heap, filter, callee)) {
    395395                if (!alreadyHasExecutable)
    396396                    stackFrame.frameType = FrameType::Unknown;
    … …  
    437437            }
    438438
    439             RELEASE_ASSERT(Heap::isPointerGCObject(filter, markedBlockSet, executable));
     439            RELEASE_ASSERT(HeapUtil::isPointerGCObjectJSCell(m_vm.heap, filter, executable));
    440440            stackFrame.frameType = FrameType::Executable;
    441441            stackFrame.executable = executable;

  • trunk/Source/JavaScriptCore/runtime/SetConstructor.cpp

    r205324 r205462  
    3030#include "GetterSetter.h"
    3131#include "IteratorOperations.h"
    32 #include "JSCJSValueInlines.h"
    33 #include "JSCellInlines.h"
     32#include "JSCInlines.h"
    3433#include "JSGlobalObject.h"
    3534#include "JSObjectInlines.h"
    … …  
    3736#include "MapData.h"
    3837#include "SetPrototype.h"
    39 #include "StructureInlines.h"
    4038
    4139namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/SetIteratorPrototype.cpp

    r205198 r205462  
    2828
    2929#include "IteratorOperations.h"
    30 #include "JSCJSValueInlines.h"
    31 #include "JSCellInlines.h"
     30#include "JSCInlines.h"
    3231#include "JSSetIterator.h"
    33 #include "StructureInlines.h"
    3432
    3533namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/SetPrototype.cpp

    r205198 r205462  
    3232#include "GetterSetter.h"
    3333#include "IteratorOperations.h"
    34 #include "JSCJSValueInlines.h"
    35 #include "JSFunctionInlines.h"
     34#include "JSCInlines.h"
    3635#include "JSSet.h"
    3736#include "JSSetIterator.h"
    3837#include "Lookup.h"
    3938#include "MapDataInlines.h"
    40 #include "StructureInlines.h"
    4139
    4240#include "SetPrototype.lut.h"

  • trunk/Source/JavaScriptCore/runtime/StringConstructor.cpp

    r205198 r205462  
    2929#include "JSCInlines.h"
    3030#include "StringPrototype.h"
     31#include <wtf/text/StringBuilder.h>
    3132
    3233namespace JSC {

  • trunk/Source/JavaScriptCore/runtime/StringIteratorPrototype.cpp

    r204912 r205462  
    2828#include "StringIteratorPrototype.h"
    2929
    30 #include "JSCJSValueInlines.h"
    31 #include "JSCellInlines.h"
     30#include "JSCInlines.h"
    3231#include "JSGlobalObject.h"
    3332#include "JSStringIterator.h"
    3433#include "ObjectConstructor.h"
    35 #include "StructureInlines.h"
    3634
    3735#include "StringIteratorPrototype.lut.h"

  • trunk/Source/JavaScriptCore/runtime/StructureInlines.h

    r202588 r205462  
    243243ALWAYS_INLINE WriteBarrier<PropertyTable>& Structure::propertyTable()
    244244{
    245     ASSERT(!globalObject() || (!globalObject()->vm().heap.isCollecting() || globalObject()->vm().heap.isHeapSnapshotting()));
    246245    return m_propertyTableUnsafe;
    247246}

  • trunk/Source/JavaScriptCore/runtime/TemplateRegistry.cpp

    r204912 r205462  
    2727#include "TemplateRegistry.h"
    2828
    29 #include "JSCJSValueInlines.h"
     29#include "JSCInlines.h"
    3030#include "JSGlobalObject.h"
    3131#include "ObjectConstructor.h"
    32 #include "StructureInlines.h"
    3332#include "WeakGCMapInlines.h"
    3433

  • trunk/Source/JavaScriptCore/runtime/TestRunnerUtils.cpp

    r204912 r205462  
    11/*
    2  * Copyright (C) 2013, 2014 Apple Inc. All rights reserved.
     2 * Copyright (C) 2013-2014, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2828
    2929#include "CodeBlock.h"
     30#include "HeapStatistics.h"
    3031#include "JSCInlines.h"
     32#include "LLIntData.h"
    3133
    3234namespace JSC {
    … …  
    151153}
    152154
     155// This is a hook called at the bitter end of some of our tests.
     156void finalizeStatsAtEndOfTesting()
     157{
     158    if (Options::logHeapStatisticsAtExit())
     159        HeapStatistics::reportSuccess();
     160    if (Options::reportLLIntStats())
     161        LLInt::Data::finalizeStats();
     162}
     163
    153164} // namespace JSC
    154165

  • trunk/Source/JavaScriptCore/runtime/TestRunnerUtils.h

    r204912 r205462  
    5454JS_EXPORT_PRIVATE unsigned numberOfOSRExitFuzzChecks();
    5555
     56JS_EXPORT_PRIVATE void finalizeStatsAtEndOfTesting();
     57
    5658} // namespace JSC
    5759

  • trunk/Source/JavaScriptCore/runtime/ThrowScope.cpp

    r205198 r205462  
    2727#include "ThrowScope.h"
    2828
    29 #include "JSCJSValueInlines.h"
     29#include "JSCInlines.h"
    3030#include "VM.h"
    3131

  • trunk/Source/JavaScriptCore/runtime/TypeProfilerLog.cpp

    r204912 r205462  
    3030#include "TypeProfilerLog.h"
    3131
    32 #include "JSCJSValueInlines.h"
     32#include "JSCInlines.h"
    3333#include "TypeLocation.h"
    3434#include <wtf/CurrentTime.h>

  • trunk/Source/JavaScriptCore/runtime/TypeSet.cpp

    r204912 r205462  
    2828
    2929#include "InspectorProtocolObjects.h"
    30 #include "JSCJSValue.h"
    31 #include "JSCJSValueInlines.h"
     30#include "JSCInlines.h"
    3231#include <wtf/text/CString.h>
    3332#include <wtf/text/WTFString.h>

  • trunk/Source/JavaScriptCore/runtime/VM.cpp

    r204994 r205462  
    7070#include "JSTemplateRegistryKey.h"
    7171#include "JSWithScope.h"
     72#include "LLIntData.h"
    7273#include "Lexer.h"
    7374#include "Lookup.h"
    … …  
    99100#include <wtf/CurrentTime.h>
    100101#include <wtf/ProcessID.h>
     102#include <wtf/SimpleStats.h>
    101103#include <wtf/StringPrintStream.h>
    102104#include <wtf/Threading.h>
    … …  
    107109#if !ENABLE(JIT)
    108110#include "CLoopStack.h"
     111#include "CLoopStackInlines.h"
    109112#endif
    110113
    … …  
    163166    , propertyNames(nullptr)
    164167    , emptyList(new MarkedArgumentBuffer)
     168    , machineCodeBytesPerBytecodeWordForBaselineJIT(std::make_unique<SimpleStats>())
    165169    , customGetterSetterFunctionMap(*this)
    166170    , stringCache(*this)
    … …  
    874878}
    875879
     880#if !ENABLE(JIT)
     881bool VM::ensureStackCapacityForCLoop(Register* newTopOfStack)
     882{
     883    return interpreter->cloopStack().ensureCapacityFor(newTopOfStack);
     884}
     885
     886bool VM::isSafeToRecurseSoftCLoop() const
     887{
     888    return interpreter->cloopStack().isSafeToRecurse();
     889}
     890#endif // !ENABLE(JIT)
     891
    876892} // namespace JSC

  • trunk/Source/JavaScriptCore/runtime/VM.h

    r205198 r205462  
    4040#include "JSCJSValue.h"
    4141#include "JSLock.h"
    42 #include "LLIntData.h"
    4342#include "MacroAssemblerCodeRef.h"
    4443#include "Microtask.h"
    … …  
    6160#include <wtf/HashMap.h>
    6261#include <wtf/HashSet.h>
    63 #include <wtf/SimpleStats.h>
    6462#include <wtf/StackBounds.h>
    6563#include <wtf/Stopwatch.h>
    … …  
    7270#include <wtf/ListHashSet.h>
    7371#endif
     72
     73namespace WTF {
     74class SimpleStats;
     75} // namespace WTF
     76using WTF::SimpleStats;
    7477
    7578namespace JSC {
    … …  
    343346    NumericStrings numericStrings;
    344347    DateInstanceCache dateInstanceCache;
    345     WTF::SimpleStats machineCodeBytesPerBytecodeWordForBaselineJIT;
     348    std::unique_ptr<SimpleStats> machineCodeBytesPerBytecodeWordForBaselineJIT;
    346349    WeakGCMap<std::pair<CustomGetterSetter*, int>, JSCustomGetterSetterFunction> customGetterSetterFunctionMap;
    347350    WeakGCMap<StringImpl*, JSString, PtrHash<StringImpl*>> stringCache;
    … …  
    642645        m_lastException = exception;
    643646    }
     647
     648#if !ENABLE(JIT)   
     649    bool ensureStackCapacityForCLoop(Register* newTopOfStack);
     650    bool isSafeToRecurseSoftCLoop() const;
     651#endif // !ENABLE(JIT)
    644652
    645653    JS_EXPORT_PRIVATE void throwException(ExecState*, Exception*);

  • trunk/Source/JavaScriptCore/runtime/VMEntryScope.h

    r205198 r205462  
    2727#define VMEntryScope_h
    2828
    29 #include "Interpreter.h"
    3029#include <wtf/StackBounds.h>
    3130#include <wtf/StackStats.h>

  • trunk/Source/JavaScriptCore/runtime/VMInlines.h

    r204912 r205462  
    3131#include "Watchdog.h"
    3232
    33 #if !ENABLE(JIT)
    34 #include "CLoopStackInlines.h"
    35 #endif
    36 
    3733namespace JSC {
    3834   
    … …  
    4339    return newTopOfStack >= m_softStackLimit;
    4440#else
    45     return interpreter->cloopStack().ensureCapacityFor(newTopOfStack);
     41    return ensureStackCapacityForCLoop(newTopOfStack);
    4642#endif
    4743   
    … …  
    5248    bool safe = isSafeToRecurse(m_softStackLimit);
    5349#if !ENABLE(JIT)
    54     safe = safe && interpreter->cloopStack().isSafeToRecurse();
     50    safe = safe && isSafeToRecurseSoftCLoop();
    5551#endif
    5652    return safe;

  • trunk/Source/JavaScriptCore/runtime/WeakMapConstructor.cpp

    r205324 r205462  
    11/*
    2  * Copyright (C) 2013 Apple, Inc. All rights reserved.
     2 * Copyright (C) 2013, 2016 Apple, Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2929#include "Error.h"
    3030#include "IteratorOperations.h"
    31 #include "JSCJSValueInlines.h"
    32 #include "JSCellInlines.h"
     31#include "JSCInlines.h"
    3332#include "JSGlobalObject.h"
    3433#include "JSObjectInlines.h"
    3534#include "JSWeakMap.h"
    36 #include "StructureInlines.h"
    3735#include "WeakMapPrototype.h"
    3836

  • trunk/Source/JavaScriptCore/runtime/WeakMapData.cpp

    r204912 r205462  
    3030#include "CopyVisitorInlines.h"
    3131#include "ExceptionHelpers.h"
    32 #include "JSCJSValueInlines.h"
    33 #include "SlotVisitorInlines.h"
     32#include "JSCInlines.h"
    3433
    3534#include <wtf/MathExtras.h>

  • trunk/Source/JavaScriptCore/runtime/WeakMapPrototype.cpp

    r205198 r205462  
    2727#include "WeakMapPrototype.h"
    2828
    29 #include "JSCJSValueInlines.h"
     29#include "JSCInlines.h"
    3030#include "JSWeakMap.h"
    31 #include "StructureInlines.h"
    3231#include "WeakMapData.h"
    3332

  • trunk/Source/JavaScriptCore/runtime/WeakSetConstructor.cpp

    r205324 r205462  
    11/*
    2  * Copyright (C) 2015 Apple, Inc. All rights reserved.
     2 * Copyright (C) 2015-2016 Apple, Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2929#include "Error.h"
    3030#include "IteratorOperations.h"
    31 #include "JSCJSValueInlines.h"
    32 #include "JSCellInlines.h"
     31#include "JSCInlines.h"
    3332#include "JSGlobalObject.h"
    3433#include "JSObjectInlines.h"
    3534#include "JSWeakSet.h"
    36 #include "StructureInlines.h"
    3735#include "WeakSetPrototype.h"
    3836

  • trunk/Source/JavaScriptCore/runtime/WeakSetPrototype.cpp

    r205198 r205462  
    2727#include "WeakSetPrototype.h"
    2828
    29 #include "JSCJSValueInlines.h"
     29#include "JSCInlines.h"
    3030#include "JSWeakSet.h"
    31 #include "StructureInlines.h"
    3231#include "WeakMapData.h"
    3332

  • trunk/Source/JavaScriptCore/testRegExp.cpp

    r204912 r205462  
    192192{
    193193    bool result = true;
    194     Vector<int, 32> outVector;
     194    Vector<int> outVector;
    195195    outVector.resize(regExpTest->expectVector.size());
    196196    int matchResult = regexp->match(vm, regExpTest->subject, regExpTest->offset, outVector);

  • trunk/Source/JavaScriptCore/tools/JSDollarVM.cpp

    r204912 r205462  
    2727#include "JSDollarVM.h"
    2828
    29 #include "JSCJSValueInlines.h"
    30 #include "StructureInlines.h"
     29#include "JSCInlines.h"
    3130
    3231namespace JSC {

  • trunk/Source/JavaScriptCore/tools/JSDollarVMPrototype.cpp

    r204912 r205462  
    147147{
    148148    MarkedBlock* candidate = MarkedBlock::blockFor(ptr);
    149     return heap->objectSpace().blocks().set().contains(candidate);
     149    if (heap->objectSpace().blocks().set().contains(candidate))
     150        return true;
     151    for (LargeAllocation* allocation : heap->objectSpace().largeAllocations()) {
     152        if (allocation->contains(ptr))
     153            return true;
     154    }
     155    return false;
    150156}
    151157

  • trunk/Source/WTF/ChangeLog

    r205362 r205462  
     12016-08-31  Filip Pizlo  <fpizlo@apple.com>
     2
     3        Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
     4        https://bugs.webkit.org/show_bug.cgi?id=160125
     5
     6        Reviewed by Geoffrey Garen and Keith Miller.
     7       
     8        I needed tryFastAlignedMalloc() so I added it.
     9
     10        * wtf/FastMalloc.cpp:
     11        (WTF::tryFastAlignedMalloc):
     12        * wtf/FastMalloc.h:
     13        * wtf/ParkingLot.cpp:
     14        (WTF::ParkingLot::forEachImpl):
     15        (WTF::ParkingLot::forEach): Deleted.
     16        * wtf/ParkingLot.h:
     17        (WTF::ParkingLot::parkConditionally):
     18        (WTF::ParkingLot::unparkOne):
     19        (WTF::ParkingLot::forEach):
     20        * wtf/ScopedLambda.h:
     21        (WTF::scopedLambdaRef):
     22        * wtf/SentinelLinkedList.h:
     23        (WTF::SentinelLinkedList::forEach):
     24        (WTF::RawNode>::takeFrom):
     25        * wtf/SimpleStats.h:
     26        (WTF::SimpleStats::operator bool):
     27        (WTF::SimpleStats::operator!): Deleted.
     28
    1292016-09-02  JF Bastien  <jfbastien@apple.com>
    230

  • trunk/Source/WTF/wtf/FastMalloc.cpp

    r204917 r205462  
    1 // Copyright (c) 2005, 2007, Google Inc. All rights reserved.
    2 
    31/*
    4  * Copyright (C) 2005-2009, 2011, 2015 Apple Inc. All rights reserved.
     2 * Copyright (c) 2005, 2007, Google Inc. All rights reserved.
     3 * Copyright (C) 2005-2009, 2011, 2015-2016 Apple Inc. All rights reserved.
    54 * Redistribution and use in source and binary forms, with or without
    65 * modification, are permitted provided that the following conditions
    … …  
    103102}
    104103
     104void* tryFastAlignedMalloc(size_t alignment, size_t size)
     105{
     106    return _aligned_malloc(size, alignment);
     107}
     108
    105109void fastAlignedFree(void* p)
    106110{
    … …  
    111115
    112116void* fastAlignedMalloc(size_t alignment, size_t size)
     117{
     118    void* p = nullptr;
     119    posix_memalign(&p, alignment, size);
     120    return p;
     121}
     122
     123void* tryFastAlignedMalloc(size_t alignment, size_t size)
    113124{
    114125    void* p = nullptr;
    … …  
    242253}
    243254
     255void* tryFastAlignedMalloc(size_t alignment, size_t size)
     256{
     257    return bmalloc::api::tryMemalign(alignment, size);
     258}
     259
    244260void fastAlignedFree(void* p)
    245261{

  • trunk/Source/WTF/wtf/FastMalloc.h

    r204917 r205462  
    11/*
    2  *  Copyright (C) 2005-2009, 2015 Apple Inc. All rights reserved.
     2 *  Copyright (C) 2005-2009, 2015-2016 Apple Inc. All rights reserved.
    33 *
    44 *  This library is free software; you can redistribute it and/or
    … …  
    5656// Allocations from fastAlignedMalloc() must be freed using fastAlignedFree().
    5757WTF_EXPORT_PRIVATE void* fastAlignedMalloc(size_t alignment, size_t);
     58WTF_EXPORT_PRIVATE void* tryFastAlignedMalloc(size_t alignment, size_t);
    5859WTF_EXPORT_PRIVATE void fastAlignedFree(void*);
    5960
    … …  
    111112using WTF::fastStrDup;
    112113using WTF::fastZeroedMalloc;
     114using WTF::tryFastAlignedMalloc;
    113115using WTF::tryFastCalloc;
    114116using WTF::tryFastMalloc;

  • trunk/Source/WTF/wtf/ParkingLot.cpp

    r204912 r205462  
    768768}
    769769
    770 NEVER_INLINE void ParkingLot::forEach(std::function<void(ThreadIdentifier, const void*)> callback)
     770NEVER_INLINE void ParkingLot::forEachImpl(const ScopedLambda<void(ThreadIdentifier, const void*)>& callback)
    771771{
    772772    Vector<Bucket*> bucketsToUnlock = lockHashtable();

  • trunk/Source/WTF/wtf/ParkingLot.h

    r204912 r205462  
    6767    static ParkResult parkConditionally(
    6868        const void* address,
    69         ValidationFunctor&& validation,
    70         BeforeSleepFunctor&& beforeSleep,
     69        const ValidationFunctor& validation,
     70        const BeforeSleepFunctor& beforeSleep,
    7171        Clock::time_point timeout)
    7272    {
    7373        return parkConditionallyImpl(
    7474            address,
    75             scopedLambda<bool()>(std::forward<ValidationFunctor>(validation)),
    76             scopedLambda<void()>(std::forward<BeforeSleepFunctor>(beforeSleep)),
     75            scopedLambdaRef<bool()>(validation),
     76            scopedLambdaRef<void()>(beforeSleep),
    7777            timeout);
    7878    }
    … …  
    125125    // WTF::Lock uses the timeToBeFair and token mechanism to implement eventual fairness.
    126126    template<typename Callback>
    127     static void unparkOne(const void* address, Callback&& callback)
     127    static void unparkOne(const void* address, const Callback& callback)
    128128    {
    129         unparkOneImpl(address, scopedLambda<intptr_t(UnparkResult)>(std::forward<Callback>(callback)));
     129        unparkOneImpl(address, scopedLambdaRef<intptr_t(UnparkResult)>(callback));
    130130    }
    131131
    … …  
    146146    // otherwise unconstrained. This method is useful primarily for debugging. It's also used by unit
    147147    // tests.
    148     WTF_EXPORT_PRIVATE static void forEach(std::function<void(ThreadIdentifier, const void*)>);
     148    template<typename Func>
     149    static void forEach(const Func& func)
     150    {
     151        forEachImpl(scopedLambdaRef<void(ThreadIdentifier, const void*)>(func));
     152    }
    149153
    150154private:
    … …  
    158162        const void* address, const ScopedLambda<intptr_t(UnparkResult)>& callback);
    159163
    160     WTF_EXPORT_PRIVATE static void forEachImpl(const std::function<void(ThreadIdentifier, const void*)>&);
     164    WTF_EXPORT_PRIVATE static void forEachImpl(const ScopedLambda<void(ThreadIdentifier, const void*)>&);
    161165};
    162166

  • trunk/Source/WTF/wtf/ScopedLambda.h

    r204912 r205462  
    127127}
    128128
     129template<typename FunctionType, typename Functor> class ScopedLambdaRefFunctor;
     130template<typename ResultType, typename... ArgumentTypes, typename Functor>
     131class ScopedLambdaRefFunctor<ResultType (ArgumentTypes...), Functor> : public ScopedLambda<ResultType (ArgumentTypes...)> {
     132public:
     133    ScopedLambdaRefFunctor(const Functor& functor)
     134        : ScopedLambda<ResultType (ArgumentTypes...)>(implFunction, this)
     135        , m_functor(&functor)
     136    {
     137    }
     138   
     139    // We need to make sure that copying and moving ScopedLambdaRefFunctor results in a
     140    // ScopedLambdaRefFunctor whose ScopedLambda supertype still points to this rather than
     141    // other.
     142    ScopedLambdaRefFunctor(const ScopedLambdaRefFunctor& other)
     143        : ScopedLambda<ResultType (ArgumentTypes...)>(implFunction, this)
     144        , m_functor(other.m_functor)
     145    {
     146    }
     147
     148    ScopedLambdaRefFunctor(ScopedLambdaRefFunctor&& other)
     149        : ScopedLambda<ResultType (ArgumentTypes...)>(implFunction, this)
     150        , m_functor(other.m_functor)
     151    {
     152    }
     153   
     154    ScopedLambdaRefFunctor& operator=(const ScopedLambdaRefFunctor& other)
     155    {
     156        m_functor = other.m_functor;
     157        return *this;
     158    }
     159   
     160    ScopedLambdaRefFunctor& operator=(ScopedLambdaRefFunctor&& other)
     161    {
     162        m_functor = other.m_functor;
     163        return *this;
     164    }
     165
     166private:
     167    static ResultType implFunction(void* argument, ArgumentTypes... arguments)
     168    {
     169        return (*static_cast<ScopedLambdaRefFunctor*>(argument)->m_functor)(arguments...);
     170    }
     171
     172    const Functor* m_functor;
     173};
     174
     175// This is for when you already refer to a functor by reference, and you know its lifetime is
     176// good. This just creates a ScopedLambda that points to your functor.
     177template<typename FunctionType, typename Functor>
     178ScopedLambdaRefFunctor<FunctionType, Functor> scopedLambdaRef(const Functor& functor)
     179{
     180    return ScopedLambdaRefFunctor<FunctionType, Functor>(functor);
     181}
     182
    129183} // namespace WTF
    130184
    131185using WTF::ScopedLambda;
    132186using WTF::scopedLambda;
     187using WTF::scopedLambdaRef;
    133188
    134189#endif // ScopedLambda_h

  • trunk/Source/WTF/wtf/SentinelLinkedList.h

    r195585 r205462  
    102102   
    103103    bool isEmpty() { return begin() == end(); }
    104 
     104   
     105    template<typename Func>
     106    void forEach(const Func& func)
     107    {
     108        for (iterator iter = begin(); iter != end();) {
     109            iterator next = iter->next();
     110            func(iter);
     111            iter = next;
     112        }
     113    }
     114   
     115    void takeFrom(SentinelLinkedList<T, RawNode>&);
     116   
    105117private:
    106118    RawNode m_headSentinel;
    … …  
    245257}
    246258
     259template <typename T, typename RawNode>
     260inline void SentinelLinkedList<T, RawNode>::takeFrom(SentinelLinkedList<T, RawNode>& other)
     261{
     262    if (other.isEmpty())
     263        return;
     264   
     265    m_tailSentinel.prev()->setNext(other.m_headSentinel.next());
     266    other.m_headSentinel.next()->setPrev(m_tailSentinel.prev());
     267   
     268    m_tailSentinel.setPrev(other.m_tailSentinel.prev());
     269    m_tailSentinel.prev()->setNext(&m_tailSentinel);
     270
     271    other.m_headSentinel.setNext(&other.m_tailSentinel);
     272    other.m_tailSentinel.setPrev(&other.m_headSentinel);
     273}
     274
    247275}
    248276

  • trunk/Source/WTF/wtf/SimpleStats.h

    r111778 r205462  
    11/*
    2  * Copyright (C) 2012 Apple Inc. All rights reserved.
     2 * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    5050    }
    5151   
    52     bool operator!() const
     52    explicit operator bool() const
    5353    {
    54         return !m_count;
     54        return !!m_count;
    5555    }
    5656   
    … …  
    111111} // namespace WTF
    112112
     113using WTF::SimpleStats;
     114
    113115#endif // SimpleStats_h
    114116

  • trunk/Source/WebCore/ChangeLog

    r205458 r205462  
     12016-08-31  Filip Pizlo  <fpizlo@apple.com>
     2
     3        Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
     4        https://bugs.webkit.org/show_bug.cgi?id=160125
     5
     6        Reviewed by Geoffrey Garen and Keith Miller.
     7
     8        No new tests because no new WebCore behavior.
     9       
     10        Just rewiring #includes.
     11
     12        * ForwardingHeaders/heap/HeapInlines.h: Added.
     13        * ForwardingHeaders/interpreter/Interpreter.h: Removed.
     14        * ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h: Added.
     15        * Modules/indexeddb/IDBCursorWithValue.cpp:
     16        * Modules/indexeddb/client/TransactionOperation.cpp:
     17        * Modules/indexeddb/server/SQLiteIDBBackingStore.cpp:
     18        * Modules/indexeddb/server/UniqueIDBDatabase.cpp:
     19        * bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp:
     20        * bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp:
     21        * bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp:
     22        * bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp:
     23        * bindings/js/JSClientRectCustom.cpp:
     24        * bindings/js/JSDOMBinding.cpp:
     25        * bindings/js/JSDOMBinding.h:
     26        * bindings/js/JSDeviceMotionEventCustom.cpp:
     27        * bindings/js/JSDeviceOrientationEventCustom.cpp:
     28        * bindings/js/JSErrorEventCustom.cpp:
     29        * bindings/js/JSIDBCursorWithValueCustom.cpp:
     30        * bindings/js/JSIDBIndexCustom.cpp:
     31        * bindings/js/JSPopStateEventCustom.cpp:
     32        * bindings/js/JSWebGL2RenderingContextCustom.cpp:
     33        * bindings/js/JSWorkerGlobalScopeCustom.cpp:
     34        * bindings/js/WorkerScriptController.cpp:
     35        * contentextensions/ContentExtensionParser.cpp:
     36        * dom/ErrorEvent.cpp:
     37        * html/HTMLCanvasElement.cpp:
     38        * html/MediaDocument.cpp:
     39        * inspector/CommandLineAPIModule.cpp:
     40        * loader/EmptyClients.cpp:
     41        * page/CaptionUserPreferences.cpp:
     42        * page/Frame.cpp:
     43        * page/PageGroup.cpp:
     44        * page/UserContentController.cpp:
     45        * platform/mock/mediasource/MockBox.cpp:
     46        * testing/GCObservation.cpp:
     47
    1482016-09-05  Fujii Hironori  <Hironori.Fujii@sony.com>
    249

  • trunk/Source/WebCore/Modules/indexeddb/IDBCursorWithValue.cpp

    r204912 r205462  
    2929#if ENABLE(INDEXED_DATABASE)
    3030
     31#include <heap/HeapInlines.h>
     32
    3133namespace WebCore {
    3234

  • trunk/Source/WebCore/Modules/indexeddb/client/TransactionOperation.cpp

    r204912 r205462  
    3030
    3131#include "IDBCursor.h"
     32#include <heap/HeapInlines.h>
    3233
    3334namespace WebCore {

  • trunk/Source/WebCore/Modules/indexeddb/server/SQLiteIDBBackingStore.cpp

    r204912 r205462  
    4646#include "SQLiteTransaction.h"
    4747#include "ThreadSafeDataBuffer.h"
     48#include <heap/HeapInlines.h>
    4849#include <heap/StrongInlines.h>
     50#include <runtime/AuxiliaryBarrierInlines.h>
    4951#include <runtime/JSCJSValueInlines.h>
    5052#include <runtime/JSGlobalObject.h>

  • trunk/Source/WebCore/Modules/indexeddb/server/UniqueIDBDatabase.cpp

    r204912 r205462  
    4040#include "SerializedScriptValue.h"
    4141#include "UniqueIDBDatabaseConnection.h"
     42#include <heap/HeapInlines.h>
     43#include <runtime/AuxiliaryBarrierInlines.h>
    4244#include <runtime/StructureInlines.h>
    4345#include <wtf/MainThread.h>

  • trunk/Source/WebCore/bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp

    r204912 r205462  
    2929#if ENABLE(APPLE_PAY)
    3030
     31#include <heap/HeapInlines.h>
    3132#include <runtime/JSCJSValueInlines.h>
    3233

  • trunk/Source/WebCore/bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp

    r204912 r205462  
    2929#if ENABLE(APPLE_PAY)
    3030
     31#include <heap/HeapInlines.h>
    3132#include <runtime/JSCJSValueInlines.h>
    3233

  • trunk/Source/WebCore/bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp

    r204912 r205462  
    2929#if ENABLE(APPLE_PAY)
    3030
     31#include <heap/HeapInlines.h>
    3132#include <runtime/JSCJSValueInlines.h>
    3233

  • trunk/Source/WebCore/bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp

    r204912 r205462  
    2929#if ENABLE(APPLE_PAY)
    3030
    31 #include <runtime/IdentifierInlines.h>
    32 #include <runtime/JSCJSValueInlines.h>
     31#include <runtime/JSCInlines.h>
    3332#include <runtime/ObjectConstructor.h>
    34 #include <runtime/StructureInlines.h>
     33#include <wtf/text/StringBuilder.h>
    3534
    3635using namespace JSC;

  • trunk/Source/WebCore/bindings/js/JSClientRectCustom.cpp

    r204912 r205462  
    2929#include "ClientRect.h"
    3030#include <bytecode/CodeBlock.h>
     31#include <heap/HeapInlines.h>
     32#include <runtime/AuxiliaryBarrierInlines.h>
    3133#include <runtime/IdentifierInlines.h>
    3234#include <runtime/JSObject.h>

  • trunk/Source/WebCore/bindings/js/JSDOMBinding.cpp

    r205198 r205462  
    3939#include <inspector/ScriptCallStack.h>
    4040#include <inspector/ScriptCallStackFactory.h>
    41 #include <interpreter/Interpreter.h>
    4241#include <runtime/DateInstance.h>
    4342#include <runtime/Error.h>
    … …  
    5049#include <wtf/MathExtras.h>
    5150#include <wtf/unicode/CharacterNames.h>
     51#include <wtf/text/StringBuilder.h>
    5252
    5353using namespace JSC;

  • trunk/Source/WebCore/bindings/js/JSDOMBinding.h

    r205422 r205462  
    3131#include "WebCoreTypedArrayController.h"
    3232#include <cstddef>
     33#include <heap/HeapInlines.h>
    3334#include <heap/SlotVisitorInlines.h>
    3435#include <heap/Weak.h>
    3536#include <heap/WeakInlines.h>
     37#include <runtime/AuxiliaryBarrierInlines.h>
    3638#include <runtime/Error.h>
    3739#include <runtime/IteratorOperations.h>

  • trunk/Source/WebCore/bindings/js/JSDeviceMotionEventCustom.cpp

    r204912 r205462  
    3232#include "DeviceMotionData.h"
    3333#include "DeviceMotionEvent.h"
     34#include <heap/HeapInlines.h>
     35#include <runtime/AuxiliaryBarrierInlines.h>
    3436#include <runtime/IdentifierInlines.h>
    3537#include <runtime/JSCJSValueInlines.h>

  • trunk/Source/WebCore/bindings/js/JSDeviceOrientationEventCustom.cpp

    r204912 r205462  
    3232#include "DeviceOrientationData.h"
    3333#include "DeviceOrientationEvent.h"
     34#include <heap/HeapInlines.h>
    3435#include <runtime/JSCJSValueInlines.h>
    3536#include <runtime/StructureInlines.h>

  • trunk/Source/WebCore/bindings/js/JSErrorEventCustom.cpp

    r204912 r205462  
    2828
    2929#include "ErrorEvent.h"
     30#include <heap/HeapInlines.h>
    3031
    3132using namespace JSC;

  • trunk/Source/WebCore/bindings/js/JSIDBCursorWithValueCustom.cpp

    r204912 r205462  
    3030
    3131#include "IDBCursorWithValue.h"
     32#include <heap/HeapInlines.h>
    3233
    3334using namespace JSC;

  • trunk/Source/WebCore/bindings/js/JSIDBIndexCustom.cpp

    r204912 r205462  
    3030
    3131#include "IDBIndex.h"
     32#include <heap/HeapInlines.h>
    3233
    3334using namespace JSC;

  • trunk/Source/WebCore/bindings/js/JSPerformanceTimingCustom.cpp

    r205286 r205462  
    3030
    3131#include "DOMWrapperWorld.h"
     32#include <heap/HeapInlines.h>
     33#include <runtime/AuxiliaryBarrierInlines.h>
    3234#include <runtime/IdentifierInlines.h>
    3335#include <runtime/JSObject.h>

  • trunk/Source/WebCore/bindings/js/JSPopStateEventCustom.cpp

    r204912 r205462  
    3434#include "DOMWrapperWorld.h"
    3535#include "JSHistory.h"
     36#include <heap/HeapInlines.h>
    3637#include <runtime/JSCJSValueInlines.h>
    3738

  • trunk/Source/WebCore/bindings/js/JSWebGL2RenderingContextCustom.cpp

    r205198 r205462  
    2929#include "JSWebGL2RenderingContext.h"
    3030
     31#include <heap/HeapInlines.h>
    3132#include <runtime/Error.h>
    3233#include "NotImplemented.h"

  • trunk/Source/WebCore/bindings/js/JSWorkerGlobalScopeCustom.cpp

    r205198 r205462  
    4242#include "WorkerLocation.h"
    4343#include "WorkerNavigator.h"
    44 #include <interpreter/Interpreter.h>
    4544
    4645#if ENABLE(WEB_SOCKETS)

  • trunk/Source/WebCore/bindings/js/WorkerScriptController.cpp

    r205198 r205462  
    4040#include <bindings/ScriptValue.h>
    4141#include <heap/StrongInlines.h>
    42 #include <interpreter/Interpreter.h>
    4342#include <runtime/Completion.h>
    4443#include <runtime/Error.h>

  • trunk/Source/WebCore/contentextensions/ContentExtensionParser.cpp

    r205324 r205462  
    3636#include "ContentExtensionsBackend.h"
    3737#include "ContentExtensionsDebugging.h"
    38 #include <JavaScriptCore/IdentifierInlines.h>
    39 #include <JavaScriptCore/JSCJSValueInlines.h>
     38#include <JavaScriptCore/JSCInlines.h>
    4039#include <JavaScriptCore/JSGlobalObject.h>
    4140#include <JavaScriptCore/JSONObject.h>
    42 #include <JavaScriptCore/JSObjectInlines.h>
    43 #include <JavaScriptCore/StructureInlines.h>
    4441#include <JavaScriptCore/VM.h>
    4542#include <wtf/CurrentTime.h>

  • trunk/Source/WebCore/dom/ErrorEvent.cpp

    r204912 r205462  
    3535#include "DOMWrapperWorld.h"
    3636#include "EventNames.h"
     37#include <heap/HeapInlines.h>
    3738
    3839using namespace JSC;

  • trunk/Source/WebCore/html/HTMLCanvasElement.cpp

    r205053 r205462  
    5252#include <runtime/JSLock.h>
    5353#include <wtf/RAMSize.h>
     54#include <wtf/text/StringBuilder.h>
    5455
    5556#if ENABLE(WEBGL)   

  • trunk/Source/WebCore/html/MediaDocument.cpp

    r205249 r205462  
    4949#include "ShadowRoot.h"
    5050#include "TypedElementDescendantIterator.h"
     51#include <wtf/text/StringBuilder.h>
    5152
    5253namespace WebCore {

  • trunk/Source/WebCore/inspector/CommandLineAPIModule.cpp

    r204912 r205462  
    3030#include "JSDOMGlobalObject.h"
    3131#include "WebInjectedScriptManager.h"
     32#include <heap/HeapInlines.h>
    3233#include <inspector/InjectedScript.h>
    3334

  • trunk/Source/WebCore/loader/EmptyClients.cpp

    r204912 r205462  
    4949#include "ThreadableWebSocketChannel.h"
    5050#include "UserContentProvider.h"
     51#include <heap/HeapInlines.h>
    5152#include <wtf/NeverDestroyed.h>
    5253

  • trunk/Source/WebCore/page/CaptionUserPreferences.cpp

    r204912 r205462  
    3939#include "UserStyleSheet.h"
    4040#include "UserStyleSheetTypes.h"
     41#include <heap/HeapInlines.h>
    4142#include <runtime/JSCellInlines.h>
    4243#include <runtime/StructureInlines.h>

  • trunk/Source/WebCore/page/Frame.cpp

    r204912 r205462  
    111111#include <wtf/RefCountedLeakCounter.h>
    112112#include <wtf/StdLibExtras.h>
     113#include <wtf/text/StringBuilder.h>
    113114#include <yarr/RegularExpression.h>
    114115

  • trunk/Source/WebCore/page/PageGroup.cpp

    r204912 r205462  
    3737#include "Settings.h"
    3838#include "StorageNamespace.h"
     39#include <heap/HeapInlines.h>
    3940#include <runtime/StructureInlines.h>
    4041#include <wtf/StdLibExtras.h>

  • trunk/Source/WebCore/page/UserContentController.cpp

    r204912 r205462  
    3030#include "UserScript.h"
    3131#include "UserStyleSheet.h"
     32#include <heap/HeapInlines.h>
    3233#include <runtime/JSCellInlines.h>
    3334#include <runtime/StructureInlines.h>

  • trunk/Source/WebCore/platform/mock/mediasource/MockBox.cpp

    r204912 r205462  
    2929#if ENABLE(MEDIA_SOURCE)
    3030
     31#include <JavaScriptCore/HeapInlines.h>
    3132#include <JavaScriptCore/JSCJSValueInlines.h>
    3233#include <JavaScriptCore/TypedArrayInlines.h>

  • trunk/Source/WebCore/testing/GCObservation.cpp

    r204912 r205462  
    2727#include "GCObservation.h"
    2828
     29#include <heap/HeapInlines.h>
     30
    2931namespace WebCore {
    3032

  • trunk/Source/WebKit2/ChangeLog

    r205460 r205462  
     12016-08-31  Filip Pizlo  <fpizlo@apple.com>
     2
     3        Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
     4        https://bugs.webkit.org/show_bug.cgi?id=160125
     5
     6        Reviewed by Geoffrey Garen and Keith Miller.
     7       
     8        Just rewiring some #includes.
     9
     10        * UIProcess/ViewGestureController.cpp:
     11        * UIProcess/WebPageProxy.cpp:
     12        * UIProcess/WebProcessPool.cpp:
     13        * UIProcess/WebProcessProxy.cpp:
     14        * WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp:
     15        * WebProcess/Plugins/Netscape/JSNPObject.cpp:
     16
    1172016-09-05  Michael Catanzaro  <mcatanzaro@igalia.com>
    218

  • trunk/Source/WebKit2/UIProcess/ViewGestureController.cpp

    r204912 r205462  
    3434#import <wtf/MathExtras.h>
    3535#import <wtf/NeverDestroyed.h>
     36#import <wtf/text/StringBuilder.h>
    3637
    3738using namespace WebCore;

  • trunk/Source/WebKit2/UIProcess/WebPageProxy.cpp

    r205412 r205462  
    126126#include <stdio.h>
    127127#include <wtf/NeverDestroyed.h>
     128#include <wtf/text/StringBuilder.h>
    128129#include <wtf/text/StringView.h>
    129130

  • trunk/Source/WebKit2/UIProcess/WebProcessPool.cpp

    r205275 r205462  
    7676#include <wtf/NeverDestroyed.h>
    7777#include <wtf/RunLoop.h>
     78#include <wtf/text/StringBuilder.h>
    7879
    7980#if ENABLE(BATTERY_STATUS)

  • trunk/Source/WebKit2/UIProcess/WebProcessProxy.cpp

    r205369 r205462  
    5858#include <wtf/RunLoop.h>
    5959#include <wtf/text/CString.h>
     60#include <wtf/text/StringBuilder.h>
    6061#include <wtf/text/WTFString.h>
    6162

  • trunk/Source/WebKit2/WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp

    r204912 r205462  
    3030#include "WebImage.h"
    3131#include <JavaScriptCore/APICast.h>
     32#include <JavaScriptCore/HeapInlines.h>
    3233#include <WebCore/Document.h>
    3334#include <WebCore/FloatRect.h>

  • trunk/Source/WebKit2/WebProcess/Plugins/Netscape/JSNPObject.cpp

    r205198 r205462  
    11/*
    2  * Copyright (C) 2010 Apple Inc. All rights reserved.
     2 * Copyright (C) 2010, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3333#include "NPRuntimeObjectMap.h"
    3434#include "NPRuntimeUtilities.h"
     35#include <JavaScriptCore/AuxiliaryBarrierInlines.h>
    3536#include <JavaScriptCore/Error.h>
    3637#include <JavaScriptCore/IdentifierInlines.h>

  • trunk/Source/bmalloc/ChangeLog

    r205215 r205462  
     12016-08-31  Filip Pizlo  <fpizlo@apple.com>
     2
     3        Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
     4        https://bugs.webkit.org/show_bug.cgi?id=160125
     5
     6        Reviewed by Geoffrey Garen and Keith Miller.
     7       
     8        I needed to tryMemalign, so I added such a thing.
     9
     10        * bmalloc/Allocator.cpp:
     11        (bmalloc::Allocator::allocate):
     12        (bmalloc::Allocator::tryAllocate):
     13        (bmalloc::Allocator::allocateImpl):
     14        * bmalloc/Allocator.h:
     15        * bmalloc/Cache.h:
     16        (bmalloc::Cache::tryAllocate):
     17        * bmalloc/bmalloc.h:
     18        (bmalloc::api::tryMemalign):
     19
    1202016-08-30  Yusuke Suzuki  <utatane.tea@gmail.com>
    221

  • trunk/Source/bmalloc/bmalloc/Allocator.cpp

    r204912 r205462  
    11/*
    2  * Copyright (C) 2014, 2015 Apple Inc. All rights reserved.
     2 * Copyright (C) 2014-2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    6565void* Allocator::allocate(size_t alignment, size_t size)
    6666{
     67    bool crashOnFailure = true;
     68    return allocateImpl(alignment, size, crashOnFailure);
     69}
     70
     71void* Allocator::tryAllocate(size_t alignment, size_t size)
     72{
     73    bool crashOnFailure = false;
     74    return allocateImpl(alignment, size, crashOnFailure);
     75}
     76
     77void* Allocator::allocateImpl(size_t alignment, size_t size, bool crashOnFailure)
     78{
    6779    BASSERT(isPowerOfTwo(alignment));
    6880
    … …  
    8193
    8294    std::lock_guard<StaticMutex> lock(PerProcess<Heap>::mutex());
    83     return PerProcess<Heap>::getFastCase()->allocateLarge(lock, alignment, size);
     95    Heap* heap = PerProcess<Heap>::getFastCase();
     96    if (crashOnFailure)
     97        return heap->allocateLarge(lock, alignment, size);
     98    return heap->tryAllocateLarge(lock, alignment, size);
    8499}
    85100

  • trunk/Source/bmalloc/bmalloc/Allocator.h

    r204912 r205462  
    11/*
    2  * Copyright (C) 2014 Apple Inc. All rights reserved.
     2 * Copyright (C) 2014, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    4444    void* tryAllocate(size_t);
    4545    void* allocate(size_t);
     46    void* tryAllocate(size_t alignment, size_t);
    4647    void* allocate(size_t alignment, size_t);
    4748    void* reallocate(void*, size_t);
    … …  
    5051
    5152private:
     53    void* allocateImpl(size_t alignment, size_t, bool crashOnFailure);
     54   
    5255    bool allocateFastCase(size_t, void*&);
    5356    void* allocateSlowCase(size_t);

  • trunk/Source/bmalloc/bmalloc/Cache.h

    r204912 r205462  
    11/*
    2  * Copyright (C) 2014 Apple Inc. All rights reserved.
     2 * Copyright (C) 2014, 2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    4242    static void* tryAllocate(size_t);
    4343    static void* allocate(size_t);
     44    static void* tryAllocate(size_t alignment, size_t);
    4445    static void* allocate(size_t alignment, size_t);
    4546    static void deallocate(void*);
    … …  
    8081}
    8182
     83inline void* Cache::tryAllocate(size_t alignment, size_t size)
     84{
     85    Cache* cache = PerThread<Cache>::getFastCase();
     86    if (!cache)
     87        return allocateSlowCaseNullCache(alignment, size);
     88    return cache->allocator().tryAllocate(alignment, size);
     89}
     90
    8291inline void* Cache::allocate(size_t alignment, size_t size)
    8392{

  • trunk/Source/bmalloc/bmalloc/bmalloc.h

    r204917 r205462  
    11/*
    2  * Copyright (C) 2014, 2015 Apple Inc. All rights reserved.
     2 * Copyright (C) 2014-2016 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    4444}
    4545
     46// Returns null on failure.
     47inline void* tryMemalign(size_t alignment, size_t size)
     48{
     49    return Cache::tryAllocate(alignment, size);
     50}
     51
    4652// Crashes on failure.
    4753inline void* memalign(size_t alignment, size_t size)

  • trunk/Tools/ChangeLog

    r205461 r205462  
     12016-08-31  Filip Pizlo  <fpizlo@apple.com>
     2
     3        Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
     4        https://bugs.webkit.org/show_bug.cgi?id=160125
     5
     6        Reviewed by Geoffrey Garen and Keith Miller.
     7
     8        * DumpRenderTree/TestRunner.cpp:
     9        * DumpRenderTree/mac/DumpRenderTree.mm:
     10        (DumpRenderTreeMain):
     11        * Scripts/run-jsc-stress-tests:
     12        * TestWebKitAPI/Tests/WTF/Vector.cpp:
     13        (TestWebKitAPI::TEST):
     14
    1152016-09-05  Michael Catanzaro  <mcatanzaro@igalia.com>
    216

  • trunk/Tools/DumpRenderTree/TestRunner.cpp

    r204918 r205462  
    3535#include "WorkQueueItem.h"
    3636#include <JavaScriptCore/APICast.h>
     37#include <JavaScriptCore/HeapInlines.h>
    3738#include <JavaScriptCore/JSContextRef.h>
    3839#include <JavaScriptCore/JSCTestRunnerUtils.h>

  • trunk/Tools/DumpRenderTree/mac/DumpRenderTree.mm

    r204945 r205462  
    5757#import "WorkQueueItem.h"
    5858#import <CoreFoundation/CoreFoundation.h>
    59 #import <JavaScriptCore/HeapStatistics.h>
    60 #import <JavaScriptCore/LLIntData.h>
    61 #import <JavaScriptCore/Options.h>
     59#import <JavaScriptCore/TestRunnerUtils.h>
    6260#import <WebCore/LogInitialization.h>
    6361#import <WebKit/DOMElement.h>
    … …  
    14301428    [WebCoreStatistics garbageCollectJavaScriptObjects];
    14311429    [WebCoreStatistics emptyCache]; // Otherwise SVGImages trigger false positives for Frame/Node counts
    1432     if (JSC::Options::logHeapStatisticsAtExit())
    1433         JSC::HeapStatistics::reportSuccess();
    1434     if (JSC::Options::reportLLIntStats())
    1435         JSC::LLInt::Data::finalizeStats();
     1430    JSC::finalizeStatsAtEndOfTesting();
    14361431    [pool release];
    14371432    returningFromMain = true;

  • trunk/Tools/Scripts/run-jsc-stress-tests

    r205387 r205462  
    422422BASE_OPTIONS = ["--useFTLJIT=false", "--useFunctionDotArguments=true", "--maxPerThreadStackUsage=1572864"]
    423423EAGER_OPTIONS = ["--thresholdForJITAfterWarmUp=10", "--thresholdForJITSoon=10", "--thresholdForOptimizeAfterWarmUp=20", "--thresholdForOptimizeAfterLongWarmUp=20", "--thresholdForOptimizeSoon=20", "--thresholdForFTLOptimizeAfterWarmUp=20", "--thresholdForFTLOptimizeSoon=20", "--maximumEvalCacheableSourceLength=150000", "--useEagerCodeBlockJettisonTiming=true"]
    424 NO_CJIT_OPTIONS = ["--useConcurrentJIT=false", "--thresholdForJITAfterWarmUp=100"]
     424NO_CJIT_OPTIONS = ["--useConcurrentJIT=false", "--thresholdForJITAfterWarmUp=100", "--scribbleFreeCells=true"]
    425425FTL_OPTIONS = ["--useFTLJIT=true"]
    426426
Note: See TracChangeset for help on using the changeset viewer.