Changeset 161615 in webkit

Timestamp:

Jan 9, 2014 6:28:27 PM (10 years ago)

Author:

mhahnenberg@apple.com

Message:

Marking should be generational
​https://bugs.webkit.org/show_bug.cgi?id=126552

Reviewed by Geoffrey Garen.

Source/JavaScriptCore:

Re-marking the same objects over and over is a waste of effort. This patch implements
the sticky mark bit algorithm (along with our already-present write barriers) to reduce
overhead during garbage collection caused by rescanning objects.

There are now two collection modes, EdenCollection and FullCollection. EdenCollections
only visit new objects or objects that were added to the remembered set by a write barrier.
FullCollections are normal collections that visit all objects regardless of their
generation.

In this patch EdenCollections do not do anything in CopiedSpace. This will be fixed in
​https://bugs.webkit.org/show_bug.cgi?id=126555.

• bytecode/CodeBlock.cpp:

(JSC::CodeBlock::visitAggregate):

• bytecode/CodeBlock.h:

(JSC::CodeBlockSet::mark):

• dfg/DFGOperations.cpp:
• heap/CodeBlockSet.cpp:

(JSC::CodeBlockSet::add):
(JSC::CodeBlockSet::traceMarked):
(JSC::CodeBlockSet::rememberCurrentlyExecutingCodeBlocks):

• heap/CodeBlockSet.h:
• heap/CopiedBlockInlines.h:

(JSC::CopiedBlock::reportLiveBytes):

• heap/CopiedSpace.cpp:

(JSC::CopiedSpace::didStartFullCollection):

• heap/CopiedSpace.h:

(JSC::CopiedSpace::heap):

• heap/Heap.cpp:

(JSC::Heap::Heap):
(JSC::Heap::didAbandon):
(JSC::Heap::markRoots):
(JSC::Heap::copyBackingStores):
(JSC::Heap::addToRememberedSet):
(JSC::Heap::collectAllGarbage):
(JSC::Heap::collect):
(JSC::Heap::didAllocate):
(JSC::Heap::writeBarrier):

• heap/Heap.h:

(JSC::Heap::isInRememberedSet):
(JSC::Heap::operationInProgress):
(JSC::Heap::shouldCollect):
(JSC::Heap::isCollecting):
(JSC::Heap::isWriteBarrierEnabled):
(JSC::Heap::writeBarrier):

• heap/HeapOperation.h:
• heap/MarkStack.cpp:

(JSC::MarkStackArray::~MarkStackArray):
(JSC::MarkStackArray::clear):
(JSC::MarkStackArray::fillVector):

• heap/MarkStack.h:
• heap/MarkedAllocator.cpp:

(JSC::isListPagedOut):
(JSC::MarkedAllocator::isPagedOut):
(JSC::MarkedAllocator::tryAllocateHelper):
(JSC::MarkedAllocator::addBlock):
(JSC::MarkedAllocator::removeBlock):
(JSC::MarkedAllocator::reset):

• heap/MarkedAllocator.h:

(JSC::MarkedAllocator::MarkedAllocator):

• heap/MarkedBlock.cpp:

(JSC::MarkedBlock::clearMarks):
(JSC::MarkedBlock::clearRememberedSet):
(JSC::MarkedBlock::clearMarksWithCollectionType):
(JSC::MarkedBlock::lastChanceToFinalize):

• heap/MarkedBlock.h: Changed atomSize to 16 bytes because we have no objects smaller

than 16 bytes. This is also to pay for the additional Bitmap for the remembered set.
(JSC::MarkedBlock::didConsumeEmptyFreeList):
(JSC::MarkedBlock::setRemembered):
(JSC::MarkedBlock::clearRemembered):
(JSC::MarkedBlock::atomicClearRemembered):
(JSC::MarkedBlock::isRemembered):

• heap/MarkedSpace.cpp:

(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::resetAllocators):
(JSC::MarkedSpace::visitWeakSets):
(JSC::MarkedSpace::reapWeakSets):
(JSC::VerifyMarked::operator()):
(JSC::MarkedSpace::clearMarks):

• heap/MarkedSpace.h:

(JSC::ClearMarks::operator()):
(JSC::ClearRememberedSet::operator()):
(JSC::MarkedSpace::didAllocateInBlock):
(JSC::MarkedSpace::clearRememberedSet):

• heap/SlotVisitor.cpp:

(JSC::SlotVisitor::~SlotVisitor):
(JSC::SlotVisitor::clearMarkStack):

• heap/SlotVisitor.h:

(JSC::SlotVisitor::markStack):
(JSC::SlotVisitor::sharedData):

• heap/SlotVisitorInlines.h:

(JSC::SlotVisitor::internalAppend):
(JSC::SlotVisitor::unconditionallyAppend):
(JSC::SlotVisitor::copyLater):
(JSC::SlotVisitor::reportExtraMemoryUsage):
(JSC::SlotVisitor::heap):

• jit/Repatch.cpp:
• runtime/JSGenericTypedArrayViewInlines.h:

(JSC::JSGenericTypedArrayView<Adaptor>::visitChildren):

• runtime/JSPropertyNameIterator.h:

(JSC::StructureRareData::setEnumerationCache):

• runtime/JSString.cpp:

(JSC::JSString::visitChildren):

• runtime/StructureRareDataInlines.h:

(JSC::StructureRareData::setPreviousID):
(JSC::StructureRareData::setObjectToStringValue):

• runtime/WeakMapData.cpp:

(JSC::WeakMapData::visitChildren):

Source/WTF:

• wtf/Bitmap.h:

(WTF::WordType>::count): Added a cast that became necessary when Bitmap
is used with smaller types than int32_t.

Location:

trunk/Source

Files:

• JavaScriptCore/ChangeLog (modified) (1 diff)
• JavaScriptCore/bytecode/CodeBlock.cpp (modified) (2 diffs)
• JavaScriptCore/bytecode/CodeBlock.h (modified) (1 diff)
• JavaScriptCore/dfg/DFGOperations.cpp (modified) (2 diffs)
• JavaScriptCore/heap/CodeBlockSet.cpp (modified) (2 diffs)
• JavaScriptCore/heap/CodeBlockSet.h (modified) (3 diffs)
• JavaScriptCore/heap/CopiedBlockInlines.h (modified) (1 diff)
• JavaScriptCore/heap/CopiedSpace.cpp (modified) (1 diff)
• JavaScriptCore/heap/CopiedSpace.h (modified) (2 diffs)
• JavaScriptCore/heap/Heap.cpp (modified) (20 diffs)
• JavaScriptCore/heap/Heap.h (modified) (9 diffs)
• JavaScriptCore/heap/HeapOperation.h (modified) (1 diff)
• JavaScriptCore/heap/MarkStack.cpp (modified) (2 diffs)
• JavaScriptCore/heap/MarkStack.h (modified) (2 diffs)
• JavaScriptCore/heap/MarkedAllocator.cpp (modified) (6 diffs)
• JavaScriptCore/heap/MarkedAllocator.h (modified) (3 diffs)
• JavaScriptCore/heap/MarkedBlock.cpp (modified) (1 diff)
• JavaScriptCore/heap/MarkedBlock.h (modified) (9 diffs)
• JavaScriptCore/heap/MarkedSpace.cpp (modified) (4 diffs)
• JavaScriptCore/heap/MarkedSpace.h (modified) (4 diffs)
• JavaScriptCore/heap/SlotVisitor.cpp (modified) (2 diffs)
• JavaScriptCore/heap/SlotVisitor.h (modified) (4 diffs)
• JavaScriptCore/heap/SlotVisitorInlines.h (modified) (4 diffs)
• JavaScriptCore/jit/Repatch.cpp (modified) (1 diff)
• JavaScriptCore/runtime/JSGenericTypedArrayViewInlines.h (modified) (1 diff)
• JavaScriptCore/runtime/JSPropertyNameIterator.h (modified) (1 diff)
• JavaScriptCore/runtime/JSString.cpp (modified) (1 diff)
• JavaScriptCore/runtime/StructureRareDataInlines.h (modified) (2 diffs)
• JavaScriptCore/runtime/WeakMapData.cpp (modified) (1 diff)
• WTF/ChangeLog (modified) (1 diff)
• WTF/wtf/Bitmap.h (modified) (1 diff)

trunk/Source/JavaScriptCore/ChangeLog

@@ +1 @@
+2014-01-07  Mark Hahnenberg  <mhahnenberg@apple.com>
+
+        Marking should be generational
+        https://bugs.webkit.org/show_bug.cgi?id=126552
+
+        Reviewed by Geoffrey Garen.
+
+        Re-marking the same objects over and over is a waste of effort. This patch implements 
+        the sticky mark bit algorithm (along with our already-present write barriers) to reduce 
+        overhead during garbage collection caused by rescanning objects.
+
+        There are now two collection modes, EdenCollection and FullCollection. EdenCollections
+        only visit new objects or objects that were added to the remembered set by a write barrier.
+        FullCollections are normal collections that visit all objects regardless of their 
+        generation.
+
+        In this patch EdenCollections do not do anything in CopiedSpace. This will be fixed in 
+        https://bugs.webkit.org/show_bug.cgi?id=126555.
+
+        * bytecode/CodeBlock.cpp:
+        (JSC::CodeBlock::visitAggregate):
+        * bytecode/CodeBlock.h:
+        (JSC::CodeBlockSet::mark):
+        * dfg/DFGOperations.cpp:
+        * heap/CodeBlockSet.cpp:
+        (JSC::CodeBlockSet::add):
+        (JSC::CodeBlockSet::traceMarked):
+        (JSC::CodeBlockSet::rememberCurrentlyExecutingCodeBlocks):
+        * heap/CodeBlockSet.h:
+        * heap/CopiedBlockInlines.h:
+        (JSC::CopiedBlock::reportLiveBytes):
+        * heap/CopiedSpace.cpp:
+        (JSC::CopiedSpace::didStartFullCollection):
+        * heap/CopiedSpace.h:
+        (JSC::CopiedSpace::heap):
+        * heap/Heap.cpp:
+        (JSC::Heap::Heap):
+        (JSC::Heap::didAbandon):
+        (JSC::Heap::markRoots):
+        (JSC::Heap::copyBackingStores):
+        (JSC::Heap::addToRememberedSet):
+        (JSC::Heap::collectAllGarbage):
+        (JSC::Heap::collect):
+        (JSC::Heap::didAllocate):
+        (JSC::Heap::writeBarrier):
+        * heap/Heap.h:
+        (JSC::Heap::isInRememberedSet):
+        (JSC::Heap::operationInProgress):
+        (JSC::Heap::shouldCollect):
+        (JSC::Heap::isCollecting):
+        (JSC::Heap::isWriteBarrierEnabled):
+        (JSC::Heap::writeBarrier):
+        * heap/HeapOperation.h:
+        * heap/MarkStack.cpp:
+        (JSC::MarkStackArray::~MarkStackArray):
+        (JSC::MarkStackArray::clear):
+        (JSC::MarkStackArray::fillVector):
+        * heap/MarkStack.h:
+        * heap/MarkedAllocator.cpp:
+        (JSC::isListPagedOut):
+        (JSC::MarkedAllocator::isPagedOut):
+        (JSC::MarkedAllocator::tryAllocateHelper):
+        (JSC::MarkedAllocator::addBlock):
+        (JSC::MarkedAllocator::removeBlock):
+        (JSC::MarkedAllocator::reset):
+        * heap/MarkedAllocator.h:
+        (JSC::MarkedAllocator::MarkedAllocator):
+        * heap/MarkedBlock.cpp:
+        (JSC::MarkedBlock::clearMarks):
+        (JSC::MarkedBlock::clearRememberedSet):
+        (JSC::MarkedBlock::clearMarksWithCollectionType):
+        (JSC::MarkedBlock::lastChanceToFinalize):
+        * heap/MarkedBlock.h: Changed atomSize to 16 bytes because we have no objects smaller
+        than 16 bytes. This is also to pay for the additional Bitmap for the remembered set.
+        (JSC::MarkedBlock::didConsumeEmptyFreeList):
+        (JSC::MarkedBlock::setRemembered):
+        (JSC::MarkedBlock::clearRemembered):
+        (JSC::MarkedBlock::atomicClearRemembered):
+        (JSC::MarkedBlock::isRemembered):
+        * heap/MarkedSpace.cpp:
+        (JSC::MarkedSpace::~MarkedSpace):
+        (JSC::MarkedSpace::resetAllocators):
+        (JSC::MarkedSpace::visitWeakSets):
+        (JSC::MarkedSpace::reapWeakSets):
+        (JSC::VerifyMarked::operator()):
+        (JSC::MarkedSpace::clearMarks):
+        * heap/MarkedSpace.h:
+        (JSC::ClearMarks::operator()):
+        (JSC::ClearRememberedSet::operator()):
+        (JSC::MarkedSpace::didAllocateInBlock):
+        (JSC::MarkedSpace::clearRememberedSet):
+        * heap/SlotVisitor.cpp:
+        (JSC::SlotVisitor::~SlotVisitor):
+        (JSC::SlotVisitor::clearMarkStack):
+        * heap/SlotVisitor.h:
+        (JSC::SlotVisitor::markStack):
+        (JSC::SlotVisitor::sharedData):
+        * heap/SlotVisitorInlines.h:
+        (JSC::SlotVisitor::internalAppend):
+        (JSC::SlotVisitor::unconditionallyAppend):
+        (JSC::SlotVisitor::copyLater):
+        (JSC::SlotVisitor::reportExtraMemoryUsage):
+        (JSC::SlotVisitor::heap):
+        * jit/Repatch.cpp:
+        * runtime/JSGenericTypedArrayViewInlines.h:
+        (JSC::JSGenericTypedArrayView<Adaptor>::visitChildren):
+        * runtime/JSPropertyNameIterator.h:
+        (JSC::StructureRareData::setEnumerationCache):
+        * runtime/JSString.cpp:
+        (JSC::JSString::visitChildren):
+        * runtime/StructureRareDataInlines.h:
+        (JSC::StructureRareData::setPreviousID):
+        (JSC::StructureRareData::setObjectToStringValue):
+        * runtime/WeakMapData.cpp:
+        (JSC::WeakMapData::visitChildren):
+
 2014-01-09  Joseph Pecoraro  <pecoraro@apple.com>
 

trunk/Source/JavaScriptCore/bytecode/CodeBlock.cpp

@@ -1955 +1955 @@
         otherBlock->visitAggregate(visitor);
 
-    visitor.reportExtraMemoryUsage(sizeof(CodeBlock));
+    visitor.reportExtraMemoryUsage(ownerExecutable(), sizeof(CodeBlock));
     if (m_jitCode)
-        visitor.reportExtraMemoryUsage(m_jitCode->size());
+        visitor.reportExtraMemoryUsage(ownerExecutable(), m_jitCode->size());
     if (m_instructions.size()) {
         // Divide by refCount() because m_instructions points to something that is shared
@@ -1963 +1963 @@
         // Having each CodeBlock report only its proportional share of the size is one way
         // of accomplishing this.
-        visitor.reportExtraMemoryUsage(m_instructions.size() * sizeof(Instruction) / m_instructions.refCount());
+        visitor.reportExtraMemoryUsage(ownerExecutable(), m_instructions.size() * sizeof(Instruction) / m_instructions.refCount());
     }
 

trunk/Source/JavaScriptCore/bytecode/CodeBlock.h

@@ -1270 +1270 @@
     
     (*iter)->m_mayBeExecuting = true;
+#if ENABLE(GGC)
+    m_currentlyExecuting.append(static_cast<CodeBlock*>(candidateCodeBlock));
+#endif
 }
 

trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp

@@ -851 +851 @@
 
     ASSERT(!object->structure()->outOfLineCapacity());
+    DeferGC deferGC(vm.heap);
     Butterfly* result = object->growOutOfLineStorage(vm, 0, initialOutOfLineCapacity);
     object->setButterflyWithoutChangingStructure(vm, result);
@@ -861 +862 @@
     NativeCallFrameTracer tracer(&vm, exec);
 
+    DeferGC deferGC(vm.heap);
     Butterfly* result = object->growOutOfLineStorage(vm, object->structure()->outOfLineCapacity(), newSize);
     object->setButterflyWithoutChangingStructure(vm, result);

trunk/Source/JavaScriptCore/heap/CodeBlockSet.cpp

@@ -46 +46 @@
 void CodeBlockSet::add(PassRefPtr<CodeBlock> codeBlock)
 {
-    bool isNewEntry = m_set.add(codeBlock.leakRef()).isNewEntry;
+    CodeBlock* block = codeBlock.leakRef();
+    bool isNewEntry = m_set.add(block).isNewEntry;
     ASSERT_UNUSED(isNewEntry, isNewEntry);
 }
@@ -102 +103 @@
         if (!codeBlock->m_mayBeExecuting)
             continue;
-        codeBlock->visitAggregate(visitor);
+        codeBlock->ownerExecutable()->methodTable()->visitChildren(codeBlock->ownerExecutable(), visitor);
     }
+}
+
+void CodeBlockSet::rememberCurrentlyExecutingCodeBlocks(Heap* heap)
+{
+#if ENABLE(GGC)
+    for (size_t i = 0; i < m_currentlyExecuting.size(); ++i)
+        heap->addToRememberedSet(m_currentlyExecuting[i]->ownerExecutable());
+    m_currentlyExecuting.clear();
+#else
+    UNUSED_PARAM(heap);
+#endif // ENABLE(GGC)
 }
 

trunk/Source/JavaScriptCore/heap/CodeBlockSet.h

@@ -31 +31 @@
 #include <wtf/PassRefPtr.h>
 #include <wtf/RefPtr.h>
+#include <wtf/Vector.h>
 
 namespace JSC {
 
 class CodeBlock;
+class Heap;
 class SlotVisitor;
 
@@ -66 +68 @@
     void traceMarked(SlotVisitor&);
 
+    // Add all currently executing CodeBlocks to the remembered set to be 
+    // re-scanned during the next collection.
+    void rememberCurrentlyExecutingCodeBlocks(Heap*);
+
 private:
     // This is not a set of RefPtr<CodeBlock> because we need to be able to find
@@ -71 +77 @@
     // and all, but that seemed like overkill.
     HashSet<CodeBlock* > m_set;
+    Vector<CodeBlock*> m_currentlyExecuting;
 };
 

trunk/Source/JavaScriptCore/heap/CopiedBlockInlines.h

@@ -43 +43 @@
     m_liveBytes += bytes;
 
+    if (isPinned())
+        return;
+
     if (!shouldEvacuate()) {
         pin();

trunk/Source/JavaScriptCore/heap/CopiedSpace.cpp

@@ -317 +317 @@
 }
 
+void CopiedSpace::didStartFullCollection()
+{
+    ASSERT(heap()->operationInProgress() == FullCollection);
+
+    ASSERT(m_fromSpace->isEmpty());
+
+    for (CopiedBlock* block = m_toSpace->head(); block; block = block->next())
+        block->didSurviveGC();
+
+    for (CopiedBlock* block = m_oversizeBlocks.head(); block; block = block->next())
+        block->didSurviveGC();
+}
+
 } // namespace JSC

trunk/Source/JavaScriptCore/heap/CopiedSpace.h

@@ -61 +61 @@
     CopiedAllocator& allocator() { return m_allocator; }
 
+    void didStartFullCollection();
+
     void startedCopying();
     void doneCopying();
@@ -80 +82 @@
 
     static CopiedBlock* blockFor(void*);
+
+    Heap* heap() const { return m_heap; }
 
 private:

trunk/Source/JavaScriptCore/heap/Heap.cpp

@@ -254 +254 @@
     , m_minBytesPerCycle(minHeapSize(m_heapType, m_ramSize))
     , m_sizeAfterLastCollect(0)
-    , m_bytesAllocatedLimit(m_minBytesPerCycle)
-    , m_bytesAllocated(0)
-    , m_bytesAbandoned(0)
+    , m_bytesAllocatedThisCycle(0)
+    , m_bytesAbandonedThisCycle(0)
+    , m_maxEdenSize(m_minBytesPerCycle)
+    , m_maxHeapSize(m_minBytesPerCycle)
+    , m_shouldDoFullCollection(false)
     , m_totalBytesVisited(0)
     , m_totalBytesCopied(0)
@@ -270 +272 @@
     , m_handleSet(vm)
     , m_isSafeToCollect(false)
-    , m_writeBarrierBuffer(128)
+    , m_writeBarrierBuffer(256)
     , m_vm(vm)
     , m_lastGCLength(0)
@@ -333 +335 @@
 {
     if (m_activityCallback)
-        m_activityCallback->didAllocate(m_bytesAllocated + m_bytesAbandoned);
-    m_bytesAbandoned += bytes;
+        m_activityCallback->didAllocate(m_bytesAllocatedThisCycle + m_bytesAbandonedThisCycle);
+    m_bytesAbandonedThisCycle += bytes;
 }
 
@@ -487 +489 @@
     visitor.setup();
     HeapRootVisitor heapRootVisitor(visitor);
+
+    Vector<const JSCell*> rememberedSet(m_slotVisitor.markStack().size());
+    m_slotVisitor.markStack().fillVector(rememberedSet);
 
     {
@@ -591 +596 @@
     }
 
+    {
+        GCPHASE(ClearRememberedSet);
+        for (unsigned i = 0; i < rememberedSet.size(); ++i) {
+            const JSCell* cell = rememberedSet[i];
+            MarkedBlock::blockFor(cell)->clearRemembered(cell);
+        }
+    }
+
     GCCOUNTER(VisitedValueCount, visitor.visitCount());
 
@@ -602 +615 @@
 #endif
 
-    m_totalBytesVisited = visitor.bytesVisited();
-    m_totalBytesCopied = visitor.bytesCopied();
+    if (m_operationInProgress == EdenCollection) {
+        m_totalBytesVisited += visitor.bytesVisited();
+        m_totalBytesCopied += visitor.bytesCopied();
+    } else {
+        ASSERT(m_operationInProgress == FullCollection);
+        m_totalBytesVisited = visitor.bytesVisited();
+        m_totalBytesCopied = visitor.bytesCopied();
+    }
 #if ENABLE(PARALLEL_GC)
     m_totalBytesVisited += m_sharedData.childBytesVisited();
@@ -616 +635 @@
 }
 
+template <HeapOperation collectionType>
 void Heap::copyBackingStores()
 {
+    if (collectionType == EdenCollection)
+        return;
+
     m_storageSpace.startedCopying();
     if (m_storageSpace.shouldDoCopyPhase()) {
@@ -628 +651 @@
         m_storageSpace.doneCopying();
         m_sharedData.didFinishCopying();
-    } else 
+    } else
         m_storageSpace.doneCopying();
 }
@@ -724 +747 @@
 }
 
+void Heap::addToRememberedSet(const JSCell* cell)
+{
+    ASSERT(cell);
+    ASSERT(!Options::enableConcurrentJIT() || !isCompilationThread());
+    if (isInRememberedSet(cell))
+        return;
+    MarkedBlock::blockFor(cell)->setRemembered(cell);
+    m_slotVisitor.unconditionallyAppend(const_cast<JSCell*>(cell));
+}
+
 void Heap::collectAllGarbage()
 {
@@ -729 +762 @@
         return;
 
+    m_shouldDoFullCollection = true;
     collect();
 
@@ -765 +799 @@
         m_vm->prepareToDiscardCode();
     }
-    
-    m_operationInProgress = Collection;
-    m_extraMemoryUsage = 0;
+
+    bool isFullCollection = m_shouldDoFullCollection;
+    if (isFullCollection) {
+        m_operationInProgress = FullCollection;
+        m_slotVisitor.clearMarkStack();
+        m_shouldDoFullCollection = false;
+        if (Options::logGC())
+            dataLog("FullCollection, ");
+    } else {
+#if ENABLE(GGC)
+        m_operationInProgress = EdenCollection;
+        if (Options::logGC())
+            dataLog("EdenCollection, ");
+#else
+        m_operationInProgress = FullCollection;
+        m_slotVisitor.clearMarkStack();
+        if (Options::logGC())
+            dataLog("FullCollection, ");
+#endif
+    }
+    if (m_operationInProgress == FullCollection)
+        m_extraMemoryUsage = 0;
 
     if (m_activityCallback)
@@ -781 +834 @@
         GCPHASE(StopAllocation);
         m_objectSpace.stopAllocating();
+        if (m_operationInProgress == FullCollection)
+            m_storageSpace.didStartFullCollection();
+    }
+
+    {
+        GCPHASE(FlushWriteBarrierBuffer);
+        if (m_operationInProgress == EdenCollection)
+            m_writeBarrierBuffer.flush(*this);
+        else
+            m_writeBarrierBuffer.reset();
     }
 
@@ -797 +860 @@
     }
 
-    {
+    if (m_operationInProgress == FullCollection) {
         m_blockSnapshot.resize(m_objectSpace.blocks().set().size());
         MarkedBlockSnapshotFunctor functor(m_blockSnapshot);
@@ -803 +866 @@
     }
 
-    copyBackingStores();
+    if (m_operationInProgress == FullCollection)
+        copyBackingStores<FullCollection>();
+    else
+        copyBackingStores<EdenCollection>();
 
     {
@@ -820 +886 @@
     }
 
-    m_sweeper->startSweeping(m_blockSnapshot);
-    m_bytesAbandoned = 0;
+    if (m_operationInProgress == FullCollection)
+        m_sweeper->startSweeping(m_blockSnapshot);
+
+    {
+        GCPHASE(AddCurrentlyExecutingCodeBlocksToRememberedSet);
+        m_codeBlocks.rememberCurrentlyExecutingCodeBlocks(this);
+    }
+
+    m_bytesAbandonedThisCycle = 0;
 
     {
@@ -832 +905 @@
         HeapStatistics::exitWithFailure();
 
+    if (m_operationInProgress == FullCollection) {
+        // To avoid pathological GC churn in very small and very large heaps, we set
+        // the new allocation limit based on the current size of the heap, with a
+        // fixed minimum.
+        m_maxHeapSize = max(minHeapSize(m_heapType, m_ramSize), proportionalHeapSize(currentHeapSize, m_ramSize));
+        m_maxEdenSize = m_maxHeapSize - currentHeapSize;
+    } else {
+        ASSERT(currentHeapSize >= m_sizeAfterLastCollect);
+        m_maxEdenSize = m_maxHeapSize - currentHeapSize;
+        double edenToOldGenerationRatio = (double)m_maxEdenSize / (double)m_maxHeapSize;
+        double minEdenToOldGenerationRatio = 1.0 / 3.0;
+        if (edenToOldGenerationRatio < minEdenToOldGenerationRatio)
+            m_shouldDoFullCollection = true;
+        m_maxHeapSize += currentHeapSize - m_sizeAfterLastCollect;
+        m_maxEdenSize = m_maxHeapSize - currentHeapSize;
+    }
+
     m_sizeAfterLastCollect = currentHeapSize;
 
-    // To avoid pathological GC churn in very small and very large heaps, we set
-    // the new allocation limit based on the current size of the heap, with a
-    // fixed minimum.
-    size_t maxHeapSize = max(minHeapSize(m_heapType, m_ramSize), proportionalHeapSize(currentHeapSize, m_ramSize));
-    m_bytesAllocatedLimit = maxHeapSize - currentHeapSize;
-
-    m_bytesAllocated = 0;
+    m_bytesAllocatedThisCycle = 0;
     double lastGCEndTime = WTF::monotonicallyIncreasingTime();
     m_lastGCLength = lastGCEndTime - lastGCStartTime;
@@ -846 +930 @@
     if (Options::recordGCPauseTimes())
         HeapStatistics::recordGCPauseTime(lastGCStartTime, lastGCEndTime);
-    RELEASE_ASSERT(m_operationInProgress == Collection);
+    RELEASE_ASSERT(m_operationInProgress == EdenCollection || m_operationInProgress == FullCollection);
 
     m_operationInProgress = NoOperation;
@@ -864 +948 @@
         dataLog(after - before, " ms, ", currentHeapSize / 1024, " kb]\n");
     }
-
-#if ENABLE(ALLOCATION_LOGGING)
-    dataLogF("JSC GC finishing collection.\n");
-#endif
 }
 
@@ -917 +997 @@
 {
     if (m_activityCallback)
-        m_activityCallback->didAllocate(m_bytesAllocated + m_bytesAbandoned);
-    m_bytesAllocated += bytes;
+        m_activityCallback->didAllocate(m_bytesAllocatedThisCycle + m_bytesAbandonedThisCycle);
+    m_bytesAllocatedThisCycle += bytes;
 }
 
@@ -993 +1073 @@
     decrementDeferralDepth();
     collectIfNecessaryOrDefer();
+}
+
+void Heap::writeBarrier(const JSCell* from)
+{
+    ASSERT_GC_OBJECT_LOOKS_VALID(const_cast<JSCell*>(from));
+    if (!from || !isMarked(from))
+        return;
+    Heap* heap = Heap::heap(from);
+    heap->addToRememberedSet(from);
 }
 

trunk/Source/JavaScriptCore/heap/Heap.h

@@ -95 +95 @@
         static void setMarked(const void*);
 
+        JS_EXPORT_PRIVATE void addToRememberedSet(const JSCell*);
+        bool isInRememberedSet(const JSCell* cell) const
+        {
+            ASSERT(cell);
+            ASSERT(!Options::enableConcurrentJIT() || !isCompilationThread());
+            return MarkedBlock::blockFor(cell)->isRemembered(cell);
+        }
         static bool isWriteBarrierEnabled();
-        static void writeBarrier(const JSCell*);
+        JS_EXPORT_PRIVATE static void writeBarrier(const JSCell*);
         static void writeBarrier(const JSCell*, JSValue);
         static void writeBarrier(const JSCell*, JSCell*);
-        static uint8_t* addressOfCardFor(JSCell*);
 
         WriteBarrierBuffer& writeBarrierBuffer() { return m_writeBarrierBuffer; }
@@ -121 +127 @@
         // true if collection is in progress
         inline bool isCollecting();
+        inline HeapOperation operationInProgress() { return m_operationInProgress; }
         // true if an allocation or collection is in progress
         inline bool isBusy();
@@ -237 +244 @@
         void markProtectedObjects(HeapRootVisitor&);
         void markTempSortVectors(HeapRootVisitor&);
+        template <HeapOperation collectionType>
         void copyBackingStores();
         void harvestWeakReferences();
@@ -258 +266 @@
         size_t m_sizeAfterLastCollect;
 
-        size_t m_bytesAllocatedLimit;
-        size_t m_bytesAllocated;
-        size_t m_bytesAbandoned;
-
+        size_t m_bytesAllocatedThisCycle;
+        size_t m_bytesAbandonedThisCycle;
+        size_t m_maxEdenSize;
+        size_t m_maxHeapSize;
+        bool m_shouldDoFullCollection;
         size_t m_totalBytesVisited;
         size_t m_totalBytesCopied;
@@ -271 +280 @@
         GCIncomingRefCountedSet<ArrayBuffer> m_arrayBuffers;
         size_t m_extraMemoryUsage;
+
+        HashSet<const JSCell*> m_copyingRememberedSet;
 
         ProtectCountSet m_protectedValues;
@@ -323 +334 @@
             return false;
         if (Options::gcMaxHeapSize())
-            return m_bytesAllocated > Options::gcMaxHeapSize() && m_isSafeToCollect && m_operationInProgress == NoOperation;
-        return m_bytesAllocated > m_bytesAllocatedLimit && m_isSafeToCollect && m_operationInProgress == NoOperation;
+            return m_bytesAllocatedThisCycle > Options::gcMaxHeapSize() && m_isSafeToCollect && m_operationInProgress == NoOperation;
+        return m_bytesAllocatedThisCycle > m_maxEdenSize && m_isSafeToCollect && m_operationInProgress == NoOperation;
     }
 
@@ -334 +345 @@
     bool Heap::isCollecting()
     {
-        return m_operationInProgress == Collection;
+        return m_operationInProgress == FullCollection || m_operationInProgress == EdenCollection;
     }
 
@@ -371 +382 @@
     inline bool Heap::isWriteBarrierEnabled()
     {
-#if ENABLE(WRITE_BARRIER_PROFILING)
+#if ENABLE(WRITE_BARRIER_PROFILING) || ENABLE(GGC)
         return true;
 #else
@@ -378 +389 @@
     }
 
-    inline void Heap::writeBarrier(const JSCell*)
-    {
+    inline void Heap::writeBarrier(const JSCell* from, JSCell* to)
+    {
+#if ENABLE(WRITE_BARRIER_PROFILING)
         WriteBarrierCounters::countWriteBarrier();
-    }
-
-    inline void Heap::writeBarrier(const JSCell*, JSCell*)
-    {
+#endif
+        if (!from || !isMarked(from))
+            return;
+        if (!to || isMarked(to))
+            return;
+        Heap::heap(from)->addToRememberedSet(from);
+    }
+
+    inline void Heap::writeBarrier(const JSCell* from, JSValue to)
+    {
+#if ENABLE(WRITE_BARRIER_PROFILING)
         WriteBarrierCounters::countWriteBarrier();
-    }
-
-    inline void Heap::writeBarrier(const JSCell*, JSValue)
-    {
-        WriteBarrierCounters::countWriteBarrier();
+#endif
+        if (!to.isCell())
+            return;
+        writeBarrier(from, to.asCell());
     }
 

trunk/Source/JavaScriptCore/heap/HeapOperation.h

@@ -29 +29 @@
 namespace JSC {
 
-enum HeapOperation { NoOperation, Allocation, Collection };
+enum HeapOperation { NoOperation, Allocation, FullCollection, EdenCollection };
 
 } // namespace JSC

trunk/Source/JavaScriptCore/heap/MarkStack.cpp

@@ -58 +58 @@
 MarkStackArray::~MarkStackArray()
 {
-    ASSERT(m_numberOfSegments == 1 && m_segments.size() == 1);
+    ASSERT(m_numberOfSegments == 1);
+    ASSERT(m_segments.size() == 1);
     m_blockAllocator.deallocate(MarkStackSegment::destroy(m_segments.removeHead()));
+    m_numberOfSegments--;
+    ASSERT(!m_numberOfSegments);
+    ASSERT(!m_segments.size());
+}
+
+void MarkStackArray::clear()
+{
+    if (!m_segments.head())
+        return;
+    MarkStackSegment* next;
+    for (MarkStackSegment* current = m_segments.head(); current->next(); current = next) {
+        next = current->next();
+        m_segments.remove(current);
+        m_blockAllocator.deallocate(MarkStackSegment::destroy(current));
+    }
+    m_top = 0;
+    m_numberOfSegments = 1;
+#if !ASSERT_DISABLED
+    m_segments.head()->m_top = 0;
+#endif
 }
 
@@ -168 +189 @@
 }
 
+void MarkStackArray::fillVector(Vector<const JSCell*>& vector)
+{
+    ASSERT(vector.size() == size());
+
+    MarkStackSegment* currentSegment = m_segments.head();
+    if (!currentSegment)
+        return;
+
+    unsigned count = 0;
+    for (unsigned i = 0; i < m_top; ++i) {
+        ASSERT(currentSegment->data()[i]);
+        vector[count++] = currentSegment->data()[i];
+    }
+
+    currentSegment = currentSegment->next();
+    while (currentSegment) {
+        for (unsigned i = 0; i < s_segmentCapacity; ++i) {
+            ASSERT(currentSegment->data()[i]);
+            vector[count++] = currentSegment->data()[i];
+        }
+        currentSegment = currentSegment->next();
+    }
+}
+
 } // namespace JSC

trunk/Source/JavaScriptCore/heap/MarkStack.h

@@ -53 +53 @@
 #include "HeapBlock.h"
 #include <wtf/StdLibExtras.h>
+#include <wtf/Vector.h>
 
 namespace JSC {
@@ -101 +102 @@
     bool isEmpty();
 
+    void fillVector(Vector<const JSCell*>&);
+    void clear();
+
 private:
     template <size_t size> struct CapacityFromSize {

trunk/Source/JavaScriptCore/heap/MarkedAllocator.cpp

@@ -11 +11 @@
 namespace JSC {
 
-bool MarkedAllocator::isPagedOut(double deadline)
+static bool isListPagedOut(double deadline, DoublyLinkedList<MarkedBlock>& list)
 {
     unsigned itersSinceLastTimeCheck = 0;
-    MarkedBlock* block = m_blockList.head();
+    MarkedBlock* block = list.head();
     while (block) {
         block = block->next();
@@ -25 +25 @@
         }
     }
+    return false;
+}
 
+bool MarkedAllocator::isPagedOut(double deadline)
+{
+    if (isListPagedOut(deadline, m_blockList))
+        return true;
     return false;
 }
@@ -37 +43 @@
         DelayedReleaseScope delayedReleaseScope(*m_markedSpace);
         if (m_currentBlock) {
-            ASSERT(m_currentBlock == m_blocksToSweep);
+            ASSERT(m_currentBlock == m_nextBlockToSweep);
             m_currentBlock->didConsumeFreeList();
-            m_blocksToSweep = m_currentBlock->next();
+            m_nextBlockToSweep = m_currentBlock->next();
         }
 
-        for (MarkedBlock*& block = m_blocksToSweep; block; block = block->next()) {
+        MarkedBlock* next;
+        for (MarkedBlock*& block = m_nextBlockToSweep; block; block = next) {
+            next = block->next();
+
             MarkedBlock::FreeList freeList = block->sweep(MarkedBlock::SweepToFreeList);
+            
             if (!freeList.head) {
                 block->didConsumeEmptyFreeList();
+                m_blockList.remove(block);
+                m_blockList.push(block);
+                if (!m_lastFullBlock)
+                    m_lastFullBlock = block;
                 continue;
             }
@@ -69 +83 @@
     m_freeList.head = head->next;
     ASSERT(head);
+    m_markedSpace->didAllocateInBlock(m_currentBlock);
     return head;
 }
@@ -137 +152 @@
     
     m_blockList.append(block);
-    m_blocksToSweep = m_currentBlock = block;
+    m_nextBlockToSweep = m_currentBlock = block;
     m_freeList = block->sweep(MarkedBlock::SweepToFreeList);
     m_markedSpace->didAddBlock(block);
@@ -148 +163 @@
         m_freeList = MarkedBlock::FreeList();
     }
-    if (m_blocksToSweep == block)
-        m_blocksToSweep = m_blocksToSweep->next();
+    if (m_nextBlockToSweep == block)
+        m_nextBlockToSweep = m_nextBlockToSweep->next();
+
+    if (block == m_lastFullBlock)
+        m_lastFullBlock = m_lastFullBlock->prev();
+    
     m_blockList.remove(block);
 }
 
+void MarkedAllocator::reset()
+{
+    m_lastActiveBlock = 0;
+    m_currentBlock = 0;
+    m_freeList = MarkedBlock::FreeList();
+    if (m_heap->operationInProgress() == FullCollection)
+        m_lastFullBlock = 0;
+
+    if (m_lastFullBlock)
+        m_nextBlockToSweep = m_lastFullBlock->next() ? m_lastFullBlock->next() : m_lastFullBlock;
+    else
+        m_nextBlockToSweep = m_blockList.head();
+}
+
 } // namespace JSC

trunk/Source/JavaScriptCore/heap/MarkedAllocator.h

@@ -53 +53 @@
     MarkedBlock* m_currentBlock;
     MarkedBlock* m_lastActiveBlock;
-    MarkedBlock* m_blocksToSweep;
+    MarkedBlock* m_nextBlockToSweep;
+    MarkedBlock* m_lastFullBlock;
     DoublyLinkedList<MarkedBlock> m_blockList;
     size_t m_cellSize;
@@ -69 +70 @@
     : m_currentBlock(0)
     , m_lastActiveBlock(0)
-    , m_blocksToSweep(0)
+    , m_nextBlockToSweep(0)
+    , m_lastFullBlock(0)
     , m_cellSize(0)
     , m_destructorType(MarkedBlock::None)
@@ -101 +103 @@
 #endif
     return head;
-}
-
-inline void MarkedAllocator::reset()
-{
-    m_lastActiveBlock = 0;
-    m_currentBlock = 0;
-    m_freeList = MarkedBlock::FreeList();
-    m_blocksToSweep = m_blockList.head();
 }
 

trunk/Source/JavaScriptCore/heap/MarkedBlock.cpp

@@ -198 +198 @@
 }
 
+void MarkedBlock::clearMarks()
+{
+    if (heap()->operationInProgress() == JSC::EdenCollection)
+        this->clearMarksWithCollectionType<EdenCollection>();
+    else
+        this->clearMarksWithCollectionType<FullCollection>();
+}
+
+void MarkedBlock::clearRememberedSet()
+{
+    m_rememberedSet.clearAll();
+}
+
+template <HeapOperation collectionType>
+void MarkedBlock::clearMarksWithCollectionType()
+{
+    ASSERT(collectionType == FullCollection || collectionType == EdenCollection);
+    HEAP_LOG_BLOCK_STATE_TRANSITION(this);
+
+    ASSERT(m_state != New && m_state != FreeListed);
+    if (collectionType == FullCollection) {
+        m_marks.clearAll();
+        m_rememberedSet.clearAll();
+    }
+
+    // This will become true at the end of the mark phase. We set it now to
+    // avoid an extra pass to do so later.
+    m_state = Marked;
+}
+
+void MarkedBlock::lastChanceToFinalize()
+{
+    m_weakSet.lastChanceToFinalize();
+
+    clearNewlyAllocated();
+    clearMarksWithCollectionType<FullCollection>();
+    sweep();
+}
+
 MarkedBlock::FreeList MarkedBlock::resumeAllocating()
 {

trunk/Source/JavaScriptCore/heap/MarkedBlock.h

@@ -26 +26 @@
 #include "HeapBlock.h"
 
+#include "HeapOperation.h"
 #include "WeakSet.h"
 #include <wtf/Bitmap.h>
@@ -73 +74 @@
 
     public:
-        static const size_t atomSize = 8; // bytes
+        static const size_t atomSize = 16; // bytes
         static const size_t atomShiftAmount = 4; // log_2(atomSize) FIXME: Change atomSize to 16.
         static const size_t blockSize = 64 * KB;
@@ -141 +142 @@
         FreeList resumeAllocating(); // Call this if you canonicalized a block for some non-collection related purpose.
         void didConsumeEmptyFreeList(); // Call this if you sweep a block, but the returned FreeList is empty.
+        void didSweepToNoAvail(); // Call this if you sweep a block and get an empty free list back.
 
         // Returns true if the "newly allocated" bitmap was non-null 
@@ -146 +148 @@
         bool clearNewlyAllocated();
         void clearMarks();
+        void clearRememberedSet();
+        template <HeapOperation collectionType>
+        void clearMarksWithCollectionType();
+
         size_t markCount();
         bool isEmpty();
@@ -161 +167 @@
         void setMarked(const void*);
         void clearMarked(const void*);
+
+        void setRemembered(const void*);
+        void clearRemembered(const void*);
+        void atomicClearRemembered(const void*);
+        bool isRemembered(const void*);
 
         bool isNewlyAllocated(const void*);
@@ -191 +202 @@
         size_t m_endAtom; // This is a fuzzy end. Always test for < m_endAtom.
 #if ENABLE(PARALLEL_GC)
-        WTF::Bitmap<atomsPerBlock, WTF::BitmapAtomic> m_marks;
+        WTF::Bitmap<atomsPerBlock, WTF::BitmapAtomic, uint8_t> m_marks;
+        WTF::Bitmap<atomsPerBlock, WTF::BitmapAtomic, uint8_t> m_rememberedSet;
 #else
-        WTF::Bitmap<atomsPerBlock, WTF::BitmapNotAtomic> m_marks;
+        WTF::Bitmap<atomsPerBlock, WTF::BitmapNotAtomic, uint8_t> m_marks;
+        WTF::Bitmap<atomsPerBlock, WTF::BitmapNotAtomic, uint8_t> m_rememberedSet;
 #endif
         OwnPtr<WTF::Bitmap<atomsPerBlock>> m_newlyAllocated;
@@ -235 +248 @@
     }
 
-    inline void MarkedBlock::lastChanceToFinalize()
-    {
-        m_weakSet.lastChanceToFinalize();
-
-        clearNewlyAllocated();
-        clearMarks();
-        sweep();
-    }
-
     inline MarkedAllocator* MarkedBlock::allocator() const
     {
@@ -292 +296 @@
 
         ASSERT(!m_newlyAllocated);
-#ifndef NDEBUG
-        for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell)
-            ASSERT(m_marks.get(i));
-#endif
         ASSERT(m_state == FreeListed);
         m_state = Marked;
     }
 
-    inline void MarkedBlock::clearMarks()
-    {
-        HEAP_LOG_BLOCK_STATE_TRANSITION(this);
-
-        ASSERT(m_state != New && m_state != FreeListed);
-        m_marks.clearAll();
-
-        // This will become true at the end of the mark phase. We set it now to
-        // avoid an extra pass to do so later.
-        m_state = Marked;
-    }
-
     inline size_t MarkedBlock::markCount()
     {
@@ -345 +333 @@
     {
         return (reinterpret_cast<Bits>(p) - reinterpret_cast<Bits>(this)) / atomSize;
+    }
+
+    inline void MarkedBlock::setRemembered(const void* p)
+    {
+        m_rememberedSet.set(atomNumber(p));
+    }
+
+    inline void MarkedBlock::clearRemembered(const void* p)
+    {
+        m_rememberedSet.clear(atomNumber(p));
+    }
+
+    inline void MarkedBlock::atomicClearRemembered(const void* p)
+    {
+        m_rememberedSet.concurrentTestAndClear(atomNumber(p));
+    }
+
+    inline bool MarkedBlock::isRemembered(const void* p)
+    {
+        return m_rememberedSet.get(atomNumber(p));
     }
 

trunk/Source/JavaScriptCore/heap/MarkedSpace.cpp

@@ -106 +106 @@
     Free free(Free::FreeAll, this);
     forEachBlock(free);
+    ASSERT(!m_blocks.set().size());
 }
 
@@ -144 +145 @@
     m_normalDestructorSpace.largeAllocator.reset();
     m_immortalStructureDestructorSpace.largeAllocator.reset();
+
+    m_blocksWithNewObjects.clear();
 }
 
@@ -149 +152 @@
 {
     VisitWeakSet visitWeakSet(heapRootVisitor);
-    forEachBlock(visitWeakSet);
+    if (m_heap->operationInProgress() == EdenCollection) {
+        for (unsigned i = 0; i < m_blocksWithNewObjects.size(); ++i)
+            visitWeakSet(m_blocksWithNewObjects[i]);
+    } else
+        forEachBlock(visitWeakSet);
 }
 
 void MarkedSpace::reapWeakSets()
 {
-    forEachBlock<ReapWeakSet>();
+    if (m_heap->operationInProgress() == EdenCollection) {
+        for (unsigned i = 0; i < m_blocksWithNewObjects.size(); ++i)
+            m_blocksWithNewObjects[i]->reapWeakSet();
+    } else
+        forEachBlock<ReapWeakSet>();
 }
 
@@ -306 +317 @@
 }
 
+#ifndef NDEBUG
+struct VerifyMarked : MarkedBlock::VoidFunctor {
+    void operator()(MarkedBlock* block) { ASSERT(block->needsSweeping()); }
+};
+#endif
+
+void MarkedSpace::clearMarks()
+{
+    if (m_heap->operationInProgress() == EdenCollection) {
+        for (unsigned i = 0; i < m_blocksWithNewObjects.size(); ++i)
+            m_blocksWithNewObjects[i]->clearMarks();
+    } else
+        forEachBlock<ClearMarks>();
+#ifndef NDEBUG
+    forEachBlock<VerifyMarked>();
+#endif
+}
+
 void MarkedSpace::willStartIterating()
 {

trunk/Source/JavaScriptCore/heap/MarkedSpace.h

@@ -47 +47 @@
 
 struct ClearMarks : MarkedBlock::VoidFunctor {
-    void operator()(MarkedBlock* block) { block->clearMarks(); }
+    void operator()(MarkedBlock* block)
+    {
+        block->clearMarks();
+    }
+};
+
+struct ClearRememberedSet : MarkedBlock::VoidFunctor {
+    void operator()(MarkedBlock* block)
+    {
+        block->clearRememberedSet();
+    }
 };
 
@@ -106 +116 @@
     void didAddBlock(MarkedBlock*);
     void didConsumeFreeList(MarkedBlock*);
+    void didAllocateInBlock(MarkedBlock*);
 
     void clearMarks();
+    void clearRememberedSet();
     void clearNewlyAllocated();
     void sweep();
@@ -151 +163 @@
     bool m_isIterating;
     MarkedBlockSet m_blocks;
+    Vector<MarkedBlock*> m_blocksWithNewObjects;
 
     DelayedReleaseScope* m_currentDelayedReleaseScope;
@@ -263 +276 @@
 }
 
-inline void MarkedSpace::clearMarks()
-{
-    forEachBlock<ClearMarks>();
+inline void MarkedSpace::didAllocateInBlock(MarkedBlock* block)
+{
+    m_blocksWithNewObjects.append(block);
+}
+
+inline void MarkedSpace::clearRememberedSet()
+{
+    forEachBlock<ClearRememberedSet>();
 }
 

trunk/Source/JavaScriptCore/heap/SlotVisitor.cpp

@@ -34 +34 @@
 SlotVisitor::~SlotVisitor()
 {
-    ASSERT(m_stack.isEmpty());
+    clearMarkStack();
 }
 
@@ -62 +62 @@
         m_shouldHashCons = false;
     }
+}
+
+void SlotVisitor::clearMarkStack()
+{
+    m_stack.clear();
 }
 

trunk/Source/JavaScriptCore/heap/SlotVisitor.h

@@ -50 +50 @@
     ~SlotVisitor();
 
+    MarkStackArray& markStack() { return m_stack; }
+
+    Heap* heap() const;
+
     void append(ConservativeRoots&);
     
@@ -62 +66 @@
     template<typename T>
     void appendUnbarrieredWeak(Weak<T>*);
+    void unconditionallyAppend(JSCell*);
     
     void addOpaqueRoot(void*);
@@ -68 +73 @@
     int opaqueRootCount();
 
-    GCThreadSharedData& sharedData() { return m_shared; }
+    GCThreadSharedData& sharedData() const { return m_shared; }
     bool isEmpty() { return m_stack.isEmpty(); }
 
     void setup();
     void reset();
+    void clearMarkStack();
 
     size_t bytesVisited() const { return m_bytesVisited; }
@@ -90 +96 @@
     void copyLater(JSCell*, CopyToken, void*, size_t);
     
-    void reportExtraMemoryUsage(size_t size);
+    void reportExtraMemoryUsage(JSCell* owner, size_t);
     
     void addWeakReferenceHarvester(WeakReferenceHarvester*);

trunk/Source/JavaScriptCore/heap/SlotVisitorInlines.h

@@ -106 +106 @@
     MARK_LOG_CHILD(*this, cell);
 
+    unconditionallyAppend(cell);
+}
+
+ALWAYS_INLINE void SlotVisitor::unconditionallyAppend(JSCell* cell)
+{
+    ASSERT(Heap::isMarked(cell));
+    m_visitCount++;
+        
     // Should never attempt to mark something that is zapped.
     ASSERT(!cell->isZapped());
@@ -219 +227 @@
 {
     ASSERT(bytes);
+    // We don't do any copying during EdenCollections.
+    ASSERT(heap()->operationInProgress() != EdenCollection);
+
     m_bytesCopied += bytes;
 
@@ -227 +238 @@
     }
 
-    if (block->isPinned())
-        return;
-
     block->reportLiveBytes(owner, token, bytes);
 }
     
-inline void SlotVisitor::reportExtraMemoryUsage(size_t size)
-{
+inline void SlotVisitor::reportExtraMemoryUsage(JSCell* owner, size_t size)
+{
+    // We don't want to double-count the extra memory that was reported in previous collections.
+    if (heap()->operationInProgress() == EdenCollection && MarkedBlock::blockFor(owner)->isRemembered(owner))
+        return;
+
     size_t* counter = &m_shared.m_vm->heap.m_extraMemoryUsage;
     
@@ -248 +260 @@
 }
 
+inline Heap* SlotVisitor::heap() const
+{
+    return &sharedData().m_vm->heap;
+}
+
 } // namespace JSC
 

trunk/Source/JavaScriptCore/jit/Repatch.cpp

@@ -40 +40 @@
 #include "RepatchBuffer.h"
 #include "ScratchRegisterAllocator.h"
+#include "StackAlignment.h"
 #include "StructureRareDataInlines.h"
 #include "StructureStubClearingWatchpoint.h"

trunk/Source/JavaScriptCore/runtime/JSGenericTypedArrayViewInlines.h

@@ -448 +448 @@
         
     case OversizeTypedArray: {
-        visitor.reportExtraMemoryUsage(thisObject->byteSize());
+        visitor.reportExtraMemoryUsage(thisObject, thisObject->byteSize());
         break;
     }

trunk/Source/JavaScriptCore/runtime/JSPropertyNameIterator.h

@@ -110 +110 @@
     }
     
-    inline void StructureRareData::setEnumerationCache(VM& vm, const Structure* owner, JSPropertyNameIterator* value)
+    inline void StructureRareData::setEnumerationCache(VM& vm, const Structure*, JSPropertyNameIterator* value)
     {
-        m_enumerationCache.set(vm, owner, value);
+        m_enumerationCache.set(vm, this, value);
     }
 

trunk/Source/JavaScriptCore/runtime/JSString.cpp

@@ -73 +73 @@
         StringImpl* impl = thisObject->m_value.impl();
         ASSERT(impl);
-        visitor.reportExtraMemoryUsage(impl->costDuringGC());
+        visitor.reportExtraMemoryUsage(thisObject, impl->costDuringGC());
     }
 }

trunk/Source/JavaScriptCore/runtime/StructureRareDataInlines.h

@@ -36 +36 @@
 }
 
-inline void StructureRareData::setPreviousID(VM& vm, Structure* transition, Structure* structure)
+inline void StructureRareData::setPreviousID(VM& vm, Structure*, Structure* structure)
 {
-    m_previous.set(vm, transition, structure);
+    m_previous.set(vm, this, structure);
 }
 
@@ -51 +51 @@
 }
 
-inline void StructureRareData::setObjectToStringValue(VM& vm, const JSCell* owner, JSString* value)
+inline void StructureRareData::setObjectToStringValue(VM& vm, const JSCell*, JSString* value)
 {
-    m_objectToStringValue.set(vm, owner, value);
+    m_objectToStringValue.set(vm, this, value);
 }
 

trunk/Source/JavaScriptCore/runtime/WeakMapData.cpp

@@ -65 +65 @@
     // This isn't exact, but it is close enough, and proportional to the actual
     // external mermory usage.
-    visitor.reportExtraMemoryUsage(thisObj->m_map.capacity() * (sizeof(JSObject*) + sizeof(WriteBarrier<Unknown>)));
+    visitor.reportExtraMemoryUsage(thisObj, thisObj->m_map.capacity() * (sizeof(JSObject*) + sizeof(WriteBarrier<Unknown>)));
 }
 

trunk/Source/WTF/ChangeLog

@@ +1 @@
+2014-01-07  Mark Hahnenberg  <mhahnenberg@apple.com>
+
+        Marking should be generational
+        https://bugs.webkit.org/show_bug.cgi?id=126552
+
+        Reviewed by Geoffrey Garen.
+
+        * wtf/Bitmap.h:
+        (WTF::WordType>::count): Added a cast that became necessary when Bitmap
+        is used with smaller types than int32_t.
+
 2014-01-09  Simon Fraser  <simon.fraser@apple.com>
 

trunk/Source/WTF/wtf/Bitmap.h

@@ -197 +197 @@
     }
     for (size_t i = start / wordSize; i < words; ++i)
-        result += WTF::bitCount(bits[i]);
+        result += WTF::bitCount(static_cast<unsigned>(bits[i]));
     return result;
 }