Changeset 155891 in webkit


Ignore:
Timestamp:
Sep 16, 2013 12:48:48 PM (11 years ago)
Author:
mhahnenberg@apple.com
Message:

MarkedBlocks shouldn't be put in Allocated state if they didn't produce a FreeList
https://bugs.webkit.org/show_bug.cgi?id=121236

Reviewed by Geoffrey Garen.

Right now, after a collection all MarkedBlocks are in the Marked block state. When lazy sweeping
happens, if a block returns an empty free list after being swept, we call didConsumeFreeList(),
which moves the block into the Allocated block state. This happens to both the block that was
just being allocated out of (i.e. m_currentBlock) as well as any blocks who are completely full.
We should distinguish between these two cases: m_currentBlock should transition to
Allocated (because we were just allocating out of it) and any subsequent block that returns an
empty free list should transition back to the Marked state. This will make the block state more
consistent with the actual state the block is in, and it will also allow us to speed up moving
all blocks the the Marked state during generational collection.

Added new RAII-style HeapIterationScope class that notifies the Heap when it is about to be
iterated and when iteration has finished. Any clients that need accurate liveness data when
iterating over the Heap now need to use a HeapIterationScope so that the state of Heap can
be properly restored after they are done iterating. No new GC-allocated objects can be created
until this object goes out of scope.

(JSC::Debugger::recompileAllJSFunctions): Added HeapIterationScope for the Recompiler iteration.

  • heap/Heap.cpp:

(JSC::Heap::willStartIterating): Callback used by HeapIterationScope to indicate that iteration of
the Heap is about to begin. This will cause cell liveness data to be canonicalized by calling stopAllocating.
(JSC::Heap::didFinishIterating): Same, but indicates that iteration has finished.
(JSC::Heap::globalObjectCount): Used HeapIterationScope.
(JSC::Heap::objectTypeCounts): Ditto.
(JSC::Heap::markDeadObjects): Ditto.
(JSC::Heap::zombifyDeadObjects): Ditto.

  • heap/Heap.h:
  • heap/HeapIterationScope.h: Added. New RAII-style object for indicating to the Heap that it's about

to be iterated or that iteration has finished.
(JSC::HeapIterationScope::HeapIterationScope):
(JSC::HeapIterationScope::~HeapIterationScope):

  • heap/HeapStatistics.cpp:

(JSC::HeapStatistics::showObjectStatistics): Used new HeapIterationScope.

  • heap/MarkedAllocator.cpp:

(JSC::MarkedAllocator::tryAllocateHelper): We now treat the case where we have just finished
allocating out of the current block differently from the case where we sweep a block and it
returns an empty free list. This was the primary point of this patch.
(JSC::MarkedAllocator::allocateSlowCase): ASSERT that nobody is currently iterating the Heap
when allocating.

  • heap/MarkedAllocator.h:

(JSC::MarkedAllocator::reset): All allocators are reset after every collection. We need to make
sure that the m_lastActiveBlock gets cleared, which it might not always because we don't call
takeCanonicalizedBlock on blocks in the large allocators.
(JSC::MarkedAllocator::stopAllocating): We shouldn't already have a last active block,
so ASSERT as much.
(JSC::MarkedAllocator::resumeAllocating): Do the opposite of what stopAllocating
does. So, if we don't have a m_lastActiveBlock then we don't have to worry about undoing anything
done by stopAllocating. If we do, then we call resumeAllocating on the block, which returns the FreeList
as it was prior to stopping allocation. We then set the current block to the last active block and
clear the last active block.

  • heap/MarkedBlock.cpp:

(JSC::MarkedBlock::resumeAllocating): Any block resuming allocation should be in
the Marked state, so ASSERT as much. We always allocate a m_newlyAllocated Bitmap if we're
FreeListed, so if we didn't allocate one then we know we were Marked when allocation was stopped,
so just return early with an empty FreeList. If we do have a non-null m_newlyAllocated Bitmap
then we need to be swept in order to rebuild our FreeList.

  • heap/MarkedBlock.h:

(JSC::MarkedBlock::didConsumeEmptyFreeList): This is called if we ever sweep a block and get back
an empty free list. Instead of transitioning to the Allocated state, we now go straight back to the
Marked state. This makes sense because we weren't actually allocated out of, so we shouldn't be in
the allocated state. Also added some ASSERTs to make sure that we're in the state that we expect: all of
our mark bits should be set and we should not have a m_newlyAllocated Bitmap.

  • heap/MarkedSpace.cpp:

(JSC::MarkedSpace::MarkedSpace):
(JSC::MarkedSpace::forEachAllocator): Added a new functor-style iteration method so that we can
easily iterate over each allocator for, e.g., stopping and resuming allocators without
duplicating code.
(JSC::StopAllocatingFunctor::operator()): New functors for use with forEachAllocator.
(JSC::MarkedSpace::stopAllocating): Ditto.
(JSC::ResumeAllocatingFunctor::operator()): Ditto.
(JSC::MarkedSpace::resumeAllocating): Ditto.
(JSC::MarkedSpace::willStartIterating): Callback that notifies MarkedSpace that it is being iterated.
Does some ASSERTs, sets a flag, canonicalizes cell liveness data by calling stopAllocating.
(JSC::MarkedSpace::didFinishIterating): Ditto, but to signal that iteration has completed.

  • heap/MarkedSpace.h:

(JSC::MarkedSpace::iterationInProgress): Returns true if a HeapIterationScope is currently active.
(JSC::MarkedSpace::forEachLiveCell): Accepts a HeapIterationScope to enforce the rule that you have to
create one prior to iterating over the Heap.
(JSC::MarkedSpace::forEachDeadCell): Ditto.

  • runtime/JSGlobalObject.cpp:

(JSC::JSGlobalObject::haveABadTime): Changed to use new HeapIterationScope.

  • runtime/VM.cpp:

(JSC::VM::releaseExecutableMemory): Ditto.

Location:
trunk/Source/JavaScriptCore
Files:
1 added
14 edited

Legend:

Unmodified
Added
Removed
  • trunk/Source/JavaScriptCore/ChangeLog

    r155889 r155891  
     12013-09-13  Mark Hahnenberg  <mhahnenberg@apple.com>
     2
     3        MarkedBlocks shouldn't be put in Allocated state if they didn't produce a FreeList
     4        https://bugs.webkit.org/show_bug.cgi?id=121236
     5
     6        Reviewed by Geoffrey Garen.
     7
     8        Right now, after a collection all MarkedBlocks are in the Marked block state. When lazy sweeping
     9        happens, if a block returns an empty free list after being swept, we call didConsumeFreeList(),
     10        which moves the block into the Allocated block state. This happens to both the block that was
     11        just being allocated out of (i.e. m_currentBlock) as well as any blocks who are completely full.
     12        We should distinguish between these two cases: m_currentBlock should transition to
     13        Allocated (because we were just allocating out of it) and any subsequent block that returns an
     14        empty free list should transition back to the Marked state. This will make the block state more
     15        consistent with the actual state the block is in, and it will also allow us to speed up moving
     16        all blocks the the Marked state during generational collection.
     17
     18        Added new RAII-style HeapIterationScope class that notifies the Heap when it is about to be
     19        iterated and when iteration has finished. Any clients that need accurate liveness data when
     20        iterating over the Heap now need to use a HeapIterationScope so that the state of Heap can
     21        be properly restored after they are done iterating. No new GC-allocated objects can be created
     22        until this object goes out of scope.
     23
     24        * JavaScriptCore.xcodeproj/project.pbxproj:
     25        * debugger/Debugger.cpp:
     26        (JSC::Debugger::recompileAllJSFunctions): Added HeapIterationScope for the Recompiler iteration.
     27        * heap/Heap.cpp:
     28        (JSC::Heap::willStartIterating): Callback used by HeapIterationScope to indicate that iteration of
     29        the Heap is about to begin. This will cause cell liveness data to be canonicalized by calling stopAllocating.
     30        (JSC::Heap::didFinishIterating): Same, but indicates that iteration has finished.
     31        (JSC::Heap::globalObjectCount): Used HeapIterationScope.
     32        (JSC::Heap::objectTypeCounts): Ditto.
     33        (JSC::Heap::markDeadObjects): Ditto.
     34        (JSC::Heap::zombifyDeadObjects): Ditto.
     35        * heap/Heap.h:
     36        * heap/HeapIterationScope.h: Added. New RAII-style object for indicating to the Heap that it's about
     37        to be iterated or that iteration has finished.
     38        (JSC::HeapIterationScope::HeapIterationScope):
     39        (JSC::HeapIterationScope::~HeapIterationScope):
     40        * heap/HeapStatistics.cpp:
     41        (JSC::HeapStatistics::showObjectStatistics): Used new HeapIterationScope.
     42        * heap/MarkedAllocator.cpp:
     43        (JSC::MarkedAllocator::tryAllocateHelper): We now treat the case where we have just finished
     44        allocating out of the current block differently from the case where we sweep a block and it
     45        returns an empty free list. This was the primary point of this patch.
     46        (JSC::MarkedAllocator::allocateSlowCase): ASSERT that nobody is currently iterating the Heap
     47        when allocating.
     48        * heap/MarkedAllocator.h:
     49        (JSC::MarkedAllocator::reset): All allocators are reset after every collection. We need to make
     50        sure that the m_lastActiveBlock gets cleared, which it might not always because we don't call
     51        takeCanonicalizedBlock on blocks in the large allocators.
     52        (JSC::MarkedAllocator::stopAllocating): We shouldn't already have a last active block,
     53        so ASSERT as much.
     54        (JSC::MarkedAllocator::resumeAllocating): Do the opposite of what stopAllocating
     55        does. So, if we don't have a m_lastActiveBlock then we don't have to worry about undoing anything
     56        done by stopAllocating. If we do, then we call resumeAllocating on the block, which returns the FreeList
     57        as it was prior to stopping allocation. We then set the current block to the last active block and
     58        clear the last active block.
     59        * heap/MarkedBlock.cpp:
     60        (JSC::MarkedBlock::resumeAllocating): Any block resuming allocation should be in
     61        the Marked state, so ASSERT as much. We always allocate a m_newlyAllocated Bitmap if we're
     62        FreeListed, so if we didn't allocate one then we know we were Marked when allocation was stopped,
     63        so just return early with an empty FreeList. If we do have a non-null m_newlyAllocated Bitmap
     64        then we need to be swept in order to rebuild our FreeList.
     65        * heap/MarkedBlock.h:
     66        (JSC::MarkedBlock::didConsumeEmptyFreeList): This is called if we ever sweep a block and get back
     67        an empty free list. Instead of transitioning to the Allocated state, we now go straight back to the
     68        Marked state. This makes sense because we weren't actually allocated out of, so we shouldn't be in
     69        the allocated state. Also added some ASSERTs to make sure that we're in the state that we expect: all of
     70        our mark bits should be set and we should not have a m_newlyAllocated Bitmap.
     71        * heap/MarkedSpace.cpp:
     72        (JSC::MarkedSpace::MarkedSpace):
     73        (JSC::MarkedSpace::forEachAllocator): Added a new functor-style iteration method so that we can
     74        easily iterate over each allocator for, e.g., stopping and resuming allocators without
     75        duplicating code.
     76        (JSC::StopAllocatingFunctor::operator()): New functors for use with forEachAllocator.
     77        (JSC::MarkedSpace::stopAllocating): Ditto.
     78        (JSC::ResumeAllocatingFunctor::operator()): Ditto.
     79        (JSC::MarkedSpace::resumeAllocating): Ditto.
     80        (JSC::MarkedSpace::willStartIterating): Callback that notifies MarkedSpace that it is being iterated.
     81        Does some ASSERTs, sets a flag, canonicalizes cell liveness data by calling stopAllocating.
     82        (JSC::MarkedSpace::didFinishIterating): Ditto, but to signal that iteration has completed.
     83        * heap/MarkedSpace.h:
     84        (JSC::MarkedSpace::iterationInProgress): Returns true if a HeapIterationScope is currently active.
     85        (JSC::MarkedSpace::forEachLiveCell): Accepts a HeapIterationScope to enforce the rule that you have to
     86        create one prior to iterating over the Heap.
     87        (JSC::MarkedSpace::forEachDeadCell): Ditto.
     88        * runtime/JSGlobalObject.cpp:
     89        (JSC::JSGlobalObject::haveABadTime): Changed to use new HeapIterationScope.
     90        * runtime/VM.cpp:
     91        (JSC::VM::releaseExecutableMemory): Ditto.
     92
    1932013-09-16  Filip Pizlo  <fpizlo@apple.com>
    294
  • trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj

    r155473 r155891  
    650650                2600B5A7152BAAA70091EE5F /* JSStringJoiner.h in Headers */ = {isa = PBXBuildFile; fileRef = 2600B5A5152BAAA70091EE5F /* JSStringJoiner.h */; };
    651651                2A48D1911772365B00C65A5F /* APICallbackFunction.h in Headers */ = {isa = PBXBuildFile; fileRef = C211B574176A224D000E2A23 /* APICallbackFunction.h */; };
     652                2AD8932B17E3868F00668276 /* HeapIterationScope.h in Headers */ = {isa = PBXBuildFile; fileRef = 2AD8932917E3868F00668276 /* HeapIterationScope.h */; };
    652653                371D842D17C98B6E00ECF994 /* libz.dylib in Frameworks */ = {isa = PBXBuildFile; fileRef = 371D842C17C98B6E00ECF994 /* libz.dylib */; };
    653654                41359CF30FDD89AD00206180 /* DateConversion.h in Headers */ = {isa = PBXBuildFile; fileRef = D21202290AD4310C00ED79B6 /* DateConversion.h */; };
     
    18131814                2600B5A4152BAAA70091EE5F /* JSStringJoiner.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = JSStringJoiner.cpp; sourceTree = "<group>"; };
    18141815                2600B5A5152BAAA70091EE5F /* JSStringJoiner.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JSStringJoiner.h; sourceTree = "<group>"; };
     1816                2AD8932917E3868F00668276 /* HeapIterationScope.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = HeapIterationScope.h; sourceTree = "<group>"; };
    18151817                371D842C17C98B6E00ECF994 /* libz.dylib */ = {isa = PBXFileReference; lastKnownFileType = "compiled.mach-o.dylib"; name = libz.dylib; path = usr/lib/libz.dylib; sourceTree = SDKROOT; };
    18161818                449097EE0F8F81B50076A327 /* FeatureDefines.xcconfig */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.xcconfig; path = FeatureDefines.xcconfig; sourceTree = "<group>"; };
     
    28512853                                0FC8150814043BCA00CFA603 /* WriteBarrierSupport.cpp */,
    28522854                                0FC8150914043BD200CFA603 /* WriteBarrierSupport.h */,
     2855                                2AD8932917E3868F00668276 /* HeapIterationScope.h */,
    28532856                        );
    28542857                        path = heap;
     
    40294032                                0F766D4415B2A3C0008F363E /* DFGRegisterSet.h in Headers */,
    40304033                                86BB09C1138E381B0056702F /* DFGRepatch.h in Headers */,
     4034                                2AD8932B17E3868F00668276 /* HeapIterationScope.h in Headers */,
    40314035                                A77A424317A0BBFD00A8DB81 /* DFGSafeToExecute.h in Headers */,
    40324036                                A741017F179DAF80002EB8BA /* DFGSaneStringGetByValSlowPathGenerator.h in Headers */,
  • trunk/Source/JavaScriptCore/debugger/Debugger.cpp

    r154797 r155891  
    2424
    2525#include "Error.h"
     26#include "HeapIterationScope.h"
    2627#include "Interpreter.h"
    2728#include "JSFunction.h"
     
    123124
    124125    Recompiler recompiler(this);
    125     vm->heap.objectSpace().forEachLiveCell(recompiler);
     126    HeapIterationScope iterationScope(vm->heap);
     127    vm->heap.objectSpace().forEachLiveCell(iterationScope, recompiler);
    126128}
    127129
  • trunk/Source/JavaScriptCore/heap/Heap.cpp

    r155413 r155891  
    3030#include "GCActivityCallback.h"
    3131#include "GCIncomingRefCountedSetInlines.h"
     32#include "HeapIterationScope.h"
    3233#include "HeapRootVisitor.h"
    3334#include "HeapStatistics.h"
     
    416417}
    417418
    418 void Heap::canonicalizeCellLivenessData()
    419 {
    420     m_objectSpace.canonicalizeCellLivenessData();
     419void Heap::willStartIterating()
     420{
     421    m_objectSpace.willStartIterating();
     422}
     423
     424void Heap::didFinishIterating()
     425{
     426    m_objectSpace.didFinishIterating();
    421427}
    422428
     
    663669size_t Heap::globalObjectCount()
    664670{
    665     return m_objectSpace.forEachLiveCell<CountIfGlobalObject>();
     671    HeapIterationScope iterationScope(*this);
     672    return m_objectSpace.forEachLiveCell<CountIfGlobalObject>(iterationScope);
    666673}
    667674
     
    678685PassOwnPtr<TypeCountSet> Heap::objectTypeCounts()
    679686{
    680     return m_objectSpace.forEachLiveCell<RecordType>();
     687    HeapIterationScope iterationScope(*this);
     688    return m_objectSpace.forEachLiveCell<RecordType>(iterationScope);
    681689}
    682690
     
    764772
    765773    {
    766         GCPHASE(Canonicalize);
    767         m_objectSpace.canonicalizeCellLivenessData();
     774        GCPHASE(StopAllocation);
     775        m_objectSpace.stopAllocating();
    768776    }
    769777
     
    876884void Heap::markDeadObjects()
    877885{
    878     m_objectSpace.forEachDeadCell<MarkObject>();
     886    HeapIterationScope iterationScope(*this);
     887    m_objectSpace.forEachDeadCell<MarkObject>(iterationScope);
    879888}
    880889
     
    955964    // Sweep now because destructors will crash once we're zombified.
    956965    m_objectSpace.sweep();
    957     m_objectSpace.forEachDeadCell<Zombify>();
     966    HeapIterationScope iterationScope(*this);
     967    m_objectSpace.forEachDeadCell<Zombify>(iterationScope);
    958968}
    959969
  • trunk/Source/JavaScriptCore/heap/Heap.h

    r155317 r155891  
    167167        HandleStack* handleStack() { return &m_handleStack; }
    168168
    169         void canonicalizeCellLivenessData();
     169        void willStartIterating();
     170        void didFinishIterating();
    170171        void getConservativeRegisterRoots(HashSet<JSCell*>& roots);
    171172
  • trunk/Source/JavaScriptCore/heap/HeapStatistics.cpp

    r139541 r155891  
    2828
    2929#include "Heap.h"
     30#include "HeapIterationScope.h"
    3031#include "JSObject.h"
    3132#include "Operations.h"
     
    236237
    237238    StorageStatistics storageStatistics;
    238     heap->m_objectSpace.forEachLiveCell(storageStatistics);
     239    {
     240        HeapIterationScope iterationScope(*heap);
     241        heap->m_objectSpace.forEachLiveCell(iterationScope, storageStatistics);
     242    }
    239243    dataLogF("wasted .property storage: %ldkB (%ld%%)\n",
    240244        static_cast<long>(
  • trunk/Source/JavaScriptCore/heap/MarkedAllocator.cpp

    r155652 r155891  
    3131{
    3232    if (!m_freeList.head) {
     33        if (m_currentBlock) {
     34            ASSERT(m_currentBlock == m_blocksToSweep);
     35            m_currentBlock->didConsumeFreeList();
     36            m_blocksToSweep = m_currentBlock->next();
     37        }
     38
    3339        for (MarkedBlock*& block = m_blocksToSweep; block; block = block->next()) {
    3440            MarkedBlock::FreeList freeList = block->sweep(MarkedBlock::SweepToFreeList);
    3541            if (!freeList.head) {
    36                 block->didConsumeFreeList();
     42                block->didConsumeEmptyFreeList();
    3743                continue;
    3844            }
    3945
    4046            if (bytes > block->cellSize()) {
    41                 block->canonicalizeCellLivenessData(freeList);
     47                block->stopAllocating(freeList);
    4248                continue;
    4349            }
     
    7783#endif
    7884   
     85    ASSERT(!m_markedSpace->isIterating());
    7986    ASSERT(!m_freeList.head);
    8087    m_heap->didAllocate(m_freeList.bytes);
  • trunk/Source/JavaScriptCore/heap/MarkedAllocator.h

    r155316 r155891  
    2323    MarkedAllocator();
    2424    void reset();
    25     void canonicalizeCellLivenessData();
     25    void stopAllocating();
     26    void resumeAllocating();
    2627    size_t cellSize() { return m_cellSize; }
    2728    MarkedBlock::DestructorType destructorType() { return m_destructorType; }
    2829    void* allocate(size_t);
    2930    Heap* heap() { return m_heap; }
    30     MarkedBlock* takeCanonicalizedBlock()
     31    MarkedBlock* takeLastActiveBlock()
    3132    {
    32         MarkedBlock* block = m_canonicalizedBlock;
    33         m_canonicalizedBlock = 0;
     33        MarkedBlock* block = m_lastActiveBlock;
     34        m_lastActiveBlock = 0;
    3435        return block;
    3536    }
     
    5152    MarkedBlock::FreeList m_freeList;
    5253    MarkedBlock* m_currentBlock;
    53     MarkedBlock* m_canonicalizedBlock;
     54    MarkedBlock* m_lastActiveBlock;
    5455    MarkedBlock* m_blocksToSweep;
    5556    DoublyLinkedList<MarkedBlock> m_blockList;
     
    6768inline MarkedAllocator::MarkedAllocator()
    6869    : m_currentBlock(0)
    69     , m_canonicalizedBlock(0)
     70    , m_lastActiveBlock(0)
    7071    , m_blocksToSweep(0)
    7172    , m_cellSize(0)
     
    104105inline void MarkedAllocator::reset()
    105106{
     107    m_lastActiveBlock = 0;
    106108    m_currentBlock = 0;
    107109    m_freeList = MarkedBlock::FreeList();
     
    109111}
    110112
    111 inline void MarkedAllocator::canonicalizeCellLivenessData()
     113inline void MarkedAllocator::stopAllocating()
    112114{
     115    ASSERT(!m_lastActiveBlock);
    113116    if (!m_currentBlock) {
    114117        ASSERT(!m_freeList.head);
     
    116119    }
    117120   
    118     m_currentBlock->canonicalizeCellLivenessData(m_freeList);
    119     m_canonicalizedBlock = m_currentBlock;
     121    m_currentBlock->stopAllocating(m_freeList);
     122    m_lastActiveBlock = m_currentBlock;
    120123    m_currentBlock = 0;
    121124    m_freeList = MarkedBlock::FreeList();
     125}
     126
     127inline void MarkedAllocator::resumeAllocating()
     128{
     129    if (!m_lastActiveBlock)
     130        return;
     131
     132    m_freeList = m_lastActiveBlock->resumeAllocating();
     133    m_currentBlock = m_lastActiveBlock;
     134    m_lastActiveBlock = 0;
    122135}
    123136
  • trunk/Source/JavaScriptCore/heap/MarkedBlock.cpp

    r148696 r155891  
    162162};
    163163
    164 void MarkedBlock::canonicalizeCellLivenessData(const FreeList& freeList)
     164void MarkedBlock::stopAllocating(const FreeList& freeList)
    165165{
    166166    HEAP_LOG_BLOCK_STATE_TRANSITION(this);
     
    200200}
    201201
     202MarkedBlock::FreeList MarkedBlock::resumeAllocating()
     203{
     204    HEAP_LOG_BLOCK_STATE_TRANSITION(this);
     205
     206    ASSERT(m_state == Marked);
     207
     208    if (!m_newlyAllocated) {
     209        // We didn't have to create a "newly allocated" bitmap. That means we were already Marked
     210        // when we last stopped allocation, so return an empty free list and stay in the Marked state.
     211        return FreeList();
     212    }
     213
     214    // Re-create our free list from before stopping allocation.
     215    return sweep(SweepToFreeList);
     216}
     217
    202218} // namespace JSC
  • trunk/Source/JavaScriptCore/heap/MarkedBlock.h

    r155652 r155891  
    133133        // of these functions:
    134134        void didConsumeFreeList(); // Call this once you've allocated all the items in the free list.
    135         void canonicalizeCellLivenessData(const FreeList&);
     135        void stopAllocating(const FreeList&);
     136        FreeList resumeAllocating(); // Call this if you canonicalized a block for some non-collection related purpose.
     137        void didConsumeEmptyFreeList(); // Call this if you sweep a block, but the returned FreeList is empty.
    136138
    137139        // Returns true if the "newly allocated" bitmap was non-null
     
    278280    }
    279281
     282    inline void MarkedBlock::didConsumeEmptyFreeList()
     283    {
     284        HEAP_LOG_BLOCK_STATE_TRANSITION(this);
     285
     286        ASSERT(!m_newlyAllocated);
     287#ifndef NDEBUG
     288        for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell)
     289            ASSERT(m_marks.get(i));
     290#endif
     291        ASSERT(m_state == FreeListed);
     292        m_state = Marked;
     293    }
     294
    280295    inline void MarkedBlock::clearMarks()
    281296    {
  • trunk/Source/JavaScriptCore/heap/MarkedSpace.cpp

    r155406 r155891  
    8181    : m_heap(heap)
    8282    , m_capacity(0)
     83    , m_isIterating(false)
    8384{
    8485    for (size_t cellSize = preciseStep; cellSize <= preciseCutoff; cellSize += preciseStep) {
     
    111112void MarkedSpace::lastChanceToFinalize()
    112113{
    113     canonicalizeCellLivenessData();
     114    stopAllocating();
    114115    forEachBlock<LastChanceToFinalize>();
    115116}
     
    151152}
    152153
    153 void MarkedSpace::canonicalizeCellLivenessData()
     154template <typename Functor>
     155void MarkedSpace::forEachAllocator()
     156{
     157    Functor functor;
     158    forEachAllocator(functor);
     159}
     160
     161template <typename Functor>
     162void MarkedSpace::forEachAllocator(Functor& functor)
    154163{
    155164    for (size_t cellSize = preciseStep; cellSize <= preciseCutoff; cellSize += preciseStep) {
    156         allocatorFor(cellSize).canonicalizeCellLivenessData();
    157         normalDestructorAllocatorFor(cellSize).canonicalizeCellLivenessData();
    158         immortalStructureDestructorAllocatorFor(cellSize).canonicalizeCellLivenessData();
     165        functor(allocatorFor(cellSize));
     166        functor(normalDestructorAllocatorFor(cellSize));
     167        functor(immortalStructureDestructorAllocatorFor(cellSize));
    159168    }
    160169
    161170    for (size_t cellSize = impreciseStep; cellSize <= impreciseCutoff; cellSize += impreciseStep) {
    162         allocatorFor(cellSize).canonicalizeCellLivenessData();
    163         normalDestructorAllocatorFor(cellSize).canonicalizeCellLivenessData();
    164         immortalStructureDestructorAllocatorFor(cellSize).canonicalizeCellLivenessData();
    165     }
    166 
    167     m_normalSpace.largeAllocator.canonicalizeCellLivenessData();
    168     m_normalDestructorSpace.largeAllocator.canonicalizeCellLivenessData();
    169     m_immortalStructureDestructorSpace.largeAllocator.canonicalizeCellLivenessData();
     171        functor(allocatorFor(cellSize));
     172        functor(normalDestructorAllocatorFor(cellSize));
     173        functor(immortalStructureDestructorAllocatorFor(cellSize));
     174    }
     175
     176    functor(m_normalSpace.largeAllocator);
     177    functor(m_normalDestructorSpace.largeAllocator);
     178    functor(m_immortalStructureDestructorSpace.largeAllocator);
     179}
     180
     181struct StopAllocatingFunctor {
     182    void operator()(MarkedAllocator& allocator) { allocator.stopAllocating(); }
     183};
     184
     185void MarkedSpace::stopAllocating()
     186{
     187    ASSERT(!isIterating());
     188    forEachAllocator<StopAllocatingFunctor>();
     189}
     190
     191struct ResumeAllocatingFunctor {
     192    void operator()(MarkedAllocator& allocator) { allocator.resumeAllocating(); }
     193};
     194
     195void MarkedSpace::resumeAllocating()
     196{
     197    ASSERT(isIterating());
     198    forEachAllocator<ResumeAllocatingFunctor>();
    170199}
    171200
     
    246275{
    247276    for (size_t i = 0; i < preciseCount; ++i) {
    248         clearNewlyAllocatedInBlock(m_normalSpace.preciseAllocators[i].takeCanonicalizedBlock());
    249         clearNewlyAllocatedInBlock(m_normalDestructorSpace.preciseAllocators[i].takeCanonicalizedBlock());
    250         clearNewlyAllocatedInBlock(m_immortalStructureDestructorSpace.preciseAllocators[i].takeCanonicalizedBlock());
     277        clearNewlyAllocatedInBlock(m_normalSpace.preciseAllocators[i].takeLastActiveBlock());
     278        clearNewlyAllocatedInBlock(m_normalDestructorSpace.preciseAllocators[i].takeLastActiveBlock());
     279        clearNewlyAllocatedInBlock(m_immortalStructureDestructorSpace.preciseAllocators[i].takeLastActiveBlock());
    251280    }
    252281
    253282    for (size_t i = 0; i < impreciseCount; ++i) {
    254         clearNewlyAllocatedInBlock(m_normalSpace.impreciseAllocators[i].takeCanonicalizedBlock());
    255         clearNewlyAllocatedInBlock(m_normalDestructorSpace.impreciseAllocators[i].takeCanonicalizedBlock());
    256         clearNewlyAllocatedInBlock(m_immortalStructureDestructorSpace.impreciseAllocators[i].takeCanonicalizedBlock());
     283        clearNewlyAllocatedInBlock(m_normalSpace.impreciseAllocators[i].takeLastActiveBlock());
     284        clearNewlyAllocatedInBlock(m_normalDestructorSpace.impreciseAllocators[i].takeLastActiveBlock());
     285        clearNewlyAllocatedInBlock(m_immortalStructureDestructorSpace.impreciseAllocators[i].takeLastActiveBlock());
    257286    }
    258287
     
    271300}
    272301
     302void MarkedSpace::willStartIterating()
     303{
     304    ASSERT(!isIterating());
     305    stopAllocating();
     306    m_isIterating = true;
     307}
     308
     309void MarkedSpace::didFinishIterating()
     310{
     311    ASSERT(isIterating());
     312    resumeAllocating();
     313    m_isIterating = false;
     314}
     315
    273316} // namespace JSC
  • trunk/Source/JavaScriptCore/heap/MarkedSpace.h

    r155406 r155891  
    3838
    3939class Heap;
     40class HeapIterationScope;
    4041class JSCell;
    4142class LiveObjectIterator;
     
    8182
    8283    MarkedBlockSet& blocks() { return m_blocks; }
    83    
    84     void canonicalizeCellLivenessData();
     84
     85    void willStartIterating();
     86    bool isIterating() { return m_isIterating; }
     87    void didFinishIterating();
     88
     89    void stopAllocating();
     90    void resumeAllocating(); // If we just stopped allocation but we didn't do a collection, we need to resume allocation.
    8591
    8692    typedef HashSet<MarkedBlock*>::iterator BlockIterator;
    8793   
    88     template<typename Functor> typename Functor::ReturnType forEachLiveCell(Functor&);
    89     template<typename Functor> typename Functor::ReturnType forEachLiveCell();
    90     template<typename Functor> typename Functor::ReturnType forEachDeadCell(Functor&);
    91     template<typename Functor> typename Functor::ReturnType forEachDeadCell();
     94    template<typename Functor> typename Functor::ReturnType forEachLiveCell(HeapIterationScope&, Functor&);
     95    template<typename Functor> typename Functor::ReturnType forEachLiveCell(HeapIterationScope&);
     96    template<typename Functor> typename Functor::ReturnType forEachDeadCell(HeapIterationScope&, Functor&);
     97    template<typename Functor> typename Functor::ReturnType forEachDeadCell(HeapIterationScope&);
    9298    template<typename Functor> typename Functor::ReturnType forEachBlock(Functor&);
    9399    template<typename Functor> typename Functor::ReturnType forEachBlock();
     
    112118    friend class LLIntOffsetsExtractor;
    113119
     120    template<typename Functor> void forEachAllocator(Functor&);
     121    template<typename Functor> void forEachAllocator();
     122
    114123    // [ 32... 128 ]
    115124    static const size_t preciseStep = MarkedBlock::atomSize;
     
    134143    Heap* m_heap;
    135144    size_t m_capacity;
     145    bool m_isIterating;
    136146    MarkedBlockSet m_blocks;
    137147};
    138148
    139 template<typename Functor> inline typename Functor::ReturnType MarkedSpace::forEachLiveCell(Functor& functor)
    140 {
    141     canonicalizeCellLivenessData();
    142 
     149template<typename Functor> inline typename Functor::ReturnType MarkedSpace::forEachLiveCell(HeapIterationScope&, Functor& functor)
     150{
     151    ASSERT(isIterating());
    143152    BlockIterator end = m_blocks.set().end();
    144153    for (BlockIterator it = m_blocks.set().begin(); it != end; ++it)
     
    147156}
    148157
    149 template<typename Functor> inline typename Functor::ReturnType MarkedSpace::forEachLiveCell()
     158template<typename Functor> inline typename Functor::ReturnType MarkedSpace::forEachLiveCell(HeapIterationScope& scope)
    150159{
    151160    Functor functor;
    152     return forEachLiveCell(functor);
    153 }
    154 
    155 template<typename Functor> inline typename Functor::ReturnType MarkedSpace::forEachDeadCell(Functor& functor)
    156 {
    157     canonicalizeCellLivenessData();
    158 
     161    return forEachLiveCell(scope, functor);
     162}
     163
     164template<typename Functor> inline typename Functor::ReturnType MarkedSpace::forEachDeadCell(HeapIterationScope&, Functor& functor)
     165{
     166    ASSERT(isIterating());
    159167    BlockIterator end = m_blocks.set().end();
    160168    for (BlockIterator it = m_blocks.set().begin(); it != end; ++it)
     
    163171}
    164172
    165 template<typename Functor> inline typename Functor::ReturnType MarkedSpace::forEachDeadCell()
     173template<typename Functor> inline typename Functor::ReturnType MarkedSpace::forEachDeadCell(HeapIterationScope& scope)
    166174{
    167175    Functor functor;
    168     return forEachDeadCell(functor);
     176    return forEachDeadCell(scope, functor);
    169177}
    170178
  • trunk/Source/JavaScriptCore/runtime/JSGlobalObject.cpp

    r155711 r155891  
    4747#include "FunctionPrototype.h"
    4848#include "GetterSetter.h"
     49#include "HeapIterationScope.h"
    4950#include "Interpreter.h"
    5051#include "JSAPIWrapperObject.h"
     
    515516    MarkedArgumentBuffer foundObjects; // Use MarkedArgumentBuffer because switchToSlowPutArrayStorage() may GC.
    516517    ObjectsWithBrokenIndexingFinder finder(foundObjects, this);
    517     vm.heap.objectSpace().forEachLiveCell(finder);
     518    {
     519        HeapIterationScope iterationScope(vm.heap);
     520        vm.heap.objectSpace().forEachLiveCell(iterationScope, finder);
     521    }
    518522    while (!foundObjects.isEmpty()) {
    519523        JSObject* object = asObject(foundObjects.last());
  • trunk/Source/JavaScriptCore/runtime/VM.cpp

    r155558 r155891  
    4343#include "GetterSetter.h"
    4444#include "Heap.h"
     45#include "HeapIterationScope.h"
    4546#include "HostCallReturnValue.h"
    4647#include "Identifier.h"
     
    522523    if (dynamicGlobalObject) {
    523524        StackPreservingRecompiler recompiler;
     525        HeapIterationScope iterationScope(heap);
    524526        HashSet<JSCell*> roots;
    525         heap.canonicalizeCellLivenessData();
    526527        heap.getConservativeRegisterRoots(roots);
    527528        HashSet<JSCell*>::iterator end = roots.end();
     
    544545               
    545546        }
    546         heap.objectSpace().forEachLiveCell<StackPreservingRecompiler>(recompiler);
     547        heap.objectSpace().forEachLiveCell<StackPreservingRecompiler>(iterationScope, recompiler);
    547548    }
    548549    m_regExpCache->invalidateCode();
Note: See TracChangeset for help on using the changeset viewer.