Changeset 252452 in webkit


Ignore:
Timestamp:
Nov 14, 2019 1:37:12 AM (4 years ago)
Author:
ysuzuki@apple.com
Message:

[JSC] BlockDirectory's bits should be compact
​https://bugs.webkit.org/show_bug.cgi?id=204149

Reviewed by Robin Morisset.

Source/JavaScriptCore:

We start applying IsoSubspace to all JSCells. This means that IsoSubspace should be small enough,
so that we can hold many IsoSubspaces without considering about memory regression.

In this patch, we introduce several things to shrink sizeof(IsoSubspace) from 528 to 384.

  1. Adjusting members to remove some paddings.
  2. Remove m_heap field since this can be got from the caller easily.
  3. Make MarkedSpace::heap() efficient: just doing pointer arithmetic.
  4. Remove m_size field from IsoSubspace since BlockDirectory knows cellSize.
  5. Introduce BlockDirectoryBits, which repalces 9 FastBitVector in BlockDirectory to this one class. Since all FastBitVector has the same size, we should not have a size field for each FastBitVector. We reuse FastBitVector's View mechanism to keep the same ergonomics while making BlockDirectoryBits much smaller. We put 9 uint32_t as Segment, and manage Vector<Segment> in this data structure. Since we touch several bits at the same time for the same block-index, this data structure is compact and efficient.
  • CMakeLists.txt:
  • JavaScriptCore.xcodeproj/project.pbxproj:
  • heap/AlignedMemoryAllocator.cpp:

(JSC::AlignedMemoryAllocator::registerDirectory):

  • heap/AlignedMemoryAllocator.h:
  • heap/Allocator.h:
  • heap/AllocatorInlines.h:

(JSC::Allocator::allocate const):

  • heap/BlockDirectory.cpp:

(JSC::BlockDirectory::BlockDirectory):
(JSC::BlockDirectory::findEmptyBlockToSteal):
(JSC::BlockDirectory::findBlockForAllocation):
(JSC::BlockDirectory::tryAllocateBlock):
(JSC::BlockDirectory::addBlock):
(JSC::BlockDirectory::removeBlock):
(JSC::BlockDirectory::prepareForAllocation):
(JSC::BlockDirectory::beginMarkingForFullCollection):
(JSC::BlockDirectory::endMarking):
(JSC::BlockDirectory::snapshotUnsweptForEdenCollection):
(JSC::BlockDirectory::snapshotUnsweptForFullCollection):
(JSC::BlockDirectory::findBlockToSweep):
(JSC::BlockDirectory::sweep):
(JSC::BlockDirectory::shrink):
(JSC::BlockDirectory::assertNoUnswept):
(JSC::BlockDirectory::parallelNotEmptyBlockSource):
(JSC::BlockDirectory::dumpBits):

  • heap/BlockDirectory.h:

(JSC::BlockDirectory::cellKind const):
(JSC::BlockDirectory::forEachBitVector):
(JSC::BlockDirectory::forEachBitVectorWithName):
(JSC::BlockDirectory::heap): Deleted.

  • heap/BlockDirectoryBits.h: Added.

(JSC::BlockDirectoryBits::BlockDirectoryBitVectorWordView::BlockDirectoryBitVectorWordView):
(JSC::BlockDirectoryBits::BlockDirectoryBitVectorWordView::numBits const):
(JSC::BlockDirectoryBits::BlockDirectoryBitVectorWordView::word const):
(JSC::BlockDirectoryBits::BlockDirectoryBitVectorWordView::word):
(JSC::BlockDirectoryBits::BlockDirectoryBitVectorWordView::clearAll):
(JSC::BlockDirectoryBits::BlockDirectoryBitVectorWordView::view const):
(JSC::BlockDirectoryBits::numBits const):
(JSC::BlockDirectoryBits::resize):
(JSC::BlockDirectoryBits::forEachSegment):

  • heap/BlockDirectoryInlines.h:

(JSC::BlockDirectory::forEachBlock):
(JSC::BlockDirectory::forEachNotEmptyBlock):

  • heap/CompleteSubspace.cpp:

(JSC::CompleteSubspace::allocatorForSlow):
(JSC::CompleteSubspace::tryAllocateSlow):

  • heap/CompleteSubspaceInlines.h:

(JSC::CompleteSubspace::allocateNonVirtual):

  • heap/IsoCellSet.cpp:

(JSC::IsoCellSet::parallelNotEmptyMarkedBlockSource):

  • heap/IsoCellSetInlines.h:

(JSC::IsoCellSet::forEachMarkedCell):

  • heap/IsoSubspace.cpp:

(JSC::IsoSubspace::IsoSubspace):
(JSC::IsoSubspace::tryAllocateFromLowerTier):

  • heap/IsoSubspace.h:

(JSC::IsoSubspace::cellSize):
(JSC::IsoSubspace::allocatorForNonVirtual):
(JSC::IsoSubspace::size const): Deleted.
(): Deleted.

  • heap/IsoSubspaceInlines.h:

(JSC::IsoSubspace::allocateNonVirtual):

  • heap/IsoSubspacePerVM.cpp:

(JSC::IsoSubspacePerVM::AutoremovingIsoSubspace::~AutoremovingIsoSubspace):

  • heap/LocalAllocator.cpp:

(JSC::LocalAllocator::allocateSlowCase):
(JSC::LocalAllocator::doTestCollectionsIfNeeded):

  • heap/LocalAllocator.h:
  • heap/LocalAllocatorInlines.h:

(JSC::LocalAllocator::allocate):

  • heap/MarkedBlock.cpp:

(JSC::MarkedBlock::Handle::dumpState):

  • heap/MarkedSpace.cpp:

(JSC::MarkedSpace::MarkedSpace):
(JSC::MarkedSpace::sweepBlocks):
(JSC::MarkedSpace::prepareForAllocation):
(JSC::MarkedSpace::visitWeakSets):
(JSC::MarkedSpace::reapWeakSets):
(JSC::MarkedSpace::prepareForMarking):
(JSC::MarkedSpace::beginMarking):
(JSC::MarkedSpace::snapshotUnswept):

  • heap/MarkedSpace.h:

(JSC::MarkedSpace::heap const): Deleted.

  • heap/MarkedSpaceInlines.h:

(JSC::MarkedSpace::heap const):

  • heap/Subspace.cpp:

(JSC::Subspace::initialize):

  • heap/Subspace.h:

Source/WTF:

  • wtf/FastBitVector.h:

(WTF::fastBitVectorArrayLength):
(WTF::FastBitVectorImpl::unsafeWords):
(WTF::FastBitVectorImpl::unsafeWords const):
(WTF::FastBitReference::FastBitReference):
(WTF::FastBitReference::operator bool const):
(WTF::FastBitReference::operator=):
(WTF::FastBitVector::at):
(WTF::FastBitVector::operator[]):
(WTF::FastBitVector::BitReference::BitReference): Deleted.
(WTF::FastBitVector::BitReference::operator bool const): Deleted.
(WTF::FastBitVector::BitReference::operator=): Deleted.

Location:
trunk/Source
Files:
1 added
29 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/JavaScriptCore/CMakeLists.txt

    r252302 r252452  
    570570    heap/AllocatorForMode.h
    571571    heap/BlockDirectory.h
     572    heap/BlockDirectoryBits.h
    572573    heap/BlockDirectoryInlines.h
    573574    heap/CellAttributes.h

  • trunk/Source/JavaScriptCore/ChangeLog

    r252422 r252452  
     12019-11-14  Yusuke Suzuki  <ysuzuki@apple.com>
     2
     3        [JSC] BlockDirectory's bits should be compact
     4        https://bugs.webkit.org/show_bug.cgi?id=204149
     5
     6        Reviewed by Robin Morisset.
     7
     8        We start applying IsoSubspace to all JSCells. This means that IsoSubspace should be small enough,
     9        so that we can hold many IsoSubspaces without considering about memory regression.
     10
     11        In this patch, we introduce several things to shrink sizeof(IsoSubspace) from 528 to 384.
     12
     13        1. Adjusting members to remove some paddings.
     14        2. Remove m_heap field since this can be got from the caller easily.
     15        3. Make MarkedSpace::heap() efficient: just doing pointer arithmetic.
     16        4. Remove m_size field from IsoSubspace since BlockDirectory knows cellSize.
     17        5. Introduce BlockDirectoryBits, which repalces 9 FastBitVector in BlockDirectory to this one class.
     18           Since all FastBitVector has the same size, we should not have a size field for each FastBitVector.
     19           We reuse FastBitVector's View mechanism to keep the same ergonomics while making BlockDirectoryBits
     20           much smaller. We put 9 uint32_t as Segment, and manage Vector<Segment> in this data structure. Since
     21           we touch several bits at the same time for the same block-index, this data structure is compact and
     22           efficient.
     23
     24        * CMakeLists.txt:
     25        * JavaScriptCore.xcodeproj/project.pbxproj:
     26        * heap/AlignedMemoryAllocator.cpp:
     27        (JSC::AlignedMemoryAllocator::registerDirectory):
     28        * heap/AlignedMemoryAllocator.h:
     29        * heap/Allocator.h:
     30        * heap/AllocatorInlines.h:
     31        (JSC::Allocator::allocate const):
     32        * heap/BlockDirectory.cpp:
     33        (JSC::BlockDirectory::BlockDirectory):
     34        (JSC::BlockDirectory::findEmptyBlockToSteal):
     35        (JSC::BlockDirectory::findBlockForAllocation):
     36        (JSC::BlockDirectory::tryAllocateBlock):
     37        (JSC::BlockDirectory::addBlock):
     38        (JSC::BlockDirectory::removeBlock):
     39        (JSC::BlockDirectory::prepareForAllocation):
     40        (JSC::BlockDirectory::beginMarkingForFullCollection):
     41        (JSC::BlockDirectory::endMarking):
     42        (JSC::BlockDirectory::snapshotUnsweptForEdenCollection):
     43        (JSC::BlockDirectory::snapshotUnsweptForFullCollection):
     44        (JSC::BlockDirectory::findBlockToSweep):
     45        (JSC::BlockDirectory::sweep):
     46        (JSC::BlockDirectory::shrink):
     47        (JSC::BlockDirectory::assertNoUnswept):
     48        (JSC::BlockDirectory::parallelNotEmptyBlockSource):
     49        (JSC::BlockDirectory::dumpBits):
     50        * heap/BlockDirectory.h:
     51        (JSC::BlockDirectory::cellKind const):
     52        (JSC::BlockDirectory::forEachBitVector):
     53        (JSC::BlockDirectory::forEachBitVectorWithName):
     54        (JSC::BlockDirectory::heap): Deleted.
     55        * heap/BlockDirectoryBits.h: Added.
     56        (JSC::BlockDirectoryBits::BlockDirectoryBitVectorWordView::BlockDirectoryBitVectorWordView):
     57        (JSC::BlockDirectoryBits::BlockDirectoryBitVectorWordView::numBits const):
     58        (JSC::BlockDirectoryBits::BlockDirectoryBitVectorWordView::word const):
     59        (JSC::BlockDirectoryBits::BlockDirectoryBitVectorWordView::word):
     60        (JSC::BlockDirectoryBits::BlockDirectoryBitVectorWordView::clearAll):
     61        (JSC::BlockDirectoryBits::BlockDirectoryBitVectorWordView::view const):
     62        (JSC::BlockDirectoryBits::numBits const):
     63        (JSC::BlockDirectoryBits::resize):
     64        (JSC::BlockDirectoryBits::forEachSegment):
     65        * heap/BlockDirectoryInlines.h:
     66        (JSC::BlockDirectory::forEachBlock):
     67        (JSC::BlockDirectory::forEachNotEmptyBlock):
     68        * heap/CompleteSubspace.cpp:
     69        (JSC::CompleteSubspace::allocatorForSlow):
     70        (JSC::CompleteSubspace::tryAllocateSlow):
     71        * heap/CompleteSubspaceInlines.h:
     72        (JSC::CompleteSubspace::allocateNonVirtual):
     73        * heap/IsoCellSet.cpp:
     74        (JSC::IsoCellSet::parallelNotEmptyMarkedBlockSource):
     75        * heap/IsoCellSetInlines.h:
     76        (JSC::IsoCellSet::forEachMarkedCell):
     77        * heap/IsoSubspace.cpp:
     78        (JSC::IsoSubspace::IsoSubspace):
     79        (JSC::IsoSubspace::tryAllocateFromLowerTier):
     80        * heap/IsoSubspace.h:
     81        (JSC::IsoSubspace::cellSize):
     82        (JSC::IsoSubspace::allocatorForNonVirtual):
     83        (JSC::IsoSubspace::size const): Deleted.
     84        (): Deleted.
     85        * heap/IsoSubspaceInlines.h:
     86        (JSC::IsoSubspace::allocateNonVirtual):
     87        * heap/IsoSubspacePerVM.cpp:
     88        (JSC::IsoSubspacePerVM::AutoremovingIsoSubspace::~AutoremovingIsoSubspace):
     89        * heap/LocalAllocator.cpp:
     90        (JSC::LocalAllocator::allocateSlowCase):
     91        (JSC::LocalAllocator::doTestCollectionsIfNeeded):
     92        * heap/LocalAllocator.h:
     93        * heap/LocalAllocatorInlines.h:
     94        (JSC::LocalAllocator::allocate):
     95        * heap/MarkedBlock.cpp:
     96        (JSC::MarkedBlock::Handle::dumpState):
     97        * heap/MarkedSpace.cpp:
     98        (JSC::MarkedSpace::MarkedSpace):
     99        (JSC::MarkedSpace::sweepBlocks):
     100        (JSC::MarkedSpace::prepareForAllocation):
     101        (JSC::MarkedSpace::visitWeakSets):
     102        (JSC::MarkedSpace::reapWeakSets):
     103        (JSC::MarkedSpace::prepareForMarking):
     104        (JSC::MarkedSpace::beginMarking):
     105        (JSC::MarkedSpace::snapshotUnswept):
     106        * heap/MarkedSpace.h:
     107        (JSC::MarkedSpace::heap const): Deleted.
     108        * heap/MarkedSpaceInlines.h:
     109        (JSC::MarkedSpace::heap const):
     110        * heap/Subspace.cpp:
     111        (JSC::Subspace::initialize):
     112        * heap/Subspace.h:
     113
    11142019-11-13  Robin Morisset  <rmorisset@apple.com>
    2115

  • trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj

    r252310 r252452  
    18181818                E3794E761B77EB97005543AE /* ModuleAnalyzer.h in Headers */ = {isa = PBXBuildFile; fileRef = E3794E741B77EB97005543AE /* ModuleAnalyzer.h */; settings = {ATTRIBUTES = (Private, ); }; };
    18191819                E3850B15226ED641009ABF9C /* DFGMinifiedIDInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = E3850B14226ED63E009ABF9C /* DFGMinifiedIDInlines.h */; };
     1820                E38652E3237CA0C900E1D5EE /* BlockDirectoryBits.h in Headers */ = {isa = PBXBuildFile; fileRef = E38652E2237CA0C800E1D5EE /* BlockDirectoryBits.h */; settings = {ATTRIBUTES = (Private, ); }; };
    18201821                E3893A1D2203A7C600E79A74 /* AsyncFromSyncIteratorPrototype.lut.h in Headers */ = {isa = PBXBuildFile; fileRef = E3893A1C2203A7C600E79A74 /* AsyncFromSyncIteratorPrototype.lut.h */; };
    18211822                E38D999C221B78BB00D50474 /* JSNonDestructibleProxy.h in Headers */ = {isa = PBXBuildFile; fileRef = E38D999A221B789F00D50474 /* JSNonDestructibleProxy.h */; settings = {ATTRIBUTES = (Private, ); }; };
    … …  
    49484949                E380D66B1F19249D00A59095 /* BuiltinNames.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = BuiltinNames.cpp; sourceTree = "<group>"; };
    49494950                E3850B14226ED63E009ABF9C /* DFGMinifiedIDInlines.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; name = DFGMinifiedIDInlines.h; path = dfg/DFGMinifiedIDInlines.h; sourceTree = "<group>"; };
     4951                E38652E2237CA0C800E1D5EE /* BlockDirectoryBits.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = BlockDirectoryBits.h; sourceTree = "<group>"; };
    49504952                E3893A1C2203A7C600E79A74 /* AsyncFromSyncIteratorPrototype.lut.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AsyncFromSyncIteratorPrototype.lut.h; sourceTree = "<group>"; };
    49514953                E38D060B1F8E814100649CF2 /* JSScriptFetchParameters.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JSScriptFetchParameters.h; sourceTree = "<group>"; };
    … …  
    61036105                                C2B916C414DA040C00CBAC86 /* BlockDirectory.cpp */,
    61046106                                C2B916C114DA014E00CBAC86 /* BlockDirectory.h */,
     6107                                E38652E2237CA0C800E1D5EE /* BlockDirectoryBits.h */,
    61056108                                0F7DF1451E2BEF680095951B /* BlockDirectoryInlines.h */,
    61066109                                0F9630351D4192C3005609D9 /* CellAttributes.cpp */,
    … …  
    90099012                                0F64B2721A784BAF006E4E66 /* BinarySwitch.h in Headers */,
    90109013                                C2B916C214DA014E00CBAC86 /* BlockDirectory.h in Headers */,
     9014                                E38652E3237CA0C900E1D5EE /* BlockDirectoryBits.h in Headers */,
    90119015                                0F7DF1461E2BEF6A0095951B /* BlockDirectoryInlines.h in Headers */,
    90129016                                BC18C3EC0E16F5CD00B34460 /* BooleanObject.h in Headers */,

  • trunk/Source/JavaScriptCore/heap/AlignedMemoryAllocator.cpp

    r245258 r252452  
    4141}
    4242
    43 void AlignedMemoryAllocator::registerDirectory(BlockDirectory* directory)
     43void AlignedMemoryAllocator::registerDirectory(Heap& heap, BlockDirectory* directory)
    4444{
    4545    RELEASE_ASSERT(!directory->nextDirectoryInAlignedMemoryAllocator());
    4646   
    4747    if (m_directories.isEmpty()) {
    48         ASSERT(!Thread::mayBeGCThread() || directory->heap()->worldIsStopped());
     48        ASSERT_UNUSED(heap, !Thread::mayBeGCThread() || heap.worldIsStopped());
    4949        for (Subspace* subspace = m_subspaces.first(); subspace; subspace = subspace->nextSubspaceInAlignedMemoryAllocator())
    5050            subspace->didCreateFirstDirectory(directory);

  • trunk/Source/JavaScriptCore/heap/AlignedMemoryAllocator.h

    r243688 r252452  
    3232
    3333class BlockDirectory;
     34class Heap;
    3435class Subspace;
    3536
    … …  
    4647    virtual void dump(PrintStream&) const = 0;
    4748
    48     void registerDirectory(BlockDirectory*);
     49    void registerDirectory(Heap&, BlockDirectory*);
    4950    BlockDirectory* firstDirectory() const { return m_directories.first(); }
    5051

  • trunk/Source/JavaScriptCore/heap/Allocator.h

    r232074 r252452  
    3232
    3333class GCDeferralContext;
     34class Heap;
    3435class LocalAllocator;
    3536
    … …  
    4647    }
    4748   
    48     void* allocate(GCDeferralContext*, AllocationFailureMode) const;
     49    void* allocate(Heap&, GCDeferralContext*, AllocationFailureMode) const;
    4950   
    5051    unsigned cellSize() const;

  • trunk/Source/JavaScriptCore/heap/AllocatorInlines.h

    r232132 r252452  
    3131namespace JSC {
    3232
    33 ALWAYS_INLINE void* Allocator::allocate(GCDeferralContext* context, AllocationFailureMode mode) const
     33ALWAYS_INLINE void* Allocator::allocate(Heap& heap, GCDeferralContext* context, AllocationFailureMode mode) const
    3434{
    35     return m_localAllocator->allocate(context, mode);
     35    return m_localAllocator->allocate(heap, context, mode);
    3636}
    3737

  • trunk/Source/JavaScriptCore/heap/BlockDirectory.cpp

    r244506 r252452  
    3939namespace JSC {
    4040
    41 BlockDirectory::BlockDirectory(Heap* heap, size_t cellSize)
     41BlockDirectory::BlockDirectory(size_t cellSize)
    4242    : m_cellSize(static_cast<unsigned>(cellSize))
    43     , m_heap(heap)
    4443{
    4544}
    … …  
    7776MarkedBlock::Handle* BlockDirectory::findEmptyBlockToSteal()
    7877{
    79     m_emptyCursor = m_empty.findBit(m_emptyCursor, true);
     78    m_emptyCursor = m_bits.empty().findBit(m_emptyCursor, true);
    8079    if (m_emptyCursor >= m_blocks.size())
    8180        return nullptr;
    … …  
    8685{
    8786    for (;;) {
    88         allocator.m_allocationCursor = (m_canAllocateButNotEmpty | m_empty).findBit(allocator.m_allocationCursor, true);
     87        allocator.m_allocationCursor = (m_bits.canAllocateButNotEmpty() | m_bits.empty()).findBit(allocator.m_allocationCursor, true);
    8988        if (allocator.m_allocationCursor >= m_blocks.size())
    9089            return nullptr;
    … …  
    9796}
    9897
    99 MarkedBlock::Handle* BlockDirectory::tryAllocateBlock()
     98MarkedBlock::Handle* BlockDirectory::tryAllocateBlock(Heap& heap)
    10099{
    101100    SuperSamplerScope superSamplerScope(false);
    102101   
    103     MarkedBlock::Handle* handle = MarkedBlock::tryCreate(*m_heap, subspace()->alignedMemoryAllocator());
     102    MarkedBlock::Handle* handle = MarkedBlock::tryCreate(heap, subspace()->alignedMemoryAllocator());
    104103    if (!handle)
    105104        return nullptr;
    … …  
    119118        m_blocks.append(block);
    120119        if (m_blocks.capacity() != oldCapacity) {
    121             forEachBitVector(
    122                 NoLockingNecessary,
    123                 [&] (FastBitVector& vector) {
    124                     ASSERT_UNUSED(vector, vector.numBits() == oldCapacity);
    125                 });
    126            
     120            ASSERT(m_bits.numBits() == oldCapacity);
    127121            ASSERT(m_blocks.capacity() > oldCapacity);
    128122           
    129123            LockHolder locker(m_bitvectorLock);
    130124            subspace()->didResizeBits(m_blocks.capacity());
    131             forEachBitVector(
    132                 locker,
    133                 [&] (FastBitVector& vector) {
    134                     vector.resize(m_blocks.capacity());
    135                 });
     125            m_bits.resize(m_blocks.capacity());
    136126        }
    137127    } else {
    … …  
    143133    forEachBitVector(
    144134        NoLockingNecessary,
    145         [&] (FastBitVector& vector) {
    146             ASSERT_UNUSED(vector, !vector[index]);
     135        [&](auto vectorRef) {
     136            ASSERT_UNUSED(vectorRef, !vectorRef[index]);
    147137        });
    148138
    … …  
    166156    forEachBitVector(
    167157        holdLock(m_bitvectorLock),
    168         [&] (FastBitVector& vector) {
    169             vector[block->index()] = false;
     158        [&](auto vectorRef) {
     159            vectorRef[block->index()] = false;
    170160        });
    171161   
    … …  
    193183    m_emptyCursor = 0;
    194184   
    195     m_eden.clearAll();
     185    m_bits.eden().clearAll();
    196186
    197187    if (UNLIKELY(Options::useImmortalObjects())) {
    … …  
    238228    // collections, so if you survived the last collection you will survive the next one so long
    239229    // as the next one is eden.
    240     m_markingNotEmpty.clearAll();
    241     m_markingRetired.clearAll();
     230    m_bits.markingNotEmpty().clearAll();
     231    m_bits.markingRetired().clearAll();
    242232}
    243233
    244234void BlockDirectory::endMarking()
    245235{
    246     m_allocated.clearAll();
     236    m_bits.allocated().clearAll();
    247237   
    248238    // It's surprising and frustrating to comprehend, but the end-of-marking flip does not need to
    … …  
    250240    // vectors.
    251241   
    252     m_empty = m_live & ~m_markingNotEmpty;
    253     m_canAllocateButNotEmpty = m_live & m_markingNotEmpty & ~m_markingRetired;
     242    m_bits.empty() = m_bits.live() & ~m_bits.markingNotEmpty();
     243    m_bits.canAllocateButNotEmpty() = m_bits.live() & m_bits.markingNotEmpty() & ~m_bits.markingRetired();
    254244
    255245    if (needsDestruction()) {
    … …  
    259249        // we just allocate a block or when we move a block from one size class to another. That doesn't
    260250        // happen here.
    261         m_destructible = m_live;
     251        m_bits.destructible() = m_bits.live();
    262252    }
    263253   
    … …  
    270260void BlockDirectory::snapshotUnsweptForEdenCollection()
    271261{
    272     m_unswept |= m_eden;
     262    m_bits.unswept() |= m_bits.eden();
    273263}
    274264
    275265void BlockDirectory::snapshotUnsweptForFullCollection()
    276266{
    277     m_unswept = m_live;
     267    m_bits.unswept() = m_bits.live();
    278268}
    279269
    280270MarkedBlock::Handle* BlockDirectory::findBlockToSweep()
    281271{
    282     m_unsweptCursor = m_unswept.findBit(m_unsweptCursor, true);
     272    m_unsweptCursor = m_bits.unswept().findBit(m_unsweptCursor, true);
    283273    if (m_unsweptCursor >= m_blocks.size())
    284274        return nullptr;
    … …  
    288278void BlockDirectory::sweep()
    289279{
    290     m_unswept.forEachSetBit(
     280    m_bits.unswept().forEachSetBit(
    291281        [&] (size_t index) {
    292282            MarkedBlock::Handle* block = m_blocks[index];
    … …  
    297287void BlockDirectory::shrink()
    298288{
    299     (m_empty & ~m_destructible).forEachSetBit(
     289    (m_bits.empty() & ~m_bits.destructible()).forEachSetBit(
    300290        [&] (size_t index) {
    301291            markedSpace().freeBlock(m_blocks[index]);
    … …  
    308298        return;
    309299   
    310     if (m_unswept.isEmpty())
     300    if (m_bits.unswept().isEmpty())
    311301        return;
    312302   
    … …  
    330320                return nullptr;
    331321            auto locker = holdLock(m_lock);
    332             m_index = m_directory.m_markingNotEmpty.findBit(m_index, true);
     322            m_index = m_directory.m_bits.markingNotEmpty().findBit(m_index, true);
    333323            if (m_index >= m_directory.m_blocks.size()) {
    334324                m_done = true;
    … …  
    358348    forEachBitVectorWithName(
    359349        NoLockingNecessary,
    360         [&] (FastBitVector&, const char* name) {
     350        [&](auto vectorRef, const char* name) {
     351            UNUSED_PARAM(vectorRef);
    361352            unsigned length = strlen(name);
    362353            maxNameLength = std::max(maxNameLength, length);
    … …  
    365356    forEachBitVectorWithName(
    366357        NoLockingNecessary,
    367         [&] (FastBitVector& vector, const char* name) {
     358        [&](auto vectorRef, const char* name) {
    368359            out.print("    ", name, ": ");
    369360            for (unsigned i = maxNameLength - strlen(name); i--;)
    370361                out.print(" ");
    371             out.print(vector, "\n");
     362            out.print(vectorRef, "\n");
    372363        });
    373364}

  • trunk/Source/JavaScriptCore/heap/BlockDirectory.h

    r240216 r252452  
    2727
    2828#include "AllocationFailureMode.h"
     29#include "BlockDirectoryBits.h"
    2930#include "CellAttributes.h"
    3031#include "FreeList.h"
    … …  
    4546class LLIntOffsetsExtractor;
    4647
    47 #define FOR_EACH_BLOCK_DIRECTORY_BIT(macro) \
    48     macro(live, Live) /* The set of block indices that have actual blocks. */\
    49     macro(empty, Empty) /* The set of all blocks that have no live objects. */ \
    50     macro(allocated, Allocated) /* The set of all blocks that are full of live objects. */\
    51     macro(canAllocateButNotEmpty, CanAllocateButNotEmpty) /* The set of all blocks are neither empty nor retired (i.e. are more than minMarkedBlockUtilization full). */ \
    52     macro(destructible, Destructible) /* The set of all blocks that may have destructors to run. */\
    53     macro(eden, Eden) /* The set of all blocks that have new objects since the last GC. */\
    54     macro(unswept, Unswept) /* The set of all blocks that could be swept by the incremental sweeper. */\
    55     \
    56     /* These are computed during marking. */\
    57     macro(markingNotEmpty, MarkingNotEmpty) /* The set of all blocks that are not empty. */ \
    58     macro(markingRetired, MarkingRetired) /* The set of all blocks that are retired. */
    59 
    60 // FIXME: We defined canAllocateButNotEmpty and empty to be exclusive:
    61 //
    62 //     canAllocateButNotEmpty & empty == 0
    63 //
    64 // Instead of calling it canAllocate and making it inclusive:
    65 //
    66 //     canAllocate & empty == empty
    67 //
    68 // The latter is probably better. I'll leave it to a future bug to fix that, since breathing on
    69 // this code leads to regressions for days, and it's not clear that making this change would
    70 // improve perf since it would not change the collector's behavior, and either way the directory
    71 // has to look at both bitvectors.
    72 // https://bugs.webkit.org/show_bug.cgi?id=162121
    73 
    7448class BlockDirectory {
    7549    WTF_MAKE_NONCOPYABLE(BlockDirectory);
    … …  
    7953
    8054public:
    81     BlockDirectory(Heap*, size_t cellSize);
     55    BlockDirectory(size_t cellSize);
    8256    ~BlockDirectory();
    8357    void setSubspace(Subspace*);
    … …  
    9973    DestructionMode destruction() const { return m_attributes.destruction; }
    10074    HeapCell::Kind cellKind() const { return m_attributes.cellKind; }
    101     Heap* heap() { return m_heap; }
    10275
    10376    bool isFreeListedCell(const void* target);
    … …  
    11689
    11790#define BLOCK_DIRECTORY_BIT_ACCESSORS(lowerBitName, capitalBitName)     \
    118     bool is ## capitalBitName(const AbstractLocker&, size_t index) const { return m_ ## lowerBitName[index]; } \
     91    bool is ## capitalBitName(const AbstractLocker&, size_t index) const { return m_bits.is ## capitalBitName(index); } \
    11992    bool is ## capitalBitName(const AbstractLocker& locker, MarkedBlock::Handle* block) const { return is ## capitalBitName(locker, block->index()); } \
    120     void setIs ## capitalBitName(const AbstractLocker&, size_t index, bool value) { m_ ## lowerBitName[index] = value; } \
     93    void setIs ## capitalBitName(const AbstractLocker&, size_t index, bool value) { m_bits.setIs ## capitalBitName(index, value); } \
    12194    void setIs ## capitalBitName(const AbstractLocker& locker, MarkedBlock::Handle* block, bool value) { setIs ## capitalBitName(locker, block->index(), value); }
    12295    FOR_EACH_BLOCK_DIRECTORY_BIT(BLOCK_DIRECTORY_BIT_ACCESSORS)
    … …  
    127100    {
    128101#define BLOCK_DIRECTORY_BIT_CALLBACK(lowerBitName, capitalBitName) \
    129         func(m_ ## lowerBitName);
     102        func(m_bits.lowerBitName());
    130103        FOR_EACH_BLOCK_DIRECTORY_BIT(BLOCK_DIRECTORY_BIT_CALLBACK);
    131104#undef BLOCK_DIRECTORY_BIT_CALLBACK
    … …  
    136109    {
    137110#define BLOCK_DIRECTORY_BIT_CALLBACK(lowerBitName, capitalBitName) \
    138         func(m_ ## lowerBitName, #capitalBitName);
     111        func(m_bits.lowerBitName(), #capitalBitName);
    139112        FOR_EACH_BLOCK_DIRECTORY_BIT(BLOCK_DIRECTORY_BIT_CALLBACK);
    140113#undef BLOCK_DIRECTORY_BIT_CALLBACK
    … …  
    167140    MarkedBlock::Handle* findBlockForAllocation(LocalAllocator&);
    168141   
    169     MarkedBlock::Handle* tryAllocateBlock();
     142    MarkedBlock::Handle* tryAllocateBlock(Heap&);
    170143   
    171144    Vector<MarkedBlock::Handle*> m_blocks;
    … …  
    174147    // Mutator uses this to guard resizing the bitvectors. Those things in the GC that may run
    175148    // concurrently to the mutator must lock this when accessing the bitvectors.
    176 #define BLOCK_DIRECTORY_BIT_DECLARATION(lowerBitName, capitalBitName) \
    177     FastBitVector m_ ## lowerBitName;
    178     FOR_EACH_BLOCK_DIRECTORY_BIT(BLOCK_DIRECTORY_BIT_DECLARATION)
    179 #undef BLOCK_DIRECTORY_BIT_DECLARATION
     149    BlockDirectoryBits m_bits;
    180150    Lock m_bitvectorLock;
    181151    Lock m_localAllocatorsLock;
    … …  
    191161    // FIXME: All of these should probably be references.
    192162    // https://bugs.webkit.org/show_bug.cgi?id=166988
    193     Heap* m_heap { nullptr };
    194163    Subspace* m_subspace { nullptr };
    195164    BlockDirectory* m_nextDirectory { nullptr };

  • trunk/Source/JavaScriptCore/heap/BlockDirectoryInlines.h

    r227617 r252452  
    3434template <typename Functor> inline void BlockDirectory::forEachBlock(const Functor& functor)
    3535{
    36     m_live.forEachSetBit(
     36    m_bits.live().forEachSetBit(
    3737        [&] (size_t index) {
    3838            functor(m_blocks[index]);
    … …  
    4242template <typename Functor> inline void BlockDirectory::forEachNotEmptyBlock(const Functor& functor)
    4343{
    44     m_markingNotEmpty.forEachSetBit(
     44    m_bits.markingNotEmpty().forEachSetBit(
    4545        [&] (size_t index) {
    4646            functor(m_blocks[index]);

  • trunk/Source/JavaScriptCore/heap/CompleteSubspace.cpp

    r252302 r252452  
    3333#include "LocalAllocatorInlines.h"
    3434#include "MarkedBlockInlines.h"
     35#include "MarkedSpaceInlines.h"
    3536#include "PreventCollectionScope.h"
    3637#include "SubspaceInlines.h"
    … …  
    8081        dataLog("Creating BlockDirectory/LocalAllocator for ", m_name, ", ", attributes(), ", ", sizeClass, ".\n");
    8182   
    82     std::unique_ptr<BlockDirectory> uniqueDirectory = makeUnique<BlockDirectory>(m_space.heap(), sizeClass);
     83    std::unique_ptr<BlockDirectory> uniqueDirectory = makeUnique<BlockDirectory>(sizeClass);
    8384    BlockDirectory* directory = uniqueDirectory.get();
    8485    m_directories.append(WTFMove(uniqueDirectory));
    … …  
    106107   
    107108    directory->setNextDirectoryInSubspace(m_firstDirectory);
    108     m_alignedMemoryAllocator->registerDirectory(directory);
     109    m_alignedMemoryAllocator->registerDirectory(m_space.heap(), directory);
    109110    WTF::storeStoreFence();
    110111    m_firstDirectory = directory;
    … …  
    128129   
    129130    if (Allocator allocator = allocatorFor(size, AllocatorForMode::EnsureAllocator))
    130         return allocator.allocate(deferralContext, AllocationFailureMode::ReturnNull);
     131        return allocator.allocate(vm.heap, deferralContext, AllocationFailureMode::ReturnNull);
    131132   
    132133    if (size <= Options::preciseAllocationCutoff()

  • trunk/Source/JavaScriptCore/heap/CompleteSubspaceInlines.h

    r251690 r252452  
    3737
    3838    if (Allocator allocator = allocatorForNonVirtual(size, AllocatorForMode::AllocatorIfExists))
    39         return allocator.allocate(deferralContext, failureMode);
     39        return allocator.allocate(vm.heap, deferralContext, failureMode);
    4040    return allocateSlow(vm, size, deferralContext, failureMode);
    4141}

  • trunk/Source/JavaScriptCore/heap/IsoCellSet.cpp

    r252298 r252452  
    6262                return nullptr;
    6363            auto locker = holdLock(m_lock);
    64             auto bits = m_directory.m_markingNotEmpty & m_set.m_blocksWithBits;
     64            auto bits = m_directory.m_bits.markingNotEmpty() & m_set.m_blocksWithBits;
    6565            m_index = bits.findBit(m_index, true);
    6666            if (m_index >= m_directory.m_blocks.size()) {

  • trunk/Source/JavaScriptCore/heap/IsoCellSetInlines.h

    r252385 r252452  
    7171{
    7272    BlockDirectory& directory = m_subspace.m_directory;
    73     (directory.m_markingNotEmpty & m_blocksWithBits).forEachSetBit(
     73    (directory.m_bits.markingNotEmpty() & m_blocksWithBits).forEachSetBit(
    7474        [&] (size_t blockIndex) {
    7575            MarkedBlock::Handle* block = directory.m_blocks[blockIndex];

  • trunk/Source/JavaScriptCore/heap/IsoSubspace.cpp

    r252385 r252452  
    3333#include "IsoSubspaceInlines.h"
    3434#include "LocalAllocatorInlines.h"
     35#include "MarkedSpaceInlines.h"
    3536
    3637namespace JSC {
    … …  
    3839IsoSubspace::IsoSubspace(CString name, Heap& heap, HeapCellType* heapCellType, size_t size, uint8_t numberOfLowerTierCells)
    3940    : Subspace(name, heap)
    40     , m_size(size)
    41     , m_directory(&heap, WTF::roundUpToMultipleOf<MarkedBlock::atomSize>(size))
     41    , m_directory(WTF::roundUpToMultipleOf<MarkedBlock::atomSize>(size))
    4242    , m_localAllocator(&m_directory)
    4343    , m_isoAlignedMemoryAllocator(makeUnique<IsoAlignedMemoryAllocator>())
    44     , m_remainingLowerTierCellCount(numberOfLowerTierCells)
    4544{
     45    m_remainingLowerTierCellCount = numberOfLowerTierCells;
     46    ASSERT(WTF::roundUpToMultipleOf<MarkedBlock::atomSize>(size) == cellSize());
    4647    ASSERT(numberOfLowerTierCells <= MarkedBlock::maxNumberOfLowerTierCells);
    4748    m_isIsoSubspace = true;
    … …  
    5152    m_directory.setSubspace(this);
    5253    m_space.addBlockDirectory(locker, &m_directory);
    53     m_alignedMemoryAllocator->registerDirectory(&m_directory);
     54    m_alignedMemoryAllocator->registerDirectory(heap, &m_directory);
    5455    m_firstDirectory = &m_directory;
    5556}
    … …  
    112113    }
    113114    if (m_remainingLowerTierCellCount) {
    114         size_t size = WTF::roundUpToMultipleOf<MarkedSpace::sizeStep>(m_size);
    115         PreciseAllocation* allocation = PreciseAllocation::createForLowerTier(*m_space.heap(), size, this, --m_remainingLowerTierCellCount);
     115        PreciseAllocation* allocation = PreciseAllocation::createForLowerTier(m_space.heap(), cellSize(), this, --m_remainingLowerTierCellCount);
    116116        return revive(allocation);
    117117    }

  • trunk/Source/JavaScriptCore/heap/IsoSubspace.h

    r252385 r252452  
    4141    JS_EXPORT_PRIVATE ~IsoSubspace();
    4242
    43     size_t size() const { return m_size; }
     43    size_t cellSize() { return m_directory.cellSize(); }
    4444
    4545    Allocator allocatorFor(size_t, AllocatorForMode) override;
    … …  
    6464    void didBeginSweepingToFreeList(MarkedBlock::Handle*) override;
    6565   
    66     size_t m_size;
    6766    BlockDirectory m_directory;
    6867    LocalAllocator m_localAllocator;
    … …  
    7069    SentinelLinkedList<PreciseAllocation, PackedRawSentinelNode<PreciseAllocation>> m_lowerTierFreeList;
    7170    SentinelLinkedList<IsoCellSet, PackedRawSentinelNode<IsoCellSet>> m_cellSets;
    72     uint8_t m_remainingLowerTierCellCount { 0 };
    7371};
    7472
    7573ALWAYS_INLINE Allocator IsoSubspace::allocatorForNonVirtual(size_t size, AllocatorForMode)
    7674{
    77     RELEASE_ASSERT(size == this->size());
     75    RELEASE_ASSERT(WTF::roundUpToMultipleOf<MarkedBlock::atomSize>(size) == cellSize());
    7876    return Allocator(&m_localAllocator);
    7977}

  • trunk/Source/JavaScriptCore/heap/IsoSubspaceInlines.h

    r252385 r252452  
    2828namespace JSC {
    2929
    30 ALWAYS_INLINE void* IsoSubspace::allocateNonVirtual(VM&, size_t size, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
     30ALWAYS_INLINE void* IsoSubspace::allocateNonVirtual(VM& vm, size_t size, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
    3131{
    32     RELEASE_ASSERT(size == this->size());
     32    RELEASE_ASSERT(WTF::roundUpToMultipleOf<MarkedBlock::atomSize>(size) == cellSize());
    3333    Allocator allocator = allocatorForNonVirtual(size, AllocatorForMode::MustAlreadyHaveAllocator);
    34     void* result = allocator.allocate(deferralContext, failureMode);
     34    void* result = allocator.allocate(vm.heap, deferralContext, failureMode);
    3535    return result;
    3636}

  • trunk/Source/JavaScriptCore/heap/IsoSubspacePerVM.cpp

    r252385 r252452  
    4242    {
    4343        auto locker = holdLock(m_perVM.m_lock);
    44         m_perVM.m_subspacePerVM.remove(&space().heap()->vm());
     44        m_perVM.m_subspacePerVM.remove(&space().heap().vm());
    4545    }
    4646

  • trunk/Source/JavaScriptCore/heap/LocalAllocator.cpp

    r252298 r252452  
    111111}
    112112
    113 void* LocalAllocator::allocateSlowCase(GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
     113void* LocalAllocator::allocateSlowCase(Heap& heap, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
    114114{
    115115    SuperSamplerScope superSamplerScope(false);
    116     Heap& heap = *m_directory->m_heap;
    117116    ASSERT(heap.vm().currentThreadIsHoldingAPILock());
    118     doTestCollectionsIfNeeded(deferralContext);
     117    doTestCollectionsIfNeeded(heap, deferralContext);
    119118
    120119    ASSERT(!m_directory->markedSpace().isIterating());
    … …  
    130129    // happens when running WebKit tests, which inject a callback into the GC's finalization.
    131130    if (UNLIKELY(m_currentBlock))
    132         return allocate(deferralContext, failureMode);
     131        return allocate(heap, deferralContext, failureMode);
    133132   
    134133    void* result = tryAllocateWithoutCollecting();
    … …  
    143142    }
    144143   
    145     MarkedBlock::Handle* block = m_directory->tryAllocateBlock();
     144    MarkedBlock::Handle* block = m_directory->tryAllocateBlock(heap);
    146145    if (!block) {
    147146        if (failureMode == AllocationFailureMode::Assert)
    … …  
    250249}
    251250
    252 void LocalAllocator::doTestCollectionsIfNeeded(GCDeferralContext* deferralContext)
     251void LocalAllocator::doTestCollectionsIfNeeded(Heap& heap, GCDeferralContext* deferralContext)
    253252{
    254253    if (!Options::slowPathAllocsBetweenGCs())
    … …  
    257256    static unsigned allocationCount = 0;
    258257    if (!allocationCount) {
    259         if (!m_directory->m_heap->isDeferred()) {
     258        if (!heap.isDeferred()) {
    260259            if (deferralContext)
    261260                deferralContext->m_shouldGC = true;
    262261            else
    263                 m_directory->m_heap->collectNow(Sync, CollectionScope::Full);
     262                heap.collectNow(Sync, CollectionScope::Full);
    264263        }
    265264    }

  • trunk/Source/JavaScriptCore/heap/LocalAllocator.h

    r244088 r252452  
    3535class BlockDirectory;
    3636class GCDeferralContext;
     37class Heap;
    3738
    3839class LocalAllocator : public BasicRawSentinelNode<LocalAllocator> {
    … …  
    4344    ~LocalAllocator();
    4445   
    45     void* allocate(GCDeferralContext*, AllocationFailureMode);
     46    void* allocate(Heap&, GCDeferralContext*, AllocationFailureMode);
    4647   
    4748    unsigned cellSize() const { return m_freeList.cellSize(); }
    … …  
    6162   
    6263    void reset();
    63     JS_EXPORT_PRIVATE void* allocateSlowCase(GCDeferralContext*, AllocationFailureMode failureMode);
     64    JS_EXPORT_PRIVATE void* allocateSlowCase(Heap&, GCDeferralContext*, AllocationFailureMode);
    6465    void didConsumeFreeList();
    6566    void* tryAllocateWithoutCollecting();
    6667    void* tryAllocateIn(MarkedBlock::Handle*);
    6768    void* allocateIn(MarkedBlock::Handle*);
    68     ALWAYS_INLINE void doTestCollectionsIfNeeded(GCDeferralContext*);
     69    ALWAYS_INLINE void doTestCollectionsIfNeeded(Heap&, GCDeferralContext*);
    6970
    7071    BlockDirectory* m_directory;

  • trunk/Source/JavaScriptCore/heap/LocalAllocatorInlines.h

    r244088 r252452  
    3131namespace JSC {
    3232
    33 ALWAYS_INLINE void* LocalAllocator::allocate(GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
     33ALWAYS_INLINE void* LocalAllocator::allocate(Heap& heap, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
    3434{
    3535    if (validateDFGDoesGC)
    36         RELEASE_ASSERT(m_directory->heap()->expectDoesGC());
     36        RELEASE_ASSERT(heap.expectDoesGC());
    3737    return m_freeList.allocate(
    3838        [&] () -> HeapCell* {
    39             sanitizeStackForVM(m_directory->heap()->vm());
    40             return static_cast<HeapCell*>(allocateSlowCase(deferralContext, failureMode));
     39            sanitizeStackForVM(heap.vm());
     40            return static_cast<HeapCell*>(allocateSlowCase(heap, deferralContext, failureMode));
    4141        });
    4242}

  • trunk/Source/JavaScriptCore/heap/MarkedBlock.cpp

    r252298 r252452  
    378378    directory()->forEachBitVectorWithName(
    379379        holdLock(directory()->bitvectorLock()),
    380         [&] (FastBitVector& bitvector, const char* name) {
    381             out.print(comma, name, ":", bitvector[index()] ? "YES" : "no");
     380        [&](auto vectorRef, const char* name) {
     381            out.print(comma, name, ":", vectorRef[index()] ? "YES" : "no");
    382382        });
    383383}

  • trunk/Source/JavaScriptCore/heap/MarkedSpace.cpp

    r252385 r252452  
    2828#include "JSCInlines.h"
    2929#include "MarkedBlockInlines.h"
     30#include "MarkedSpaceInlines.h"
    3031#include <wtf/ListDump.h>
    3132
    … …  
    196197
    197198MarkedSpace::MarkedSpace(Heap* heap)
    198     : m_heap(heap)
    199 {
     199{
     200    ASSERT_UNUSED(heap, heap == &this->heap());
    200201    initializeSizeClassForStepSize();
    201202}
    … …  
    235236void MarkedSpace::sweepBlocks()
    236237{
    237     m_heap->sweeper().stopSweeping();
     238    heap().sweeper().stopSweeping();
    238239    forEachDirectory(
    239240        [&] (BlockDirectory& directory) -> IterationStatus {
    … …  
    271272void MarkedSpace::prepareForAllocation()
    272273{
    273     ASSERT(!Thread::mayBeGCThread() || m_heap->worldIsStopped());
     274    ASSERT(!Thread::mayBeGCThread() || heap().worldIsStopped());
    274275    for (Subspace* subspace : m_subspaces)
    275276        subspace->prepareForAllocation();
    … …  
    277278    m_activeWeakSets.takeFrom(m_newActiveWeakSets);
    278279   
    279     if (m_heap->collectionScope() == CollectionScope::Eden)
     280    if (heap().collectionScope() == CollectionScope::Eden)
    280281        m_preciseAllocationsNurseryOffsetForSweep = m_preciseAllocationsNurseryOffset;
    281282    else
    … …  
    299300    m_newActiveWeakSets.forEach(visit);
    300301   
    301     if (m_heap->collectionScope() == CollectionScope::Full)
     302    if (heap().collectionScope() == CollectionScope::Full)
    302303        m_activeWeakSets.forEach(visit);
    303304}
    … …  
    311312    m_newActiveWeakSets.forEach(visit);
    312313   
    313     if (m_heap->collectionScope() == CollectionScope::Full)
     314    if (heap().collectionScope() == CollectionScope::Full)
    314315        m_activeWeakSets.forEach(visit);
    315316}
    … …  
    357358void MarkedSpace::prepareForMarking()
    358359{
    359     if (m_heap->collectionScope() == CollectionScope::Eden)
     360    if (heap().collectionScope() == CollectionScope::Eden)
    360361        m_preciseAllocationsOffsetForThisCollection = m_preciseAllocationsNurseryOffset;
    361362    else
    … …  
    417418void MarkedSpace::beginMarking()
    418419{
    419     if (m_heap->collectionScope() == CollectionScope::Full) {
     420    if (heap().collectionScope() == CollectionScope::Full) {
    420421        forEachDirectory(
    421422            [&] (BlockDirectory& directory) -> IterationStatus {
    … …  
    552553void MarkedSpace::snapshotUnswept()
    553554{
    554     if (m_heap->collectionScope() == CollectionScope::Eden) {
     555    if (heap().collectionScope() == CollectionScope::Eden) {
    555556        forEachDirectory(
    556557            [&] (BlockDirectory& directory) -> IterationStatus {

  • trunk/Source/JavaScriptCore/heap/MarkedSpace.h

    r252385 r252452  
    9696    ~MarkedSpace();
    9797   
    98     Heap* heap() const { return m_heap; }
     98    Heap& heap() const;
    9999   
    100100    void lastChanceToFinalize(); // Must call stopAllocatingForGood first.
    … …  
    214214    PreciseAllocation** m_preciseAllocationsForThisCollectionEnd { nullptr };
    215215
    216     Heap* m_heap;
    217216    size_t m_capacity { 0 };
    218217    HeapVersion m_markingVersion { initialVersion };

  • trunk/Source/JavaScriptCore/heap/MarkedSpaceInlines.h

    r252302 r252452  
    3030
    3131namespace JSC {
     32
     33ALWAYS_INLINE Heap& MarkedSpace::heap() const
     34{
     35    return *bitwise_cast<Heap*>(bitwise_cast<uintptr_t>(this) - OBJECT_OFFSETOF(Heap, m_objectSpace));
     36}
    3237
    3338template<typename Functor> inline void MarkedSpace::forEachLiveCell(HeapIterationScope&, const Functor& functor)

  • trunk/Source/JavaScriptCore/heap/Subspace.cpp

    r252385 r252452  
    3232#include "JSCInlines.h"
    3333#include "MarkedBlockInlines.h"
     34#include "MarkedSpaceInlines.h"
    3435#include "ParallelSourceAdapter.h"
    3536#include "PreventCollectionScope.h"
    … …  
    5051    m_directoryForEmptyAllocation = m_alignedMemoryAllocator->firstDirectory();
    5152
    52     Heap& heap = *m_space.heap();
     53    Heap& heap = m_space.heap();
    5354    heap.objectSpace().m_subspaces.append(this);
    5455    m_alignedMemoryAllocator->registerSubspace(this);

  • trunk/Source/JavaScriptCore/heap/Subspace.h

    r252385 r252452  
    120120
    121121    bool m_isIsoSubspace { false };
     122protected:
     123    uint8_t m_remainingLowerTierCellCount { 0 };
    122124};
    123125

  • trunk/Source/WTF/ChangeLog

    r252345 r252452  
     12019-11-14  Yusuke Suzuki  <ysuzuki@apple.com>
     2
     3        [JSC] BlockDirectory's bits should be compact
     4        https://bugs.webkit.org/show_bug.cgi?id=204149
     5
     6        Reviewed by Robin Morisset.
     7
     8        * wtf/FastBitVector.h:
     9        (WTF::fastBitVectorArrayLength):
     10        (WTF::FastBitVectorImpl::unsafeWords):
     11        (WTF::FastBitVectorImpl::unsafeWords const):
     12        (WTF::FastBitReference::FastBitReference):
     13        (WTF::FastBitReference::operator bool const):
     14        (WTF::FastBitReference::operator=):
     15        (WTF::FastBitVector::at):
     16        (WTF::FastBitVector::operator[]):
     17        (WTF::FastBitVector::BitReference::BitReference): Deleted.
     18        (WTF::FastBitVector::BitReference::operator bool const): Deleted.
     19        (WTF::FastBitVector::BitReference::operator=): Deleted.
     20
    1212019-11-11  Ross Kirsling  <ross.kirsling@sony.com>
    222

  • trunk/Source/WTF/wtf/FastBitVector.h

    r248546 r252452  
    3636class PrintStream;
    3737
    38 inline size_t fastBitVectorArrayLength(size_t numBits) { return (numBits + 31) / 32; }
     38inline constexpr size_t fastBitVectorArrayLength(size_t numBits) { return (numBits + 31) / 32; }
    3939
    4040class FastBitVectorWordView {
    … …  
    422422   
    423423    typename Words::ViewType wordView() const { return m_words.view(); }
     424
     425    Words& unsafeWords() { return m_words; }
     426    const Words& unsafeWords() const { return m_words; }
    424427   
    425428private:
    … …  
    436439    Words m_words;
    437440};
     441
     442class FastBitReference {
     443    WTF_MAKE_FAST_ALLOCATED;
     444public:
     445    FastBitReference() = default;
     446
     447    FastBitReference(uint32_t* word, uint32_t mask)
     448        : m_word(word)
     449        , m_mask(mask)
     450    {
     451    }
     452
     453    explicit operator bool() const
     454    {
     455        return !!(*m_word & m_mask);
     456    }
     457
     458    FastBitReference& operator=(bool value)
     459    {
     460        if (value)
     461            *m_word |= m_mask;
     462        else
     463            *m_word &= ~m_mask;
     464        return *this;
     465    }
     466
     467private:
     468    uint32_t* m_word { nullptr };
     469    uint32_t m_mask { 0 };
     470};
     471
     472
    438473
    439474class FastBitVector : public FastBitVectorImpl<FastBitVectorWordOwner> {
    … …  
    519554    }
    520555   
    521     class BitReference {
    522     public:
    523         BitReference() { }
    524        
    525         BitReference(uint32_t* word, uint32_t mask)
    526             : m_word(word)
    527             , m_mask(mask)
    528         {
    529         }
    530        
    531         explicit operator bool() const
    532         {
    533             return !!(*m_word & m_mask);
    534         }
    535        
    536         BitReference& operator=(bool value)
    537         {
    538             if (value)
    539                 *m_word |= m_mask;
    540             else
    541                 *m_word &= ~m_mask;
    542             return *this;
    543         }
    544        
    545     private:
    546         uint32_t* m_word { nullptr };
    547         uint32_t m_mask { 0 };
    548     };
    549    
    550     BitReference at(size_t index)
     556    FastBitReference at(size_t index)
    551557    {
    552558        ASSERT_WITH_SECURITY_IMPLICATION(index < numBits());
    553         return BitReference(&m_words.word(index >> 5), 1 << (index & 31));
    554     }
    555    
    556     BitReference operator[](size_t index)
     559        return FastBitReference(&m_words.word(index >> 5), 1 << (index & 31));
     560    }
     561   
     562    FastBitReference operator[](size_t index)
    557563    {
    558564        return at(index);
Note: See TracChangeset for help on using the changeset viewer.