⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 243688 in webkit


Ignore:
Timestamp:
Mar 31, 2019, 11:51:11 PM (7 years ago)
Author:
ysuzuki@apple.com
Message:

[JSC] Butterfly allocation from LargeAllocation should try "realloc" behavior if collector thread is not active
​https://bugs.webkit.org/show_bug.cgi?id=196160

Reviewed by Saam Barati.

Source/JavaScriptCore:

"realloc" can be effective in terms of peak/current memory footprint when realloc succeeds because,

  1. It does not allocate additional memory while expanding a vector
  2. It does not deallocate an old memory, just reusing the current memory by expanding, so that memory footprint is tight even before scavenging

We found that we can "realloc" large butterflies in certain conditions are met because,

  1. If it goes to LargeAllocation, this memory region is never reused until GC sweeps it.
  2. Butterflies are owned by owner JSObjects, so we know the lifetime of Butterflies.

This patch attempts to use "realloc" onto butterflies if,

  1. Butterflies are allocated in LargeAllocation kind
  2. Concurrent collector is not active
  3. Butterflies do not have property storage

The condition (2) is required to avoid deallocating butterflies while the concurrent collector looks into it. The condition (3) is
also required to avoid deallocating butterflies while the concurrent compiler looks into it.

We also change LargeAllocation mechanism to using "malloc" and "free" instead of "posix_memalign". This allows us to use "realloc"
safely in all the platforms. Since LargeAllocation uses alignment to distinguish LargeAllocation and MarkedBlock, we manually adjust
16B alignment by allocating 8B more memory in "malloc".

Speedometer2 and JetStream2 are neutral. RAMification shows about 1% progression (even in some of JIT tests).

  • heap/AlignedMemoryAllocator.h:
  • heap/CompleteSubspace.cpp:

(JSC::CompleteSubspace::tryAllocateSlow):
(JSC::CompleteSubspace::reallocateLargeAllocationNonVirtual):

  • heap/CompleteSubspace.h:
  • heap/FastMallocAlignedMemoryAllocator.cpp:

(JSC::FastMallocAlignedMemoryAllocator::tryAllocateMemory):
(JSC::FastMallocAlignedMemoryAllocator::freeMemory):
(JSC::FastMallocAlignedMemoryAllocator::tryReallocateMemory):

  • heap/FastMallocAlignedMemoryAllocator.h:
  • heap/GigacageAlignedMemoryAllocator.cpp:

(JSC::GigacageAlignedMemoryAllocator::tryAllocateMemory):
(JSC::GigacageAlignedMemoryAllocator::freeMemory):
(JSC::GigacageAlignedMemoryAllocator::tryReallocateMemory):

  • heap/GigacageAlignedMemoryAllocator.h:
  • heap/IsoAlignedMemoryAllocator.cpp:

(JSC::IsoAlignedMemoryAllocator::tryAllocateMemory):
(JSC::IsoAlignedMemoryAllocator::freeMemory):
(JSC::IsoAlignedMemoryAllocator::tryReallocateMemory):

  • heap/IsoAlignedMemoryAllocator.h:
  • heap/LargeAllocation.cpp:

(JSC::isAlignedForLargeAllocation):
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::tryReallocate):
(JSC::LargeAllocation::LargeAllocation):
(JSC::LargeAllocation::destroy):

  • heap/LargeAllocation.h:

(JSC::LargeAllocation::indexInSpace):
(JSC::LargeAllocation::setIndexInSpace):
(JSC::LargeAllocation::basePointer const):

  • heap/MarkedSpace.cpp:

(JSC::MarkedSpace::sweepLargeAllocations):
(JSC::MarkedSpace::prepareForConservativeScan):

  • heap/WeakSet.h:

(JSC::WeakSet::isTriviallyDestructible const):

  • runtime/Butterfly.h:
  • runtime/ButterflyInlines.h:

(JSC::Butterfly::reallocArrayRightIfPossible):

  • runtime/JSObject.cpp:

(JSC::JSObject::ensureLengthSlow):

Source/WTF:

  • wtf/FastMalloc.h:

(WTF::FastMalloc::tryRealloc):

  • wtf/Gigacage.cpp:

(Gigacage::tryRealloc):

  • wtf/Gigacage.h:
Location:
trunk/Source
Files:
21 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/JavaScriptCore/ChangeLog

    r243683 r243688  
     12019-03-31  Yusuke Suzuki  <ysuzuki@apple.com>
     2
     3        [JSC] Butterfly allocation from LargeAllocation should try "realloc" behavior if collector thread is not active
     4        https://bugs.webkit.org/show_bug.cgi?id=196160
     5
     6        Reviewed by Saam Barati.
     7
     8        "realloc" can be effective in terms of peak/current memory footprint when realloc succeeds because,
     9
     10        1. It does not allocate additional memory while expanding a vector
     11        2. It does not deallocate an old memory, just reusing the current memory by expanding, so that memory footprint is tight even before scavenging
     12
     13        We found that we can "realloc" large butterflies in certain conditions are met because,
     14
     15        1. If it goes to LargeAllocation, this memory region is never reused until GC sweeps it.
     16        2. Butterflies are owned by owner JSObjects, so we know the lifetime of Butterflies.
     17
     18        This patch attempts to use "realloc" onto butterflies if,
     19
     20        1. Butterflies are allocated in LargeAllocation kind
     21        2. Concurrent collector is not active
     22        3. Butterflies do not have property storage
     23
     24        The condition (2) is required to avoid deallocating butterflies while the concurrent collector looks into it. The condition (3) is
     25        also required to avoid deallocating butterflies while the concurrent compiler looks into it.
     26
     27        We also change LargeAllocation mechanism to using "malloc" and "free" instead of "posix_memalign". This allows us to use "realloc"
     28        safely in all the platforms. Since LargeAllocation uses alignment to distinguish LargeAllocation and MarkedBlock, we manually adjust
     29        16B alignment by allocating 8B more memory in "malloc".
     30
     31        Speedometer2 and JetStream2 are neutral. RAMification shows about 1% progression (even in some of JIT tests).
     32
     33        * heap/AlignedMemoryAllocator.h:
     34        * heap/CompleteSubspace.cpp:
     35        (JSC::CompleteSubspace::tryAllocateSlow):
     36        (JSC::CompleteSubspace::reallocateLargeAllocationNonVirtual):
     37        * heap/CompleteSubspace.h:
     38        * heap/FastMallocAlignedMemoryAllocator.cpp:
     39        (JSC::FastMallocAlignedMemoryAllocator::tryAllocateMemory):
     40        (JSC::FastMallocAlignedMemoryAllocator::freeMemory):
     41        (JSC::FastMallocAlignedMemoryAllocator::tryReallocateMemory):
     42        * heap/FastMallocAlignedMemoryAllocator.h:
     43        * heap/GigacageAlignedMemoryAllocator.cpp:
     44        (JSC::GigacageAlignedMemoryAllocator::tryAllocateMemory):
     45        (JSC::GigacageAlignedMemoryAllocator::freeMemory):
     46        (JSC::GigacageAlignedMemoryAllocator::tryReallocateMemory):
     47        * heap/GigacageAlignedMemoryAllocator.h:
     48        * heap/IsoAlignedMemoryAllocator.cpp:
     49        (JSC::IsoAlignedMemoryAllocator::tryAllocateMemory):
     50        (JSC::IsoAlignedMemoryAllocator::freeMemory):
     51        (JSC::IsoAlignedMemoryAllocator::tryReallocateMemory):
     52        * heap/IsoAlignedMemoryAllocator.h:
     53        * heap/LargeAllocation.cpp:
     54        (JSC::isAlignedForLargeAllocation):
     55        (JSC::LargeAllocation::tryCreate):
     56        (JSC::LargeAllocation::tryReallocate):
     57        (JSC::LargeAllocation::LargeAllocation):
     58        (JSC::LargeAllocation::destroy):
     59        * heap/LargeAllocation.h:
     60        (JSC::LargeAllocation::indexInSpace):
     61        (JSC::LargeAllocation::setIndexInSpace):
     62        (JSC::LargeAllocation::basePointer const):
     63        * heap/MarkedSpace.cpp:
     64        (JSC::MarkedSpace::sweepLargeAllocations):
     65        (JSC::MarkedSpace::prepareForConservativeScan):
     66        * heap/WeakSet.h:
     67        (JSC::WeakSet::isTriviallyDestructible const):
     68        * runtime/Butterfly.h:
     69        * runtime/ButterflyInlines.h:
     70        (JSC::Butterfly::reallocArrayRightIfPossible):
     71        * runtime/JSObject.cpp:
     72        (JSC::JSObject::ensureLengthSlow):
     73
    1742019-03-31  Sam Weinig  <weinig@apple.com>
    275

  • trunk/Source/JavaScriptCore/heap/AlignedMemoryAllocator.h

    r226822 r243688  
    5151    void registerSubspace(Subspace*);
    5252
     53    // Some of derived memory allocators do not have these features because they do not use them.
     54    // For example, IsoAlignedMemoryAllocator does not have "realloc" feature since it never extends / shrinks the allocated memory region.
     55    virtual void* tryAllocateMemory(size_t) = 0;
     56    virtual void freeMemory(void*) = 0;
     57    virtual void* tryReallocateMemory(void*, size_t) = 0;
     58
    5359private:
    5460    SinglyLinkedListWithTail<BlockDirectory> m_directories;

  • trunk/Source/JavaScriptCore/heap/CompleteSubspace.cpp

    r241927 r243688  
    141141   
    142142    size = WTF::roundUpToMultipleOf<MarkedSpace::sizeStep>(size);
    143     LargeAllocation* allocation = LargeAllocation::tryCreate(vm.heap, size, this);
     143    LargeAllocation* allocation = LargeAllocation::tryCreate(vm.heap, size, this, m_space.m_largeAllocations.size());
    144144    if (!allocation)
    145145        return nullptr;
    146146   
    147147    m_space.m_largeAllocations.append(allocation);
     148    ASSERT(allocation->indexInSpace() == m_space.m_largeAllocations.size() - 1);
    148149    vm.heap.didAllocate(size);
    149150    m_space.m_capacity += size;
    … …  
    154155}
    155156
     157void* CompleteSubspace::reallocateLargeAllocationNonVirtual(VM& vm, HeapCell* oldCell, size_t size, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
     158{
     159    if (validateDFGDoesGC)
     160        RELEASE_ASSERT(vm.heap.expectDoesGC());
     161
     162    // The following conditions are met in Butterfly for example.
     163    ASSERT(oldCell->isLargeAllocation());
     164
     165    LargeAllocation* oldAllocation = &oldCell->largeAllocation();
     166    ASSERT(oldAllocation->cellSize() <= size);
     167    ASSERT(oldAllocation->weakSet().isTriviallyDestructible());
     168    ASSERT(oldAllocation->attributes().destruction == DoesNotNeedDestruction);
     169    ASSERT(oldAllocation->attributes().cellKind == HeapCell::Auxiliary);
     170    ASSERT(size > MarkedSpace::largeCutoff);
     171
     172    sanitizeStackForVM(&vm);
     173
     174    if (size <= Options::largeAllocationCutoff()
     175        && size <= MarkedSpace::largeCutoff) {
     176        dataLog("FATAL: attampting to allocate small object using large allocation.\n");
     177        dataLog("Requested allocation size: ", size, "\n");
     178        RELEASE_ASSERT_NOT_REACHED();
     179    }
     180
     181    vm.heap.collectIfNecessaryOrDefer(deferralContext);
     182
     183    size = WTF::roundUpToMultipleOf<MarkedSpace::sizeStep>(size);
     184    size_t difference = size - oldAllocation->cellSize();
     185    unsigned oldIndexInSpace = oldAllocation->indexInSpace();
     186    if (oldAllocation->isOnList())
     187        oldAllocation->remove();
     188
     189    LargeAllocation* allocation = oldAllocation->tryReallocate(size, this);
     190    if (!allocation) {
     191        RELEASE_ASSERT(failureMode != AllocationFailureMode::Assert);
     192        m_largeAllocations.append(oldAllocation);
     193        return nullptr;
     194    }
     195    ASSERT(oldIndexInSpace == allocation->indexInSpace());
     196
     197    m_space.m_largeAllocations[oldIndexInSpace] = allocation;
     198    vm.heap.didAllocate(difference);
     199    m_space.m_capacity += difference;
     200
     201    m_largeAllocations.append(allocation);
     202
     203    return allocation->cell();
     204}
     205
    156206} // namespace JSC
    157207

  • trunk/Source/JavaScriptCore/heap/CompleteSubspace.h

    r232132 r243688  
    4545    void* allocate(VM&, size_t, GCDeferralContext*, AllocationFailureMode) override;
    4646    void* allocateNonVirtual(VM&, size_t, GCDeferralContext*, AllocationFailureMode);
     47    void* reallocateLargeAllocationNonVirtual(VM&, HeapCell*, size_t, GCDeferralContext*, AllocationFailureMode);
    4748   
    4849    static ptrdiff_t offsetOfAllocatorForSizeStep() { return OBJECT_OFFSETOF(CompleteSubspace, m_allocatorForSizeStep); }

  • trunk/Source/JavaScriptCore/heap/FastMallocAlignedMemoryAllocator.cpp

    r220352 r243688  
    5555}
    5656
     57void* FastMallocAlignedMemoryAllocator::tryAllocateMemory(size_t size)
     58{
     59    return FastMalloc::tryMalloc(size);
     60}
     61
     62void FastMallocAlignedMemoryAllocator::freeMemory(void* pointer)
     63{
     64    FastMalloc::free(pointer);
     65}
     66
     67void* FastMallocAlignedMemoryAllocator::tryReallocateMemory(void* pointer, size_t size)
     68{
     69    return FastMalloc::tryRealloc(pointer, size);
     70}
     71
    5772} // namespace JSC
    5873

  • trunk/Source/JavaScriptCore/heap/FastMallocAlignedMemoryAllocator.h

    r220352 r243688  
    3939   
    4040    void dump(PrintStream&) const override;
     41
     42    void* tryAllocateMemory(size_t) override;
     43    void freeMemory(void*) override;
     44    void* tryReallocateMemory(void*, size_t) override;
    4145};
    4246

  • trunk/Source/JavaScriptCore/heap/GigacageAlignedMemoryAllocator.cpp

    r220352 r243688  
    5353}
    5454
     55void* GigacageAlignedMemoryAllocator::tryAllocateMemory(size_t size)
     56{
     57    return Gigacage::tryMalloc(m_kind, size);
     58}
     59
     60void GigacageAlignedMemoryAllocator::freeMemory(void* pointer)
     61{
     62    Gigacage::free(m_kind, pointer);
     63}
     64
     65void* GigacageAlignedMemoryAllocator::tryReallocateMemory(void* pointer, size_t size)
     66{
     67    return Gigacage::tryRealloc(m_kind, pointer, size);
     68}
     69
    5570} // namespace JSC
    5671

  • trunk/Source/JavaScriptCore/heap/GigacageAlignedMemoryAllocator.h

    r220352 r243688  
    4141    void dump(PrintStream&) const override;
    4242
     43    void* tryAllocateMemory(size_t) override;
     44    void freeMemory(void*) override;
     45    void* tryReallocateMemory(void*, size_t) override;
     46
    4347private:
    4448    Gigacage::Kind m_kind;

  • trunk/Source/JavaScriptCore/heap/IsoAlignedMemoryAllocator.cpp

    r230187 r243688  
    8989}
    9090
     91void* IsoAlignedMemoryAllocator::tryAllocateMemory(size_t)
     92{
     93    RELEASE_ASSERT_NOT_REACHED();
     94}
     95
     96void IsoAlignedMemoryAllocator::freeMemory(void*)
     97{
     98    RELEASE_ASSERT_NOT_REACHED();
     99}
     100
     101void* IsoAlignedMemoryAllocator::tryReallocateMemory(void*, size_t)
     102{
     103    RELEASE_ASSERT_NOT_REACHED();
     104}
     105
    91106} // namespace JSC
    92107

  • trunk/Source/JavaScriptCore/heap/IsoAlignedMemoryAllocator.h

    r240216 r243688  
    4040    void dump(PrintStream&) const override;
    4141
     42    void* tryAllocateMemory(size_t) override;
     43    void freeMemory(void*) override;
     44    void* tryReallocateMemory(void*, size_t) override;
     45
    4246private:
    4347    Vector<void*> m_blocks;

  • trunk/Source/JavaScriptCore/heap/LargeAllocation.cpp

    r243667 r243688  
    3535namespace JSC {
    3636
    37 LargeAllocation* LargeAllocation::tryCreate(Heap& heap, size_t size, Subspace* subspace)
     37static inline bool isAlignedForLargeAllocation(void* memory)
     38{
     39    uintptr_t allocatedPointer = bitwise_cast<uintptr_t>(memory);
     40    return !(allocatedPointer & (LargeAllocation::alignment - 1));
     41}
     42
     43LargeAllocation* LargeAllocation::tryCreate(Heap& heap, size_t size, Subspace* subspace, unsigned indexInSpace)
    3844{
    3945    if (validateDFGDoesGC)
    4046        RELEASE_ASSERT(heap.expectDoesGC());
    4147
    42     size_t allocationSize = headerSize() + size;
     48    size_t adjustedAlignmentAllocationSize = headerSize() + size + halfAlignment;
     49    static_assert(halfAlignment == 8, "We assume that memory returned by malloc has alignment >= 8.");
    4350   
    44     void* space = subspace->alignedMemoryAllocator()->tryAllocateAlignedMemory(alignment, allocationSize);
     51    // We must use tryAllocateMemory instead of tryAllocateAlignedMemory since we want to use "realloc" feature.
     52    void* space = subspace->alignedMemoryAllocator()->tryAllocateMemory(adjustedAlignmentAllocationSize);
    4553    if (!space)
    4654        return nullptr;
     55
     56    bool adjustedAlignment = false;
     57    if (!isAlignedForLargeAllocation(space)) {
     58        space = bitwise_cast<void*>(bitwise_cast<uintptr_t>(space) + halfAlignment);
     59        adjustedAlignment = true;
     60        ASSERT(isAlignedForLargeAllocation(space));
     61    }
    4762   
    4863    if (scribbleFreeCells())
    4964        scribble(space, size);
    50     return new (NotNull, space) LargeAllocation(heap, size, subspace);
    51 }
    52 
    53 LargeAllocation::LargeAllocation(Heap& heap, size_t size, Subspace* subspace)
     65    return new (NotNull, space) LargeAllocation(heap, size, subspace, indexInSpace, adjustedAlignment);
     66}
     67
     68LargeAllocation* LargeAllocation::tryReallocate(size_t size, Subspace* subspace)
     69{
     70    size_t adjustedAlignmentAllocationSize = headerSize() + size + halfAlignment;
     71    static_assert(halfAlignment == 8, "We assume that memory returned by malloc has alignment >= 8.");
     72
     73    ASSERT(subspace == m_subspace);
     74
     75    unsigned oldCellSize = m_cellSize;
     76    bool oldAdjustedAlignment = m_adjustedAlignment;
     77    void* oldBasePointer = basePointer();
     78
     79    void* newBasePointer = subspace->alignedMemoryAllocator()->tryReallocateMemory(oldBasePointer, adjustedAlignmentAllocationSize);
     80    if (!newBasePointer)
     81        return nullptr;
     82
     83    LargeAllocation* newAllocation = bitwise_cast<LargeAllocation*>(newBasePointer);
     84    bool newAdjustedAlignment = false;
     85    if (!isAlignedForLargeAllocation(newBasePointer)) {
     86        newAdjustedAlignment = true;
     87        newAllocation = bitwise_cast<LargeAllocation*>(bitwise_cast<uintptr_t>(newBasePointer) + halfAlignment);
     88        ASSERT(isAlignedForLargeAllocation(static_cast<void*>(newAllocation)));
     89    }
     90
     91    // We have 4 patterns.
     92    // oldAdjustedAlignment = true  newAdjustedAlignment = true  => Do nothing.
     93    // oldAdjustedAlignment = true  newAdjustedAlignment = false => Shift forward by halfAlignment
     94    // oldAdjustedAlignment = false newAdjustedAlignment = true  => Shift backward by halfAlignment
     95    // oldAdjustedAlignment = false newAdjustedAlignment = false => Do nothing.
     96
     97    if (oldAdjustedAlignment != newAdjustedAlignment) {
     98        if (oldAdjustedAlignment) {
     99            ASSERT(!newAdjustedAlignment);
     100            ASSERT(newAllocation == newBasePointer);
     101            // Old   [ 8 ][  content  ]
     102            // Now   [   ][  content  ]
     103            // New   [  content  ]...
     104            memmove(newBasePointer, bitwise_cast<char*>(newBasePointer) + halfAlignment, oldCellSize + LargeAllocation::headerSize());
     105        } else {
     106            ASSERT(newAdjustedAlignment);
     107            ASSERT(newAllocation != newBasePointer);
     108            ASSERT(newAllocation == bitwise_cast<void*>(bitwise_cast<char*>(newBasePointer) + halfAlignment));
     109            // Old   [  content  ]
     110            // Now   [  content  ][   ]
     111            // New   [ 8 ][  content  ]
     112            memmove(bitwise_cast<char*>(newBasePointer) + halfAlignment, newBasePointer, oldCellSize + LargeAllocation::headerSize());
     113        }
     114    }
     115
     116    newAllocation->m_cellSize = size;
     117    newAllocation->m_adjustedAlignment = newAdjustedAlignment;
     118    return newAllocation;
     119}
     120
     121LargeAllocation::LargeAllocation(Heap& heap, size_t size, Subspace* subspace, unsigned indexInSpace, bool adjustedAlignment)
    54122    : m_cellSize(size)
     123    , m_indexInSpace(indexInSpace)
    55124    , m_isNewlyAllocated(true)
    56125    , m_hasValidCell(true)
     126    , m_adjustedAlignment(adjustedAlignment)
    57127    , m_attributes(subspace->attributes())
    58128    , m_subspace(subspace)
    … …  
    116186{
    117187    AlignedMemoryAllocator* allocator = m_subspace->alignedMemoryAllocator();
     188    void* basePointer = this->basePointer();
    118189    this->~LargeAllocation();
    119     allocator->freeAlignedMemory(this);
     190    allocator->freeMemory(basePointer);
    120191}
    121192

  • trunk/Source/JavaScriptCore/heap/LargeAllocation.h

    r226822 r243688  
    4040class LargeAllocation : public BasicRawSentinelNode<LargeAllocation> {
    4141public:
    42     static LargeAllocation* tryCreate(Heap&, size_t, Subspace*);
     42    static LargeAllocation* tryCreate(Heap&, size_t, Subspace*, unsigned indexInSpace);
     43
     44    LargeAllocation* tryReallocate(size_t, Subspace*);
    4345   
    4446    ~LargeAllocation();
    … …  
    6668    VM* vm() const { return m_weakSet.vm(); }
    6769    WeakSet& weakSet() { return m_weakSet; }
     70
     71    unsigned indexInSpace() { return m_indexInSpace; }
     72    void setIndexInSpace(unsigned indexInSpace) { m_indexInSpace = indexInSpace; }
    6873   
    6974    void shrink();
    … …  
    141146    void dump(PrintStream&) const;
    142147   
    143 private:
    144     LargeAllocation(Heap&, size_t, Subspace*);
    145    
    146148    static const unsigned alignment = MarkedBlock::atomSize;
    147149    static const unsigned halfAlignment = alignment / 2;
    148150
     151private:
     152    LargeAllocation(Heap&, size_t, Subspace*, unsigned indexInSpace, bool adjustedAlignment);
     153   
    149154    static unsigned headerSize();
     155
     156    void* basePointer() const;
    150157   
    151158    size_t m_cellSize;
    152     bool m_isNewlyAllocated;
    153     bool m_hasValidCell;
     159    unsigned m_indexInSpace { 0 };
     160    bool m_isNewlyAllocated : 1;
     161    bool m_hasValidCell : 1;
     162    bool m_adjustedAlignment : 1;
    154163    Atomic<bool> m_isMarked;
    155164    CellAttributes m_attributes;
    … …  
    163172}
    164173
     174inline void* LargeAllocation::basePointer() const
     175{
     176    if (m_adjustedAlignment)
     177        return bitwise_cast<char*>(this) - halfAlignment;
     178    return bitwise_cast<void*>(this);
     179}
     180
    165181} // namespace JSC
    166182

  • trunk/Source/JavaScriptCore/heap/MarkedSpace.cpp

    r240965 r243688  
    251251            continue;
    252252        }
     253        allocation->setIndexInSpace(dstIndex);
    253254        m_largeAllocations[dstIndex++] = allocation;
    254255    }
    … …  
    328329            return a < b;
    329330        });
     331    unsigned index = m_largeAllocationsOffsetForThisCollection;
     332    for (auto* start = m_largeAllocationsForThisCollectionBegin; start != m_largeAllocationsForThisCollectionEnd; ++start, ++index) {
     333        (*start)->setIndexInSpace(index);
     334        ASSERT(m_largeAllocations[index] == *start);
     335        ASSERT(m_largeAllocations[index]->indexInSpace() == index);
     336    }
    330337}
    331338

  • trunk/Source/JavaScriptCore/heap/WeakSet.h

    r210844 r243688  
    5353
    5454    bool isEmpty() const;
     55    bool isTriviallyDestructible() const;
    5556
    5657    void visit(SlotVisitor&);
    … …  
    9798}
    9899
     100inline bool WeakSet::isTriviallyDestructible() const
     101{
     102    if (!m_blocks.isEmpty())
     103        return false;
     104    if (isOnList())
     105        return false;
     106    return true;
     107}
     108
    99109inline void WeakSet::deallocate(WeakImpl* weakImpl)
    100110{

  • trunk/Source/JavaScriptCore/runtime/Butterfly.h

    r239324 r243688  
    223223    Butterfly* growArrayRight(VM&, JSObject* intendedOwner, Structure* oldStructure, size_t propertyCapacity, bool hadIndexingHeader, size_t oldIndexingPayloadSizeInBytes, size_t newIndexingPayloadSizeInBytes); // Assumes that preCapacity is zero, and asserts as much.
    224224    Butterfly* growArrayRight(VM&, JSObject* intendedOwner, Structure*, size_t newIndexingPayloadSizeInBytes);
     225
     226    Butterfly* reallocArrayRightIfPossible(VM&, GCDeferralContext&, JSObject* intendedOwner, Structure* oldStructure, size_t propertyCapacity, bool hadIndexingHeader, size_t oldIndexingPayloadSizeInBytes, size_t newIndexingPayloadSizeInBytes); // Assumes that preCapacity is zero, and asserts as much.
     227
    225228    Butterfly* resizeArray(VM&, JSObject* intendedOwner, size_t propertyCapacity, bool oldHasIndexingHeader, size_t oldIndexingPayloadSizeInBytes, size_t newPreCapacity, bool newHasIndexingHeader, size_t newIndexingPayloadSizeInBytes);
    226229    Butterfly* resizeArray(VM&, JSObject* intendedOwner, Structure*, size_t newPreCapacity, size_t newIndexingPayloadSizeInBytes); // Assumes that you're not changing whether or not the object has an indexing header.

  • trunk/Source/JavaScriptCore/runtime/ButterflyInlines.h

    r232951 r243688  
    195195}
    196196
     197inline Butterfly* Butterfly::reallocArrayRightIfPossible(
     198    VM& vm, GCDeferralContext& deferralContext, JSObject* intendedOwner, Structure* oldStructure, size_t propertyCapacity,
     199    bool hadIndexingHeader, size_t oldIndexingPayloadSizeInBytes,
     200    size_t newIndexingPayloadSizeInBytes)
     201{
     202    ASSERT_UNUSED(oldStructure, !indexingHeader()->preCapacity(oldStructure));
     203    ASSERT_UNUSED(intendedOwner, hadIndexingHeader == oldStructure->hasIndexingHeader(intendedOwner));
     204
     205    void* theBase = base(0, propertyCapacity);
     206    size_t oldSize = totalSize(0, propertyCapacity, hadIndexingHeader, oldIndexingPayloadSizeInBytes);
     207    size_t newSize = totalSize(0, propertyCapacity, true, newIndexingPayloadSizeInBytes);
     208    ASSERT(newSize >= oldSize);
     209
     210    // We can eagerly destroy butterfly backed by LargeAllocation if (1) concurrent collector is not active and (2) the butterfly does not contain any property storage.
     211    // This is because during deallocation concurrent collector can access butterfly and DFG concurrent compilers accesses properties.
     212    // Objects with no properties are common in arrays, and we are focusing on very large array crafted by repeating Array#push, so... that's fine!
     213    bool canRealloc = !propertyCapacity && !vm.heap.mutatorShouldBeFenced() && bitwise_cast<HeapCell*>(theBase)->isLargeAllocation();
     214    if (canRealloc) {
     215        void* newBase = vm.jsValueGigacageAuxiliarySpace.reallocateLargeAllocationNonVirtual(vm, bitwise_cast<HeapCell*>(theBase), newSize, &deferralContext, AllocationFailureMode::ReturnNull);
     216        if (!newBase)
     217            return nullptr;
     218        return fromBase(newBase, 0, propertyCapacity);
     219    }
     220
     221    void* newBase = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(vm, newSize, &deferralContext, AllocationFailureMode::ReturnNull);
     222    if (!newBase)
     223        return nullptr;
     224    memcpy(newBase, theBase, oldSize);
     225    return fromBase(newBase, 0, propertyCapacity);
     226}
     227
    197228inline Butterfly* Butterfly::resizeArray(
    198229    VM& vm, JSObject* intendedOwner, size_t propertyCapacity, bool oldHasIndexingHeader,

  • trunk/Source/JavaScriptCore/runtime/JSObject.cpp

    r243299 r243688  
    3131#include "ErrorConstructor.h"
    3232#include "Exception.h"
     33#include "GCDeferralContextInlines.h"
    3334#include "GetterSetter.h"
    3435#include "HeapSnapshotBuilder.h"
    … …  
    33583359    unsigned propertyCapacity = structure->outOfLineCapacity();
    33593360   
     3361    GCDeferralContext deferralContext(vm.heap);
     3362    DisallowGC disallowGC;
    33603363    unsigned availableOldLength =
    33613364        Butterfly::availableContiguousVectorLength(propertyCapacity, oldVectorLength);
    … …  
    33693372        newVectorLength = Butterfly::optimalContiguousVectorLength(
    33703373            propertyCapacity, std::min(length * 2, MAX_STORAGE_VECTOR_LENGTH));
    3371         butterfly = butterfly->growArrayRight(
    3372             vm, this, structure, propertyCapacity, true,
     3374        butterfly = butterfly->reallocArrayRightIfPossible(
     3375            vm, deferralContext, this, structure, propertyCapacity, true,
    33733376            oldVectorLength * sizeof(EncodedJSValue),
    33743377            newVectorLength * sizeof(EncodedJSValue));

  • trunk/Source/WTF/ChangeLog

    r243682 r243688  
     12019-03-31  Yusuke Suzuki  <ysuzuki@apple.com>
     2
     3        [JSC] Butterfly allocation from LargeAllocation should try "realloc" behavior if collector thread is not active
     4        https://bugs.webkit.org/show_bug.cgi?id=196160
     5
     6        Reviewed by Saam Barati.
     7
     8        * wtf/FastMalloc.h:
     9        (WTF::FastMalloc::tryRealloc):
     10        * wtf/Gigacage.cpp:
     11        (Gigacage::tryRealloc):
     12        * wtf/Gigacage.h:
     13
    1142019-03-31  Andy Estes  <aestes@apple.com>
    215

  • trunk/Source/WTF/wtf/FastMalloc.h

    r237577 r243688  
    202202   
    203203    static void* realloc(void* p, size_t size) { return fastRealloc(p, size); }
     204
     205    static void* tryRealloc(void* p, size_t size)
     206    {
     207        auto result = tryFastRealloc(p, size);
     208        void* realResult;
     209        if (result.getValue(realResult))
     210            return realResult;
     211        return nullptr;
     212    }
    204213   
    205214    static void free(void* p) { fastFree(p); }

  • trunk/Source/WTF/wtf/Gigacage.cpp

    r240175 r243688  
    4040{
    4141    return FastMalloc::tryMalloc(size);
     42}
     43
     44void* tryRealloc(Kind, void* pointer, size_t size)
     45{
     46    return FastMalloc::tryRealloc(pointer, size);
    4247}
    4348
    … …  
    9095{
    9196    void* result = bmalloc::api::tryMalloc(size, bmalloc::heapKind(kind));
     97    WTF::compilerFence();
     98    return result;
     99}
     100
     101void* tryRealloc(Kind kind, void* pointer, size_t size)
     102{
     103    void* result = bmalloc::api::tryRealloc(pointer, size, bmalloc::heapKind(kind));
    92104    WTF::compilerFence();
    93105    return result;

  • trunk/Source/WTF/wtf/Gigacage.h

    r240175 r243688  
    121121inline void alignedFree(Kind, void* p) { fastAlignedFree(p); }
    122122WTF_EXPORT_PRIVATE void* tryMalloc(Kind, size_t size);
     123WTF_EXPORT_PRIVATE void* tryRealloc(Kind, void*, size_t);
    123124inline void free(Kind, void* p) { fastFree(p); }
    124125
    … …  
    135136WTF_EXPORT_PRIVATE void alignedFree(Kind, void*);
    136137WTF_EXPORT_PRIVATE void* tryMalloc(Kind, size_t);
     138WTF_EXPORT_PRIVATE void* tryRealloc(Kind, void*, size_t);
    137139WTF_EXPORT_PRIVATE void free(Kind, void*);
    138140
Note: See TracChangeset for help on using the changeset viewer.