⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 278278 in webkit


Ignore:
Timestamp:
May 31, 2021, 8:04:24 AM (5 years ago)
Author:
msaboff@apple.com
Message:

[bmalloc] Make adaptive scavenging more precise
​https://bugs.webkit.org/show_bug.cgi?id=226237

Reviewed by Geoffrey Garen.

This patch re-enables the adaptive scavenger for macOS.
It is much more precise when calling madvise() by keeping track of the
maximum extent of physically mapped memory in a LargeRange. For example,
in the prior code, we'd return a small range back to a Gigacage LargeRange
whithout any physical pages. When we scavenge that LargeRange, we madvise()
the whole Gigacage range. Although this didn't cause correctness issues,
it was quite wasteful.

In the tests I did with this patch compared to the earlier adaptive change,
the number of madvise calls we made drops by ~18% and the time spent in those
calls dropped ~13% on an Apple silicon mac and ~30% on an x86 mac.
This was measured using dtrace running JetStream2 from the command line.
RAMification results improved ~1% over the adptive change.

There is the possible future optimization where we also keep track of the
first address of physically mapped memory in a LargeRange. Since bmalloc
allocates memory from lower addresses first, it is thought that the change
in this patch is sufficient to reduce not only the number of madvise calls,
but the time it takes to make those calls.

  • bmalloc/BPlatform.h:
  • bmalloc/Heap.cpp:

(bmalloc::Heap::decommitLargeRange):
(bmalloc::Heap::scavenge):
(bmalloc::Heap::allocateSmallChunk):
(bmalloc::Heap::deallocateSmallChunk):
(bmalloc::Heap::allocateSmallPage):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::allocateLarge):
(bmalloc::Heap::tryAllocateLargeChunk):
(bmalloc::Heap::shrinkLarge):
(bmalloc::Heap::deallocateLarge):
(bmalloc::Heap::scavengeToHighWatermark): Deleted.

  • bmalloc/Heap.h:
  • bmalloc/IsoDirectory.h:
  • bmalloc/IsoDirectoryInlines.h:

(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::scavenge):
(bmalloc::passedNumPages>::scavengeToHighWatermark): Deleted.

  • bmalloc/IsoHeapImpl.h:
  • bmalloc/IsoHeapImplInlines.h:

(bmalloc::IsoHeapImpl<Config>::scavengeToHighWatermark): Deleted.

  • bmalloc/LargeMap.cpp:

(bmalloc::LargeMap::add):

  • bmalloc/LargeRange.h:

(bmalloc::LargeRange::LargeRange):
(bmalloc::LargeRange::physicalExtent const):
(bmalloc::LargeRange::setPhysicalExtent):
(bmalloc::LargeRange::resetPhysicalExtent):
(bmalloc::LargeRange::setUsedSinceLastScavenge):
(bmalloc::merge):
(bmalloc::LargeRange::split const):
(): Deleted.

  • bmalloc/Scavenger.cpp:

(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::scavenge):
(bmalloc::Scavenger::threadRunLoop):
(bmalloc::Scavenger::timeSinceLastPartialScavenge): Deleted.
(bmalloc::Scavenger::partialScavenge): Deleted.

  • bmalloc/Scavenger.h:
  • bmalloc/SmallPage.h:

(bmalloc::SmallPage::setUsedSinceLastScavenge):

Location:
trunk/Source/bmalloc
Files:
14 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/bmalloc/ChangeLog

    r277898 r278278  
     12021-05-31  Michael Saboff  <msaboff@apple.com>
     2
     3        [bmalloc] Make adaptive scavenging more precise
     4        https://bugs.webkit.org/show_bug.cgi?id=226237
     5
     6        Reviewed by Geoffrey Garen.
     7
     8        This patch re-enables the adaptive scavenger for macOS.
     9        It is much more precise when calling madvise() by keeping track of the
     10        maximum extent of physically mapped memory in a LargeRange.  For example,
     11        in the prior code, we'd return a small range back to a Gigacage LargeRange
     12        whithout any physical pages.  When we scavenge that LargeRange, we madvise()
     13        the whole Gigacage range.  Although this didn't cause correctness issues,
     14        it was quite wasteful.
     15
     16        In the tests I did with this patch compared to the earlier adaptive change,
     17        the number of madvise calls we made drops by ~18% and the time spent in those
     18        calls dropped ~13% on an Apple silicon mac and ~30% on an x86 mac.
     19        This was measured using dtrace running JetStream2 from the command line.
     20        RAMification results improved ~1% over the adptive change.
     21
     22        There is the possible future optimization where we also keep track of the
     23        first address of physically mapped memory in a LargeRange.  Since bmalloc
     24        allocates memory from lower addresses first, it is thought that the change
     25        in this patch is sufficient to reduce not only the number of madvise calls,
     26        but the time it takes to make those calls.
     27
     28        * bmalloc/BPlatform.h:
     29        * bmalloc/Heap.cpp:
     30        (bmalloc::Heap::decommitLargeRange):
     31        (bmalloc::Heap::scavenge):
     32        (bmalloc::Heap::allocateSmallChunk):
     33        (bmalloc::Heap::deallocateSmallChunk):
     34        (bmalloc::Heap::allocateSmallPage):
     35        (bmalloc::Heap::splitAndAllocate):
     36        (bmalloc::Heap::allocateLarge):
     37        (bmalloc::Heap::tryAllocateLargeChunk):
     38        (bmalloc::Heap::shrinkLarge):
     39        (bmalloc::Heap::deallocateLarge):
     40        (bmalloc::Heap::scavengeToHighWatermark): Deleted.
     41        * bmalloc/Heap.h:
     42        * bmalloc/IsoDirectory.h:
     43        * bmalloc/IsoDirectoryInlines.h:
     44        (bmalloc::passedNumPages>::takeFirstEligible):
     45        (bmalloc::passedNumPages>::scavenge):
     46        (bmalloc::passedNumPages>::scavengeToHighWatermark): Deleted.
     47        * bmalloc/IsoHeapImpl.h:
     48        * bmalloc/IsoHeapImplInlines.h:
     49        (bmalloc::IsoHeapImpl<Config>::scavengeToHighWatermark): Deleted.
     50        * bmalloc/LargeMap.cpp:
     51        (bmalloc::LargeMap::add):
     52        * bmalloc/LargeRange.h:
     53        (bmalloc::LargeRange::LargeRange):
     54        (bmalloc::LargeRange::physicalExtent const):
     55        (bmalloc::LargeRange::setPhysicalExtent):
     56        (bmalloc::LargeRange::resetPhysicalExtent):
     57        (bmalloc::LargeRange::setUsedSinceLastScavenge):
     58        (bmalloc::merge):
     59        (bmalloc::LargeRange::split const):
     60        (): Deleted.
     61        * bmalloc/Scavenger.cpp:
     62        (bmalloc::Scavenger::Scavenger):
     63        (bmalloc::Scavenger::scavenge):
     64        (bmalloc::Scavenger::threadRunLoop):
     65        (bmalloc::Scavenger::timeSinceLastPartialScavenge): Deleted.
     66        (bmalloc::Scavenger::partialScavenge): Deleted.
     67        * bmalloc/Scavenger.h:
     68        * bmalloc/SmallPage.h:
     69        (bmalloc::SmallPage::setUsedSinceLastScavenge):
     70
    1712021-05-21  Michael Saboff  <msaboff@apple.com>
    272

  • trunk/Source/bmalloc/bmalloc/BPlatform.h

    r277898 r278278  
    310310#define ENABLE_PHYSICAL_PAGE_MAP 0
    311311
    312 #if BPLATFORM(MAC)
    313 #define BUSE_PARTIAL_SCAVENGE 1
    314 #else
    315 #define BUSE_PARTIAL_SCAVENGE 0
    316 #endif
    317 
    318312#if !defined(BUSE_PRECOMPUTED_CONSTANTS_VMPAGE4K)
    319313#define BUSE_PRECOMPUTED_CONSTANTS_VMPAGE4K 1

  • trunk/Source/bmalloc/bmalloc/Heap.cpp

    r277898 r278278  
    6666        ptrdiff_t offset = roundDownToMultipleOf(vmPageSize(), random[1] % (gigacageSize - size));
    6767        void* base = reinterpret_cast<unsigned char*>(gigacageBasePtr) + offset;
    68         m_largeFree.add(LargeRange(base, size, 0, 0));
     68        m_largeFree.add(LargeRange(base, size, 0, 0, base));
    6969    }
    7070#endif
    … …  
    109109    m_footprint -= range.totalPhysicalSize();
    110110    m_freeableMemory -= range.totalPhysicalSize();
    111     decommitter.addLazy(range.begin(), range.size());
    112     m_hasPendingDecommits = true;
     111    if (range.totalPhysicalSize()) {
     112        decommitter.addLazy(range.begin(), range.physicalEnd() - range.begin());
     113        m_hasPendingDecommits = true;
     114    }
    113115    range.setStartPhysicalSize(0);
    114116    range.setTotalPhysicalSize(0);
     117    range.clearPhysicalEnd();
    115118    BASSERT(range.isEligibile());
    116119    range.setEligible(false);
    … …  
    120123}
    121124
    122 #if BUSE(PARTIAL_SCAVENGE)
    123 void Heap::scavenge(UniqueLockHolder& lock, BulkDecommit& decommitter)
    124 #else
    125125void Heap::scavenge(UniqueLockHolder& lock, BulkDecommit& decommitter, size_t& deferredDecommits)
    126 #endif
    127126{
    128127    for (auto& list : m_freePages) {
    … …  
    131130                if (!page->hasPhysicalPages())
    132131                    continue;
    133 #if !BUSE(PARTIAL_SCAVENGE)
    134132                if (page->usedSinceLastScavenge()) {
    135133                    page->clearUsedSinceLastScavenge();
    … …  
    137135                    continue;
    138136                }
    139 #endif
    140137
    141138                size_t pageSize = bmalloc::pageSize(&list - &m_freePages[0]);
    … …  
    158155
    159156    for (LargeRange& range : m_largeFree) {
    160 #if BUSE(PARTIAL_SCAVENGE)
    161         m_highWatermark = std::min(m_highWatermark, static_cast<void*>(range.begin()));
    162 #else
    163157        if (range.usedSinceLastScavenge()) {
    164158            range.clearUsedSinceLastScavenge();
    … …  
    166160            continue;
    167161        }
    168 #endif
    169162        decommitLargeRange(lock, range, decommitter);
    170163    }
    171 
    172 #if BUSE(PARTIAL_SCAVENGE)
    173     m_freeableMemory = 0;
    174 #endif
    175 }
    176 
    177 #if BUSE(PARTIAL_SCAVENGE)
    178 void Heap::scavengeToHighWatermark(UniqueLockHolder& lock, BulkDecommit& decommitter)
    179 {
    180     void* newHighWaterMark = nullptr;
    181     for (LargeRange& range : m_largeFree) {
    182         if (range.begin() <= m_highWatermark)
    183             newHighWaterMark = std::min(newHighWaterMark, static_cast<void*>(range.begin()));
    184         else
    185             decommitLargeRange(lock, range, decommitter);
    186     }
    187     m_highWatermark = newHighWaterMark;
    188 }
    189 #endif
     164}
    190165
    191166void Heap::deallocateLineCache(UniqueLockHolder&, LineCache& lineCache)
    … …  
    219194        m_objectTypes.set(lock, chunk, ObjectType::Small);
    220195
    221         size_t accountedInFreeable = 0;
    222196        forEachPage(chunk, pageSize, [&](SmallPage* page) {
    223197            page->setHasPhysicalPages(true);
    224 #if !BUSE(PARTIAL_SCAVENGE)
    225198            page->setUsedSinceLastScavenge();
    226 #endif
    227199            page->setHasFreeLines(lock, true);
    228200            chunk->freePages().push(page);
    229             accountedInFreeable += pageSize;
    230201        });
    231202
    232         m_freeableMemory += accountedInFreeable;
    233 
    234         auto metadataSize = Chunk::metadataSize(pageSize);
    235         vmDeallocatePhysicalPagesSloppy(chunk->address(sizeof(Chunk)), metadataSize - sizeof(Chunk));
    236 
    237         auto decommitSize = chunkSize - metadataSize - accountedInFreeable;
    238         if (decommitSize > 0)
    239             vmDeallocatePhysicalPagesSloppy(chunk->address(chunkSize - decommitSize), decommitSize);
     203        m_freeableMemory += chunkSize;
    240204
    241205        m_scavenger->schedule(0);
    … …  
    254218    size_t size = m_largeAllocated.remove(chunk);
    255219    size_t totalPhysicalSize = size;
    256 
    257     size_t accountedInFreeable = 0;
    258 
    259     bool hasPhysicalPages = true;
    260     forEachPage(chunk, pageSize(pageClass), [&](SmallPage* page) {
     220    size_t chunkPageSize = pageSize(pageClass);
     221    SmallPage* firstPageWithoutPhysicalPages = nullptr;
     222
     223    void* physicalEnd = chunk->address(chunk->metadataSize(chunkPageSize));
     224    forEachPage(chunk, chunkPageSize, [&](SmallPage* page) {
    261225        size_t physicalSize = physicalPageSizeSloppy(page->begin()->begin(), pageSize(pageClass));
    262226        if (!page->hasPhysicalPages()) {
    263227            totalPhysicalSize -= physicalSize;
    264             hasPhysicalPages = false;
     228            if (!firstPageWithoutPhysicalPages)
     229                firstPageWithoutPhysicalPages = page;
    265230        } else
    266             accountedInFreeable += physicalSize;
     231            physicalEnd = page->begin()->begin() + physicalSize;
    267232    });
    268233
    269     m_freeableMemory -= accountedInFreeable;
    270     m_freeableMemory += totalPhysicalSize;
    271 
    272     size_t startPhysicalSize = hasPhysicalPages ? size : 0;
    273     m_largeFree.add(LargeRange(chunk, size, startPhysicalSize, totalPhysicalSize));
     234    size_t startPhysicalSize = firstPageWithoutPhysicalPages ? firstPageWithoutPhysicalPages->begin()->begin() - chunk->bytes() : size;
     235
     236    m_largeFree.add(LargeRange(chunk, size, startPhysicalSize, totalPhysicalSize, chunk->address(startPhysicalSize)));
    274237}
    275238
    … …  
    284247        return m_lineCache[sizeClass].popFront();
    285248
    286     m_scavenger->didStartGrowing();
    287    
    288249    SmallPage* page = [&]() -> SmallPage* {
    289250        size_t pageClass = m_constants.pageClass(sizeClass);
    … …  
    315276#endif
    316277        }
    317 #if !BUSE(PARTIAL_SCAVENGE)
    318278        page->setUsedSinceLastScavenge();
    319 #endif
    320279
    321280        return page;
    … …  
    526485        range.setStartPhysicalSize(range.size());
    527486        range.setTotalPhysicalSize(range.size());
     487        range.setPhysicalEnd(range.begin() + range.size());
    528488#if ENABLE_PHYSICAL_PAGE_MAP
    529489        m_physicalPageMap.commit(range.begin(), range.size());
    … …  
    561521    BASSERT(isPowerOfTwo(alignment));
    562522   
    563     m_scavenger->didStartGrowing();
    564    
    565523    size_t roundedSize = size ? roundUpToMultipleOf(largeAlignment, size) : largeAlignment;
    566524    ASSERT_OR_RETURN_ON_FAILURE(roundedSize >= size); // Check for overflow
    … …  
    591549
    592550    void* result = splitAndAllocate(lock, range, alignment, size).begin();
    593 #if BUSE(PARTIAL_SCAVENGE)
    594     m_highWatermark = std::max(m_highWatermark, result);
    595 #endif
    596551    ASSERT_OR_RETURN_ON_FAILURE(result);
    597552    return result;
    … …  
    622577#endif
    623578
    624     return LargeRange(memory, size, 0, 0);
     579    return LargeRange(memory, size, 0, 0, memory);
    625580}
    626581
    … …  
    635590
    636591    size_t size = m_largeAllocated.remove(object.begin());
    637     LargeRange range = LargeRange(object, size, size);
     592    LargeRange range = LargeRange(object, size, size, object.begin() + size);
    638593    splitAndAllocate(lock, range, alignment, newSize);
    639594
    … …  
    644599{
    645600    size_t size = m_largeAllocated.remove(object);
    646     m_largeFree.add(LargeRange(object, size, size, size));
     601    m_largeFree.add(LargeRange(object, size, size, size, static_cast<char*>(object) + size));
    647602    m_freeableMemory += size;
    648603    m_scavenger->schedule(size);

  • trunk/Source/bmalloc/bmalloc/Heap.h

    r277898 r278278  
    7575    void shrinkLarge(UniqueLockHolder&, const Range&, size_t);
    7676
    77 #if BUSE(PARTIAL_SCAVENGE)
    78     void scavengeToHighWatermark(UniqueLockHolder&, BulkDecommit&);
    79     void scavenge(UniqueLockHolder&, BulkDecommit&);
    80 #else
    8177    void scavenge(UniqueLockHolder&, BulkDecommit&, size_t& deferredDecommits);
    82 #endif
    8378    void scavenge(UniqueLockHolder&, BulkDecommit&, size_t& freed, size_t goal);
    8479
    … …  
    148143    PhysicalPageMap m_physicalPageMap;
    149144#endif
    150    
    151 #if BUSE(PARTIAL_SCAVENGE)
    152     void* m_highWatermark { nullptr };
    153 #endif
    154145};
    155146

  • trunk/Source/bmalloc/bmalloc/IsoDirectory.h

    r277898 r278278  
    7777    // pages as being decommitted. It's the caller's job to do the actual decommitting.
    7878    void scavenge(const LockHolder&, Vector<DeferredDecommit>&);
    79 #if BUSE(PARTIAL_SCAVENGE)
    80     void scavengeToHighWatermark(const LockHolder&, Vector<DeferredDecommit>&);
    81 #endif
    8279
    8380    template<typename Func>
    … …  
    9491    Bits<numPages> m_committed;
    9592    unsigned m_firstEligibleOrDecommitted { 0 };
    96 #if BUSE(PARTIAL_SCAVENGE)
    97     unsigned m_highWatermark { 0 };
    98 #endif
    9993};
    10094

  • trunk/Source/bmalloc/bmalloc/IsoDirectoryInlines.h

    r277898 r278278  
    5151        return EligibilityKind::Full;
    5252
    53 #if BUSE(PARTIAL_SCAVENGE)
    54     m_highWatermark = std::max(pageIndex, m_highWatermark);
    55 #endif
    56 
    5753    Scavenger& scavenger = *Scavenger::get();
    58     scavenger.didStartGrowing();
    5954   
    6055    IsoPage<Config>* page = m_pages[pageIndex].get();
    … …  
    147142            scavengePage(locker, index, decommits);
    148143        });
    149 #if BUSE(PARTIAL_SCAVENGE)
    150     m_highWatermark = 0;
    151 #endif
    152144}
    153 
    154 #if BUSE(PARTIAL_SCAVENGE)
    155 template<typename Config, unsigned passedNumPages>
    156 void IsoDirectory<Config, passedNumPages>::scavengeToHighWatermark(const LockHolder& locker, Vector<DeferredDecommit>& decommits)
    157 {
    158     (m_empty & m_committed).forEachSetBit(
    159         [&] (size_t index) {
    160             if (index > m_highWatermark)
    161                 scavengePage(locker, index, decommits);
    162         });
    163     m_highWatermark = 0;
    164 }
    165 #endif
    166145
    167146template<typename Config, unsigned passedNumPages>

  • trunk/Source/bmalloc/bmalloc/IsoHeapImpl.h

    r277898 r278278  
    5050   
    5151    virtual void scavenge(Vector<DeferredDecommit>&) = 0;
    52 #if BUSE(PARTIAL_SCAVENGE)
    53     virtual void scavengeToHighWatermark(Vector<DeferredDecommit>&) = 0;
    54 #endif
    5552   
    5653    void scavengeNow();
    … …  
    113110   
    114111    void scavenge(Vector<DeferredDecommit>&) override;
    115 #if BUSE(PARTIAL_SCAVENGE)
    116     void scavengeToHighWatermark(Vector<DeferredDecommit>&) override;
    117 #endif
    118112
    119113    unsigned allocatorOffset();

  • trunk/Source/bmalloc/bmalloc/IsoHeapImplInlines.h

    r277898 r278278  
    122122}
    123123
    124 #if BUSE(PARTIAL_SCAVENGE)
    125 template<typename Config>
    126 void IsoHeapImpl<Config>::scavengeToHighWatermark(Vector<DeferredDecommit>& decommits)
    127 {
    128     LockHolder locker(this->lock);
    129     if (!m_directoryHighWatermark)
    130         m_inlineDirectory.scavengeToHighWatermark(locker, decommits);
    131     for (IsoDirectoryPage<Config>* page = m_headDirectory.get(); page; page = page->next) {
    132         if (page->index() >= m_directoryHighWatermark)
    133             page->payload.scavengeToHighWatermark(locker, decommits);
    134     }
    135     m_directoryHighWatermark = 0;
    136 }
    137 #endif
    138 
    139124inline size_t IsoHeapImplBase::freeableMemory()
    140125{

  • trunk/Source/bmalloc/bmalloc/IsoSharedHeapInlines.h

    r254781 r278278  
    6464{
    6565    Scavenger& scavenger = *Scavenger::get();
    66     scavenger.didStartGrowing();
    6766    scavenger.scheduleIfUnderMemoryPressure(IsoSharedPage::pageSize);
    6867

  • trunk/Source/bmalloc/bmalloc/LargeMap.cpp

    r277898 r278278  
    7777    }
    7878
    79 #if !BUSE(PARTIAL_SCAVENGE)
    8079    merged.setUsedSinceLastScavenge();
    81 #endif
    8280    m_free.push(merged);
    8381}

  • trunk/Source/bmalloc/bmalloc/LargeRange.h

    r277898 r278278  
    3838        , m_startPhysicalSize(0)
    3939        , m_totalPhysicalSize(0)
    40 #if !BUSE(PARTIAL_SCAVENGE)
     40        , m_physicalEnd(begin())
    4141        , m_isEligible(true)
    4242        , m_usedSinceLastScavenge(false)
    43 #endif
    4443    {
    4544    }
    4645
    47     LargeRange(const Range& other, size_t startPhysicalSize, size_t totalPhysicalSize)
     46    LargeRange(const Range& other, size_t startPhysicalSize, size_t totalPhysicalSize, void* physicalEnd)
    4847        : Range(other)
    4948        , m_startPhysicalSize(startPhysicalSize)
    5049        , m_totalPhysicalSize(totalPhysicalSize)
    51 #if !BUSE(PARTIAL_SCAVENGE)
     50        , m_physicalEnd(static_cast<char*>(physicalEnd))
    5251        , m_isEligible(true)
    5352        , m_usedSinceLastScavenge(false)
    54 #endif
    5553    {
    5654        BASSERT(this->size() >= this->totalPhysicalSize());
    … …  
    5856    }
    5957
    60 #if BUSE(PARTIAL_SCAVENGE)
    61     LargeRange(void* begin, size_t size, size_t startPhysicalSize, size_t totalPhysicalSize)
     58    LargeRange(void* begin, size_t size, size_t startPhysicalSize, size_t totalPhysicalSize, void* physicalEnd, bool usedSinceLastScavenge = false)
    6259        : Range(begin, size)
    6360        , m_startPhysicalSize(startPhysicalSize)
    6461        , m_totalPhysicalSize(totalPhysicalSize)
    65     {
    66         BASSERT(this->size() >= this->totalPhysicalSize());
    67         BASSERT(this->totalPhysicalSize() >= this->startPhysicalSize());
    68     }
    69 #else
    70     LargeRange(void* begin, size_t size, size_t startPhysicalSize, size_t totalPhysicalSize, bool usedSinceLastScavenge = false)
    71         : Range(begin, size)
    72         , m_startPhysicalSize(startPhysicalSize)
    73         , m_totalPhysicalSize(totalPhysicalSize)
     62        , m_physicalEnd(static_cast<char*>(physicalEnd))
    7463        , m_isEligible(true)
    7564        , m_usedSinceLastScavenge(usedSinceLastScavenge)
    … …  
    7867        BASSERT(this->totalPhysicalSize() >= this->startPhysicalSize());
    7968    }
    80 #endif
    8169
    8270    // Returns a lower bound on physical size at the start of the range. Ranges that
    … …  
    9987    size_t totalPhysicalSize() const { return m_totalPhysicalSize; }
    10088    void setTotalPhysicalSize(size_t totalPhysicalSize) { m_totalPhysicalSize = totalPhysicalSize; }
     89   
     90    // This is the address past the end of physical memory in this range.
     91    // When decomitting this range, we decommitt [begin(), physicalEnd).
     92    char* physicalEnd() const { return m_physicalEnd; }
     93    void setPhysicalEnd(void* physicalEnd) { m_physicalEnd = static_cast<char*>(physicalEnd); }
     94    void clearPhysicalEnd() { m_physicalEnd = begin(); }
    10195
    10296    std::pair<LargeRange, LargeRange> split(size_t) const;
    … …  
    10599    bool isEligibile() const { return m_isEligible; }
    106100
    107 #if !BUSE(PARTIAL_SCAVENGE)
    108101    bool usedSinceLastScavenge() const { return m_usedSinceLastScavenge; }
    109102    void clearUsedSinceLastScavenge() { m_usedSinceLastScavenge = false; }
    110103    void setUsedSinceLastScavenge() { m_usedSinceLastScavenge = true; }
    111 #endif
    112104
    113105    bool operator<(const void* other) const { return begin() < other; }
    … …  
    117109    size_t m_startPhysicalSize;
    118110    size_t m_totalPhysicalSize;
    119 #if BUSE(PARTIAL_SCAVENGE)
    120     bool m_isEligible { true };
    121 #else
     111    char* m_physicalEnd;
    122112    unsigned m_isEligible: 1;
    123113    unsigned m_usedSinceLastScavenge: 1;
    124 #endif
    125114};
    126115
    … …  
    145134{
    146135    const LargeRange& left = std::min(a, b);
    147 #if !BUSE(PARTIAL_SCAVENGE)
     136    const LargeRange& right = std::max(a, b);
     137    void* physicalEnd = right.totalPhysicalSize() ? right.physicalEnd() : left.physicalEnd();
    148138    bool mergedUsedSinceLastScavenge = a.usedSinceLastScavenge() || b.usedSinceLastScavenge();
    149 #endif
    150139    if (left.size() == left.startPhysicalSize()) {
    151140        return LargeRange(
    … …  
    153142            a.size() + b.size(),
    154143            a.startPhysicalSize() + b.startPhysicalSize(),
    155             a.totalPhysicalSize() + b.totalPhysicalSize()
    156 #if !BUSE(PARTIAL_SCAVENGE)
     144            a.totalPhysicalSize() + b.totalPhysicalSize(),
     145            physicalEnd
    157146            , mergedUsedSinceLastScavenge
    158 #endif
    159147        );
    160148       
    … …  
    165153        a.size() + b.size(),
    166154        left.startPhysicalSize(),
    167         a.totalPhysicalSize() + b.totalPhysicalSize()
    168 #if !BUSE(PARTIAL_SCAVENGE)
     155        a.totalPhysicalSize() + b.totalPhysicalSize(),
     156        physicalEnd
    169157        , mergedUsedSinceLastScavenge
    170 #endif
    171158    );
    172159}
    … …  
    176163    BASSERT(leftSize <= this->size());
    177164    size_t rightSize = this->size() - leftSize;
     165    char* physicalEnd = this->physicalEnd();
    178166
    179167    if (leftSize <= startPhysicalSize()) {
    180168        BASSERT(totalPhysicalSize() >= leftSize);
    181         LargeRange left(begin(), leftSize, leftSize, leftSize);
    182         LargeRange right(left.end(), rightSize, startPhysicalSize() - leftSize, totalPhysicalSize() - leftSize);
     169        LargeRange left(begin(), leftSize, leftSize, leftSize, std::min(physicalEnd, begin() + leftSize));
     170        LargeRange right(left.end(), rightSize, startPhysicalSize() - leftSize, totalPhysicalSize() - leftSize, std::max(physicalEnd, left.end()));
    183171        return std::make_pair(left, right);
    184172    }
    … …  
    195183    }
    196184
    197     LargeRange left(begin(), leftSize, startPhysicalSize(), leftTotalPhysicalSize);
    198     LargeRange right(left.end(), rightSize, 0, rightTotalPhysicalSize);
     185    LargeRange left(begin(), leftSize, startPhysicalSize(), leftTotalPhysicalSize, std::min(physicalEnd, begin() + leftSize));
     186    LargeRange right(left.end(), rightSize, 0, rightTotalPhysicalSize, std::max(physicalEnd, left.end()));
    199187    return std::make_pair(left, right);
    200188}

  • trunk/Source/bmalloc/bmalloc/Scavenger.cpp

    r277898 r278278  
    8686    dispatch_release(queue);
    8787#endif
    88 #if BUSE(PARTIAL_SCAVENGE)
    89     m_waitTime = std::chrono::milliseconds(m_isInMiniMode ? 200 : 2000);
    90 #else
    9188    m_waitTime = std::chrono::milliseconds(10);
    92 #endif
    9389
    9490    m_thread = std::thread(&threadEntryPoint, this);
    … …  
    121117}
    122118
    123 void Scavenger::didStartGrowing()
    124 {
    125     // We don't really need to lock here, since this is just a heuristic.
    126     m_isProbablyGrowing = true;
    127 }
    128 
    129119void Scavenger::scheduleIfUnderMemoryPressure(size_t bytes)
    130120{
    … …  
    147137        return;
    148138
    149     m_isProbablyGrowing = false;
    150139    run(lock);
    151140}
    … …  
    159148        return;
    160149   
    161     m_isProbablyGrowing = false;
    162150    runSoon(lock);
    163151}
    … …  
    188176}
    189177
    190 #if BUSE(PARTIAL_SCAVENGE)
    191 std::chrono::milliseconds Scavenger::timeSinceLastPartialScavenge()
    192 {
    193     UniqueLockHolder lock(mutex());
    194     return std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - m_lastPartialScavengeTime);
    195 }
    196 #endif
    197 
    198178void Scavenger::enableMiniMode()
    199179{
    … …  
    221201        {
    222202            PrintTime printTime("\nfull scavenge under lock time");
    223 #if !BUSE(PARTIAL_SCAVENGE)
    224203            size_t deferredDecommits = 0;
    225 #endif
    226204            UniqueLockHolder lock(Heap::mutex());
    227205            for (unsigned i = numHeaps; i--;) {
    228206                if (!isActiveHeapKind(static_cast<HeapKind>(i)))
    229207                    continue;
    230 #if BUSE(PARTIAL_SCAVENGE)
    231                 PerProcess<PerHeapKind<Heap>>::get()->at(i).scavenge(lock, decommitter);
    232 #else
    233208                PerProcess<PerHeapKind<Heap>>::get()->at(i).scavenge(lock, decommitter, deferredDecommits);
    234 #endif
    235209            }
    236210            decommitter.processEager();
    237211
    238 #if !BUSE(PARTIAL_SCAVENGE)
    239212            if (deferredDecommits)
    240213                m_state = State::RunSoon;
    241 #endif
    242214        }
    243215
    … …  
    280252}
    281253
    282 #if BUSE(PARTIAL_SCAVENGE)
    283 void Scavenger::partialScavenge()
    284 {
    285     if (!m_isEnabled)
    286         return;
    287 
    288     UniqueLockHolder lock(m_scavengingMutex);
    289 
    290     if (verbose) {
    291         fprintf(stderr, "--------------------------------\n");
    292         fprintf(stderr, "--before partial scavenging--\n");
    293         dumpStats();
    294     }
    295 
    296     {
    297         BulkDecommit decommitter;
    298         {
    299             PrintTime printTime("\npartialScavenge under lock time");
    300             UniqueLockHolder lock(Heap::mutex());
    301             for (unsigned i = numHeaps; i--;) {
    302                 if (!isActiveHeapKind(static_cast<HeapKind>(i)))
    303                     continue;
    304                 Heap& heap = PerProcess<PerHeapKind<Heap>>::get()->at(i);
    305                 size_t freeableMemory = heap.freeableMemory(lock);
    306                 if (freeableMemory < 4 * MB)
    307                     continue;
    308                 heap.scavengeToHighWatermark(lock, decommitter);
    309             }
    310 
    311             decommitter.processEager();
    312         }
    313 
    314         {
    315             PrintTime printTime("partialScavenge lazy decommit time");
    316             decommitter.processLazy();
    317         }
    318 
    319         {
    320             PrintTime printTime("partialScavenge mark all as eligible time");
    321             LockHolder lock(Heap::mutex());
    322             for (unsigned i = numHeaps; i--;) {
    323                 if (!isActiveHeapKind(static_cast<HeapKind>(i)))
    324                     continue;
    325                 Heap& heap = PerProcess<PerHeapKind<Heap>>::get()->at(i);
    326                 heap.markAllLargeAsEligibile(lock);
    327             }
    328         }
    329     }
    330 
    331     {
    332         RELEASE_BASSERT(!m_deferredDecommits.size());
    333         AllIsoHeaps::get()->forEach(
    334             [&] (IsoHeapImplBase& heap) {
    335                 heap.scavengeToHighWatermark(m_deferredDecommits);
    336             });
    337         IsoHeapImplBase::finishScavenging(m_deferredDecommits);
    338         m_deferredDecommits.shrink(0);
    339     }
    340 
    341     if (verbose) {
    342         fprintf(stderr, "--after partial scavenging--\n");
    343         dumpStats();
    344         fprintf(stderr, "--------------------------------\n");
    345     }
    346 
    347     {
    348         UniqueLockHolder lock(mutex());
    349         m_lastPartialScavengeTime = std::chrono::steady_clock::now();
    350     }
    351 }
    352 #endif
    353 
    354254size_t Scavenger::freeableMemory()
    355255{
    … …  
    433333        }
    434334
    435 #if BUSE(PARTIAL_SCAVENGE)
    436         enum class ScavengeMode {
    437             None,
    438             Partial,
    439             Full
    440         };
    441 
    442         size_t freeableMemory = this->freeableMemory();
    443 
    444         ScavengeMode scavengeMode = [&] {
    445             auto timeSinceLastFullScavenge = this->timeSinceLastFullScavenge();
    446             auto timeSinceLastPartialScavenge = this->timeSinceLastPartialScavenge();
    447             auto timeSinceLastScavenge = std::min(timeSinceLastPartialScavenge, timeSinceLastFullScavenge);
    448 
    449             if (isUnderMemoryPressure() && freeableMemory > 1 * MB && timeSinceLastScavenge > std::chrono::milliseconds(5))
    450                 return ScavengeMode::Full;
    451 
    452             if (!m_isProbablyGrowing) {
    453                 if (timeSinceLastFullScavenge < std::chrono::milliseconds(1000) && !m_isInMiniMode)
    454                     return ScavengeMode::Partial;
    455                 return ScavengeMode::Full;
    456             }
    457 
    458             if (m_isInMiniMode) {
    459                 if (timeSinceLastFullScavenge < std::chrono::milliseconds(200))
    460                     return ScavengeMode::Partial;
    461                 return ScavengeMode::Full;
    462             }
    463 
    464 #if BCPU(X86_64)
    465             auto partialScavengeInterval = std::chrono::milliseconds(12000);
    466 #else
    467             auto partialScavengeInterval = std::chrono::milliseconds(8000);
    468 #endif
    469             if (timeSinceLastScavenge < partialScavengeInterval) {
    470                 // Rate limit partial scavenges.
    471                 return ScavengeMode::None;
    472             }
    473             if (freeableMemory < 25 * MB)
    474                 return ScavengeMode::None;
    475             if (5 * freeableMemory < footprint())
    476                 return ScavengeMode::None;
    477             return ScavengeMode::Partial;
    478         }();
    479 
    480         m_isProbablyGrowing = false;
    481 
    482         switch (scavengeMode) {
    483         case ScavengeMode::None: {
    484             runSoon();
    485             break;
    486         }
    487         case ScavengeMode::Partial: {
    488             partialScavenge();
    489             runSoon();
    490             break;
    491         }
    492         case ScavengeMode::Full: {
    493             scavenge();
    494             break;
    495         }
    496         }
    497 #else
    498335        std::chrono::steady_clock::time_point start { std::chrono::steady_clock::now() };
    499336       
    … …  
    510347        // https://bugs.webkit.org/show_bug.cgi?id=203987
    511348        if (!m_isInMiniMode) {
    512             timeSpentScavenging *= 150;
     349            timeSpentScavenging *= s_newWaitMultiplier;
    513350            std::chrono::milliseconds newWaitTime = std::chrono::duration_cast<std::chrono::milliseconds>(timeSpentScavenging);
    514             m_waitTime = std::min(std::max(newWaitTime, std::chrono::milliseconds(100)), std::chrono::milliseconds(10000));
     351            m_waitTime = std::min(std::max(newWaitTime, std::chrono::milliseconds(s_minWaitTimeMilliseconds)), std::chrono::milliseconds(s_maxWaitTimeMilliseconds));
    515352        }
    516353
    517354        if (verbose)
    518355            fprintf(stderr, "new wait time %lldms\n", static_cast<long long int>(m_waitTime.count()));
    519 #endif
    520356    }
    521357}

  • trunk/Source/bmalloc/bmalloc/Scavenger.h

    r277898 r278278  
    6060    void runSoon();
    6161   
    62     BEXPORT void didStartGrowing();
    6362    BEXPORT void scheduleIfUnderMemoryPressure(size_t bytes);
    6463    BEXPORT void schedule(size_t bytes);
    … …  
    9392
    9493    std::chrono::milliseconds timeSinceLastFullScavenge();
    95 #if BUSE(PARTIAL_SCAVENGE)
    96     std::chrono::milliseconds timeSinceLastPartialScavenge();
    97     void partialScavenge();
    98 #endif
    9994
    10095    std::atomic<State> m_state { State::Sleep };
    10196    size_t m_scavengerBytes { 0 };
    10297    std::chrono::milliseconds m_waitTime;
    103     bool m_isProbablyGrowing { false };
    10498    bool m_isInMiniMode { false };
    10599   
    … …  
    109103    std::thread m_thread;
    110104    std::chrono::steady_clock::time_point m_lastFullScavengeTime { std::chrono::steady_clock::now() };
    111 #if BUSE(PARTIAL_SCAVENGE)
    112     std::chrono::steady_clock::time_point m_lastPartialScavengeTime { std::chrono::steady_clock::now() };
    113 #endif
    114105
    115106#if BOS(DARWIN)
    … …  
    118109#endif
    119110   
     111#if BPLATFORM(MAC)
     112    const unsigned s_newWaitMultiplier = 300;
     113    const unsigned s_minWaitTimeMilliseconds = 750;
     114    const unsigned s_maxWaitTimeMilliseconds = 20000;
     115#else
     116    const unsigned s_newWaitMultiplier = 150;
     117    const unsigned s_minWaitTimeMilliseconds = 100;
     118    const unsigned s_maxWaitTimeMilliseconds = 10000;
     119#endif
     120
    120121    Vector<DeferredDecommit> m_deferredDecommits;
    121122    bool m_isEnabled { true };

  • trunk/Source/bmalloc/bmalloc/SmallPage.h

    r277898 r278278  
    5252    void setHasPhysicalPages(bool hasPhysicalPages) { m_hasPhysicalPages = hasPhysicalPages; }
    5353
    54 #if !BUSE(PARTIAL_SCAVENGE)
    5554    bool usedSinceLastScavenge() { return m_usedSinceLastScavenge; }
    5655    void clearUsedSinceLastScavenge() { m_usedSinceLastScavenge = false; }
    5756    void setUsedSinceLastScavenge() { m_usedSinceLastScavenge = true; }
    58 #endif
    5957
    6058    SmallLine* begin();
    … …  
    6664    unsigned char m_hasFreeLines: 1;
    6765    unsigned char m_hasPhysicalPages: 1;
    68 #if !BUSE(PARTIAL_SCAVENGE)
    6966    unsigned char m_usedSinceLastScavenge: 1;
    70 #endif
    7167    unsigned char m_refCount: 7;
    7268    unsigned char m_sizeClass;
Note: See TracChangeset for help on using the changeset viewer.