Changeset 220352 in webkit


Ignore:
Timestamp:
Aug 7, 2017 2:31:49 PM (7 years ago)
Author:
fpizlo@apple.com
Message:

Primitive auxiliaries and JSValue auxiliaries should have separate gigacages
https://bugs.webkit.org/show_bug.cgi?id=174919

Reviewed by Keith Miller.
Source/bmalloc:


This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of
HeapKind, PrimitiveGigacage and JSValueGigacage.

The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and
NO_INLINE into BNO_INLINE.

  • bmalloc.xcodeproj/project.pbxproj:
  • bmalloc/Allocator.cpp:

(bmalloc::Allocator::refillAllocatorSlowCase):
(bmalloc::Allocator::refillAllocator):
(bmalloc::Allocator::allocateLarge):
(bmalloc::Allocator::allocateLogSizeClass):

  • bmalloc/AsyncTask.h:
  • bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h.
  • bmalloc/Cache.cpp:

(bmalloc::Cache::tryAllocateSlowCaseNullCache):
(bmalloc::Cache::allocateSlowCaseNullCache):
(bmalloc::Cache::deallocateSlowCaseNullCache):
(bmalloc::Cache::reallocateSlowCaseNullCache):

  • bmalloc/Deallocator.cpp:
  • bmalloc/Gigacage.cpp:

(Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks):
(Gigacage::ensureGigacage):
(Gigacage::disablePrimitiveGigacage):
(Gigacage::addPrimitiveDisableCallback):
(Gigacage::removePrimitiveDisableCallback):
(Gigacage::Callbacks::Callbacks): Deleted.
(Gigacage::disableGigacage): Deleted.
(Gigacage::addDisableCallback): Deleted.
(Gigacage::removeDisableCallback): Deleted.

  • bmalloc/Gigacage.h:

(Gigacage::name):
(Gigacage::basePtr):
(Gigacage::forEachKind):
(Gigacage::caged):
(Gigacage::isCaged):

  • bmalloc/Heap.cpp:

(bmalloc::Heap::Heap):
(bmalloc::Heap::usingGigacage):
(bmalloc::Heap::gigacageBasePtr):

  • bmalloc/Heap.h:
  • bmalloc/HeapKind.h:

(bmalloc::isGigacage):
(bmalloc::gigacageKind):
(bmalloc::heapKind):

  • bmalloc/Inline.h: Removed.
  • bmalloc/Map.h:
  • bmalloc/PerProcess.h:

(bmalloc::PerProcess<T>::getFastCase):
(bmalloc::PerProcess<T>::get):
(bmalloc::PerProcess<T>::getSlowCase):

  • bmalloc/PerThread.h:

(bmalloc::PerThread<T>::getFastCase):

  • bmalloc/Vector.h:

(bmalloc::Vector<T>::push):
(bmalloc::Vector<T>::shrinkCapacity):
(bmalloc::Vector<T>::growCapacity):

Source/JavaScriptCore:


This adapts JSC to there being two gigacages.

To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than
singletons. I don't think we were gaining anything by making them be singletons.

This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of
gigacages. We'll have one of those allocators per cage.

From there, this change teaches everyone who previously knew about cages that there are two cages.
This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is
easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's
not so obvious, so this change introduces some helpers to make it easy to define what cage you want
to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h

A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and
CagedPtr. This removes one layer of "get()" calls from a bunch of places.

(JSC::AccessCase::generateImpl):

  • dfg/DFGSpeculativeJIT.cpp:

(JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
(JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileNewTypedArray):
(JSC::DFG::SpeculativeJIT::emitAllocateButterfly):

  • ftl/FTLLowerDFGToB3.cpp:

(JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
(JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray):
(JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
(JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
(JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
(JSC::FTL::DFG::LowerDFGToB3::allocateJSArray):
(JSC::FTL::DFG::LowerDFGToB3::caged):

  • heap/FastMallocAlignedMemoryAllocator.cpp:

(JSC::FastMallocAlignedMemoryAllocator::instance): Deleted.

  • heap/FastMallocAlignedMemoryAllocator.h:
  • heap/GigacageAlignedMemoryAllocator.cpp:

(JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator):
(JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory):
(JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory):
(JSC::GigacageAlignedMemoryAllocator::dump const):
(JSC::GigacageAlignedMemoryAllocator::instance): Deleted.

  • heap/GigacageAlignedMemoryAllocator.h:
  • jsc.cpp:

(primitiveGigacageDisabled):
(jscmain):
(gigacageDisabled): Deleted.

  • llint/LowLevelInterpreter64.asm:
  • runtime/ArrayBuffer.cpp:

(JSC::ArrayBufferContents::tryAllocate):
(JSC::ArrayBuffer::createAdopted):
(JSC::ArrayBuffer::createFromBytes):

  • runtime/AuxiliaryBarrier.h:
  • runtime/ButterflyInlines.h:

(JSC::Butterfly::createUninitialized):
(JSC::Butterfly::tryCreate):
(JSC::Butterfly::growArrayRight):

  • runtime/CagedBarrierPtr.h: Added.

(JSC::CagedBarrierPtr::CagedBarrierPtr):
(JSC::CagedBarrierPtr::clear):
(JSC::CagedBarrierPtr::set):
(JSC::CagedBarrierPtr::get const):
(JSC::CagedBarrierPtr::getMayBeNull const):
(JSC::CagedBarrierPtr::operator== const):
(JSC::CagedBarrierPtr::operator!= const):
(JSC::CagedBarrierPtr::operator bool const):
(JSC::CagedBarrierPtr::setWithoutBarrier):
(JSC::CagedBarrierPtr::operator* const):
(JSC::CagedBarrierPtr::operator-> const):
(JSC::CagedBarrierPtr::operator[] const):

  • runtime/DirectArguments.cpp:

(JSC::DirectArguments::overrideThings):
(JSC::DirectArguments::unmapArgument):

  • runtime/DirectArguments.h:

(JSC::DirectArguments::isMappedArgument const):

  • runtime/GenericArguments.h:
  • runtime/GenericArgumentsInlines.h:

(JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor):
(JSC::GenericArguments<Type>::setModifiedArgumentDescriptor):
(JSC::GenericArguments<Type>::isModifiedArgumentDescriptor):

  • runtime/HashMapImpl.cpp:

(JSC::HashMapImpl<HashMapBucket>::visitChildren):

  • runtime/HashMapImpl.h:

(JSC::HashMapBuffer::create):
(JSC::HashMapImpl::buffer const):
(JSC::HashMapImpl::rehash):

  • runtime/JSArray.cpp:

(JSC::JSArray::tryCreateUninitializedRestricted):
(JSC::JSArray::unshiftCountSlowCase):
(JSC::JSArray::setLength):
(JSC::JSArray::pop):
(JSC::JSArray::push):
(JSC::JSArray::fastSlice):
(JSC::JSArray::shiftCountWithArrayStorage):
(JSC::JSArray::shiftCountWithAnyIndexingType):
(JSC::JSArray::unshiftCountWithAnyIndexingType):
(JSC::JSArray::fillArgList):
(JSC::JSArray::copyToArguments):

  • runtime/JSArray.h:

(JSC::JSArray::tryCreate):

  • runtime/JSArrayBufferView.cpp:

(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
(JSC::JSArrayBufferView::finalize):

  • runtime/JSLock.cpp:

(JSC::JSLock::didAcquireLock):

  • runtime/JSObject.cpp:

(JSC::JSObject::heapSnapshot):
(JSC::JSObject::getOwnPropertySlotByIndex):
(JSC::JSObject::putByIndex):
(JSC::JSObject::enterDictionaryIndexingMode):
(JSC::JSObject::createInitialIndexedStorage):
(JSC::JSObject::createArrayStorage):
(JSC::JSObject::convertUndecidedToInt32):
(JSC::JSObject::convertUndecidedToDouble):
(JSC::JSObject::convertUndecidedToContiguous):
(JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements):
(JSC::JSObject::convertUndecidedToArrayStorage):
(JSC::JSObject::convertInt32ToDouble):
(JSC::JSObject::convertInt32ToContiguous):
(JSC::JSObject::convertInt32ToArrayStorage):
(JSC::JSObject::convertDoubleToContiguous):
(JSC::JSObject::convertDoubleToArrayStorage):
(JSC::JSObject::convertContiguousToArrayStorage):
(JSC::JSObject::setIndexQuicklyToUndecided):
(JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode):
(JSC::JSObject::deletePropertyByIndex):
(JSC::JSObject::getOwnPropertyNames):
(JSC::JSObject::putIndexedDescriptor):
(JSC::JSObject::defineOwnIndexedProperty):
(JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes):
(JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength):
(JSC::JSObject::getNewVectorLength):
(JSC::JSObject::ensureLengthSlow):
(JSC::JSObject::reallocateAndShrinkButterfly):
(JSC::JSObject::allocateMoreOutOfLineStorage):
(JSC::JSObject::getEnumerableLength):

  • runtime/JSObject.h:

(JSC::JSObject::getArrayLength const):
(JSC::JSObject::getVectorLength):
(JSC::JSObject::putDirectIndex):
(JSC::JSObject::canGetIndexQuickly):
(JSC::JSObject::getIndexQuickly):
(JSC::JSObject::tryGetIndexQuickly const):
(JSC::JSObject::canSetIndexQuickly):
(JSC::JSObject::setIndexQuickly):
(JSC::JSObject::initializeIndex):
(JSC::JSObject::initializeIndexWithoutBarrier):
(JSC::JSObject::hasSparseMap):
(JSC::JSObject::inSparseIndexingMode):
(JSC::JSObject::butterfly const):
(JSC::JSObject::butterfly):
(JSC::JSObject::outOfLineStorage const):
(JSC::JSObject::outOfLineStorage):
(JSC::JSObject::ensureInt32):
(JSC::JSObject::ensureDouble):
(JSC::JSObject::ensureContiguous):
(JSC::JSObject::ensureArrayStorage):
(JSC::JSObject::arrayStorage):
(JSC::JSObject::arrayStorageOrNull):
(JSC::JSObject::ensureLength):

  • runtime/RegExpMatchesArray.h:

(JSC::tryCreateUninitializedRegExpMatchesArray):

  • runtime/VM.cpp:

(JSC::VM::VM):
(JSC::VM::~VM):
(JSC::VM::primitiveGigacageDisabledCallback):
(JSC::VM::primitiveGigacageDisabled):
(JSC::VM::gigacageDisabledCallback): Deleted.
(JSC::VM::gigacageDisabled): Deleted.

  • runtime/VM.h:

(JSC::VM::gigacageAuxiliarySpace):
(JSC::VM::firePrimitiveGigacageEnabledIfNecessary):
(JSC::VM::primitiveGigacageEnabled):
(JSC::VM::fireGigacageEnabledIfNecessary): Deleted.
(JSC::VM::gigacageEnabled): Deleted.

  • wasm/WasmMemory.cpp:

(JSC::Wasm::Memory::create):
(JSC::Wasm::Memory::~Memory):
(JSC::Wasm::Memory::grow):

Source/WebCore:

No new tests because no change in behavior.

Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be
specific about which Gigacage we're using.

  • bindings/js/WebCoreJSClientData.cpp:

(WebCore::JSVMClientData::JSVMClientData):

  • platform/graphics/cocoa/GPUBufferMetal.mm:

(WebCore::GPUBuffer::GPUBuffer):

Source/WebKit:


The disable callback is all about the primitive gigacage.

  • WebProcess/WebProcess.cpp:

(WebKit::primitiveGigacageDisabled):
(WebKit::m_webSQLiteDatabaseTracker):
(WebKit::gigacageDisabled): Deleted.

Source/WTF:


This mirrors the changes from bmalloc/Gigacage.h.

Also it teaches CagedPtr how to reason about multiple gigacages.

  • wtf/CagedPtr.h:

(WTF::CagedPtr::get const):
(WTF::CagedPtr::operator[] const):

  • wtf/Gigacage.cpp:

(Gigacage::tryMalloc):
(Gigacage::tryAllocateVirtualPages):
(Gigacage::freeVirtualPages):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):

  • wtf/Gigacage.h:

(Gigacage::disablePrimitiveGigacage):
(Gigacage::addPrimitiveDisableCallback):
(Gigacage::removePrimitiveDisableCallback):
(Gigacage::name):
(Gigacage::basePtr):
(Gigacage::caged):
(Gigacage::isCaged):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
(Gigacage::disableGigacage): Deleted.
(Gigacage::addDisableCallback): Deleted.
(Gigacage::removeDisableCallback): Deleted.

Location:
trunk/Source
Files:
1 added
54 edited
1 moved

Legend:

Unmodified
Added
Removed
  • trunk/Source/JavaScriptCore/ChangeLog

    r220346 r220352  
     12017-08-06  Filip Pizlo  <fpizlo@apple.com>
     2
     3        Primitive auxiliaries and JSValue auxiliaries should have separate gigacages
     4        https://bugs.webkit.org/show_bug.cgi?id=174919
     5
     6        Reviewed by Keith Miller.
     7       
     8        This adapts JSC to there being two gigacages.
     9       
     10        To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than
     11        singletons. I don't think we were gaining anything by making them be singletons.
     12       
     13        This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of
     14        gigacages. We'll have one of those allocators per cage.
     15       
     16        From there, this change teaches everyone who previously knew about cages that there are two cages.
     17        This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is
     18        easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's
     19        not so obvious, so this change introduces some helpers to make it easy to define what cage you want
     20        to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h
     21       
     22        A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and
     23        CagedPtr. This removes one layer of "get()" calls from a bunch of places.
     24
     25        * JavaScriptCore.xcodeproj/project.pbxproj:
     26        * bytecode/AccessCase.cpp:
     27        (JSC::AccessCase::generateImpl):
     28        * dfg/DFGSpeculativeJIT.cpp:
     29        (JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
     30        (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
     31        (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
     32        (JSC::DFG::SpeculativeJIT::compileNewTypedArray):
     33        (JSC::DFG::SpeculativeJIT::emitAllocateButterfly):
     34        * ftl/FTLLowerDFGToB3.cpp:
     35        (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
     36        (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
     37        (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray):
     38        (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
     39        (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
     40        (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
     41        (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray):
     42        (JSC::FTL::DFG::LowerDFGToB3::caged):
     43        * heap/FastMallocAlignedMemoryAllocator.cpp:
     44        (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted.
     45        * heap/FastMallocAlignedMemoryAllocator.h:
     46        * heap/GigacageAlignedMemoryAllocator.cpp:
     47        (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator):
     48        (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory):
     49        (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory):
     50        (JSC::GigacageAlignedMemoryAllocator::dump const):
     51        (JSC::GigacageAlignedMemoryAllocator::instance): Deleted.
     52        * heap/GigacageAlignedMemoryAllocator.h:
     53        * jsc.cpp:
     54        (primitiveGigacageDisabled):
     55        (jscmain):
     56        (gigacageDisabled): Deleted.
     57        * llint/LowLevelInterpreter64.asm:
     58        * runtime/ArrayBuffer.cpp:
     59        (JSC::ArrayBufferContents::tryAllocate):
     60        (JSC::ArrayBuffer::createAdopted):
     61        (JSC::ArrayBuffer::createFromBytes):
     62        * runtime/AuxiliaryBarrier.h:
     63        * runtime/ButterflyInlines.h:
     64        (JSC::Butterfly::createUninitialized):
     65        (JSC::Butterfly::tryCreate):
     66        (JSC::Butterfly::growArrayRight):
     67        * runtime/CagedBarrierPtr.h: Added.
     68        (JSC::CagedBarrierPtr::CagedBarrierPtr):
     69        (JSC::CagedBarrierPtr::clear):
     70        (JSC::CagedBarrierPtr::set):
     71        (JSC::CagedBarrierPtr::get const):
     72        (JSC::CagedBarrierPtr::getMayBeNull const):
     73        (JSC::CagedBarrierPtr::operator== const):
     74        (JSC::CagedBarrierPtr::operator!= const):
     75        (JSC::CagedBarrierPtr::operator bool const):
     76        (JSC::CagedBarrierPtr::setWithoutBarrier):
     77        (JSC::CagedBarrierPtr::operator* const):
     78        (JSC::CagedBarrierPtr::operator-> const):
     79        (JSC::CagedBarrierPtr::operator[] const):
     80        * runtime/DirectArguments.cpp:
     81        (JSC::DirectArguments::overrideThings):
     82        (JSC::DirectArguments::unmapArgument):
     83        * runtime/DirectArguments.h:
     84        (JSC::DirectArguments::isMappedArgument const):
     85        * runtime/GenericArguments.h:
     86        * runtime/GenericArgumentsInlines.h:
     87        (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor):
     88        (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor):
     89        (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor):
     90        * runtime/HashMapImpl.cpp:
     91        (JSC::HashMapImpl<HashMapBucket>::visitChildren):
     92        * runtime/HashMapImpl.h:
     93        (JSC::HashMapBuffer::create):
     94        (JSC::HashMapImpl::buffer const):
     95        (JSC::HashMapImpl::rehash):
     96        * runtime/JSArray.cpp:
     97        (JSC::JSArray::tryCreateUninitializedRestricted):
     98        (JSC::JSArray::unshiftCountSlowCase):
     99        (JSC::JSArray::setLength):
     100        (JSC::JSArray::pop):
     101        (JSC::JSArray::push):
     102        (JSC::JSArray::fastSlice):
     103        (JSC::JSArray::shiftCountWithArrayStorage):
     104        (JSC::JSArray::shiftCountWithAnyIndexingType):
     105        (JSC::JSArray::unshiftCountWithAnyIndexingType):
     106        (JSC::JSArray::fillArgList):
     107        (JSC::JSArray::copyToArguments):
     108        * runtime/JSArray.h:
     109        (JSC::JSArray::tryCreate):
     110        * runtime/JSArrayBufferView.cpp:
     111        (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
     112        (JSC::JSArrayBufferView::finalize):
     113        * runtime/JSLock.cpp:
     114        (JSC::JSLock::didAcquireLock):
     115        * runtime/JSObject.cpp:
     116        (JSC::JSObject::heapSnapshot):
     117        (JSC::JSObject::getOwnPropertySlotByIndex):
     118        (JSC::JSObject::putByIndex):
     119        (JSC::JSObject::enterDictionaryIndexingMode):
     120        (JSC::JSObject::createInitialIndexedStorage):
     121        (JSC::JSObject::createArrayStorage):
     122        (JSC::JSObject::convertUndecidedToInt32):
     123        (JSC::JSObject::convertUndecidedToDouble):
     124        (JSC::JSObject::convertUndecidedToContiguous):
     125        (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements):
     126        (JSC::JSObject::convertUndecidedToArrayStorage):
     127        (JSC::JSObject::convertInt32ToDouble):
     128        (JSC::JSObject::convertInt32ToContiguous):
     129        (JSC::JSObject::convertInt32ToArrayStorage):
     130        (JSC::JSObject::convertDoubleToContiguous):
     131        (JSC::JSObject::convertDoubleToArrayStorage):
     132        (JSC::JSObject::convertContiguousToArrayStorage):
     133        (JSC::JSObject::setIndexQuicklyToUndecided):
     134        (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode):
     135        (JSC::JSObject::deletePropertyByIndex):
     136        (JSC::JSObject::getOwnPropertyNames):
     137        (JSC::JSObject::putIndexedDescriptor):
     138        (JSC::JSObject::defineOwnIndexedProperty):
     139        (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes):
     140        (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength):
     141        (JSC::JSObject::getNewVectorLength):
     142        (JSC::JSObject::ensureLengthSlow):
     143        (JSC::JSObject::reallocateAndShrinkButterfly):
     144        (JSC::JSObject::allocateMoreOutOfLineStorage):
     145        (JSC::JSObject::getEnumerableLength):
     146        * runtime/JSObject.h:
     147        (JSC::JSObject::getArrayLength const):
     148        (JSC::JSObject::getVectorLength):
     149        (JSC::JSObject::putDirectIndex):
     150        (JSC::JSObject::canGetIndexQuickly):
     151        (JSC::JSObject::getIndexQuickly):
     152        (JSC::JSObject::tryGetIndexQuickly const):
     153        (JSC::JSObject::canSetIndexQuickly):
     154        (JSC::JSObject::setIndexQuickly):
     155        (JSC::JSObject::initializeIndex):
     156        (JSC::JSObject::initializeIndexWithoutBarrier):
     157        (JSC::JSObject::hasSparseMap):
     158        (JSC::JSObject::inSparseIndexingMode):
     159        (JSC::JSObject::butterfly const):
     160        (JSC::JSObject::butterfly):
     161        (JSC::JSObject::outOfLineStorage const):
     162        (JSC::JSObject::outOfLineStorage):
     163        (JSC::JSObject::ensureInt32):
     164        (JSC::JSObject::ensureDouble):
     165        (JSC::JSObject::ensureContiguous):
     166        (JSC::JSObject::ensureArrayStorage):
     167        (JSC::JSObject::arrayStorage):
     168        (JSC::JSObject::arrayStorageOrNull):
     169        (JSC::JSObject::ensureLength):
     170        * runtime/RegExpMatchesArray.h:
     171        (JSC::tryCreateUninitializedRegExpMatchesArray):
     172        * runtime/VM.cpp:
     173        (JSC::VM::VM):
     174        (JSC::VM::~VM):
     175        (JSC::VM::primitiveGigacageDisabledCallback):
     176        (JSC::VM::primitiveGigacageDisabled):
     177        (JSC::VM::gigacageDisabledCallback): Deleted.
     178        (JSC::VM::gigacageDisabled): Deleted.
     179        * runtime/VM.h:
     180        (JSC::VM::gigacageAuxiliarySpace):
     181        (JSC::VM::firePrimitiveGigacageEnabledIfNecessary):
     182        (JSC::VM::primitiveGigacageEnabled):
     183        (JSC::VM::fireGigacageEnabledIfNecessary): Deleted.
     184        (JSC::VM::gigacageEnabled): Deleted.
     185        * wasm/WasmMemory.cpp:
     186        (JSC::Wasm::Memory::create):
     187        (JSC::Wasm::Memory::~Memory):
     188        (JSC::Wasm::Memory::grow):
     189
    11902017-08-07  Commit Queue  <commit-queue@webkit.org>
    2191
  • trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj

    r220318 r220352  
    899899                0FEC3C5A1F33A48900F59B6C /* GigacageAlignedMemoryAllocator.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FEC3C581F33A48900F59B6C /* GigacageAlignedMemoryAllocator.cpp */; };
    900900                0FEC3C5B1F33A48900F59B6C /* GigacageAlignedMemoryAllocator.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC3C591F33A48900F59B6C /* GigacageAlignedMemoryAllocator.h */; };
     901                0FEC3C601F379F5300F59B6C /* CagedBarrierPtr.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC3C5F1F379F5300F59B6C /* CagedBarrierPtr.h */; settings = {ATTRIBUTES = (Private, ); }; };
    901902                0FEC84FE1BDACDAC0080FF74 /* B3ArgumentRegValue.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FEC84B41BDACDAC0080FF74 /* B3ArgumentRegValue.cpp */; };
    902903                0FEC84FF1BDACDAC0080FF74 /* B3ArgumentRegValue.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84B51BDACDAC0080FF74 /* B3ArgumentRegValue.h */; };
     
    34973498                0FEC3C581F33A48900F59B6C /* GigacageAlignedMemoryAllocator.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = GigacageAlignedMemoryAllocator.cpp; sourceTree = "<group>"; };
    34983499                0FEC3C591F33A48900F59B6C /* GigacageAlignedMemoryAllocator.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = GigacageAlignedMemoryAllocator.h; sourceTree = "<group>"; };
     3500                0FEC3C5F1F379F5300F59B6C /* CagedBarrierPtr.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = CagedBarrierPtr.h; sourceTree = "<group>"; };
    34993501                0FEC84B41BDACDAC0080FF74 /* B3ArgumentRegValue.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = B3ArgumentRegValue.cpp; path = b3/B3ArgumentRegValue.cpp; sourceTree = "<group>"; };
    35003502                0FEC84B51BDACDAC0080FF74 /* B3ArgumentRegValue.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = B3ArgumentRegValue.h; path = b3/B3ArgumentRegValue.h; sourceTree = "<group>"; };
     
    66236625                                0FB7F38B15ED8E3800F167B2 /* Butterfly.h */,
    66246626                                0FB7F38C15ED8E3800F167B2 /* ButterflyInlines.h */,
     6627                                0FEC3C5F1F379F5300F59B6C /* CagedBarrierPtr.h */,
    66256628                                BCA62DFE0E2826230004F30D /* CallData.cpp */,
    66266629                                145C507F0D9DF63B0088F6B9 /* CallData.h */,
     
    86408643                                A704D90617A0BAA8006BA554 /* DFGInPlaceAbstractState.h in Headers */,
    86418644                                0F2BDC21151E803B00CD8910 /* DFGInsertionSet.h in Headers */,
     8645                                0FEC3C601F379F5300F59B6C /* CagedBarrierPtr.h in Headers */,
    86428646                                0F300B7C18AB1B1400A6D72E /* DFGIntegerCheckCombiningPhase.h in Headers */,
    86438647                                0F898F321B27689F0083A33C /* DFGIntegerRangeOptimizationPhase.h in Headers */,
  • trunk/Source/JavaScriptCore/bytecode/AccessCase.cpp

    r220322 r220352  
    859859
    860860            if (allocatingInline) {
    861                 MarkedAllocator* allocator = vm.auxiliarySpace.allocatorFor(newSize);
     861                MarkedAllocator* allocator = vm.jsValueGigacageAuxiliarySpace.allocatorFor(newSize);
    862862
    863863                if (!allocator) {
  • trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp

    r220219 r220352  
    110110   
    111111    if (size) {
    112         if (MarkedAllocator* allocator = m_jit.vm()->auxiliarySpace.allocatorFor(size)) {
     112        if (MarkedAllocator* allocator = m_jit.vm()->jsValueGigacageAuxiliarySpace.allocatorFor(size)) {
    113113            m_jit.move(TrustedImmPtr(allocator), scratchGPR);
    114114            m_jit.emitAllocate(storageGPR, allocator, scratchGPR, scratch2GPR, slowCases);
     
    78697869    size_t size = initialOutOfLineCapacity * sizeof(JSValue);
    78707870
    7871     MarkedAllocator* allocator = m_jit.vm()->auxiliarySpace.allocatorFor(size);
     7871    MarkedAllocator* allocator = m_jit.vm()->jsValueGigacageAuxiliarySpace.allocatorFor(size);
    78727872
    78737873    if (!allocator || node->transition()->previous->couldHaveIndexingHeader()) {
     
    79147914    ASSERT(newSize == node->transition()->next->outOfLineCapacity() * sizeof(JSValue));
    79157915   
    7916     MarkedAllocator* allocator = m_jit.vm()->auxiliarySpace.allocatorFor(newSize);
     7916    MarkedAllocator* allocator = m_jit.vm()->jsValueGigacageAuxiliarySpace.allocatorFor(newSize);
    79177917
    79187918    if (!allocator || node->transition()->previous->couldHaveIndexingHeader()) {
     
    84958495    }
    84968496    m_jit.emitAllocateVariableSized(
    8497         storageGPR, m_jit.vm()->auxiliarySpace, scratchGPR, scratchGPR,
     8497        storageGPR, m_jit.vm()->primitiveGigacageAuxiliarySpace, scratchGPR, scratchGPR,
    84988498        scratchGPR2, slowCases);
    84998499   
     
    1011310113    m_jit.add32(TrustedImm32(sizeof(IndexingHeader)), scratch1, scratch2);
    1011410114    m_jit.emitAllocateVariableSized(
    10115         storageResultGPR, m_jit.vm()->auxiliarySpace, scratch2, scratch1, scratch3, slowCases);
     10115        storageResultGPR, m_jit.vm()->jsValueGigacageAuxiliarySpace, scratch2, scratch1, scratch3, slowCases);
    1011610116    m_jit.addPtr(TrustedImm32(sizeof(IndexingHeader)), storageResultGPR);
    1011710117
  • trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp

    r220219 r220352  
    32363236        LValue butterfly = m_out.loadPtr(lowCell(m_node->child1()), m_heaps.JSObject_butterfly);
    32373237        if (m_node->op() != GetButterflyWithoutCaging)
    3238             butterfly = caged(butterfly);
     3238            butterfly = caged(Gigacage::JSValue, butterfly);
    32393239        setStorage(butterfly);
    32403240    }
     
    32733273
    32743274        DFG_ASSERT(m_graph, m_node, isTypedView(m_node->arrayMode().typedArrayType()));
    3275         setStorage(caged(m_out.loadPtr(cell, m_heaps.JSArrayBufferView_vector)));
     3275        setStorage(caged(Gigacage::Primitive, m_out.loadPtr(cell, m_heaps.JSArrayBufferView_vector)));
    32763276    }
    32773277   
     
    50415041            }
    50425042       
    5043             LValue allocator = allocatorForSize(vm().auxiliarySpace, byteSize, slowCase);
     5043            LValue allocator = allocatorForSize(vm().primitiveGigacageAuxiliarySpace, byteSize, slowCase);
    50445044            LValue storage = allocateHeapCell(allocator, slowCase);
    50455045           
     
    88708870        int32_t offsetOfFirstProperty = static_cast<int32_t>(offsetInButterfly(firstOutOfLineOffset)) * sizeof(EncodedJSValue);
    88718871        ValueFromBlock outOfLineResult = m_out.anchor(
    8872             m_out.load64(m_out.baseIndex(m_heaps.properties.atAnyNumber(), caged(storage), realIndex, ScaleEight, offsetOfFirstProperty)));
     8872            m_out.load64(m_out.baseIndex(m_heaps.properties.atAnyNumber(), caged(Gigacage::JSValue, storage), realIndex, ScaleEight, offsetOfFirstProperty)));
    88738873        m_out.jump(continuation);
    88748874
     
    90509050               
    90519051                LValue startOfStorage = allocateHeapCell(
    9052                     allocatorForSize(vm().auxiliarySpace, butterflySize, slowPath),
     9052                    allocatorForSize(vm().jsValueGigacageAuxiliarySpace, butterflySize, slowPath),
    90539053                    slowPath);
    90549054
     
    1000610006
    1000710007        size_t sizeInBytes = sizeInValues * sizeof(JSValue);
    10008         MarkedAllocator* allocator = vm().auxiliarySpace.allocatorFor(sizeInBytes);
     10008        MarkedAllocator* allocator = vm().jsValueGigacageAuxiliarySpace.allocatorFor(sizeInBytes);
    1000910009        LValue startOfStorage = allocateHeapCell(m_out.constIntPtr(allocator), slowPath);
    1001010010        ValueFromBlock fastButterfly = m_out.anchor(
     
    1123411234            payloadSize, m_out.constIntPtr(sizeof(IndexingHeader)));
    1123511235           
    11236         LValue allocator = allocatorForSize(vm().auxiliarySpace, butterflySize, failCase);
     11236        LValue allocator = allocatorForSize(vm().jsValueGigacageAuxiliarySpace, butterflySize, failCase);
    1123711237        LValue startOfStorage = allocateHeapCell(allocator, failCase);
    1123811238           
     
    1161511615    }
    1161611616   
    11617     LValue caged(LValue ptr)
    11618     {
    11619         if (vm().gigacageEnabled().isStillValid()) {
    11620             m_graph.watchpoints().addLazily(vm().gigacageEnabled());
    11621            
    11622             LValue basePtr = m_out.constIntPtr(g_gigacageBasePtr);
    11623             LValue mask = m_out.constIntPtr(GIGACAGE_MASK);
    11624            
    11625             // We don't have to worry about B3 messing up the bitAnd. Also, we want to get B3's excellent
    11626             // codegen for 2-operand andq on x86-64.
    11627             LValue masked = m_out.bitAnd(ptr, mask);
    11628            
    11629             // But B3 will currently mess up the code generation of this add. Basically, any offset from what we
    11630             // compute here will get reassociated and folded with g_gigacageBasePtr. There's a world in which
    11631             // moveConstants() observes that it needs to reassociate in order to hoist the big constants. But
    11632             // it's much easier to just block B3's badness here. That's what we do for now.
    11633             PatchpointValue* patchpoint = m_out.patchpoint(pointerType());
    11634             patchpoint->appendSomeRegister(basePtr);
    11635             patchpoint->appendSomeRegister(masked);
    11636             patchpoint->setGenerator(
    11637                 [] (CCallHelpers& jit, const StackmapGenerationParams& params) {
    11638                     jit.addPtr(params[1].gpr(), params[2].gpr(), params[0].gpr());
    11639                 });
    11640             patchpoint->effects = Effects::none();
    11641             return patchpoint;
    11642         }
    11643        
    11644         return ptr;
     11617    LValue caged(Gigacage::Kind kind, LValue ptr)
     11618    {
     11619        if (kind == Gigacage::Primitive) {
     11620            if (vm().primitiveGigacageEnabled().isStillValid())
     11621                m_graph.watchpoints().addLazily(vm().primitiveGigacageEnabled());
     11622            else
     11623                return ptr;
     11624        }
     11625       
     11626        LValue basePtr = m_out.constIntPtr(Gigacage::basePtr(kind));
     11627        LValue mask = m_out.constIntPtr(GIGACAGE_MASK);
     11628       
     11629        // We don't have to worry about B3 messing up the bitAnd. Also, we want to get B3's excellent
     11630        // codegen for 2-operand andq on x86-64.
     11631        LValue masked = m_out.bitAnd(ptr, mask);
     11632       
     11633        // But B3 will currently mess up the code generation of this add. Basically, any offset from what we
     11634        // compute here will get reassociated and folded with Gigacage::basePtr. There's a world in which
     11635        // moveConstants() observes that it needs to reassociate in order to hoist the big constants. But
     11636        // it's much easier to just block B3's badness here. That's what we do for now.
     11637        PatchpointValue* patchpoint = m_out.patchpoint(pointerType());
     11638        patchpoint->appendSomeRegister(basePtr);
     11639        patchpoint->appendSomeRegister(masked);
     11640        patchpoint->setGenerator(
     11641            [] (CCallHelpers& jit, const StackmapGenerationParams& params) {
     11642                jit.addPtr(params[1].gpr(), params[2].gpr(), params[0].gpr());
     11643            });
     11644        patchpoint->effects = Effects::none();
     11645        return patchpoint;
    1164511646    }
    1164611647   
  • trunk/Source/JavaScriptCore/heap/FastMallocAlignedMemoryAllocator.cpp

    r220291 r220352  
    3232namespace JSC {
    3333
    34 FastMallocAlignedMemoryAllocator& FastMallocAlignedMemoryAllocator::instance()
    35 {
    36     static FastMallocAlignedMemoryAllocator* result;
    37     static std::once_flag onceFlag;
    38     std::call_once(
    39         onceFlag,
    40         [] {
    41             result = new FastMallocAlignedMemoryAllocator();
    42         });
    43     return *result;
    44 }
    45 
    4634FastMallocAlignedMemoryAllocator::FastMallocAlignedMemoryAllocator()
    4735{
  • trunk/Source/JavaScriptCore/heap/FastMallocAlignedMemoryAllocator.h

    r220291 r220352  
    3232class FastMallocAlignedMemoryAllocator : public AlignedMemoryAllocator {
    3333public:
    34     JS_EXPORT_PRIVATE static FastMallocAlignedMemoryAllocator& instance();
    35 
     34    FastMallocAlignedMemoryAllocator();
    3635    ~FastMallocAlignedMemoryAllocator();
    3736   
     
    4039   
    4140    void dump(PrintStream&) const override;
    42 
    43 private:
    44     FastMallocAlignedMemoryAllocator();
    4541};
    4642
  • trunk/Source/JavaScriptCore/heap/GigacageAlignedMemoryAllocator.cpp

    r220291 r220352  
    2727#include "GigacageAlignedMemoryAllocator.h"
    2828
    29 #include <mutex>
    30 #include <wtf/Gigacage.h>
    31 
    3229namespace JSC {
    3330
    34 GigacageAlignedMemoryAllocator& GigacageAlignedMemoryAllocator::instance()
    35 {
    36     static GigacageAlignedMemoryAllocator* result;
    37     static std::once_flag onceFlag;
    38     std::call_once(
    39         onceFlag,
    40         [] {
    41             result = new GigacageAlignedMemoryAllocator();
    42         });
    43     return *result;
    44 }
    45 
    46 GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator()
     31GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator(Gigacage::Kind kind)
     32    : m_kind(kind)
    4733{
    4834}
     
    5440void* GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory(size_t alignment, size_t size)
    5541{
    56     return Gigacage::tryAlignedMalloc(alignment, size);
     42    return Gigacage::tryAlignedMalloc(m_kind, alignment, size);
    5743}
    5844
    5945void GigacageAlignedMemoryAllocator::freeAlignedMemory(void* basePtr)
    6046{
    61     Gigacage::alignedFree(basePtr);
     47    Gigacage::alignedFree(m_kind, basePtr);
    6248}
    6349
    6450void GigacageAlignedMemoryAllocator::dump(PrintStream& out) const
    6551{
    66     out.print("Gigacage");
     52    out.print(Gigacage::name(m_kind), "Gigacage");
    6753}
    6854
  • trunk/Source/JavaScriptCore/heap/GigacageAlignedMemoryAllocator.h

    r220291 r220352  
    2727
    2828#include "AlignedMemoryAllocator.h"
     29#include <wtf/Gigacage.h>
    2930
    3031namespace JSC {
     
    3233class GigacageAlignedMemoryAllocator : public AlignedMemoryAllocator {
    3334public:
    34     // FIXME: This shouldn't be a singleton. There should be different instances for primaries, JSValues,
    35     // and other things.
    36     // https://bugs.webkit.org/show_bug.cgi?id=174919
    37     static GigacageAlignedMemoryAllocator& instance();
    38 
     35    GigacageAlignedMemoryAllocator(Gigacage::Kind);
    3936    ~GigacageAlignedMemoryAllocator();
    4037   
     
    4542
    4643private:
    47     GigacageAlignedMemoryAllocator();
     44    Gigacage::Kind m_kind;
    4845};
    4946
  • trunk/Source/JavaScriptCore/jsc.cpp

    r220148 r220352  
    38033803}
    38043804
    3805 static void gigacageDisabled(void*)
    3806 {
    3807     dataLog("Gigacage disabled! Aborting.\n");
     3805static void primitiveGigacageDisabled(void*)
     3806{
     3807    dataLog("Primitive gigacage disabled! Aborting.\n");
    38083808    UNREACHABLE_FOR_PLATFORM();
    38093809}
     
    38283828#endif
    38293829    if (Gigacage::shouldBeEnabled())
    3830         Gigacage::addDisableCallback(gigacageDisabled, nullptr);
     3830        Gigacage::addPrimitiveDisableCallback(primitiveGigacageDisabled, nullptr);
    38313831
    38323832    int result;
  • trunk/Source/JavaScriptCore/llint/LowLevelInterpreter64.asm

    r220184 r220352  
    377377end
    378378
    379 macro loadCaged(source, dest, scratch)
     379macro loadCaged(basePtr, source, dest, scratch)
    380380    loadp source, dest
    381381    if GIGACAGE_ENABLED and not C_LOOP
    382         loadp _g_gigacageBasePtr, scratch
     382        loadp basePtr, scratch
    383383        btpz scratch, .done
    384384        andp constexpr GIGACAGE_MASK, dest
     
    12101210macro loadPropertyAtVariableOffset(propertyOffsetAsInt, objectAndStorage, value)
    12111211    bilt propertyOffsetAsInt, firstOutOfLineOffset, .isInline
    1212     loadCaged(JSObject::m_butterfly[objectAndStorage], objectAndStorage, value)
     1212    loadCaged(_g_jsValueGigacageBasePtr, JSObject::m_butterfly[objectAndStorage], objectAndStorage, value)
    12131213    negi propertyOffsetAsInt
    12141214    sxi2q propertyOffsetAsInt, propertyOffsetAsInt
     
    12231223macro storePropertyAtVariableOffset(propertyOffsetAsInt, objectAndStorage, value, scratch)
    12241224    bilt propertyOffsetAsInt, firstOutOfLineOffset, .isInline
    1225     loadCaged(JSObject::m_butterfly[objectAndStorage], objectAndStorage, scratch)
     1225    loadCaged(_g_jsValueGigacageBasePtr, JSObject::m_butterfly[objectAndStorage], objectAndStorage, scratch)
    12261226    negi propertyOffsetAsInt
    12271227    sxi2q propertyOffsetAsInt, propertyOffsetAsInt
     
    12991299    btiz t2, IndexingShapeMask, .opGetArrayLengthSlow
    13001300    loadisFromInstruction(1, t1)
    1301     loadCaged(JSObject::m_butterfly[t3], t0, t2)
     1301    loadCaged(_g_jsValueGigacageBasePtr, JSObject::m_butterfly[t3], t0, t2)
    13021302    loadi -sizeof IndexingHeader + IndexingHeader::u.lengths.publicLength[t0], t0
    13031303    bilt t0, 0, .opGetArrayLengthSlow
     
    14821482    loadConstantOrVariableInt32(t3, t1, .opGetByValSlow)
    14831483    sxi2q t1, t1
    1484     loadCaged(JSObject::m_butterfly[t0], t3, t5)
     1484    loadCaged(_g_jsValueGigacageBasePtr, JSObject::m_butterfly[t0], t3, t5)
    14851485    andi IndexingShapeMask, t2
    14861486    bieq t2, Int32Shape, .opGetByValIsContiguous
     
    15291529   
    15301530    # Sweet, now we know that we have a typed array. Do some basic things now.
    1531     loadCaged(JSArrayBufferView::m_vector[t0], t3, t5)
     1531    loadCaged(_g_primitiveGigacageBasePtr, JSArrayBufferView::m_vector[t0], t3, t5)
    15321532    biaeq t1, JSArrayBufferView::m_length[t0], .opGetByValSlow
    15331533   
     
    16201620    loadConstantOrVariableInt32(t0, t3, .opPutByValSlow)
    16211621    sxi2q t3, t3
    1622     loadCaged(JSObject::m_butterfly[t1], t0, t5)
     1622    loadCaged(_g_jsValueGigacageBasePtr, JSObject::m_butterfly[t1], t0, t5)
    16231623    andi IndexingShapeMask, t2
    16241624    bineq t2, Int32Shape, .opPutByValNotInt32
  • trunk/Source/JavaScriptCore/runtime/ArrayBuffer.cpp

    r220330 r220352  
    107107    if (!size)
    108108        size = 1; // Make sure malloc actually allocates something, but not too much. We use null to mean that the buffer is neutered.
    109     m_data = Gigacage::tryMalloc(size);
     109    m_data = Gigacage::tryMalloc(Gigacage::Primitive, size);
    110110    if (!m_data) {
    111111        reset();
     
    117117
    118118    m_sizeInBytes = numElements * elementByteSize;
    119     m_destructor = [] (void* p) { Gigacage::free(p); };
     119    m_destructor = [] (void* p) { Gigacage::free(Gigacage::Primitive, p); };
    120120}
    121121
     
    188188Ref<ArrayBuffer> ArrayBuffer::createAdopted(const void* data, unsigned byteLength)
    189189{
    190     return createFromBytes(data, byteLength, [] (void* p) { Gigacage::free(p); });
     190    return createFromBytes(data, byteLength, [] (void* p) { Gigacage::free(Gigacage::Primitive, p); });
    191191}
    192192
     
    199199Ref<ArrayBuffer> ArrayBuffer::createFromBytes(const void* data, unsigned byteLength, ArrayBufferDestructorFunction&& destructor)
    200200{
    201     if (data && byteLength && !Gigacage::isCaged(data))
    202         Gigacage::disableGigacage();
     201    if (data && byteLength && !Gigacage::isCaged(Gigacage::Primitive, data))
     202        Gigacage::disablePrimitiveGigacage();
    203203   
    204204    ArrayBufferContents contents(const_cast<void*>(data), byteLength, WTFMove(destructor));
  • trunk/Source/JavaScriptCore/runtime/AuxiliaryBarrier.h

    r206525 r220352  
    3737class AuxiliaryBarrier {
    3838public:
     39    typedef T Type;
     40   
    3941    AuxiliaryBarrier(): m_value() { }
    4042   
  • trunk/Source/JavaScriptCore/runtime/ButterflyInlines.h

    r211043 r220352  
    6363{
    6464    size_t size = totalSize(preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes);
    65     void* base = vm.auxiliarySpace.allocate(size);
     65    void* base = vm.jsValueGigacageAuxiliarySpace.allocate(size);
    6666    Butterfly* result = fromBase(base, preCapacity, propertyCapacity);
    6767    return result;
     
    7171{
    7272    size_t size = totalSize(preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes);
    73     void* base = vm.auxiliarySpace.tryAllocate(size);
     73    void* base = vm.jsValueGigacageAuxiliarySpace.tryAllocate(size);
    7474    if (!base)
    7575        return nullptr;
     
    149149    size_t oldSize = totalSize(0, propertyCapacity, hadIndexingHeader, oldIndexingPayloadSizeInBytes);
    150150    size_t newSize = totalSize(0, propertyCapacity, true, newIndexingPayloadSizeInBytes);
    151     void* newBase = vm.auxiliarySpace.tryAllocate(newSize);
     151    void* newBase = vm.jsValueGigacageAuxiliarySpace.tryAllocate(newSize);
    152152    if (!newBase)
    153153        return nullptr;
  • trunk/Source/JavaScriptCore/runtime/DirectArguments.cpp

    r217108 r220352  
    119119    putDirect(vm, vm.propertyNames->iteratorSymbol, globalObject()->arrayProtoValuesFunction(), DontEnum);
    120120   
    121     void* backingStore = vm.auxiliarySpace.tryAllocate(mappedArgumentsSize());
     121    void* backingStore = vm.gigacageAuxiliarySpace(m_mappedArguments.kind).tryAllocate(mappedArgumentsSize());
    122122    RELEASE_ASSERT(backingStore);
    123123    bool* overrides = static_cast<bool*>(backingStore);
     
    136136{
    137137    overrideThingsIfNecessary(vm);
    138     m_mappedArguments.get()[index] = true;
     138    m_mappedArguments[index] = true;
    139139}
    140140
  • trunk/Source/JavaScriptCore/runtime/DirectArguments.h

    r215345 r220352  
    11/*
    2  * Copyright (C) 2015-2016 Apple Inc. All rights reserved.
     2 * Copyright (C) 2015-2017 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
     
    2626#pragma once
    2727
    28 #include "AuxiliaryBarrier.h"
     28#include "CagedBarrierPtr.h"
    2929#include "DirectArgumentsOffset.h"
    3030#include "GenericArguments.h"
     31#include <wtf/CagedPtr.h>
    3132
    3233namespace JSC {
     
    7475    bool isMappedArgument(uint32_t i) const
    7576    {
    76         return i < m_length && (!m_mappedArguments || !m_mappedArguments.get()[i]);
     77        return i < m_length && (!m_mappedArguments || !m_mappedArguments[i]);
    7778    }
    7879
     
    165166    uint32_t m_length; // Always the actual length of captured arguments and never what was stored into the length property.
    166167    uint32_t m_minCapacity; // The max of this and length determines the capacity of this object. It may be the actual capacity, or maybe something smaller. We arrange it this way to be kind to the JITs.
    167     AuxiliaryBarrier<bool*> m_mappedArguments; // If non-null, it means that length, callee, and caller are fully materialized properties.
     168    CagedBarrierPtr<Gigacage::Primitive, bool> m_mappedArguments; // If non-null, it means that length, callee, and caller are fully materialized properties.
    168169};
    169170
  • trunk/Source/JavaScriptCore/runtime/GenericArguments.h

    r210146 r220352  
    11/*
    2  * Copyright (C) 2015 Apple Inc. All rights reserved.
     2 * Copyright (C) 2015-2017 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
     
    2626#pragma once
    2727
     28#include "CagedBarrierPtr.h"
    2829#include "JSObject.h"
    2930
     
    6162    void copyToArguments(ExecState*, VirtualRegister firstElementDest, unsigned offset, unsigned length);
    6263   
    63     AuxiliaryBarrier<bool*> m_modifiedArgumentsDescriptor;
     64    CagedBarrierPtr<Gigacage::Primitive, bool> m_modifiedArgumentsDescriptor;
    6465};
    6566
  • trunk/Source/JavaScriptCore/runtime/GenericArgumentsInlines.h

    r219433 r220352  
    262262
    263263    if (argsLength) {
    264         void* backingStore = vm.auxiliarySpace.tryAllocate(WTF::roundUpToMultipleOf<8>(argsLength));
     264        void* backingStore = vm.gigacageAuxiliarySpace(m_modifiedArgumentsDescriptor.kind).tryAllocate(WTF::roundUpToMultipleOf<8>(argsLength));
    265265        RELEASE_ASSERT(backingStore);
    266266        bool* modifiedArguments = static_cast<bool*>(backingStore);
     
    283283    initModifiedArgumentsDescriptorIfNecessary(vm, length);
    284284    if (index < length)
    285         m_modifiedArgumentsDescriptor.get()[index] = true;
     285        m_modifiedArgumentsDescriptor[index] = true;
    286286}
    287287
     
    292292        return false;
    293293    if (index < length)
    294         return m_modifiedArgumentsDescriptor.get()[index];
     294        return m_modifiedArgumentsDescriptor[index];
    295295    return false;
    296296}
  • trunk/Source/JavaScriptCore/runtime/HashMapImpl.cpp

    r217192 r220352  
    6363    visitor.append(thisObject->m_tail);
    6464   
    65     if (HashMapBufferType* buffer = thisObject->m_buffer.get())
     65    if (HashMapBufferType* buffer = thisObject->m_buffer.getMayBeNull())
    6666        visitor.markAuxiliary(buffer);
    6767}
  • trunk/Source/JavaScriptCore/runtime/HashMapImpl.h

    r217808 r220352  
    189189        auto scope = DECLARE_THROW_SCOPE(vm);
    190190        size_t allocationSize = HashMapBuffer::allocationSize(capacity);
    191         void* data = vm.auxiliarySpace.tryAllocate(allocationSize);
     191        void* data = vm.jsValueGigacageAuxiliarySpace.tryAllocate(allocationSize);
    192192        if (!data) {
    193193            throwOutOfMemoryError(exec, scope);
     
    303303    ALWAYS_INLINE HashMapBucketType** buffer() const
    304304    {
    305         return m_buffer.get()->buffer();
     305        return m_buffer->buffer();
    306306    }
    307307
     
    592592            RETURN_IF_EXCEPTION(scope, void());
    593593        } else {
    594             m_buffer.get()->reset(m_capacity);
     594            m_buffer->reset(m_capacity);
    595595            assertBufferIsEmpty();
    596596        }
     
    656656    WriteBarrier<HashMapBucketType> m_head;
    657657    WriteBarrier<HashMapBucketType> m_tail;
    658     AuxiliaryBarrier<HashMapBufferType*> m_buffer;
     658    CagedBarrierPtr<Gigacage::JSValue, HashMapBufferType> m_buffer;
    659659    uint32_t m_keyCount;
    660660    uint32_t m_deleteCount;
  • trunk/Source/JavaScriptCore/runtime/JSArray.cpp

    r220165 r220352  
    8282
    8383        unsigned vectorLength = Butterfly::optimalContiguousVectorLength(structure, initialLength);
    84         void* temp = vm.auxiliarySpace.tryAllocate(deferralContext, Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)));
     84        void* temp = vm.jsValueGigacageAuxiliarySpace.tryAllocate(deferralContext, Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)));
    8585        if (UNLIKELY(!temp))
    8686            return nullptr;
     
    9999        static const unsigned indexBias = 0;
    100100        unsigned vectorLength = ArrayStorage::optimalVectorLength(indexBias, structure, initialLength);
    101         void* temp = vm.auxiliarySpace.tryAllocate(deferralContext, Butterfly::totalSize(indexBias, outOfLineStorage, true, ArrayStorage::sizeFor(vectorLength)));
     101        void* temp = vm.jsValueGigacageAuxiliarySpace.tryAllocate(deferralContext, Butterfly::totalSize(indexBias, outOfLineStorage, true, ArrayStorage::sizeFor(vectorLength)));
    102102        if (UNLIKELY(!temp))
    103103            return nullptr;
     
    361361    } else {
    362362        size_t newSize = Butterfly::totalSize(0, propertyCapacity, true, ArrayStorage::sizeFor(desiredCapacity));
    363         newAllocBase = vm.auxiliarySpace.tryAllocate(newSize);
     363        newAllocBase = vm.jsValueGigacageAuxiliarySpace.tryAllocate(newSize);
    364364        if (!newAllocBase)
    365365            return false;
     
    551551    auto scope = DECLARE_THROW_SCOPE(vm);
    552552
    553     Butterfly* butterfly = m_butterfly.get().getMayBeNull();
     553    Butterfly* butterfly = m_butterfly.getMayBeNull();
    554554    switch (indexingType()) {
    555555    case ArrayClass:
     
    621621    auto scope = DECLARE_THROW_SCOPE(vm);
    622622
    623     Butterfly* butterfly = m_butterfly.get().getMayBeNull();
     623    Butterfly* butterfly = m_butterfly.getMayBeNull();
    624624   
    625625    switch (indexingType()) {
     
    723723    auto scope = DECLARE_THROW_SCOPE(vm);
    724724
    725     Butterfly* butterfly = m_butterfly.get().getMayBeNull();
     725    Butterfly* butterfly = m_butterfly.getMayBeNull();
    726726   
    727727    switch (indexingType()) {
     
    891891        auto& resultButterfly = *resultArray->butterfly();
    892892        if (arrayType == ArrayWithDouble)
    893             memcpy(resultButterfly.contiguousDouble().data(), m_butterfly.get()->contiguousDouble().data() + startIndex, sizeof(JSValue) * count);
     893            memcpy(resultButterfly.contiguousDouble().data(), m_butterfly->contiguousDouble().data() + startIndex, sizeof(JSValue) * count);
    894894        else
    895             memcpy(resultButterfly.contiguous().data(), m_butterfly.get()->contiguous().data() + startIndex, sizeof(JSValue) * count);
     895            memcpy(resultButterfly.contiguous().data(), m_butterfly->contiguous().data() + startIndex, sizeof(JSValue) * count);
    896896        resultButterfly.setPublicLength(count);
    897897
     
    971971        // the start of the Butterfly, which needs to point at the first indexed property in the used
    972972        // portion of the vector.
    973         Butterfly* butterfly = m_butterfly.get()->shift(structure(), count);
     973        Butterfly* butterfly = m_butterfly->shift(structure(), count);
    974974        setButterfly(vm, butterfly);
    975975        storage = butterfly->arrayStorage();
     
    10161016    RELEASE_ASSERT(count > 0);
    10171017
    1018     Butterfly* butterfly = m_butterfly.get().getMayBeNull();
     1018    Butterfly* butterfly = m_butterfly.getMayBeNull();
    10191019   
    10201020    switch (indexingType()) {
     
    11721172    auto scope = DECLARE_THROW_SCOPE(vm);
    11731173
    1174     Butterfly* butterfly = m_butterfly.get().getMayBeNull();
     1174    Butterfly* butterfly = m_butterfly.getMayBeNull();
    11751175   
    11761176    switch (indexingType()) {
     
    11951195            return false;
    11961196        }
    1197         butterfly = m_butterfly.get().getMayBeNull();
     1197        butterfly = m_butterfly.getMayBeNull();
    11981198
    11991199        // We have to check for holes before we start moving things around so that we don't get halfway
     
    12391239            return false;
    12401240        }
    1241         butterfly = m_butterfly.get().getMayBeNull();
     1241        butterfly = m_butterfly.getMayBeNull();
    12421242       
    12431243        // We have to check for holes before we start moving things around so that we don't get halfway
     
    12821282    WriteBarrier<Unknown>* vector;
    12831283
    1284     Butterfly* butterfly = m_butterfly.get().getMayBeNull();
     1284    Butterfly* butterfly = m_butterfly.getMayBeNull();
    12851285   
    12861286    switch (indexingType()) {
     
    13551355    ASSERT(length == this->length());
    13561356
    1357     Butterfly* butterfly = m_butterfly.get().getMayBeNull();
     1357    Butterfly* butterfly = m_butterfly.getMayBeNull();
    13581358    switch (indexingType()) {
    13591359    case ArrayClass:
  • trunk/Source/JavaScriptCore/runtime/JSArray.h

    r219636 r220352  
    232232
    233233        unsigned vectorLength = Butterfly::optimalContiguousVectorLength(structure, initialLength);
    234         void* temp = vm.auxiliarySpace.tryAllocate(nullptr, Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)));
     234        void* temp = vm.jsValueGigacageAuxiliarySpace.tryAllocate(nullptr, Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)));
    235235        if (!temp)
    236236            return nullptr;
  • trunk/Source/JavaScriptCore/runtime/JSArrayBufferView.cpp

    r220118 r220352  
    6767        size_t size = sizeOf(length, elementSize);
    6868        if (size) {
    69             temp = vm.auxiliarySpace.tryAllocate(nullptr, size);
     69            temp = vm.primitiveGigacageAuxiliarySpace.tryAllocate(nullptr, size);
    7070            if (!temp)
    7171                return;
     
    9191   
    9292    size_t size = static_cast<size_t>(length) * static_cast<size_t>(elementSize);
    93     m_vector = Gigacage::tryMalloc(size);
     93    m_vector = Gigacage::tryMalloc(Gigacage::Primitive, size);
    9494    if (!m_vector)
    9595        return;
     
    193193    ASSERT(thisObject->m_mode == OversizeTypedArray || thisObject->m_mode == WastefulTypedArray);
    194194    if (thisObject->m_mode == OversizeTypedArray)
    195         Gigacage::free(thisObject->m_vector.get());
     195        Gigacage::free(Gigacage::Primitive, thisObject->m_vector.get());
    196196}
    197197
  • trunk/Source/JavaScriptCore/runtime/JSLock.cpp

    r220186 r220352  
    158158    m_vm->traps().notifyGrabAllLocks();
    159159   
    160     m_vm->fireGigacageEnabledIfNecessary();
     160    m_vm->firePrimitiveGigacageEnabledIfNecessary();
    161161
    162162#if ENABLE(SAMPLING_PROFILER)
  • trunk/Source/JavaScriptCore/runtime/JSObject.cpp

    r220165 r220352  
    464464    }
    465465
    466     Butterfly* butterfly = thisObject->m_butterfly.get().getMayBeNull();
     466    Butterfly* butterfly = thisObject->m_butterfly.getMayBeNull();
    467467    if (butterfly) {
    468468        WriteBarrier<Unknown>* data = nullptr;
     
    614614       
    615615    case ALL_ARRAY_STORAGE_INDEXING_TYPES: {
    616         ArrayStorage* storage = thisObject->m_butterfly.get()->arrayStorage();
     616        ArrayStorage* storage = thisObject->m_butterfly->arrayStorage();
    617617        if (i >= storage->length())
    618618            return false;
     
    876876    case NonArrayWithArrayStorage:
    877877    case ArrayWithArrayStorage: {
    878         ArrayStorage* storage = thisObject->m_butterfly.get()->arrayStorage();
     878        ArrayStorage* storage = thisObject->m_butterfly->arrayStorage();
    879879       
    880880        if (propertyName >= storage->vectorLength())
     
    898898    case NonArrayWithSlowPutArrayStorage:
    899899    case ArrayWithSlowPutArrayStorage: {
    900         ArrayStorage* storage = thisObject->m_butterfly.get()->arrayStorage();
     900        ArrayStorage* storage = thisObject->m_butterfly->arrayStorage();
    901901       
    902902        if (propertyName >= storage->vectorLength())
     
    980980        break;
    981981    case ALL_ARRAY_STORAGE_INDEXING_TYPES:
    982         enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, m_butterfly.get()->arrayStorage());
     982        enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, m_butterfly->arrayStorage());
    983983        break;
    984984       
     
    10121012    unsigned vectorLength = Butterfly::optimalContiguousVectorLength(propertyCapacity, length);
    10131013    Butterfly* newButterfly = Butterfly::createOrGrowArrayRight(
    1014         m_butterfly.get().getMayBeNull(), vm, this, structure, propertyCapacity, false, 0,
     1014        m_butterfly.getMayBeNull(), vm, this, structure, propertyCapacity, false, 0,
    10151015        sizeof(EncodedJSValue) * vectorLength);
    10161016    newButterfly->setPublicLength(length);
     
    11001100    ASSERT_UNUSED(oldType, !hasIndexedProperties(oldType));
    11011101
    1102     Butterfly* newButterfly = createArrayStorageButterfly(vm, this, oldStructure, length, vectorLength, m_butterfly.get().getMayBeNull());
     1102    Butterfly* newButterfly = createArrayStorageButterfly(vm, this, oldStructure, length, vectorLength, m_butterfly.getMayBeNull());
    11031103    ArrayStorage* result = newButterfly->arrayStorage();
    11041104    Structure* newStructure = Structure::nonPropertyTransition(vm, oldStructure, oldStructure->suggestedArrayStorageTransition());
     
    11181118    ASSERT(hasUndecided(indexingType()));
    11191119
    1120     Butterfly* butterfly = m_butterfly.get().getMayBeNull();
     1120    Butterfly* butterfly = m_butterfly.getMayBeNull();
    11211121    for (unsigned i = butterfly->vectorLength(); i--;)
    11221122        butterfly->contiguousInt32()[i].setWithoutWriteBarrier(JSValue());
    11231123
    11241124    setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateInt32));
    1125     return m_butterfly.get()->contiguousInt32();
     1125    return m_butterfly->contiguousInt32();
    11261126}
    11271127
     
    11301130    ASSERT(hasUndecided(indexingType()));
    11311131
    1132     Butterfly* butterfly = m_butterfly.get().get();
     1132    Butterfly* butterfly = m_butterfly.get();
    11331133    for (unsigned i = butterfly->vectorLength(); i--;)
    11341134        butterfly->contiguousDouble()[i] = PNaN;
    11351135   
    11361136    setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateDouble));
    1137     return m_butterfly.get()->contiguousDouble();
     1137    return m_butterfly->contiguousDouble();
    11381138}
    11391139
     
    11421142    ASSERT(hasUndecided(indexingType()));
    11431143
    1144     Butterfly* butterfly = m_butterfly.get().get();
     1144    Butterfly* butterfly = m_butterfly.get();
    11451145    for (unsigned i = butterfly->vectorLength(); i--;)
    11461146        butterfly->contiguous()[i].setWithoutWriteBarrier(JSValue());
     
    11481148    WTF::storeStoreFence();
    11491149    setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateContiguous));
    1150     return m_butterfly.get()->contiguous();
     1150    return m_butterfly->contiguous();
    11511151}
    11521152
     
    11541154{
    11551155    Structure* structure = this->structure(vm);
    1156     unsigned publicLength = m_butterfly.get()->publicLength();
     1156    unsigned publicLength = m_butterfly->publicLength();
    11571157    unsigned propertyCapacity = structure->outOfLineCapacity();
    11581158    unsigned propertySize = structure->outOfLineSize();
     
    11631163    memcpy(
    11641164        newButterfly->propertyStorage() - propertySize,
    1165         m_butterfly.get()->propertyStorage() - propertySize,
     1165        m_butterfly->propertyStorage() - propertySize,
    11661166        propertySize * sizeof(EncodedJSValue));
    11671167   
     
    11811181    ASSERT(hasUndecided(indexingType()));
    11821182
    1183     unsigned vectorLength = m_butterfly.get()->vectorLength();
     1183    unsigned vectorLength = m_butterfly->vectorLength();
    11841184    ArrayStorage* storage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
    11851185   
     
    12041204    ASSERT(hasInt32(indexingType()));
    12051205
    1206     Butterfly* butterfly = m_butterfly.get().get();
     1206    Butterfly* butterfly = m_butterfly.get();
    12071207    for (unsigned i = butterfly->vectorLength(); i--;) {
    12081208        WriteBarrier<Unknown>* current = &butterfly->contiguousInt32()[i];
     
    12191219   
    12201220    setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateDouble));
    1221     return m_butterfly.get()->contiguousDouble();
     1221    return m_butterfly->contiguousDouble();
    12221222}
    12231223
     
    12271227   
    12281228    setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateContiguous));
    1229     return m_butterfly.get()->contiguous();
     1229    return m_butterfly->contiguous();
    12301230}
    12311231
     
    12351235    ASSERT(hasInt32(indexingType()));
    12361236
    1237     unsigned vectorLength = m_butterfly.get()->vectorLength();
     1237    unsigned vectorLength = m_butterfly->vectorLength();
    12381238    ArrayStorage* newStorage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
    1239     Butterfly* butterfly = m_butterfly.get().get();
     1239    Butterfly* butterfly = m_butterfly.get();
    12401240    for (unsigned i = 0; i < vectorLength; i++) {
    12411241        JSValue v = butterfly->contiguous()[i].get();
     
    12621262    ASSERT(hasDouble(indexingType()));
    12631263
    1264     Butterfly* butterfly = m_butterfly.get().get();
     1264    Butterfly* butterfly = m_butterfly.get();
    12651265    for (unsigned i = butterfly->vectorLength(); i--;) {
    12661266        double* current = &butterfly->contiguousDouble()[i];
     
    12771277    WTF::storeStoreFence();
    12781278    setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateContiguous));
    1279     return m_butterfly.get()->contiguous();
     1279    return m_butterfly->contiguous();
    12801280}
    12811281
     
    12851285    ASSERT(hasDouble(indexingType()));
    12861286
    1287     unsigned vectorLength = m_butterfly.get()->vectorLength();
     1287    unsigned vectorLength = m_butterfly->vectorLength();
    12881288    ArrayStorage* newStorage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
    1289     Butterfly* butterfly = m_butterfly.get().get();
     1289    Butterfly* butterfly = m_butterfly.get();
    12901290    for (unsigned i = 0; i < vectorLength; i++) {
    12911291        double value = butterfly->contiguousDouble()[i];
     
    13161316    ASSERT(hasContiguous(indexingType()));
    13171317
    1318     unsigned vectorLength = m_butterfly.get()->vectorLength();
     1318    unsigned vectorLength = m_butterfly->vectorLength();
    13191319    ArrayStorage* newStorage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
    1320     Butterfly* butterfly = m_butterfly.get().get();
     1320    Butterfly* butterfly = m_butterfly.get();
    13211321    for (unsigned i = 0; i < vectorLength; i++) {
    13221322        JSValue v = butterfly->contiguous()[i].get();
     
    14091409void JSObject::setIndexQuicklyToUndecided(VM& vm, unsigned index, JSValue value)
    14101410{
    1411     ASSERT(index < m_butterfly.get()->publicLength());
    1412     ASSERT(index < m_butterfly.get()->vectorLength());
     1411    ASSERT(index < m_butterfly->publicLength());
     1412    ASSERT(index < m_butterfly->vectorLength());
    14131413    convertUndecidedForValue(vm, value);
    14141414    setIndexQuickly(vm, index, value);
     
    15781578       
    15791579    case ALL_ARRAY_STORAGE_INDEXING_TYPES:
    1580         return enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, m_butterfly.get()->arrayStorage());
     1580        return enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, m_butterfly->arrayStorage());
    15811581       
    15821582    default:
     
    18631863       
    18641864    case ALL_ARRAY_STORAGE_INDEXING_TYPES: {
    1865         ArrayStorage* storage = thisObject->m_butterfly.get()->arrayStorage();
     1865        ArrayStorage* storage = thisObject->m_butterfly->arrayStorage();
    18661866       
    18671867        if (i < storage->vectorLength()) {
     
    21692169           
    21702170        case ALL_ARRAY_STORAGE_INDEXING_TYPES: {
    2171             ArrayStorage* storage = object->m_butterfly.get()->arrayStorage();
     2171            ArrayStorage* storage = object->m_butterfly->arrayStorage();
    21722172           
    21732173            unsigned usedVectorLength = std::min(storage->length(), storage->vectorLength());
     
    23262326{
    23272327    VM& vm = exec->vm();
    2328     auto map = m_butterfly.get()->arrayStorage()->m_sparseMap.get();
     2328    auto map = m_butterfly->arrayStorage()->m_sparseMap.get();
    23292329
    23302330    if (descriptor.isDataDescriptor()) {
     
    23962396        notifyPresenceOfIndexedAccessors(vm);
    23972397
    2398     SparseArrayValueMap* map = m_butterfly.get()->arrayStorage()->m_sparseMap.get();
     2398    SparseArrayValueMap* map = m_butterfly->arrayStorage()->m_sparseMap.get();
    23992399    RELEASE_ASSERT(map);
    24002400   
     
    24282428
    24292429        putIndexedDescriptor(exec, entryInMap, descriptor, defaults);
    2430         Butterfly* butterfly = m_butterfly.get().get();
     2430        Butterfly* butterfly = m_butterfly.get();
    24312431        if (index >= butterfly->arrayStorage()->length())
    24322432            butterfly->arrayStorage()->setLength(index + 1);
     
    25602560    ASSERT(!indexingShouldBeSparse());
    25612561
    2562     Butterfly* butterfly = m_butterfly.get().get();
     2562    Butterfly* butterfly = m_butterfly.get();
    25632563   
    25642564    // For us to get here, the index is either greater than the public length, or greater than
     
    25822582        return false;
    25832583    }
    2584     butterfly = m_butterfly.get().get();
     2584    butterfly = m_butterfly.get();
    25852585
    25862586    RELEASE_ASSERT(i < butterfly->vectorLength());
     
    28702870    case ALL_INT32_INDEXING_TYPES: {
    28712871        if (attributes) {
    2872             if (i < m_butterfly.get()->vectorLength())
     2872            if (i < m_butterfly->vectorLength())
    28732873                return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, ensureArrayStorageExistsAndEnterDictionaryIndexingMode(vm));
    28742874            return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, convertInt32ToArrayStorage(vm));
     
    28842884    case ALL_DOUBLE_INDEXING_TYPES: {
    28852885        if (attributes) {
    2886             if (i < m_butterfly.get()->vectorLength())
     2886            if (i < m_butterfly->vectorLength())
    28872887                return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, ensureArrayStorageExistsAndEnterDictionaryIndexingMode(vm));
    28882888            return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, convertDoubleToArrayStorage(vm));
     
    29032903    case ALL_CONTIGUOUS_INDEXING_TYPES: {
    29042904        if (attributes) {
    2905             if (i < m_butterfly.get()->vectorLength())
     2905            if (i < m_butterfly->vectorLength())
    29062906                return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, ensureArrayStorageExistsAndEnterDictionaryIndexingMode(vm));
    29072907            return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, convertContiguousToArrayStorage(vm));
     
    29132913    case ALL_ARRAY_STORAGE_INDEXING_TYPES:
    29142914        if (attributes) {
    2915             if (i < m_butterfly.get()->vectorLength())
     2915            if (i < m_butterfly->vectorLength())
    29162916                return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, ensureArrayStorageExistsAndEnterDictionaryIndexingMode(vm));
    29172917        }
     
    30163016        if (ArrayStorage* storage = arrayStorageOrNull())
    30173017            indexBias = storage->m_indexBias;
    3018         vectorLength = m_butterfly.get()->vectorLength();
    3019         length = m_butterfly.get()->publicLength();
     3018        vectorLength = m_butterfly->vectorLength();
     3019        length = m_butterfly->publicLength();
    30203020    }
    30213021
     
    31333133bool JSObject::ensureLengthSlow(VM& vm, unsigned length)
    31343134{
    3135     Butterfly* butterfly = m_butterfly.get().get();
     3135    Butterfly* butterfly = m_butterfly.get();
    31363136   
    31373137    ASSERT(length <= MAX_STORAGE_VECTOR_LENGTH);
     
    31893189    ASSERT(length <= MAX_STORAGE_VECTOR_LENGTH);
    31903190    ASSERT(hasContiguous(indexingType()) || hasInt32(indexingType()) || hasDouble(indexingType()) || hasUndecided(indexingType()));
    3191     ASSERT(m_butterfly.get()->vectorLength() > length);
    3192     ASSERT(!m_butterfly.get()->indexingHeader()->preCapacity(structure()));
     3191    ASSERT(m_butterfly->vectorLength() > length);
     3192    ASSERT(!m_butterfly->indexingHeader()->preCapacity(structure()));
    31933193
    31943194    DeferGC deferGC(vm.heap);
    3195     Butterfly* newButterfly = m_butterfly.get()->resizeArray(vm, this, structure(), 0, ArrayStorage::sizeFor(length));
     3195    Butterfly* newButterfly = m_butterfly->resizeArray(vm, this, structure(), 0, ArrayStorage::sizeFor(length));
    31963196    newButterfly->setVectorLength(length);
    31973197    newButterfly->setPublicLength(length);
     
    32073207    // capacity, since we might have already mutated the structure in-place.
    32083208
    3209     return Butterfly::createOrGrowPropertyStorage(m_butterfly.get().getMayBeNull(), vm, this, structure(vm), oldSize, newSize);
     3209    return Butterfly::createOrGrowPropertyStorage(m_butterfly.getMayBeNull(), vm, this, structure(vm), oldSize, newSize);
    32103210}
    32113211
     
    35533553       
    35543554    case ALL_ARRAY_STORAGE_INDEXING_TYPES: {
    3555         ArrayStorage* storage = object->m_butterfly.get()->arrayStorage();
     3555        ArrayStorage* storage = object->m_butterfly->arrayStorage();
    35563556        if (storage->m_sparseMap.get())
    35573557            return 0;
  • trunk/Source/JavaScriptCore/runtime/JSObject.h

    r220165 r220352  
    2525#include "ArrayConventions.h"
    2626#include "ArrayStorage.h"
    27 #include "AuxiliaryBarrier.h"
    2827#include "Butterfly.h"
    2928#include "CPU.h"
     29#include "CagedBarrierPtr.h"
    3030#include "CallFrame.h"
    3131#include "ClassInfo.h"
     
    4444#include "JSString.h"
    4545#include "SparseArrayValueMap.h"
    46 #include <wtf/CagedPtr.h>
    4746#include <wtf/StdLibExtras.h>
    4847
     
    179178        if (!hasIndexedProperties(indexingType()))
    180179            return 0;
    181         return m_butterfly.get()->publicLength();
     180        return m_butterfly->publicLength();
    182181    }
    183182       
     
    186185        if (!hasIndexedProperties(indexingType()))
    187186            return 0;
    188         return m_butterfly.get()->vectorLength();
     187        return m_butterfly->vectorLength();
    189188    }
    190189   
     
    222221            case ALL_CONTIGUOUS_INDEXING_TYPES:
    223222            case ALL_ARRAY_STORAGE_INDEXING_TYPES:
    224                 return propertyName < m_butterfly.get()->vectorLength();
     223                return propertyName < m_butterfly->vectorLength();
    225224            default:
    226225                RELEASE_ASSERT_NOT_REACHED();
     
    256255    bool canGetIndexQuickly(unsigned i)
    257256    {
    258         Butterfly* butterfly = m_butterfly.get().getMayBeNull();
     257        Butterfly* butterfly = m_butterfly.getMayBeNull();
    259258        switch (indexingType()) {
    260259        case ALL_BLANK_INDEXING_TYPES:
     
    282281    JSValue getIndexQuickly(unsigned i)
    283282    {
    284         Butterfly* butterfly = m_butterfly.get().get();
     283        Butterfly* butterfly = m_butterfly.get();
    285284        switch (indexingType()) {
    286285        case ALL_INT32_INDEXING_TYPES:
     
    300299    JSValue tryGetIndexQuickly(unsigned i) const
    301300    {
    302         Butterfly* butterfly = m_butterfly.get().getMayBeNull();
     301        Butterfly* butterfly = m_butterfly.getMayBeNull();
    303302        switch (indexingType()) {
    304303        case ALL_BLANK_INDEXING_TYPES:
     
    354353    bool canSetIndexQuickly(unsigned i)
    355354    {
    356         Butterfly* butterfly = m_butterfly.get().getMayBeNull();
     355        Butterfly* butterfly = m_butterfly.getMayBeNull();
    357356        switch (indexingType()) {
    358357        case ALL_BLANK_INDEXING_TYPES:
     
    377376    void setIndexQuickly(VM& vm, unsigned i, JSValue v)
    378377    {
    379         Butterfly* butterfly = m_butterfly.get().get();
     378        Butterfly* butterfly = m_butterfly.get();
    380379        switch (indexingType()) {
    381380        case ALL_INT32_INDEXING_TYPES: {
     
    437436    {
    438437        VM& vm = scope.vm();
    439         Butterfly* butterfly = m_butterfly.get().get();
     438        Butterfly* butterfly = m_butterfly.get();
    440439        switch (indexingType) {
    441440        case ALL_UNDECIDED_INDEXING_TYPES: {
     
    494493    ALWAYS_INLINE void initializeIndexWithoutBarrier(ObjectInitializationScope&, unsigned i, JSValue v, IndexingType indexingType)
    495494    {
    496         Butterfly* butterfly = m_butterfly.get().get();
     495        Butterfly* butterfly = m_butterfly.get();
    497496        switch (indexingType) {
    498497        case ALL_UNDECIDED_INDEXING_TYPES: {
     
    543542            return false;
    544543        case ALL_ARRAY_STORAGE_INDEXING_TYPES:
    545             return !!m_butterfly.get()->arrayStorage()->m_sparseMap;
     544            return !!m_butterfly->arrayStorage()->m_sparseMap;
    546545        default:
    547546            RELEASE_ASSERT_NOT_REACHED();
     
    560559            return false;
    561560        case ALL_ARRAY_STORAGE_INDEXING_TYPES:
    562             return m_butterfly.get()->arrayStorage()->inSparseMode();
     561            return m_butterfly->arrayStorage()->inSparseMode();
    563562        default:
    564563            RELEASE_ASSERT_NOT_REACHED();
     
    670669    }
    671670       
    672     const Butterfly* butterfly() const { return m_butterfly.get().getMayBeNull(); }
    673     Butterfly* butterfly() { return m_butterfly.get().getMayBeNull(); }
     671    const Butterfly* butterfly() const { return m_butterfly.getMayBeNull(); }
     672    Butterfly* butterfly() { return m_butterfly.getMayBeNull(); }
    674673   
    675     ConstPropertyStorage outOfLineStorage() const { return m_butterfly.get()->propertyStorage(); }
    676     PropertyStorage outOfLineStorage() { return m_butterfly.get()->propertyStorage(); }
     674    ConstPropertyStorage outOfLineStorage() const { return m_butterfly->propertyStorage(); }
     675    PropertyStorage outOfLineStorage() { return m_butterfly->propertyStorage(); }
    677676
    678677    const WriteBarrierBase<Unknown>* locationForOffset(PropertyOffset offset) const
     
    808807    {
    809808        if (LIKELY(hasInt32(indexingType())))
    810             return m_butterfly.get()->contiguousInt32();
     809            return m_butterfly->contiguousInt32();
    811810           
    812811        return ensureInt32Slow(vm);
     
    820819    {
    821820        if (LIKELY(hasDouble(indexingType())))
    822             return m_butterfly.get()->contiguousDouble();
     821            return m_butterfly->contiguousDouble();
    823822           
    824823        return ensureDoubleSlow(vm);
     
    830829    {
    831830        if (LIKELY(hasContiguous(indexingType())))
    832             return m_butterfly.get()->contiguous();
     831            return m_butterfly->contiguous();
    833832           
    834833        return ensureContiguousSlow(vm);
     
    842841    {
    843842        if (LIKELY(hasAnyArrayStorage(indexingType())))
    844             return m_butterfly.get()->arrayStorage();
     843            return m_butterfly->arrayStorage();
    845844
    846845        return ensureArrayStorageSlow(vm);
     
    894893    {
    895894        ASSERT(hasAnyArrayStorage(indexingType()));
    896         return m_butterfly.get()->arrayStorage();
     895        return m_butterfly->arrayStorage();
    897896    }
    898897       
     
    903902        switch (indexingType()) {
    904903        case ALL_ARRAY_STORAGE_INDEXING_TYPES:
    905             return m_butterfly.get()->arrayStorage();
     904            return m_butterfly->arrayStorage();
    906905               
    907906        default:
     
    969968        ASSERT(hasContiguous(indexingType()) || hasInt32(indexingType()) || hasDouble(indexingType()) || hasUndecided(indexingType()));
    970969
    971         if (m_butterfly.get()->vectorLength() < length) {
     970        if (m_butterfly->vectorLength() < length) {
    972971            if (!ensureLengthSlow(vm, length))
    973972                return false;
    974973        }
    975974           
    976         if (m_butterfly.get()->publicLength() < length)
    977             m_butterfly.get()->setPublicLength(length);
     975        if (m_butterfly->publicLength() < length)
     976            m_butterfly->setPublicLength(length);
    978977        return true;
    979978    }
     
    10471046
    10481047protected:
    1049     AuxiliaryBarrier<CagedPtr<Butterfly>> m_butterfly;
     1048    CagedBarrierPtr<Gigacage::JSValue, Butterfly> m_butterfly;
    10501049#if USE(JSVALUE32_64)
    10511050private:
  • trunk/Source/JavaScriptCore/runtime/RegExpMatchesArray.h

    r215885 r220352  
    4242    JSGlobalObject* globalObject = structure->globalObject();
    4343    bool createUninitialized = globalObject->isOriginalArrayStructure(structure);
    44     void* temp = vm.auxiliarySpace.tryAllocate(deferralContext, Butterfly::totalSize(0, structure->outOfLineCapacity(), true, vectorLength * sizeof(EncodedJSValue)));
     44    void* temp = vm.jsValueGigacageAuxiliarySpace.tryAllocate(deferralContext, Butterfly::totalSize(0, structure->outOfLineCapacity(), true, vectorLength * sizeof(EncodedJSValue)));
    4545    if (UNLIKELY(!temp))
    4646        return nullptr;
  • trunk/Source/JavaScriptCore/runtime/VM.cpp

    r220322 r220352  
    166166#endif // USE(CF)
    167167    , heap(this, heapType)
    168     , auxiliarySpace("Auxiliary", heap, AllocatorAttributes(DoesNotNeedDestruction, HeapCell::Auxiliary), &GigacageAlignedMemoryAllocator::instance())
    169     , cellSpace("JSCell", heap, AllocatorAttributes(DoesNotNeedDestruction, HeapCell::JSCell), &FastMallocAlignedMemoryAllocator::instance())
    170     , destructibleCellSpace("Destructible JSCell", heap, AllocatorAttributes(NeedsDestruction, HeapCell::JSCell), &FastMallocAlignedMemoryAllocator::instance())
    171     , stringSpace("JSString", heap, &FastMallocAlignedMemoryAllocator::instance())
    172     , destructibleObjectSpace("JSDestructibleObject", heap, &FastMallocAlignedMemoryAllocator::instance())
    173     , eagerlySweptDestructibleObjectSpace("Eagerly Swept JSDestructibleObject", heap, &FastMallocAlignedMemoryAllocator::instance())
    174     , segmentedVariableObjectSpace("JSSegmentedVariableObjectSpace", heap, &FastMallocAlignedMemoryAllocator::instance())
     168    , fastMallocAllocator(std::make_unique<FastMallocAlignedMemoryAllocator>())
     169    , primitiveGigacageAllocator(std::make_unique<GigacageAlignedMemoryAllocator>(Gigacage::Primitive))
     170    , jsValueGigacageAllocator(std::make_unique<GigacageAlignedMemoryAllocator>(Gigacage::JSValue))
     171    , primitiveGigacageAuxiliarySpace("Primitive Gigacage Auxiliary", heap, AllocatorAttributes(DoesNotNeedDestruction, HeapCell::Auxiliary), primitiveGigacageAllocator.get())
     172    , jsValueGigacageAuxiliarySpace("JSValue Gigacage Auxiliary", heap, AllocatorAttributes(DoesNotNeedDestruction, HeapCell::Auxiliary), jsValueGigacageAllocator.get())
     173    , cellSpace("JSCell", heap, AllocatorAttributes(DoesNotNeedDestruction, HeapCell::JSCell), fastMallocAllocator.get())
     174    , destructibleCellSpace("Destructible JSCell", heap, AllocatorAttributes(NeedsDestruction, HeapCell::JSCell), fastMallocAllocator.get())
     175    , stringSpace("JSString", heap, fastMallocAllocator.get())
     176    , destructibleObjectSpace("JSDestructibleObject", heap, fastMallocAllocator.get())
     177    , eagerlySweptDestructibleObjectSpace("Eagerly Swept JSDestructibleObject", heap, fastMallocAllocator.get())
     178    , segmentedVariableObjectSpace("JSSegmentedVariableObjectSpace", heap, fastMallocAllocator.get())
    175179#if ENABLE(WEBASSEMBLY)
    176     , webAssemblyCodeBlockSpace("JSWebAssemblyCodeBlockSpace", heap, &FastMallocAlignedMemoryAllocator::instance())
     180    , webAssemblyCodeBlockSpace("JSWebAssemblyCodeBlockSpace", heap, fastMallocAllocator.get())
    177181#endif
    178182    , vmType(vmType)
     
    212216    , m_builtinExecutables(std::make_unique<BuiltinExecutables>(*this))
    213217    , m_typeProfilerEnabledCount(0)
    214     , m_gigacageEnabled(IsWatched)
     218    , m_primitiveGigacageEnabled(IsWatched)
    215219    , m_controlFlowProfilerEnabledCount(0)
    216220    , m_shadowChicken(std::make_unique<ShadowChicken>())
     
    291295#endif
    292296   
    293     Gigacage::addDisableCallback(gigacageDisabledCallback, this);
     297    Gigacage::addPrimitiveDisableCallback(primitiveGigacageDisabledCallback, this);
    294298
    295299    heap.notifyIsSafeToCollect();
     
    355359    auto destructionLocker = holdLock(s_destructionLock.read());
    356360   
    357     Gigacage::removeDisableCallback(gigacageDisabledCallback, this);
     361    Gigacage::removePrimitiveDisableCallback(primitiveGigacageDisabledCallback, this);
    358362    promiseDeferredTimer->stopRunningTasks();
    359363#if ENABLE(WEBASSEMBLY)
     
    424428}
    425429
    426 void VM::gigacageDisabledCallback(void* argument)
    427 {
    428     static_cast<VM*>(argument)->gigacageDisabled();
    429 }
    430 
    431 void VM::gigacageDisabled()
     430void VM::primitiveGigacageDisabledCallback(void* argument)
     431{
     432    static_cast<VM*>(argument)->primitiveGigacageDisabled();
     433}
     434
     435void VM::primitiveGigacageDisabled()
    432436{
    433437    if (m_apiLock->currentThreadIsHoldingLock()) {
    434         m_gigacageEnabled.fireAll(*this, "Gigacage disabled");
     438        m_primitiveGigacageEnabled.fireAll(*this, "Primitive gigacage disabled");
    435439        return;
    436440    }
     
    438442    // This is totally racy, and that's OK. The point is, it's up to the user to ensure that they pass the
    439443    // uncaged buffer in a nicely synchronized manner.
    440     m_needToFireGigacageEnabled = true;
     444    m_needToFirePrimitiveGigacageEnabled = true;
    441445}
    442446
  • trunk/Source/JavaScriptCore/runtime/VM.h

    r220291 r220352  
    6363#include <wtf/DoublyLinkedList.h>
    6464#include <wtf/Forward.h>
     65#include <wtf/Gigacage.h>
    6566#include <wtf/HashMap.h>
    6667#include <wtf/HashSet.h>
     
    9697class Exception;
    9798class ExceptionScope;
     99class FastMallocAlignedMemoryAllocator;
     100class GigacageAlignedMemoryAllocator;
    98101class HandleStack;
    99102class TypeProfiler;
     
    286289    Heap heap;
    287290   
    288     Subspace auxiliarySpace;
     291    std::unique_ptr<FastMallocAlignedMemoryAllocator> fastMallocAllocator;
     292    std::unique_ptr<GigacageAlignedMemoryAllocator> primitiveGigacageAllocator;
     293    std::unique_ptr<GigacageAlignedMemoryAllocator> jsValueGigacageAllocator;
     294   
     295    Subspace primitiveGigacageAuxiliarySpace; // Typed arrays, strings, bitvectors, etc go here.
     296    Subspace jsValueGigacageAuxiliarySpace; // Butterflies, arrays of JSValues, etc go here.
     297
     298    // We make cross-cutting assumptions about typed arrays being in the primitive Gigacage and butterflies
     299    // being in the JSValue gigacage. For some types, it's super obvious where they should go, and so we
     300    // can hardcode that fact. But sometimes it's not clear, so we abstract it by having a Gigacage::Kind
     301    // constant somewhere.
     302    // FIXME: Maybe it would be better if everyone abstracted this?
     303    // https://bugs.webkit.org/show_bug.cgi?id=175248
     304    ALWAYS_INLINE Subspace& gigacageAuxiliarySpace(Gigacage::Kind kind)
     305    {
     306        switch (kind) {
     307        case Gigacage::Primitive:
     308            return primitiveGigacageAuxiliarySpace;
     309        case Gigacage::JSValue:
     310            return jsValueGigacageAuxiliarySpace;
     311        }
     312        RELEASE_ASSERT_NOT_REACHED();
     313        return primitiveGigacageAuxiliarySpace;
     314    }
    289315   
    290316    // Whenever possible, use subspaceFor<CellType>(vm) to get one of these subspaces.
     
    525551    void setLastStackTop(void*);
    526552   
    527     void fireGigacageEnabledIfNecessary()
    528     {
    529         if (m_needToFireGigacageEnabled) {
    530             m_needToFireGigacageEnabled = false;
    531             m_gigacageEnabled.fireAll(*this, "Gigacage disabled asynchronously");
     553    void firePrimitiveGigacageEnabledIfNecessary()
     554    {
     555        if (m_needToFirePrimitiveGigacageEnabled) {
     556            m_needToFirePrimitiveGigacageEnabled = false;
     557            m_primitiveGigacageEnabled.fireAll(*this, "Primitive gigacage disabled asynchronously");
    532558        }
    533559    }
     
    634660    JS_EXPORT_PRIVATE void addImpureProperty(const String&);
    635661   
    636     InlineWatchpointSet& gigacageEnabled() { return m_gigacageEnabled; }
     662    InlineWatchpointSet& primitiveGigacageEnabled() { return m_primitiveGigacageEnabled; }
    637663
    638664    BuiltinExecutables* builtinExecutables() { return m_builtinExecutables.get(); }
     
    742768#endif
    743769   
    744     static void gigacageDisabledCallback(void*);
    745     void gigacageDisabled();
     770    static void primitiveGigacageDisabledCallback(void*);
     771    void primitiveGigacageDisabled();
    746772
    747773#if ENABLE(ASSEMBLER)
     
    788814    std::unique_ptr<TypeProfilerLog> m_typeProfilerLog;
    789815    unsigned m_typeProfilerEnabledCount;
    790     bool m_needToFireGigacageEnabled { false };
    791     InlineWatchpointSet m_gigacageEnabled;
     816    bool m_needToFirePrimitiveGigacageEnabled { false };
     817    InlineWatchpointSet m_primitiveGigacageEnabled;
    792818    FunctionHasExecutedCache m_functionHasExecutedCache;
    793819    std::unique_ptr<ControlFlowProfiler> m_controlFlowProfiler;
  • trunk/Source/JavaScriptCore/wasm/WasmMemory.cpp

    r220118 r220352  
    101101                return MemoryResult(nullptr, MemoryResult::SyncGCAndRetry);
    102102           
    103             void* result = Gigacage::tryAllocateVirtualPages(Memory::fastMappedBytes());
     103            void* result = Gigacage::tryAllocateVirtualPages(Gigacage::Primitive, Memory::fastMappedBytes());
    104104            if (!result)
    105105                return MemoryResult(nullptr, MemoryResult::SyncGCAndRetry);
     
    122122        {
    123123            auto holder = holdLock(m_lock);
    124             Gigacage::freeVirtualPages(basePtr, Memory::fastMappedBytes());
     124            Gigacage::freeVirtualPages(Gigacage::Primitive, basePtr, Memory::fastMappedBytes());
    125125            m_memories.removeFirst(basePtr);
    126126        }
     
    313313        return adoptRef(new Memory(initial, maximum));
    314314   
    315     void* slowMemory = Gigacage::tryAlignedMalloc(WTF::pageSize(), initialBytes);
     315    void* slowMemory = Gigacage::tryAlignedMalloc(Gigacage::Primitive, WTF::pageSize(), initialBytes);
    316316    if (!slowMemory) {
    317317        memoryManager().freePhysicalBytes(initialBytes);
     
    332332            break;
    333333        case MemoryMode::BoundsChecking:
    334             Gigacage::alignedFree(m_memory);
     334            Gigacage::alignedFree(Gigacage::Primitive, m_memory);
    335335            break;
    336336        }
     
    379379        RELEASE_ASSERT(maximum().bytes() != 0);
    380380       
    381         void* newMemory = Gigacage::tryAlignedMalloc(WTF::pageSize(), desiredSize);
     381        void* newMemory = Gigacage::tryAlignedMalloc(Gigacage::Primitive, WTF::pageSize(), desiredSize);
    382382        if (!newMemory)
    383383            return false;
     
    385385        memset(static_cast<char*>(newMemory) + m_size, 0, desiredSize - m_size);
    386386        if (m_memory)
    387             Gigacage::alignedFree(m_memory);
     387            Gigacage::alignedFree(Gigacage::Primitive, m_memory);
    388388        m_memory = newMemory;
    389389        m_mappedCapacity = desiredSize;
  • trunk/Source/WTF/ChangeLog

    r220336 r220352  
     12017-08-06  Filip Pizlo  <fpizlo@apple.com>
     2
     3        Primitive auxiliaries and JSValue auxiliaries should have separate gigacages
     4        https://bugs.webkit.org/show_bug.cgi?id=174919
     5
     6        Reviewed by Keith Miller.
     7       
     8        This mirrors the changes from bmalloc/Gigacage.h.
     9       
     10        Also it teaches CagedPtr how to reason about multiple gigacages.
     11
     12        * wtf/CagedPtr.h:
     13        (WTF::CagedPtr::get const):
     14        (WTF::CagedPtr::operator[] const):
     15        * wtf/Gigacage.cpp:
     16        (Gigacage::tryMalloc):
     17        (Gigacage::tryAllocateVirtualPages):
     18        (Gigacage::freeVirtualPages):
     19        (Gigacage::tryAlignedMalloc):
     20        (Gigacage::alignedFree):
     21        (Gigacage::free):
     22        * wtf/Gigacage.h:
     23        (Gigacage::disablePrimitiveGigacage):
     24        (Gigacage::addPrimitiveDisableCallback):
     25        (Gigacage::removePrimitiveDisableCallback):
     26        (Gigacage::name):
     27        (Gigacage::basePtr):
     28        (Gigacage::caged):
     29        (Gigacage::isCaged):
     30        (Gigacage::tryAlignedMalloc):
     31        (Gigacage::alignedFree):
     32        (Gigacage::free):
     33        (Gigacage::disableGigacage): Deleted.
     34        (Gigacage::addDisableCallback): Deleted.
     35        (Gigacage::removeDisableCallback): Deleted.
     36
    1372017-08-07  Brian Burg  <bburg@apple.com>
    238
  • trunk/Source/WTF/wtf/CagedPtr.h

    r220165 r220352  
    3030namespace WTF {
    3131
    32 template<typename T>
     32template<Gigacage::Kind passedKind, typename T>
    3333class CagedPtr {
    3434public:
     35    static constexpr Gigacage::Kind kind = passedKind;
     36   
    3537    CagedPtr(T* ptr = nullptr)
    3638        : m_ptr(ptr)
     
    4143    {
    4244        ASSERT(m_ptr);
    43         return Gigacage::caged(m_ptr);
     45        return Gigacage::caged(kind, m_ptr);
    4446    }
    4547   
     
    6870    T& operator*() const { return *get(); }
    6971    T* operator->() const { return get(); }
     72
     73    template<typename IndexType>
     74    T& operator[](IndexType index) const { return get()[index]; }
    7075   
    7176private:
  • trunk/Source/WTF/wtf/Gigacage.cpp

    r220118 r220352  
    3434
    3535extern "C" {
    36 const void* g_gigacageBasePtr;
     36void* const g_gigacageBasePtr;
    3737}
    3838
    3939namespace Gigacage {
    4040
    41 void* tryMalloc(size_t size)
     41void* tryMalloc(Kind, size_t size)
    4242{
    4343    auto result = tryFastMalloc(size);
     
    4848}
    4949
    50 void* tryAllocateVirtualPages(size_t size)
     50void* tryAllocateVirtualPages(Kind, size_t size)
    5151{
    5252    return OSAllocator::reserveUncommitted(size);
    5353}
    5454
    55 void freeVirtualPages(void* basePtr, size_t size)
     55void freeVirtualPages(Kind, void* basePtr, size_t size)
    5656{
    5757    OSAllocator::releaseDecommitted(basePtr, size);
     
    6868// https://bugs.webkit.org/show_bug.cgi?id=175035
    6969
    70 void* tryAlignedMalloc(size_t alignment, size_t size)
     70void* tryAlignedMalloc(Kind kind, size_t alignment, size_t size)
    7171{
    72     void* result = bmalloc::api::tryMemalign(alignment, size, bmalloc::HeapKind::Gigacage);
     72    void* result = bmalloc::api::tryMemalign(alignment, size, bmalloc::heapKind(kind));
    7373    WTF::compilerFence();
    7474    return result;
    7575}
    7676
    77 void alignedFree(void* p)
     77void alignedFree(Kind kind, void* p)
    7878{
    79     bmalloc::api::free(p, bmalloc::HeapKind::Gigacage);
     79    if (!p)
     80        return;
     81    RELEASE_ASSERT(isCaged(kind, p));
     82    bmalloc::api::free(p, bmalloc::heapKind(kind));
    8083    WTF::compilerFence();
    8184}
    8285
    83 void* tryMalloc(size_t size)
     86void* tryMalloc(Kind kind, size_t size)
    8487{
    85     void* result = bmalloc::api::tryMalloc(size, bmalloc::HeapKind::Gigacage);
     88    void* result = bmalloc::api::tryMalloc(size, bmalloc::heapKind(kind));
    8689    WTF::compilerFence();
    8790    return result;
    8891}
    8992
    90 void free(void* p)
     93void free(Kind kind, void* p)
    9194{
    92     bmalloc::api::free(p, bmalloc::HeapKind::Gigacage);
     95    if (!p)
     96        return;
     97    RELEASE_ASSERT(isCaged(kind, p));
     98    bmalloc::api::free(p, bmalloc::heapKind(kind));
    9399    WTF::compilerFence();
    94100}
    95101
    96 void* tryAllocateVirtualPages(size_t size)
     102void* tryAllocateVirtualPages(Kind kind, size_t size)
    97103{
    98     void* result = bmalloc::api::tryLargeMemalignVirtual(WTF::pageSize(), size, bmalloc::HeapKind::Gigacage);
     104    void* result = bmalloc::api::tryLargeMemalignVirtual(WTF::pageSize(), size, bmalloc::heapKind(kind));
    99105    WTF::compilerFence();
    100106    return result;
    101107}
    102108
    103 void freeVirtualPages(void* basePtr, size_t)
     109void freeVirtualPages(Kind kind, void* basePtr, size_t)
    104110{
    105     bmalloc::api::freeLargeVirtual(basePtr, bmalloc::HeapKind::Gigacage);
     111    if (!basePtr)
     112        return;
     113    RELEASE_ASSERT(isCaged(kind, basePtr));
     114    bmalloc::api::freeLargeVirtual(basePtr, bmalloc::heapKind(kind));
    106115    WTF::compilerFence();
    107116}
  • trunk/Source/WTF/wtf/Gigacage.h

    r220148 r220352  
    3333
    3434extern "C" {
    35 extern WTF_EXPORTDATA const void* g_gigacageBasePtr;
     35extern WTF_EXPORTDATA void* const g_gigacageBasePtr;
    3636}
    3737
    3838namespace Gigacage {
    3939
     40enum Kind {
     41    Primitive,
     42    JSValue
     43};
     44
    4045inline void ensureGigacage() { }
    41 inline void disableGigacage() { }
     46inline void disablePrimitiveGigacage() { }
    4247inline bool shouldBeEnabled() { return false; }
    4348
    44 inline void addDisableCallback(void (*)(void*), void*) { }
    45 inline void removeDisableCallback(void (*)(void*), void*) { }
     49inline void addPrimitiveDisableCallback(void (*)(void*), void*) { }
     50inline void removePrimitiveDisableCallback(void (*)(void*), void*) { }
     51
     52ALWAYS_INLINE const char* name(Kind kind)
     53{
     54    switch (kind) {
     55    case Primitive:
     56        return "Primitive";
     57    case JSValue:
     58        return "JSValue";
     59    }
     60    RELEASE_ASSERT_NOT_REACHED();
     61    return nullptr;
     62}
     63
     64ALWAYS_INLINE void* basePtr(Kind)
     65{
     66    return g_gigacageBasePtr;
     67}
    4668
    4769template<typename T>
    48 inline T* caged(T* ptr) { return ptr; }
     70inline T* caged(Kind, T* ptr) { return ptr; }
    4971
    50 inline bool isCaged(const void*) { return false; }
     72inline bool isCaged(Kind, const void*) { return false; }
    5173
    52 inline void* tryAlignedMalloc(size_t alignment, size_t size) { return tryFastAlignedMalloc(alignment, size); }
    53 inline void alignedFree(void* p) { fastAlignedFree(p); }
    54 WTF_EXPORT_PRIVATE void* tryMalloc(size_t size);
    55 inline void free(void* p) { fastFree(p); }
     74inline void* tryAlignedMalloc(Kind, size_t alignment, size_t size) { return tryFastAlignedMalloc(alignment, size); }
     75inline void alignedFree(Kind, void* p) { fastAlignedFree(p); }
     76WTF_EXPORT_PRIVATE void* tryMalloc(Kind, size_t size);
     77inline void free(Kind, void* p) { fastFree(p); }
    5678
    57 WTF_EXPORT_PRIVATE void* tryAllocateVirtualPages(size_t size);
    58 WTF_EXPORT_PRIVATE void freeVirtualPages(void* basePtr, size_t size);
     79WTF_EXPORT_PRIVATE void* tryAllocateVirtualPages(Kind, size_t size);
     80WTF_EXPORT_PRIVATE void freeVirtualPages(Kind, void* basePtr, size_t size);
    5981
    6082} // namespace Gigacage
     
    6486namespace Gigacage {
    6587
    66 WTF_EXPORT_PRIVATE void* tryAlignedMalloc(size_t alignment, size_t size);
    67 WTF_EXPORT_PRIVATE void alignedFree(void*);
    68 WTF_EXPORT_PRIVATE void* tryMalloc(size_t);
    69 WTF_EXPORT_PRIVATE void free(void*);
     88WTF_EXPORT_PRIVATE void* tryAlignedMalloc(Kind, size_t alignment, size_t size);
     89WTF_EXPORT_PRIVATE void alignedFree(Kind, void*);
     90WTF_EXPORT_PRIVATE void* tryMalloc(Kind, size_t);
     91WTF_EXPORT_PRIVATE void free(Kind, void*);
    7092
    71 WTF_EXPORT_PRIVATE void* tryAllocateVirtualPages(size_t size);
    72 WTF_EXPORT_PRIVATE void freeVirtualPages(void* basePtr, size_t size);
     93WTF_EXPORT_PRIVATE void* tryAllocateVirtualPages(Kind, size_t size);
     94WTF_EXPORT_PRIVATE void freeVirtualPages(Kind, void* basePtr, size_t size);
    7395
    7496} // namespace Gigacage
  • trunk/Source/WebCore/ChangeLog

    r220351 r220352  
     12017-08-06  Filip Pizlo  <fpizlo@apple.com>
     2
     3        Primitive auxiliaries and JSValue auxiliaries should have separate gigacages
     4        https://bugs.webkit.org/show_bug.cgi?id=174919
     5
     6        Reviewed by Keith Miller.
     7
     8        No new tests because no change in behavior.
     9       
     10        Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be
     11        specific about which Gigacage we're using.
     12
     13        * bindings/js/WebCoreJSClientData.cpp:
     14        (WebCore::JSVMClientData::JSVMClientData):
     15        * platform/graphics/cocoa/GPUBufferMetal.mm:
     16        (WebCore::GPUBuffer::GPUBuffer):
     17
    1182017-08-07  Basuke Suzuki  <Basuke.Suzuki@sony.com>
    219
  • trunk/Source/WebCore/bindings/js/WebCoreJSClientData.cpp

    r220291 r220352  
    4545    : m_builtinFunctions(vm)
    4646    , m_builtinNames(&vm)
    47     , m_outputConstraintSpace("WebCore Wrapper w/ Output Constraint", vm.heap, &FastMallocAlignedMemoryAllocator::instance())
    48     , m_globalObjectOutputConstraintSpace("WebCore Global Object w/ Output Constraint", vm.heap, &FastMallocAlignedMemoryAllocator::instance())
     47    , m_outputConstraintSpace("WebCore Wrapper w/ Output Constraint", vm.heap, vm.fastMallocAllocator.get())
     48    , m_globalObjectOutputConstraintSpace("WebCore Global Object w/ Output Constraint", vm.heap, vm.fastMallocAllocator.get())
    4949{
    5050}
  • trunk/Source/WebCore/platform/graphics/cocoa/GPUBufferMetal.mm

    r220145 r220352  
    4646    size_t pageSize = WTF::pageSize();
    4747    size_t pageAlignedSize = roundUpToMultipleOf(pageSize, data->byteLength());
    48     void* pageAlignedCopy = Gigacage::tryAlignedMalloc(pageSize, pageAlignedSize);
     48    void* pageAlignedCopy = Gigacage::tryAlignedMalloc(Gigacage::Primitive, pageSize, pageAlignedSize);
    4949    if (!pageAlignedCopy)
    5050        return;
    5151    memcpy(pageAlignedCopy, data->baseAddress(), data->byteLength());
    52     m_contents = ArrayBuffer::createFromBytes(pageAlignedCopy, data->byteLength(), [] (void* ptr) { Gigacage::alignedFree(ptr); });
     52    m_contents = ArrayBuffer::createFromBytes(pageAlignedCopy, data->byteLength(), [] (void* ptr) { Gigacage::alignedFree(Gigacage::Primitive, ptr); });
    5353    m_contents->ref();
    5454    ArrayBuffer* capturedContents = m_contents.get();
  • trunk/Source/WebKit/ChangeLog

    r220336 r220352  
     12017-08-06  Filip Pizlo  <fpizlo@apple.com>
     2
     3        Primitive auxiliaries and JSValue auxiliaries should have separate gigacages
     4        https://bugs.webkit.org/show_bug.cgi?id=174919
     5
     6        Reviewed by Keith Miller.
     7       
     8        The disable callback is all about the primitive gigacage.
     9
     10        * WebProcess/WebProcess.cpp:
     11        (WebKit::primitiveGigacageDisabled):
     12        (WebKit::m_webSQLiteDatabaseTracker):
     13        (WebKit::gigacageDisabled): Deleted.
     14
    1152017-08-07  Brian Burg  <bburg@apple.com>
    216
  • trunk/Source/WebKit/WebProcess/WebProcess.cpp

    r220148 r220352  
    147147namespace WebKit {
    148148
    149 static void gigacageDisabled(void*)
     149static void primitiveGigacageDisabled(void*)
    150150{
    151151    UNREACHABLE_FOR_PLATFORM();
     
    204204
    205205    if (Gigacage::shouldBeEnabled())
    206         Gigacage::addDisableCallback(gigacageDisabled, nullptr);
     206        Gigacage::addPrimitiveDisableCallback(primitiveGigacageDisabled, nullptr);
    207207}
    208208
  • trunk/Source/bmalloc/ChangeLog

    r220154 r220352  
     12017-08-06  Filip Pizlo  <fpizlo@apple.com>
     2
     3        Primitive auxiliaries and JSValue auxiliaries should have separate gigacages
     4        https://bugs.webkit.org/show_bug.cgi?id=174919
     5
     6        Reviewed by Keith Miller.
     7       
     8        This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of
     9        HeapKind, PrimitiveGigacage and JSValueGigacage.
     10       
     11        The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and
     12        NO_INLINE into BNO_INLINE.
     13
     14        * bmalloc.xcodeproj/project.pbxproj:
     15        * bmalloc/Allocator.cpp:
     16        (bmalloc::Allocator::refillAllocatorSlowCase):
     17        (bmalloc::Allocator::refillAllocator):
     18        (bmalloc::Allocator::allocateLarge):
     19        (bmalloc::Allocator::allocateLogSizeClass):
     20        * bmalloc/AsyncTask.h:
     21        * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h.
     22        * bmalloc/Cache.cpp:
     23        (bmalloc::Cache::tryAllocateSlowCaseNullCache):
     24        (bmalloc::Cache::allocateSlowCaseNullCache):
     25        (bmalloc::Cache::deallocateSlowCaseNullCache):
     26        (bmalloc::Cache::reallocateSlowCaseNullCache):
     27        * bmalloc/Deallocator.cpp:
     28        * bmalloc/Gigacage.cpp:
     29        (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks):
     30        (Gigacage::ensureGigacage):
     31        (Gigacage::disablePrimitiveGigacage):
     32        (Gigacage::addPrimitiveDisableCallback):
     33        (Gigacage::removePrimitiveDisableCallback):
     34        (Gigacage::Callbacks::Callbacks): Deleted.
     35        (Gigacage::disableGigacage): Deleted.
     36        (Gigacage::addDisableCallback): Deleted.
     37        (Gigacage::removeDisableCallback): Deleted.
     38        * bmalloc/Gigacage.h:
     39        (Gigacage::name):
     40        (Gigacage::basePtr):
     41        (Gigacage::forEachKind):
     42        (Gigacage::caged):
     43        (Gigacage::isCaged):
     44        * bmalloc/Heap.cpp:
     45        (bmalloc::Heap::Heap):
     46        (bmalloc::Heap::usingGigacage):
     47        (bmalloc::Heap::gigacageBasePtr):
     48        * bmalloc/Heap.h:
     49        * bmalloc/HeapKind.h:
     50        (bmalloc::isGigacage):
     51        (bmalloc::gigacageKind):
     52        (bmalloc::heapKind):
     53        * bmalloc/Inline.h: Removed.
     54        * bmalloc/Map.h:
     55        * bmalloc/PerProcess.h:
     56        (bmalloc::PerProcess<T>::getFastCase):
     57        (bmalloc::PerProcess<T>::get):
     58        (bmalloc::PerProcess<T>::getSlowCase):
     59        * bmalloc/PerThread.h:
     60        (bmalloc::PerThread<T>::getFastCase):
     61        * bmalloc/Vector.h:
     62        (bmalloc::Vector<T>::push):
     63        (bmalloc::Vector<T>::shrinkCapacity):
     64        (bmalloc::Vector<T>::growCapacity):
     65
    1662017-08-02  Filip Pizlo  <fpizlo@apple.com>
    267
  • trunk/Source/bmalloc/bmalloc.xcodeproj/project.pbxproj

    r220118 r220352  
    5252                14DD78C718F48D7500950702 /* BAssert.h in Headers */ = {isa = PBXBuildFile; fileRef = 1413E468189EEDE400546D68 /* BAssert.h */; settings = {ATTRIBUTES = (Private, ); }; };
    5353                14DD78C818F48D7500950702 /* FixedVector.h in Headers */ = {isa = PBXBuildFile; fileRef = 14D9DB4517F2447100EAAB79 /* FixedVector.h */; settings = {ATTRIBUTES = (Private, ); }; };
    54                 14DD78C918F48D7500950702 /* Inline.h in Headers */ = {isa = PBXBuildFile; fileRef = 1413E460189DCE1E00546D68 /* Inline.h */; settings = {ATTRIBUTES = (Private, ); }; };
     54                14DD78C918F48D7500950702 /* BInline.h in Headers */ = {isa = PBXBuildFile; fileRef = 1413E460189DCE1E00546D68 /* BInline.h */; settings = {ATTRIBUTES = (Private, ); }; };
    5555                14DD78CA18F48D7500950702 /* Mutex.h in Headers */ = {isa = PBXBuildFile; fileRef = 144DCED617A649D90093B2F2 /* Mutex.h */; settings = {ATTRIBUTES = (Private, ); }; };
    5656                14DD78CB18F48D7500950702 /* PerProcess.h in Headers */ = {isa = PBXBuildFile; fileRef = 14446A0717A61FA400F9EA1D /* PerProcess.h */; settings = {ATTRIBUTES = (Private, ); }; };
     
    9595                140FA00419CE4B6800FFD3C8 /* LineMetadata.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = LineMetadata.h; path = bmalloc/LineMetadata.h; sourceTree = "<group>"; };
    9696                14105E8318E14374003A106E /* ObjectType.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = ObjectType.cpp; path = bmalloc/ObjectType.cpp; sourceTree = "<group>"; };
    97                 1413E460189DCE1E00546D68 /* Inline.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = Inline.h; path = bmalloc/Inline.h; sourceTree = "<group>"; };
     97                1413E460189DCE1E00546D68 /* BInline.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = BInline.h; path = bmalloc/BInline.h; sourceTree = "<group>"; };
    9898                1413E462189DE1CD00546D68 /* BumpAllocator.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; lineEnding = 0; name = BumpAllocator.h; path = bmalloc/BumpAllocator.h; sourceTree = "<group>"; };
    9999                1413E468189EEDE400546D68 /* BAssert.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = BAssert.h; path = bmalloc/BAssert.h; sourceTree = "<group>"; };
     
    288288                                1413E468189EEDE400546D68 /* BAssert.h */,
    289289                                0F5BF1721F23C5710029D91D /* BExport.h */,
     290                                1413E460189DCE1E00546D68 /* BInline.h */,
    290291                                14C919C818FCC59F0028DB43 /* BPlatform.h */,
    291292                                14D9DB4517F2447100EAAB79 /* FixedVector.h */,
    292293                                0F5BF1461F22A8B10029D91D /* HeapKind.h */,
    293                                 1413E460189DCE1E00546D68 /* Inline.h */,
    294294                                141D9AFF1C8E51C0000ABBA0 /* List.h */,
    295295                                4426E27E1C838EE0008EB042 /* Logging.cpp */,
     
    354354                                1400274918F89C1300115C97 /* Heap.h in Headers */,
    355355                                0F5BF1491F22A8D80029D91D /* PerHeapKind.h in Headers */,
    356                                 14DD78C918F48D7500950702 /* Inline.h in Headers */,
     356                                14DD78C918F48D7500950702 /* BInline.h in Headers */,
    357357                                144C07F51C7B70260051BB6A /* LargeMap.h in Headers */,
    358358                                14C8992D1CC578330027A057 /* LargeRange.h in Headers */,
  • trunk/Source/bmalloc/bmalloc/Allocator.cpp

    r220118 r220352  
    150150}
    151151
    152 NO_INLINE void Allocator::refillAllocatorSlowCase(BumpAllocator& allocator, size_t sizeClass)
     152BNO_INLINE void Allocator::refillAllocatorSlowCase(BumpAllocator& allocator, size_t sizeClass)
    153153{
    154154    BumpRangeCache& bumpRangeCache = m_bumpRangeCaches[sizeClass];
     
    159159}
    160160
    161 INLINE void Allocator::refillAllocator(BumpAllocator& allocator, size_t sizeClass)
     161BINLINE void Allocator::refillAllocator(BumpAllocator& allocator, size_t sizeClass)
    162162{
    163163    BumpRangeCache& bumpRangeCache = m_bumpRangeCaches[sizeClass];
     
    167167}
    168168
    169 NO_INLINE void* Allocator::allocateLarge(size_t size)
     169BNO_INLINE void* Allocator::allocateLarge(size_t size)
    170170{
    171171    std::lock_guard<StaticMutex> lock(Heap::mutex());
     
    173173}
    174174
    175 NO_INLINE void* Allocator::allocateLogSizeClass(size_t size)
     175BNO_INLINE void* Allocator::allocateLogSizeClass(size_t size)
    176176{
    177177    size_t sizeClass = bmalloc::sizeClass(size);
  • trunk/Source/bmalloc/bmalloc/AsyncTask.h

    r217456 r220352  
    2828
    2929#include "BAssert.h"
    30 #include "Inline.h"
     30#include "BInline.h"
    3131#include "Mutex.h"
    3232#include "Sizes.h"
  • trunk/Source/bmalloc/bmalloc/BInline.h

    r220351 r220352  
    2424 */
    2525
    26 #ifndef Inline_h
    27 #define Inline_h
     26#ifndef BInline_h
     27#define BInline_h
    2828
    29 #define INLINE __attribute__((always_inline)) inline
     29#define BINLINE __attribute__((always_inline)) inline
    3030
    31 #define NO_INLINE __attribute__((noinline))
     31#define BNO_INLINE __attribute__((noinline))
    3232
    33 #endif // Inline_h
     33#endif // BInline_h
  • trunk/Source/bmalloc/bmalloc/Cache.cpp

    r220118 r220352  
    2424 */
    2525
     26#include "BInline.h"
    2627#include "Cache.h"
    2728#include "Heap.h"
    28 #include "Inline.h"
    2929#include "PerProcess.h"
    3030
     
    4747}
    4848
    49 NO_INLINE void* Cache::tryAllocateSlowCaseNullCache(HeapKind heapKind, size_t size)
     49BNO_INLINE void* Cache::tryAllocateSlowCaseNullCache(HeapKind heapKind, size_t size)
    5050{
    5151    return PerThread<PerHeapKind<Cache>>::getSlowCase()->at(heapKind).allocator().tryAllocate(size);
    5252}
    5353
    54 NO_INLINE void* Cache::allocateSlowCaseNullCache(HeapKind heapKind, size_t size)
     54BNO_INLINE void* Cache::allocateSlowCaseNullCache(HeapKind heapKind, size_t size)
    5555{
    5656    return PerThread<PerHeapKind<Cache>>::getSlowCase()->at(heapKind).allocator().allocate(size);
    5757}
    5858
    59 NO_INLINE void* Cache::allocateSlowCaseNullCache(HeapKind heapKind, size_t alignment, size_t size)
     59BNO_INLINE void* Cache::allocateSlowCaseNullCache(HeapKind heapKind, size_t alignment, size_t size)
    6060{
    6161    return PerThread<PerHeapKind<Cache>>::getSlowCase()->at(heapKind).allocator().allocate(alignment, size);
    6262}
    6363
    64 NO_INLINE void Cache::deallocateSlowCaseNullCache(HeapKind heapKind, void* object)
     64BNO_INLINE void Cache::deallocateSlowCaseNullCache(HeapKind heapKind, void* object)
    6565{
    6666    PerThread<PerHeapKind<Cache>>::getSlowCase()->at(heapKind).deallocator().deallocate(object);
    6767}
    6868
    69 NO_INLINE void* Cache::reallocateSlowCaseNullCache(HeapKind heapKind, void* object, size_t newSize)
     69BNO_INLINE void* Cache::reallocateSlowCaseNullCache(HeapKind heapKind, void* object, size_t newSize)
    7070{
    7171    return PerThread<PerHeapKind<Cache>>::getSlowCase()->at(heapKind).allocator().reallocate(object, newSize);
  • trunk/Source/bmalloc/bmalloc/Deallocator.cpp

    r220118 r220352  
    2525
    2626#include "BAssert.h"
     27#include "BInline.h"
    2728#include "Chunk.h"
    2829#include "Deallocator.h"
    2930#include "DebugHeap.h"
    3031#include "Heap.h"
    31 #include "Inline.h"
    3232#include "Object.h"
    3333#include "PerProcess.h"
  • trunk/Source/bmalloc/bmalloc/Gigacage.cpp

    r220148 r220352  
    3535// FIXME: Ask dyld to put this in its own page, and mprotect the page after we ensure the gigacage.
    3636// https://bugs.webkit.org/show_bug.cgi?id=174972
    37 void* g_gigacageBasePtr;
     37void* g_primitiveGigacageBasePtr;
     38void* g_jsValueGigacageBasePtr;
    3839
    3940using namespace bmalloc;
     
    5455};
    5556
    56 struct Callbacks {
    57     Callbacks(std::lock_guard<StaticMutex>&) { }
     57struct PrimitiveDisableCallbacks {
     58    PrimitiveDisableCallbacks(std::lock_guard<StaticMutex>&) { }
    5859   
    5960    Vector<Callback> callbacks;
     
    7071                return;
    7172           
    72             void* basePtr = tryVMAllocate(GIGACAGE_SIZE, GIGACAGE_SIZE + GIGACAGE_RUNWAY);
    73             if (!basePtr)
    74                 return;
    75            
    76             vmDeallocatePhysicalPages(basePtr, GIGACAGE_SIZE + GIGACAGE_RUNWAY);
    77            
    78             g_gigacageBasePtr = basePtr;
     73            forEachKind(
     74                [&] (Kind kind) {
     75                    // FIXME: Randomize where this goes.
     76                    // https://bugs.webkit.org/show_bug.cgi?id=175245
     77                    basePtr(kind) = tryVMAllocate(GIGACAGE_SIZE, GIGACAGE_SIZE + GIGACAGE_RUNWAY);
     78                    if (!basePtr(kind)) {
     79                        fprintf(stderr, "FATAL: Could not allocate %s gigacage.\n", name(kind));
     80                        BCRASH();
     81                    }
     82                   
     83                    vmDeallocatePhysicalPages(basePtr(kind), GIGACAGE_SIZE + GIGACAGE_RUNWAY);
     84                });
    7985        });
    8086#endif // GIGACAGE_ENABLED
    8187}
    8288
    83 void disableGigacage()
     89void disablePrimitiveGigacage()
    8490{
    8591    ensureGigacage();
    86     if (!g_gigacageBasePtr) {
     92    if (!g_primitiveGigacageBasePtr) {
    8793        // It was never enabled. That means that we never even saved any callbacks. Or, we had already disabled
    8894        // it before, and already called the callbacks.
     
    9096    }
    9197   
    92     Callbacks& callbacks = *PerProcess<Callbacks>::get();
    93     std::unique_lock<StaticMutex> lock(PerProcess<Callbacks>::mutex());
     98    PrimitiveDisableCallbacks& callbacks = *PerProcess<PrimitiveDisableCallbacks>::get();
     99    std::unique_lock<StaticMutex> lock(PerProcess<PrimitiveDisableCallbacks>::mutex());
    94100    for (Callback& callback : callbacks.callbacks)
    95101        callback.function(callback.argument);
    96102    callbacks.callbacks.shrink(0);
    97     g_gigacageBasePtr = nullptr;
     103    g_primitiveGigacageBasePtr = nullptr;
    98104}
    99105
    100 void addDisableCallback(void (*function)(void*), void* argument)
     106void addPrimitiveDisableCallback(void (*function)(void*), void* argument)
    101107{
    102108    ensureGigacage();
    103     if (!g_gigacageBasePtr) {
     109    if (!g_primitiveGigacageBasePtr) {
    104110        // It was already disabled or we were never able to enable it.
    105111        function(argument);
     
    107113    }
    108114   
    109     Callbacks& callbacks = *PerProcess<Callbacks>::get();
    110     std::unique_lock<StaticMutex> lock(PerProcess<Callbacks>::mutex());
     115    PrimitiveDisableCallbacks& callbacks = *PerProcess<PrimitiveDisableCallbacks>::get();
     116    std::unique_lock<StaticMutex> lock(PerProcess<PrimitiveDisableCallbacks>::mutex());
    111117    callbacks.callbacks.push(Callback(function, argument));
    112118}
    113119
    114 void removeDisableCallback(void (*function)(void*), void* argument)
     120void removePrimitiveDisableCallback(void (*function)(void*), void* argument)
    115121{
    116     Callbacks& callbacks = *PerProcess<Callbacks>::get();
    117     std::unique_lock<StaticMutex> lock(PerProcess<Callbacks>::mutex());
     122    PrimitiveDisableCallbacks& callbacks = *PerProcess<PrimitiveDisableCallbacks>::get();
     123    std::unique_lock<StaticMutex> lock(PerProcess<PrimitiveDisableCallbacks>::mutex());
    118124    for (size_t i = 0; i < callbacks.callbacks.size(); ++i) {
    119125        if (callbacks.callbacks[i].function == function
  • trunk/Source/bmalloc/bmalloc/Gigacage.h

    r220165 r220352  
    2828#include "BAssert.h"
    2929#include "BExport.h"
     30#include "BInline.h"
    3031#include "BPlatform.h"
    3132#include <inttypes.h>
     
    4546#endif
    4647
    47 extern "C" BEXPORT void* g_gigacageBasePtr;
     48extern "C" BEXPORT void* g_primitiveGigacageBasePtr;
     49extern "C" BEXPORT void* g_jsValueGigacageBasePtr;
    4850
    4951namespace Gigacage {
    5052
     53enum Kind {
     54    Primitive,
     55    JSValue
     56};
     57
    5158BEXPORT void ensureGigacage();
    5259
    53 BEXPORT void disableGigacage();
     60BEXPORT void disablePrimitiveGigacage();
    5461
    55 // This will call the disable callback immediately if the Gigacage is currently disabled.
    56 BEXPORT void addDisableCallback(void (*)(void*), void*);
    57 BEXPORT void removeDisableCallback(void (*)(void*), void*);
     62// This will call the disable callback immediately if the Primitive Gigacage is currently disabled.
     63BEXPORT void addPrimitiveDisableCallback(void (*)(void*), void*);
     64BEXPORT void removePrimitiveDisableCallback(void (*)(void*), void*);
     65
     66BINLINE const char* name(Kind kind)
     67{
     68    switch (kind) {
     69    case Primitive:
     70        return "Primitive";
     71    case JSValue:
     72        return "JSValue";
     73    }
     74    BCRASH();
     75    return nullptr;
     76}
     77
     78BINLINE void*& basePtr(Kind kind)
     79{
     80    switch (kind) {
     81    case Primitive:
     82        return g_primitiveGigacageBasePtr;
     83    case JSValue:
     84        return g_jsValueGigacageBasePtr;
     85    }
     86    BCRASH();
     87    return g_primitiveGigacageBasePtr;
     88}
     89
     90template<typename Func>
     91void forEachKind(const Func& func)
     92{
     93    func(Primitive);
     94    func(JSValue);
     95}
    5896
    5997template<typename T>
    60 T* caged(T* ptr)
     98BINLINE T* caged(Kind kind, T* ptr)
    6199{
    62100    BASSERT(ptr);
    63     void* gigacageBasePtr = g_gigacageBasePtr;
     101    void* gigacageBasePtr = basePtr(kind);
    64102    if (!gigacageBasePtr)
    65103        return ptr;
     
    69107}
    70108
    71 inline bool isCaged(const void* ptr)
     109BINLINE bool isCaged(Kind kind, const void* ptr)
    72110{
    73     return caged(ptr) == ptr;
     111    return caged(kind, ptr) == ptr;
    74112}
    75113
  • trunk/Source/bmalloc/bmalloc/Heap.cpp

    r220154 r220352  
    6060#if GIGACAGE_ENABLED
    6161        if (usingGigacage()) {
    62             RELEASE_BASSERT(g_gigacageBasePtr);
    63             m_largeFree.add(LargeRange(g_gigacageBasePtr, GIGACAGE_SIZE, 0));
     62            RELEASE_BASSERT(gigacageBasePtr());
     63            m_largeFree.add(LargeRange(gigacageBasePtr(), GIGACAGE_SIZE, 0));
    6464        }
    6565#endif
     
    7171bool Heap::usingGigacage()
    7272{
    73     return m_kind == HeapKind::Gigacage && g_gigacageBasePtr;
     73    return isGigacage(m_kind) && gigacageBasePtr();
     74}
     75
     76void* Heap::gigacageBasePtr()
     77{
     78    return Gigacage::basePtr(gigacageKind(m_kind));
    7479}
    7580
  • trunk/Source/bmalloc/bmalloc/Heap.h

    r220148 r220352  
    9090   
    9191    bool usingGigacage();
     92    void* gigacageBasePtr(); // May crash if !usingGigacage().
    9293   
    9394    void initializeLineMetadata();
  • trunk/Source/bmalloc/bmalloc/HeapKind.h

    r220118 r220352  
    2626#pragma once
    2727
     28#include "BAssert.h"
     29#include "BInline.h"
     30#include "Gigacage.h"
     31
    2832namespace bmalloc {
    2933
    3034enum class HeapKind {
    3135    Primary,
    32     Gigacage
     36    PrimitiveGigacage,
     37    JSValueGigacage
    3338};
    3439
    35 static constexpr unsigned numHeaps = 2;
     40static constexpr unsigned numHeaps = 3;
     41
     42BINLINE bool isGigacage(HeapKind heapKind)
     43{
     44    switch (heapKind) {
     45    case HeapKind::Primary:
     46        return false;
     47    case HeapKind::PrimitiveGigacage:
     48    case HeapKind::JSValueGigacage:
     49        return true;
     50    }
     51    BCRASH();
     52    return false;
     53}
     54
     55BINLINE Gigacage::Kind gigacageKind(HeapKind kind)
     56{
     57    switch (kind) {
     58    case HeapKind::Primary:
     59        BCRASH();
     60        return Gigacage::Primitive;
     61    case HeapKind::PrimitiveGigacage:
     62        return Gigacage::Primitive;
     63    case HeapKind::JSValueGigacage:
     64        return Gigacage::JSValue;
     65    }
     66    BCRASH();
     67    return Gigacage::Primitive;
     68}
     69
     70BINLINE HeapKind heapKind(Gigacage::Kind kind)
     71{
     72    switch (kind) {
     73    case Gigacage::Primitive:
     74        return HeapKind::PrimitiveGigacage;
     75    case Gigacage::JSValue:
     76        return HeapKind::JSValueGigacage;
     77    }
     78    BCRASH();
     79    return HeapKind::Primary;
     80}
    3681
    3782} // namespace bmalloc
  • trunk/Source/bmalloc/bmalloc/Map.h

    r219702 r220352  
    2727#define Map_h
    2828
    29 #include "Inline.h"
     29#include "BInline.h"
    3030#include "Sizes.h"
    3131#include "Vector.h"
  • trunk/Source/bmalloc/bmalloc/PerProcess.h

    r220079 r220352  
    2727#define PerProcess_h
    2828
    29 #include "Inline.h"
     29#include "BInline.h"
    3030#include "Sizes.h"
    3131#include "StaticMutex.h"
     
    7272
    7373template<typename T>
    74 INLINE T* PerProcess<T>::getFastCase()
     74BINLINE T* PerProcess<T>::getFastCase()
    7575{
    7676    return s_object.load(std::memory_order_consume);
     
    7878
    7979template<typename T>
    80 INLINE T* PerProcess<T>::get()
     80BINLINE T* PerProcess<T>::get()
    8181{
    8282    T* object = getFastCase();
     
    8787
    8888template<typename T>
    89 NO_INLINE T* PerProcess<T>::getSlowCase()
     89BNO_INLINE T* PerProcess<T>::getSlowCase()
    9090{
    9191    std::lock_guard<StaticMutex> lock(s_mutex);
  • trunk/Source/bmalloc/bmalloc/PerThread.h

    r220118 r220352  
    2727#define PerThread_h
    2828
     29#include "BInline.h"
    2930#include "BPlatform.h"
    30 #include "Inline.h"
    3131#include "PerHeapKind.h"
    3232#include "VMAllocate.h"
     
    116116
    117117template<typename T>
    118 INLINE T* PerThread<T>::getFastCase()
     118BINLINE T* PerThread<T>::getFastCase()
    119119{
    120120    return static_cast<T*>(PerThreadStorage<T>::get());
  • trunk/Source/bmalloc/bmalloc/Vector.h

    r201772 r220352  
    2727#define Vector_h
    2828
    29 #include "Inline.h"
     29#include "BInline.h"
    3030#include "VMAllocate.h"
    3131#include <cstddef>
     
    120120
    121121template<typename T>
    122 INLINE void Vector<T>::push(const T& value)
     122BINLINE void Vector<T>::push(const T& value)
    123123{
    124124    if (m_size == m_capacity)
     
    213213
    214214template<typename T>
    215 NO_INLINE void Vector<T>::shrinkCapacity()
     215BNO_INLINE void Vector<T>::shrinkCapacity()
    216216{
    217217    size_t newCapacity = max(initialCapacity(), m_capacity / shrinkFactor);
     
    220220
    221221template<typename T>
    222 NO_INLINE void Vector<T>::growCapacity()
     222BNO_INLINE void Vector<T>::growCapacity()
    223223{
    224224    size_t newCapacity = max(initialCapacity(), m_size * growFactor);
Note: See TracChangeset for help on using the changeset viewer.