Changeset 220352 in webkit
- Timestamp:
- Aug 7, 2017 2:31:49 PM (7 years ago)
- Location:
- trunk/Source
- Files:
-
- 1 added
- 54 edited
- 1 moved
Legend:
- Unmodified
- Added
- Removed
-
trunk/Source/JavaScriptCore/ChangeLog
r220346 r220352 1 2017-08-06 Filip Pizlo <fpizlo@apple.com> 2 3 Primitive auxiliaries and JSValue auxiliaries should have separate gigacages 4 https://bugs.webkit.org/show_bug.cgi?id=174919 5 6 Reviewed by Keith Miller. 7 8 This adapts JSC to there being two gigacages. 9 10 To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than 11 singletons. I don't think we were gaining anything by making them be singletons. 12 13 This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of 14 gigacages. We'll have one of those allocators per cage. 15 16 From there, this change teaches everyone who previously knew about cages that there are two cages. 17 This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is 18 easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's 19 not so obvious, so this change introduces some helpers to make it easy to define what cage you want 20 to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h 21 22 A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and 23 CagedPtr. This removes one layer of "get()" calls from a bunch of places. 24 25 * JavaScriptCore.xcodeproj/project.pbxproj: 26 * bytecode/AccessCase.cpp: 27 (JSC::AccessCase::generateImpl): 28 * dfg/DFGSpeculativeJIT.cpp: 29 (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): 30 (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): 31 (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): 32 (JSC::DFG::SpeculativeJIT::compileNewTypedArray): 33 (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): 34 * ftl/FTLLowerDFGToB3.cpp: 35 (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): 36 (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): 37 (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): 38 (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): 39 (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): 40 (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): 41 (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): 42 (JSC::FTL::DFG::LowerDFGToB3::caged): 43 * heap/FastMallocAlignedMemoryAllocator.cpp: 44 (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. 45 * heap/FastMallocAlignedMemoryAllocator.h: 46 * heap/GigacageAlignedMemoryAllocator.cpp: 47 (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): 48 (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): 49 (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): 50 (JSC::GigacageAlignedMemoryAllocator::dump const): 51 (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. 52 * heap/GigacageAlignedMemoryAllocator.h: 53 * jsc.cpp: 54 (primitiveGigacageDisabled): 55 (jscmain): 56 (gigacageDisabled): Deleted. 57 * llint/LowLevelInterpreter64.asm: 58 * runtime/ArrayBuffer.cpp: 59 (JSC::ArrayBufferContents::tryAllocate): 60 (JSC::ArrayBuffer::createAdopted): 61 (JSC::ArrayBuffer::createFromBytes): 62 * runtime/AuxiliaryBarrier.h: 63 * runtime/ButterflyInlines.h: 64 (JSC::Butterfly::createUninitialized): 65 (JSC::Butterfly::tryCreate): 66 (JSC::Butterfly::growArrayRight): 67 * runtime/CagedBarrierPtr.h: Added. 68 (JSC::CagedBarrierPtr::CagedBarrierPtr): 69 (JSC::CagedBarrierPtr::clear): 70 (JSC::CagedBarrierPtr::set): 71 (JSC::CagedBarrierPtr::get const): 72 (JSC::CagedBarrierPtr::getMayBeNull const): 73 (JSC::CagedBarrierPtr::operator== const): 74 (JSC::CagedBarrierPtr::operator!= const): 75 (JSC::CagedBarrierPtr::operator bool const): 76 (JSC::CagedBarrierPtr::setWithoutBarrier): 77 (JSC::CagedBarrierPtr::operator* const): 78 (JSC::CagedBarrierPtr::operator-> const): 79 (JSC::CagedBarrierPtr::operator[] const): 80 * runtime/DirectArguments.cpp: 81 (JSC::DirectArguments::overrideThings): 82 (JSC::DirectArguments::unmapArgument): 83 * runtime/DirectArguments.h: 84 (JSC::DirectArguments::isMappedArgument const): 85 * runtime/GenericArguments.h: 86 * runtime/GenericArgumentsInlines.h: 87 (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): 88 (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): 89 (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): 90 * runtime/HashMapImpl.cpp: 91 (JSC::HashMapImpl<HashMapBucket>::visitChildren): 92 * runtime/HashMapImpl.h: 93 (JSC::HashMapBuffer::create): 94 (JSC::HashMapImpl::buffer const): 95 (JSC::HashMapImpl::rehash): 96 * runtime/JSArray.cpp: 97 (JSC::JSArray::tryCreateUninitializedRestricted): 98 (JSC::JSArray::unshiftCountSlowCase): 99 (JSC::JSArray::setLength): 100 (JSC::JSArray::pop): 101 (JSC::JSArray::push): 102 (JSC::JSArray::fastSlice): 103 (JSC::JSArray::shiftCountWithArrayStorage): 104 (JSC::JSArray::shiftCountWithAnyIndexingType): 105 (JSC::JSArray::unshiftCountWithAnyIndexingType): 106 (JSC::JSArray::fillArgList): 107 (JSC::JSArray::copyToArguments): 108 * runtime/JSArray.h: 109 (JSC::JSArray::tryCreate): 110 * runtime/JSArrayBufferView.cpp: 111 (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): 112 (JSC::JSArrayBufferView::finalize): 113 * runtime/JSLock.cpp: 114 (JSC::JSLock::didAcquireLock): 115 * runtime/JSObject.cpp: 116 (JSC::JSObject::heapSnapshot): 117 (JSC::JSObject::getOwnPropertySlotByIndex): 118 (JSC::JSObject::putByIndex): 119 (JSC::JSObject::enterDictionaryIndexingMode): 120 (JSC::JSObject::createInitialIndexedStorage): 121 (JSC::JSObject::createArrayStorage): 122 (JSC::JSObject::convertUndecidedToInt32): 123 (JSC::JSObject::convertUndecidedToDouble): 124 (JSC::JSObject::convertUndecidedToContiguous): 125 (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): 126 (JSC::JSObject::convertUndecidedToArrayStorage): 127 (JSC::JSObject::convertInt32ToDouble): 128 (JSC::JSObject::convertInt32ToContiguous): 129 (JSC::JSObject::convertInt32ToArrayStorage): 130 (JSC::JSObject::convertDoubleToContiguous): 131 (JSC::JSObject::convertDoubleToArrayStorage): 132 (JSC::JSObject::convertContiguousToArrayStorage): 133 (JSC::JSObject::setIndexQuicklyToUndecided): 134 (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): 135 (JSC::JSObject::deletePropertyByIndex): 136 (JSC::JSObject::getOwnPropertyNames): 137 (JSC::JSObject::putIndexedDescriptor): 138 (JSC::JSObject::defineOwnIndexedProperty): 139 (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): 140 (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): 141 (JSC::JSObject::getNewVectorLength): 142 (JSC::JSObject::ensureLengthSlow): 143 (JSC::JSObject::reallocateAndShrinkButterfly): 144 (JSC::JSObject::allocateMoreOutOfLineStorage): 145 (JSC::JSObject::getEnumerableLength): 146 * runtime/JSObject.h: 147 (JSC::JSObject::getArrayLength const): 148 (JSC::JSObject::getVectorLength): 149 (JSC::JSObject::putDirectIndex): 150 (JSC::JSObject::canGetIndexQuickly): 151 (JSC::JSObject::getIndexQuickly): 152 (JSC::JSObject::tryGetIndexQuickly const): 153 (JSC::JSObject::canSetIndexQuickly): 154 (JSC::JSObject::setIndexQuickly): 155 (JSC::JSObject::initializeIndex): 156 (JSC::JSObject::initializeIndexWithoutBarrier): 157 (JSC::JSObject::hasSparseMap): 158 (JSC::JSObject::inSparseIndexingMode): 159 (JSC::JSObject::butterfly const): 160 (JSC::JSObject::butterfly): 161 (JSC::JSObject::outOfLineStorage const): 162 (JSC::JSObject::outOfLineStorage): 163 (JSC::JSObject::ensureInt32): 164 (JSC::JSObject::ensureDouble): 165 (JSC::JSObject::ensureContiguous): 166 (JSC::JSObject::ensureArrayStorage): 167 (JSC::JSObject::arrayStorage): 168 (JSC::JSObject::arrayStorageOrNull): 169 (JSC::JSObject::ensureLength): 170 * runtime/RegExpMatchesArray.h: 171 (JSC::tryCreateUninitializedRegExpMatchesArray): 172 * runtime/VM.cpp: 173 (JSC::VM::VM): 174 (JSC::VM::~VM): 175 (JSC::VM::primitiveGigacageDisabledCallback): 176 (JSC::VM::primitiveGigacageDisabled): 177 (JSC::VM::gigacageDisabledCallback): Deleted. 178 (JSC::VM::gigacageDisabled): Deleted. 179 * runtime/VM.h: 180 (JSC::VM::gigacageAuxiliarySpace): 181 (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): 182 (JSC::VM::primitiveGigacageEnabled): 183 (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. 184 (JSC::VM::gigacageEnabled): Deleted. 185 * wasm/WasmMemory.cpp: 186 (JSC::Wasm::Memory::create): 187 (JSC::Wasm::Memory::~Memory): 188 (JSC::Wasm::Memory::grow): 189 1 190 2017-08-07 Commit Queue <commit-queue@webkit.org> 2 191 -
trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj
r220318 r220352 899 899 0FEC3C5A1F33A48900F59B6C /* GigacageAlignedMemoryAllocator.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FEC3C581F33A48900F59B6C /* GigacageAlignedMemoryAllocator.cpp */; }; 900 900 0FEC3C5B1F33A48900F59B6C /* GigacageAlignedMemoryAllocator.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC3C591F33A48900F59B6C /* GigacageAlignedMemoryAllocator.h */; }; 901 0FEC3C601F379F5300F59B6C /* CagedBarrierPtr.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC3C5F1F379F5300F59B6C /* CagedBarrierPtr.h */; settings = {ATTRIBUTES = (Private, ); }; }; 901 902 0FEC84FE1BDACDAC0080FF74 /* B3ArgumentRegValue.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FEC84B41BDACDAC0080FF74 /* B3ArgumentRegValue.cpp */; }; 902 903 0FEC84FF1BDACDAC0080FF74 /* B3ArgumentRegValue.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84B51BDACDAC0080FF74 /* B3ArgumentRegValue.h */; }; … … 3497 3498 0FEC3C581F33A48900F59B6C /* GigacageAlignedMemoryAllocator.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = GigacageAlignedMemoryAllocator.cpp; sourceTree = "<group>"; }; 3498 3499 0FEC3C591F33A48900F59B6C /* GigacageAlignedMemoryAllocator.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = GigacageAlignedMemoryAllocator.h; sourceTree = "<group>"; }; 3500 0FEC3C5F1F379F5300F59B6C /* CagedBarrierPtr.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = CagedBarrierPtr.h; sourceTree = "<group>"; }; 3499 3501 0FEC84B41BDACDAC0080FF74 /* B3ArgumentRegValue.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = B3ArgumentRegValue.cpp; path = b3/B3ArgumentRegValue.cpp; sourceTree = "<group>"; }; 3500 3502 0FEC84B51BDACDAC0080FF74 /* B3ArgumentRegValue.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = B3ArgumentRegValue.h; path = b3/B3ArgumentRegValue.h; sourceTree = "<group>"; }; … … 6623 6625 0FB7F38B15ED8E3800F167B2 /* Butterfly.h */, 6624 6626 0FB7F38C15ED8E3800F167B2 /* ButterflyInlines.h */, 6627 0FEC3C5F1F379F5300F59B6C /* CagedBarrierPtr.h */, 6625 6628 BCA62DFE0E2826230004F30D /* CallData.cpp */, 6626 6629 145C507F0D9DF63B0088F6B9 /* CallData.h */, … … 8640 8643 A704D90617A0BAA8006BA554 /* DFGInPlaceAbstractState.h in Headers */, 8641 8644 0F2BDC21151E803B00CD8910 /* DFGInsertionSet.h in Headers */, 8645 0FEC3C601F379F5300F59B6C /* CagedBarrierPtr.h in Headers */, 8642 8646 0F300B7C18AB1B1400A6D72E /* DFGIntegerCheckCombiningPhase.h in Headers */, 8643 8647 0F898F321B27689F0083A33C /* DFGIntegerRangeOptimizationPhase.h in Headers */, -
trunk/Source/JavaScriptCore/bytecode/AccessCase.cpp
r220322 r220352 859 859 860 860 if (allocatingInline) { 861 MarkedAllocator* allocator = vm. auxiliarySpace.allocatorFor(newSize);861 MarkedAllocator* allocator = vm.jsValueGigacageAuxiliarySpace.allocatorFor(newSize); 862 862 863 863 if (!allocator) { -
trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp
r220219 r220352 110 110 111 111 if (size) { 112 if (MarkedAllocator* allocator = m_jit.vm()-> auxiliarySpace.allocatorFor(size)) {112 if (MarkedAllocator* allocator = m_jit.vm()->jsValueGigacageAuxiliarySpace.allocatorFor(size)) { 113 113 m_jit.move(TrustedImmPtr(allocator), scratchGPR); 114 114 m_jit.emitAllocate(storageGPR, allocator, scratchGPR, scratch2GPR, slowCases); … … 7869 7869 size_t size = initialOutOfLineCapacity * sizeof(JSValue); 7870 7870 7871 MarkedAllocator* allocator = m_jit.vm()-> auxiliarySpace.allocatorFor(size);7871 MarkedAllocator* allocator = m_jit.vm()->jsValueGigacageAuxiliarySpace.allocatorFor(size); 7872 7872 7873 7873 if (!allocator || node->transition()->previous->couldHaveIndexingHeader()) { … … 7914 7914 ASSERT(newSize == node->transition()->next->outOfLineCapacity() * sizeof(JSValue)); 7915 7915 7916 MarkedAllocator* allocator = m_jit.vm()-> auxiliarySpace.allocatorFor(newSize);7916 MarkedAllocator* allocator = m_jit.vm()->jsValueGigacageAuxiliarySpace.allocatorFor(newSize); 7917 7917 7918 7918 if (!allocator || node->transition()->previous->couldHaveIndexingHeader()) { … … 8495 8495 } 8496 8496 m_jit.emitAllocateVariableSized( 8497 storageGPR, m_jit.vm()-> auxiliarySpace, scratchGPR, scratchGPR,8497 storageGPR, m_jit.vm()->primitiveGigacageAuxiliarySpace, scratchGPR, scratchGPR, 8498 8498 scratchGPR2, slowCases); 8499 8499 … … 10113 10113 m_jit.add32(TrustedImm32(sizeof(IndexingHeader)), scratch1, scratch2); 10114 10114 m_jit.emitAllocateVariableSized( 10115 storageResultGPR, m_jit.vm()-> auxiliarySpace, scratch2, scratch1, scratch3, slowCases);10115 storageResultGPR, m_jit.vm()->jsValueGigacageAuxiliarySpace, scratch2, scratch1, scratch3, slowCases); 10116 10116 m_jit.addPtr(TrustedImm32(sizeof(IndexingHeader)), storageResultGPR); 10117 10117 -
trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp
r220219 r220352 3236 3236 LValue butterfly = m_out.loadPtr(lowCell(m_node->child1()), m_heaps.JSObject_butterfly); 3237 3237 if (m_node->op() != GetButterflyWithoutCaging) 3238 butterfly = caged( butterfly);3238 butterfly = caged(Gigacage::JSValue, butterfly); 3239 3239 setStorage(butterfly); 3240 3240 } … … 3273 3273 3274 3274 DFG_ASSERT(m_graph, m_node, isTypedView(m_node->arrayMode().typedArrayType())); 3275 setStorage(caged( m_out.loadPtr(cell, m_heaps.JSArrayBufferView_vector)));3275 setStorage(caged(Gigacage::Primitive, m_out.loadPtr(cell, m_heaps.JSArrayBufferView_vector))); 3276 3276 } 3277 3277 … … 5041 5041 } 5042 5042 5043 LValue allocator = allocatorForSize(vm(). auxiliarySpace, byteSize, slowCase);5043 LValue allocator = allocatorForSize(vm().primitiveGigacageAuxiliarySpace, byteSize, slowCase); 5044 5044 LValue storage = allocateHeapCell(allocator, slowCase); 5045 5045 … … 8870 8870 int32_t offsetOfFirstProperty = static_cast<int32_t>(offsetInButterfly(firstOutOfLineOffset)) * sizeof(EncodedJSValue); 8871 8871 ValueFromBlock outOfLineResult = m_out.anchor( 8872 m_out.load64(m_out.baseIndex(m_heaps.properties.atAnyNumber(), caged( storage), realIndex, ScaleEight, offsetOfFirstProperty)));8872 m_out.load64(m_out.baseIndex(m_heaps.properties.atAnyNumber(), caged(Gigacage::JSValue, storage), realIndex, ScaleEight, offsetOfFirstProperty))); 8873 8873 m_out.jump(continuation); 8874 8874 … … 9050 9050 9051 9051 LValue startOfStorage = allocateHeapCell( 9052 allocatorForSize(vm(). auxiliarySpace, butterflySize, slowPath),9052 allocatorForSize(vm().jsValueGigacageAuxiliarySpace, butterflySize, slowPath), 9053 9053 slowPath); 9054 9054 … … 10006 10006 10007 10007 size_t sizeInBytes = sizeInValues * sizeof(JSValue); 10008 MarkedAllocator* allocator = vm(). auxiliarySpace.allocatorFor(sizeInBytes);10008 MarkedAllocator* allocator = vm().jsValueGigacageAuxiliarySpace.allocatorFor(sizeInBytes); 10009 10009 LValue startOfStorage = allocateHeapCell(m_out.constIntPtr(allocator), slowPath); 10010 10010 ValueFromBlock fastButterfly = m_out.anchor( … … 11234 11234 payloadSize, m_out.constIntPtr(sizeof(IndexingHeader))); 11235 11235 11236 LValue allocator = allocatorForSize(vm(). auxiliarySpace, butterflySize, failCase);11236 LValue allocator = allocatorForSize(vm().jsValueGigacageAuxiliarySpace, butterflySize, failCase); 11237 11237 LValue startOfStorage = allocateHeapCell(allocator, failCase); 11238 11238 … … 11615 11615 } 11616 11616 11617 LValue caged(LValue ptr) 11618 { 11619 if (vm().gigacageEnabled().isStillValid()) { 11620 m_graph.watchpoints().addLazily(vm().gigacageEnabled()); 11621 11622 LValue basePtr = m_out.constIntPtr(g_gigacageBasePtr); 11623 LValue mask = m_out.constIntPtr(GIGACAGE_MASK); 11624 11625 // We don't have to worry about B3 messing up the bitAnd. Also, we want to get B3's excellent 11626 // codegen for 2-operand andq on x86-64. 11627 LValue masked = m_out.bitAnd(ptr, mask); 11628 11629 // But B3 will currently mess up the code generation of this add. Basically, any offset from what we 11630 // compute here will get reassociated and folded with g_gigacageBasePtr. There's a world in which 11631 // moveConstants() observes that it needs to reassociate in order to hoist the big constants. But 11632 // it's much easier to just block B3's badness here. That's what we do for now. 11633 PatchpointValue* patchpoint = m_out.patchpoint(pointerType()); 11634 patchpoint->appendSomeRegister(basePtr); 11635 patchpoint->appendSomeRegister(masked); 11636 patchpoint->setGenerator( 11637 [] (CCallHelpers& jit, const StackmapGenerationParams& params) { 11638 jit.addPtr(params[1].gpr(), params[2].gpr(), params[0].gpr()); 11639 }); 11640 patchpoint->effects = Effects::none(); 11641 return patchpoint; 11642 } 11643 11644 return ptr; 11617 LValue caged(Gigacage::Kind kind, LValue ptr) 11618 { 11619 if (kind == Gigacage::Primitive) { 11620 if (vm().primitiveGigacageEnabled().isStillValid()) 11621 m_graph.watchpoints().addLazily(vm().primitiveGigacageEnabled()); 11622 else 11623 return ptr; 11624 } 11625 11626 LValue basePtr = m_out.constIntPtr(Gigacage::basePtr(kind)); 11627 LValue mask = m_out.constIntPtr(GIGACAGE_MASK); 11628 11629 // We don't have to worry about B3 messing up the bitAnd. Also, we want to get B3's excellent 11630 // codegen for 2-operand andq on x86-64. 11631 LValue masked = m_out.bitAnd(ptr, mask); 11632 11633 // But B3 will currently mess up the code generation of this add. Basically, any offset from what we 11634 // compute here will get reassociated and folded with Gigacage::basePtr. There's a world in which 11635 // moveConstants() observes that it needs to reassociate in order to hoist the big constants. But 11636 // it's much easier to just block B3's badness here. That's what we do for now. 11637 PatchpointValue* patchpoint = m_out.patchpoint(pointerType()); 11638 patchpoint->appendSomeRegister(basePtr); 11639 patchpoint->appendSomeRegister(masked); 11640 patchpoint->setGenerator( 11641 [] (CCallHelpers& jit, const StackmapGenerationParams& params) { 11642 jit.addPtr(params[1].gpr(), params[2].gpr(), params[0].gpr()); 11643 }); 11644 patchpoint->effects = Effects::none(); 11645 return patchpoint; 11645 11646 } 11646 11647 -
trunk/Source/JavaScriptCore/heap/FastMallocAlignedMemoryAllocator.cpp
r220291 r220352 32 32 namespace JSC { 33 33 34 FastMallocAlignedMemoryAllocator& FastMallocAlignedMemoryAllocator::instance()35 {36 static FastMallocAlignedMemoryAllocator* result;37 static std::once_flag onceFlag;38 std::call_once(39 onceFlag,40 [] {41 result = new FastMallocAlignedMemoryAllocator();42 });43 return *result;44 }45 46 34 FastMallocAlignedMemoryAllocator::FastMallocAlignedMemoryAllocator() 47 35 { -
trunk/Source/JavaScriptCore/heap/FastMallocAlignedMemoryAllocator.h
r220291 r220352 32 32 class FastMallocAlignedMemoryAllocator : public AlignedMemoryAllocator { 33 33 public: 34 JS_EXPORT_PRIVATE static FastMallocAlignedMemoryAllocator& instance(); 35 34 FastMallocAlignedMemoryAllocator(); 36 35 ~FastMallocAlignedMemoryAllocator(); 37 36 … … 40 39 41 40 void dump(PrintStream&) const override; 42 43 private:44 FastMallocAlignedMemoryAllocator();45 41 }; 46 42 -
trunk/Source/JavaScriptCore/heap/GigacageAlignedMemoryAllocator.cpp
r220291 r220352 27 27 #include "GigacageAlignedMemoryAllocator.h" 28 28 29 #include <mutex>30 #include <wtf/Gigacage.h>31 32 29 namespace JSC { 33 30 34 GigacageAlignedMemoryAllocator& GigacageAlignedMemoryAllocator::instance() 35 { 36 static GigacageAlignedMemoryAllocator* result; 37 static std::once_flag onceFlag; 38 std::call_once( 39 onceFlag, 40 [] { 41 result = new GigacageAlignedMemoryAllocator(); 42 }); 43 return *result; 44 } 45 46 GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator() 31 GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator(Gigacage::Kind kind) 32 : m_kind(kind) 47 33 { 48 34 } … … 54 40 void* GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory(size_t alignment, size_t size) 55 41 { 56 return Gigacage::tryAlignedMalloc( alignment, size);42 return Gigacage::tryAlignedMalloc(m_kind, alignment, size); 57 43 } 58 44 59 45 void GigacageAlignedMemoryAllocator::freeAlignedMemory(void* basePtr) 60 46 { 61 Gigacage::alignedFree( basePtr);47 Gigacage::alignedFree(m_kind, basePtr); 62 48 } 63 49 64 50 void GigacageAlignedMemoryAllocator::dump(PrintStream& out) const 65 51 { 66 out.print( "Gigacage");52 out.print(Gigacage::name(m_kind), "Gigacage"); 67 53 } 68 54 -
trunk/Source/JavaScriptCore/heap/GigacageAlignedMemoryAllocator.h
r220291 r220352 27 27 28 28 #include "AlignedMemoryAllocator.h" 29 #include <wtf/Gigacage.h> 29 30 30 31 namespace JSC { … … 32 33 class GigacageAlignedMemoryAllocator : public AlignedMemoryAllocator { 33 34 public: 34 // FIXME: This shouldn't be a singleton. There should be different instances for primaries, JSValues, 35 // and other things. 36 // https://bugs.webkit.org/show_bug.cgi?id=174919 37 static GigacageAlignedMemoryAllocator& instance(); 38 35 GigacageAlignedMemoryAllocator(Gigacage::Kind); 39 36 ~GigacageAlignedMemoryAllocator(); 40 37 … … 45 42 46 43 private: 47 Gigacage AlignedMemoryAllocator();44 Gigacage::Kind m_kind; 48 45 }; 49 46 -
trunk/Source/JavaScriptCore/jsc.cpp
r220148 r220352 3803 3803 } 3804 3804 3805 static void gigacageDisabled(void*)3806 { 3807 dataLog(" Gigacage disabled! Aborting.\n");3805 static void primitiveGigacageDisabled(void*) 3806 { 3807 dataLog("Primitive gigacage disabled! Aborting.\n"); 3808 3808 UNREACHABLE_FOR_PLATFORM(); 3809 3809 } … … 3828 3828 #endif 3829 3829 if (Gigacage::shouldBeEnabled()) 3830 Gigacage::add DisableCallback(gigacageDisabled, nullptr);3830 Gigacage::addPrimitiveDisableCallback(primitiveGigacageDisabled, nullptr); 3831 3831 3832 3832 int result; -
trunk/Source/JavaScriptCore/llint/LowLevelInterpreter64.asm
r220184 r220352 377 377 end 378 378 379 macro loadCaged( source, dest, scratch)379 macro loadCaged(basePtr, source, dest, scratch) 380 380 loadp source, dest 381 381 if GIGACAGE_ENABLED and not C_LOOP 382 loadp _g_gigacageBasePtr, scratch382 loadp basePtr, scratch 383 383 btpz scratch, .done 384 384 andp constexpr GIGACAGE_MASK, dest … … 1210 1210 macro loadPropertyAtVariableOffset(propertyOffsetAsInt, objectAndStorage, value) 1211 1211 bilt propertyOffsetAsInt, firstOutOfLineOffset, .isInline 1212 loadCaged( JSObject::m_butterfly[objectAndStorage], objectAndStorage, value)1212 loadCaged(_g_jsValueGigacageBasePtr, JSObject::m_butterfly[objectAndStorage], objectAndStorage, value) 1213 1213 negi propertyOffsetAsInt 1214 1214 sxi2q propertyOffsetAsInt, propertyOffsetAsInt … … 1223 1223 macro storePropertyAtVariableOffset(propertyOffsetAsInt, objectAndStorage, value, scratch) 1224 1224 bilt propertyOffsetAsInt, firstOutOfLineOffset, .isInline 1225 loadCaged( JSObject::m_butterfly[objectAndStorage], objectAndStorage, scratch)1225 loadCaged(_g_jsValueGigacageBasePtr, JSObject::m_butterfly[objectAndStorage], objectAndStorage, scratch) 1226 1226 negi propertyOffsetAsInt 1227 1227 sxi2q propertyOffsetAsInt, propertyOffsetAsInt … … 1299 1299 btiz t2, IndexingShapeMask, .opGetArrayLengthSlow 1300 1300 loadisFromInstruction(1, t1) 1301 loadCaged( JSObject::m_butterfly[t3], t0, t2)1301 loadCaged(_g_jsValueGigacageBasePtr, JSObject::m_butterfly[t3], t0, t2) 1302 1302 loadi -sizeof IndexingHeader + IndexingHeader::u.lengths.publicLength[t0], t0 1303 1303 bilt t0, 0, .opGetArrayLengthSlow … … 1482 1482 loadConstantOrVariableInt32(t3, t1, .opGetByValSlow) 1483 1483 sxi2q t1, t1 1484 loadCaged( JSObject::m_butterfly[t0], t3, t5)1484 loadCaged(_g_jsValueGigacageBasePtr, JSObject::m_butterfly[t0], t3, t5) 1485 1485 andi IndexingShapeMask, t2 1486 1486 bieq t2, Int32Shape, .opGetByValIsContiguous … … 1529 1529 1530 1530 # Sweet, now we know that we have a typed array. Do some basic things now. 1531 loadCaged( JSArrayBufferView::m_vector[t0], t3, t5)1531 loadCaged(_g_primitiveGigacageBasePtr, JSArrayBufferView::m_vector[t0], t3, t5) 1532 1532 biaeq t1, JSArrayBufferView::m_length[t0], .opGetByValSlow 1533 1533 … … 1620 1620 loadConstantOrVariableInt32(t0, t3, .opPutByValSlow) 1621 1621 sxi2q t3, t3 1622 loadCaged( JSObject::m_butterfly[t1], t0, t5)1622 loadCaged(_g_jsValueGigacageBasePtr, JSObject::m_butterfly[t1], t0, t5) 1623 1623 andi IndexingShapeMask, t2 1624 1624 bineq t2, Int32Shape, .opPutByValNotInt32 -
trunk/Source/JavaScriptCore/runtime/ArrayBuffer.cpp
r220330 r220352 107 107 if (!size) 108 108 size = 1; // Make sure malloc actually allocates something, but not too much. We use null to mean that the buffer is neutered. 109 m_data = Gigacage::tryMalloc( size);109 m_data = Gigacage::tryMalloc(Gigacage::Primitive, size); 110 110 if (!m_data) { 111 111 reset(); … … 117 117 118 118 m_sizeInBytes = numElements * elementByteSize; 119 m_destructor = [] (void* p) { Gigacage::free( p); };119 m_destructor = [] (void* p) { Gigacage::free(Gigacage::Primitive, p); }; 120 120 } 121 121 … … 188 188 Ref<ArrayBuffer> ArrayBuffer::createAdopted(const void* data, unsigned byteLength) 189 189 { 190 return createFromBytes(data, byteLength, [] (void* p) { Gigacage::free( p); });190 return createFromBytes(data, byteLength, [] (void* p) { Gigacage::free(Gigacage::Primitive, p); }); 191 191 } 192 192 … … 199 199 Ref<ArrayBuffer> ArrayBuffer::createFromBytes(const void* data, unsigned byteLength, ArrayBufferDestructorFunction&& destructor) 200 200 { 201 if (data && byteLength && !Gigacage::isCaged( data))202 Gigacage::disable Gigacage();201 if (data && byteLength && !Gigacage::isCaged(Gigacage::Primitive, data)) 202 Gigacage::disablePrimitiveGigacage(); 203 203 204 204 ArrayBufferContents contents(const_cast<void*>(data), byteLength, WTFMove(destructor)); -
trunk/Source/JavaScriptCore/runtime/AuxiliaryBarrier.h
r206525 r220352 37 37 class AuxiliaryBarrier { 38 38 public: 39 typedef T Type; 40 39 41 AuxiliaryBarrier(): m_value() { } 40 42 -
trunk/Source/JavaScriptCore/runtime/ButterflyInlines.h
r211043 r220352 63 63 { 64 64 size_t size = totalSize(preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes); 65 void* base = vm. auxiliarySpace.allocate(size);65 void* base = vm.jsValueGigacageAuxiliarySpace.allocate(size); 66 66 Butterfly* result = fromBase(base, preCapacity, propertyCapacity); 67 67 return result; … … 71 71 { 72 72 size_t size = totalSize(preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes); 73 void* base = vm. auxiliarySpace.tryAllocate(size);73 void* base = vm.jsValueGigacageAuxiliarySpace.tryAllocate(size); 74 74 if (!base) 75 75 return nullptr; … … 149 149 size_t oldSize = totalSize(0, propertyCapacity, hadIndexingHeader, oldIndexingPayloadSizeInBytes); 150 150 size_t newSize = totalSize(0, propertyCapacity, true, newIndexingPayloadSizeInBytes); 151 void* newBase = vm. auxiliarySpace.tryAllocate(newSize);151 void* newBase = vm.jsValueGigacageAuxiliarySpace.tryAllocate(newSize); 152 152 if (!newBase) 153 153 return nullptr; -
trunk/Source/JavaScriptCore/runtime/DirectArguments.cpp
r217108 r220352 119 119 putDirect(vm, vm.propertyNames->iteratorSymbol, globalObject()->arrayProtoValuesFunction(), DontEnum); 120 120 121 void* backingStore = vm. auxiliarySpace.tryAllocate(mappedArgumentsSize());121 void* backingStore = vm.gigacageAuxiliarySpace(m_mappedArguments.kind).tryAllocate(mappedArgumentsSize()); 122 122 RELEASE_ASSERT(backingStore); 123 123 bool* overrides = static_cast<bool*>(backingStore); … … 136 136 { 137 137 overrideThingsIfNecessary(vm); 138 m_mappedArguments .get()[index] = true;138 m_mappedArguments[index] = true; 139 139 } 140 140 -
trunk/Source/JavaScriptCore/runtime/DirectArguments.h
r215345 r220352 1 1 /* 2 * Copyright (C) 2015-201 6Apple Inc. All rights reserved.2 * Copyright (C) 2015-2017 Apple Inc. All rights reserved. 3 3 * 4 4 * Redistribution and use in source and binary forms, with or without … … 26 26 #pragma once 27 27 28 #include " AuxiliaryBarrier.h"28 #include "CagedBarrierPtr.h" 29 29 #include "DirectArgumentsOffset.h" 30 30 #include "GenericArguments.h" 31 #include <wtf/CagedPtr.h> 31 32 32 33 namespace JSC { … … 74 75 bool isMappedArgument(uint32_t i) const 75 76 { 76 return i < m_length && (!m_mappedArguments || !m_mappedArguments .get()[i]);77 return i < m_length && (!m_mappedArguments || !m_mappedArguments[i]); 77 78 } 78 79 … … 165 166 uint32_t m_length; // Always the actual length of captured arguments and never what was stored into the length property. 166 167 uint32_t m_minCapacity; // The max of this and length determines the capacity of this object. It may be the actual capacity, or maybe something smaller. We arrange it this way to be kind to the JITs. 167 AuxiliaryBarrier<bool*> m_mappedArguments; // If non-null, it means that length, callee, and caller are fully materialized properties.168 CagedBarrierPtr<Gigacage::Primitive, bool> m_mappedArguments; // If non-null, it means that length, callee, and caller are fully materialized properties. 168 169 }; 169 170 -
trunk/Source/JavaScriptCore/runtime/GenericArguments.h
r210146 r220352 1 1 /* 2 * Copyright (C) 2015 Apple Inc. All rights reserved.2 * Copyright (C) 2015-2017 Apple Inc. All rights reserved. 3 3 * 4 4 * Redistribution and use in source and binary forms, with or without … … 26 26 #pragma once 27 27 28 #include "CagedBarrierPtr.h" 28 29 #include "JSObject.h" 29 30 … … 61 62 void copyToArguments(ExecState*, VirtualRegister firstElementDest, unsigned offset, unsigned length); 62 63 63 AuxiliaryBarrier<bool*> m_modifiedArgumentsDescriptor;64 CagedBarrierPtr<Gigacage::Primitive, bool> m_modifiedArgumentsDescriptor; 64 65 }; 65 66 -
trunk/Source/JavaScriptCore/runtime/GenericArgumentsInlines.h
r219433 r220352 262 262 263 263 if (argsLength) { 264 void* backingStore = vm. auxiliarySpace.tryAllocate(WTF::roundUpToMultipleOf<8>(argsLength));264 void* backingStore = vm.gigacageAuxiliarySpace(m_modifiedArgumentsDescriptor.kind).tryAllocate(WTF::roundUpToMultipleOf<8>(argsLength)); 265 265 RELEASE_ASSERT(backingStore); 266 266 bool* modifiedArguments = static_cast<bool*>(backingStore); … … 283 283 initModifiedArgumentsDescriptorIfNecessary(vm, length); 284 284 if (index < length) 285 m_modifiedArgumentsDescriptor .get()[index] = true;285 m_modifiedArgumentsDescriptor[index] = true; 286 286 } 287 287 … … 292 292 return false; 293 293 if (index < length) 294 return m_modifiedArgumentsDescriptor .get()[index];294 return m_modifiedArgumentsDescriptor[index]; 295 295 return false; 296 296 } -
trunk/Source/JavaScriptCore/runtime/HashMapImpl.cpp
r217192 r220352 63 63 visitor.append(thisObject->m_tail); 64 64 65 if (HashMapBufferType* buffer = thisObject->m_buffer.get ())65 if (HashMapBufferType* buffer = thisObject->m_buffer.getMayBeNull()) 66 66 visitor.markAuxiliary(buffer); 67 67 } -
trunk/Source/JavaScriptCore/runtime/HashMapImpl.h
r217808 r220352 189 189 auto scope = DECLARE_THROW_SCOPE(vm); 190 190 size_t allocationSize = HashMapBuffer::allocationSize(capacity); 191 void* data = vm. auxiliarySpace.tryAllocate(allocationSize);191 void* data = vm.jsValueGigacageAuxiliarySpace.tryAllocate(allocationSize); 192 192 if (!data) { 193 193 throwOutOfMemoryError(exec, scope); … … 303 303 ALWAYS_INLINE HashMapBucketType** buffer() const 304 304 { 305 return m_buffer .get()->buffer();305 return m_buffer->buffer(); 306 306 } 307 307 … … 592 592 RETURN_IF_EXCEPTION(scope, void()); 593 593 } else { 594 m_buffer .get()->reset(m_capacity);594 m_buffer->reset(m_capacity); 595 595 assertBufferIsEmpty(); 596 596 } … … 656 656 WriteBarrier<HashMapBucketType> m_head; 657 657 WriteBarrier<HashMapBucketType> m_tail; 658 AuxiliaryBarrier<HashMapBufferType*> m_buffer;658 CagedBarrierPtr<Gigacage::JSValue, HashMapBufferType> m_buffer; 659 659 uint32_t m_keyCount; 660 660 uint32_t m_deleteCount; -
trunk/Source/JavaScriptCore/runtime/JSArray.cpp
r220165 r220352 82 82 83 83 unsigned vectorLength = Butterfly::optimalContiguousVectorLength(structure, initialLength); 84 void* temp = vm. auxiliarySpace.tryAllocate(deferralContext, Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)));84 void* temp = vm.jsValueGigacageAuxiliarySpace.tryAllocate(deferralContext, Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue))); 85 85 if (UNLIKELY(!temp)) 86 86 return nullptr; … … 99 99 static const unsigned indexBias = 0; 100 100 unsigned vectorLength = ArrayStorage::optimalVectorLength(indexBias, structure, initialLength); 101 void* temp = vm. auxiliarySpace.tryAllocate(deferralContext, Butterfly::totalSize(indexBias, outOfLineStorage, true, ArrayStorage::sizeFor(vectorLength)));101 void* temp = vm.jsValueGigacageAuxiliarySpace.tryAllocate(deferralContext, Butterfly::totalSize(indexBias, outOfLineStorage, true, ArrayStorage::sizeFor(vectorLength))); 102 102 if (UNLIKELY(!temp)) 103 103 return nullptr; … … 361 361 } else { 362 362 size_t newSize = Butterfly::totalSize(0, propertyCapacity, true, ArrayStorage::sizeFor(desiredCapacity)); 363 newAllocBase = vm. auxiliarySpace.tryAllocate(newSize);363 newAllocBase = vm.jsValueGigacageAuxiliarySpace.tryAllocate(newSize); 364 364 if (!newAllocBase) 365 365 return false; … … 551 551 auto scope = DECLARE_THROW_SCOPE(vm); 552 552 553 Butterfly* butterfly = m_butterfly.get ().getMayBeNull();553 Butterfly* butterfly = m_butterfly.getMayBeNull(); 554 554 switch (indexingType()) { 555 555 case ArrayClass: … … 621 621 auto scope = DECLARE_THROW_SCOPE(vm); 622 622 623 Butterfly* butterfly = m_butterfly.get ().getMayBeNull();623 Butterfly* butterfly = m_butterfly.getMayBeNull(); 624 624 625 625 switch (indexingType()) { … … 723 723 auto scope = DECLARE_THROW_SCOPE(vm); 724 724 725 Butterfly* butterfly = m_butterfly.get ().getMayBeNull();725 Butterfly* butterfly = m_butterfly.getMayBeNull(); 726 726 727 727 switch (indexingType()) { … … 891 891 auto& resultButterfly = *resultArray->butterfly(); 892 892 if (arrayType == ArrayWithDouble) 893 memcpy(resultButterfly.contiguousDouble().data(), m_butterfly .get()->contiguousDouble().data() + startIndex, sizeof(JSValue) * count);893 memcpy(resultButterfly.contiguousDouble().data(), m_butterfly->contiguousDouble().data() + startIndex, sizeof(JSValue) * count); 894 894 else 895 memcpy(resultButterfly.contiguous().data(), m_butterfly .get()->contiguous().data() + startIndex, sizeof(JSValue) * count);895 memcpy(resultButterfly.contiguous().data(), m_butterfly->contiguous().data() + startIndex, sizeof(JSValue) * count); 896 896 resultButterfly.setPublicLength(count); 897 897 … … 971 971 // the start of the Butterfly, which needs to point at the first indexed property in the used 972 972 // portion of the vector. 973 Butterfly* butterfly = m_butterfly .get()->shift(structure(), count);973 Butterfly* butterfly = m_butterfly->shift(structure(), count); 974 974 setButterfly(vm, butterfly); 975 975 storage = butterfly->arrayStorage(); … … 1016 1016 RELEASE_ASSERT(count > 0); 1017 1017 1018 Butterfly* butterfly = m_butterfly.get ().getMayBeNull();1018 Butterfly* butterfly = m_butterfly.getMayBeNull(); 1019 1019 1020 1020 switch (indexingType()) { … … 1172 1172 auto scope = DECLARE_THROW_SCOPE(vm); 1173 1173 1174 Butterfly* butterfly = m_butterfly.get ().getMayBeNull();1174 Butterfly* butterfly = m_butterfly.getMayBeNull(); 1175 1175 1176 1176 switch (indexingType()) { … … 1195 1195 return false; 1196 1196 } 1197 butterfly = m_butterfly.get ().getMayBeNull();1197 butterfly = m_butterfly.getMayBeNull(); 1198 1198 1199 1199 // We have to check for holes before we start moving things around so that we don't get halfway … … 1239 1239 return false; 1240 1240 } 1241 butterfly = m_butterfly.get ().getMayBeNull();1241 butterfly = m_butterfly.getMayBeNull(); 1242 1242 1243 1243 // We have to check for holes before we start moving things around so that we don't get halfway … … 1282 1282 WriteBarrier<Unknown>* vector; 1283 1283 1284 Butterfly* butterfly = m_butterfly.get ().getMayBeNull();1284 Butterfly* butterfly = m_butterfly.getMayBeNull(); 1285 1285 1286 1286 switch (indexingType()) { … … 1355 1355 ASSERT(length == this->length()); 1356 1356 1357 Butterfly* butterfly = m_butterfly.get ().getMayBeNull();1357 Butterfly* butterfly = m_butterfly.getMayBeNull(); 1358 1358 switch (indexingType()) { 1359 1359 case ArrayClass: -
trunk/Source/JavaScriptCore/runtime/JSArray.h
r219636 r220352 232 232 233 233 unsigned vectorLength = Butterfly::optimalContiguousVectorLength(structure, initialLength); 234 void* temp = vm. auxiliarySpace.tryAllocate(nullptr, Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)));234 void* temp = vm.jsValueGigacageAuxiliarySpace.tryAllocate(nullptr, Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue))); 235 235 if (!temp) 236 236 return nullptr; -
trunk/Source/JavaScriptCore/runtime/JSArrayBufferView.cpp
r220118 r220352 67 67 size_t size = sizeOf(length, elementSize); 68 68 if (size) { 69 temp = vm. auxiliarySpace.tryAllocate(nullptr, size);69 temp = vm.primitiveGigacageAuxiliarySpace.tryAllocate(nullptr, size); 70 70 if (!temp) 71 71 return; … … 91 91 92 92 size_t size = static_cast<size_t>(length) * static_cast<size_t>(elementSize); 93 m_vector = Gigacage::tryMalloc( size);93 m_vector = Gigacage::tryMalloc(Gigacage::Primitive, size); 94 94 if (!m_vector) 95 95 return; … … 193 193 ASSERT(thisObject->m_mode == OversizeTypedArray || thisObject->m_mode == WastefulTypedArray); 194 194 if (thisObject->m_mode == OversizeTypedArray) 195 Gigacage::free( thisObject->m_vector.get());195 Gigacage::free(Gigacage::Primitive, thisObject->m_vector.get()); 196 196 } 197 197 -
trunk/Source/JavaScriptCore/runtime/JSLock.cpp
r220186 r220352 158 158 m_vm->traps().notifyGrabAllLocks(); 159 159 160 m_vm->fire GigacageEnabledIfNecessary();160 m_vm->firePrimitiveGigacageEnabledIfNecessary(); 161 161 162 162 #if ENABLE(SAMPLING_PROFILER) -
trunk/Source/JavaScriptCore/runtime/JSObject.cpp
r220165 r220352 464 464 } 465 465 466 Butterfly* butterfly = thisObject->m_butterfly.get ().getMayBeNull();466 Butterfly* butterfly = thisObject->m_butterfly.getMayBeNull(); 467 467 if (butterfly) { 468 468 WriteBarrier<Unknown>* data = nullptr; … … 614 614 615 615 case ALL_ARRAY_STORAGE_INDEXING_TYPES: { 616 ArrayStorage* storage = thisObject->m_butterfly .get()->arrayStorage();616 ArrayStorage* storage = thisObject->m_butterfly->arrayStorage(); 617 617 if (i >= storage->length()) 618 618 return false; … … 876 876 case NonArrayWithArrayStorage: 877 877 case ArrayWithArrayStorage: { 878 ArrayStorage* storage = thisObject->m_butterfly .get()->arrayStorage();878 ArrayStorage* storage = thisObject->m_butterfly->arrayStorage(); 879 879 880 880 if (propertyName >= storage->vectorLength()) … … 898 898 case NonArrayWithSlowPutArrayStorage: 899 899 case ArrayWithSlowPutArrayStorage: { 900 ArrayStorage* storage = thisObject->m_butterfly .get()->arrayStorage();900 ArrayStorage* storage = thisObject->m_butterfly->arrayStorage(); 901 901 902 902 if (propertyName >= storage->vectorLength()) … … 980 980 break; 981 981 case ALL_ARRAY_STORAGE_INDEXING_TYPES: 982 enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, m_butterfly .get()->arrayStorage());982 enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, m_butterfly->arrayStorage()); 983 983 break; 984 984 … … 1012 1012 unsigned vectorLength = Butterfly::optimalContiguousVectorLength(propertyCapacity, length); 1013 1013 Butterfly* newButterfly = Butterfly::createOrGrowArrayRight( 1014 m_butterfly.get ().getMayBeNull(), vm, this, structure, propertyCapacity, false, 0,1014 m_butterfly.getMayBeNull(), vm, this, structure, propertyCapacity, false, 0, 1015 1015 sizeof(EncodedJSValue) * vectorLength); 1016 1016 newButterfly->setPublicLength(length); … … 1100 1100 ASSERT_UNUSED(oldType, !hasIndexedProperties(oldType)); 1101 1101 1102 Butterfly* newButterfly = createArrayStorageButterfly(vm, this, oldStructure, length, vectorLength, m_butterfly.get ().getMayBeNull());1102 Butterfly* newButterfly = createArrayStorageButterfly(vm, this, oldStructure, length, vectorLength, m_butterfly.getMayBeNull()); 1103 1103 ArrayStorage* result = newButterfly->arrayStorage(); 1104 1104 Structure* newStructure = Structure::nonPropertyTransition(vm, oldStructure, oldStructure->suggestedArrayStorageTransition()); … … 1118 1118 ASSERT(hasUndecided(indexingType())); 1119 1119 1120 Butterfly* butterfly = m_butterfly.get ().getMayBeNull();1120 Butterfly* butterfly = m_butterfly.getMayBeNull(); 1121 1121 for (unsigned i = butterfly->vectorLength(); i--;) 1122 1122 butterfly->contiguousInt32()[i].setWithoutWriteBarrier(JSValue()); 1123 1123 1124 1124 setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateInt32)); 1125 return m_butterfly .get()->contiguousInt32();1125 return m_butterfly->contiguousInt32(); 1126 1126 } 1127 1127 … … 1130 1130 ASSERT(hasUndecided(indexingType())); 1131 1131 1132 Butterfly* butterfly = m_butterfly.get() .get();1132 Butterfly* butterfly = m_butterfly.get(); 1133 1133 for (unsigned i = butterfly->vectorLength(); i--;) 1134 1134 butterfly->contiguousDouble()[i] = PNaN; 1135 1135 1136 1136 setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateDouble)); 1137 return m_butterfly .get()->contiguousDouble();1137 return m_butterfly->contiguousDouble(); 1138 1138 } 1139 1139 … … 1142 1142 ASSERT(hasUndecided(indexingType())); 1143 1143 1144 Butterfly* butterfly = m_butterfly.get() .get();1144 Butterfly* butterfly = m_butterfly.get(); 1145 1145 for (unsigned i = butterfly->vectorLength(); i--;) 1146 1146 butterfly->contiguous()[i].setWithoutWriteBarrier(JSValue()); … … 1148 1148 WTF::storeStoreFence(); 1149 1149 setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateContiguous)); 1150 return m_butterfly .get()->contiguous();1150 return m_butterfly->contiguous(); 1151 1151 } 1152 1152 … … 1154 1154 { 1155 1155 Structure* structure = this->structure(vm); 1156 unsigned publicLength = m_butterfly .get()->publicLength();1156 unsigned publicLength = m_butterfly->publicLength(); 1157 1157 unsigned propertyCapacity = structure->outOfLineCapacity(); 1158 1158 unsigned propertySize = structure->outOfLineSize(); … … 1163 1163 memcpy( 1164 1164 newButterfly->propertyStorage() - propertySize, 1165 m_butterfly .get()->propertyStorage() - propertySize,1165 m_butterfly->propertyStorage() - propertySize, 1166 1166 propertySize * sizeof(EncodedJSValue)); 1167 1167 … … 1181 1181 ASSERT(hasUndecided(indexingType())); 1182 1182 1183 unsigned vectorLength = m_butterfly .get()->vectorLength();1183 unsigned vectorLength = m_butterfly->vectorLength(); 1184 1184 ArrayStorage* storage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength); 1185 1185 … … 1204 1204 ASSERT(hasInt32(indexingType())); 1205 1205 1206 Butterfly* butterfly = m_butterfly.get() .get();1206 Butterfly* butterfly = m_butterfly.get(); 1207 1207 for (unsigned i = butterfly->vectorLength(); i--;) { 1208 1208 WriteBarrier<Unknown>* current = &butterfly->contiguousInt32()[i]; … … 1219 1219 1220 1220 setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateDouble)); 1221 return m_butterfly .get()->contiguousDouble();1221 return m_butterfly->contiguousDouble(); 1222 1222 } 1223 1223 … … 1227 1227 1228 1228 setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateContiguous)); 1229 return m_butterfly .get()->contiguous();1229 return m_butterfly->contiguous(); 1230 1230 } 1231 1231 … … 1235 1235 ASSERT(hasInt32(indexingType())); 1236 1236 1237 unsigned vectorLength = m_butterfly .get()->vectorLength();1237 unsigned vectorLength = m_butterfly->vectorLength(); 1238 1238 ArrayStorage* newStorage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength); 1239 Butterfly* butterfly = m_butterfly.get() .get();1239 Butterfly* butterfly = m_butterfly.get(); 1240 1240 for (unsigned i = 0; i < vectorLength; i++) { 1241 1241 JSValue v = butterfly->contiguous()[i].get(); … … 1262 1262 ASSERT(hasDouble(indexingType())); 1263 1263 1264 Butterfly* butterfly = m_butterfly.get() .get();1264 Butterfly* butterfly = m_butterfly.get(); 1265 1265 for (unsigned i = butterfly->vectorLength(); i--;) { 1266 1266 double* current = &butterfly->contiguousDouble()[i]; … … 1277 1277 WTF::storeStoreFence(); 1278 1278 setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateContiguous)); 1279 return m_butterfly .get()->contiguous();1279 return m_butterfly->contiguous(); 1280 1280 } 1281 1281 … … 1285 1285 ASSERT(hasDouble(indexingType())); 1286 1286 1287 unsigned vectorLength = m_butterfly .get()->vectorLength();1287 unsigned vectorLength = m_butterfly->vectorLength(); 1288 1288 ArrayStorage* newStorage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength); 1289 Butterfly* butterfly = m_butterfly.get() .get();1289 Butterfly* butterfly = m_butterfly.get(); 1290 1290 for (unsigned i = 0; i < vectorLength; i++) { 1291 1291 double value = butterfly->contiguousDouble()[i]; … … 1316 1316 ASSERT(hasContiguous(indexingType())); 1317 1317 1318 unsigned vectorLength = m_butterfly .get()->vectorLength();1318 unsigned vectorLength = m_butterfly->vectorLength(); 1319 1319 ArrayStorage* newStorage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength); 1320 Butterfly* butterfly = m_butterfly.get() .get();1320 Butterfly* butterfly = m_butterfly.get(); 1321 1321 for (unsigned i = 0; i < vectorLength; i++) { 1322 1322 JSValue v = butterfly->contiguous()[i].get(); … … 1409 1409 void JSObject::setIndexQuicklyToUndecided(VM& vm, unsigned index, JSValue value) 1410 1410 { 1411 ASSERT(index < m_butterfly .get()->publicLength());1412 ASSERT(index < m_butterfly .get()->vectorLength());1411 ASSERT(index < m_butterfly->publicLength()); 1412 ASSERT(index < m_butterfly->vectorLength()); 1413 1413 convertUndecidedForValue(vm, value); 1414 1414 setIndexQuickly(vm, index, value); … … 1578 1578 1579 1579 case ALL_ARRAY_STORAGE_INDEXING_TYPES: 1580 return enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, m_butterfly .get()->arrayStorage());1580 return enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, m_butterfly->arrayStorage()); 1581 1581 1582 1582 default: … … 1863 1863 1864 1864 case ALL_ARRAY_STORAGE_INDEXING_TYPES: { 1865 ArrayStorage* storage = thisObject->m_butterfly .get()->arrayStorage();1865 ArrayStorage* storage = thisObject->m_butterfly->arrayStorage(); 1866 1866 1867 1867 if (i < storage->vectorLength()) { … … 2169 2169 2170 2170 case ALL_ARRAY_STORAGE_INDEXING_TYPES: { 2171 ArrayStorage* storage = object->m_butterfly .get()->arrayStorage();2171 ArrayStorage* storage = object->m_butterfly->arrayStorage(); 2172 2172 2173 2173 unsigned usedVectorLength = std::min(storage->length(), storage->vectorLength()); … … 2326 2326 { 2327 2327 VM& vm = exec->vm(); 2328 auto map = m_butterfly .get()->arrayStorage()->m_sparseMap.get();2328 auto map = m_butterfly->arrayStorage()->m_sparseMap.get(); 2329 2329 2330 2330 if (descriptor.isDataDescriptor()) { … … 2396 2396 notifyPresenceOfIndexedAccessors(vm); 2397 2397 2398 SparseArrayValueMap* map = m_butterfly .get()->arrayStorage()->m_sparseMap.get();2398 SparseArrayValueMap* map = m_butterfly->arrayStorage()->m_sparseMap.get(); 2399 2399 RELEASE_ASSERT(map); 2400 2400 … … 2428 2428 2429 2429 putIndexedDescriptor(exec, entryInMap, descriptor, defaults); 2430 Butterfly* butterfly = m_butterfly.get() .get();2430 Butterfly* butterfly = m_butterfly.get(); 2431 2431 if (index >= butterfly->arrayStorage()->length()) 2432 2432 butterfly->arrayStorage()->setLength(index + 1); … … 2560 2560 ASSERT(!indexingShouldBeSparse()); 2561 2561 2562 Butterfly* butterfly = m_butterfly.get() .get();2562 Butterfly* butterfly = m_butterfly.get(); 2563 2563 2564 2564 // For us to get here, the index is either greater than the public length, or greater than … … 2582 2582 return false; 2583 2583 } 2584 butterfly = m_butterfly.get() .get();2584 butterfly = m_butterfly.get(); 2585 2585 2586 2586 RELEASE_ASSERT(i < butterfly->vectorLength()); … … 2870 2870 case ALL_INT32_INDEXING_TYPES: { 2871 2871 if (attributes) { 2872 if (i < m_butterfly .get()->vectorLength())2872 if (i < m_butterfly->vectorLength()) 2873 2873 return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, ensureArrayStorageExistsAndEnterDictionaryIndexingMode(vm)); 2874 2874 return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, convertInt32ToArrayStorage(vm)); … … 2884 2884 case ALL_DOUBLE_INDEXING_TYPES: { 2885 2885 if (attributes) { 2886 if (i < m_butterfly .get()->vectorLength())2886 if (i < m_butterfly->vectorLength()) 2887 2887 return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, ensureArrayStorageExistsAndEnterDictionaryIndexingMode(vm)); 2888 2888 return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, convertDoubleToArrayStorage(vm)); … … 2903 2903 case ALL_CONTIGUOUS_INDEXING_TYPES: { 2904 2904 if (attributes) { 2905 if (i < m_butterfly .get()->vectorLength())2905 if (i < m_butterfly->vectorLength()) 2906 2906 return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, ensureArrayStorageExistsAndEnterDictionaryIndexingMode(vm)); 2907 2907 return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, convertContiguousToArrayStorage(vm)); … … 2913 2913 case ALL_ARRAY_STORAGE_INDEXING_TYPES: 2914 2914 if (attributes) { 2915 if (i < m_butterfly .get()->vectorLength())2915 if (i < m_butterfly->vectorLength()) 2916 2916 return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, ensureArrayStorageExistsAndEnterDictionaryIndexingMode(vm)); 2917 2917 } … … 3016 3016 if (ArrayStorage* storage = arrayStorageOrNull()) 3017 3017 indexBias = storage->m_indexBias; 3018 vectorLength = m_butterfly .get()->vectorLength();3019 length = m_butterfly .get()->publicLength();3018 vectorLength = m_butterfly->vectorLength(); 3019 length = m_butterfly->publicLength(); 3020 3020 } 3021 3021 … … 3133 3133 bool JSObject::ensureLengthSlow(VM& vm, unsigned length) 3134 3134 { 3135 Butterfly* butterfly = m_butterfly.get() .get();3135 Butterfly* butterfly = m_butterfly.get(); 3136 3136 3137 3137 ASSERT(length <= MAX_STORAGE_VECTOR_LENGTH); … … 3189 3189 ASSERT(length <= MAX_STORAGE_VECTOR_LENGTH); 3190 3190 ASSERT(hasContiguous(indexingType()) || hasInt32(indexingType()) || hasDouble(indexingType()) || hasUndecided(indexingType())); 3191 ASSERT(m_butterfly .get()->vectorLength() > length);3192 ASSERT(!m_butterfly .get()->indexingHeader()->preCapacity(structure()));3191 ASSERT(m_butterfly->vectorLength() > length); 3192 ASSERT(!m_butterfly->indexingHeader()->preCapacity(structure())); 3193 3193 3194 3194 DeferGC deferGC(vm.heap); 3195 Butterfly* newButterfly = m_butterfly .get()->resizeArray(vm, this, structure(), 0, ArrayStorage::sizeFor(length));3195 Butterfly* newButterfly = m_butterfly->resizeArray(vm, this, structure(), 0, ArrayStorage::sizeFor(length)); 3196 3196 newButterfly->setVectorLength(length); 3197 3197 newButterfly->setPublicLength(length); … … 3207 3207 // capacity, since we might have already mutated the structure in-place. 3208 3208 3209 return Butterfly::createOrGrowPropertyStorage(m_butterfly.get ().getMayBeNull(), vm, this, structure(vm), oldSize, newSize);3209 return Butterfly::createOrGrowPropertyStorage(m_butterfly.getMayBeNull(), vm, this, structure(vm), oldSize, newSize); 3210 3210 } 3211 3211 … … 3553 3553 3554 3554 case ALL_ARRAY_STORAGE_INDEXING_TYPES: { 3555 ArrayStorage* storage = object->m_butterfly .get()->arrayStorage();3555 ArrayStorage* storage = object->m_butterfly->arrayStorage(); 3556 3556 if (storage->m_sparseMap.get()) 3557 3557 return 0; -
trunk/Source/JavaScriptCore/runtime/JSObject.h
r220165 r220352 25 25 #include "ArrayConventions.h" 26 26 #include "ArrayStorage.h" 27 #include "AuxiliaryBarrier.h"28 27 #include "Butterfly.h" 29 28 #include "CPU.h" 29 #include "CagedBarrierPtr.h" 30 30 #include "CallFrame.h" 31 31 #include "ClassInfo.h" … … 44 44 #include "JSString.h" 45 45 #include "SparseArrayValueMap.h" 46 #include <wtf/CagedPtr.h>47 46 #include <wtf/StdLibExtras.h> 48 47 … … 179 178 if (!hasIndexedProperties(indexingType())) 180 179 return 0; 181 return m_butterfly .get()->publicLength();180 return m_butterfly->publicLength(); 182 181 } 183 182 … … 186 185 if (!hasIndexedProperties(indexingType())) 187 186 return 0; 188 return m_butterfly .get()->vectorLength();187 return m_butterfly->vectorLength(); 189 188 } 190 189 … … 222 221 case ALL_CONTIGUOUS_INDEXING_TYPES: 223 222 case ALL_ARRAY_STORAGE_INDEXING_TYPES: 224 return propertyName < m_butterfly .get()->vectorLength();223 return propertyName < m_butterfly->vectorLength(); 225 224 default: 226 225 RELEASE_ASSERT_NOT_REACHED(); … … 256 255 bool canGetIndexQuickly(unsigned i) 257 256 { 258 Butterfly* butterfly = m_butterfly.get ().getMayBeNull();257 Butterfly* butterfly = m_butterfly.getMayBeNull(); 259 258 switch (indexingType()) { 260 259 case ALL_BLANK_INDEXING_TYPES: … … 282 281 JSValue getIndexQuickly(unsigned i) 283 282 { 284 Butterfly* butterfly = m_butterfly.get() .get();283 Butterfly* butterfly = m_butterfly.get(); 285 284 switch (indexingType()) { 286 285 case ALL_INT32_INDEXING_TYPES: … … 300 299 JSValue tryGetIndexQuickly(unsigned i) const 301 300 { 302 Butterfly* butterfly = m_butterfly.get ().getMayBeNull();301 Butterfly* butterfly = m_butterfly.getMayBeNull(); 303 302 switch (indexingType()) { 304 303 case ALL_BLANK_INDEXING_TYPES: … … 354 353 bool canSetIndexQuickly(unsigned i) 355 354 { 356 Butterfly* butterfly = m_butterfly.get ().getMayBeNull();355 Butterfly* butterfly = m_butterfly.getMayBeNull(); 357 356 switch (indexingType()) { 358 357 case ALL_BLANK_INDEXING_TYPES: … … 377 376 void setIndexQuickly(VM& vm, unsigned i, JSValue v) 378 377 { 379 Butterfly* butterfly = m_butterfly.get() .get();378 Butterfly* butterfly = m_butterfly.get(); 380 379 switch (indexingType()) { 381 380 case ALL_INT32_INDEXING_TYPES: { … … 437 436 { 438 437 VM& vm = scope.vm(); 439 Butterfly* butterfly = m_butterfly.get() .get();438 Butterfly* butterfly = m_butterfly.get(); 440 439 switch (indexingType) { 441 440 case ALL_UNDECIDED_INDEXING_TYPES: { … … 494 493 ALWAYS_INLINE void initializeIndexWithoutBarrier(ObjectInitializationScope&, unsigned i, JSValue v, IndexingType indexingType) 495 494 { 496 Butterfly* butterfly = m_butterfly.get() .get();495 Butterfly* butterfly = m_butterfly.get(); 497 496 switch (indexingType) { 498 497 case ALL_UNDECIDED_INDEXING_TYPES: { … … 543 542 return false; 544 543 case ALL_ARRAY_STORAGE_INDEXING_TYPES: 545 return !!m_butterfly .get()->arrayStorage()->m_sparseMap;544 return !!m_butterfly->arrayStorage()->m_sparseMap; 546 545 default: 547 546 RELEASE_ASSERT_NOT_REACHED(); … … 560 559 return false; 561 560 case ALL_ARRAY_STORAGE_INDEXING_TYPES: 562 return m_butterfly .get()->arrayStorage()->inSparseMode();561 return m_butterfly->arrayStorage()->inSparseMode(); 563 562 default: 564 563 RELEASE_ASSERT_NOT_REACHED(); … … 670 669 } 671 670 672 const Butterfly* butterfly() const { return m_butterfly.get ().getMayBeNull(); }673 Butterfly* butterfly() { return m_butterfly.get ().getMayBeNull(); }671 const Butterfly* butterfly() const { return m_butterfly.getMayBeNull(); } 672 Butterfly* butterfly() { return m_butterfly.getMayBeNull(); } 674 673 675 ConstPropertyStorage outOfLineStorage() const { return m_butterfly .get()->propertyStorage(); }676 PropertyStorage outOfLineStorage() { return m_butterfly .get()->propertyStorage(); }674 ConstPropertyStorage outOfLineStorage() const { return m_butterfly->propertyStorage(); } 675 PropertyStorage outOfLineStorage() { return m_butterfly->propertyStorage(); } 677 676 678 677 const WriteBarrierBase<Unknown>* locationForOffset(PropertyOffset offset) const … … 808 807 { 809 808 if (LIKELY(hasInt32(indexingType()))) 810 return m_butterfly .get()->contiguousInt32();809 return m_butterfly->contiguousInt32(); 811 810 812 811 return ensureInt32Slow(vm); … … 820 819 { 821 820 if (LIKELY(hasDouble(indexingType()))) 822 return m_butterfly .get()->contiguousDouble();821 return m_butterfly->contiguousDouble(); 823 822 824 823 return ensureDoubleSlow(vm); … … 830 829 { 831 830 if (LIKELY(hasContiguous(indexingType()))) 832 return m_butterfly .get()->contiguous();831 return m_butterfly->contiguous(); 833 832 834 833 return ensureContiguousSlow(vm); … … 842 841 { 843 842 if (LIKELY(hasAnyArrayStorage(indexingType()))) 844 return m_butterfly .get()->arrayStorage();843 return m_butterfly->arrayStorage(); 845 844 846 845 return ensureArrayStorageSlow(vm); … … 894 893 { 895 894 ASSERT(hasAnyArrayStorage(indexingType())); 896 return m_butterfly .get()->arrayStorage();895 return m_butterfly->arrayStorage(); 897 896 } 898 897 … … 903 902 switch (indexingType()) { 904 903 case ALL_ARRAY_STORAGE_INDEXING_TYPES: 905 return m_butterfly .get()->arrayStorage();904 return m_butterfly->arrayStorage(); 906 905 907 906 default: … … 969 968 ASSERT(hasContiguous(indexingType()) || hasInt32(indexingType()) || hasDouble(indexingType()) || hasUndecided(indexingType())); 970 969 971 if (m_butterfly .get()->vectorLength() < length) {970 if (m_butterfly->vectorLength() < length) { 972 971 if (!ensureLengthSlow(vm, length)) 973 972 return false; 974 973 } 975 974 976 if (m_butterfly .get()->publicLength() < length)977 m_butterfly .get()->setPublicLength(length);975 if (m_butterfly->publicLength() < length) 976 m_butterfly->setPublicLength(length); 978 977 return true; 979 978 } … … 1047 1046 1048 1047 protected: 1049 AuxiliaryBarrier<CagedPtr<Butterfly>> m_butterfly;1048 CagedBarrierPtr<Gigacage::JSValue, Butterfly> m_butterfly; 1050 1049 #if USE(JSVALUE32_64) 1051 1050 private: -
trunk/Source/JavaScriptCore/runtime/RegExpMatchesArray.h
r215885 r220352 42 42 JSGlobalObject* globalObject = structure->globalObject(); 43 43 bool createUninitialized = globalObject->isOriginalArrayStructure(structure); 44 void* temp = vm. auxiliarySpace.tryAllocate(deferralContext, Butterfly::totalSize(0, structure->outOfLineCapacity(), true, vectorLength * sizeof(EncodedJSValue)));44 void* temp = vm.jsValueGigacageAuxiliarySpace.tryAllocate(deferralContext, Butterfly::totalSize(0, structure->outOfLineCapacity(), true, vectorLength * sizeof(EncodedJSValue))); 45 45 if (UNLIKELY(!temp)) 46 46 return nullptr; -
trunk/Source/JavaScriptCore/runtime/VM.cpp
r220322 r220352 166 166 #endif // USE(CF) 167 167 , heap(this, heapType) 168 , auxiliarySpace("Auxiliary", heap, AllocatorAttributes(DoesNotNeedDestruction, HeapCell::Auxiliary), &GigacageAlignedMemoryAllocator::instance()) 169 , cellSpace("JSCell", heap, AllocatorAttributes(DoesNotNeedDestruction, HeapCell::JSCell), &FastMallocAlignedMemoryAllocator::instance()) 170 , destructibleCellSpace("Destructible JSCell", heap, AllocatorAttributes(NeedsDestruction, HeapCell::JSCell), &FastMallocAlignedMemoryAllocator::instance()) 171 , stringSpace("JSString", heap, &FastMallocAlignedMemoryAllocator::instance()) 172 , destructibleObjectSpace("JSDestructibleObject", heap, &FastMallocAlignedMemoryAllocator::instance()) 173 , eagerlySweptDestructibleObjectSpace("Eagerly Swept JSDestructibleObject", heap, &FastMallocAlignedMemoryAllocator::instance()) 174 , segmentedVariableObjectSpace("JSSegmentedVariableObjectSpace", heap, &FastMallocAlignedMemoryAllocator::instance()) 168 , fastMallocAllocator(std::make_unique<FastMallocAlignedMemoryAllocator>()) 169 , primitiveGigacageAllocator(std::make_unique<GigacageAlignedMemoryAllocator>(Gigacage::Primitive)) 170 , jsValueGigacageAllocator(std::make_unique<GigacageAlignedMemoryAllocator>(Gigacage::JSValue)) 171 , primitiveGigacageAuxiliarySpace("Primitive Gigacage Auxiliary", heap, AllocatorAttributes(DoesNotNeedDestruction, HeapCell::Auxiliary), primitiveGigacageAllocator.get()) 172 , jsValueGigacageAuxiliarySpace("JSValue Gigacage Auxiliary", heap, AllocatorAttributes(DoesNotNeedDestruction, HeapCell::Auxiliary), jsValueGigacageAllocator.get()) 173 , cellSpace("JSCell", heap, AllocatorAttributes(DoesNotNeedDestruction, HeapCell::JSCell), fastMallocAllocator.get()) 174 , destructibleCellSpace("Destructible JSCell", heap, AllocatorAttributes(NeedsDestruction, HeapCell::JSCell), fastMallocAllocator.get()) 175 , stringSpace("JSString", heap, fastMallocAllocator.get()) 176 , destructibleObjectSpace("JSDestructibleObject", heap, fastMallocAllocator.get()) 177 , eagerlySweptDestructibleObjectSpace("Eagerly Swept JSDestructibleObject", heap, fastMallocAllocator.get()) 178 , segmentedVariableObjectSpace("JSSegmentedVariableObjectSpace", heap, fastMallocAllocator.get()) 175 179 #if ENABLE(WEBASSEMBLY) 176 , webAssemblyCodeBlockSpace("JSWebAssemblyCodeBlockSpace", heap, &FastMallocAlignedMemoryAllocator::instance())180 , webAssemblyCodeBlockSpace("JSWebAssemblyCodeBlockSpace", heap, fastMallocAllocator.get()) 177 181 #endif 178 182 , vmType(vmType) … … 212 216 , m_builtinExecutables(std::make_unique<BuiltinExecutables>(*this)) 213 217 , m_typeProfilerEnabledCount(0) 214 , m_ gigacageEnabled(IsWatched)218 , m_primitiveGigacageEnabled(IsWatched) 215 219 , m_controlFlowProfilerEnabledCount(0) 216 220 , m_shadowChicken(std::make_unique<ShadowChicken>()) … … 291 295 #endif 292 296 293 Gigacage::add DisableCallback(gigacageDisabledCallback, this);297 Gigacage::addPrimitiveDisableCallback(primitiveGigacageDisabledCallback, this); 294 298 295 299 heap.notifyIsSafeToCollect(); … … 355 359 auto destructionLocker = holdLock(s_destructionLock.read()); 356 360 357 Gigacage::remove DisableCallback(gigacageDisabledCallback, this);361 Gigacage::removePrimitiveDisableCallback(primitiveGigacageDisabledCallback, this); 358 362 promiseDeferredTimer->stopRunningTasks(); 359 363 #if ENABLE(WEBASSEMBLY) … … 424 428 } 425 429 426 void VM:: gigacageDisabledCallback(void* argument)427 { 428 static_cast<VM*>(argument)-> gigacageDisabled();429 } 430 431 void VM:: gigacageDisabled()430 void VM::primitiveGigacageDisabledCallback(void* argument) 431 { 432 static_cast<VM*>(argument)->primitiveGigacageDisabled(); 433 } 434 435 void VM::primitiveGigacageDisabled() 432 436 { 433 437 if (m_apiLock->currentThreadIsHoldingLock()) { 434 m_ gigacageEnabled.fireAll(*this, "Gigacage disabled");438 m_primitiveGigacageEnabled.fireAll(*this, "Primitive gigacage disabled"); 435 439 return; 436 440 } … … 438 442 // This is totally racy, and that's OK. The point is, it's up to the user to ensure that they pass the 439 443 // uncaged buffer in a nicely synchronized manner. 440 m_needToFire GigacageEnabled = true;444 m_needToFirePrimitiveGigacageEnabled = true; 441 445 } 442 446 -
trunk/Source/JavaScriptCore/runtime/VM.h
r220291 r220352 63 63 #include <wtf/DoublyLinkedList.h> 64 64 #include <wtf/Forward.h> 65 #include <wtf/Gigacage.h> 65 66 #include <wtf/HashMap.h> 66 67 #include <wtf/HashSet.h> … … 96 97 class Exception; 97 98 class ExceptionScope; 99 class FastMallocAlignedMemoryAllocator; 100 class GigacageAlignedMemoryAllocator; 98 101 class HandleStack; 99 102 class TypeProfiler; … … 286 289 Heap heap; 287 290 288 Subspace auxiliarySpace; 291 std::unique_ptr<FastMallocAlignedMemoryAllocator> fastMallocAllocator; 292 std::unique_ptr<GigacageAlignedMemoryAllocator> primitiveGigacageAllocator; 293 std::unique_ptr<GigacageAlignedMemoryAllocator> jsValueGigacageAllocator; 294 295 Subspace primitiveGigacageAuxiliarySpace; // Typed arrays, strings, bitvectors, etc go here. 296 Subspace jsValueGigacageAuxiliarySpace; // Butterflies, arrays of JSValues, etc go here. 297 298 // We make cross-cutting assumptions about typed arrays being in the primitive Gigacage and butterflies 299 // being in the JSValue gigacage. For some types, it's super obvious where they should go, and so we 300 // can hardcode that fact. But sometimes it's not clear, so we abstract it by having a Gigacage::Kind 301 // constant somewhere. 302 // FIXME: Maybe it would be better if everyone abstracted this? 303 // https://bugs.webkit.org/show_bug.cgi?id=175248 304 ALWAYS_INLINE Subspace& gigacageAuxiliarySpace(Gigacage::Kind kind) 305 { 306 switch (kind) { 307 case Gigacage::Primitive: 308 return primitiveGigacageAuxiliarySpace; 309 case Gigacage::JSValue: 310 return jsValueGigacageAuxiliarySpace; 311 } 312 RELEASE_ASSERT_NOT_REACHED(); 313 return primitiveGigacageAuxiliarySpace; 314 } 289 315 290 316 // Whenever possible, use subspaceFor<CellType>(vm) to get one of these subspaces. … … 525 551 void setLastStackTop(void*); 526 552 527 void fire GigacageEnabledIfNecessary()528 { 529 if (m_needToFire GigacageEnabled) {530 m_needToFire GigacageEnabled = false;531 m_ gigacageEnabled.fireAll(*this, "Gigacage disabled asynchronously");553 void firePrimitiveGigacageEnabledIfNecessary() 554 { 555 if (m_needToFirePrimitiveGigacageEnabled) { 556 m_needToFirePrimitiveGigacageEnabled = false; 557 m_primitiveGigacageEnabled.fireAll(*this, "Primitive gigacage disabled asynchronously"); 532 558 } 533 559 } … … 634 660 JS_EXPORT_PRIVATE void addImpureProperty(const String&); 635 661 636 InlineWatchpointSet& gigacageEnabled() { return m_gigacageEnabled; }662 InlineWatchpointSet& primitiveGigacageEnabled() { return m_primitiveGigacageEnabled; } 637 663 638 664 BuiltinExecutables* builtinExecutables() { return m_builtinExecutables.get(); } … … 742 768 #endif 743 769 744 static void gigacageDisabledCallback(void*);745 void gigacageDisabled();770 static void primitiveGigacageDisabledCallback(void*); 771 void primitiveGigacageDisabled(); 746 772 747 773 #if ENABLE(ASSEMBLER) … … 788 814 std::unique_ptr<TypeProfilerLog> m_typeProfilerLog; 789 815 unsigned m_typeProfilerEnabledCount; 790 bool m_needToFire GigacageEnabled { false };791 InlineWatchpointSet m_ gigacageEnabled;816 bool m_needToFirePrimitiveGigacageEnabled { false }; 817 InlineWatchpointSet m_primitiveGigacageEnabled; 792 818 FunctionHasExecutedCache m_functionHasExecutedCache; 793 819 std::unique_ptr<ControlFlowProfiler> m_controlFlowProfiler; -
trunk/Source/JavaScriptCore/wasm/WasmMemory.cpp
r220118 r220352 101 101 return MemoryResult(nullptr, MemoryResult::SyncGCAndRetry); 102 102 103 void* result = Gigacage::tryAllocateVirtualPages( Memory::fastMappedBytes());103 void* result = Gigacage::tryAllocateVirtualPages(Gigacage::Primitive, Memory::fastMappedBytes()); 104 104 if (!result) 105 105 return MemoryResult(nullptr, MemoryResult::SyncGCAndRetry); … … 122 122 { 123 123 auto holder = holdLock(m_lock); 124 Gigacage::freeVirtualPages( basePtr, Memory::fastMappedBytes());124 Gigacage::freeVirtualPages(Gigacage::Primitive, basePtr, Memory::fastMappedBytes()); 125 125 m_memories.removeFirst(basePtr); 126 126 } … … 313 313 return adoptRef(new Memory(initial, maximum)); 314 314 315 void* slowMemory = Gigacage::tryAlignedMalloc( WTF::pageSize(), initialBytes);315 void* slowMemory = Gigacage::tryAlignedMalloc(Gigacage::Primitive, WTF::pageSize(), initialBytes); 316 316 if (!slowMemory) { 317 317 memoryManager().freePhysicalBytes(initialBytes); … … 332 332 break; 333 333 case MemoryMode::BoundsChecking: 334 Gigacage::alignedFree( m_memory);334 Gigacage::alignedFree(Gigacage::Primitive, m_memory); 335 335 break; 336 336 } … … 379 379 RELEASE_ASSERT(maximum().bytes() != 0); 380 380 381 void* newMemory = Gigacage::tryAlignedMalloc( WTF::pageSize(), desiredSize);381 void* newMemory = Gigacage::tryAlignedMalloc(Gigacage::Primitive, WTF::pageSize(), desiredSize); 382 382 if (!newMemory) 383 383 return false; … … 385 385 memset(static_cast<char*>(newMemory) + m_size, 0, desiredSize - m_size); 386 386 if (m_memory) 387 Gigacage::alignedFree( m_memory);387 Gigacage::alignedFree(Gigacage::Primitive, m_memory); 388 388 m_memory = newMemory; 389 389 m_mappedCapacity = desiredSize; -
trunk/Source/WTF/ChangeLog
r220336 r220352 1 2017-08-06 Filip Pizlo <fpizlo@apple.com> 2 3 Primitive auxiliaries and JSValue auxiliaries should have separate gigacages 4 https://bugs.webkit.org/show_bug.cgi?id=174919 5 6 Reviewed by Keith Miller. 7 8 This mirrors the changes from bmalloc/Gigacage.h. 9 10 Also it teaches CagedPtr how to reason about multiple gigacages. 11 12 * wtf/CagedPtr.h: 13 (WTF::CagedPtr::get const): 14 (WTF::CagedPtr::operator[] const): 15 * wtf/Gigacage.cpp: 16 (Gigacage::tryMalloc): 17 (Gigacage::tryAllocateVirtualPages): 18 (Gigacage::freeVirtualPages): 19 (Gigacage::tryAlignedMalloc): 20 (Gigacage::alignedFree): 21 (Gigacage::free): 22 * wtf/Gigacage.h: 23 (Gigacage::disablePrimitiveGigacage): 24 (Gigacage::addPrimitiveDisableCallback): 25 (Gigacage::removePrimitiveDisableCallback): 26 (Gigacage::name): 27 (Gigacage::basePtr): 28 (Gigacage::caged): 29 (Gigacage::isCaged): 30 (Gigacage::tryAlignedMalloc): 31 (Gigacage::alignedFree): 32 (Gigacage::free): 33 (Gigacage::disableGigacage): Deleted. 34 (Gigacage::addDisableCallback): Deleted. 35 (Gigacage::removeDisableCallback): Deleted. 36 1 37 2017-08-07 Brian Burg <bburg@apple.com> 2 38 -
trunk/Source/WTF/wtf/CagedPtr.h
r220165 r220352 30 30 namespace WTF { 31 31 32 template< typename T>32 template<Gigacage::Kind passedKind, typename T> 33 33 class CagedPtr { 34 34 public: 35 static constexpr Gigacage::Kind kind = passedKind; 36 35 37 CagedPtr(T* ptr = nullptr) 36 38 : m_ptr(ptr) … … 41 43 { 42 44 ASSERT(m_ptr); 43 return Gigacage::caged( m_ptr);45 return Gigacage::caged(kind, m_ptr); 44 46 } 45 47 … … 68 70 T& operator*() const { return *get(); } 69 71 T* operator->() const { return get(); } 72 73 template<typename IndexType> 74 T& operator[](IndexType index) const { return get()[index]; } 70 75 71 76 private: -
trunk/Source/WTF/wtf/Gigacage.cpp
r220118 r220352 34 34 35 35 extern "C" { 36 const void*g_gigacageBasePtr;36 void* const g_gigacageBasePtr; 37 37 } 38 38 39 39 namespace Gigacage { 40 40 41 void* tryMalloc( size_t size)41 void* tryMalloc(Kind, size_t size) 42 42 { 43 43 auto result = tryFastMalloc(size); … … 48 48 } 49 49 50 void* tryAllocateVirtualPages( size_t size)50 void* tryAllocateVirtualPages(Kind, size_t size) 51 51 { 52 52 return OSAllocator::reserveUncommitted(size); 53 53 } 54 54 55 void freeVirtualPages( void* basePtr, size_t size)55 void freeVirtualPages(Kind, void* basePtr, size_t size) 56 56 { 57 57 OSAllocator::releaseDecommitted(basePtr, size); … … 68 68 // https://bugs.webkit.org/show_bug.cgi?id=175035 69 69 70 void* tryAlignedMalloc( size_t alignment, size_t size)70 void* tryAlignedMalloc(Kind kind, size_t alignment, size_t size) 71 71 { 72 void* result = bmalloc::api::tryMemalign(alignment, size, bmalloc:: HeapKind::Gigacage);72 void* result = bmalloc::api::tryMemalign(alignment, size, bmalloc::heapKind(kind)); 73 73 WTF::compilerFence(); 74 74 return result; 75 75 } 76 76 77 void alignedFree( void* p)77 void alignedFree(Kind kind, void* p) 78 78 { 79 bmalloc::api::free(p, bmalloc::HeapKind::Gigacage); 79 if (!p) 80 return; 81 RELEASE_ASSERT(isCaged(kind, p)); 82 bmalloc::api::free(p, bmalloc::heapKind(kind)); 80 83 WTF::compilerFence(); 81 84 } 82 85 83 void* tryMalloc( size_t size)86 void* tryMalloc(Kind kind, size_t size) 84 87 { 85 void* result = bmalloc::api::tryMalloc(size, bmalloc:: HeapKind::Gigacage);88 void* result = bmalloc::api::tryMalloc(size, bmalloc::heapKind(kind)); 86 89 WTF::compilerFence(); 87 90 return result; 88 91 } 89 92 90 void free( void* p)93 void free(Kind kind, void* p) 91 94 { 92 bmalloc::api::free(p, bmalloc::HeapKind::Gigacage); 95 if (!p) 96 return; 97 RELEASE_ASSERT(isCaged(kind, p)); 98 bmalloc::api::free(p, bmalloc::heapKind(kind)); 93 99 WTF::compilerFence(); 94 100 } 95 101 96 void* tryAllocateVirtualPages( size_t size)102 void* tryAllocateVirtualPages(Kind kind, size_t size) 97 103 { 98 void* result = bmalloc::api::tryLargeMemalignVirtual(WTF::pageSize(), size, bmalloc:: HeapKind::Gigacage);104 void* result = bmalloc::api::tryLargeMemalignVirtual(WTF::pageSize(), size, bmalloc::heapKind(kind)); 99 105 WTF::compilerFence(); 100 106 return result; 101 107 } 102 108 103 void freeVirtualPages( void* basePtr, size_t)109 void freeVirtualPages(Kind kind, void* basePtr, size_t) 104 110 { 105 bmalloc::api::freeLargeVirtual(basePtr, bmalloc::HeapKind::Gigacage); 111 if (!basePtr) 112 return; 113 RELEASE_ASSERT(isCaged(kind, basePtr)); 114 bmalloc::api::freeLargeVirtual(basePtr, bmalloc::heapKind(kind)); 106 115 WTF::compilerFence(); 107 116 } -
trunk/Source/WTF/wtf/Gigacage.h
r220148 r220352 33 33 34 34 extern "C" { 35 extern WTF_EXPORTDATA const void*g_gigacageBasePtr;35 extern WTF_EXPORTDATA void* const g_gigacageBasePtr; 36 36 } 37 37 38 38 namespace Gigacage { 39 39 40 enum Kind { 41 Primitive, 42 JSValue 43 }; 44 40 45 inline void ensureGigacage() { } 41 inline void disable Gigacage() { }46 inline void disablePrimitiveGigacage() { } 42 47 inline bool shouldBeEnabled() { return false; } 43 48 44 inline void addDisableCallback(void (*)(void*), void*) { } 45 inline void removeDisableCallback(void (*)(void*), void*) { } 49 inline void addPrimitiveDisableCallback(void (*)(void*), void*) { } 50 inline void removePrimitiveDisableCallback(void (*)(void*), void*) { } 51 52 ALWAYS_INLINE const char* name(Kind kind) 53 { 54 switch (kind) { 55 case Primitive: 56 return "Primitive"; 57 case JSValue: 58 return "JSValue"; 59 } 60 RELEASE_ASSERT_NOT_REACHED(); 61 return nullptr; 62 } 63 64 ALWAYS_INLINE void* basePtr(Kind) 65 { 66 return g_gigacageBasePtr; 67 } 46 68 47 69 template<typename T> 48 inline T* caged( T* ptr) { return ptr; }70 inline T* caged(Kind, T* ptr) { return ptr; } 49 71 50 inline bool isCaged( const void*) { return false; }72 inline bool isCaged(Kind, const void*) { return false; } 51 73 52 inline void* tryAlignedMalloc( size_t alignment, size_t size) { return tryFastAlignedMalloc(alignment, size); }53 inline void alignedFree( void* p) { fastAlignedFree(p); }54 WTF_EXPORT_PRIVATE void* tryMalloc( size_t size);55 inline void free( void* p) { fastFree(p); }74 inline void* tryAlignedMalloc(Kind, size_t alignment, size_t size) { return tryFastAlignedMalloc(alignment, size); } 75 inline void alignedFree(Kind, void* p) { fastAlignedFree(p); } 76 WTF_EXPORT_PRIVATE void* tryMalloc(Kind, size_t size); 77 inline void free(Kind, void* p) { fastFree(p); } 56 78 57 WTF_EXPORT_PRIVATE void* tryAllocateVirtualPages( size_t size);58 WTF_EXPORT_PRIVATE void freeVirtualPages( void* basePtr, size_t size);79 WTF_EXPORT_PRIVATE void* tryAllocateVirtualPages(Kind, size_t size); 80 WTF_EXPORT_PRIVATE void freeVirtualPages(Kind, void* basePtr, size_t size); 59 81 60 82 } // namespace Gigacage … … 64 86 namespace Gigacage { 65 87 66 WTF_EXPORT_PRIVATE void* tryAlignedMalloc( size_t alignment, size_t size);67 WTF_EXPORT_PRIVATE void alignedFree( void*);68 WTF_EXPORT_PRIVATE void* tryMalloc( size_t);69 WTF_EXPORT_PRIVATE void free( void*);88 WTF_EXPORT_PRIVATE void* tryAlignedMalloc(Kind, size_t alignment, size_t size); 89 WTF_EXPORT_PRIVATE void alignedFree(Kind, void*); 90 WTF_EXPORT_PRIVATE void* tryMalloc(Kind, size_t); 91 WTF_EXPORT_PRIVATE void free(Kind, void*); 70 92 71 WTF_EXPORT_PRIVATE void* tryAllocateVirtualPages( size_t size);72 WTF_EXPORT_PRIVATE void freeVirtualPages( void* basePtr, size_t size);93 WTF_EXPORT_PRIVATE void* tryAllocateVirtualPages(Kind, size_t size); 94 WTF_EXPORT_PRIVATE void freeVirtualPages(Kind, void* basePtr, size_t size); 73 95 74 96 } // namespace Gigacage -
trunk/Source/WebCore/ChangeLog
r220351 r220352 1 2017-08-06 Filip Pizlo <fpizlo@apple.com> 2 3 Primitive auxiliaries and JSValue auxiliaries should have separate gigacages 4 https://bugs.webkit.org/show_bug.cgi?id=174919 5 6 Reviewed by Keith Miller. 7 8 No new tests because no change in behavior. 9 10 Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be 11 specific about which Gigacage we're using. 12 13 * bindings/js/WebCoreJSClientData.cpp: 14 (WebCore::JSVMClientData::JSVMClientData): 15 * platform/graphics/cocoa/GPUBufferMetal.mm: 16 (WebCore::GPUBuffer::GPUBuffer): 17 1 18 2017-08-07 Basuke Suzuki <Basuke.Suzuki@sony.com> 2 19 -
trunk/Source/WebCore/bindings/js/WebCoreJSClientData.cpp
r220291 r220352 45 45 : m_builtinFunctions(vm) 46 46 , m_builtinNames(&vm) 47 , m_outputConstraintSpace("WebCore Wrapper w/ Output Constraint", vm.heap, &FastMallocAlignedMemoryAllocator::instance())48 , m_globalObjectOutputConstraintSpace("WebCore Global Object w/ Output Constraint", vm.heap, &FastMallocAlignedMemoryAllocator::instance())47 , m_outputConstraintSpace("WebCore Wrapper w/ Output Constraint", vm.heap, vm.fastMallocAllocator.get()) 48 , m_globalObjectOutputConstraintSpace("WebCore Global Object w/ Output Constraint", vm.heap, vm.fastMallocAllocator.get()) 49 49 { 50 50 } -
trunk/Source/WebCore/platform/graphics/cocoa/GPUBufferMetal.mm
r220145 r220352 46 46 size_t pageSize = WTF::pageSize(); 47 47 size_t pageAlignedSize = roundUpToMultipleOf(pageSize, data->byteLength()); 48 void* pageAlignedCopy = Gigacage::tryAlignedMalloc( pageSize, pageAlignedSize);48 void* pageAlignedCopy = Gigacage::tryAlignedMalloc(Gigacage::Primitive, pageSize, pageAlignedSize); 49 49 if (!pageAlignedCopy) 50 50 return; 51 51 memcpy(pageAlignedCopy, data->baseAddress(), data->byteLength()); 52 m_contents = ArrayBuffer::createFromBytes(pageAlignedCopy, data->byteLength(), [] (void* ptr) { Gigacage::alignedFree( ptr); });52 m_contents = ArrayBuffer::createFromBytes(pageAlignedCopy, data->byteLength(), [] (void* ptr) { Gigacage::alignedFree(Gigacage::Primitive, ptr); }); 53 53 m_contents->ref(); 54 54 ArrayBuffer* capturedContents = m_contents.get(); -
trunk/Source/WebKit/ChangeLog
r220336 r220352 1 2017-08-06 Filip Pizlo <fpizlo@apple.com> 2 3 Primitive auxiliaries and JSValue auxiliaries should have separate gigacages 4 https://bugs.webkit.org/show_bug.cgi?id=174919 5 6 Reviewed by Keith Miller. 7 8 The disable callback is all about the primitive gigacage. 9 10 * WebProcess/WebProcess.cpp: 11 (WebKit::primitiveGigacageDisabled): 12 (WebKit::m_webSQLiteDatabaseTracker): 13 (WebKit::gigacageDisabled): Deleted. 14 1 15 2017-08-07 Brian Burg <bburg@apple.com> 2 16 -
trunk/Source/WebKit/WebProcess/WebProcess.cpp
r220148 r220352 147 147 namespace WebKit { 148 148 149 static void gigacageDisabled(void*)149 static void primitiveGigacageDisabled(void*) 150 150 { 151 151 UNREACHABLE_FOR_PLATFORM(); … … 204 204 205 205 if (Gigacage::shouldBeEnabled()) 206 Gigacage::add DisableCallback(gigacageDisabled, nullptr);206 Gigacage::addPrimitiveDisableCallback(primitiveGigacageDisabled, nullptr); 207 207 } 208 208 -
trunk/Source/bmalloc/ChangeLog
r220154 r220352 1 2017-08-06 Filip Pizlo <fpizlo@apple.com> 2 3 Primitive auxiliaries and JSValue auxiliaries should have separate gigacages 4 https://bugs.webkit.org/show_bug.cgi?id=174919 5 6 Reviewed by Keith Miller. 7 8 This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of 9 HeapKind, PrimitiveGigacage and JSValueGigacage. 10 11 The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and 12 NO_INLINE into BNO_INLINE. 13 14 * bmalloc.xcodeproj/project.pbxproj: 15 * bmalloc/Allocator.cpp: 16 (bmalloc::Allocator::refillAllocatorSlowCase): 17 (bmalloc::Allocator::refillAllocator): 18 (bmalloc::Allocator::allocateLarge): 19 (bmalloc::Allocator::allocateLogSizeClass): 20 * bmalloc/AsyncTask.h: 21 * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. 22 * bmalloc/Cache.cpp: 23 (bmalloc::Cache::tryAllocateSlowCaseNullCache): 24 (bmalloc::Cache::allocateSlowCaseNullCache): 25 (bmalloc::Cache::deallocateSlowCaseNullCache): 26 (bmalloc::Cache::reallocateSlowCaseNullCache): 27 * bmalloc/Deallocator.cpp: 28 * bmalloc/Gigacage.cpp: 29 (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): 30 (Gigacage::ensureGigacage): 31 (Gigacage::disablePrimitiveGigacage): 32 (Gigacage::addPrimitiveDisableCallback): 33 (Gigacage::removePrimitiveDisableCallback): 34 (Gigacage::Callbacks::Callbacks): Deleted. 35 (Gigacage::disableGigacage): Deleted. 36 (Gigacage::addDisableCallback): Deleted. 37 (Gigacage::removeDisableCallback): Deleted. 38 * bmalloc/Gigacage.h: 39 (Gigacage::name): 40 (Gigacage::basePtr): 41 (Gigacage::forEachKind): 42 (Gigacage::caged): 43 (Gigacage::isCaged): 44 * bmalloc/Heap.cpp: 45 (bmalloc::Heap::Heap): 46 (bmalloc::Heap::usingGigacage): 47 (bmalloc::Heap::gigacageBasePtr): 48 * bmalloc/Heap.h: 49 * bmalloc/HeapKind.h: 50 (bmalloc::isGigacage): 51 (bmalloc::gigacageKind): 52 (bmalloc::heapKind): 53 * bmalloc/Inline.h: Removed. 54 * bmalloc/Map.h: 55 * bmalloc/PerProcess.h: 56 (bmalloc::PerProcess<T>::getFastCase): 57 (bmalloc::PerProcess<T>::get): 58 (bmalloc::PerProcess<T>::getSlowCase): 59 * bmalloc/PerThread.h: 60 (bmalloc::PerThread<T>::getFastCase): 61 * bmalloc/Vector.h: 62 (bmalloc::Vector<T>::push): 63 (bmalloc::Vector<T>::shrinkCapacity): 64 (bmalloc::Vector<T>::growCapacity): 65 1 66 2017-08-02 Filip Pizlo <fpizlo@apple.com> 2 67 -
trunk/Source/bmalloc/bmalloc.xcodeproj/project.pbxproj
r220118 r220352 52 52 14DD78C718F48D7500950702 /* BAssert.h in Headers */ = {isa = PBXBuildFile; fileRef = 1413E468189EEDE400546D68 /* BAssert.h */; settings = {ATTRIBUTES = (Private, ); }; }; 53 53 14DD78C818F48D7500950702 /* FixedVector.h in Headers */ = {isa = PBXBuildFile; fileRef = 14D9DB4517F2447100EAAB79 /* FixedVector.h */; settings = {ATTRIBUTES = (Private, ); }; }; 54 14DD78C918F48D7500950702 /* Inline.h in Headers */ = {isa = PBXBuildFile; fileRef = 1413E460189DCE1E00546D68 /*Inline.h */; settings = {ATTRIBUTES = (Private, ); }; };54 14DD78C918F48D7500950702 /* BInline.h in Headers */ = {isa = PBXBuildFile; fileRef = 1413E460189DCE1E00546D68 /* BInline.h */; settings = {ATTRIBUTES = (Private, ); }; }; 55 55 14DD78CA18F48D7500950702 /* Mutex.h in Headers */ = {isa = PBXBuildFile; fileRef = 144DCED617A649D90093B2F2 /* Mutex.h */; settings = {ATTRIBUTES = (Private, ); }; }; 56 56 14DD78CB18F48D7500950702 /* PerProcess.h in Headers */ = {isa = PBXBuildFile; fileRef = 14446A0717A61FA400F9EA1D /* PerProcess.h */; settings = {ATTRIBUTES = (Private, ); }; }; … … 95 95 140FA00419CE4B6800FFD3C8 /* LineMetadata.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = LineMetadata.h; path = bmalloc/LineMetadata.h; sourceTree = "<group>"; }; 96 96 14105E8318E14374003A106E /* ObjectType.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = ObjectType.cpp; path = bmalloc/ObjectType.cpp; sourceTree = "<group>"; }; 97 1413E460189DCE1E00546D68 /* Inline.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = Inline.h; path = bmalloc/Inline.h; sourceTree = "<group>"; };97 1413E460189DCE1E00546D68 /* BInline.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = BInline.h; path = bmalloc/BInline.h; sourceTree = "<group>"; }; 98 98 1413E462189DE1CD00546D68 /* BumpAllocator.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; lineEnding = 0; name = BumpAllocator.h; path = bmalloc/BumpAllocator.h; sourceTree = "<group>"; }; 99 99 1413E468189EEDE400546D68 /* BAssert.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = BAssert.h; path = bmalloc/BAssert.h; sourceTree = "<group>"; }; … … 288 288 1413E468189EEDE400546D68 /* BAssert.h */, 289 289 0F5BF1721F23C5710029D91D /* BExport.h */, 290 1413E460189DCE1E00546D68 /* BInline.h */, 290 291 14C919C818FCC59F0028DB43 /* BPlatform.h */, 291 292 14D9DB4517F2447100EAAB79 /* FixedVector.h */, 292 293 0F5BF1461F22A8B10029D91D /* HeapKind.h */, 293 1413E460189DCE1E00546D68 /* Inline.h */,294 294 141D9AFF1C8E51C0000ABBA0 /* List.h */, 295 295 4426E27E1C838EE0008EB042 /* Logging.cpp */, … … 354 354 1400274918F89C1300115C97 /* Heap.h in Headers */, 355 355 0F5BF1491F22A8D80029D91D /* PerHeapKind.h in Headers */, 356 14DD78C918F48D7500950702 /* Inline.h in Headers */,356 14DD78C918F48D7500950702 /* BInline.h in Headers */, 357 357 144C07F51C7B70260051BB6A /* LargeMap.h in Headers */, 358 358 14C8992D1CC578330027A057 /* LargeRange.h in Headers */, -
trunk/Source/bmalloc/bmalloc/Allocator.cpp
r220118 r220352 150 150 } 151 151 152 NO_INLINE void Allocator::refillAllocatorSlowCase(BumpAllocator& allocator, size_t sizeClass)152 BNO_INLINE void Allocator::refillAllocatorSlowCase(BumpAllocator& allocator, size_t sizeClass) 153 153 { 154 154 BumpRangeCache& bumpRangeCache = m_bumpRangeCaches[sizeClass]; … … 159 159 } 160 160 161 INLINE void Allocator::refillAllocator(BumpAllocator& allocator, size_t sizeClass)161 BINLINE void Allocator::refillAllocator(BumpAllocator& allocator, size_t sizeClass) 162 162 { 163 163 BumpRangeCache& bumpRangeCache = m_bumpRangeCaches[sizeClass]; … … 167 167 } 168 168 169 NO_INLINE void* Allocator::allocateLarge(size_t size)169 BNO_INLINE void* Allocator::allocateLarge(size_t size) 170 170 { 171 171 std::lock_guard<StaticMutex> lock(Heap::mutex()); … … 173 173 } 174 174 175 NO_INLINE void* Allocator::allocateLogSizeClass(size_t size)175 BNO_INLINE void* Allocator::allocateLogSizeClass(size_t size) 176 176 { 177 177 size_t sizeClass = bmalloc::sizeClass(size); -
trunk/Source/bmalloc/bmalloc/AsyncTask.h
r217456 r220352 28 28 29 29 #include "BAssert.h" 30 #include " Inline.h"30 #include "BInline.h" 31 31 #include "Mutex.h" 32 32 #include "Sizes.h" -
trunk/Source/bmalloc/bmalloc/BInline.h
r220351 r220352 24 24 */ 25 25 26 #ifndef Inline_h27 #define Inline_h26 #ifndef BInline_h 27 #define BInline_h 28 28 29 #define INLINE __attribute__((always_inline)) inline29 #define BINLINE __attribute__((always_inline)) inline 30 30 31 #define NO_INLINE __attribute__((noinline))31 #define BNO_INLINE __attribute__((noinline)) 32 32 33 #endif // Inline_h33 #endif // BInline_h -
trunk/Source/bmalloc/bmalloc/Cache.cpp
r220118 r220352 24 24 */ 25 25 26 #include "BInline.h" 26 27 #include "Cache.h" 27 28 #include "Heap.h" 28 #include "Inline.h"29 29 #include "PerProcess.h" 30 30 … … 47 47 } 48 48 49 NO_INLINE void* Cache::tryAllocateSlowCaseNullCache(HeapKind heapKind, size_t size)49 BNO_INLINE void* Cache::tryAllocateSlowCaseNullCache(HeapKind heapKind, size_t size) 50 50 { 51 51 return PerThread<PerHeapKind<Cache>>::getSlowCase()->at(heapKind).allocator().tryAllocate(size); 52 52 } 53 53 54 NO_INLINE void* Cache::allocateSlowCaseNullCache(HeapKind heapKind, size_t size)54 BNO_INLINE void* Cache::allocateSlowCaseNullCache(HeapKind heapKind, size_t size) 55 55 { 56 56 return PerThread<PerHeapKind<Cache>>::getSlowCase()->at(heapKind).allocator().allocate(size); 57 57 } 58 58 59 NO_INLINE void* Cache::allocateSlowCaseNullCache(HeapKind heapKind, size_t alignment, size_t size)59 BNO_INLINE void* Cache::allocateSlowCaseNullCache(HeapKind heapKind, size_t alignment, size_t size) 60 60 { 61 61 return PerThread<PerHeapKind<Cache>>::getSlowCase()->at(heapKind).allocator().allocate(alignment, size); 62 62 } 63 63 64 NO_INLINE void Cache::deallocateSlowCaseNullCache(HeapKind heapKind, void* object)64 BNO_INLINE void Cache::deallocateSlowCaseNullCache(HeapKind heapKind, void* object) 65 65 { 66 66 PerThread<PerHeapKind<Cache>>::getSlowCase()->at(heapKind).deallocator().deallocate(object); 67 67 } 68 68 69 NO_INLINE void* Cache::reallocateSlowCaseNullCache(HeapKind heapKind, void* object, size_t newSize)69 BNO_INLINE void* Cache::reallocateSlowCaseNullCache(HeapKind heapKind, void* object, size_t newSize) 70 70 { 71 71 return PerThread<PerHeapKind<Cache>>::getSlowCase()->at(heapKind).allocator().reallocate(object, newSize); -
trunk/Source/bmalloc/bmalloc/Deallocator.cpp
r220118 r220352 25 25 26 26 #include "BAssert.h" 27 #include "BInline.h" 27 28 #include "Chunk.h" 28 29 #include "Deallocator.h" 29 30 #include "DebugHeap.h" 30 31 #include "Heap.h" 31 #include "Inline.h"32 32 #include "Object.h" 33 33 #include "PerProcess.h" -
trunk/Source/bmalloc/bmalloc/Gigacage.cpp
r220148 r220352 35 35 // FIXME: Ask dyld to put this in its own page, and mprotect the page after we ensure the gigacage. 36 36 // https://bugs.webkit.org/show_bug.cgi?id=174972 37 void* g_gigacageBasePtr; 37 void* g_primitiveGigacageBasePtr; 38 void* g_jsValueGigacageBasePtr; 38 39 39 40 using namespace bmalloc; … … 54 55 }; 55 56 56 struct Callbacks {57 Callbacks(std::lock_guard<StaticMutex>&) { }57 struct PrimitiveDisableCallbacks { 58 PrimitiveDisableCallbacks(std::lock_guard<StaticMutex>&) { } 58 59 59 60 Vector<Callback> callbacks; … … 70 71 return; 71 72 72 void* basePtr = tryVMAllocate(GIGACAGE_SIZE, GIGACAGE_SIZE + GIGACAGE_RUNWAY); 73 if (!basePtr) 74 return; 75 76 vmDeallocatePhysicalPages(basePtr, GIGACAGE_SIZE + GIGACAGE_RUNWAY); 77 78 g_gigacageBasePtr = basePtr; 73 forEachKind( 74 [&] (Kind kind) { 75 // FIXME: Randomize where this goes. 76 // https://bugs.webkit.org/show_bug.cgi?id=175245 77 basePtr(kind) = tryVMAllocate(GIGACAGE_SIZE, GIGACAGE_SIZE + GIGACAGE_RUNWAY); 78 if (!basePtr(kind)) { 79 fprintf(stderr, "FATAL: Could not allocate %s gigacage.\n", name(kind)); 80 BCRASH(); 81 } 82 83 vmDeallocatePhysicalPages(basePtr(kind), GIGACAGE_SIZE + GIGACAGE_RUNWAY); 84 }); 79 85 }); 80 86 #endif // GIGACAGE_ENABLED 81 87 } 82 88 83 void disable Gigacage()89 void disablePrimitiveGigacage() 84 90 { 85 91 ensureGigacage(); 86 if (!g_ gigacageBasePtr) {92 if (!g_primitiveGigacageBasePtr) { 87 93 // It was never enabled. That means that we never even saved any callbacks. Or, we had already disabled 88 94 // it before, and already called the callbacks. … … 90 96 } 91 97 92 Callbacks& callbacks = *PerProcess<Callbacks>::get();93 std::unique_lock<StaticMutex> lock(PerProcess< Callbacks>::mutex());98 PrimitiveDisableCallbacks& callbacks = *PerProcess<PrimitiveDisableCallbacks>::get(); 99 std::unique_lock<StaticMutex> lock(PerProcess<PrimitiveDisableCallbacks>::mutex()); 94 100 for (Callback& callback : callbacks.callbacks) 95 101 callback.function(callback.argument); 96 102 callbacks.callbacks.shrink(0); 97 g_ gigacageBasePtr = nullptr;103 g_primitiveGigacageBasePtr = nullptr; 98 104 } 99 105 100 void add DisableCallback(void (*function)(void*), void* argument)106 void addPrimitiveDisableCallback(void (*function)(void*), void* argument) 101 107 { 102 108 ensureGigacage(); 103 if (!g_ gigacageBasePtr) {109 if (!g_primitiveGigacageBasePtr) { 104 110 // It was already disabled or we were never able to enable it. 105 111 function(argument); … … 107 113 } 108 114 109 Callbacks& callbacks = *PerProcess<Callbacks>::get();110 std::unique_lock<StaticMutex> lock(PerProcess< Callbacks>::mutex());115 PrimitiveDisableCallbacks& callbacks = *PerProcess<PrimitiveDisableCallbacks>::get(); 116 std::unique_lock<StaticMutex> lock(PerProcess<PrimitiveDisableCallbacks>::mutex()); 111 117 callbacks.callbacks.push(Callback(function, argument)); 112 118 } 113 119 114 void remove DisableCallback(void (*function)(void*), void* argument)120 void removePrimitiveDisableCallback(void (*function)(void*), void* argument) 115 121 { 116 Callbacks& callbacks = *PerProcess<Callbacks>::get();117 std::unique_lock<StaticMutex> lock(PerProcess< Callbacks>::mutex());122 PrimitiveDisableCallbacks& callbacks = *PerProcess<PrimitiveDisableCallbacks>::get(); 123 std::unique_lock<StaticMutex> lock(PerProcess<PrimitiveDisableCallbacks>::mutex()); 118 124 for (size_t i = 0; i < callbacks.callbacks.size(); ++i) { 119 125 if (callbacks.callbacks[i].function == function -
trunk/Source/bmalloc/bmalloc/Gigacage.h
r220165 r220352 28 28 #include "BAssert.h" 29 29 #include "BExport.h" 30 #include "BInline.h" 30 31 #include "BPlatform.h" 31 32 #include <inttypes.h> … … 45 46 #endif 46 47 47 extern "C" BEXPORT void* g_gigacageBasePtr; 48 extern "C" BEXPORT void* g_primitiveGigacageBasePtr; 49 extern "C" BEXPORT void* g_jsValueGigacageBasePtr; 48 50 49 51 namespace Gigacage { 50 52 53 enum Kind { 54 Primitive, 55 JSValue 56 }; 57 51 58 BEXPORT void ensureGigacage(); 52 59 53 BEXPORT void disable Gigacage();60 BEXPORT void disablePrimitiveGigacage(); 54 61 55 // This will call the disable callback immediately if the Gigacage is currently disabled. 56 BEXPORT void addDisableCallback(void (*)(void*), void*); 57 BEXPORT void removeDisableCallback(void (*)(void*), void*); 62 // This will call the disable callback immediately if the Primitive Gigacage is currently disabled. 63 BEXPORT void addPrimitiveDisableCallback(void (*)(void*), void*); 64 BEXPORT void removePrimitiveDisableCallback(void (*)(void*), void*); 65 66 BINLINE const char* name(Kind kind) 67 { 68 switch (kind) { 69 case Primitive: 70 return "Primitive"; 71 case JSValue: 72 return "JSValue"; 73 } 74 BCRASH(); 75 return nullptr; 76 } 77 78 BINLINE void*& basePtr(Kind kind) 79 { 80 switch (kind) { 81 case Primitive: 82 return g_primitiveGigacageBasePtr; 83 case JSValue: 84 return g_jsValueGigacageBasePtr; 85 } 86 BCRASH(); 87 return g_primitiveGigacageBasePtr; 88 } 89 90 template<typename Func> 91 void forEachKind(const Func& func) 92 { 93 func(Primitive); 94 func(JSValue); 95 } 58 96 59 97 template<typename T> 60 T* caged(T* ptr)98 BINLINE T* caged(Kind kind, T* ptr) 61 99 { 62 100 BASSERT(ptr); 63 void* gigacageBasePtr = g_gigacageBasePtr;101 void* gigacageBasePtr = basePtr(kind); 64 102 if (!gigacageBasePtr) 65 103 return ptr; … … 69 107 } 70 108 71 inline bool isCaged(const void* ptr)109 BINLINE bool isCaged(Kind kind, const void* ptr) 72 110 { 73 return caged( ptr) == ptr;111 return caged(kind, ptr) == ptr; 74 112 } 75 113 -
trunk/Source/bmalloc/bmalloc/Heap.cpp
r220154 r220352 60 60 #if GIGACAGE_ENABLED 61 61 if (usingGigacage()) { 62 RELEASE_BASSERT(g _gigacageBasePtr);63 m_largeFree.add(LargeRange(g _gigacageBasePtr, GIGACAGE_SIZE, 0));62 RELEASE_BASSERT(gigacageBasePtr()); 63 m_largeFree.add(LargeRange(gigacageBasePtr(), GIGACAGE_SIZE, 0)); 64 64 } 65 65 #endif … … 71 71 bool Heap::usingGigacage() 72 72 { 73 return m_kind == HeapKind::Gigacage && g_gigacageBasePtr; 73 return isGigacage(m_kind) && gigacageBasePtr(); 74 } 75 76 void* Heap::gigacageBasePtr() 77 { 78 return Gigacage::basePtr(gigacageKind(m_kind)); 74 79 } 75 80 -
trunk/Source/bmalloc/bmalloc/Heap.h
r220148 r220352 90 90 91 91 bool usingGigacage(); 92 void* gigacageBasePtr(); // May crash if !usingGigacage(). 92 93 93 94 void initializeLineMetadata(); -
trunk/Source/bmalloc/bmalloc/HeapKind.h
r220118 r220352 26 26 #pragma once 27 27 28 #include "BAssert.h" 29 #include "BInline.h" 30 #include "Gigacage.h" 31 28 32 namespace bmalloc { 29 33 30 34 enum class HeapKind { 31 35 Primary, 32 Gigacage 36 PrimitiveGigacage, 37 JSValueGigacage 33 38 }; 34 39 35 static constexpr unsigned numHeaps = 2; 40 static constexpr unsigned numHeaps = 3; 41 42 BINLINE bool isGigacage(HeapKind heapKind) 43 { 44 switch (heapKind) { 45 case HeapKind::Primary: 46 return false; 47 case HeapKind::PrimitiveGigacage: 48 case HeapKind::JSValueGigacage: 49 return true; 50 } 51 BCRASH(); 52 return false; 53 } 54 55 BINLINE Gigacage::Kind gigacageKind(HeapKind kind) 56 { 57 switch (kind) { 58 case HeapKind::Primary: 59 BCRASH(); 60 return Gigacage::Primitive; 61 case HeapKind::PrimitiveGigacage: 62 return Gigacage::Primitive; 63 case HeapKind::JSValueGigacage: 64 return Gigacage::JSValue; 65 } 66 BCRASH(); 67 return Gigacage::Primitive; 68 } 69 70 BINLINE HeapKind heapKind(Gigacage::Kind kind) 71 { 72 switch (kind) { 73 case Gigacage::Primitive: 74 return HeapKind::PrimitiveGigacage; 75 case Gigacage::JSValue: 76 return HeapKind::JSValueGigacage; 77 } 78 BCRASH(); 79 return HeapKind::Primary; 80 } 36 81 37 82 } // namespace bmalloc -
trunk/Source/bmalloc/bmalloc/Map.h
r219702 r220352 27 27 #define Map_h 28 28 29 #include " Inline.h"29 #include "BInline.h" 30 30 #include "Sizes.h" 31 31 #include "Vector.h" -
trunk/Source/bmalloc/bmalloc/PerProcess.h
r220079 r220352 27 27 #define PerProcess_h 28 28 29 #include " Inline.h"29 #include "BInline.h" 30 30 #include "Sizes.h" 31 31 #include "StaticMutex.h" … … 72 72 73 73 template<typename T> 74 INLINE T* PerProcess<T>::getFastCase()74 BINLINE T* PerProcess<T>::getFastCase() 75 75 { 76 76 return s_object.load(std::memory_order_consume); … … 78 78 79 79 template<typename T> 80 INLINE T* PerProcess<T>::get()80 BINLINE T* PerProcess<T>::get() 81 81 { 82 82 T* object = getFastCase(); … … 87 87 88 88 template<typename T> 89 NO_INLINE T* PerProcess<T>::getSlowCase()89 BNO_INLINE T* PerProcess<T>::getSlowCase() 90 90 { 91 91 std::lock_guard<StaticMutex> lock(s_mutex); -
trunk/Source/bmalloc/bmalloc/PerThread.h
r220118 r220352 27 27 #define PerThread_h 28 28 29 #include "BInline.h" 29 30 #include "BPlatform.h" 30 #include "Inline.h"31 31 #include "PerHeapKind.h" 32 32 #include "VMAllocate.h" … … 116 116 117 117 template<typename T> 118 INLINE T* PerThread<T>::getFastCase()118 BINLINE T* PerThread<T>::getFastCase() 119 119 { 120 120 return static_cast<T*>(PerThreadStorage<T>::get()); -
trunk/Source/bmalloc/bmalloc/Vector.h
r201772 r220352 27 27 #define Vector_h 28 28 29 #include " Inline.h"29 #include "BInline.h" 30 30 #include "VMAllocate.h" 31 31 #include <cstddef> … … 120 120 121 121 template<typename T> 122 INLINE void Vector<T>::push(const T& value)122 BINLINE void Vector<T>::push(const T& value) 123 123 { 124 124 if (m_size == m_capacity) … … 213 213 214 214 template<typename T> 215 NO_INLINE void Vector<T>::shrinkCapacity()215 BNO_INLINE void Vector<T>::shrinkCapacity() 216 216 { 217 217 size_t newCapacity = max(initialCapacity(), m_capacity / shrinkFactor); … … 220 220 221 221 template<typename T> 222 NO_INLINE void Vector<T>::growCapacity()222 BNO_INLINE void Vector<T>::growCapacity() 223 223 { 224 224 size_t newCapacity = max(initialCapacity(), m_size * growFactor);
Note: See TracChangeset
for help on using the changeset viewer.