Changeset 287968 in webkit


Ignore:
Timestamp:
Jan 12, 2022 6:52:00 PM (2 years ago)
Author:
fpizlo@apple.com
Message:

[libpas] thread_local_cache should not be allocated in the compact heap (cherry pick 11afcedfb5968f6894379ff1a41dd449ba7745f6)
​https://bugs.webkit.org/show_bug.cgi?id=235096

Reviewed by Yusuke Suzuki.

Thread local caches can get quite large because of how libpas uses them, we can allocate one per
thread, and we reallocate them with exponential resizing, so there's a lot of wasted space and a
decent amount of fragmentation. This shows up as occasional crashes trying to allocate a thread local
cache out of the compact heap.

This moves thread local caches out of the compact heap. They were only ever there because partial views
sometimes need to point to the local_allocator's bitvector, but that's part of the thread local cache.
So, that means that either the partial views' bits pointer cannot be a compact pointer, or the
thread_local_cache needs to be in the compact heap. So, the thread_local_cache ended up in the compact
heap to keep that pointer small.

This change works around the problem: it's rare that the partial views' bits pointer points at
the local_allocator's bits, and none of the fast path cases where we access that pointer will ever see
it in that state. So, this makes the pointer either point to a utility-heap-allocated box that contains
the full pointer, or it points at the actual array allocated in the compact heap. The utility heap is
in the compact heap, so the compact pointer can point at either one. The implementation of this is
encapsulated as pas_lenient_compact_ptr. It's a bit gross; storing to it only works when you're holding
the heap lock, for example.

This is perf-neutral on Speedometer. This is perf-neutral on JS2 cli with full JSC isoheaps (i.e. the
patch from bug 231938). It's a 0.4% regression on RAMification with full JSC isoheaps, but I'm not
going to worry about that because trunk doesn't have full JSC isoheaps, and JSC isoheaps requires some
change like this to work reliably (currently it'll randomly run out of compact heap).

  • bmalloc.xcodeproj/project.pbxproj:
  • libpas/libpas.xcodeproj/project.pbxproj:
  • libpas/src/libpas/pas_enumerate_segregated_heaps.c:

(enumerate_partial_view):

  • libpas/src/libpas/pas_full_alloc_bits_inlines.h:

(pas_full_alloc_bits_create_for_partial_but_not_primordial):
(pas_full_alloc_bits_create_for_partial):

  • libpas/src/libpas/pas_lenient_compact_ptr.h: Added.
  • libpas/src/libpas/pas_lenient_compact_ptr_inlines.h: Added.
  • libpas/src/libpas/pas_lenient_compact_unsigned_ptr.c: Added.
  • libpas/src/libpas/pas_lenient_compact_unsigned_ptr.h: Added.
  • libpas/src/libpas/pas_local_allocator.c:

(pas_local_allocator_move):

  • libpas/src/libpas/pas_local_allocator_inlines.h:

(pas_local_allocator_set_up_free_bits):
(pas_local_allocator_start_allocating_in_primordial_partial_view):
(pas_local_allocator_bless_primordial_partial_view_before_stopping):

  • libpas/src/libpas/pas_segregated_partial_view.c:

(pas_segregated_partial_view_create):
(compute_summary):

  • libpas/src/libpas/pas_segregated_partial_view.h:
  • libpas/src/libpas/pas_segregated_view_allocator_inlines.h:

(pas_segregated_view_will_start_allocating):

  • libpas/src/libpas/pas_thread_local_cache.c:

(deallocate):
(allocate_cache):

  • libpas/src/test/LotsOfHeapsAndThreads.cpp: Added.

(std::testLotsOfHeapsAndThreads):
(addLotsOfHeapsAndThreadsTests):

  • libpas/src/test/TestHarness.cpp:

(main):

Location:
trunk/Source/bmalloc
Files:
5 added
12 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/bmalloc/ChangeLog

    r287956 r287968  
     12022-01-12  Filip Pizlo  <fpizlo@apple.com>
     2
     3        [libpas] thread_local_cache should not be allocated in the compact heap (cherry pick 11afcedfb5968f6894379ff1a41dd449ba7745f6)
     4        https://bugs.webkit.org/show_bug.cgi?id=235096
     5
     6        Reviewed by Yusuke Suzuki.
     7
     8        Thread local caches can get quite large because of how libpas uses them, we can allocate one per
     9        thread, and we reallocate them with exponential resizing, so there's a lot of wasted space and a
     10        decent amount of fragmentation. This shows up as occasional crashes trying to allocate a thread local
     11        cache out of the compact heap.
     12
     13        This moves thread local caches out of the compact heap. They were only ever there because partial views
     14        sometimes need to point to the local_allocator's bitvector, but that's part of the thread local cache.
     15        So, that means that either the partial views' bits pointer cannot be a compact pointer, or the
     16        thread_local_cache needs to be in the compact heap. So, the thread_local_cache ended up in the compact
     17        heap to keep that pointer small.
     18
     19        This change works around the problem: it's rare that the partial views' bits pointer points at
     20        the local_allocator's bits, and none of the fast path cases where we access that pointer will ever see
     21        it in that state. So, this makes the pointer either point to a utility-heap-allocated box that contains
     22        the full pointer, or it points at the actual array allocated in the compact heap. The utility heap is
     23        in the compact heap, so the compact pointer can point at either one. The implementation of this is
     24        encapsulated as pas_lenient_compact_ptr. It's a bit gross; storing to it only works when you're holding
     25        the heap lock, for example.
     26
     27        This is perf-neutral on Speedometer. This is perf-neutral on JS2 cli with full JSC isoheaps (i.e. the
     28        patch from bug 231938). It's a 0.4% regression on RAMification with full JSC isoheaps, but I'm not
     29        going to worry about that because trunk doesn't have full JSC isoheaps, and JSC isoheaps requires some
     30        change like this to work reliably (currently it'll randomly run out of compact heap).
     31
     32        * bmalloc.xcodeproj/project.pbxproj:
     33        * libpas/libpas.xcodeproj/project.pbxproj:
     34        * libpas/src/libpas/pas_enumerate_segregated_heaps.c:
     35        (enumerate_partial_view):
     36        * libpas/src/libpas/pas_full_alloc_bits_inlines.h:
     37        (pas_full_alloc_bits_create_for_partial_but_not_primordial):
     38        (pas_full_alloc_bits_create_for_partial):
     39        * libpas/src/libpas/pas_lenient_compact_ptr.h: Added.
     40        * libpas/src/libpas/pas_lenient_compact_ptr_inlines.h: Added.
     41        * libpas/src/libpas/pas_lenient_compact_unsigned_ptr.c: Added.
     42        * libpas/src/libpas/pas_lenient_compact_unsigned_ptr.h: Added.
     43        * libpas/src/libpas/pas_local_allocator.c:
     44        (pas_local_allocator_move):
     45        * libpas/src/libpas/pas_local_allocator_inlines.h:
     46        (pas_local_allocator_set_up_free_bits):
     47        (pas_local_allocator_start_allocating_in_primordial_partial_view):
     48        (pas_local_allocator_bless_primordial_partial_view_before_stopping):
     49        * libpas/src/libpas/pas_segregated_partial_view.c:
     50        (pas_segregated_partial_view_create):
     51        (compute_summary):
     52        * libpas/src/libpas/pas_segregated_partial_view.h:
     53        * libpas/src/libpas/pas_segregated_view_allocator_inlines.h:
     54        (pas_segregated_view_will_start_allocating):
     55        * libpas/src/libpas/pas_thread_local_cache.c:
     56        (deallocate):
     57        (allocate_cache):
     58        * libpas/src/test/LotsOfHeapsAndThreads.cpp: Added.
     59        (std::testLotsOfHeapsAndThreads):
     60        (addLotsOfHeapsAndThreadsTests):
     61        * libpas/src/test/TestHarness.cpp:
     62        (main):
     63
    1642022-01-12  Elliott Williams  <emw@apple.com>
    265

  • trunk/Source/bmalloc/bmalloc.xcodeproj/project.pbxproj

    r287956 r287968  
    617617                2C48134527406A3E006CAB55 /* pas_large_expendable_memory.h in Headers */ = {isa = PBXBuildFile; fileRef = 2C48133F27406A3E006CAB55 /* pas_large_expendable_memory.h */; settings = {ATTRIBUTES = (Private, ); }; };
    618618                2C91E552271CE47A00D67FF9 /* pas_size_lookup_mode.h in Headers */ = {isa = PBXBuildFile; fileRef = 2C91E551271CE47A00D67FF9 /* pas_size_lookup_mode.h */; settings = {ATTRIBUTES = (Private, ); }; };
     619                2C971D04278E798700C9E129 /* pas_lenient_compact_ptr_inlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 2C971D00278E798700C9E129 /* pas_lenient_compact_ptr_inlines.h */; };
     620                2C971D05278E798700C9E129 /* pas_lenient_compact_ptr.h in Headers */ = {isa = PBXBuildFile; fileRef = 2C971D01278E798700C9E129 /* pas_lenient_compact_ptr.h */; settings = {ATTRIBUTES = (Private, ); }; };
     621                2C971D06278E798700C9E129 /* pas_lenient_compact_unsigned_ptr.h in Headers */ = {isa = PBXBuildFile; fileRef = 2C971D02278E798700C9E129 /* pas_lenient_compact_unsigned_ptr.h */; settings = {ATTRIBUTES = (Private, ); }; };
     622                2C971D07278E798700C9E129 /* pas_lenient_compact_unsigned_ptr.c in Sources */ = {isa = PBXBuildFile; fileRef = 2C971D03278E798700C9E129 /* pas_lenient_compact_unsigned_ptr.c */; };
    619623                2CE2AE2D27596DEB00D02BBC /* pas_segregated_page_role.h in Headers */ = {isa = PBXBuildFile; fileRef = 2CE2AE2627596DEA00D02BBC /* pas_segregated_page_role.h */; settings = {ATTRIBUTES = (Private, ); }; };
    620624                2CE2AE2E27596DEB00D02BBC /* pas_page_base_and_kind.h in Headers */ = {isa = PBXBuildFile; fileRef = 2CE2AE2727596DEA00D02BBC /* pas_page_base_and_kind.h */; settings = {ATTRIBUTES = (Private, ); }; };
    … …  
    12761280                2C48133F27406A3E006CAB55 /* pas_large_expendable_memory.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = pas_large_expendable_memory.h; path = libpas/src/libpas/pas_large_expendable_memory.h; sourceTree = "<group>"; };
    12771281                2C91E551271CE47A00D67FF9 /* pas_size_lookup_mode.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = pas_size_lookup_mode.h; path = libpas/src/libpas/pas_size_lookup_mode.h; sourceTree = "<group>"; };
     1282                2C971D00278E798700C9E129 /* pas_lenient_compact_ptr_inlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = pas_lenient_compact_ptr_inlines.h; path = libpas/src/libpas/pas_lenient_compact_ptr_inlines.h; sourceTree = "<group>"; };
     1283                2C971D01278E798700C9E129 /* pas_lenient_compact_ptr.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = pas_lenient_compact_ptr.h; path = libpas/src/libpas/pas_lenient_compact_ptr.h; sourceTree = "<group>"; };
     1284                2C971D02278E798700C9E129 /* pas_lenient_compact_unsigned_ptr.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = pas_lenient_compact_unsigned_ptr.h; path = libpas/src/libpas/pas_lenient_compact_unsigned_ptr.h; sourceTree = "<group>"; };
     1285                2C971D03278E798700C9E129 /* pas_lenient_compact_unsigned_ptr.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; name = pas_lenient_compact_unsigned_ptr.c; path = libpas/src/libpas/pas_lenient_compact_unsigned_ptr.c; sourceTree = "<group>"; };
    12781286                2CE2AE2627596DEA00D02BBC /* pas_segregated_page_role.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = pas_segregated_page_role.h; path = libpas/src/libpas/pas_segregated_page_role.h; sourceTree = "<group>"; };
    12791287                2CE2AE2727596DEA00D02BBC /* pas_page_base_and_kind.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = pas_page_base_and_kind.h; path = libpas/src/libpas/pas_page_base_and_kind.h; sourceTree = "<group>"; };
    … …  
    17021710                                0FC40AA32451498E00876DA0 /* pas_large_utility_free_heap.c */,
    17031711                                0FC40AF12451499300876DA0 /* pas_large_utility_free_heap.h */,
     1712                                2C971D00278E798700C9E129 /* pas_lenient_compact_ptr_inlines.h */,
     1713                                2C971D01278E798700C9E129 /* pas_lenient_compact_ptr.h */,
     1714                                2C971D03278E798700C9E129 /* pas_lenient_compact_unsigned_ptr.c */,
     1715                                2C971D02278E798700C9E129 /* pas_lenient_compact_unsigned_ptr.h */,
    17041716                                0F8A812425F83E2400790B4A /* pas_line_word_config.h */,
    17051717                                0FC40A7D2451498B00876DA0 /* pas_list_direction.h */,
    … …  
    21802192                                0F7EB8351F9541B000F1ABCB /* IsoTLSEntryInlines.h in Headers */,
    21812193                                0F7EB8461F9541B000F1ABCB /* IsoTLSInlines.h in Headers */,
     2194                                2C971D04278E798700C9E129 /* pas_lenient_compact_ptr_inlines.h in Headers */,
    21822195                                0F7EB8401F9541B000F1ABCB /* IsoTLSLayout.h in Headers */,
    21832196                                0F5193EF266C42AD00483A2C /* jit_heap.h in Headers */,
    … …  
    21872200                                14C8992D1CC578330027A057 /* LargeRange.h in Headers */,
    21882201                                140FA00519CE4B6800FFD3C8 /* LineMetadata.h in Headers */,
     2202                                2C971D06278E798700C9E129 /* pas_lenient_compact_unsigned_ptr.h in Headers */,
    21892203                                141D9B001C8E51C0000ABBA0 /* List.h in Headers */,
    21902204                                4426E2811C838EE0008EB042 /* Logging.h in Headers */,
    … …  
    23132327                                0FC409E92451496400876DA0 /* pas_ensure_heap_with_page_caches.h in Headers */,
    23142328                                0F87002C25AF89C9000E1ABF /* pas_enumerable_page_malloc.h in Headers */,
     2329                                2C971D05278E798700C9E129 /* pas_lenient_compact_ptr.h in Headers */,
    23152330                                0F87002E25AF89C9000E1ABF /* pas_enumerable_range_list.h in Headers */,
    23162331                                0F87FFEC25AF897C000E1ABF /* pas_enumerate_bitfit_heaps.h in Headers */,
    … …  
    26842699                                E378A9DF246B68720029C2BB /* ObjectTypeTable.cpp in Sources */,
    26852700                                0F18F84725C3467700721C2A /* pagesize64k_heap.c in Sources */,
     2701                                2C971D07278E798700C9E129 /* pas_lenient_compact_unsigned_ptr.c in Sources */,
    26862702                                0F18F84925C3467700721C2A /* pagesize64k_heap_config.c in Sources */,
    26872703                                0FC4095D2451494400876DA0 /* pas_alignment.c in Sources */,

  • trunk/Source/bmalloc/libpas/libpas.xcodeproj/project.pbxproj

    r287192 r287968  
    572572                2C85DC4127128F0F00367905 /* pas_try_allocate_intrinsic.h in Headers */ = {isa = PBXBuildFile; fileRef = 2C85DC4027128F0F00367905 /* pas_try_allocate_intrinsic.h */; };
    573573                2C91E5502718DA9A00D67FF9 /* pas_size_lookup_mode.h in Headers */ = {isa = PBXBuildFile; fileRef = 2C91E54F2718DA9A00D67FF9 /* pas_size_lookup_mode.h */; };
     574                2C971CFA278E11C300C9E129 /* pas_lenient_compact_ptr.h in Headers */ = {isa = PBXBuildFile; fileRef = 2C971CF8278E11C300C9E129 /* pas_lenient_compact_ptr.h */; };
     575                2C971CFB278E11C300C9E129 /* pas_lenient_compact_unsigned_ptr.h in Headers */ = {isa = PBXBuildFile; fileRef = 2C971CF9278E11C300C9E129 /* pas_lenient_compact_unsigned_ptr.h */; };
     576                2C971CFE278E136300C9E129 /* pas_lenient_compact_unsigned_ptr.c in Sources */ = {isa = PBXBuildFile; fileRef = 2C971CFC278E136300C9E129 /* pas_lenient_compact_unsigned_ptr.c */; };
     577                2C971CFF278E136300C9E129 /* pas_lenient_compact_ptr_inlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 2C971CFD278E136300C9E129 /* pas_lenient_compact_ptr_inlines.h */; };
     578                2CB9B153278F6C85003A8C1B /* LotsOfHeapsAndThreads.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 2CB9B152278F6C85003A8C1B /* LotsOfHeapsAndThreads.cpp */; };
    574579                2CE2AE35275A953E00D02BBC /* BitfitTests.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 2CE2AE34275A953E00D02BBC /* BitfitTests.cpp */; };
    575580/* End PBXBuildFile section */
    … …  
    12601265                2C85DC4027128F0F00367905 /* pas_try_allocate_intrinsic.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = pas_try_allocate_intrinsic.h; sourceTree = "<group>"; };
    12611266                2C91E54F2718DA9A00D67FF9 /* pas_size_lookup_mode.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = pas_size_lookup_mode.h; sourceTree = "<group>"; };
     1267                2C971CF8278E11C300C9E129 /* pas_lenient_compact_ptr.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = pas_lenient_compact_ptr.h; sourceTree = "<group>"; };
     1268                2C971CF9278E11C300C9E129 /* pas_lenient_compact_unsigned_ptr.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = pas_lenient_compact_unsigned_ptr.h; sourceTree = "<group>"; };
     1269                2C971CFC278E136300C9E129 /* pas_lenient_compact_unsigned_ptr.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = pas_lenient_compact_unsigned_ptr.c; sourceTree = "<group>"; };
     1270                2C971CFD278E136300C9E129 /* pas_lenient_compact_ptr_inlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = pas_lenient_compact_ptr_inlines.h; sourceTree = "<group>"; };
     1271                2CB9B152278F6C85003A8C1B /* LotsOfHeapsAndThreads.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = LotsOfHeapsAndThreads.cpp; sourceTree = "<group>"; };
    12621272                2CE2AE34275A953E00D02BBC /* BitfitTests.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = BitfitTests.cpp; sourceTree = "<group>"; };
    12631273/* End PBXFileReference section */
    … …  
    13551365                                2CE2AE34275A953E00D02BBC /* BitfitTests.cpp */,
    13561366                                0F53181022C954ED003F7B6A /* BitvectorTests.cpp */,
    1357                                 2B2A589B2742D815005EE07C /* PGMTests.cpp */,
    13581367                                0F31A66723E8B336002C0CA3 /* CartesianTreeTests.cpp */,
    13591368                                0FC682362129D4EB003C6A13 /* CoalignTests.cpp */,
    … …  
    13691378                                0F5B6093235E919900CAE629 /* IsoHeapReservedMemoryTests.cpp */,
    13701379                                0F5193E6266AE5D400483A2C /* JITHeapTests.cpp */,
    1371                                 2B8BCAC62739A47900B1A46F /* PGMTests.cpp */,
    13721380                                0FC64190213745FA0040CE5E /* LargeFreeHeapTests.cpp */,
    13731381                                0FEB6666231C87BC009C001B /* LargeSharingPoolDump.cpp */,
    … …  
    13751383                                0FEB6669231DA86F009C001B /* LargeSharingPoolTests.cpp */,
    13761384                                0FEA45BE236CDAED00B5A375 /* LockFreeReadPtrPtrHashtableTests.cpp */,
     1385                                2CB9B152278F6C85003A8C1B /* LotsOfHeapsAndThreads.cpp */,
    13771386                                0FD22D0722CD8A7500B21841 /* MinHeapTests.cpp */,
     1387                                2B2A589B2742D815005EE07C /* PGMTests.cpp */,
    13781388                                0F5E483923D69F610046DA5C /* RaceTests.cpp */,
    13791389                                0FF08F3522A59DB300386575 /* RedBlackTreeTests.cpp */,
    … …  
    14111421                                0F516EE624561A60004E2B8D /* iso_heap_innards.h */,
    14121422                                0F529DDE2455460C00385A8C /* iso_heap_ref.h */,
    1413                                 2B2A58982742D802005EE07C /* pas_probabilistic_guard_malloc_allocator.c */,
    1414                                 2B2A58972742D802005EE07C /* pas_probabilistic_guard_malloc_allocator.h */,
    14151423                                0FE7EE74229F010F004F4166 /* iso_heap.c */,
    14161424                                0FE7EE75229F010F004F4166 /* iso_heap.h */,
    … …  
    16861694                                0F6D547523C573E000F40DBB /* pas_large_utility_free_heap.c */,
    16871695                                0F6D547223C573E000F40DBB /* pas_large_utility_free_heap.h */,
     1696                                2C971CFD278E136300C9E129 /* pas_lenient_compact_ptr_inlines.h */,
     1697                                2C971CF8278E11C300C9E129 /* pas_lenient_compact_ptr.h */,
     1698                                2C971CFC278E136300C9E129 /* pas_lenient_compact_unsigned_ptr.c */,
     1699                                2C971CF9278E11C300C9E129 /* pas_lenient_compact_unsigned_ptr.h */,
    16881700                                0F8A802725EC588800790B4A /* pas_line_word_config.h */,
    16891701                                0F31A64323E3509E002C0CA3 /* pas_list_direction.h */,
    … …  
    17431755                                0F4F61C925A4CD3B008B4A82 /* pas_payload_reservation_page_list.c */,
    17441756                                0F4F61C825A4CD3B008B4A82 /* pas_payload_reservation_page_list.h */,
    1745                                 2B8BCAC32739A45900B1A46F /* pas_probabilistic_guard_malloc_allocator.c */,
    1746                                 2B8BCAC22739A45900B1A46F /* pas_probabilistic_guard_malloc_allocator.h */,
    17471757                                0F6D547323C573E000F40DBB /* pas_physical_memory_synchronization_style.h */,
    17481758                                0FF248F722FC9EDA0077202E /* pas_physical_memory_transaction.c */,
    … …  
    17501760                                0FDEA578228E23450085E340 /* pas_primitive_heap_ref.c */,
    17511761                                0FDEA577228E23440085E340 /* pas_primitive_heap_ref.h */,
     1762                                2B2A58982742D802005EE07C /* pas_probabilistic_guard_malloc_allocator.c */,
     1763                                2B2A58972742D802005EE07C /* pas_probabilistic_guard_malloc_allocator.h */,
    17521764                                0F9A1CAD2559961300C8D11B /* pas_promote_intrinsic_heap.h */,
    17531765                                0F4F615A259BF6EC008B4A82 /* pas_ptr_hash_map.h */,
    … …  
    20512063                                0FE7EE0422960142004F4166 /* pas_bitvector.h in Headers */,
    20522064                                0FE7EE0522960142004F4166 /* pas_bootstrap_free_heap.h in Headers */,
     2065                                2C971CFB278E11C300C9E129 /* pas_lenient_compact_unsigned_ptr.h in Headers */,
    20532066                                0F5B6084235E572000CAE629 /* pas_bootstrap_heap_page_provider.h in Headers */,
    20542067                                0FE7EE0822960142004F4166 /* pas_cares_about_size_mode.h in Headers */,
    … …  
    21472160                                0FD48B3A23A9ABB30026C46D /* pas_full_alloc_bits_inlines.h in Headers */,
    21482161                                0FF08F3A22A5B40300386575 /* pas_generic_large_free_heap.h in Headers */,
     2162                                2C971CFA278E11C300C9E129 /* pas_lenient_compact_ptr.h in Headers */,
    21492163                                0FE7EE60229E14FA004F4166 /* pas_get_allocation_size.h in Headers */,
    21502164                                0FE7EE64229E14FA004F4166 /* pas_get_heap.h in Headers */,
    … …  
    22542268                                0F19326C22F73E8500FBA713 /* pas_scavenger.h in Headers */,
    22552269                                0F68127222BD4BF40036A02B /* pas_segmented_vector.h in Headers */,
     2270                                2C971CFF278E136300C9E129 /* pas_lenient_compact_ptr_inlines.h in Headers */,
    22562271                                0F148710269B7518006887A9 /* pas_segregated_deallocation_mode.h in Headers */,
    22572272                                0FD48B5623A9ABB30026C46D /* pas_segregated_directory.h in Headers */,
    … …  
    26002615                                0FF248FD230CC4CB0077202E /* IsoHeapPageSharingTests.cpp in Sources */,
    26012616                                0FD48B6723B589910026C46D /* IsoHeapPartialAndBaselineTests.cpp in Sources */,
     2617                                2CB9B153278F6C85003A8C1B /* LotsOfHeapsAndThreads.cpp in Sources */,
    26022618                                0F5B6094235E919900CAE629 /* IsoHeapReservedMemoryTests.cpp in Sources */,
    26032619                                2CE2AE35275A953E00D02BBC /* BitfitTests.cpp in Sources */,
    … …  
    26822698                                0F19325622ED04A600FBA713 /* pas_epoch.c in Sources */,
    26832699                                0FD48B3423A9ABB30026C46D /* pas_exclusive_view_template_memo_table.c in Sources */,
     2700                                2C971CFE278E136300C9E129 /* pas_lenient_compact_unsigned_ptr.c in Sources */,
    26842701                                2C481328273F341A006CAB55 /* pas_expendable_memory.c in Sources */,
    26852702                                0FE7EDBA22960142004F4166 /* pas_extended_gcd.c in Sources */,

  • trunk/Source/bmalloc/libpas/src/libpas/pas_enumerate_segregated_heaps.c

    r287192 r287968  
    11/*
    2  * Copyright (c) 2020-2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2020-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    516516        pas_log("Found allocator = %p\n", allocator);
    517517
    518     full_alloc_bits.bits = pas_compact_tagged_unsigned_ptr_load_remote(enumerator, &view->alloc_bits);
     518    /* This is so weird: the size we pass is only valid when view->alloc_bits is pointing at the
     519       local_allocator's bits. But that's the only time that load_remote will go down the path where it needs
     520       to know the size. So, it's fine, I guess. */
     521    full_alloc_bits.bits = pas_lenient_compact_unsigned_ptr_load_remote(
     522        enumerator, &view->alloc_bits, pas_segregated_page_config_num_alloc_bytes(*page_config));
    519523    full_alloc_bits.word_index_begin = view->alloc_bits_offset;
    520524    full_alloc_bits.word_index_end = view->alloc_bits_offset + view->alloc_bits_size;

  • trunk/Source/bmalloc/libpas/src/libpas/pas_full_alloc_bits_inlines.h

    r282556 r287968  
    11/*
    2  * Copyright (c) 2019-2020 Apple Inc. All rights reserved.
     2 * Copyright (c) 2019-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    5050
    5151static PAS_ALWAYS_INLINE pas_full_alloc_bits
     52pas_full_alloc_bits_create_for_partial_but_not_primordial(pas_segregated_view view)
     53{
     54    pas_segregated_partial_view* partial_view;
     55   
     56    partial_view = pas_segregated_view_get_partial(view);
     57
     58    return pas_full_alloc_bits_create(
     59        pas_lenient_compact_unsigned_ptr_load_compact_non_null(&partial_view->alloc_bits),
     60        partial_view->alloc_bits_offset,
     61        partial_view->alloc_bits_offset + partial_view->alloc_bits_size);
     62}
     63
     64static PAS_ALWAYS_INLINE pas_full_alloc_bits
    5265pas_full_alloc_bits_create_for_partial(pas_segregated_view view)
    5366{
    … …  
    5770
    5871    return pas_full_alloc_bits_create(
    59         pas_compact_tagged_unsigned_ptr_load_non_null(&partial_view->alloc_bits),
     72        pas_lenient_compact_unsigned_ptr_load(&partial_view->alloc_bits),
    6073        partial_view->alloc_bits_offset,
    6174        partial_view->alloc_bits_offset + partial_view->alloc_bits_size);

  • trunk/Source/bmalloc/libpas/src/libpas/pas_local_allocator.c

    r286493 r287968  
    11/*
    2  * Copyright (c) 2018-2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2018-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    119119    pas_segregated_shared_view* shared_view;
    120120
     121    pas_heap_lock_assert_held(); /* Needed to modify a lenient_compact_ptr. */
     122
    121123    directory = pas_segregated_view_get_size_directory(src->view);
    122124   
    … …  
    136138       happens to hold the ownership lock. */
    137139    pas_lock_lock(&shared_view->ownership_lock);
    138     if (pas_compact_tagged_unsigned_ptr_load(&partial_view->alloc_bits) == (unsigned*)src->bits)
    139         pas_compact_tagged_unsigned_ptr_store(&partial_view->alloc_bits, (unsigned*)dst->bits);
     140    if (pas_lenient_compact_unsigned_ptr_load(&partial_view->alloc_bits) == (unsigned*)src->bits)
     141        pas_lenient_compact_unsigned_ptr_store(&partial_view->alloc_bits, (unsigned*)dst->bits);
    140142    pas_lock_unlock(&shared_view->ownership_lock);
    141143}

  • trunk/Source/bmalloc/libpas/src/libpas/pas_local_allocator_inlines.h

    r287192 r287968  
    11/*
    2  * Copyright (c) 2018-2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2018-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    353353    partial_view_as_view = pas_segregated_partial_view_as_view((pas_segregated_partial_view*)view);
    354354   
    355     full_alloc_bits = pas_full_alloc_bits_create_for_partial(partial_view_as_view);
     355    full_alloc_bits = pas_full_alloc_bits_create_for_partial_but_not_primordial(partial_view_as_view);
    356356   
    357357    allocator->view = partial_view_as_view;
    … …  
    600600       
    601601        shared_view = pas_segregated_shared_page_directory_find_first_eligible(
    602             shared_page_directory, size, alignment,
    603             pas_segregated_page_config_heap_lock_hold_mode(page_config));
     602            shared_page_directory, size, alignment, pas_lock_is_not_held);
    604603
    605604        PAS_ASSERT(shared_view);
    606605       
    607         pas_lock_lock_conditionally(
    608             &shared_view->commit_lock,
    609             pas_segregated_page_config_heap_lock_hold_mode(page_config));
     606        pas_lock_lock(&shared_view->commit_lock);
    610607
    611608        handle = pas_segregated_shared_view_commit_page_if_necessary(
    612609            shared_view, heap, shared_page_directory, view, page_config);
    613610        if (!handle) {
    614             pas_lock_unlock_conditionally(
    615                 &shared_view->commit_lock,
    616                 pas_segregated_page_config_heap_lock_hold_mode(page_config));
     611            pas_lock_unlock(&shared_view->commit_lock);
    617612            return false;
    618613        }
    … …  
    664659        pas_zero_memory(allocator->bits, pas_segregated_page_config_num_alloc_bytes(page_config));
    665660
    666         /* Doing this helps heap introspection but isn't otherwise necessary. */
    667         pas_compact_tagged_unsigned_ptr_store(&view->alloc_bits, (unsigned*)allocator->bits);
    668         PAS_ASSERT(!view->alloc_bits_offset);
    669         PAS_ASSERT((uint8_t)pas_segregated_page_config_num_alloc_words(page_config)
    670                    == pas_segregated_page_config_num_alloc_words(page_config));
    671         view->alloc_bits_size = (uint8_t)pas_segregated_page_config_num_alloc_words(page_config);
    672 
    673661        pas_local_allocator_set_up_primordial_bump(
    674662            allocator, view, handle, page, &held_lock, bump_result,
    … …  
    680668        view->is_attached_to_shared_handle = true;
    681669
     670        pas_lock_unlock(&shared_view->commit_lock);
     671
     672        /* This code to set the view's alloc_bits is primarily for heap enumeration.
     673           
     674           If the enumerator ran right now then (before we do this stuff), then it would see:
     675       
     676           - A bunch of page alloc bits set for the objects we are about to allocate.
     677           - No partial views say that they own those objects.
     678           - Local allocator claims to own those objects.
     679       
     680           So, we would report that those objects are not live right now. I believe they would show up
     681           as "meta" allocations, not even as objects.
     682       
     683           Immediately after settings the view's alloc_bits and alloc_bits_size, we will know that the
     684           objects exist (view alloc bits are set), but they are dead (despite page alloc bits also being
     685           set, the objects are part of the bump range). */
     686        if (!pas_heap_lock_try_lock()) {
     687            pas_lock_switch(&held_lock, NULL);
     688            pas_heap_lock_lock();
     689            pas_segregated_page_switch_lock(page, &held_lock, page_config);
     690        }
     691
     692        pas_lenient_compact_unsigned_ptr_store(&view->alloc_bits, (unsigned*)allocator->bits);
     693        PAS_ASSERT(!view->alloc_bits_offset);
     694        PAS_ASSERT((uint8_t)pas_segregated_page_config_num_alloc_words(page_config)
     695                   == pas_segregated_page_config_num_alloc_words(page_config));
     696        pas_compiler_fence();
     697        view->alloc_bits_size = (uint8_t)pas_segregated_page_config_num_alloc_words(page_config);
    682698        pas_lock_switch(&held_lock, NULL);
    683         pas_lock_unlock_conditionally(
    684             &shared_view->commit_lock,
    685             pas_segregated_page_config_heap_lock_hold_mode(page_config));
    686        
     699        pas_heap_lock_unlock();
     700
    687701        return true;
    688702    }
    … …  
    776790    PAS_ASSERT((uint8_t)alloc_bits_offset == alloc_bits_offset);
    777791    view->alloc_bits_offset = (uint8_t)alloc_bits_offset;
     792   
     793    /* We hold the page lock. Lock ordering says that we cannot acquire the heap lock when
     794       we are holding the page lock.
     795       
     796       Luckily, this is a fine place to drop the page lock.
     797       
     798       Therefore, we try-lock the heap lock. It's always safe to do that. Usually it will just
     799       succeed. But if it fails, we will drop the page lock and then acquire both of them. */
     800    if (!pas_heap_lock_try_lock_conditionally(heap_lock_hold_mode)) {
     801        pas_segregated_page_unlock(page, page_config);
     802        pas_heap_lock_lock();
     803        pas_segregated_page_lock(page, page_config);
     804    }
    778805
    779806    if (alloc_bits_size == 1)
    780807        alloc_bits = &view->inline_alloc_bits - alloc_bits_offset;
    781808    else {
    782         /* We hold the page lock. Lock ordering says that we cannot acquire the heap lock when
    783            we are holding the page lock.
    784            
    785            Luckily, this is a fine place to drop the page lock.
    786            
    787            Therefore, we try-lock the heap lock. It's always safe to do that. Usually it will just
    788            succeed. But if it fails, we will drop the page lock and then acquire both of them. */
    789         if (!pas_heap_lock_try_lock_conditionally(heap_lock_hold_mode)) {
    790             pas_segregated_page_unlock(page, page_config);
    791             pas_heap_lock_lock();
    792             pas_segregated_page_lock(page, page_config);
    793         }
    794 
    795809        alloc_bits = (unsigned*)pas_immortal_heap_allocate_with_manual_alignment(
    796810            alloc_bits_size * sizeof(unsigned),
    … …  
    798812            "pas_segregated_partial_view/alloc_bits",
    799813            pas_object_allocation) - alloc_bits_offset;
    800 
    801         pas_heap_lock_unlock_conditionally(heap_lock_hold_mode);
    802814    }
    803815   
    … …  
    808820    pas_store_store_fence();
    809821
    810     pas_compact_tagged_unsigned_ptr_store(&view->alloc_bits, alloc_bits);
     822    pas_lenient_compact_unsigned_ptr_store(&view->alloc_bits, alloc_bits);
     823   
     824    pas_heap_lock_unlock_conditionally(heap_lock_hold_mode);
    811825}
    812826

  • trunk/Source/bmalloc/libpas/src/libpas/pas_segregated_partial_view.c

    r285853 r287968  
    11/*
    2  * Copyright (c) 2019-2020 Apple Inc. All rights reserved.
     2 * Copyright (c) 2019-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    6565    result->alloc_bits_offset = 0;
    6666    result->alloc_bits_size = 0;
    67     pas_compact_tagged_unsigned_ptr_store(&result->alloc_bits, NULL);
     67    pas_lenient_compact_unsigned_ptr_store(&result->alloc_bits, NULL);
    6868
    6969    result->inline_alloc_bits = 0; /* There isn't a real big need to do this, but it helps keep
    … …  
    187187    shared_view = pas_compact_segregated_shared_view_ptr_load_non_null(&view->shared_view);
    188188
    189     full_alloc_bits = pas_compact_tagged_unsigned_ptr_load_non_null(&view->alloc_bits);
     189    full_alloc_bits = pas_lenient_compact_unsigned_ptr_load(&view->alloc_bits);
    190190
    191191    if (shared_view->is_owned) {

  • trunk/Source/bmalloc/libpas/src/libpas/pas_segregated_partial_view.h

    r282556 r287968  
    11/*
    2  * Copyright (c) 2019-2020 Apple Inc. All rights reserved.
     2 * Copyright (c) 2019-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2929#include "pas_compact_segregated_shared_view_ptr.h"
    3030#include "pas_compact_segregated_size_directory_ptr.h"
    31 #include "pas_compact_tagged_unsigned_ptr.h"
     31#include "pas_lenient_compact_unsigned_ptr.h"
     32#include "pas_page_granule_use_count.h"
    3233#include "pas_segregated_page_config.h"
    33 #include "pas_page_granule_use_count.h"
    3434#include "pas_segregated_view.h"
    3535#include "pas_utils.h"
    … …  
    4949    pas_compact_segregated_shared_view_ptr shared_view;
    5050    pas_compact_segregated_size_directory_ptr directory;
    51     pas_compact_tagged_unsigned_ptr alloc_bits;
     51    pas_lenient_compact_unsigned_ptr alloc_bits;
    5252
    5353    /* The index can be small since we would never create a high-indexed partial view. That would not

  • trunk/Source/bmalloc/libpas/src/libpas/pas_segregated_view_allocator_inlines.h

    r287192 r287968  
    11/*
    2  * Copyright (c) 2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2021-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    263263            unsigned word_index;
    264264       
    265             full_alloc_bits = pas_compact_tagged_unsigned_ptr_load_non_null(&partial->alloc_bits);
     265            full_alloc_bits = pas_lenient_compact_unsigned_ptr_load_compact_non_null(&partial->alloc_bits);
    266266       
    267267            begin_word_index = partial->alloc_bits_offset;

  • trunk/Source/bmalloc/libpas/src/libpas/pas_thread_local_cache.c

    r286587 r287968  
    3131
    3232#include "pas_all_heap_configs.h"
    33 #include "pas_compact_large_utility_free_heap.h"
    3433#include "pas_debug_heap.h"
    3534#include "pas_heap_lock.h"
     35#include "pas_large_utility_free_heap.h"
    3636#include "pas_log.h"
    3737#include "pas_monotonic_time.h"
    … …  
    6464    char* begin;
    6565
    66     pas_compact_large_utility_free_heap_deallocate(
     66    pas_large_utility_free_heap_deallocate(
    6767        thread_local_cache->should_stop_bitvector,
    6868        PAS_BITVECTOR_NUM_BYTES(thread_local_cache->allocator_index_capacity));
    6969   
    7070    begin = (char*)thread_local_cache;
    71     pas_compact_large_utility_free_heap_deallocate(
     71    pas_large_utility_free_heap_deallocate(
    7272        begin,
    7373        pas_thread_local_cache_size_for_allocator_index_capacity(
    … …  
    130130        printf("Cache size: %zu\n", size);
    131131   
    132     result = (pas_thread_local_cache*)pas_compact_large_utility_free_heap_allocate(size, "pas_thread_local_cache");
     132    result = (pas_thread_local_cache*)pas_large_utility_free_heap_allocate(size, "pas_thread_local_cache");
    133133
    134134    pas_zero_memory(result, size);
    135135
    136     result->should_stop_bitvector = (unsigned int*)pas_compact_large_utility_free_heap_allocate(
     136    result->should_stop_bitvector = (unsigned int*)pas_large_utility_free_heap_allocate(
    137137        PAS_BITVECTOR_NUM_BYTES(allocator_index_capacity),
    138138        "pas_thread_local_cache/should_stop_bitvector");

  • trunk/Source/bmalloc/libpas/src/test/TestHarness.cpp

    r287192 r287968  
    11/*
    2  * Copyright (c) 2018-2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2018-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    354354void addLargeSharingPoolTests();
    355355void addLockFreeReadPtrPtrHashtableTests();
     356void addLotsOfHeapsAndThreadsTests();
    356357void addMinHeapTests();
    357358void addPGMTests();
    … …  
    718719    ADD_SUITE(LargeSharingPool);
    719720    ADD_SUITE(LockFreeReadPtrPtrHashtable);
     721    ADD_SUITE(LotsOfHeapsAndThreads);
    720722    ADD_SUITE(MinHeap);
    721723    ADD_SUITE(PGM);
Note: See TracChangeset for help on using the changeset viewer.