Changeset 289590 in webkit


Ignore:
Timestamp:
Feb 10, 2022, 3:49:33 PM (4 years ago)
Author:
fpizlo@apple.com
Message:

[libpas] jit_heap should support the segregated heap
​https://bugs.webkit.org/show_bug.cgi?id=235497

Reviewed by Yusuke Suzuki.

Source/bmalloc:

One of the things that libpas provides is a fast implementation of executable memory allocation, which
libpas calls "jit_heap". The initial implementation of this only used libpas's bitfit and large
algorithms. Bitfit was used for smaller objects and large was used for larger objects. The libpas
segregated heap was disabled in jit_heap. This made sense because:

  • Bitfit and large support the shrink() call, where we shrink an allocation in-place. Executable allocation supports this to aid branch compaction (we compact code after we copy it into its final resting place, which is after we allocate the memory - so after we finish compaction, we can only shrink the allocation in-place or not at all).
  • Segregated heaps have some virtual address space overheads, and memory overheads, that the bitfit and large heaps don't have. This happens because segreated heaps will dedicate each page to objects of exactly one size forever.

However, segregated heaps are substantially faster than bitfit heaps. They are faster under serial
allocation thanks to the fact that no locks need to be acquired for a typical allocation or
deallocation. They are even more faster under parallel allocation thanks to thread-local caching.

This patch enables segregated heaps for the smallest jit_heap allocations (<=2000 bytes). The cutoff
threshold is runtime configurable, so it's possible to disable segregated heaps at runtime by setting
the cutoff to 0.

With the 2000 threshold, this appears to be a 0.3% Speedometer2 speed-up.

While there is a theoretical possibility of memory overhead, I haven't seen it. If we are using jump
islands then the virtual memory overhead of segregated heaps will just mean that we use jump islands
more frequently, which would manifest as a performance regression -- but I've not seen any such
regression. So, this disables the segregated heap if the JIT pool size is too small (see changes to
ExecutableAllocator.cpp in JSC).

  • libpas/libpas.xcodeproj/project.pbxproj:
  • libpas/src/libpas/jit_heap.c:

(jit_heap_shrink):

  • libpas/src/libpas/jit_heap_config.c:

(jit_small_segregated_allocate_page):
(jit_small_segregated_create_page_header):
(jit_small_destroy_page_header):
(jit_small_segregated_shared_page_directory_selector):
(jit_small_bitfit_allocate_page):
(jit_small_bitfit_create_page_header):
(jit_medium_bitfit_create_page_header):
(jit_medium_destroy_page_header):
(jit_prepare_to_enumerate):
(jit_small_bitfit_destroy_page_header): Deleted.
(jit_medium_bitfit_destroy_page_header): Deleted.

  • libpas/src/libpas/jit_heap_config.h:

(jit_small_page_header_for_boundary):
(jit_small_boundary_for_page_header):
(jit_medium_page_header_for_boundary):
(jit_medium_boundary_for_page_header):
(jit_heap_config_page_header):
(jit_small_bitfit_page_header_for_boundary): Deleted.
(jit_small_bitfit_boundary_for_page_header): Deleted.
(jit_medium_bitfit_page_header_for_boundary): Deleted.
(jit_medium_bitfit_boundary_for_page_header): Deleted.

  • libpas/src/libpas/jit_heap_config_root_data.h:
  • libpas/src/libpas/pas_bitfit_heap.c:

(pas_bitfit_heap_select_variant):
(pas_bitfit_heap_construct_and_insert_size_class):

  • libpas/src/libpas/pas_bitfit_heap.h:
  • libpas/src/libpas/pas_bitfit_page_config.h:

(pas_bitfit_page_config_is_enabled):

  • libpas/src/libpas/pas_heap_config.h:

(pas_heap_config_segregated_heap_min_align_shift):
(pas_heap_config_segregated_heap_min_align):

  • libpas/src/libpas/pas_segregated_heap.c:

(min_align_for_heap):
(min_object_size_for_heap):
(max_segregated_object_size_for_heap):
(max_bitfit_object_size_for_heap):
(max_object_size_for_heap):
(pas_segregated_heap_ensure_allocator_index):
(pas_segregated_heap_ensure_size_directory_for_size):
(min_object_size_for_heap_config): Deleted.
(max_segregated_object_size_for_heap_config): Deleted.
(max_bitfit_object_size_for_heap_config): Deleted.
(max_object_size_for_heap_config): Deleted.

  • libpas/src/libpas/pas_segregated_heap.h:

(pas_segregated_heap_index_for_size):
(pas_segregated_heap_size_for_index):

  • libpas/src/libpas/pas_segregated_page.c:

(pas_segregated_page_construct):

  • libpas/src/libpas/pas_segregated_page_config.h:

(pas_segregated_page_config_is_enabled):

  • libpas/src/libpas/pas_segregated_page_config_kind.def:
  • libpas/src/libpas/pas_segregated_page_config_kind_and_role.h:
  • libpas/src/libpas/pas_segregated_size_directory.c:

(pas_segregated_size_directory_create):

  • libpas/src/test/IsoHeapChaosTests.cpp:

(std::addAllTests):

  • libpas/src/test/JITHeapTests.cpp:

(std::testAllocationSize):
(addJITHeapTests):

  • libpas/src/test/TestHarness.cpp:

(main):

  • libpas/src/test/ViewCacheTests.cpp: Added.

(std::setupConfig):
(std::testDisableViewCacheUsingBoundForNoViewCache):
(std::testEnableViewCacheAtSomeBoundForNoViewCache):
(addViewCacheTests):

Source/JavaScriptCore:

Disable the jit_heap segregated heap if the executable pool size is too small.

  • jit/ExecutableAllocator.cpp:

(JSC::initializeJITPageReservation):

Location:
trunk/Source
Files:
1 added
24 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/JavaScriptCore/ChangeLog

    r289515 r289590  
     12022-01-24  Filip Pizlo  <fpizlo@apple.com>
     2
     3        [libpas] jit_heap should support the segregated heap
     4        https://bugs.webkit.org/show_bug.cgi?id=235497
     5
     6        Reviewed by Yusuke Suzuki.
     7
     8        Disable the jit_heap segregated heap if the executable pool size is too small.
     9
     10        * jit/ExecutableAllocator.cpp:
     11        (JSC::initializeJITPageReservation):
     12
    1132022-02-09  Yusuke Suzuki  <ysuzuki@apple.com>
    214

  • trunk/Source/JavaScriptCore/jit/ExecutableAllocator.cpp

    r288870 r289590  
    11/*
    2  * Copyright (C) 2008-2021 Apple Inc. All rights reserved.
     2 * Copyright (C) 2008-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    4444#if USE(LIBPAS_JIT_HEAP)
    4545#include <bmalloc/jit_heap.h>
     46#include <bmalloc/jit_heap_config.h>
    4647#else
    4748#include <wtf/MetaAllocator.h>
    … …  
    8586using namespace WTF;
    8687
     88#if USE(LIBPAS_JIT_HEAP)
     89static constexpr size_t minimumPoolSizeForSegregatedHeap = 256 * MB;
     90#endif
     91
    8792#if defined(FIXED_EXECUTABLE_MEMORY_POOL_SIZE_IN_MB) && FIXED_EXECUTABLE_MEMORY_POOL_SIZE_IN_MB > 0
    8893static constexpr size_t fixedExecutableMemoryPoolSize = FIXED_EXECUTABLE_MEMORY_POOL_SIZE_IN_MB * MB;
    … …  
    354359    }
    355360    reservation.size = std::max(roundUpToMultipleOf(pageSize(), reservation.size), pageSize() * 2);
     361
     362#if USE(LIBPAS_JIT_HEAP)
     363    if (reservation.size < minimumPoolSizeForSegregatedHeap)
     364        jit_heap_runtime_config.max_segregated_object_size = 0;
     365#endif
    356366
    357367    auto tryCreatePageReservation = [] (size_t reservationSize) {
    … …  
    771781            RELEASE_ASSERT(!m_end);
    772782            m_start = reinterpret_cast<uintptr_t>(start);
    773             m_end = m_start + sizeInBytes; 
     783            m_end = m_start + sizeInBytes;
    774784            jit_heap_add_fresh_memory(pas_range_create(m_start, m_end));
    775785        }

  • trunk/Source/JavaScriptCore/jit/ExecutableMemoryHandle.h

    r288599 r289590  
    2929#include <wtf/MetaAllocatorPtr.h>
    3030#include <wtf/ThreadSafeRefCounted.h>
    31 #include <bmalloc/jit_heap.h>
    3231#else
    3332#include <wtf/MetaAllocatorHandle.h>

  • trunk/Source/bmalloc/ChangeLog

    r289579 r289590  
     12022-01-23  Filip Pizlo  <fpizlo@apple.com>
     2
     3        [libpas] jit_heap should support the segregated heap
     4        https://bugs.webkit.org/show_bug.cgi?id=235497
     5
     6        Reviewed by Yusuke Suzuki.
     7
     8        One of the things that libpas provides is a fast implementation of executable memory allocation, which
     9        libpas calls "jit_heap". The initial implementation of this only used libpas's bitfit and large
     10        algorithms. Bitfit was used for smaller objects and large was used for larger objects. The libpas
     11        segregated heap was disabled in jit_heap. This made sense because:
     12
     13        - Bitfit and large support the shrink() call, where we shrink an allocation in-place. Executable
     14          allocation supports this to aid branch compaction (we compact code after we copy it into its final
     15          resting place, which is after we allocate the memory - so after we finish compaction, we can only
     16          shrink the allocation in-place or not at all).
     17
     18        - Segregated heaps have some virtual address space overheads, and memory overheads, that the bitfit
     19          and large heaps don't have. This happens because segreated heaps will dedicate each page to objects
     20          of exactly one size forever.
     21
     22        However, segregated heaps are substantially faster than bitfit heaps. They are faster under serial
     23        allocation thanks to the fact that no locks need to be acquired for a typical allocation or
     24        deallocation. They are even more faster under parallel allocation thanks to thread-local caching.
     25
     26        This patch enables segregated heaps for the smallest jit_heap allocations (<=2000 bytes). The cutoff
     27        threshold is runtime configurable, so it's possible to disable segregated heaps at runtime by setting
     28        the cutoff to 0.
     29
     30        With the 2000 threshold, this appears to be a 0.3% Speedometer2 speed-up.
     31
     32        While there is a theoretical possibility of memory overhead, I haven't seen it. If we are using jump
     33        islands then the virtual memory overhead of segregated heaps will just mean that we use jump islands
     34        more frequently, which would manifest as a performance regression -- but I've not seen any such
     35        regression. So, this disables the segregated heap if the JIT pool size is too small (see changes to
     36        ExecutableAllocator.cpp in JSC).
     37
     38        * libpas/libpas.xcodeproj/project.pbxproj:
     39        * libpas/src/libpas/jit_heap.c:
     40        (jit_heap_shrink):
     41        * libpas/src/libpas/jit_heap_config.c:
     42        (jit_small_segregated_allocate_page):
     43        (jit_small_segregated_create_page_header):
     44        (jit_small_destroy_page_header):
     45        (jit_small_segregated_shared_page_directory_selector):
     46        (jit_small_bitfit_allocate_page):
     47        (jit_small_bitfit_create_page_header):
     48        (jit_medium_bitfit_create_page_header):
     49        (jit_medium_destroy_page_header):
     50        (jit_prepare_to_enumerate):
     51        (jit_small_bitfit_destroy_page_header): Deleted.
     52        (jit_medium_bitfit_destroy_page_header): Deleted.
     53        * libpas/src/libpas/jit_heap_config.h:
     54        (jit_small_page_header_for_boundary):
     55        (jit_small_boundary_for_page_header):
     56        (jit_medium_page_header_for_boundary):
     57        (jit_medium_boundary_for_page_header):
     58        (jit_heap_config_page_header):
     59        (jit_small_bitfit_page_header_for_boundary): Deleted.
     60        (jit_small_bitfit_boundary_for_page_header): Deleted.
     61        (jit_medium_bitfit_page_header_for_boundary): Deleted.
     62        (jit_medium_bitfit_boundary_for_page_header): Deleted.
     63        * libpas/src/libpas/jit_heap_config_root_data.h:
     64        * libpas/src/libpas/pas_bitfit_heap.c:
     65        (pas_bitfit_heap_select_variant):
     66        (pas_bitfit_heap_construct_and_insert_size_class):
     67        * libpas/src/libpas/pas_bitfit_heap.h:
     68        * libpas/src/libpas/pas_bitfit_page_config.h:
     69        (pas_bitfit_page_config_is_enabled):
     70        * libpas/src/libpas/pas_heap_config.h:
     71        (pas_heap_config_segregated_heap_min_align_shift):
     72        (pas_heap_config_segregated_heap_min_align):
     73        * libpas/src/libpas/pas_segregated_heap.c:
     74        (min_align_for_heap):
     75        (min_object_size_for_heap):
     76        (max_segregated_object_size_for_heap):
     77        (max_bitfit_object_size_for_heap):
     78        (max_object_size_for_heap):
     79        (pas_segregated_heap_ensure_allocator_index):
     80        (pas_segregated_heap_ensure_size_directory_for_size):
     81        (min_object_size_for_heap_config): Deleted.
     82        (max_segregated_object_size_for_heap_config): Deleted.
     83        (max_bitfit_object_size_for_heap_config): Deleted.
     84        (max_object_size_for_heap_config): Deleted.
     85        * libpas/src/libpas/pas_segregated_heap.h:
     86        (pas_segregated_heap_index_for_size):
     87        (pas_segregated_heap_size_for_index):
     88        * libpas/src/libpas/pas_segregated_page.c:
     89        (pas_segregated_page_construct):
     90        * libpas/src/libpas/pas_segregated_page_config.h:
     91        (pas_segregated_page_config_is_enabled):
     92        * libpas/src/libpas/pas_segregated_page_config_kind.def:
     93        * libpas/src/libpas/pas_segregated_page_config_kind_and_role.h:
     94        * libpas/src/libpas/pas_segregated_size_directory.c:
     95        (pas_segregated_size_directory_create):
     96        * libpas/src/test/IsoHeapChaosTests.cpp:
     97        (std::addAllTests):
     98        * libpas/src/test/JITHeapTests.cpp:
     99        (std::testAllocationSize):
     100        (addJITHeapTests):
     101        * libpas/src/test/TestHarness.cpp:
     102        (main):
     103        * libpas/src/test/ViewCacheTests.cpp: Added.
     104        (std::setupConfig):
     105        (std::testDisableViewCacheUsingBoundForNoViewCache):
     106        (std::testEnableViewCacheAtSomeBoundForNoViewCache):
     107        (addViewCacheTests):
     108
    11092022-01-18  Filip Pizlo  <fpizlo@apple.com>
    2110

  • trunk/Source/bmalloc/bmalloc.xcodeproj/project.pbxproj

    r289579 r289590  
    3939                0F516EE52456184F004E2B8D /* pas_internal_config.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F516EE42456184F004E2B8D /* pas_internal_config.h */; settings = {ATTRIBUTES = (Private, ); }; };
    4040                0F516EE924561AE9004E2B8D /* iso_heap_innards.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F516EE824561AE8004E2B8D /* iso_heap_innards.h */; settings = {ATTRIBUTES = (Private, ); }; };
    41                 0F5193EE266C42AD00483A2C /* jit_heap_config.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F5193E8266C42AC00483A2C /* jit_heap_config.h */; };
     41                0F5193EE266C42AD00483A2C /* jit_heap_config.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F5193E8266C42AC00483A2C /* jit_heap_config.h */; settings = {ATTRIBUTES = (Private, ); }; };
    4242                0F5193EF266C42AD00483A2C /* jit_heap.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F5193E9266C42AC00483A2C /* jit_heap.h */; settings = {ATTRIBUTES = (Private, ); }; };
    43                 0F5193F0266C42AD00483A2C /* jit_heap_config_root_data.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F5193EA266C42AC00483A2C /* jit_heap_config_root_data.h */; };
     43                0F5193F0266C42AD00483A2C /* jit_heap_config_root_data.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F5193EA266C42AC00483A2C /* jit_heap_config_root_data.h */; settings = {ATTRIBUTES = (Private, ); }; };
    4444                0F5193F1266C42AD00483A2C /* jit_heap_config.c in Sources */ = {isa = PBXBuildFile; fileRef = 0F5193EB266C42AC00483A2C /* jit_heap_config.c */; };
    4545                0F5193F2266C42AD00483A2C /* jit_heap.c in Sources */ = {isa = PBXBuildFile; fileRef = 0F5193EC266C42AC00483A2C /* jit_heap.c */; };

  • trunk/Source/bmalloc/libpas/libpas.xcodeproj/project.pbxproj

    r289579 r289590  
    555555                2C11E88F2728A783002162D0 /* pas_simple_type.c in Sources */ = {isa = PBXBuildFile; fileRef = 2C11E88D2728A783002162D0 /* pas_simple_type.c */; };
    556556                2C11E8912728A893002162D0 /* bmalloc_type.c in Sources */ = {isa = PBXBuildFile; fileRef = 2C11E8902728A893002162D0 /* bmalloc_type.c */; };
     557                2C16DAE0279E07290042E919 /* ViewCacheTests.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 2C16DADF279E07290042E919 /* ViewCacheTests.cpp */; };
    557558                2C340000275815D0005565CB /* pas_segregated_page_config_kind_and_role.c in Sources */ = {isa = PBXBuildFile; fileRef = 2C34FFFF275815D0005565CB /* pas_segregated_page_config_kind_and_role.c */; };
    558559                2C34000227581687005565CB /* pas_page_base_config.c in Sources */ = {isa = PBXBuildFile; fileRef = 2C34000127581687005565CB /* pas_page_base_config.c */; };
    … …  
    12561257                2C11E88D2728A783002162D0 /* pas_simple_type.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = pas_simple_type.c; sourceTree = "<group>"; };
    12571258                2C11E8902728A893002162D0 /* bmalloc_type.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = bmalloc_type.c; sourceTree = "<group>"; };
     1259                2C16DADF279E07290042E919 /* ViewCacheTests.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ViewCacheTests.cpp; sourceTree = "<group>"; };
    12581260                2C34000127581687005565CB /* pas_page_base_config.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = pas_page_base_config.c; sourceTree = "<group>"; };
    12591261                2C34FFF527571D2F005565CB /* pas_page_base_and_kind.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = pas_page_base_and_kind.h; sourceTree = "<group>"; };
    … …  
    14171419                                0F8700DE25B14C68000E1ABF /* TSDTests.cpp */,
    14181420                                0F53181222C9609D003F7B6A /* UtilsTests.cpp */,
     1421                                2C16DADF279E07290042E919 /* ViewCacheTests.cpp */,
    14191422                        );
    14201423                        path = test;
    … …  
    26442647                                0F53181122C954ED003F7B6A /* BitvectorTests.cpp in Sources */,
    26452648                                0F31A66823E8B336002C0CA3 /* CartesianTreeTests.cpp in Sources */,
     2649                                2C16DAE0279E07290042E919 /* ViewCacheTests.cpp in Sources */,
    26462650                                0FC682382129D4EE003C6A13 /* CoalignTests.cpp in Sources */,
    26472651                                2C48132D273F4159006CAB55 /* ExpendableMemoryTests.cpp in Sources */,

  • trunk/Source/bmalloc/libpas/src/libpas/jit_heap.c

    r285853 r289590  
    11/*
    2  * Copyright (c) 2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2021-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    8787void jit_heap_shrink(void* object, size_t new_size)
    8888{
    89     bool result;
    90     result = pas_try_shrink(object, new_size, JIT_HEAP_CONFIG);
    91     PAS_ASSERT(result);
     89    /* NOTE: the shrink call will fail (return false) for segregated allocations, and that's fine because we
     90       only use segregated allocations for smaller sizes (so the amount of potential memory savings from
     91       shrinking is small). */
     92    pas_try_shrink(object, new_size, JIT_HEAP_CONFIG);
    9293}
    9394

  • trunk/Source/bmalloc/libpas/src/libpas/jit_heap_config.c

    r289579 r289590  
    11/*
    2  * Copyright (c) 2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2021-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3838#include "pas_heap_config_inlines.h"
    3939#include "pas_root.h"
     40#include "pas_segregated_page_config_inlines.h"
    4041#include "pas_stream.h"
    4142
    … …  
    103104};
    104105
    105 pas_page_header_table jit_small_bitfit_page_header_table =
     106pas_page_header_table jit_small_page_header_table =
    106107    PAS_PAGE_HEADER_TABLE_INITIALIZER(JIT_SMALL_PAGE_SIZE);
    107 pas_page_header_table jit_medium_bitfit_page_header_table =
     108pas_page_header_table jit_medium_page_header_table =
    108109    PAS_PAGE_HEADER_TABLE_INITIALIZER(JIT_MEDIUM_PAGE_SIZE);
    109110
    … …  
    115116    .directory_size_bound_for_partial_views = 0,
    116117    .directory_size_bound_for_baseline_allocators = 0,
    117     .directory_size_bound_for_no_view_cache = UINT_MAX,
    118     .max_segregated_object_size = 0,
     118    .directory_size_bound_for_no_view_cache = 0,
     119    .max_segregated_object_size = 2000,
    119120    .max_bitfit_object_size = UINT_MAX,
    120     .view_cache_capacity_for_object_size = pas_heap_runtime_config_zero_view_cache_capacity
     121    .view_cache_capacity_for_object_size = pas_heap_runtime_config_aggressive_view_cache_capacity
    121122};
    122123
    123124jit_heap_config_root_data jit_root_data = {
    124     .small_bitfit_page_header_table = &jit_small_bitfit_page_header_table,
    125     .medium_bitfit_page_header_table = &jit_medium_bitfit_page_header_table
     125    .small_page_header_table = &jit_small_page_header_table,
     126    .medium_page_header_table = &jit_medium_page_header_table
    126127};
    127128
    … …  
    143144}
    144145
    145 void* jit_small_bitfit_allocate_page(
    146     pas_segregated_heap* heap, pas_physical_memory_transaction* transaction)
    147 {
     146void* jit_small_segregated_allocate_page(
     147    pas_segregated_heap* heap, pas_physical_memory_transaction* transaction, pas_segregated_page_role role)
     148{
     149    PAS_ASSERT(role == pas_segregated_page_exclusive_role);
    148150    PAS_UNUSED_PARAM(heap);
    149151    PAS_UNUSED_PARAM(transaction);
    … …  
    152154}
    153155
     156pas_page_base* jit_small_segregated_create_page_header(
     157    void* boundary, pas_page_kind kind, pas_lock_hold_mode heap_lock_hold_mode)
     158{
     159    pas_page_base* result;
     160    PAS_ASSERT(kind == pas_small_exclusive_segregated_page_kind);
     161    pas_heap_lock_lock_conditionally(heap_lock_hold_mode);
     162    result = pas_page_header_table_add(
     163        &jit_small_page_header_table,
     164        JIT_SMALL_PAGE_SIZE,
     165        pas_segregated_page_header_size(JIT_HEAP_CONFIG.small_segregated_config,
     166                                        pas_segregated_page_exclusive_role),
     167        boundary);
     168    pas_heap_lock_unlock_conditionally(heap_lock_hold_mode);
     169    return result;
     170}
     171
     172void jit_small_destroy_page_header(pas_page_base* page, pas_lock_hold_mode heap_lock_hold_mode)
     173{
     174    pas_heap_lock_lock_conditionally(heap_lock_hold_mode);
     175    pas_page_header_table_remove(&jit_small_page_header_table,
     176                                 JIT_SMALL_PAGE_SIZE,
     177                                 page);
     178    pas_heap_lock_unlock_conditionally(heap_lock_hold_mode);
     179}
     180
     181pas_segregated_shared_page_directory* jit_small_segregated_shared_page_directory_selector(
     182    pas_segregated_heap* heap, pas_segregated_size_directory* directory)
     183{
     184    PAS_UNUSED_PARAM(heap);
     185    PAS_UNUSED_PARAM(directory);
     186    PAS_ASSERT(!"Not implemented");
     187    return NULL;
     188}
     189
     190void* jit_small_bitfit_allocate_page(
     191    pas_segregated_heap* heap, pas_physical_memory_transaction* transaction)
     192{
     193    PAS_UNUSED_PARAM(heap);
     194    PAS_UNUSED_PARAM(transaction);
     195    return (void*)allocate_from_fresh(
     196        JIT_SMALL_PAGE_SIZE, pas_alignment_create_traditional(JIT_SMALL_PAGE_SIZE)).begin;
     197}
     198
    154199pas_page_base* jit_small_bitfit_create_page_header(
    155200    void* boundary, pas_page_kind kind, pas_lock_hold_mode heap_lock_hold_mode)
    … …  
    158203    PAS_ASSERT(kind == pas_small_bitfit_page_kind);
    159204    pas_heap_lock_lock_conditionally(heap_lock_hold_mode);
    160     result = pas_page_header_table_add(&jit_small_bitfit_page_header_table,
     205    result = pas_page_header_table_add(&jit_small_page_header_table,
    161206                                       JIT_SMALL_PAGE_SIZE,
    162207                                       pas_bitfit_page_header_size(JIT_HEAP_CONFIG.small_bitfit_config),
    … …  
    166211}
    167212
    168 void jit_small_bitfit_destroy_page_header(
    169     pas_page_base* page, pas_lock_hold_mode heap_lock_hold_mode)
    170 {
    171     pas_heap_lock_lock_conditionally(heap_lock_hold_mode);
    172     pas_page_header_table_remove(&jit_small_bitfit_page_header_table,
    173                                  JIT_SMALL_PAGE_SIZE,
    174                                  page);
    175     pas_heap_lock_unlock_conditionally(heap_lock_hold_mode);
    176 }
    177 
    178213void* jit_medium_bitfit_allocate_page(
    179214    pas_segregated_heap* heap, pas_physical_memory_transaction* transaction)
    … …  
    191226    PAS_ASSERT(kind == pas_medium_bitfit_page_kind);
    192227    pas_heap_lock_lock_conditionally(heap_lock_hold_mode);
    193     result = pas_page_header_table_add(&jit_medium_bitfit_page_header_table,
     228    result = pas_page_header_table_add(&jit_medium_page_header_table,
    194229                                       JIT_MEDIUM_PAGE_SIZE,
    195230                                       pas_bitfit_page_header_size(JIT_HEAP_CONFIG.medium_bitfit_config),
    … …  
    199234}
    200235
    201 void jit_medium_bitfit_destroy_page_header(
     236void jit_medium_destroy_page_header(
    202237    pas_page_base* page, pas_lock_hold_mode heap_lock_hold_mode)
    203238{
    204239    pas_heap_lock_lock_conditionally(heap_lock_hold_mode);
    205     pas_page_header_table_remove(&jit_medium_bitfit_page_header_table,
     240    pas_page_header_table_remove(&jit_medium_page_header_table,
    206241                                 JIT_MEDIUM_PAGE_SIZE,
    207242                                 page);
    … …  
    270305            enumerator,
    271306            (pas_page_header_table*)pas_enumerator_read(
    272                 enumerator, root_data->small_bitfit_page_header_table, sizeof(pas_page_header_table))))
     307                enumerator, root_data->small_page_header_table, sizeof(pas_page_header_table))))
    273308        return NULL;
    274309   
    … …  
    277312            enumerator,
    278313            (pas_page_header_table*)pas_enumerator_read(
    279                 enumerator, root_data->medium_bitfit_page_header_table, sizeof(pas_page_header_table))))
     314                enumerator, root_data->medium_page_header_table, sizeof(pas_page_header_table))))
    280315        return NULL;
    281316   
    … …  
    332367}
    333368
     369PAS_SEGREGATED_PAGE_CONFIG_SPECIALIZATION_DEFINITIONS(
     370    jit_small_segregated_page_config, JIT_HEAP_CONFIG.small_segregated_config);
    334371PAS_BITFIT_PAGE_CONFIG_SPECIALIZATION_DEFINITIONS(
    335372    jit_small_bitfit_page_config, JIT_HEAP_CONFIG.small_bitfit_config);

  • trunk/Source/bmalloc/libpas/src/libpas/jit_heap_config.h

    r289579 r289590  
    11/*
    2  * Copyright (c) 2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2021-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3737PAS_BEGIN_EXTERN_C;
    3838
    39 #define JIT_SMALL_MIN_ALIGN_SHIFT 2u
    40 #define JIT_SMALL_MIN_ALIGN (1u << JIT_SMALL_MIN_ALIGN_SHIFT)
     39#define JIT_SMALL_SEGREGATED_MIN_ALIGN_SHIFT 4u
     40#define JIT_SMALL_SEGREGATED_MIN_ALIGN (1u << JIT_SMALL_SEGREGATED_MIN_ALIGN_SHIFT)
     41#define JIT_SMALL_BITFIT_MIN_ALIGN_SHIFT 2u
     42#define JIT_SMALL_BITFIT_MIN_ALIGN (1u << JIT_SMALL_BITFIT_MIN_ALIGN_SHIFT)
    4143#define JIT_SMALL_PAGE_SIZE 16384u
    4244#define JIT_SMALL_GRANULE_SIZE 16384u
    43 #define JIT_MEDIUM_MIN_ALIGN_SHIFT 8u
    44 #define JIT_MEDIUM_MIN_ALIGN (1u << JIT_MEDIUM_MIN_ALIGN_SHIFT)
     45#define JIT_MEDIUM_BITFIT_MIN_ALIGN_SHIFT 8u
     46#define JIT_MEDIUM_BITFIT_MIN_ALIGN (1u << JIT_MEDIUM_BITFIT_MIN_ALIGN_SHIFT)
    4547#define JIT_MEDIUM_PAGE_SIZE 131072u
    4648#if PAS_ARM64
    … …  
    6769PAS_API pas_page_base* jit_page_header_for_boundary_remote(pas_enumerator* enumerator, void* boundary);
    6870
    69 static PAS_ALWAYS_INLINE pas_page_base* jit_small_bitfit_page_header_for_boundary(void* boundary);
    70 static PAS_ALWAYS_INLINE void* jit_small_bitfit_boundary_for_page_header(pas_page_base* page);
     71static PAS_ALWAYS_INLINE pas_page_base* jit_small_page_header_for_boundary(void* boundary);
     72static PAS_ALWAYS_INLINE void* jit_small_boundary_for_page_header(pas_page_base* page);
     73PAS_API void* jit_small_segregated_allocate_page(
     74    pas_segregated_heap* heap, pas_physical_memory_transaction* transaction, pas_segregated_page_role role);
     75PAS_API pas_page_base* jit_small_segregated_create_page_header(
     76    void* boundary, pas_page_kind kind, pas_lock_hold_mode heap_lock_hold_mode);
     77PAS_API void jit_small_destroy_page_header(
     78    pas_page_base* page, pas_lock_hold_mode heap_lock_hold_mode);
     79PAS_API pas_segregated_shared_page_directory* jit_small_segregated_shared_page_directory_selector(
     80    pas_segregated_heap* heap, pas_segregated_size_directory* directory);
     81
     82PAS_SEGREGATED_PAGE_CONFIG_SPECIALIZATION_DECLARATIONS(jit_small_segregated_page_config);
     83
    7184PAS_API void* jit_small_bitfit_allocate_page(
    7285    pas_segregated_heap* heap, pas_physical_memory_transaction* transaction);
    7386PAS_API pas_page_base* jit_small_bitfit_create_page_header(
    7487    void* boundary, pas_page_kind kind, pas_lock_hold_mode heap_lock_hold_mode);
    75 PAS_API void jit_small_bitfit_destroy_page_header(
    76     pas_page_base* page, pas_lock_hold_mode heap_lock_hold_mode);
    7788
    7889PAS_BITFIT_PAGE_CONFIG_SPECIALIZATION_DECLARATIONS(jit_small_bitfit_page_config);
    7990
    80 static PAS_ALWAYS_INLINE pas_page_base* jit_medium_bitfit_page_header_for_boundary(void* boundary);
     91static PAS_ALWAYS_INLINE pas_page_base* jit_medium_page_header_for_boundary(void* boundary);
     92static PAS_ALWAYS_INLINE void* jit_medium_boundary_for_page_header(pas_page_base* page);
    8193PAS_API void* jit_medium_bitfit_allocate_page(
    8294    pas_segregated_heap* heap, pas_physical_memory_transaction* transaction);
    8395PAS_API pas_page_base* jit_medium_bitfit_create_page_header(
    8496    void* boundary, pas_page_kind kind, pas_lock_hold_mode heap_lock_hold_mode);
    85 PAS_API void jit_medium_bitfit_destroy_page_header(
     97PAS_API void jit_medium_destroy_page_header(
    8698    pas_page_base* page, pas_lock_hold_mode heap_lock_hold_mode);
    8799
    … …  
    122134            .page_config_ptr = &jit_heap_config.variant_lowercase ## _bitfit_config.base, \
    123135            .page_config_kind = pas_page_config_kind_bitfit, \
    124             .min_align_shift = JIT_ ## variant_uppercase ## _MIN_ALIGN_SHIFT, \
     136            .min_align_shift = JIT_ ## variant_uppercase ## _BITFIT_MIN_ALIGN_SHIFT, \
    125137            .page_size = JIT_ ## variant_uppercase ## _PAGE_SIZE, \
    126138            .granule_size = JIT_ ## variant_uppercase ## _GRANULE_SIZE, \
    127139            .max_object_size = \
    128                 PAS_BITFIT_MAX_FREE_MAX_VALID << JIT_ ## variant_uppercase ## _MIN_ALIGN_SHIFT, \
    129             .page_header_for_boundary = jit_ ## variant_lowercase ## _bitfit_page_header_for_boundary, \
    130             .boundary_for_page_header = jit_ ## variant_lowercase ## _bitfit_boundary_for_page_header, \
     140                PAS_BITFIT_MAX_FREE_MAX_VALID << JIT_ ## variant_uppercase ## _BITFIT_MIN_ALIGN_SHIFT, \
     141            .page_header_for_boundary = jit_ ## variant_lowercase ## _page_header_for_boundary, \
     142            .boundary_for_page_header = jit_ ## variant_lowercase ## _boundary_for_page_header, \
    131143            .page_header_for_boundary_remote = jit_page_header_for_boundary_remote, \
    132144            .create_page_header = jit_ ## variant_lowercase ## _bitfit_create_page_header, \
    133             .destroy_page_header = jit_ ## variant_lowercase ## _bitfit_destroy_page_header \
     145            .destroy_page_header = jit_ ## variant_lowercase ## _destroy_page_header \
    134146        }, \
    135147        .variant = pas_ ## variant_lowercase ## _bitfit_page_config_variant, \
    … …  
    148160        .get_type_alignment = jit_type_alignment, \
    149161        .dump_type = jit_type_dump, \
    150         .large_alignment = JIT_SMALL_MIN_ALIGN, \
     162        .large_alignment = PAS_MIN_CONST(JIT_SMALL_SEGREGATED_MIN_ALIGN, JIT_SMALL_BITFIT_MIN_ALIGN), \
    151163        .small_segregated_config = { \
    152164            .base = { \
    153                 .is_enabled = false \
    154             } \
     165                .is_enabled = true, \
     166                .heap_config_ptr = &jit_heap_config, \
     167                .page_config_ptr = &jit_heap_config.small_segregated_config.base, \
     168                .page_config_kind = pas_page_config_kind_segregated, \
     169                .min_align_shift = JIT_SMALL_SEGREGATED_MIN_ALIGN_SHIFT, \
     170                .page_size = JIT_SMALL_PAGE_SIZE, \
     171                .granule_size = JIT_SMALL_GRANULE_SIZE, \
     172                .max_object_size = PAS_MAX_OBJECT_SIZE(JIT_SMALL_PAGE_SIZE), \
     173                .page_header_for_boundary = jit_small_page_header_for_boundary, \
     174                .boundary_for_page_header = jit_small_boundary_for_page_header, \
     175                .page_header_for_boundary_remote = jit_page_header_for_boundary_remote, \
     176                .create_page_header = jit_small_segregated_create_page_header, \
     177                .destroy_page_header = jit_small_destroy_page_header \
     178            }, \
     179            .variant = pas_small_segregated_page_config_variant, \
     180            .kind = pas_segregated_page_config_kind_jit_small_segregated, \
     181            .wasteage_handicap = 1., \
     182            .sharing_shift = PAS_SMALL_SHARING_SHIFT, \
     183            .num_alloc_bits = PAS_BASIC_SEGREGATED_NUM_ALLOC_BITS(JIT_SMALL_SEGREGATED_MIN_ALIGN_SHIFT, \
     184                                                                  JIT_SMALL_PAGE_SIZE), \
     185            .shared_payload_offset = 0, \
     186            .exclusive_payload_offset = 0, \
     187            .shared_payload_size = 0, \
     188            .exclusive_payload_size = JIT_SMALL_PAGE_SIZE, \
     189            .shared_logging_mode = pas_segregated_deallocation_no_logging_mode, \
     190            .exclusive_logging_mode = pas_segregated_deallocation_size_oblivious_logging_mode, \
     191            .use_reversed_current_word = PAS_ARM64, \
     192            .check_deallocation = false, \
     193            .enable_empty_word_eligibility_optimization_for_shared = false, \
     194            .enable_empty_word_eligibility_optimization_for_exclusive = true, \
     195            .enable_view_cache = true, \
     196            .page_allocator = jit_small_segregated_allocate_page, \
     197            .shared_page_directory_selector = jit_small_segregated_shared_page_directory_selector, \
     198            PAS_SEGREGATED_PAGE_CONFIG_SPECIALIZATIONS(jit_small_segregated_page_config) \
    155199        }, \
    156200        .medium_segregated_config = { \
    … …  
    192236
    193237PAS_API extern pas_large_heap_physical_page_sharing_cache jit_large_fresh_memory_heap;
    194 PAS_API extern pas_page_header_table jit_small_bitfit_page_header_table;
    195 PAS_API extern pas_page_header_table jit_medium_bitfit_page_header_table;
     238PAS_API extern pas_page_header_table jit_small_page_header_table;
     239PAS_API extern pas_page_header_table jit_medium_page_header_table;
    196240PAS_API extern pas_heap_runtime_config jit_heap_runtime_config;
    197241PAS_API extern jit_heap_config_root_data jit_root_data;
    198242
    199 static PAS_ALWAYS_INLINE pas_page_base* jit_small_bitfit_page_header_for_boundary(void* boundary)
     243static PAS_ALWAYS_INLINE pas_page_base* jit_small_page_header_for_boundary(void* boundary)
    200244{
    201245    return pas_page_header_table_get_for_boundary(
    202         &jit_small_bitfit_page_header_table, JIT_SMALL_PAGE_SIZE, boundary);
    203 }
    204 
    205 static PAS_ALWAYS_INLINE void* jit_small_bitfit_boundary_for_page_header(pas_page_base* page)
     246        &jit_small_page_header_table, JIT_SMALL_PAGE_SIZE, boundary);
     247}
     248
     249static PAS_ALWAYS_INLINE void* jit_small_boundary_for_page_header(pas_page_base* page)
    206250{
    207251    return pas_page_header_table_get_boundary(
    208         &jit_small_bitfit_page_header_table, JIT_SMALL_PAGE_SIZE, page);
    209 }
    210 
    211 static PAS_ALWAYS_INLINE pas_page_base* jit_medium_bitfit_page_header_for_boundary(void* boundary)
     252        &jit_small_page_header_table, JIT_SMALL_PAGE_SIZE, page);
     253}
     254
     255static PAS_ALWAYS_INLINE pas_page_base* jit_medium_page_header_for_boundary(void* boundary)
    212256{
    213257    return pas_page_header_table_get_for_boundary(
    214         &jit_medium_bitfit_page_header_table, JIT_MEDIUM_PAGE_SIZE, boundary);
    215 }
    216 
    217 static PAS_ALWAYS_INLINE void* jit_medium_bitfit_boundary_for_page_header(pas_page_base* page)
     258        &jit_medium_page_header_table, JIT_MEDIUM_PAGE_SIZE, boundary);
     259}
     260
     261static PAS_ALWAYS_INLINE void* jit_medium_boundary_for_page_header(pas_page_base* page)
    218262{
    219263    return pas_page_header_table_get_boundary(
    220         &jit_medium_bitfit_page_header_table, JIT_MEDIUM_PAGE_SIZE, page);
     264        &jit_medium_page_header_table, JIT_MEDIUM_PAGE_SIZE, page);
    221265}
    222266
    … …  
    226270
    227271    result = pas_page_header_table_get_for_address(
    228         &jit_small_bitfit_page_header_table, JIT_SMALL_PAGE_SIZE, (void*)begin);
     272        &jit_small_page_header_table, JIT_SMALL_PAGE_SIZE, (void*)begin);
    229273    if (result)
    230274        return result;
    231275
    232276    return pas_page_header_table_get_for_address(
    233         &jit_medium_bitfit_page_header_table, JIT_MEDIUM_PAGE_SIZE, (void*)begin);
     277        &jit_medium_page_header_table, JIT_MEDIUM_PAGE_SIZE, (void*)begin);
    234278}
    235279

  • trunk/Source/bmalloc/libpas/src/libpas/jit_heap_config_root_data.h

    r279867 r289590  
    11/*
    2  * Copyright (c) 2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2021-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3737
    3838struct jit_heap_config_root_data {
    39     pas_page_header_table* small_bitfit_page_header_table;
    40     pas_page_header_table* medium_bitfit_page_header_table;
     39    pas_page_header_table* small_page_header_table;
     40    pas_page_header_table* medium_page_header_table;
    4141};
    4242

  • trunk/Source/bmalloc/libpas/src/libpas/pas_bitfit_heap.c

    r286493 r289590  
    11/*
    2  * Copyright (c) 2020-2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2020-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    6161
    6262pas_bitfit_variant_selection pas_bitfit_heap_select_variant(size_t requested_object_size,
    63                                                             pas_heap_config* config)
     63                                                            pas_heap_config* config,
     64                                                            pas_heap_runtime_config* runtime_config)
    6465{
    6566    static const bool verbose = false;
    … …  
    8081
    8182        page_config = *pas_heap_config_bitfit_page_config_ptr_for_variant(config, variant);
    82         if (!pas_bitfit_page_config_is_enabled(page_config))
     83        if (!pas_bitfit_page_config_is_enabled(page_config, runtime_config))
    8384            continue;
    8485
    … …  
    126127                                                     pas_bitfit_size_class* size_class,
    127128                                                     unsigned object_size,
    128                                                      pas_heap_config* config)
     129                                                     pas_heap_config* config,
     130                                                     pas_heap_runtime_config* runtime_config)
    129131{
    130132    static const bool verbose = false;
    … …  
    140142    }
    141143
    142     best = pas_bitfit_heap_select_variant(object_size, config);
     144    best = pas_bitfit_heap_select_variant(object_size, config, runtime_config);
    143145
    144146    pas_heap_lock_assert_held();

  • trunk/Source/bmalloc/libpas/src/libpas/pas_bitfit_heap.h

    r286493 r289590  
    11/*
    2  * Copyright (c) 2020-2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2020-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3434struct pas_bitfit_heap;
    3535struct pas_heap_config;
     36struct pas_heap_runtime_config;
    3637struct pas_segregated_size_directory;
    3738struct pas_segregated_heap;
    3839typedef struct pas_bitfit_heap pas_bitfit_heap;
    3940typedef struct pas_heap_config pas_heap_config;
     41typedef struct pas_heap_runtime_config pas_heap_runtime_config;
    4042typedef struct pas_segregated_size_directory pas_segregated_size_directory;
    4143typedef struct pas_segregated_heap pas_segregated_heap;
    … …  
    6264
    6365PAS_API pas_bitfit_variant_selection
    64 pas_bitfit_heap_select_variant(size_t object_size, pas_heap_config* config);
     66pas_bitfit_heap_select_variant(size_t object_size,
     67                               pas_heap_config* config,
     68                               pas_heap_runtime_config* runtime_config);
    6569
    6670PAS_API void pas_bitfit_heap_construct_and_insert_size_class(pas_bitfit_heap* heap,
    6771                                                             pas_bitfit_size_class* size_class,
    6872                                                             unsigned object_size,
    69                                                              pas_heap_config* config);
     73                                                             pas_heap_config* config,
     74                                                             pas_heap_runtime_config* runtime_config);
    7075
    7176PAS_API pas_heap_summary pas_bitfit_heap_compute_summary(pas_bitfit_heap* heap);

  • trunk/Source/bmalloc/libpas/src/libpas/pas_bitfit_page_config.h

    r289579 r289590  
    11/*
    2  * Copyright (c) 2020-2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2020-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3232#include "pas_bitvector.h"
    3333#include "pas_fast_path_allocation_result.h"
     34#include "pas_heap_runtime_config.h"
    3435#include "pas_page_base_config.h"
    3536#include "pas_page_malloc.h"
    … …  
    136137        pas_bitfit_page* page, uintptr_t begin, size_t new_size)
    137138
    138 static inline bool pas_bitfit_page_config_is_enabled(pas_bitfit_page_config config)
     139static inline bool pas_bitfit_page_config_is_enabled(pas_bitfit_page_config config,
     140                                                     pas_heap_runtime_config* runtime_config)
    139141{
    140142    if (!config.base.is_enabled)
     143        return false;
     144    /* Doing this check here is not super necessary, but it's sort of nice for cases where we have a heap
     145       that sometimes uses bitfit exclusively or sometimes uses segregated exclusively and that's selected
     146       by selecting or mutating runtime_configs. This is_enabled function is only called as part of the math
     147       that sets up size classes, at least for now, so the implications of not doing this check are rather
     148       tiny. */
     149    if (!runtime_config->max_bitfit_object_size)
    141150        return false;
    142151    switch (config.variant) {

  • trunk/Source/bmalloc/libpas/src/libpas/pas_heap_config.h

    r289579 r289590  
    11/*
    2  * Copyright (c) 2018-2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2018-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    309309}
    310310
     311static PAS_ALWAYS_INLINE size_t pas_heap_config_segregated_heap_min_align_shift(pas_heap_config config)
     312{
     313    size_t result;
     314    result = SIZE_MAX;
     315    if (config.small_bitfit_config.base.is_enabled)
     316        result = PAS_MIN(result, config.small_bitfit_config.base.min_align_shift);
     317    if (config.small_segregated_config.base.is_enabled)
     318        result = PAS_MIN(result, config.small_segregated_config.base.min_align_shift);
     319    PAS_ASSERT(result != SIZE_MAX);
     320    return result;
     321}
     322
     323static PAS_ALWAYS_INLINE size_t pas_heap_config_segregated_heap_min_align(pas_heap_config config)
     324{
     325    return (size_t)1 << pas_heap_config_segregated_heap_min_align_shift(config);
     326}
     327
    311328/* Returns true if we were the first to active it. Must hold the heap lock to call this. This is
    312329   permissive of recursive initialization: in that case, it will just pretend that the config is

  • trunk/Source/bmalloc/libpas/src/libpas/pas_segregated_heap.c

    r289579 r289590  
    5353                                                           const char* where);
    5454
    55 static size_t min_object_size_for_heap_config(pas_heap_config* config)
     55static size_t min_align_for_heap(pas_segregated_heap* heap,
     56                                 pas_heap_config* config)
    5657{
    5758    pas_segregated_page_config_variant segregated_variant;
    … …  
    6162        page_config =
    6263            pas_heap_config_segregated_page_config_ptr_for_variant(config, segregated_variant);
    63         if (!pas_segregated_page_config_is_enabled(*page_config))
     64        if (!pas_segregated_page_config_is_enabled(*page_config, heap->runtime_config))
    6465            continue;
    6566        return pas_segregated_page_config_min_align(*page_config);
    … …  
    7172        page_config =
    7273            pas_heap_config_bitfit_page_config_ptr_for_variant(config, bitfit_variant);
    73         if (!pas_bitfit_page_config_is_enabled(*page_config))
     74        if (!pas_bitfit_page_config_is_enabled(*page_config, heap->runtime_config))
    7475            continue;
    7576        return pas_page_base_config_min_align(page_config->base);
    … …  
    8081}
    8182
     83static size_t min_object_size_for_heap(pas_segregated_heap* heap,
     84                                       pas_heap_config* config)
     85{
     86    return min_align_for_heap(heap, config);
     87}
     88
    8289static size_t max_object_size_for_page_config(pas_heap* parent_heap,
    8390                                              pas_page_base_config* page_config)
    … …  
    99106}
    100107
    101 static size_t max_segregated_object_size_for_heap_config(pas_heap* parent_heap,
    102                                                          pas_segregated_heap* heap,
    103                                                          pas_heap_config* config)
     108static size_t max_segregated_object_size_for_heap(pas_heap* parent_heap,
     109                                                  pas_segregated_heap* heap,
     110                                                  pas_heap_config* config)
    104111{
    105112    static const bool verbose = false;
    … …  
    112119       
    113120        page_config = pas_heap_config_segregated_page_config_ptr_for_variant(config, variant);
    114         if (!pas_segregated_page_config_is_enabled(*page_config))
     121        if (!pas_segregated_page_config_is_enabled(*page_config, heap->runtime_config))
    115122            continue;
    116123
    … …  
    126133}
    127134
    128 static size_t max_bitfit_object_size_for_heap_config(pas_heap* parent_heap,
    129                                                      pas_segregated_heap* heap,
    130                                                      pas_heap_config* config)
     135static size_t max_bitfit_object_size_for_heap(pas_heap* parent_heap,
     136                                              pas_segregated_heap* heap,
     137                                              pas_heap_config* config)
    131138{
    132139    static const bool verbose = false;
    … …  
    146153        }
    147154
    148         if (!pas_bitfit_page_config_is_enabled(*page_config)) {
     155        if (!pas_bitfit_page_config_is_enabled(*page_config, heap->runtime_config)) {
    149156            if (verbose) {
    150157                pas_log("Not considering %s because it's disabled.\n",
    … …  
    183190}
    184191
    185 static size_t max_object_size_for_heap_config(pas_heap* parent_heap,
    186                                               pas_segregated_heap* heap,
    187                                               pas_heap_config* config)
    188 {
    189     return PAS_MAX(max_segregated_object_size_for_heap_config(parent_heap, heap, config),
    190                    max_bitfit_object_size_for_heap_config(parent_heap, heap, config));
     192static size_t max_object_size_for_heap(pas_heap* parent_heap,
     193                                       pas_segregated_heap* heap,
     194                                       pas_heap_config* config)
     195{
     196    return PAS_MAX(max_segregated_object_size_for_heap(parent_heap, heap, config),
     197                   max_bitfit_object_size_for_heap(parent_heap, heap, config));
    191198}
    192199
    … …  
    848855    pas_heap_lock_assert_held();
    849856   
    850     PAS_ASSERT(directory->object_size >= min_object_size_for_heap_config(config));
     857    PAS_ASSERT(directory->object_size >= min_object_size_for_heap(heap, config));
    851858
    852859    rematerialize_size_lookup_if_necessary(heap, config, cached_index);
    … …  
    13291336    pas_segregated_heap_medium_directory_tuple* medium_tuple;
    13301337    bool is_utility;
     1338    size_t dynamic_min_align;
     1339    size_t static_min_align;
    13311340    size_t type_alignment;
    13321341
    … …  
    14241433   
    14251434    index = pas_segregated_heap_index_for_size(size, *config);
     1435
     1436    if (verbose)
     1437        pas_log("index = %zu\n", index);
     1438   
    14261439    object_size = pas_segregated_heap_size_for_index(index, *config);
     1440
     1441    if (verbose)
     1442        pas_log("object_size = %zu\n", object_size);
    14271443
    14281444    if (object_size < size) {
    … …  
    14321448    }
    14331449
    1434     object_size = PAS_MAX(min_object_size_for_heap_config(config), object_size);
    1435 
    1436     PAS_ASSERT(pas_is_aligned(
    1437                    object_size, pas_segregated_page_config_min_align(config->small_segregated_config)));
    1438 
    1439     alignment = PAS_MAX(alignment,
    1440                         pas_segregated_page_config_min_align(config->small_segregated_config));
     1450    object_size = PAS_MAX(min_object_size_for_heap(heap, config), object_size);
     1451
     1452    if (verbose)
     1453        pas_log("object_size after accounting for min_object_size = %zu\n", object_size);
     1454
     1455    static_min_align = pas_heap_config_segregated_heap_min_align(*config);
     1456    dynamic_min_align = min_align_for_heap(heap, config);
     1457    PAS_ASSERT(dynamic_min_align >= static_min_align);
     1458
     1459    PAS_ASSERT(pas_is_aligned(object_size, alignment));
     1460    PAS_ASSERT(pas_is_aligned(object_size, static_min_align));
     1461
     1462    object_size = pas_round_up_to_power_of_2(object_size, dynamic_min_align);
     1463   
     1464    alignment = PAS_MAX(alignment, dynamic_min_align);
    14411465
    14421466    if (alignment > type_alignment) {
    … …  
    14671491       immediately give up. Do this before ensure_size_lookup so that heaps that are only used
    14681492       for large object allocation don't allocate any small heap meta-data. */
    1469     if (object_size > max_object_size_for_heap_config(parent_heap, heap, config)) {
     1493    if (object_size > max_object_size_for_heap(parent_heap, heap, config)) {
    14701494        if (verbose)
    14711495            pas_log("It's too big.\n");
    … …  
    17051729                page_config_ptr =
    17061730                    pas_heap_config_segregated_page_config_ptr_for_variant(config, variant);
    1707                 if (!pas_segregated_page_config_is_enabled(*page_config_ptr))
     1731                if (!pas_segregated_page_config_is_enabled(*page_config_ptr, heap->runtime_config))
    17081732                    continue;
    17091733               
    … …  
    17491773            PAS_ASSERT(object_size <= heap->runtime_config->max_bitfit_object_size);
    17501774            PAS_TESTING_ASSERT(
    1751                 object_size <= max_bitfit_object_size_for_heap_config(parent_heap, heap, config));
     1775                object_size <= max_bitfit_object_size_for_heap(parent_heap, heap, config));
    17521776            best_bytes_dirtied_per_object =
    1753                 pas_bitfit_heap_select_variant(object_size, config).object_size;
     1777                pas_bitfit_heap_select_variant(object_size, config, heap->runtime_config).object_size;
    17541778            PAS_ASSERT(!is_utility);
    17551779        }

  • trunk/Source/bmalloc/libpas/src/libpas/pas_segregated_heap.h

    r288342 r289590  
    11/*
    2  * Copyright (c) 2018-2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2018-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    156156pas_segregated_heap_index_for_size(size_t size, pas_heap_config config)
    157157{
    158     return (size + pas_segregated_page_config_min_align(config.small_segregated_config) - 1)
    159         >> config.small_segregated_config.base.min_align_shift;
     158    size_t min_align_shift;
     159    size_t min_align;
     160    min_align_shift = pas_heap_config_segregated_heap_min_align_shift(config);
     161    min_align = (size_t)1 << min_align_shift;
     162    return (size + min_align - 1) >> min_align_shift;
    160163}
    161164
    … …  
    163166pas_segregated_heap_size_for_index(size_t index, pas_heap_config config)
    164167{
    165     return index << config.small_segregated_config.base.min_align_shift;
     168    return index << pas_heap_config_segregated_heap_min_align_shift(config);
    166169}
    167170

  • trunk/Source/bmalloc/libpas/src/libpas/pas_segregated_page.c

    r289579 r289590  
    244244        if (pas_segregated_size_directory_view_cache_capacity(directory)) {
    245245            PAS_ASSERT(directory->view_cache_index);
    246             PAS_ASSERT(directory->view_cache_index < (pas_allocator_index)UINT_MAX);
    247246            page->view_cache_index = directory->view_cache_index;
    248247        } else

  • trunk/Source/bmalloc/libpas/src/libpas/pas_segregated_page_config.h

    r287192 r289590  
    11/*
    2  * Copyright (c) 2018-2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2018-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3030#include "pas_bitvector.h"
    3131#include "pas_config.h"
     32#include "pas_heap_runtime_config.h"
    3233#include "pas_local_allocator_refill_mode.h"
    3334#include "pas_lock.h"
    34 #include "pas_segregated_page_config_kind.h"
    3535#include "pas_page_base_config.h"
    3636#include "pas_page_granule_use_count.h"
    3737#include "pas_page_sharing_mode.h"
    3838#include "pas_segregated_deallocation_logging_mode.h"
     39#include "pas_segregated_page_config_kind.h"
    3940#include "pas_segregated_page_config_variant.h"
    4041#include "pas_segregated_page_role.h"
    … …  
    223224    ((object_payload_size) / (min_num_objects))
    224225
    225 static inline bool pas_segregated_page_config_is_enabled(pas_segregated_page_config config)
     226static inline bool pas_segregated_page_config_is_enabled(pas_segregated_page_config config,
     227                                                         pas_heap_runtime_config* runtime_config)
    226228{
    227229    if (!config.base.is_enabled)
     230        return false;
     231    /* Doing this check here is not super necessary, but it's sort of nice for cases where we have a heap
     232       that sometimes uses bitfit exclusively or sometimes uses segregated exclusively and that's selected
     233       by selecting or mutating runtime_configs. This is_enabled function is only called as part of the math
     234       that sets up size classes, at least for now, so the implications of not doing this check are rather
     235       tiny. */
     236    if (!runtime_config->max_segregated_object_size)
    228237        return false;
    229238    switch (config.variant) {

  • trunk/Source/bmalloc/libpas/src/libpas/pas_segregated_page_config_kind.def

    r286493 r289590  
    11/*
    2  * Copyright (c) 2019-2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2019-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    105105#endif
    106106
     107#if PAS_ENABLE_JIT
     108PAS_DEFINE_SEGREGATED_PAGE_CONFIG_KIND(jit_small_segregated,
     109                                       JIT_HEAP_CONFIG.small_segregated_config)
     110#endif

  • trunk/Source/bmalloc/libpas/src/libpas/pas_segregated_page_config_kind_and_role.h

    r286493 r289590  
    11/*
    2  * Copyright (c) 2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2021-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    4242typedef enum pas_segregated_page_config_kind_and_role pas_segregated_page_config_kind_and_role;
    4343
    44 #define PAS_SEGREGATED_PAGE_CONFIG_KIND_AND_ROLE_NUM_BITS 5u
     44#define PAS_SEGREGATED_PAGE_CONFIG_KIND_AND_ROLE_NUM_BITS 6u
    4545#define PAS_SEGREGATED_PAGE_CONFIG_KIND_AND_ROLE_MASK \
    4646    ((1u << PAS_SEGREGATED_PAGE_CONFIG_KIND_AND_ROLE_NUM_BITS) - 1u)

  • trunk/Source/bmalloc/libpas/src/libpas/pas_segregated_size_directory.c

    r289579 r289590  
    11/*
    2  * Copyright (c) 2019-2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2019-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    131131       
    132132        pas_bitfit_heap_construct_and_insert_size_class(
    133             bitfit_heap, bitfit_size_class, object_size, heap_config);
     133            bitfit_heap, bitfit_size_class, object_size, heap_config, heap->runtime_config);
    134134    }
    135135   

  • trunk/Source/bmalloc/libpas/src/test/IsoHeapChaosTests.cpp

    r287383 r289590  
    11/*
    2  * Copyright (c) 2019-2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2019-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    12081208
    12091209        ADD_GROUP(addTheTests(1, true));
     1210
     1211        {
     1212            ForceBitfit forceBitfit;
     1213            ADD_GROUP(addTheTests(1, true));
     1214        }
    12101215    }
    12111216
    … …  
    12281233
    12291234        ADD_GROUP(addTheTests(1, false));
     1235
     1236        {
     1237            ForceBitfit forceBitfit;
     1238            ADD_GROUP(addTheTests(1, false));
     1239        }
    12301240    }
    12311241#endif // PAS_ENABLE_HOTBIT

  • trunk/Source/bmalloc/libpas/src/test/JITHeapTests.cpp

    r279867 r289590  
    11/*
    2  * Copyright (c) 2021 Apple Inc. All rights reserved.
     2 * Copyright (c) 2021-2022 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    7070}
    7171
     72void testAllocationSize(size_t requestedSize, size_t actualSize)
     73{
     74    CHECK_EQUAL(jit_heap_get_size(jit_heap_try_allocate(requestedSize)), actualSize);
     75}
     76
    7277} // anonymous namespace
    7378
    … …  
    7984    BootJITHeap bootJITHeap;
    8085
    81     ADD_TEST(testAllocateShrinkAndAllocate(0, 0, 0, 0, 0, 4));
    82     ADD_TEST(testAllocateShrinkAndAllocate(0, 0, 128, 64, 64, 4));
    83     ADD_TEST(testAllocateShrinkAndAllocate(32, 10, 128, 64, 64, 4));
    84     ADD_TEST(testAllocateShrinkAndAllocate(32, 10, 1000, 500, 1000, 4));
    85     ADD_TEST(testAllocateShrinkAndAllocate(0, 0, 2048, 512, 1100, 256));
    86     ADD_TEST(testAllocateShrinkAndAllocate(32, 10, 2048, 512, 1100, 256));
    87     ADD_TEST(testAllocateShrinkAndAllocate(1100, 10, 2048, 512, 1100, 256));
    88     ADD_TEST(testAllocateShrinkAndAllocate(0, 0, 100000, 10000, 80000, 4));
     86    {
     87        ForceBitfit forceBitfit;
     88        ADD_TEST(testAllocateShrinkAndAllocate(0, 0, 0, 0, 0, 4));
     89        ADD_TEST(testAllocateShrinkAndAllocate(0, 0, 128, 64, 64, 4));
     90        ADD_TEST(testAllocateShrinkAndAllocate(32, 10, 128, 64, 64, 4));
     91        ADD_TEST(testAllocateShrinkAndAllocate(32, 10, 1000, 500, 1000, 4));
     92        ADD_TEST(testAllocateShrinkAndAllocate(0, 0, 2048, 512, 1100, 256));
     93        ADD_TEST(testAllocateShrinkAndAllocate(32, 10, 2048, 512, 1100, 256));
     94        ADD_TEST(testAllocateShrinkAndAllocate(1100, 10, 2048, 512, 1100, 256));
     95        ADD_TEST(testAllocateShrinkAndAllocate(0, 0, 100000, 10000, 80000, 4));
     96    }
     97   
     98    ADD_TEST(testAllocationSize(4, 16));
     99    ADD_TEST(testAllocationSize(8, 16));
     100    ADD_TEST(testAllocationSize(12, 16));
     101    ADD_TEST(testAllocationSize(16, 16));
     102    ADD_TEST(testAllocationSize(20, 32));
     103    {
     104        ForceBitfit forceBitfit;
     105        ADD_TEST(testAllocationSize(4, 4));
     106        ADD_TEST(testAllocationSize(8, 8));
     107        ADD_TEST(testAllocationSize(12, 12));
     108        ADD_TEST(testAllocationSize(16, 16));
     109        ADD_TEST(testAllocationSize(20, 20));
     110    }
    89111#endif // PAS_ENABLE_JIT
    90112}

  • trunk/Source/bmalloc/libpas/src/test/TestHarness.cpp

    r289579 r289590  
    379379void addThingyAndUtilityHeapAllocationTests();
    380380void addUtilsTests();
     381void addViewCacheTests();
    381382
    382383void testSucceeded()
    … …  
    745746    ADD_SUITE(TSD);
    746747    ADD_SUITE(Utils);
     748    ADD_SUITE(ViewCache);
    747749   
    748750    string filter;
Note: See TracChangeset for help on using the changeset viewer.