Changeset 242252 in webkit


Ignore:
Timestamp:
Feb 28, 2019 7:13:31 PM (5 years ago)
Author:
ysuzuki@apple.com
Message:

[JSC] sizeof(JSString) should be 16
​https://bugs.webkit.org/show_bug.cgi?id=194375

Reviewed by Saam Barati.

JSTests:

  • microbenchmarks/make-rope.js: Added.

(makeRope):

  • stress/to-lower-case-intrinsic-on-empty-rope.js: We no longer allow 0 length JSString except for jsEmptyString singleton per VM.

(returnRope.helper): Deleted.
(returnRope): Deleted.

Source/JavaScriptCore:

This patch reduces sizeof(JSString) from 24 to 16 to fit it into GC heap cell atom. And it also reduces sizeof(JSRopeString) from 48 to 32.
Both classes cut 16 bytes per instance in GC allocation. This new layout is used in 64bit architectures which has little endianess.

JSString no longer has length and flags directly. JSString has String, and we query information to this String instead of holding duplicate
information in JSString. We embed isRope bit into this String's pointer so that we can convert JSRopeString to JSString in an atomic manner.
We emit store-store fence before we put String pointer. This should exist even before this patch, so this patch also fixes one concurrency issue.

The old JSRopeString separately had JSString* fibers along with String. In this patch, we merge the first JSString* fiber and String pointer
storage into one to reduce the size of JSRopeString. JSRopeString has three pointer width storage. We pick 48bit effective address of JSString*
fibers to compress three fibers + length + flags into three pointer width storage.

In 64bit architecture, JSString and JSRopeString have the following memory layout to make sizeof(JSString) == 16 and sizeof(JSRopeString) == 32.
JSString has only one pointer. We use it for String. length() and is8Bit() queries go to StringImpl. In JSRopeString, we reuse the above pointer
place for the 1st fiber. JSRopeString has three fibers so its size is 48. To keep length and is8Bit flag information in JSRopeString, JSRopeString
encodes these information into the fiber pointers. is8Bit flag is encoded in the 1st fiber pointer. length is embedded directly, and two fibers
are compressed into 12bytes. isRope information is encoded in the first fiber's LSB.

Since length of JSRopeString should be frequently accessed compared to each fiber, we put length in contiguous 32byte field, and compress 2nd
and 3rd fibers into the following 80byte fields. One problem is that now 2nd and 3rd fibers are split. Storing and loading 2nd and 3rd fibers
are not one pointer load operation. To make concurrent collector work correctly, we must initialize 2nd and 3rd fibers at JSRopeString creation
and we must not modify these part later.

0 8 10 16 32 48

JSString [ ID ][ header ][ String pointer 0]
JSRopeString [ ID ][ header ][ flags ][ 1st fiber 1][ length ][2nd lower32][2nd upper16][3rd lower16][3rd upper32]


isRope bit

Since fibers in JSRopeString are not initialized in atomic pointer store manner, we must initialize all the fiber fields at JSRopeString creation.
To achieve this, we modify our JSRopeString::RopeBuilder implementation not to create half-baked JSRopeString.

This patch also makes an empty JSString singleton per VM. This makes evaluation of JSString in boolean context one pointer comparison. This is
critical in this change since this patch enlarges the code necessary to get length from JSString in JIT. Without this guarantee, our code of boolean
context evaluation is bloated. This patch hides all the JSString::create and JSRopeString::create in the private permission. JSString and JSRopeString
creation is only allowed from jsString and related helper functions and they return a singleton empty JSString if the length is zero. We also change
JSRopeString::RopeBuilder not to construct an empty JSRopeString.

This patch is performance neutral in Speedometer2 and JetStream2. And it improves RAMification by 2.7%.

  • JavaScriptCore.xcodeproj/project.pbxproj:
  • assembler/MacroAssemblerARM64.h:

(JSC::MacroAssemblerARM64::storeZero16):

  • assembler/MacroAssemblerX86Common.h:

(JSC::MacroAssemblerX86Common::storeZero16):
(JSC::MacroAssemblerX86Common::store16):

  • bytecode/AccessCase.cpp:

(JSC::AccessCase::generateImpl):

  • bytecode/InlineAccess.cpp:

(JSC::InlineAccess::dumpCacheSizesAndCrash):
(JSC::linkCodeInline):
(JSC::InlineAccess::isCacheableStringLength):
(JSC::InlineAccess::generateStringLength):

  • bytecode/InlineAccess.h:

(JSC::InlineAccess::sizeForPropertyAccess):
(JSC::InlineAccess::sizeForPropertyReplace):
(JSC::InlineAccess::sizeForLengthAccess):

  • dfg/DFGOperations.cpp:
  • dfg/DFGOperations.h:
  • dfg/DFGSpeculativeJIT.cpp:

(JSC::DFG::SpeculativeJIT::compileStringSlice):
(JSC::DFG::SpeculativeJIT::compileToLowerCase):
(JSC::DFG::SpeculativeJIT::compileGetCharCodeAt):
(JSC::DFG::SpeculativeJIT::compileGetByValOnString):
(JSC::DFG::SpeculativeJIT::compileStringEquality):
(JSC::DFG::SpeculativeJIT::compileStringZeroLength):
(JSC::DFG::SpeculativeJIT::compileLogicalNotStringOrOther):
(JSC::DFG::SpeculativeJIT::emitStringBranch):
(JSC::DFG::SpeculativeJIT::emitStringOrOtherBranch):
(JSC::DFG::SpeculativeJIT::compileGetIndexedPropertyStorage):
(JSC::DFG::SpeculativeJIT::compileGetArrayLength):
(JSC::DFG::SpeculativeJIT::emitPopulateSliceIndex):
(JSC::DFG::SpeculativeJIT::compileArraySlice):
(JSC::DFG::SpeculativeJIT::compileArrayIndexOf):
(JSC::DFG::SpeculativeJIT::speculateStringIdentAndLoadStorage):
(JSC::DFG::SpeculativeJIT::emitSwitchCharStringJump):
(JSC::DFG::SpeculativeJIT::emitSwitchStringOnString):
(JSC::DFG::SpeculativeJIT::compileMakeRope): Deleted.

  • dfg/DFGSpeculativeJIT.h:
  • dfg/DFGSpeculativeJIT32_64.cpp:

(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileMakeRope):

  • dfg/DFGSpeculativeJIT64.cpp:

(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileMakeRope):

  • ftl/FTLAbstractHeapRepository.cpp:

(JSC::FTL::AbstractHeapRepository::AbstractHeapRepository):

  • ftl/FTLAbstractHeapRepository.h:
  • ftl/FTLLowerDFGToB3.cpp:

(JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::compileGetArrayLength):
(JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt):
(JSC::FTL::DFG::LowerDFGToB3::compileCompareStrictEq):
(JSC::FTL::DFG::LowerDFGToB3::compileStringToUntypedStrictEquality):
(JSC::FTL::DFG::LowerDFGToB3::compileSwitch):
(JSC::FTL::DFG::LowerDFGToB3::mapHashString):
(JSC::FTL::DFG::LowerDFGToB3::compileMapHash):
(JSC::FTL::DFG::LowerDFGToB3::compileHasOwnProperty):
(JSC::FTL::DFG::LowerDFGToB3::compileStringSlice):
(JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase):
(JSC::FTL::DFG::LowerDFGToB3::stringsEqual):
(JSC::FTL::DFG::LowerDFGToB3::boolify):
(JSC::FTL::DFG::LowerDFGToB3::switchString):
(JSC::FTL::DFG::LowerDFGToB3::isRopeString):
(JSC::FTL::DFG::LowerDFGToB3::isNotRopeString):
(JSC::FTL::DFG::LowerDFGToB3::speculateStringIdent):

  • jit/AssemblyHelpers.cpp:

(JSC::AssemblyHelpers::emitConvertValueToBoolean):
(JSC::AssemblyHelpers::branchIfValue):

  • jit/AssemblyHelpers.h:

(JSC::AssemblyHelpers::branchIfRopeStringImpl):
(JSC::AssemblyHelpers::branchIfNotRopeStringImpl):

  • jit/JITInlines.h:

(JSC::JIT::emitLoadCharacterString):

  • jit/Repatch.cpp:

(JSC::tryCacheGetByID):

  • jit/ThunkGenerators.cpp:

(JSC::stringGetByValGenerator):
(JSC::stringCharLoad):

  • llint/LowLevelInterpreter.asm:
  • llint/LowLevelInterpreter32_64.asm:
  • llint/LowLevelInterpreter64.asm:
  • runtime/JSString.cpp:

(JSC::JSString::createEmptyString):
(JSC::JSRopeString::RopeBuilder<RecordOverflow>::expand):
(JSC::JSString::dumpToStream):
(JSC::JSString::estimatedSize):
(JSC::JSString::visitChildren):
(JSC::JSRopeString::resolveRopeInternal8 const):
(JSC::JSRopeString::resolveRopeInternal8NoSubstring const):
(JSC::JSRopeString::resolveRopeInternal16 const):
(JSC::JSRopeString::resolveRopeInternal16NoSubstring const):
(JSC::JSRopeString::resolveRopeToAtomicString const):
(JSC::JSRopeString::convertToNonRope const):
(JSC::JSRopeString::resolveRopeToExistingAtomicString const):
(JSC::JSRopeString::resolveRopeWithFunction const):
(JSC::JSRopeString::resolveRope const):
(JSC::JSRopeString::resolveRopeSlowCase8 const):
(JSC::JSRopeString::resolveRopeSlowCase const):
(JSC::JSRopeString::outOfMemory const):
(JSC::JSRopeString::visitFibers): Deleted.
(JSC::JSRopeString::clearFibers const): Deleted.

  • runtime/JSString.h:

(JSC::JSString::uninitializedValueInternal const):
(JSC::JSString::valueInternal const):
(JSC::JSString::JSString):
(JSC::JSString::finishCreation):
(JSC::JSString::create):
(JSC::JSString::offsetOfValue):
(JSC::JSString::isRope const):
(JSC::JSString::is8Bit const):
(JSC::JSString::length const):
(JSC::JSString::tryGetValueImpl const):
(JSC::JSString::toAtomicString const):
(JSC::JSString::toExistingAtomicString const):
(JSC::JSString::value const):
(JSC::JSString::tryGetValue const):
(JSC::JSRopeString::unsafeView const):
(JSC::JSRopeString::viewWithUnderlyingString const):
(JSC::JSString::unsafeView const):
(JSC::JSString::viewWithUnderlyingString const):
(JSC::JSString::offsetOfLength): Deleted.
(JSC::JSString::offsetOfFlags): Deleted.
(JSC::JSString::setIs8Bit const): Deleted.
(JSC::JSString::setLength): Deleted.
(JSC::JSString::string): Deleted.
(JSC::jsStringBuilder): Deleted.

  • runtime/JSStringInlines.h:

(JSC::JSString::~JSString):
(JSC::JSString::equal const):

  • runtime/ObjectPrototype.cpp:

(JSC::objectProtoFuncToString):

  • runtime/RegExpMatchesArray.h:

(JSC::createRegExpMatchesArray):

  • runtime/RegExpObjectInlines.h:

(JSC::collectMatches):

  • runtime/RegExpPrototype.cpp:

(JSC::regExpProtoFuncSplitFast):

  • runtime/SmallStrings.cpp:

(JSC::SmallStrings::initializeCommonStrings):
(JSC::SmallStrings::createEmptyString): Deleted.

  • runtime/SmallStrings.h:
  • runtime/StringPrototype.cpp:

(JSC::stringProtoFuncSlice):

  • runtime/StringPrototypeInlines.h: Added.

(JSC::stringSlice):

Source/WTF:

  • wtf/text/StringImpl.h:

(WTF::StringImpl::flagIs8Bit):
(WTF::StringImpl::flagIsAtomic):
(WTF::StringImpl::flagIsSymbol):
(WTF::StringImpl::maskStringKind):

  • wtf/text/WTFString.cpp:

(WTF::nullString):

  • wtf/text/WTFString.h:
Location:
trunk
Files:
2 added
40 edited

Legend:

Unmodified
Added
Removed

  • trunk/JSTests/ChangeLog

    r242213 r242252  
     12019-02-28  Yusuke Suzuki  <ysuzuki@apple.com>
     2
     3        [JSC] sizeof(JSString) should be 16
     4        https://bugs.webkit.org/show_bug.cgi?id=194375
     5
     6        Reviewed by Saam Barati.
     7
     8        * microbenchmarks/make-rope.js: Added.
     9        (makeRope):
     10        * stress/to-lower-case-intrinsic-on-empty-rope.js: We no longer allow 0 length JSString except for jsEmptyString singleton per VM.
     11        (returnRope.helper): Deleted.
     12        (returnRope): Deleted.
     13
    1142019-02-28  Yusuke Suzuki  <ysuzuki@apple.com>
    215

  • trunk/JSTests/stress/to-lower-case-intrinsic-on-empty-rope.js

    r207377 r242252  
    44}
    55
    6 function returnRope() {
     6function returnEmptyString() {
    77    function helper(r, s) {
    88        // I'm paranoid about RegExp matching constant folding.
    … …  
    1212    return helper(/(b*)fo/, "fo");
    1313}
    14 noInline(returnRope);
     14noInline(returnEmptyString);
    1515
    1616function lower(a) {
    … …  
    2020
    2121for (let i = 0; i < 10000; i++) {
    22     let rope = returnRope();
    23     assert(!rope.length);
    24     assert(isRope(rope)); // Keep this test future proofed. If this stops returning a rope, we should try to find another way to return an empty rope.
    25     lower(rope);
     22    let notRope = returnEmptyString();
     23    assert(!notRope.length);
     24    assert(!isRope(notRope));
     25    lower(notRope);
    2626}

  • trunk/Source/JavaScriptCore/ChangeLog

    r242247 r242252  
     12019-02-28  Yusuke Suzuki  <ysuzuki@apple.com>
     2
     3        [JSC] sizeof(JSString) should be 16
     4        https://bugs.webkit.org/show_bug.cgi?id=194375
     5
     6        Reviewed by Saam Barati.
     7
     8        This patch reduces sizeof(JSString) from 24 to 16 to fit it into GC heap cell atom. And it also reduces sizeof(JSRopeString) from 48 to 32.
     9        Both classes cut 16 bytes per instance in GC allocation. This new layout is used in 64bit architectures which has little endianess.
     10
     11        JSString no longer has length and flags directly. JSString has String, and we query information to this String instead of holding duplicate
     12        information in JSString. We embed isRope bit into this String's pointer so that we can convert JSRopeString to JSString in an atomic manner.
     13        We emit store-store fence before we put String pointer. This should exist even before this patch, so this patch also fixes one concurrency issue.
     14
     15        The old JSRopeString separately had JSString* fibers along with String. In this patch, we merge the first JSString* fiber and String pointer
     16        storage into one to reduce the size of JSRopeString. JSRopeString has three pointer width storage. We pick 48bit effective address of JSString*
     17        fibers to compress three fibers + length + flags into three pointer width storage.
     18
     19        In 64bit architecture, JSString and JSRopeString have the following memory layout to make sizeof(JSString) == 16 and sizeof(JSRopeString) == 32.
     20        JSString has only one pointer. We use it for String. length() and is8Bit() queries go to StringImpl. In JSRopeString, we reuse the above pointer
     21        place for the 1st fiber. JSRopeString has three fibers so its size is 48. To keep length and is8Bit flag information in JSRopeString, JSRopeString
     22        encodes these information into the fiber pointers. is8Bit flag is encoded in the 1st fiber pointer. length is embedded directly, and two fibers
     23        are compressed into 12bytes. isRope information is encoded in the first fiber's LSB.
     24
     25        Since length of JSRopeString should be frequently accessed compared to each fiber, we put length in contiguous 32byte field, and compress 2nd
     26        and 3rd fibers into the following 80byte fields. One problem is that now 2nd and 3rd fibers are split. Storing and loading 2nd and 3rd fibers
     27        are not one pointer load operation. To make concurrent collector work correctly, we must initialize 2nd and 3rd fibers at JSRopeString creation
     28        and we must not modify these part later.
     29
     30                     0                        8        10               16                       32                                     48
     31        JSString     [   ID      ][  header  ][   String pointer      0]
     32        JSRopeString [   ID      ][  header  ][ flags ][ 1st fiber    1][  length  ][2nd lower32][2nd upper16][3rd lower16][3rd upper32]
     33                                                                      ^
     34                                                                   isRope bit
     35
     36        Since fibers in JSRopeString are not initialized in atomic pointer store manner, we must initialize all the fiber fields at JSRopeString creation.
     37        To achieve this, we modify our JSRopeString::RopeBuilder implementation not to create half-baked JSRopeString.
     38
     39        This patch also makes an empty JSString singleton per VM. This makes evaluation of JSString in boolean context one pointer comparison. This is
     40        critical in this change since this patch enlarges the code necessary to get length from JSString in JIT. Without this guarantee, our code of boolean
     41        context evaluation is bloated. This patch hides all the JSString::create and JSRopeString::create in the private permission. JSString and JSRopeString
     42        creation is only allowed from jsString and related helper functions and they return a singleton empty JSString if the length is zero. We also change
     43        JSRopeString::RopeBuilder not to construct an empty JSRopeString.
     44
     45        This patch is performance neutral in Speedometer2 and JetStream2. And it improves RAMification by 2.7%.
     46
     47        * JavaScriptCore.xcodeproj/project.pbxproj:
     48        * assembler/MacroAssemblerARM64.h:
     49        (JSC::MacroAssemblerARM64::storeZero16):
     50        * assembler/MacroAssemblerX86Common.h:
     51        (JSC::MacroAssemblerX86Common::storeZero16):
     52        (JSC::MacroAssemblerX86Common::store16):
     53        * bytecode/AccessCase.cpp:
     54        (JSC::AccessCase::generateImpl):
     55        * bytecode/InlineAccess.cpp:
     56        (JSC::InlineAccess::dumpCacheSizesAndCrash):
     57        (JSC::linkCodeInline):
     58        (JSC::InlineAccess::isCacheableStringLength):
     59        (JSC::InlineAccess::generateStringLength):
     60        * bytecode/InlineAccess.h:
     61        (JSC::InlineAccess::sizeForPropertyAccess):
     62        (JSC::InlineAccess::sizeForPropertyReplace):
     63        (JSC::InlineAccess::sizeForLengthAccess):
     64        * dfg/DFGOperations.cpp:
     65        * dfg/DFGOperations.h:
     66        * dfg/DFGSpeculativeJIT.cpp:
     67        (JSC::DFG::SpeculativeJIT::compileStringSlice):
     68        (JSC::DFG::SpeculativeJIT::compileToLowerCase):
     69        (JSC::DFG::SpeculativeJIT::compileGetCharCodeAt):
     70        (JSC::DFG::SpeculativeJIT::compileGetByValOnString):
     71        (JSC::DFG::SpeculativeJIT::compileStringEquality):
     72        (JSC::DFG::SpeculativeJIT::compileStringZeroLength):
     73        (JSC::DFG::SpeculativeJIT::compileLogicalNotStringOrOther):
     74        (JSC::DFG::SpeculativeJIT::emitStringBranch):
     75        (JSC::DFG::SpeculativeJIT::emitStringOrOtherBranch):
     76        (JSC::DFG::SpeculativeJIT::compileGetIndexedPropertyStorage):
     77        (JSC::DFG::SpeculativeJIT::compileGetArrayLength):
     78        (JSC::DFG::SpeculativeJIT::emitPopulateSliceIndex):
     79        (JSC::DFG::SpeculativeJIT::compileArraySlice):
     80        (JSC::DFG::SpeculativeJIT::compileArrayIndexOf):
     81        (JSC::DFG::SpeculativeJIT::speculateStringIdentAndLoadStorage):
     82        (JSC::DFG::SpeculativeJIT::emitSwitchCharStringJump):
     83        (JSC::DFG::SpeculativeJIT::emitSwitchStringOnString):
     84        (JSC::DFG::SpeculativeJIT::compileMakeRope): Deleted.
     85        * dfg/DFGSpeculativeJIT.h:
     86        * dfg/DFGSpeculativeJIT32_64.cpp:
     87        (JSC::DFG::SpeculativeJIT::compile):
     88        (JSC::DFG::SpeculativeJIT::compileMakeRope):
     89        * dfg/DFGSpeculativeJIT64.cpp:
     90        (JSC::DFG::SpeculativeJIT::compile):
     91        (JSC::DFG::SpeculativeJIT::compileMakeRope):
     92        * ftl/FTLAbstractHeapRepository.cpp:
     93        (JSC::FTL::AbstractHeapRepository::AbstractHeapRepository):
     94        * ftl/FTLAbstractHeapRepository.h:
     95        * ftl/FTLLowerDFGToB3.cpp:
     96        (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
     97        (JSC::FTL::DFG::LowerDFGToB3::compileGetArrayLength):
     98        (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
     99        (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt):
     100        (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt):
     101        (JSC::FTL::DFG::LowerDFGToB3::compileCompareStrictEq):
     102        (JSC::FTL::DFG::LowerDFGToB3::compileStringToUntypedStrictEquality):
     103        (JSC::FTL::DFG::LowerDFGToB3::compileSwitch):
     104        (JSC::FTL::DFG::LowerDFGToB3::mapHashString):
     105        (JSC::FTL::DFG::LowerDFGToB3::compileMapHash):
     106        (JSC::FTL::DFG::LowerDFGToB3::compileHasOwnProperty):
     107        (JSC::FTL::DFG::LowerDFGToB3::compileStringSlice):
     108        (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase):
     109        (JSC::FTL::DFG::LowerDFGToB3::stringsEqual):
     110        (JSC::FTL::DFG::LowerDFGToB3::boolify):
     111        (JSC::FTL::DFG::LowerDFGToB3::switchString):
     112        (JSC::FTL::DFG::LowerDFGToB3::isRopeString):
     113        (JSC::FTL::DFG::LowerDFGToB3::isNotRopeString):
     114        (JSC::FTL::DFG::LowerDFGToB3::speculateStringIdent):
     115        * jit/AssemblyHelpers.cpp:
     116        (JSC::AssemblyHelpers::emitConvertValueToBoolean):
     117        (JSC::AssemblyHelpers::branchIfValue):
     118        * jit/AssemblyHelpers.h:
     119        (JSC::AssemblyHelpers::branchIfRopeStringImpl):
     120        (JSC::AssemblyHelpers::branchIfNotRopeStringImpl):
     121        * jit/JITInlines.h:
     122        (JSC::JIT::emitLoadCharacterString):
     123        * jit/Repatch.cpp:
     124        (JSC::tryCacheGetByID):
     125        * jit/ThunkGenerators.cpp:
     126        (JSC::stringGetByValGenerator):
     127        (JSC::stringCharLoad):
     128        * llint/LowLevelInterpreter.asm:
     129        * llint/LowLevelInterpreter32_64.asm:
     130        * llint/LowLevelInterpreter64.asm:
     131        * runtime/JSString.cpp:
     132        (JSC::JSString::createEmptyString):
     133        (JSC::JSRopeString::RopeBuilder<RecordOverflow>::expand):
     134        (JSC::JSString::dumpToStream):
     135        (JSC::JSString::estimatedSize):
     136        (JSC::JSString::visitChildren):
     137        (JSC::JSRopeString::resolveRopeInternal8 const):
     138        (JSC::JSRopeString::resolveRopeInternal8NoSubstring const):
     139        (JSC::JSRopeString::resolveRopeInternal16 const):
     140        (JSC::JSRopeString::resolveRopeInternal16NoSubstring const):
     141        (JSC::JSRopeString::resolveRopeToAtomicString const):
     142        (JSC::JSRopeString::convertToNonRope const):
     143        (JSC::JSRopeString::resolveRopeToExistingAtomicString const):
     144        (JSC::JSRopeString::resolveRopeWithFunction const):
     145        (JSC::JSRopeString::resolveRope const):
     146        (JSC::JSRopeString::resolveRopeSlowCase8 const):
     147        (JSC::JSRopeString::resolveRopeSlowCase const):
     148        (JSC::JSRopeString::outOfMemory const):
     149        (JSC::JSRopeString::visitFibers): Deleted.
     150        (JSC::JSRopeString::clearFibers const): Deleted.
     151        * runtime/JSString.h:
     152        (JSC::JSString::uninitializedValueInternal const):
     153        (JSC::JSString::valueInternal const):
     154        (JSC::JSString::JSString):
     155        (JSC::JSString::finishCreation):
     156        (JSC::JSString::create):
     157        (JSC::JSString::offsetOfValue):
     158        (JSC::JSString::isRope const):
     159        (JSC::JSString::is8Bit const):
     160        (JSC::JSString::length const):
     161        (JSC::JSString::tryGetValueImpl const):
     162        (JSC::JSString::toAtomicString const):
     163        (JSC::JSString::toExistingAtomicString const):
     164        (JSC::JSString::value const):
     165        (JSC::JSString::tryGetValue const):
     166        (JSC::JSRopeString::unsafeView const):
     167        (JSC::JSRopeString::viewWithUnderlyingString const):
     168        (JSC::JSString::unsafeView const):
     169        (JSC::JSString::viewWithUnderlyingString const):
     170        (JSC::JSString::offsetOfLength): Deleted.
     171        (JSC::JSString::offsetOfFlags): Deleted.
     172        (JSC::JSString::setIs8Bit const): Deleted.
     173        (JSC::JSString::setLength): Deleted.
     174        (JSC::JSString::string): Deleted.
     175        (JSC::jsStringBuilder): Deleted.
     176        * runtime/JSStringInlines.h:
     177        (JSC::JSString::~JSString):
     178        (JSC::JSString::equal const):
     179        * runtime/ObjectPrototype.cpp:
     180        (JSC::objectProtoFuncToString):
     181        * runtime/RegExpMatchesArray.h:
     182        (JSC::createRegExpMatchesArray):
     183        * runtime/RegExpObjectInlines.h:
     184        (JSC::collectMatches):
     185        * runtime/RegExpPrototype.cpp:
     186        (JSC::regExpProtoFuncSplitFast):
     187        * runtime/SmallStrings.cpp:
     188        (JSC::SmallStrings::initializeCommonStrings):
     189        (JSC::SmallStrings::createEmptyString): Deleted.
     190        * runtime/SmallStrings.h:
     191        * runtime/StringPrototype.cpp:
     192        (JSC::stringProtoFuncSlice):
     193        * runtime/StringPrototypeInlines.h: Added.
     194        (JSC::stringSlice):
     195
    11962019-02-28  Saam barati  <sbarati@apple.com>
    2197

  • trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj

    r242123 r242252  
    17401740                E322E5A71DA644A8006E7709 /* FTLSnippetParams.h in Headers */ = {isa = PBXBuildFile; fileRef = E322E5A51DA644A4006E7709 /* FTLSnippetParams.h */; };
    17411741                E325956421FDA2C9008EDC9C /* RegExpGlobalDataInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = E325956321FDA2C8008EDC9C /* RegExpGlobalDataInlines.h */; };
     1742                E325A36022211590007349A1 /* StringPrototypeInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = E325A35F2221158A007349A1 /* StringPrototypeInlines.h */; };
    17421743                E3282BBB1FE930AF00EDAF71 /* YarrErrorCode.h in Headers */ = {isa = PBXBuildFile; fileRef = E3282BBA1FE930A400EDAF71 /* YarrErrorCode.h */; settings = {ATTRIBUTES = (Private, ); }; };
    17431744                E328C6C71DA4304500D255FD /* MaxFrameExtentForSlowPathCall.h in Headers */ = {isa = PBXBuildFile; fileRef = 65860177185A8F5E00030EEE /* MaxFrameExtentForSlowPathCall.h */; settings = {ATTRIBUTES = (Private, ); }; };
    … …  
    46704671                E322E5A51DA644A4006E7709 /* FTLSnippetParams.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = FTLSnippetParams.h; path = ftl/FTLSnippetParams.h; sourceTree = "<group>"; };
    46714672                E325956321FDA2C8008EDC9C /* RegExpGlobalDataInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = RegExpGlobalDataInlines.h; sourceTree = "<group>"; };
     4673                E325A35F2221158A007349A1 /* StringPrototypeInlines.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = StringPrototypeInlines.h; sourceTree = "<group>"; };
    46724674                E326C4961ECBEF5700A9A905 /* ClassInfo.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ClassInfo.cpp; sourceTree = "<group>"; };
    46734675                E3282BB91FE930A300EDAF71 /* YarrErrorCode.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = YarrErrorCode.cpp; path = yarr/YarrErrorCode.cpp; sourceTree = "<group>"; };
    … …  
    71677169                                BC18C3C50E16EE3300B34460 /* StringPrototype.cpp */,
    71687170                                BC18C3C60E16EE3300B34460 /* StringPrototype.h */,
     7171                                E325A35F2221158A007349A1 /* StringPrototypeInlines.h */,
    71697172                                93345A8712D838C400302BE3 /* StringRecursionChecker.cpp */,
    71707173                                93345A8812D838C400302BE3 /* StringRecursionChecker.h */,
    … …  
    96989701                                BC18C4680E16F5CD00B34460 /* StringObject.h in Headers */,
    96999702                                BC18C46A0E16F5CD00B34460 /* StringPrototype.h in Headers */,
     9703                                E325A36022211590007349A1 /* StringPrototypeInlines.h in Headers */,
    97009704                                142E313B134FF0A600AFADB5 /* Strong.h in Headers */,
    97019705                                145722861437E140005FDE26 /* StrongInlines.h in Headers */,

  • trunk/Source/JavaScriptCore/assembler/MacroAssemblerARM64.h

    r239427 r242252  
    15411541    }
    15421542
     1543    void storeZero16(ImplicitAddress address)
     1544    {
     1545        store16(ARM64Registers::zr, address);
     1546    }
     1547
     1548    void storeZero16(BaseIndex address)
     1549    {
     1550        store16(ARM64Registers::zr, address);
     1551    }
     1552
    15431553    void store8(RegisterID src, BaseIndex address)
    15441554    {

  • trunk/Source/JavaScriptCore/assembler/MacroAssemblerX86Common.h

    r239427 r242252  
    13491349    }
    13501350
     1351    void storeZero16(ImplicitAddress address)
     1352    {
     1353        store16(TrustedImm32(0), address);
     1354    }
     1355
     1356    void storeZero16(BaseIndex address)
     1357    {
     1358        store16(TrustedImm32(0), address);
     1359    }
     1360
    13511361    void store8(TrustedImm32 imm, Address address)
    13521362    {
    … …  
    14351445    }
    14361446
    1437     void store16(TrustedImm32 imm, Address address)
     1447    void store16(TrustedImm32 imm, ImplicitAddress address)
    14381448    {
    14391449        m_assembler.movw_im(static_cast<int16_t>(imm.m_value), address.offset, address.base);

  • trunk/Source/JavaScriptCore/bytecode/AccessCase.cpp

    r242123 r242252  
    12141214       
    12151215    case StringLength: {
    1216         jit.load32(CCallHelpers::Address(baseGPR, JSString::offsetOfLength()), valueRegs.payloadGPR());
    1217         jit.boxInt32(valueRegs.payloadGPR(), valueRegs);
     1216        jit.loadPtr(CCallHelpers::Address(baseGPR, JSString::offsetOfValue()), scratchGPR);
     1217        auto isRope = jit.branchIfRopeStringImpl(scratchGPR);
     1218        jit.load32(CCallHelpers::Address(scratchGPR, StringImpl::lengthMemoryOffset()), scratchGPR);
     1219        auto done = jit.jump();
     1220
     1221        isRope.link(&jit);
     1222        jit.load32(CCallHelpers::Address(baseGPR, JSRopeString::offsetOfLength()), scratchGPR);
     1223
     1224        done.link(&jit);
     1225        jit.boxInt32(scratchGPR, valueRegs);
    12181226        state.succeed();
    12191227        return;

  • trunk/Source/JavaScriptCore/bytecode/InlineAccess.cpp

    r236584 r242252  
    5151        CCallHelpers jit;
    5252
     53        GPRReg scratchGPR = value;
    5354        jit.patchableBranch8(
    5455            CCallHelpers::NotEqual,
    5556            CCallHelpers::Address(base, JSCell::typeInfoTypeOffset()),
    5657            CCallHelpers::TrustedImm32(StringType));
    57         jit.load32(CCallHelpers::Address(base, JSString::offsetOfLength()), regs.payloadGPR());
     58
     59        jit.loadPtr(CCallHelpers::Address(base, JSString::offsetOfValue()), scratchGPR);
     60        auto isRope = jit.branchIfRopeStringImpl(scratchGPR);
     61        jit.load32(CCallHelpers::Address(scratchGPR, StringImpl::lengthMemoryOffset()), regs.payloadGPR());
     62        auto done = jit.jump();
     63
     64        isRope.link(&jit);
     65        jit.load32(CCallHelpers::Address(base, JSRopeString::offsetOfLength()), regs.payloadGPR());
     66
     67        done.link(&jit);
    5868        jit.boxInt32(regs.payloadGPR(), regs);
    5969
    … …  
    144154{
    145155    if (jit.m_assembler.buffer().codeSize() <= stubInfo.patch.inlineSize()) {
    146         bool needsBranchCompaction = false;
     156        bool needsBranchCompaction = true;
    147157        LinkBuffer linkBuffer(jit, stubInfo.patch.start, stubInfo.patch.inlineSize(), JITCompilationMustSucceed, needsBranchCompaction);
    148158        ASSERT(linkBuffer.isValid());
    … …  
    157167    // of randomness. It's helpful to flip this on when running tests or browsing
    158168    // the web just to see how often it fails. You don't want an IC size that always fails.
    159     const bool failIfCantInline = false;
     169    constexpr bool failIfCantInline = false;
    160170    if (failIfCantInline) {
    161171        dataLog("Failure for: ", name, "\n");
    … …  
    289299}
    290300
     301bool InlineAccess::isCacheableStringLength(StructureStubInfo& stubInfo)
     302{
     303    return hasFreeRegister(stubInfo);
     304}
     305
    291306bool InlineAccess::generateStringLength(StructureStubInfo& stubInfo)
    292307{
    293     CCallHelpers jit;
    294 
    295     GPRReg base = stubInfo.baseGPR();
    296     JSValueRegs value = stubInfo.valueRegs();
     308    ASSERT(isCacheableStringLength(stubInfo));
     309
     310    CCallHelpers jit;
     311
     312    GPRReg base = stubInfo.baseGPR();
     313    JSValueRegs value = stubInfo.valueRegs();
     314    GPRReg scratch = getScratchRegister(stubInfo);
    297315
    298316    auto branchToSlowPath = jit.patchableBranch8(
    … …  
    300318        CCallHelpers::Address(base, JSCell::typeInfoTypeOffset()),
    301319        CCallHelpers::TrustedImm32(StringType));
    302     jit.load32(CCallHelpers::Address(base, JSString::offsetOfLength()), value.payloadGPR());
     320
     321    jit.loadPtr(CCallHelpers::Address(base, JSString::offsetOfValue()), scratch);
     322    auto isRope = jit.branchIfRopeStringImpl(scratch);
     323    jit.load32(CCallHelpers::Address(scratch, StringImpl::lengthMemoryOffset()), value.payloadGPR());
     324    auto done = jit.jump();
     325
     326    isRope.link(&jit);
     327    jit.load32(CCallHelpers::Address(base, JSRopeString::offsetOfLength()), value.payloadGPR());
     328
     329    done.link(&jit);
    303330    jit.boxInt32(value.payloadGPR(), value);
    304331

  • trunk/Source/JavaScriptCore/bytecode/InlineAccess.h

    r238439 r242252  
    4545    {
    4646#if CPU(X86_64)
    47         return 23;
     47        return 26;
    4848#elif CPU(X86)
    4949        return 27;
    … …  
    6363    {
    6464#if CPU(X86_64)
    65         return 23;
     65        return 26;
    6666#elif CPU(X86)
    6767        return 27;
    … …  
    8484    {
    8585#if CPU(X86_64)
    86         size_t size = 26;
     86        size_t size = 43;
    8787#elif CPU(X86)
    8888        size_t size = 27;
    8989#elif CPU(ARM64)
    90         size_t size = 32;
     90        size_t size = 44;
    9191#elif CPU(ARM_THUMB2)
    9292        size_t size = 30;
    … …  
    103103    static bool generateSelfPropertyReplace(StructureStubInfo&, Structure*, PropertyOffset);
    104104    static bool isCacheableArrayLength(StructureStubInfo&, JSArray*);
     105    static bool isCacheableStringLength(StructureStubInfo&);
    105106    static bool generateArrayLength(StructureStubInfo&, JSArray*);
    106107    static void rewireStubAsJump(StructureStubInfo&, CodeLocationLabel<JITStubRoutinePtrTag>);
    … …  
    112113    // its placeholder code here, and log the size. That way we
    113114    // can intelligently choose sizes on various platforms.
    114     NO_RETURN_DUE_TO_CRASH static void dumpCacheSizesAndCrash();
     115    JS_EXPORT_PRIVATE NO_RETURN_DUE_TO_CRASH static void dumpCacheSizesAndCrash();
    115116};
    116117

  • trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp

    r242192 r242252  
    7272#include "ScopedArguments.h"
    7373#include "StringConstructor.h"
     74#include "StringPrototypeInlines.h"
    7475#include "SuperSampler.h"
    7576#include "Symbol.h"
    … …  
    21562157    RETURN_IF_EXCEPTION(scope, nullptr);
    21572158    return jsSubstring(exec, string, from, span);
     2159}
     2160
     2161JSCell* JIT_OPERATION operationStringSlice(ExecState* exec, JSCell* cell, int32_t start, int32_t end)
     2162{
     2163    VM& vm = exec->vm();
     2164    NativeCallFrameTracer tracer(&vm, exec);
     2165    auto scope = DECLARE_THROW_SCOPE(vm);
     2166
     2167    auto string = jsCast<JSString*>(cell)->value(exec);
     2168    RETURN_IF_EXCEPTION(scope, nullptr);
     2169    static_assert(static_cast<uint64_t>(JSString::MaxLength) <= static_cast<uint64_t>(std::numeric_limits<int32_t>::max()), "");
     2170
     2171    scope.release();
     2172    return stringSlice(exec, WTFMove(string), start, end);
    21582173}
    21592174

  • trunk/Source/JavaScriptCore/dfg/DFGOperations.h

    r240327 r242252  
    204204
    205205JSCell* JIT_OPERATION operationStringSubstr(ExecState*, JSCell*, int32_t, int32_t);
     206JSCell* JIT_OPERATION operationStringSlice(ExecState*, JSCell*, int32_t, int32_t);
    206207JSString* JIT_OPERATION operationStringValueOf(ExecState*, EncodedJSValue);
    207208JSString* JIT_OPERATION operationToLowerCase(ExecState*, JSString*, uint32_t);

  • trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp

    r242123 r242252  
    15311531{
    15321532    SpeculateCellOperand string(this, node->child1());
     1533
     1534    GPRReg stringGPR = string.gpr();
     1535
     1536    speculateString(node->child1(), stringGPR);
     1537
     1538    SpeculateInt32Operand start(this, node->child2());
     1539    GPRReg startGPR = start.gpr();
     1540
     1541    Optional<SpeculateInt32Operand> end;
     1542    Optional<GPRReg> endGPR;
     1543    if (node->child3()) {
     1544        end.emplace(this, node->child3());
     1545        endGPR.emplace(end->gpr());
     1546    }
     1547
     1548    GPRTemporary temp(this);
     1549    GPRReg tempGPR = temp.gpr();
     1550
     1551    m_jit.loadPtr(CCallHelpers::Address(stringGPR, JSString::offsetOfValue()), tempGPR);
     1552    auto isRope = m_jit.branchIfRopeStringImpl(tempGPR);
     1553
     1554    GPRTemporary temp2(this);
    15331555    GPRTemporary startIndex(this);
    1534     GPRTemporary temp(this);
    1535     GPRTemporary temp2(this);
    1536 
    1537     GPRReg stringGPR = string.gpr();
     1556
     1557    GPRReg temp2GPR = temp2.gpr();
    15381558    GPRReg startIndexGPR = startIndex.gpr();
    1539     GPRReg tempGPR = temp.gpr();
    1540     GPRReg temp2GPR = temp2.gpr();
    1541 
    1542     speculateString(node->child1(), stringGPR);
    1543 
    15441559    {
    1545         m_jit.load32(JITCompiler::Address(stringGPR, JSString::offsetOfLength()), temp2GPR);
    1546 
    1547         emitPopulateSliceIndex(node->child2(), temp2GPR, startIndexGPR);
     1560        m_jit.load32(MacroAssembler::Address(tempGPR, StringImpl::lengthMemoryOffset()), temp2GPR);
     1561
     1562        emitPopulateSliceIndex(node->child2(), startGPR, temp2GPR, startIndexGPR);
     1563
    15481564        if (node->child3())
    1549             emitPopulateSliceIndex(node->child3(), temp2GPR, tempGPR);
     1565            emitPopulateSliceIndex(node->child3(), endGPR.value(), temp2GPR, tempGPR);
    15501566        else
    15511567            m_jit.move(temp2GPR, tempGPR);
    … …  
    15631579    slowCases.append(m_jit.branch32(MacroAssembler::NotEqual, tempGPR, TrustedImm32(1)));
    15641580
     1581    // Refill StringImpl* here.
    15651582    m_jit.loadPtr(MacroAssembler::Address(stringGPR, JSString::offsetOfValue()), temp2GPR);
    1566     slowCases.append(m_jit.branchTestPtr(MacroAssembler::Zero, temp2GPR));
    1567 
    15681583    m_jit.loadPtr(MacroAssembler::Address(temp2GPR, StringImpl::dataOffset()), tempGPR);
    15691584
    … …  
    15871602    m_jit.loadPtr(tempGPR, tempGPR);
    15881603
    1589     addSlowPathGenerator(
    1590         slowPathCall(
    1591             bigCharacter, this, operationSingleCharacterString, tempGPR, tempGPR));
    1592 
    1593     addSlowPathGenerator(
    1594         slowPathCall(
    1595             slowCases, this, operationStringSubstr, tempGPR, stringGPR, startIndexGPR, tempGPR));
     1604    addSlowPathGenerator(slowPathCall(bigCharacter, this, operationSingleCharacterString, tempGPR, tempGPR));
     1605
     1606    addSlowPathGenerator(slowPathCall(slowCases, this, operationStringSubstr, tempGPR, stringGPR, startIndexGPR, tempGPR));
     1607
     1608    if (endGPR)
     1609        addSlowPathGenerator(slowPathCall(isRope, this, operationStringSlice, tempGPR, stringGPR, startGPR, *endGPR));
     1610    else
     1611        addSlowPathGenerator(slowPathCall(isRope, this, operationStringSlice, tempGPR, stringGPR, startGPR, TrustedImm32(std::numeric_limits<int32_t>::max())));
    15961612
    15971613    doneCases.link(&m_jit);
    … …  
    16211637
    16221638    m_jit.loadPtr(MacroAssembler::Address(stringGPR, JSString::offsetOfValue()), tempGPR);
    1623     slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, tempGPR));
    1624 
     1639    slowPath.append(m_jit.branchIfRopeStringImpl(tempGPR));
    16251640    slowPath.append(m_jit.branchTest32(
    16261641        MacroAssembler::Zero, MacroAssembler::Address(tempGPR, StringImpl::flagsOffset()),
    … …  
    21292144    ASSERT(speculationChecked(m_state.forNode(node->child1()).m_type, SpecString));
    21302145
    2131     // unsigned comparison so we can filter out negative indices and indices that are too large
    2132     speculationCheck(Uncountable, JSValueRegs(), 0, m_jit.branch32(MacroAssembler::AboveOrEqual, indexReg, MacroAssembler::Address(stringReg, JSString::offsetOfLength())));
    2133 
    21342146    GPRTemporary scratch(this);
    21352147    GPRReg scratchReg = scratch.gpr();
    21362148
    21372149    m_jit.loadPtr(MacroAssembler::Address(stringReg, JSString::offsetOfValue()), scratchReg);
     2150   
     2151    // unsigned comparison so we can filter out negative indices and indices that are too large
     2152    speculationCheck(Uncountable, JSValueRegs(), 0, m_jit.branch32(MacroAssembler::AboveOrEqual, indexReg, CCallHelpers::Address(scratchReg, StringImpl::lengthMemoryOffset())));
    21382153
    21392154    // Load the character into scratchReg
    … …  
    21762191
    21772192    // unsigned comparison so we can filter out negative indices and indices that are too large
     2193    m_jit.loadPtr(MacroAssembler::Address(baseReg, JSString::offsetOfValue()), scratchReg);
    21782194    JITCompiler::Jump outOfBounds = m_jit.branch32(
    21792195        MacroAssembler::AboveOrEqual, propertyReg,
    2180         MacroAssembler::Address(baseReg, JSString::offsetOfLength()));
     2196        MacroAssembler::Address(scratchReg, StringImpl::lengthMemoryOffset()));
    21812197    if (node->arrayMode().isInBounds())
    21822198        speculationCheck(OutOfBounds, JSValueRegs(), 0, outOfBounds);
    2183 
    2184     m_jit.loadPtr(MacroAssembler::Address(baseReg, JSString::offsetOfValue()), scratchReg);
    21852199
    21862200    // Load the character into scratchReg
    … …  
    43294343        break;
    43304344    }
    4331 }
    4332 
    4333 void SpeculativeJIT::compileMakeRope(Node* node)
    4334 {
    4335     ASSERT(node->child1().useKind() == KnownStringUse);
    4336     ASSERT(node->child2().useKind() == KnownStringUse);
    4337     ASSERT(!node->child3() || node->child3().useKind() == KnownStringUse);
    4338    
    4339     SpeculateCellOperand op1(this, node->child1());
    4340     SpeculateCellOperand op2(this, node->child2());
    4341     SpeculateCellOperand op3(this, node->child3());
    4342     GPRTemporary result(this);
    4343     GPRTemporary allocator(this);
    4344     GPRTemporary scratch(this);
    4345    
    4346     GPRReg opGPRs[3];
    4347     unsigned numOpGPRs;
    4348     opGPRs[0] = op1.gpr();
    4349     opGPRs[1] = op2.gpr();
    4350     if (node->child3()) {
    4351         opGPRs[2] = op3.gpr();
    4352         numOpGPRs = 3;
    4353     } else {
    4354         opGPRs[2] = InvalidGPRReg;
    4355         numOpGPRs = 2;
    4356     }
    4357     GPRReg resultGPR = result.gpr();
    4358     GPRReg allocatorGPR = allocator.gpr();
    4359     GPRReg scratchGPR = scratch.gpr();
    4360    
    4361     JITCompiler::JumpList slowPath;
    4362     Allocator allocatorValue = allocatorForNonVirtualConcurrently<JSRopeString>(*m_jit.vm(), sizeof(JSRopeString), AllocatorForMode::AllocatorIfExists);
    4363     emitAllocateJSCell(resultGPR, JITAllocator::constant(allocatorValue), allocatorGPR, TrustedImmPtr(m_jit.graph().registerStructure(m_jit.vm()->stringStructure.get())), scratchGPR, slowPath);
    4364        
    4365     m_jit.storePtr(TrustedImmPtr(nullptr), JITCompiler::Address(resultGPR, JSString::offsetOfValue()));
    4366     for (unsigned i = 0; i < numOpGPRs; ++i)
    4367         m_jit.storePtr(opGPRs[i], JITCompiler::Address(resultGPR, JSRopeString::offsetOfFibers() + sizeof(WriteBarrier<JSString>) * i));
    4368     for (unsigned i = numOpGPRs; i < JSRopeString::s_maxInternalRopeLength; ++i)
    4369         m_jit.storePtr(TrustedImmPtr(nullptr), JITCompiler::Address(resultGPR, JSRopeString::offsetOfFibers() + sizeof(WriteBarrier<JSString>) * i));
    4370     m_jit.load16(JITCompiler::Address(opGPRs[0], JSString::offsetOfFlags()), scratchGPR);
    4371     m_jit.load32(JITCompiler::Address(opGPRs[0], JSString::offsetOfLength()), allocatorGPR);
    4372     if (!ASSERT_DISABLED) {
    4373         JITCompiler::Jump ok = m_jit.branch32(
    4374             JITCompiler::GreaterThanOrEqual, allocatorGPR, TrustedImm32(0));
    4375         m_jit.abortWithReason(DFGNegativeStringLength);
    4376         ok.link(&m_jit);
    4377     }
    4378     for (unsigned i = 1; i < numOpGPRs; ++i) {
    4379         m_jit.and16(JITCompiler::Address(opGPRs[i], JSString::offsetOfFlags()), scratchGPR);
    4380         speculationCheck(
    4381             Uncountable, JSValueSource(), nullptr,
    4382             m_jit.branchAdd32(
    4383                 JITCompiler::Overflow,
    4384                 JITCompiler::Address(opGPRs[i], JSString::offsetOfLength()), allocatorGPR));
    4385     }
    4386     m_jit.and32(JITCompiler::TrustedImm32(JSString::Is8Bit), scratchGPR);
    4387     m_jit.store16(scratchGPR, JITCompiler::Address(resultGPR, JSString::offsetOfFlags()));
    4388     if (!ASSERT_DISABLED) {
    4389         JITCompiler::Jump ok = m_jit.branch32(
    4390             JITCompiler::GreaterThanOrEqual, allocatorGPR, TrustedImm32(0));
    4391         m_jit.abortWithReason(DFGNegativeStringLength);
    4392         ok.link(&m_jit);
    4393     }
    4394     m_jit.store32(allocatorGPR, JITCompiler::Address(resultGPR, JSString::offsetOfLength()));
    4395    
    4396     m_jit.mutatorFence(*m_jit.vm());
    4397    
    4398     switch (numOpGPRs) {
    4399     case 2:
    4400         addSlowPathGenerator(slowPathCall(
    4401             slowPath, this, operationMakeRope2, resultGPR, opGPRs[0], opGPRs[1]));
    4402         break;
    4403     case 3:
    4404         addSlowPathGenerator(slowPathCall(
    4405             slowPath, this, operationMakeRope3, resultGPR, opGPRs[0], opGPRs[1], opGPRs[2]));
    4406         break;
    4407     default:
    4408         RELEASE_ASSERT_NOT_REACHED();
    4409         break;
    4410     }
    4411    
    4412     cellResult(resultGPR, node);
    44134345}
    44144346
    … …  
    63216253    falseCase.append(fastFalse);
    63226254
    6323     m_jit.load32(MacroAssembler::Address(leftGPR, JSString::offsetOfLength()), lengthGPR);
     6255    m_jit.loadPtr(MacroAssembler::Address(leftGPR, JSString::offsetOfValue()), leftTempGPR);
     6256    m_jit.loadPtr(MacroAssembler::Address(rightGPR, JSString::offsetOfValue()), rightTempGPR);
     6257
     6258    slowCase.append(m_jit.branchIfRopeStringImpl(leftTempGPR));
     6259    slowCase.append(m_jit.branchIfRopeStringImpl(rightTempGPR));
     6260
     6261    m_jit.load32(MacroAssembler::Address(leftTempGPR, StringImpl::lengthMemoryOffset()), lengthGPR);
    63246262   
    63256263    falseCase.append(m_jit.branch32(
    63266264        MacroAssembler::NotEqual,
    6327         MacroAssembler::Address(rightGPR, JSString::offsetOfLength()),
     6265        MacroAssembler::Address(rightTempGPR, StringImpl::lengthMemoryOffset()),
    63286266        lengthGPR));
    63296267   
    63306268    trueCase.append(m_jit.branchTest32(MacroAssembler::Zero, lengthGPR));
    6331    
    6332     m_jit.loadPtr(MacroAssembler::Address(leftGPR, JSString::offsetOfValue()), leftTempGPR);
    6333     m_jit.loadPtr(MacroAssembler::Address(rightGPR, JSString::offsetOfValue()), rightTempGPR);
    6334    
    6335     slowCase.append(m_jit.branchTestPtr(MacroAssembler::Zero, leftTempGPR));
    6336     slowCase.append(m_jit.branchTestPtr(MacroAssembler::Zero, rightTempGPR));
    63376269   
    63386270    slowCase.append(m_jit.branchTest32(
    … …  
    66396571    GPRReg eqGPR = eq.gpr();
    66406572
    6641     // Fetch the length field from the string object.
    6642     m_jit.test32(MacroAssembler::Zero, MacroAssembler::Address(strGPR, JSString::offsetOfLength()), MacroAssembler::TrustedImm32(-1), eqGPR);
    6643 
     6573    m_jit.move(TrustedImmPtr::weakPointer(m_jit.graph(), jsEmptyString(m_jit.vm())), eqGPR);
     6574    m_jit.comparePtr(CCallHelpers::Equal, strGPR, eqGPR, eqGPR);
    66446575    unblessedBooleanResult(eqGPR, node);
    66456576}
    … …  
    66566587    DFG_TYPE_CHECK(
    66576588        valueRegs, node->child1(), (~SpecCellCheck) | SpecString, m_jit.branchIfNotString(cellGPR));
    6658     m_jit.test32(
    6659         JITCompiler::Zero, JITCompiler::Address(cellGPR, JSString::offsetOfLength()),
    6660         JITCompiler::TrustedImm32(-1), tempGPR);
    6661     JITCompiler::Jump done = m_jit.jump();
     6589
     6590    m_jit.move(TrustedImmPtr::weakPointer(m_jit.graph(), jsEmptyString(m_jit.vm())), tempGPR);
     6591    m_jit.comparePtr(CCallHelpers::Equal, cellGPR, tempGPR, tempGPR);
     6592    auto done = m_jit.jump();
     6593
    66626594    notCell.link(&m_jit);
    66636595    DFG_TYPE_CHECK(
    66646596        valueRegs, node->child1(), SpecCellCheck | SpecOther, m_jit.branchIfNotOther(valueRegs, tempGPR));
    66656597    m_jit.move(TrustedImm32(1), tempGPR);
     6598
    66666599    done.link(&m_jit);
    6667 
    66686600    unblessedBooleanResult(tempGPR, node);
    66696601
    … …  
    66736605{
    66746606    SpeculateCellOperand str(this, nodeUse);
    6675     speculateString(nodeUse, str.gpr());
    6676     branchTest32(JITCompiler::NonZero, MacroAssembler::Address(str.gpr(), JSString::offsetOfLength()), taken);
    6677     jump(notTaken);
     6607
     6608    GPRReg strGPR = str.gpr();
     6609
     6610    speculateString(nodeUse, strGPR);
     6611
     6612    branchPtr(CCallHelpers::Equal, strGPR, TrustedImmPtr::weakPointer(m_jit.graph(), jsEmptyString(m_jit.vm())), notTaken);
     6613    jump(taken);
     6614
    66786615    noResult(m_currentNode);
    66796616}
    … …  
    66896626    GPRReg cellGPR = valueRegs.payloadGPR();
    66906627    DFG_TYPE_CHECK(valueRegs, nodeUse, (~SpecCellCheck) | SpecString, m_jit.branchIfNotString(cellGPR));
    6691     branchTest32(
    6692         JITCompiler::Zero, JITCompiler::Address(cellGPR, JSString::offsetOfLength()),
    6693         JITCompiler::TrustedImm32(-1), notTaken);
     6628
     6629    branchPtr(CCallHelpers::Equal, cellGPR, TrustedImmPtr::weakPointer(m_jit.graph(), jsEmptyString(m_jit.vm())), notTaken);
    66946630    jump(taken, ForceJump);
     6631
    66956632    notCell.link(&m_jit);
    66966633    DFG_TYPE_CHECK(
    … …  
    67416678        addSlowPathGenerator(
    67426679            slowPathCall(
    6743                 m_jit.branchTest32(MacroAssembler::Zero, storageReg),
     6680                m_jit.branchIfRopeStringImpl(storageReg),
    67446681                this, operationResolveRope, storageReg, baseReg));
    67456682
    … …  
    69936930        SpeculateCellOperand base(this, node->child1());
    69946931        GPRTemporary result(this, Reuse, base);
     6932        GPRTemporary temp(this);
    69956933        GPRReg baseGPR = base.gpr();
    69966934        GPRReg resultGPR = result.gpr();
    6997         m_jit.load32(MacroAssembler::Address(baseGPR, JSString::offsetOfLength()), resultGPR);
     6935        GPRReg tempGPR = temp.gpr();
     6936
     6937        m_jit.loadPtr(MacroAssembler::Address(baseGPR, JSString::offsetOfValue()), tempGPR);
     6938        auto isRope = m_jit.branchIfRopeStringImpl(tempGPR);
     6939        m_jit.load32(MacroAssembler::Address(tempGPR, StringImpl::lengthMemoryOffset()), resultGPR);
     6940        auto done = m_jit.jump();
     6941
     6942        isRope.link(&m_jit);
     6943        m_jit.load32(CCallHelpers::Address(baseGPR, JSRopeString::offsetOfLength()), resultGPR);
     6944
     6945        done.link(&m_jit);
    69986946        int32Result(resultGPR, node);
    69996947        break;
    … …  
    81408088}
    81418089
    8142 void SpeculativeJIT::emitPopulateSliceIndex(Edge& target, GPRReg length, GPRReg result)
     8090void SpeculativeJIT::emitPopulateSliceIndex(Edge& target, Optional<GPRReg> indexGPR, GPRReg lengthGPR, GPRReg resultGPR)
    81438091{
    81448092    if (target->isInt32Constant()) {
    81458093        int32_t value = target->asInt32();
    81468094        if (value == 0) {
    8147             m_jit.move(TrustedImm32(0), result);
     8095            m_jit.move(TrustedImm32(0), resultGPR);
    81488096            return;
    81498097        }
    … …  
    81518099        MacroAssembler::JumpList done;
    81528100        if (value > 0) {
    8153             m_jit.move(TrustedImm32(value), result);
    8154             done.append(m_jit.branch32(MacroAssembler::BelowOrEqual, result, length));
    8155             m_jit.move(length, result);
     8101            m_jit.move(TrustedImm32(value), resultGPR);
     8102            done.append(m_jit.branch32(MacroAssembler::BelowOrEqual, resultGPR, lengthGPR));
     8103            m_jit.move(lengthGPR, resultGPR);
    81568104        } else {
    81578105            ASSERT(value != 0);
    8158             m_jit.move(length, result);
    8159             done.append(m_jit.branchAdd32(MacroAssembler::PositiveOrZero, TrustedImm32(value), result));
    8160             m_jit.move(TrustedImm32(0), result);
     8106            m_jit.move(lengthGPR, resultGPR);
     8107            done.append(m_jit.branchAdd32(MacroAssembler::PositiveOrZero, TrustedImm32(value), resultGPR));
     8108            m_jit.move(TrustedImm32(0), resultGPR);
    81618109        }
    81628110        done.link(&m_jit);
    … …  
    81648112    }
    81658113
    8166     SpeculateInt32Operand index(this, target);
    8167     GPRReg indexGPR = index.gpr();
     8114    Optional<SpeculateInt32Operand> index;
     8115    if (!indexGPR) {
     8116        index.emplace(this, target);
     8117        indexGPR = index->gpr();
     8118    }
    81688119    MacroAssembler::JumpList done;
    81698120
    8170     auto isPositive = m_jit.branch32(MacroAssembler::GreaterThanOrEqual, indexGPR, TrustedImm32(0));
    8171     m_jit.move(length, result);
    8172     done.append(m_jit.branchAdd32(MacroAssembler::PositiveOrZero, indexGPR, result));
    8173     m_jit.move(TrustedImm32(0), result);
     8121    auto isPositive = m_jit.branch32(MacroAssembler::GreaterThanOrEqual, indexGPR.value(), TrustedImm32(0));
     8122    m_jit.move(lengthGPR, resultGPR);
     8123    done.append(m_jit.branchAdd32(MacroAssembler::PositiveOrZero, indexGPR.value(), resultGPR));
     8124    m_jit.move(TrustedImm32(0), resultGPR);
    81748125    done.append(m_jit.jump());
    81758126
    81768127    isPositive.link(&m_jit);
    8177     m_jit.move(indexGPR, result);
    8178     done.append(m_jit.branch32(MacroAssembler::BelowOrEqual, result, length));
    8179     m_jit.move(length, result);
     8128    m_jit.move(indexGPR.value(), resultGPR);
     8129    done.append(m_jit.branch32(MacroAssembler::BelowOrEqual, resultGPR, lengthGPR));
     8130    m_jit.move(lengthGPR, resultGPR);
    81808131
    81818132    done.link(&m_jit);
    … …  
    82058156
    82068157        if (node->numChildren() == 4)
    8207             emitPopulateSliceIndex(m_jit.graph().varArgChild(node, 2), lengthGPR, tempGPR);
     8158            emitPopulateSliceIndex(m_jit.graph().varArgChild(node, 2), WTF::nullopt, lengthGPR, tempGPR);
    82088159        else
    82098160            m_jit.move(lengthGPR, tempGPR);
    … …  
    82158166            GPRTemporary tempStartIndex(this);
    82168167            GPRReg startGPR = tempStartIndex.gpr();
    8217             emitPopulateSliceIndex(m_jit.graph().varArgChild(node, 1), lengthGPR, startGPR);
     8168            emitPopulateSliceIndex(m_jit.graph().varArgChild(node, 1), WTF::nullopt, lengthGPR, startGPR);
    82188169
    82198170            auto tooBig = m_jit.branch32(MacroAssembler::Above, startGPR, tempGPR);
    … …  
    83048255        m_jit.load32(MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()), tempValue);
    83058256        if (node->numChildren() == 4)
    8306             emitPopulateSliceIndex(m_jit.graph().varArgChild(node, 2), tempValue, tempGPR);
     8257            emitPopulateSliceIndex(m_jit.graph().varArgChild(node, 2), WTF::nullopt, tempValue, tempGPR);
    83078258        else
    83088259            m_jit.move(tempValue, tempGPR);
    8309         emitPopulateSliceIndex(m_jit.graph().varArgChild(node, 1), tempValue, loadIndex);
     8260        emitPopulateSliceIndex(m_jit.graph().varArgChild(node, 1), WTF::nullopt, tempValue, loadIndex);
    83108261    }
    83118262
    … …  
    83618312
    83628313    if (node->numChildren() == 4)
    8363         emitPopulateSliceIndex(m_jit.graph().varArgChild(node, 2), lengthGPR, indexGPR);
     8314        emitPopulateSliceIndex(m_jit.graph().varArgChild(node, 2), WTF::nullopt, lengthGPR, indexGPR);
    83648315    else
    83658316        m_jit.move(TrustedImm32(0), indexGPR);
    … …  
    1013810089    speculationCheck(
    1013910090        BadType, JSValueSource::unboxedCell(string), edge,
    10140         m_jit.branchTestPtr(MacroAssembler::Zero, storage));
     10091        m_jit.branchIfRopeStringImpl(storage));
    1014110092    speculationCheck(
    1014210093        BadType, JSValueSource::unboxedCell(string), edge, m_jit.branchTest32(
    … …  
    1052910480    SwitchData* data, GPRReg value, GPRReg scratch)
    1053010481{
     10482    m_jit.loadPtr(MacroAssembler::Address(value, JSString::offsetOfValue()), scratch);
     10483    auto isRope = m_jit.branchIfRopeStringImpl(scratch);
     10484
    1053110485    addBranch(
    1053210486        m_jit.branch32(
    1053310487            MacroAssembler::NotEqual,
    10534             MacroAssembler::Address(value, JSString::offsetOfLength()),
     10488            MacroAssembler::Address(scratch, StringImpl::lengthMemoryOffset()),
    1053510489            TrustedImm32(1)),
    1053610490        data->fallThrough.block);
    1053710491   
    10538     m_jit.loadPtr(MacroAssembler::Address(value, JSString::offsetOfValue()), scratch);
    10539    
    10540     addSlowPathGenerator(
    10541         slowPathCall(
    10542             m_jit.branchTestPtr(MacroAssembler::Zero, scratch),
    10543             this, operationResolveRope, scratch, value));
     10492    addSlowPathGenerator(slowPathCall(isRope, this, operationResolveRope, scratch, value));
    1054410493   
    1054510494    m_jit.loadPtr(MacroAssembler::Address(scratch, StringImpl::dataOffset()), value);
    … …  
    1078410733    GPRReg tempGPR = temp.gpr();
    1078510734   
    10786     m_jit.load32(MacroAssembler::Address(string, JSString::offsetOfLength()), lengthGPR);
     10735    MacroAssembler::JumpList slowCases;
    1078710736    m_jit.loadPtr(MacroAssembler::Address(string, JSString::offsetOfValue()), tempGPR);
    10788    
    10789     MacroAssembler::JumpList slowCases;
    10790     slowCases.append(m_jit.branchTestPtr(MacroAssembler::Zero, tempGPR));
     10737    slowCases.append(m_jit.branchIfRopeStringImpl(tempGPR));
     10738    m_jit.load32(MacroAssembler::Address(tempGPR, StringImpl::lengthMemoryOffset()), lengthGPR);
     10739   
    1079110740    slowCases.append(m_jit.branchTest32(
    1079210741        MacroAssembler::Zero,

  • trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h

    r242123 r242252  
    15431543    void emitGetCallee(CodeOrigin, GPRReg calleeGPR);
    15441544    void emitGetArgumentStart(CodeOrigin, GPRReg startGPR);
    1545     void emitPopulateSliceIndex(Edge&, GPRReg length, GPRReg result);
     1545    void emitPopulateSliceIndex(Edge&, Optional<GPRReg> indexGPR, GPRReg lengthGPR, GPRReg resultGPR);
    15461546   
    15471547    // Generate an OSR exit fuzz check. Returns Jump() if OSR exit fuzz is not enabled, or if

  • trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp

    r240327 r242252  
    38813881            speculateString(node->child2(), keyRegs.payloadGPR());
    38823882            m_jit.loadPtr(MacroAssembler::Address(keyRegs.payloadGPR(), JSString::offsetOfValue()), implGPR);
    3883             slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, implGPR));
     3883            slowPath.append(m_jit.branchIfRopeStringImpl(implGPR));
    38843884            slowPath.append(m_jit.branchTest32(
    38853885                MacroAssembler::Zero, MacroAssembler::Address(implGPR, StringImpl::flagsOffset()),
    … …  
    38913891            auto isNotString = m_jit.branchIfNotString(keyRegs.payloadGPR());
    38923892            m_jit.loadPtr(MacroAssembler::Address(keyRegs.payloadGPR(), JSString::offsetOfValue()), implGPR);
    3893             slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, implGPR));
     3893            slowPath.append(m_jit.branchIfRopeStringImpl(implGPR));
    38943894            slowPath.append(m_jit.branchTest32(
    38953895                MacroAssembler::Zero, MacroAssembler::Address(implGPR, StringImpl::flagsOffset()),
    … …  
    41964196}
    41974197
     4198void SpeculativeJIT::compileMakeRope(Node* node)
     4199{
     4200    ASSERT(node->child1().useKind() == KnownStringUse);
     4201    ASSERT(node->child2().useKind() == KnownStringUse);
     4202    ASSERT(!node->child3() || node->child3().useKind() == KnownStringUse);
     4203   
     4204    SpeculateCellOperand op1(this, node->child1());
     4205    SpeculateCellOperand op2(this, node->child2());
     4206    SpeculateCellOperand op3(this, node->child3());
     4207   
     4208    GPRReg opGPRs[3];
     4209    unsigned numOpGPRs;
     4210    opGPRs[0] = op1.gpr();
     4211    opGPRs[1] = op2.gpr();
     4212    if (node->child3()) {
     4213        opGPRs[2] = op3.gpr();
     4214        numOpGPRs = 3;
     4215    } else {
     4216        opGPRs[2] = InvalidGPRReg;
     4217        numOpGPRs = 2;
     4218    }
     4219
     4220    flushRegisters();
     4221    GPRFlushedCallResult result(this);
     4222    GPRReg resultGPR = result.gpr();
     4223    switch (numOpGPRs) {
     4224    case 2:
     4225        callOperation(operationMakeRope2, resultGPR, opGPRs[0], opGPRs[1]);
     4226        m_jit.exceptionCheck();
     4227        break;
     4228    case 3:
     4229        callOperation(operationMakeRope3, resultGPR, opGPRs[0], opGPRs[1], opGPRs[2]);
     4230        m_jit.exceptionCheck();
     4231        break;
     4232    default:
     4233        RELEASE_ASSERT_NOT_REACHED();
     4234        break;
     4235    }
     4236   
     4237    cellResult(resultGPR, node);
     4238}
     4239
    41984240#endif
    41994241

  • trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp

    r242015 r242252  
    40394039
    40404040            m_jit.loadPtr(MacroAssembler::Address(inputGPR, JSString::offsetOfValue()), resultGPR);
    4041             slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, resultGPR));
     4041            slowPath.append(m_jit.branchIfRopeStringImpl(resultGPR));
    40424042            m_jit.load32(MacroAssembler::Address(resultGPR, StringImpl::flagsOffset()), resultGPR);
    40434043            m_jit.urshift32(MacroAssembler::TrustedImm32(StringImpl::s_flagCount), resultGPR);
    … …  
    40774077        straightHash.append(m_jit.branchIfNotString(inputGPR));
    40784078        m_jit.loadPtr(MacroAssembler::Address(inputGPR, JSString::offsetOfValue()), resultGPR);
    4079         slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, resultGPR));
     4079        slowPath.append(m_jit.branchIfRopeStringImpl(resultGPR));
    40804080        m_jit.load32(MacroAssembler::Address(resultGPR, StringImpl::flagsOffset()), resultGPR);
    40814081        m_jit.urshift32(MacroAssembler::TrustedImm32(StringImpl::s_flagCount), resultGPR);
    … …  
    44514451            speculateString(node->child2(), keyGPR);
    44524452            m_jit.loadPtr(MacroAssembler::Address(keyGPR, JSString::offsetOfValue()), implGPR);
    4453             slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, implGPR));
     4453            slowPath.append(m_jit.branchIfRopeStringImpl(implGPR));
    44544454            slowPath.append(m_jit.branchTest32(
    44554455                MacroAssembler::Zero, MacroAssembler::Address(implGPR, StringImpl::flagsOffset()),
    … …  
    44614461            auto isNotString = m_jit.branchIfNotString(keyGPR);
    44624462            m_jit.loadPtr(MacroAssembler::Address(keyGPR, JSString::offsetOfValue()), implGPR);
    4463             slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, implGPR));
     4463            slowPath.append(m_jit.branchIfRopeStringImpl(implGPR));
    44644464            slowPath.append(m_jit.branchTest32(
    44654465                MacroAssembler::Zero, MacroAssembler::Address(implGPR, StringImpl::flagsOffset()),
    … …  
    52315231}
    52325232
     5233void SpeculativeJIT::compileMakeRope(Node* node)
     5234{
     5235    ASSERT(node->child1().useKind() == KnownStringUse);
     5236    ASSERT(node->child2().useKind() == KnownStringUse);
     5237    ASSERT(!node->child3() || node->child3().useKind() == KnownStringUse);
     5238
     5239    SpeculateCellOperand op1(this, node->child1());
     5240    SpeculateCellOperand op2(this, node->child2());
     5241    SpeculateCellOperand op3(this, node->child3());
     5242    GPRTemporary result(this);
     5243    GPRTemporary allocator(this);
     5244    GPRTemporary scratch(this);
     5245    GPRTemporary scratch2(this);
     5246
     5247    Edge edges[3] = {
     5248        node->child1(),
     5249        node->child2(),
     5250        node->child3()
     5251    };
     5252    GPRReg opGPRs[3];
     5253    unsigned numOpGPRs;
     5254    opGPRs[0] = op1.gpr();
     5255    opGPRs[1] = op2.gpr();
     5256    if (node->child3()) {
     5257        opGPRs[2] = op3.gpr();
     5258        numOpGPRs = 3;
     5259    } else {
     5260        opGPRs[2] = InvalidGPRReg;
     5261        numOpGPRs = 2;
     5262    }
     5263    GPRReg resultGPR = result.gpr();
     5264    GPRReg allocatorGPR = allocator.gpr();
     5265    GPRReg scratchGPR = scratch.gpr();
     5266    GPRReg scratch2GPR = scratch2.gpr();
     5267
     5268    CCallHelpers::JumpList slowPath;
     5269    Allocator allocatorValue = allocatorForNonVirtualConcurrently<JSRopeString>(*m_jit.vm(), sizeof(JSRopeString), AllocatorForMode::AllocatorIfExists);
     5270    emitAllocateJSCell(resultGPR, JITAllocator::constant(allocatorValue), allocatorGPR, TrustedImmPtr(m_jit.graph().registerStructure(m_jit.vm()->stringStructure.get())), scratchGPR, slowPath);
     5271
     5272    m_jit.orPtr(TrustedImm32(JSString::isRopeInPointer), opGPRs[0], allocatorGPR);
     5273    m_jit.storePtr(allocatorGPR, CCallHelpers::Address(resultGPR, JSRopeString::offsetOfFiber0()));
     5274
     5275    m_jit.move(opGPRs[1], scratchGPR);
     5276    m_jit.store32(scratchGPR, CCallHelpers::Address(resultGPR, JSRopeString::offsetOfFiber1Lower()));
     5277    m_jit.rshiftPtr(TrustedImm32(32), scratchGPR);
     5278    m_jit.store16(scratchGPR, CCallHelpers::Address(resultGPR, JSRopeString::offsetOfFiber1Upper()));
     5279
     5280    if (numOpGPRs == 3) {
     5281        m_jit.move(opGPRs[2], scratchGPR);
     5282        m_jit.store32(scratchGPR, CCallHelpers::Address(resultGPR, JSRopeString::offsetOfFiber2Lower()));
     5283        m_jit.rshiftPtr(TrustedImm32(32), scratchGPR);
     5284        m_jit.store16(scratchGPR, CCallHelpers::Address(resultGPR, JSRopeString::offsetOfFiber2Upper()));
     5285    } else {
     5286        m_jit.storeZero32(CCallHelpers::Address(resultGPR, JSRopeString::offsetOfFiber2Lower()));
     5287        m_jit.storeZero16(CCallHelpers::Address(resultGPR, JSRopeString::offsetOfFiber2Upper()));
     5288    }
     5289
     5290    {
     5291        if (JSString* string = edges[0]->dynamicCastConstant<JSString*>(*m_jit.vm())) {
     5292            m_jit.move(TrustedImm32(string->is8Bit() ? StringImpl::flagIs8Bit() : 0), scratchGPR);
     5293            m_jit.move(TrustedImm32(string->length()), allocatorGPR);
     5294        } else {
     5295            bool canBeRope = !m_state.forNode(edges[0]).isType(SpecStringIdent);
     5296            m_jit.loadPtr(CCallHelpers::Address(opGPRs[0], JSString::offsetOfValue()), scratch2GPR);
     5297            CCallHelpers::Jump isRope;
     5298            if (canBeRope)
     5299                isRope = m_jit.branchIfRopeStringImpl(scratch2GPR);
     5300
     5301            m_jit.load32(CCallHelpers::Address(scratch2GPR, StringImpl::flagsOffset()), scratchGPR);
     5302            m_jit.load32(CCallHelpers::Address(scratch2GPR, StringImpl::lengthMemoryOffset()), allocatorGPR);
     5303
     5304            if (canBeRope) {
     5305                auto done = m_jit.jump();
     5306
     5307                isRope.link(&m_jit);
     5308                m_jit.load16(CCallHelpers::Address(opGPRs[0], JSRopeString::offsetOfFlags()), scratchGPR);
     5309                m_jit.load32(CCallHelpers::Address(opGPRs[0], JSRopeString::offsetOfLength()), allocatorGPR);
     5310                done.link(&m_jit);
     5311            }
     5312        }
     5313
     5314        if (!ASSERT_DISABLED) {
     5315            CCallHelpers::Jump ok = m_jit.branch32(
     5316                CCallHelpers::GreaterThanOrEqual, allocatorGPR, TrustedImm32(0));
     5317            m_jit.abortWithReason(DFGNegativeStringLength);
     5318            ok.link(&m_jit);
     5319        }
     5320    }
     5321
     5322    for (unsigned i = 1; i < numOpGPRs; ++i) {
     5323        if (JSString* string = edges[i]->dynamicCastConstant<JSString*>(*m_jit.vm())) {
     5324            m_jit.and32(TrustedImm32(string->is8Bit() ? StringImpl::flagIs8Bit() : 0), scratchGPR);
     5325            speculationCheck(
     5326                Uncountable, JSValueSource(), nullptr,
     5327                m_jit.branchAdd32(
     5328                    CCallHelpers::Overflow,
     5329                    TrustedImm32(string->length()), allocatorGPR));
     5330        } else {
     5331            bool canBeRope = !m_state.forNode(edges[i]).isType(SpecStringIdent);
     5332            m_jit.loadPtr(CCallHelpers::Address(opGPRs[i], JSString::offsetOfValue()), scratch2GPR);
     5333            CCallHelpers::Jump isRope;
     5334            if (canBeRope)
     5335                isRope = m_jit.branchIfRopeStringImpl(scratch2GPR);
     5336
     5337            m_jit.and16(CCallHelpers::Address(scratch2GPR, StringImpl::flagsOffset()), scratchGPR);
     5338            speculationCheck(
     5339                Uncountable, JSValueSource(), nullptr,
     5340                m_jit.branchAdd32(
     5341                    CCallHelpers::Overflow,
     5342                    CCallHelpers::Address(scratch2GPR, StringImpl::lengthMemoryOffset()), allocatorGPR));
     5343            if (canBeRope) {
     5344                auto done = m_jit.jump();
     5345
     5346                isRope.link(&m_jit);
     5347                m_jit.and16(CCallHelpers::Address(opGPRs[i], JSRopeString::offsetOfFlags()), scratchGPR);
     5348                m_jit.load32(CCallHelpers::Address(opGPRs[i], JSRopeString::offsetOfLength()), scratch2GPR);
     5349                speculationCheck(
     5350                    Uncountable, JSValueSource(), nullptr,
     5351                    m_jit.branchAdd32(
     5352                        CCallHelpers::Overflow, scratch2GPR, allocatorGPR));
     5353                done.link(&m_jit);
     5354            }
     5355        }
     5356    }
     5357    m_jit.store16(scratchGPR, CCallHelpers::Address(resultGPR, JSRopeString::offsetOfFlags()));
     5358    if (!ASSERT_DISABLED) {
     5359        CCallHelpers::Jump ok = m_jit.branch32(
     5360            CCallHelpers::GreaterThanOrEqual, allocatorGPR, TrustedImm32(0));
     5361        m_jit.abortWithReason(DFGNegativeStringLength);
     5362        ok.link(&m_jit);
     5363    }
     5364    m_jit.store32(allocatorGPR, CCallHelpers::Address(resultGPR, JSRopeString::offsetOfLength()));
     5365    auto isNonEmptyString = m_jit.branchTest32(CCallHelpers::NonZero, allocatorGPR);
     5366
     5367    m_jit.move(TrustedImmPtr::weakPointer(m_jit.graph(), jsEmptyString(&m_jit.graph().m_vm)), resultGPR);
     5368
     5369    isNonEmptyString.link(&m_jit);
     5370    m_jit.mutatorFence(*m_jit.vm());
     5371
     5372    switch (numOpGPRs) {
     5373    case 2:
     5374        addSlowPathGenerator(slowPathCall(
     5375            slowPath, this, operationMakeRope2, resultGPR, opGPRs[0], opGPRs[1]));
     5376        break;
     5377    case 3:
     5378        addSlowPathGenerator(slowPathCall(
     5379            slowPath, this, operationMakeRope3, resultGPR, opGPRs[0], opGPRs[1], opGPRs[2]));
     5380        break;
     5381    default:
     5382        RELEASE_ASSERT_NOT_REACHED();
     5383        break;
     5384    }
     5385
     5386    cellResult(resultGPR, node);
     5387}
     5388
    52335389#endif
    52345390

  • trunk/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.cpp

    r240593 r242252  
    6161    , ArrayStorage_publicLength(Butterfly_publicLength)
    6262    , ArrayStorage_vectorLength(Butterfly_vectorLength)
    63     , JSBigInt_length(JSBigIntOrString_length)
    64     , JSString_length(JSBigIntOrString_length)
    6563   
    6664#define INDEXED_ABSTRACT_HEAP_INITIALIZATION(name, offset, size) , name(&root, #name, offset, size)
    … …  
    7270#undef NUMBERED_ABSTRACT_HEAP_INITIALIZATION
    7371
     72    , JSString_value(JSRopeString_fiber0)
     73
    7474    , absolute(&root, "absolute")
    7575{
    … …  
    8080    RELEASE_ASSERT(JSCell_indexingTypeAndMisc.offset() + 3 == JSCell_cellState.offset());
    8181
    82     RELEASE_ASSERT(JSBigInt::offsetOfLength() == JSString::offsetOfLength());
    83 
    8482    JSCell_structureID.changeParent(&JSCell_header);
    8583    JSCell_usefulBytes.changeParent(&JSCell_header);
    … …  
    8886    JSCell_typeInfoFlags.changeParent(&JSCell_usefulBytes);
    8987    JSCell_cellState.changeParent(&JSCell_usefulBytes);
     88    JSRopeString_flags.changeParent(&JSRopeString_fiber0);
    9089
    9190    RELEASE_ASSERT(!JSCell_freeListNext.offset());

  • trunk/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h

    r240593 r242252  
    6565    macro(JSArrayBufferView_mode, JSArrayBufferView::offsetOfMode()) \
    6666    macro(JSArrayBufferView_vector, JSArrayBufferView::offsetOfVector()) \
    67     macro(JSBigIntOrString_length, JSBigInt::offsetOfLength()) \
     67    macro(JSBigInt_length, JSBigInt::offsetOfLength()) \
    6868    macro(JSCell_cellState, JSCell::cellStateOffset()) \
    6969    macro(JSCell_header, 0) \
    … …  
    8989    macro(JSPropertyNameEnumerator_endStructurePropertyIndex, JSPropertyNameEnumerator::endStructurePropertyIndexOffset()) \
    9090    macro(JSPropertyNameEnumerator_indexLength, JSPropertyNameEnumerator::indexedLengthOffset()) \
     91    macro(JSRopeString_flags, JSRopeString::offsetOfFlags()) \
     92    macro(JSRopeString_fiber0, JSRopeString::offsetOfFiber0()) \
     93    macro(JSRopeString_length, JSRopeString::offsetOfLength()) \
     94    macro(JSRopeString_fiber1Lower, JSRopeString::offsetOfFiber1Lower()) \
     95    macro(JSRopeString_fiber1Upper, JSRopeString::offsetOfFiber1Upper()) \
     96    macro(JSRopeString_fiber2Lower, JSRopeString::offsetOfFiber2Lower()) \
     97    macro(JSRopeString_fiber2Upper, JSRopeString::offsetOfFiber2Upper()) \
    9198    macro(JSScope_next, JSScope::offsetOfNext()) \
    92     macro(JSString_flags, JSString::offsetOfFlags()) \
    93     macro(JSString_value, JSString::offsetOfValue()) \
    9499    macro(JSSymbolTableObject_symbolTable, JSSymbolTableObject::offsetOfSymbolTable()) \
    95100    macro(JSWrapperObject_internalValue, JSWrapperObject::internalValueOffset()) \
    … …  
    141146    macro(JSLexicalEnvironment_variables, JSLexicalEnvironment::offsetOfVariables(), sizeof(EncodedJSValue)) \
    142147    macro(JSPropertyNameEnumerator_cachedPropertyNamesVectorContents, 0, sizeof(WriteBarrier<JSString>)) \
    143     macro(JSRopeString_fibers, JSRopeString::offsetOfFibers(), sizeof(WriteBarrier<JSString>)) \
    144148    macro(ScopedArguments_Storage_storage, 0, sizeof(EncodedJSValue)) \
    145149    macro(WriteBarrierBuffer_bufferContents, 0, sizeof(JSCell*)) \
    … …  
    181185    AbstractHeap& ArrayStorage_publicLength;
    182186    AbstractHeap& ArrayStorage_vectorLength;
    183     AbstractHeap& JSBigInt_length;
    184     AbstractHeap& JSString_length;
    185187   
    186188#define INDEXED_ABSTRACT_HEAP_DECLARATION(name, offset, size) IndexedAbstractHeap name;
    … …  
    191193    FOR_EACH_NUMBERED_ABSTRACT_HEAP(NUMBERED_ABSTRACT_HEAP_DECLARATION)
    192194#undef NUMBERED_ABSTRACT_HEAP_DECLARATION
     195
     196    AbstractHeap& JSString_value;
    193197
    194198    AbsoluteAbstractHeap absolute;

  • trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp

    r242123 r242252  
    36533653            ValueFromBlock fastResult = m_out.anchor(fastResultValue);
    36543654           
    3655             m_out.branch(
    3656                 m_out.notNull(fastResultValue), usually(continuation), rarely(slowPath));
     3655            m_out.branch(isRopeString(cell, m_node->child1()), rarely(slowPath), usually(continuation));
    36573656           
    36583657            LBasicBlock lastNext = m_out.appendTo(slowPath, continuation);
    … …  
    38263825        case Array::String: {
    38273826            LValue string = lowCell(m_node->child1());
    3828             setInt32(m_out.load32NonNegative(string, m_heaps.JSString_length));
     3827
     3828            LBasicBlock ropePath = m_out.newBlock();
     3829            LBasicBlock nonRopePath = m_out.newBlock();
     3830            LBasicBlock continuation = m_out.newBlock();
     3831
     3832            m_out.branch(isRopeString(string, m_node->child1()), rarely(ropePath), usually(nonRopePath));
     3833
     3834            LBasicBlock lastNext = m_out.appendTo(ropePath, nonRopePath);
     3835            ValueFromBlock ropeLength = m_out.anchor(m_out.load32NonNegative(string, m_heaps.JSRopeString_length));
     3836            m_out.jump(continuation);
     3837
     3838            m_out.appendTo(nonRopePath, continuation);
     3839            ValueFromBlock nonRopeLength = m_out.anchor(m_out.load32NonNegative(m_out.loadPtr(string, m_heaps.JSString_value), m_heaps.StringImpl_length));
     3840            m_out.jump(continuation);
     3841
     3842            m_out.appendTo(continuation, lastNext);
     3843            setInt32(m_out.phi(Int32, ropeLength, nonRopeLength));
    38293844            return;
    38303845        }
    … …  
    64886503    void compileMakeRope()
    64896504    {
     6505        struct FlagsAndLength {
     6506            LValue flags;
     6507            LValue length;
     6508        };
     6509
     6510        Edge edges[3] = {
     6511            m_node->child1(),
     6512            m_node->child2(),
     6513            m_node->child3(),
     6514        };
    64906515        LValue kids[3];
    64916516        unsigned numKids;
    6492         kids[0] = lowCell(m_node->child1());
    6493         kids[1] = lowCell(m_node->child2());
    6494         if (m_node->child3()) {
    6495             kids[2] = lowCell(m_node->child3());
     6517        kids[0] = lowCell(edges[0]);
     6518        kids[1] = lowCell(edges[1]);
     6519        if (edges[2]) {
     6520            kids[2] = lowCell(edges[2]);
    64966521            numKids = 3;
    64976522        } else {
    … …  
    65036528        LBasicBlock continuation = m_out.newBlock();
    65046529       
    6505         LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath);
    6506        
    65076530        Allocator allocator = allocatorForNonVirtualConcurrently<JSRopeString>(vm(), sizeof(JSRopeString), AllocatorForMode::AllocatorIfExists);
    65086531       
    … …  
    65106533            m_out.constIntPtr(allocator.localAllocator()), vm().stringStructure.get(), slowPath);
    65116534       
    6512         m_out.storePtr(m_out.intPtrZero, result, m_heaps.JSString_value);
    6513         for (unsigned i = 0; i < numKids; ++i)
    6514             m_out.storePtr(kids[i], result, m_heaps.JSRopeString_fibers[i]);
    6515         for (unsigned i = numKids; i < JSRopeString::s_maxInternalRopeLength; ++i)
    6516             m_out.storePtr(m_out.intPtrZero, result, m_heaps.JSRopeString_fibers[i]);
    6517         LValue flags = m_out.load16ZeroExt32(kids[0], m_heaps.JSString_flags);
    6518         LValue length = m_out.load32(kids[0], m_heaps.JSString_length);
     6535        m_out.storePtr(m_out.bitOr(kids[0], m_out.constIntPtr(JSString::isRopeInPointer)), result, m_heaps.JSRopeString_fiber0);
     6536
     6537        m_out.store32(m_out.castToInt32(kids[1]), result, m_heaps.JSRopeString_fiber1Lower);
     6538        m_out.store32As16(m_out.castToInt32(m_out.lShr(kids[1], m_out.constInt32(32))), result, m_heaps.JSRopeString_fiber1Upper);
     6539
     6540        if (numKids == 3) {
     6541            m_out.store32(m_out.castToInt32(kids[2]), result, m_heaps.JSRopeString_fiber2Lower);
     6542            m_out.store32As16(m_out.castToInt32(m_out.lShr(kids[2], m_out.constInt32(32))), result, m_heaps.JSRopeString_fiber2Upper);
     6543        } else {
     6544            m_out.store32(m_out.int32Zero, result, m_heaps.JSRopeString_fiber2Lower);
     6545            m_out.store32As16(m_out.int32Zero, result, m_heaps.JSRopeString_fiber2Upper);
     6546        }
     6547
     6548        auto getFlagsAndLength = [&] (Edge& edge, LValue child) {
     6549            if (JSString* string = edge->dynamicCastConstant<JSString*>(vm())) {
     6550                return FlagsAndLength {
     6551                    m_out.constInt32(string->is8Bit() ? StringImpl::flagIs8Bit() : 0),
     6552                    m_out.constInt32(string->length())
     6553                };
     6554            }
     6555
     6556            LBasicBlock continuation = m_out.newBlock();
     6557            LBasicBlock ropeCase = m_out.newBlock();
     6558            LBasicBlock notRopeCase = m_out.newBlock();
     6559
     6560            m_out.branch(isRopeString(child, edge), unsure(ropeCase), unsure(notRopeCase));
     6561
     6562            LBasicBlock lastNext = m_out.appendTo(ropeCase, notRopeCase);
     6563            ValueFromBlock flagsForRope = m_out.anchor(m_out.load16ZeroExt32(child, m_heaps.JSRopeString_flags));
     6564            ValueFromBlock lengthForRope = m_out.anchor(m_out.load32NonNegative(child, m_heaps.JSRopeString_length));
     6565            m_out.jump(continuation);
     6566
     6567            m_out.appendTo(notRopeCase, continuation);
     6568            LValue stringImpl = m_out.loadPtr(child, m_heaps.JSString_value);
     6569            ValueFromBlock flagsForNonRope = m_out.anchor(m_out.load32NonNegative(stringImpl, m_heaps.StringImpl_hashAndFlags));
     6570            ValueFromBlock lengthForNonRope = m_out.anchor(m_out.load32NonNegative(stringImpl, m_heaps.StringImpl_length));
     6571            m_out.jump(continuation);
     6572
     6573            m_out.appendTo(continuation, lastNext);
     6574            return FlagsAndLength {
     6575                m_out.phi(Int32, flagsForRope, flagsForNonRope),
     6576                m_out.phi(Int32, lengthForRope, lengthForNonRope)
     6577            };
     6578        };
     6579
     6580        FlagsAndLength flagsAndLength = getFlagsAndLength(edges[0], kids[0]);
    65196581        for (unsigned i = 1; i < numKids; ++i) {
    6520             flags = m_out.bitAnd(flags, m_out.load16ZeroExt32(kids[i], m_heaps.JSString_flags));
    6521             CheckValue* lengthCheck = m_out.speculateAdd(
    6522                 length, m_out.load32(kids[i], m_heaps.JSString_length));
    6523             blessSpeculation(lengthCheck, Uncountable, noValue(), nullptr, m_origin);
    6524             length = lengthCheck;
    6525         }
    6526         m_out.store32As16(
    6527             m_out.bitAnd(m_out.constInt32(JSString::Is8Bit), flags),
    6528             result, m_heaps.JSString_flags);
    6529         m_out.store32(length, result, m_heaps.JSString_length);
     6582            auto mergeFlagsAndLength = [&] (Edge& edge, LValue child, FlagsAndLength previousFlagsAndLength) {
     6583                FlagsAndLength flagsAndLength = getFlagsAndLength(edge, child);
     6584                LValue flags = m_out.bitAnd(previousFlagsAndLength.flags, flagsAndLength.flags);
     6585                CheckValue* lengthCheck = m_out.speculateAdd(previousFlagsAndLength.length, flagsAndLength.length);
     6586                blessSpeculation(lengthCheck, Uncountable, noValue(), nullptr, m_origin);
     6587                return FlagsAndLength {
     6588                    flags,
     6589                    lengthCheck
     6590                };
     6591            };
     6592            flagsAndLength = mergeFlagsAndLength(edges[i], kids[i], flagsAndLength);
     6593        }
     6594        m_out.store32As16(flagsAndLength.flags, result, m_heaps.JSRopeString_flags);
     6595        m_out.store32(flagsAndLength.length, result, m_heaps.JSRopeString_length);
    65306596       
    65316597        mutatorFence();
    6532         ValueFromBlock fastResult = m_out.anchor(result);
     6598        ValueFromBlock fastResult = m_out.anchor(m_out.select(m_out.isZero32(flagsAndLength.length), weakPointer(jsEmptyString(&m_graph.m_vm)), result));
    65336599        m_out.jump(continuation);
    65346600       
    6535         m_out.appendTo(slowPath, continuation);
     6601        LBasicBlock lastNext = m_out.appendTo(slowPath, continuation);
    65366602        LValue slowResultValue;
    65376603        VM& vm = this->vm();
    … …  
    65746640        LBasicBlock continuation = m_out.newBlock();
    65756641           
     6642        LValue stringImpl = m_out.loadPtr(base, m_heaps.JSString_value);
    65766643        m_out.branch(
    65776644            m_out.aboveOrEqual(
    6578                 index, m_out.load32NonNegative(base, m_heaps.JSString_length)),
     6645                index, m_out.load32NonNegative(stringImpl, m_heaps.StringImpl_length)),
    65796646            rarely(slowPath), usually(fastPath));
    65806647           
    65816648        LBasicBlock lastNext = m_out.appendTo(fastPath, slowPath);
    6582            
    6583         LValue stringImpl = m_out.loadPtr(base, m_heaps.JSString_value);
    65846649           
    65856650        LBasicBlock is8Bit = m_out.newBlock();
    … …  
    66846749        LValue storage = lowStorage(m_node->child3());
    66856750       
     6751        LValue stringImpl = m_out.loadPtr(base, m_heaps.JSString_value);
     6752
    66866753        speculate(
    66876754            Uncountable, noValue(), 0,
    66886755            m_out.aboveOrEqual(
    6689                 index, m_out.load32NonNegative(base, m_heaps.JSString_length)));
    6690        
    6691         LValue stringImpl = m_out.loadPtr(base, m_heaps.JSString_value);
     6756                index, m_out.load32NonNegative(stringImpl, m_heaps.StringImpl_length)));
    66926757       
    66936758        m_out.branch(
    … …  
    72037268            speculateString(m_node->child2(), right);
    72047269           
    7205             ValueFromBlock slowResult = m_out.anchor(stringsEqual(left, right));
     7270            ValueFromBlock slowResult = m_out.anchor(stringsEqual(left, right, m_node->child1(), m_node->child2()));
    72067271            m_out.jump(continuation);
    72077272
    … …  
    73777442        // Full String compare.
    73787443        m_out.appendTo(testStringEquality, continuation);
    7379         ValueFromBlock slowResult = m_out.anchor(stringsEqual(leftString, rightValue));
     7444        ValueFromBlock slowResult = m_out.anchor(stringsEqual(leftString, rightValue, stringEdge, untypedEdge));
    73807445        m_out.jump(continuation);
    73817446
    … …  
    90279092            LBasicBlock continuation = m_out.newBlock();
    90289093           
     9094            ValueFromBlock fastValue = m_out.anchor(m_out.loadPtr(stringValue, m_heaps.JSString_value));
     9095            m_out.branch(
     9096                isRopeString(stringValue, m_node->child1()),
     9097                rarely(needResolution), usually(resolved));
     9098
     9099            LBasicBlock lastNext = m_out.appendTo(needResolution, resolved);
     9100            ValueFromBlock slowValue = m_out.anchor(
     9101                vmCall(pointerType(), m_out.operation(operationResolveRope), m_callFrame, stringValue));
     9102            m_out.jump(resolved);
     9103
     9104            m_out.appendTo(resolved, lengthIs1);
     9105            LValue value = m_out.phi(pointerType(), fastValue, slowValue);
    90299106            m_out.branch(
    90309107                m_out.notEqual(
    9031                     m_out.load32NonNegative(stringValue, m_heaps.JSString_length),
     9108                    m_out.load32NonNegative(value, m_heaps.StringImpl_length),
    90329109                    m_out.int32One),
    90339110                unsure(lowBlock(data->fallThrough.block)), unsure(lengthIs1));
    9034            
    9035             LBasicBlock lastNext = m_out.appendTo(lengthIs1, needResolution);
    9036             Vector<ValueFromBlock, 2> values;
    9037             LValue fastValue = m_out.loadPtr(stringValue, m_heaps.JSString_value);
    9038             values.append(m_out.anchor(fastValue));
    9039             m_out.branch(m_out.isNull(fastValue), rarely(needResolution), usually(resolved));
    9040            
    9041             m_out.appendTo(needResolution, resolved);
    9042             values.append(m_out.anchor(
    9043                 vmCall(pointerType(), m_out.operation(operationResolveRope), m_callFrame, stringValue)));
    9044             m_out.jump(resolved);
    9045            
    9046             m_out.appendTo(resolved, is8Bit);
    9047             LValue value = m_out.phi(pointerType(), values);
     9111
     9112            m_out.appendTo(lengthIs1, is8Bit);
    90489113            LValue characterData = m_out.loadPtr(value, m_heaps.StringImpl_data);
    90499114            m_out.branch(
    … …  
    90879152               
    90889153            case StringUse: {
    9089                 switchString(data, lowString(m_node->child1()));
     9154                switchString(data, lowString(m_node->child1()), m_node->child1());
    90909155                return;
    90919156            }
    … …  
    91099174                m_out.appendTo(isStringBlock, lastNext);
    91109175               
    9111                 switchString(data, value);
     9176                switchString(data, value, m_node->child1());
    91129177                return;
    91139178            }
    … …  
    94479512    }
    94489513
    9449     LValue mapHashString(LValue string)
     9514    LValue mapHashString(LValue string, Edge& edge)
    94509515    {
    94519516        LBasicBlock nonEmptyStringCase = m_out.newBlock();
    … …  
    94539518        LBasicBlock continuation = m_out.newBlock();
    94549519
     9520        m_out.branch(isRopeString(string, edge), rarely(slowCase), usually(nonEmptyStringCase));
     9521
     9522        LBasicBlock lastNext = m_out.appendTo(nonEmptyStringCase, slowCase);
    94559523        LValue stringImpl = m_out.loadPtr(string, m_heaps.JSString_value);
    9456         m_out.branch(
    9457             m_out.equal(stringImpl, m_out.constIntPtr(0)), unsure(slowCase), unsure(nonEmptyStringCase));
    9458 
    9459         LBasicBlock lastNext = m_out.appendTo(nonEmptyStringCase, slowCase);
    94609524        LValue hash = m_out.lShr(m_out.load32(stringImpl, m_heaps.StringImpl_hashAndFlags), m_out.constInt32(StringImpl::s_flagCount));
    94619525        ValueFromBlock nonEmptyStringHashResult = m_out.anchor(hash);
    … …  
    94969560
    94979561            LBasicBlock lastNext = m_out.appendTo(isString, notString);
    9498             ValueFromBlock stringResult = m_out.anchor(mapHashString(value));
     9562            ValueFromBlock stringResult = m_out.anchor(mapHashString(value, m_node->child1()));
    94999563            m_out.jump(continuation);
    95009564
    … …  
    95109574        case StringUse: {
    95119575            LValue string = lowString(m_node->child1());
    9512             setInt32(mapHashString(string));
     9576            setInt32(mapHashString(string, m_node->child1()));
    95139577            return;
    95149578        }
    … …  
    95379601
    95389602        m_out.appendTo(isStringCase, nonEmptyStringCase);
     9603        m_out.branch(isRopeString(value, m_node->child1()), rarely(slowCase), usually(nonEmptyStringCase));
     9604
     9605        m_out.appendTo(nonEmptyStringCase, straightHash);
    95399606        LValue stringImpl = m_out.loadPtr(value, m_heaps.JSString_value);
    9540         m_out.branch(
    9541             m_out.equal(stringImpl, m_out.constIntPtr(0)), rarely(slowCase), usually(nonEmptyStringCase));
    9542 
    9543         m_out.appendTo(nonEmptyStringCase, straightHash);
    95449607        LValue hash = m_out.lShr(m_out.load32(stringImpl, m_heaps.StringImpl_hashAndFlags), m_out.constInt32(StringImpl::s_flagCount));
    95459608        ValueFromBlock nonEmptyStringHashResult = m_out.anchor(hash);
    … …  
    1013910202
    1014010203            keyAsValue = lowString(m_node->child2());
     10204            m_out.branch(isNotRopeString(keyAsValue, m_node->child2()), usually(isNonEmptyString), rarely(slowCase));
     10205
     10206            lastNext = m_out.appendTo(isNonEmptyString, isAtomicString);
    1014110207            uniquedStringImpl = m_out.loadPtr(keyAsValue, m_heaps.JSString_value);
    10142             m_out.branch(m_out.notNull(uniquedStringImpl), usually(isNonEmptyString), rarely(slowCase));
    10143 
    10144             lastNext = m_out.appendTo(isNonEmptyString, isAtomicString);
    1014510208            LValue isNotAtomic = m_out.testIsZero32(m_out.load32(uniquedStringImpl, m_heaps.StringImpl_hashAndFlags), m_out.constInt32(StringImpl::flagIsAtomic()));
    1014610209            m_out.branch(isNotAtomic, rarely(slowCase), usually(isAtomicString));
    … …  
    1017010233
    1017110234            m_out.appendTo(isStringCase, isNonEmptyString);
     10235            m_out.branch(isNotRopeString(keyAsValue, m_node->child2()), usually(isNonEmptyString), rarely(slowCase));
     10236
     10237            m_out.appendTo(isNonEmptyString, notStringCase);
    1017210238            LValue implFromString = m_out.loadPtr(keyAsValue, m_heaps.JSString_value);
    1017310239            ValueFromBlock stringResult = m_out.anchor(implFromString);
    10174             m_out.branch(m_out.notNull(implFromString), usually(isNonEmptyString), rarely(slowCase));
    10175 
    10176             m_out.appendTo(isNonEmptyString, notStringCase);
    1017710240            LValue isNotAtomic = m_out.testIsZero32(m_out.load32(implFromString, m_heaps.StringImpl_hashAndFlags), m_out.constInt32(StringImpl::flagIsAtomic()));
    1017810241            m_out.branch(isNotAtomic, rarely(slowCase), usually(hasUniquedStringImpl));
    … …  
    1191211975    void compileStringSlice()
    1191311976    {
     11977        LBasicBlock lengthCheckCase = m_out.newBlock();
    1191411978        LBasicBlock emptyCase = m_out.newBlock();
    1191511979        LBasicBlock notEmptyCase = m_out.newBlock();
    1191611980        LBasicBlock oneCharCase = m_out.newBlock();
    11917         LBasicBlock bitCheckCase = m_out.newBlock();
    1191811981        LBasicBlock is8Bit = m_out.newBlock();
    1191911982        LBasicBlock is16Bit = m_out.newBlock();
    … …  
    1192111984        LBasicBlock bigCharacter = m_out.newBlock();
    1192211985        LBasicBlock slowCase = m_out.newBlock();
     11986        LBasicBlock ropeSlowCase = m_out.newBlock();
    1192311987        LBasicBlock continuation = m_out.newBlock();
    1192411988
    1192511989        LValue string = lowString(m_node->child1());
    11926         LValue length = m_out.load32NonNegative(string, m_heaps.JSString_length);
    1192711990        LValue start = lowInt32(m_node->child2());
    1192811991        LValue end = nullptr;
    1192911992        if (m_node->child3())
    1193011993            end = lowInt32(m_node->child3());
    11931 
     11994        else
     11995            end = m_out.constInt32(std::numeric_limits<int32_t>::max());
     11996        m_out.branch(isRopeString(string, m_node->child1()), rarely(ropeSlowCase), usually(lengthCheckCase));
     11997
     11998        LBasicBlock lastNext = m_out.appendTo(lengthCheckCase, emptyCase);
     11999        LValue stringImpl = m_out.loadPtr(string, m_heaps.JSString_value);
     12000        LValue length = m_out.load32NonNegative(stringImpl, m_heaps.StringImpl_length);
    1193212001        auto range = populateSliceRange(start, end, length);
    1193312002        LValue from = range.first;
    1193412003        LValue to = range.second;
    11935 
    1193612004        LValue span = m_out.sub(to, from);
    1193712005        m_out.branch(m_out.lessThanOrEqual(span, m_out.int32Zero), unsure(emptyCase), unsure(notEmptyCase));
    1193812006
    11939         Vector<ValueFromBlock, 4> results;
    11940 
    11941         LBasicBlock lastNext = m_out.appendTo(emptyCase, notEmptyCase);
     12007        Vector<ValueFromBlock, 5> results;
     12008
     12009        m_out.appendTo(emptyCase, notEmptyCase);
    1194212010        results.append(m_out.anchor(weakPointer(jsEmptyString(&vm()))));
    1194312011        m_out.jump(continuation);
    … …  
    1194612014        m_out.branch(m_out.equal(span, m_out.int32One), unsure(oneCharCase), unsure(slowCase));
    1194712015
    11948         m_out.appendTo(oneCharCase, bitCheckCase);
    11949         LValue stringImpl = m_out.loadPtr(string, m_heaps.JSString_value);
    11950         m_out.branch(m_out.isNull(stringImpl), unsure(slowCase), unsure(bitCheckCase));
    11951 
    11952         m_out.appendTo(bitCheckCase, is8Bit);
     12016        m_out.appendTo(oneCharCase, is8Bit);
    1195312017        LValue storage = m_out.loadPtr(stringImpl, m_heaps.StringImpl_data);
    1195412018        m_out.branch(
    … …  
    1195912023
    1196012024        m_out.appendTo(is8Bit, is16Bit);
    11961         // FIXME: Need to cage strings!
    11962         // https://bugs.webkit.org/show_bug.cgi?id=174924
    1196312025        ValueFromBlock char8Bit = m_out.anchor(m_out.load8ZeroExt32(m_out.baseIndex(m_heaps.characters8, storage, m_out.zeroExtPtr(from))));
    1196412026        m_out.jump(bitsContinuation);
    … …  
    1198412046        m_out.jump(continuation);
    1198512047
    11986         m_out.appendTo(slowCase, continuation);
     12048        m_out.appendTo(slowCase, ropeSlowCase);
    1198712049        results.append(m_out.anchor(vmCall(pointerType(), m_out.operation(operationStringSubstr), m_callFrame, string, from, span)));
     12050        m_out.jump(continuation);
     12051
     12052        m_out.appendTo(ropeSlowCase, continuation);
     12053        results.append(m_out.anchor(vmCall(pointerType(), m_out.operation(operationStringSlice), m_callFrame, string, start, end)));
    1198812054        m_out.jump(continuation);
    1198912055
    … …  
    1200412070        ValueFromBlock startIndex = m_out.anchor(m_out.constInt32(0));
    1200512071        ValueFromBlock startIndexForCall = m_out.anchor(m_out.constInt32(0));
     12072        m_out.branch(isRopeString(string, m_node->child1()),
     12073            unsure(slowPath), unsure(notRope));
     12074
     12075        LBasicBlock lastNext = m_out.appendTo(notRope, is8Bit);
    1200612076        LValue impl = m_out.loadPtr(string, m_heaps.JSString_value);
    12007         m_out.branch(m_out.isZero64(impl),
    12008             unsure(slowPath), unsure(notRope));
    12009 
    12010         LBasicBlock lastNext = m_out.appendTo(notRope, is8Bit);
    12011 
    1201212077        m_out.branch(
    1201312078            m_out.testIsZero32(
    … …  
    1278012845    }
    1278112846
    12782     LValue stringsEqual(LValue leftJSString, LValue rightJSString)
     12847    LValue stringsEqual(LValue leftJSString, LValue rightJSString, Edge leftJSStringEdge = Edge(), Edge rightJSStringEdge = Edge())
    1278312848    {
    1278412849        LBasicBlock notTriviallyUnequalCase = m_out.newBlock();
    … …  
    1279512860        LBasicBlock continuation = m_out.newBlock();
    1279612861
    12797         LValue length = m_out.load32(leftJSString, m_heaps.JSString_length);
    12798 
    12799         m_out.branch(
    12800             m_out.notEqual(length, m_out.load32(rightJSString, m_heaps.JSString_length)),
    12801             unsure(falseCase), unsure(notTriviallyUnequalCase));
    12802 
    12803         LBasicBlock lastNext = m_out.appendTo(notTriviallyUnequalCase, notEmptyCase);
    12804 
    12805         m_out.branch(m_out.isZero32(length), unsure(trueCase), unsure(notEmptyCase));
    12806 
    12807         m_out.appendTo(notEmptyCase, leftReadyCase);
    12808 
     12862        m_out.branch(isRopeString(leftJSString, leftJSStringEdge), rarely(slowCase), usually(leftReadyCase));
     12863
     12864        LBasicBlock lastNext = m_out.appendTo(leftReadyCase, rightReadyCase);
     12865        m_out.branch(isRopeString(rightJSString, rightJSStringEdge), rarely(slowCase), usually(rightReadyCase));
     12866
     12867        m_out.appendTo(rightReadyCase, notTriviallyUnequalCase);
    1280912868        LValue left = m_out.loadPtr(leftJSString, m_heaps.JSString_value);
    1281012869        LValue right = m_out.loadPtr(rightJSString, m_heaps.JSString_value);
    12811 
    12812         m_out.branch(m_out.notNull(left), usually(leftReadyCase), rarely(slowCase));
    12813 
    12814         m_out.appendTo(leftReadyCase, rightReadyCase);
    12815        
    12816         m_out.branch(m_out.notNull(right), usually(rightReadyCase), rarely(slowCase));
    12817 
    12818         m_out.appendTo(rightReadyCase, left8BitCase);
    12819 
     12870        LValue length = m_out.load32(left, m_heaps.StringImpl_length);
     12871        m_out.branch(
     12872            m_out.notEqual(length, m_out.load32(right, m_heaps.StringImpl_length)),
     12873            unsure(falseCase), unsure(notTriviallyUnequalCase));
     12874
     12875        m_out.appendTo(notTriviallyUnequalCase, notEmptyCase);
     12876        m_out.branch(m_out.isZero32(length), unsure(trueCase), unsure(notEmptyCase));
     12877
     12878        m_out.appendTo(notEmptyCase, left8BitCase);
    1282012879        m_out.branch(
    1282112880            m_out.testIsZero32(
    … …  
    1282512884
    1282612885        m_out.appendTo(left8BitCase, right8BitCase);
    12827 
    1282812886        m_out.branch(
    1282912887            m_out.testIsZero32(
    … …  
    1354713605                    edge, CellCaseSpeculatesObject, SpeculateNullOrUndefined,
    1354813606                    ManualOperandSpeculation));
    13549         case StringUse: {
    13550             LValue stringValue = lowString(edge);
    13551             LValue length = m_out.load32NonNegative(stringValue, m_heaps.JSString_length);
    13552             return m_out.notEqual(length, m_out.int32Zero);
    13553         }
     13607        case StringUse:
     13608            return m_out.notEqual(lowString(edge), weakPointer(jsEmptyString(&m_graph.m_vm)));
    1355413609        case StringOrOtherUse: {
    1355513610            LValue value = lowJSValue(edge, ManualOperandSpeculation);
    … …  
    1356213617           
    1356313618            LBasicBlock lastNext = m_out.appendTo(cellCase, notCellCase);
    13564            
    1356513619            FTL_TYPE_CHECK(jsValueValue(value), edge, (~SpecCellCheck) | SpecString, isNotString(value));
    13566             LValue length = m_out.load32NonNegative(value, m_heaps.JSString_length);
    13567             ValueFromBlock cellResult = m_out.anchor(m_out.notEqual(length, m_out.int32Zero));
     13620            ValueFromBlock stringResult = m_out.anchor(m_out.notEqual(value, weakPointer(jsEmptyString(&m_graph.m_vm))));
    1356813621            m_out.jump(continuation);
    13569            
     13622
    1357013623            m_out.appendTo(notCellCase, continuation);
    13571            
    1357213624            FTL_TYPE_CHECK(jsValueValue(value), edge, SpecCellCheck | SpecOther, isNotOther(value));
    1357313625            ValueFromBlock notCellResult = m_out.anchor(m_out.booleanFalse);
    1357413626            m_out.jump(continuation);
     13627
    1357513628            m_out.appendTo(continuation, lastNext);
    13576 
    13577             return m_out.phi(Int32, cellResult, notCellResult);
     13629            return m_out.phi(Int32, stringResult, notCellResult);
    1357813630        }
    1357913631        case UntypedUse: {
    … …  
    1359813650            LBasicBlock cellCase = m_out.newBlock();
    1359913651            LBasicBlock notStringCase = m_out.newBlock();
    13600             LBasicBlock stringOrBigIntCase = m_out.newBlock();
     13652            LBasicBlock stringCase = m_out.newBlock();
     13653            LBasicBlock bigIntCase = m_out.newBlock();
    1360113654            LBasicBlock notStringOrBigIntCase = m_out.newBlock();
    1360213655            LBasicBlock notCellCase = m_out.newBlock();
    … …  
    1361413667            m_out.branch(
    1361513668                isString(value, provenType(edge) & SpecCell),
    13616                 unsure(stringOrBigIntCase), unsure(notStringCase));
    13617            
    13618             m_out.appendTo(notStringCase, stringOrBigIntCase);
     13669                unsure(stringCase), unsure(notStringCase));
     13670           
     13671            m_out.appendTo(notStringCase, stringCase);
    1361913672            m_out.branch(
    1362013673                isBigInt(value, provenType(edge) & (SpecCell - SpecString)),
    13621                 unsure(stringOrBigIntCase), unsure(notStringOrBigIntCase));
    13622 
    13623             m_out.appendTo(stringOrBigIntCase, notStringOrBigIntCase);
    13624             LValue nonZeroCell = m_out.notZero32(
    13625                 m_out.load32NonNegative(value, m_heaps.JSBigIntOrString_length));
    13626             results.append(m_out.anchor(nonZeroCell));
     13674                unsure(bigIntCase), unsure(notStringOrBigIntCase));
     13675
     13676            m_out.appendTo(stringCase, bigIntCase);
     13677            results.append(m_out.anchor(m_out.notEqual(value, weakPointer(jsEmptyString(&m_graph.m_vm)))));
     13678            m_out.jump(continuation);
     13679
     13680            m_out.appendTo(bigIntCase, notStringOrBigIntCase);
     13681            LValue nonZeroBigInt = m_out.notZero32(
     13682                m_out.load32NonNegative(value, m_heaps.JSBigInt_length));
     13683            results.append(m_out.anchor(nonZeroBigInt));
    1362713684            m_out.jump(continuation);
    1362813685           
    … …  
    1388813945    }
    1388913946   
    13890     void switchString(SwitchData* data, LValue string)
     13947    void switchString(SwitchData* data, LValue string, Edge& edge)
    1389113948    {
    1389213949        bool canDoBinarySwitch = true;
    … …  
    1391113968        }
    1391213969       
    13913         LValue stringImpl = m_out.loadPtr(string, m_heaps.JSString_value);
    13914         LValue length = m_out.load32(string, m_heaps.JSString_length);
    13915        
    1391613970        LBasicBlock hasImplBlock = m_out.newBlock();
    1391713971        LBasicBlock is8BitBlock = m_out.newBlock();
    1391813972        LBasicBlock slowBlock = m_out.newBlock();
    1391913973       
    13920         m_out.branch(m_out.isNull(stringImpl), unsure(slowBlock), unsure(hasImplBlock));
     13974        m_out.branch(isRopeString(string, edge), unsure(slowBlock), unsure(hasImplBlock));
    1392113975       
    1392213976        LBasicBlock lastNext = m_out.appendTo(hasImplBlock, is8BitBlock);
     13977
     13978        LValue stringImpl = m_out.loadPtr(string, m_heaps.JSString_value);
     13979        LValue length = m_out.load32(stringImpl, m_heaps.StringImpl_length);
    1392313980       
    1392413981        m_out.branch(
    … …  
    1561515672    }
    1561615673
     15674    LValue isRopeString(LValue string, Edge edge = Edge())
     15675    {
     15676        if (edge) {
     15677            if (!((provenType(edge) & SpecString) & ~SpecStringIdent))
     15678                return m_out.booleanFalse;
     15679            if (JSValue value = provenValue(edge)) {
     15680                if (value.isCell() && value.asCell()->type() == StringType && !asString(value)->isRope())
     15681                    return m_out.booleanFalse;
     15682            }
     15683        }
     15684
     15685        return m_out.testNonZeroPtr(m_out.loadPtr(string, m_heaps.JSString_value), m_out.constIntPtr(JSString::isRopeInPointer));
     15686    }
     15687
     15688    LValue isNotRopeString(LValue string, Edge edge = Edge())
     15689    {
     15690        if (edge) {
     15691            if (!((provenType(edge) & SpecString) & ~SpecStringIdent))
     15692                return m_out.booleanTrue;
     15693            if (JSValue value = provenValue(edge)) {
     15694                if (value.isCell() && value.asCell()->type() == StringType && !asString(value)->isRope())
     15695                    return m_out.booleanTrue;
     15696            }
     15697        }
     15698
     15699        return m_out.testIsZeroPtr(m_out.loadPtr(string, m_heaps.JSString_value), m_out.constIntPtr(JSString::isRopeInPointer));
     15700    }
     15701
    1561715702    LValue isNotSymbol(LValue cell, SpeculatedType type = SpecFullTop)
    1561815703    {
    … …  
    1601216097            return;
    1601316098       
    16014         speculate(BadType, jsValueValue(string), edge.node(), m_out.isNull(stringImpl));
     16099        speculate(BadType, jsValueValue(string), edge.node(), isRopeString(string));
    1601516100        speculate(
    1601616101            BadType, jsValueValue(string), edge.node(),

  • trunk/Source/JavaScriptCore/jit/AssemblyHelpers.cpp

    r242123 r242252  
    728728
    729729    isString.link(this);
     730    move(TrustedImmPtr(jsEmptyString(&vm)), result);
     731    comparePtr(invert ? Equal : NotEqual, value.payloadGPR(), result, result);
     732    done.append(jump());
     733
    730734    isBigInt.link(this);
    731     RELEASE_ASSERT(JSString::offsetOfLength() == JSBigInt::offsetOfLength());
    732735    load32(Address(value.payloadGPR(), JSBigInt::offsetOfLength()), result);
    733736    compare32(invert ? Equal : NotEqual, result, TrustedImm32(0), result);
    … …  
    815818
    816819    isString.link(this);
     820    truthy.append(branchPtr(invert ? Equal : NotEqual, value.payloadGPR(), TrustedImmPtr(jsEmptyString(&vm))));
     821    done.append(jump());
     822
    817823    isBigInt.link(this);
    818     RELEASE_ASSERT(JSString::offsetOfLength() == JSBigInt::offsetOfLength());
    819824    truthy.append(branchTest32(invert ? Zero : NonZero, Address(value.payloadGPR(), JSBigInt::offsetOfLength())));
    820825    done.append(jump());

  • trunk/Source/JavaScriptCore/jit/AssemblyHelpers.h

    r242123 r242252  
    10741074    {
    10751075        return branchDouble(DoubleEqual, fpr, fpr);
     1076    }
     1077
     1078    Jump branchIfRopeStringImpl(GPRReg stringImplGPR)
     1079    {
     1080        return branchTestPtr(NonZero, stringImplGPR, TrustedImm32(JSString::isRopeInPointer));
     1081    }
     1082
     1083    Jump branchIfNotRopeStringImpl(GPRReg stringImplGPR)
     1084    {
     1085        return branchTestPtr(Zero, stringImplGPR, TrustedImm32(JSString::isRopeInPointer));
    10761086    }
    10771087

  • trunk/Source/JavaScriptCore/jit/JITInlines.h

    r240041 r242252  
    9595{
    9696    failures.append(branchIfNotString(src));
    97     failures.append(branch32(NotEqual, MacroAssembler::Address(src, JSString::offsetOfLength()), TrustedImm32(1)));
    9897    loadPtr(MacroAssembler::Address(src, JSString::offsetOfValue()), dst);
    99     failures.append(branchTest32(Zero, dst));
     98    failures.append(branchIfRopeStringImpl(dst));
     99    failures.append(branch32(NotEqual, MacroAssembler::Address(dst, StringImpl::lengthMemoryOffset()), TrustedImm32(1)));
    100100    loadPtr(MacroAssembler::Address(dst, StringImpl::flagsOffset()), regT1);
    101101    loadPtr(MacroAssembler::Address(dst, StringImpl::dataOffset()), dst);

  • trunk/Source/JavaScriptCore/jit/Repatch.cpp

    r242123 r242252  
    211211                newCase = AccessCase::create(vm, codeBlock, AccessCase::ArrayLength);
    212212            } else if (isJSString(baseCell)) {
    213                 if (stubInfo.cacheType == CacheType::Unset) {
     213                if (stubInfo.cacheType == CacheType::Unset && InlineAccess::isCacheableStringLength(stubInfo)) {
    214214                    bool generatedCodeInline = InlineAccess::generateStringLength(stubInfo);
    215215                    if (generatedCodeInline) {

  • trunk/Source/JavaScriptCore/jit/ThunkGenerators.cpp

    r242123 r242252  
    623623
    624624    // Load string length to regT2, and start the process of loading the data pointer into regT0
    625     jit.load32(JSInterfaceJIT::Address(stringGPR, JSString::offsetOfLength()), scratchGPR);
    626625    jit.loadPtr(JSInterfaceJIT::Address(stringGPR, JSString::offsetOfValue()), stringGPR);
    627     failures.append(jit.branchTestPtr(JSInterfaceJIT::Zero, stringGPR));
     626    failures.append(jit.branchIfRopeStringImpl(stringGPR));
     627    jit.load32(JSInterfaceJIT::Address(stringGPR, StringImpl::lengthMemoryOffset()), scratchGPR);
    628628
    629629    // Do an unsigned compare to simultaneously filter negative indices as well as indices that are too large
    … …  
    662662
    663663    // Load string length to regT2, and start the process of loading the data pointer into regT0
    664     jit.load32(MacroAssembler::Address(SpecializedThunkJIT::regT0, JSString::offsetOfLength()), SpecializedThunkJIT::regT2);
    665664    jit.loadPtr(MacroAssembler::Address(SpecializedThunkJIT::regT0, JSString::offsetOfValue()), SpecializedThunkJIT::regT0);
    666     jit.appendFailure(jit.branchTest32(MacroAssembler::Zero, SpecializedThunkJIT::regT0));
     665    jit.appendFailure(jit.branchIfRopeStringImpl(SpecializedThunkJIT::regT0));
     666    jit.load32(MacroAssembler::Address(SpecializedThunkJIT::regT0, StringImpl::lengthMemoryOffset()), SpecializedThunkJIT::regT2);
    667667
    668668    // load index

  • trunk/Source/JavaScriptCore/llint/LowLevelInterpreter.asm

    r242096 r242252  
    483483
    484484# String flags.
     485const isRopeInPointer = constexpr JSString::isRopeInPointer
    485486const HashFlags8BitBuffer = constexpr StringImpl::s_hashFlag8BitBuffer
    486487

  • trunk/Source/JavaScriptCore/llint/LowLevelInterpreter32_64.asm

    r242067 r242252  
    17891789    bineq t1, CellTag, .opSwitchCharFallThrough
    17901790    bbneq JSCell::m_type[t0], StringType, .opSwitchCharFallThrough
    1791     bineq JSString::m_length[t0], 1, .opSwitchCharFallThrough
    1792     loadp JSString::m_value[t0], t0
    1793     btpz  t0, .opSwitchOnRope
     1791    loadp JSString::m_fiber[t0], t0
     1792    btpnz t0, isRopeInPointer, .opSwitchOnRope
     1793    bineq StringImpl::m_length[t0], 1, .opSwitchCharFallThrough
    17941794    loadp StringImpl::m_data8[t0], t1
    17951795    btinz StringImpl::m_hashAndFlags[t0], HashFlags8BitBuffer, .opSwitchChar8Bit

  • trunk/Source/JavaScriptCore/llint/LowLevelInterpreter64.asm

    r242096 r242252  
    18781878    btqnz t1, tagMask, .opSwitchCharFallThrough
    18791879    bbneq JSCell::m_type[t1], StringType, .opSwitchCharFallThrough
    1880     bineq JSString::m_length[t1], 1, .opSwitchCharFallThrough
    1881     loadp JSString::m_value[t1], t0
    1882     btpz  t0, .opSwitchOnRope
     1880    loadp JSString::m_fiber[t1], t0
     1881    btpnz t0, isRopeInPointer, .opSwitchOnRope
     1882    bineq StringImpl::m_length[t0], 1, .opSwitchCharFallThrough
    18831883    loadp StringImpl::m_data8[t0], t1
    18841884    btinz StringImpl::m_hashAndFlags[t0], HashFlags8BitBuffer, .opSwitchChar8Bit

  • trunk/Source/JavaScriptCore/runtime/JSString.cpp

    r239427 r242252  
    4141}
    4242
     43JSString* JSString::createEmptyString(VM& vm)
     44{
     45    JSString* newString = new (NotNull, allocateCell<JSString>(vm.heap)) JSString(vm, *StringImpl::empty());
     46    newString->finishCreation(vm);
     47    return newString;
     48}
     49
    4350template<>
    4451void JSRopeString::RopeBuilder<RecordOverflow>::expand()
    4552{
    4653    RELEASE_ASSERT(!this->hasOverflowed());
    47     ASSERT(m_index == JSRopeString::s_maxInternalRopeLength);
    48     JSString* jsString = m_jsString;
    49     m_jsString = jsStringBuilder(&m_vm);
    50     m_index = 0;
    51     append(jsString);
     54    ASSERT(m_strings.size() == JSRopeString::s_maxInternalRopeLength);
     55    static_assert(3 == JSRopeString::s_maxInternalRopeLength, "");
     56    ASSERT(m_length);
     57    ASSERT(asString(m_strings.at(0))->length());
     58    ASSERT(asString(m_strings.at(1))->length());
     59    ASSERT(asString(m_strings.at(2))->length());
     60
     61    JSString* string = JSRopeString::create(m_vm, asString(m_strings.at(0)), asString(m_strings.at(1)), asString(m_strings.at(2)));
     62    ASSERT(string->length() == m_length);
     63    m_strings.clear();
     64    m_strings.append(string);
    5265}
    5366
    … …  
    6275    const JSString* thisObject = jsCast<const JSString*>(cell);
    6376    out.printf("<%p, %s, [%u], ", thisObject, thisObject->className(vm), thisObject->length());
    64     if (thisObject->isRope())
     77    uintptr_t pointer = thisObject->m_fiber;
     78    if (pointer & isRopeInPointer)
    6579        out.printf("[rope]");
    6680    else {
    67         WTF::StringImpl* ourImpl = thisObject->m_value.impl();
    68         if (ourImpl->is8Bit())
    69             out.printf("[8 %p]", ourImpl->characters8());
    70         else
    71             out.printf("[16 %p]", ourImpl->characters16());
     81        if (WTF::StringImpl* ourImpl = bitwise_cast<StringImpl*>(pointer)) {
     82            if (ourImpl->is8Bit())
     83                out.printf("[8 %p]", ourImpl->characters8());
     84            else
     85                out.printf("[16 %p]", ourImpl->characters16());
     86        }
    7287    }
    7388    out.printf(">");
    … …  
    87102{
    88103    JSString* thisObject = asString(cell);
    89     if (thisObject->isRope())
     104    uintptr_t pointer = thisObject->m_fiber;
     105    if (pointer & isRopeInPointer)
    90106        return Base::estimatedSize(cell, vm);
    91     return Base::estimatedSize(cell, vm) + thisObject->m_value.impl()->costDuringGC();
     107    return Base::estimatedSize(cell, vm) + bitwise_cast<StringImpl*>(pointer)->costDuringGC();
    92108}
    93109
    … …  
    97113    Base::visitChildren(thisObject, visitor);
    98114   
    99     if (thisObject->isRope())
    100         static_cast<JSRopeString*>(thisObject)->visitFibers(visitor);
    101     if (StringImpl* impl = thisObject->m_value.impl())
     115    uintptr_t pointer = thisObject->m_fiber;
     116    if (pointer & isRopeInPointer) {
     117        if ((pointer & JSRopeString::stringMask) == JSRopeString::substringSentinel()) {
     118            visitor.appendUnbarriered(static_cast<JSRopeString*>(thisObject)->fiber1());
     119            return;
     120        }
     121        for (unsigned index = 0; index < JSRopeString::s_maxInternalRopeLength; ++index) {
     122            JSString* fiber = nullptr;
     123            switch (index) {
     124            case 0:
     125                fiber = bitwise_cast<JSString*>(pointer & JSRopeString::stringMask);
     126                break;
     127            case 1:
     128                fiber = static_cast<JSRopeString*>(thisObject)->fiber1();
     129                break;
     130            case 2:
     131                fiber = static_cast<JSRopeString*>(thisObject)->fiber2();
     132                break;
     133            default:
     134                ASSERT_NOT_REACHED();
     135                return;
     136            }
     137            if (!fiber)
     138                break;
     139            visitor.appendUnbarriered(fiber);
     140        }
     141        return;
     142    }
     143    if (StringImpl* impl = bitwise_cast<StringImpl*>(pointer))
    102144        visitor.reportExtraMemoryVisited(impl->costDuringGC());
    103145}
    104146
    105 void JSRopeString::visitFibers(SlotVisitor& visitor)
     147static const unsigned maxLengthForOnStackResolve = 2048;
     148
     149void JSRopeString::resolveRopeInternal8(LChar* buffer) const
    106150{
    107151    if (isSubstring()) {
    108         visitor.append(substringBase());
    109         return;
    110     }
    111     for (size_t i = 0; i < s_maxInternalRopeLength && fiber(i); ++i)
    112         visitor.append(fiber(i));
    113 }
    114 
    115 static const unsigned maxLengthForOnStackResolve = 2048;
    116 
    117 void JSRopeString::resolveRopeInternal8(LChar* buffer) const
    118 {
    119     if (isSubstring()) {
    120         StringImpl::copyCharacters(buffer, substringBase()->m_value.characters8() + substringOffset(), length());
     152        StringImpl::copyCharacters(buffer, substringBase()->valueInternal().characters8() + substringOffset(), length());
    121153        return;
    122154    }
    … …  
    136168    LChar* position = buffer;
    137169    for (size_t i = 0; i < s_maxInternalRopeLength && fiber(i); ++i) {
    138         const StringImpl& fiberString = *fiber(i)->m_value.impl();
     170        const StringImpl& fiberString = *fiber(i)->valueInternal().impl();
    139171        unsigned length = fiberString.length();
    140172        StringImpl::copyCharacters(position, fiberString.characters8(), length);
    … …  
    148180    if (isSubstring()) {
    149181        StringImpl::copyCharacters(
    150             buffer, substringBase()->m_value.characters16() + substringOffset(), length());
     182            buffer, substringBase()->valueInternal().characters16() + substringOffset(), length());
    151183        return;
    152184    }
    … …  
    166198    UChar* position = buffer;
    167199    for (size_t i = 0; i < s_maxInternalRopeLength && fiber(i); ++i) {
    168         const StringImpl& fiberString = *fiber(i)->m_value.impl();
     200        const StringImpl& fiberString = *fiber(i)->valueInternal().impl();
    169201        unsigned length = fiberString.length();
    170202        if (fiberString.is8Bit())
    … …  
    177209}
    178210
    179 void JSRopeString::resolveRopeToAtomicString(ExecState* exec) const
     211AtomicString JSRopeString::resolveRopeToAtomicString(ExecState* exec) const
    180212{
    181213    VM& vm = exec->vm();
    … …  
    183215
    184216    if (length() > maxLengthForOnStackResolve) {
    185         resolveRope(exec);
    186         RETURN_IF_EXCEPTION(scope, void());
    187         m_value = AtomicString(m_value);
    188         setIs8Bit(m_value.impl()->is8Bit());
    189         return;
     217        scope.release();
     218        return resolveRopeWithFunction(exec, [&] (Ref<StringImpl>&& newImpl) {
     219            return AtomicStringImpl::add(newImpl.ptr());
     220        });
    190221    }
    191222
    … …  
    193224        LChar buffer[maxLengthForOnStackResolve];
    194225        resolveRopeInternal8(buffer);
    195         m_value = AtomicString(buffer, length());
    196         setIs8Bit(m_value.impl()->is8Bit());
     226        convertToNonRope(AtomicStringImpl::add(buffer, length()));
    197227    } else {
    198228        UChar buffer[maxLengthForOnStackResolve];
    199229        resolveRopeInternal16(buffer);
    200         m_value = AtomicString(buffer, length());
    201         setIs8Bit(m_value.impl()->is8Bit());
    202     }
    203 
    204     clearFibers();
     230        convertToNonRope(AtomicStringImpl::add(buffer, length()));
     231    }
    205232
    206233    // If we resolved a string that didn't previously exist, notify the heap that we've grown.
    207     if (m_value.impl()->hasOneRef())
    208         vm.heap.reportExtraMemoryAllocated(m_value.impl()->cost());
    209 }
    210 
    211 void JSRopeString::clearFibers() const
    212 {
    213     for (size_t i = 0; i < s_maxInternalRopeLength; ++i)
    214         u[i].number = 0;
     234    if (valueInternal().impl()->hasOneRef())
     235        vm.heap.reportExtraMemoryAllocated(valueInternal().impl()->cost());
     236    return valueInternal();
     237}
     238
     239inline void JSRopeString::convertToNonRope(String&& string) const
     240{
     241    // Concurrent compiler threads can access String held by JSString. So we always emit
     242    // store-store barrier here to ensure concurrent compiler threads see initialized String.
     243    ASSERT(JSString::isRope());
     244    WTF::storeStoreFence();
     245    new (&uninitializedValueInternal()) String(WTFMove(string));
     246    static_assert(sizeof(String) == sizeof(RefPtr<StringImpl>), "JSString's String initialization must be done in one pointer move.");
     247    // We do not clear the trailing fibers and length information (fiber1 and fiber2) because we could be reading the length concurrently.
     248    ASSERT(!JSString::isRope());
    215249}
    216250
    217251RefPtr<AtomicStringImpl> JSRopeString::resolveRopeToExistingAtomicString(ExecState* exec) const
    218252{
     253    VM& vm = exec->vm();
     254    auto scope = DECLARE_THROW_SCOPE(vm);
     255
    219256    if (length() > maxLengthForOnStackResolve) {
    220         resolveRope(exec);
    221         if (RefPtr<AtomicStringImpl> existingAtomicString = AtomicStringImpl::lookUp(m_value.impl())) {
    222             m_value = *existingAtomicString;
    223             setIs8Bit(m_value.impl()->is8Bit());
    224             clearFibers();
    225             return existingAtomicString;
    226         }
    227         return nullptr;
     257        RefPtr<AtomicStringImpl> existingAtomicString;
     258        resolveRopeWithFunction(exec, [&] (Ref<StringImpl>&& newImpl) -> Ref<StringImpl> {
     259            existingAtomicString = AtomicStringImpl::lookUp(newImpl.ptr());
     260            if (existingAtomicString)
     261                return makeRef(*existingAtomicString);
     262            return WTFMove(newImpl);
     263        });
     264        RETURN_IF_EXCEPTION(scope, nullptr);
     265        return existingAtomicString;
    228266    }
    229267   
    … …  
    232270        resolveRopeInternal8(buffer);
    233271        if (RefPtr<AtomicStringImpl> existingAtomicString = AtomicStringImpl::lookUp(buffer, length())) {
    234             m_value = *existingAtomicString;
    235             setIs8Bit(m_value.impl()->is8Bit());
    236             clearFibers();
     272            convertToNonRope(*existingAtomicString);
    237273            return existingAtomicString;
    238274        }
    … …  
    241277        resolveRopeInternal16(buffer);
    242278        if (RefPtr<AtomicStringImpl> existingAtomicString = AtomicStringImpl::lookUp(buffer, length())) {
    243             m_value = *existingAtomicString;
    244             setIs8Bit(m_value.impl()->is8Bit());
    245             clearFibers();
     279            convertToNonRope(*existingAtomicString);
    246280            return existingAtomicString;
    247281        }
    … …  
    251285}
    252286
    253 void JSRopeString::resolveRope(ExecState* nullOrExecForOOM) const
     287template<typename Function>
     288const String& JSRopeString::resolveRopeWithFunction(ExecState* nullOrExecForOOM, Function&& function) const
    254289{
    255290    ASSERT(isRope());
    256291   
     292    VM& vm = *this->vm();
    257293    if (isSubstring()) {
    258294        ASSERT(!substringBase()->isRope());
    259         m_value = substringBase()->m_value.substringSharingImpl(substringOffset(), length());
    260         substringBase().clear();
    261         return;
     295        auto newImpl = substringBase()->valueInternal().substringSharingImpl(substringOffset(), length());
     296        convertToNonRope(function(newImpl.releaseImpl().releaseNonNull()));
     297        return valueInternal();
    262298    }
    263299   
    264300    if (is8Bit()) {
    265301        LChar* buffer;
    266         if (auto newImpl = StringImpl::tryCreateUninitialized(length(), buffer)) {
    267             Heap::heap(this)->reportExtraMemoryAllocated(newImpl->cost());
    268             m_value = WTFMove(newImpl);
    269         } else {
     302        auto newImpl = StringImpl::tryCreateUninitialized(length(), buffer);
     303        if (!newImpl) {
    270304            outOfMemory(nullOrExecForOOM);
    271             return;
    272         }
     305            return nullString();
     306        }
     307        vm.heap.reportExtraMemoryAllocated(newImpl->cost());
     308
    273309        resolveRopeInternal8NoSubstring(buffer);
    274         clearFibers();
    275         ASSERT(!isRope());
    276         return;
     310        convertToNonRope(function(newImpl.releaseNonNull()));
     311        return valueInternal();
    277312    }
    278313   
    279314    UChar* buffer;
    280     if (auto newImpl = StringImpl::tryCreateUninitialized(length(), buffer)) {
    281         Heap::heap(this)->reportExtraMemoryAllocated(newImpl->cost());
    282         m_value = WTFMove(newImpl);
    283     } else {
     315    auto newImpl = StringImpl::tryCreateUninitialized(length(), buffer);
     316    if (!newImpl) {
    284317        outOfMemory(nullOrExecForOOM);
    285         return;
    286     }
     318        return nullString();
     319    }
     320    vm.heap.reportExtraMemoryAllocated(newImpl->cost());
    287321   
    288322    resolveRopeInternal16NoSubstring(buffer);
    289     clearFibers();
    290     ASSERT(!isRope());
     323    convertToNonRope(function(newImpl.releaseNonNull()));
     324    return valueInternal();
     325}
     326
     327const String& JSRopeString::resolveRope(ExecState* nullOrExecForOOM) const
     328{
     329    return resolveRopeWithFunction(nullOrExecForOOM, [] (Ref<StringImpl>&& newImpl) {
     330        return WTFMove(newImpl);
     331    });
    291332}
    292333
    … …  
    307348   
    308349    for (size_t i = 0; i < s_maxInternalRopeLength && fiber(i); ++i)
    309         workQueue.append(fiber(i).get());
     350        workQueue.append(fiber(i));
    310351
    311352    while (!workQueue.isEmpty()) {
    … …  
    319360            if (!currentFiberAsRope->isSubstring()) {
    320361                for (size_t i = 0; i < s_maxInternalRopeLength && currentFiberAsRope->fiber(i); ++i)
    321                     workQueue.append(currentFiberAsRope->fiber(i).get());
     362                    workQueue.append(currentFiberAsRope->fiber(i));
    322363                continue;
    323364            }
    324365            ASSERT(!currentFiberAsRope->substringBase()->isRope());
    325366            characters =
    326                 currentFiberAsRope->substringBase()->m_value.characters8() +
     367                currentFiberAsRope->substringBase()->valueInternal().characters8() +
    327368                currentFiberAsRope->substringOffset();
    328369        } else
    329             characters = currentFiber->m_value.characters8();
     370            characters = currentFiber->valueInternal().characters8();
    330371       
    331372        unsigned length = currentFiber->length();
    … …  
    343384
    344385    for (size_t i = 0; i < s_maxInternalRopeLength && fiber(i); ++i)
    345         workQueue.append(fiber(i).get());
     386        workQueue.append(fiber(i));
    346387
    347388    while (!workQueue.isEmpty()) {
    … …  
    354395                ASSERT(!currentFiberAsRope->substringBase()->isRope());
    355396                StringImpl* string = static_cast<StringImpl*>(
    356                     currentFiberAsRope->substringBase()->m_value.impl());
     397                    currentFiberAsRope->substringBase()->valueInternal().impl());
    357398                unsigned offset = currentFiberAsRope->substringOffset();
    358399                unsigned length = currentFiberAsRope->length();
    … …  
    365406            }
    366407            for (size_t i = 0; i < s_maxInternalRopeLength && currentFiberAsRope->fiber(i); ++i)
    367                 workQueue.append(currentFiberAsRope->fiber(i).get());
     408                workQueue.append(currentFiberAsRope->fiber(i));
    368409            continue;
    369410        }
    370411
    371         StringImpl* string = static_cast<StringImpl*>(currentFiber->m_value.impl());
     412        StringImpl* string = static_cast<StringImpl*>(currentFiber->valueInternal().impl());
    372413        unsigned length = string->length();
    373414        position -= length;
    … …  
    383424void JSRopeString::outOfMemory(ExecState* nullOrExecForOOM) const
    384425{
    385     clearFibers();
    386426    ASSERT(isRope());
    387     ASSERT(m_value.isNull());
    388427    if (nullOrExecForOOM) {
    389428        VM& vm = nullOrExecForOOM->vm();

  • trunk/Source/JavaScriptCore/runtime/JSString.h

    r242123 r242252  
    2323#pragma once
    2424
     25#include "ArgList.h"
    2526#include "CallFrame.h"
    2627#include "CommonIdentifiers.h"
    … …  
    3233#include <array>
    3334#include <wtf/CheckedArithmetic.h>
     35#include <wtf/ForbidHeapAllocation.h>
    3436#include <wtf/text/StringView.h>
    3537
    … …  
    6264JSString* jsOwnedString(ExecState*, const String&);
    6365
    64 JSRopeString* jsStringBuilder(VM*);
    65 
    6666bool isJSString(JSCell*);
    6767bool isJSString(JSValue);
    … …  
    7373};
    7474
     75
     76// In 64bit architecture, JSString and JSRopeString have the following memory layout to make sizeof(JSString) == 16 and sizeof(JSRopeString) == 32.
     77// JSString has only one pointer. We use it for String. length() and is8Bit() queries go to StringImpl. In JSRopeString, we reuse the above pointer
     78// place for the 1st fiber. JSRopeString has three fibers so its size is 48. To keep length and is8Bit flag information in JSRopeString, JSRopeString
     79// encodes these information into the fiber pointers. is8Bit flag is encoded in the 1st fiber pointer. length is embedded directly, and two fibers
     80// are compressed into 12bytes. isRope information is encoded in the first fiber's LSB.
     81//
     82// Since length of JSRopeString should be frequently accessed compared to each fiber, we put length in contiguous 32byte field, and compress 2nd
     83// and 3rd fibers into the following 80byte fields. One problem is that now 2nd and 3rd fibers are split. Storing and loading 2nd and 3rd fibers
     84// are not one pointer load operation. To make concurrent collector work correctly, we must initialize 2nd and 3rd fibers at JSRopeString creation
     85// and we must not modify these part later.
     86//
     87//              0                        8        10               16                       32                                     48
     88// JSString     [   ID      ][  header  ][   String pointer      0]
     89// JSRopeString [   ID      ][  header  ][ flags ][ 1st fiber    1][  length  ][2nd lower32][2nd upper16][3rd lower16][3rd upper32]
     90//                                                               ^
     91//                                                            isRope bit
    7592class JSString : public JSCell {
    7693public:
    … …  
    8198    friend class MarkStack;
    8299    friend class SlotVisitor;
     100    friend class SmallStrings;
    83101
    84102    typedef JSCell Base;
    … …  
    102120    static_assert(MaxLength == String::MaxLength, "");
    103121
     122    static constexpr uintptr_t isRopeInPointer = 0x1;
     123
    104124private:
     125    String& uninitializedValueInternal() const
     126    {
     127        return *bitwise_cast<String*>(&m_fiber);
     128    }
     129
     130    String& valueInternal() const
     131    {
     132        ASSERT(!isRope());
     133        return uninitializedValueInternal();
     134    }
     135
    105136    JSString(VM& vm, Ref<StringImpl>&& value)
    106137        : JSCell(vm, vm.stringStructure.get())
    107         , m_value(WTFMove(value))
    108     {
     138    {
     139        new (&uninitializedValueInternal()) String(WTFMove(value));
    109140    }
    110141
    111142    JSString(VM& vm)
    112143        : JSCell(vm, vm.stringStructure.get())
     144        , m_fiber(isRopeInPointer)
    113145    {
    114146    }
    … …  
    116148    void finishCreation(VM& vm, unsigned length)
    117149    {
    118         ASSERT(!m_value.isNull());
     150        ASSERT_UNUSED(length, length > 0);
     151        ASSERT(!valueInternal().isNull());
    119152        Base::finishCreation(vm);
    120         setLength(length);
    121         setIs8Bit(m_value.impl()->is8Bit());
    122153    }
    123154
    124155    void finishCreation(VM& vm, unsigned length, size_t cost)
    125156    {
    126         ASSERT(!m_value.isNull());
     157        ASSERT_UNUSED(length, length > 0);
     158        ASSERT(!valueInternal().isNull());
    127159        Base::finishCreation(vm);
    128         setLength(length);
    129         setIs8Bit(m_value.impl()->is8Bit());
    130160        vm.heap.reportExtraMemoryAllocated(cost);
    131161    }
    132162
    133 protected:
    134     void finishCreation(VM& vm)
    135     {
    136         Base::finishCreation(vm);
    137         setLength(0);
    138         setIs8Bit(true);
    139     }
    140 
    141 public:
     163    static JSString* createEmptyString(VM&);
     164
    142165    static JSString* create(VM& vm, Ref<StringImpl>&& value)
    143166    {
    144167        unsigned length = value->length();
     168        ASSERT(length > 0);
    145169        size_t cost = value->cost();
    146170        JSString* newString = new (NotNull, allocateCell<JSString>(vm.heap)) JSString(vm, WTFMove(value));
    … …  
    156180    }
    157181
     182protected:
     183    void finishCreation(VM& vm)
     184    {
     185        Base::finishCreation(vm);
     186    }
     187
     188public:
     189    ~JSString();
     190
    158191    Identifier toIdentifier(ExecState*) const;
    159192    AtomicString toAtomicString(ExecState*) const;
    … …  
    166199    inline const String& tryGetValue(bool allocationAllowed = true) const;
    167200    const StringImpl* tryGetValueImpl() const;
    168     ALWAYS_INLINE unsigned length() const { return m_length; }
     201    ALWAYS_INLINE unsigned length() const;
    169202
    170203    JSValue toPrimitive(ExecState*, PreferredPrimitiveType) const;
    … …  
    183216    static Structure* createStructure(VM&, JSGlobalObject*, JSValue);
    184217
    185     static size_t offsetOfLength() { return OBJECT_OFFSETOF(JSString, m_length); }
    186     static size_t offsetOfFlags() { return OBJECT_OFFSETOF(JSString, m_flags); }
    187     static size_t offsetOfValue() { return OBJECT_OFFSETOF(JSString, m_value); }
     218    static ptrdiff_t offsetOfValue() { return OBJECT_OFFSETOF(JSString, m_fiber); }
    188219
    189220    DECLARE_EXPORT_INFO;
    … …  
    193224    static void visitChildren(JSCell*, SlotVisitor&);
    194225
    195     enum {
    196         Is8Bit = 1u
    197     };
    198 
    199     bool isRope() const { return m_value.isNull(); }
     226    ALWAYS_INLINE bool isRope() const
     227    {
     228        return m_fiber & isRopeInPointer;
     229    }
     230
     231    bool is8Bit() const;
    200232
    201233protected:
    … …  
    204236    JS_EXPORT_PRIVATE bool equalSlowCase(ExecState*, JSString* other) const;
    205237    bool isSubstring() const;
    206     bool is8Bit() const { return m_flags & Is8Bit; }
    207     void setIs8Bit(bool flag) const
    208     {
    209         if (flag)
    210             m_flags |= Is8Bit;
    211         else
    212             m_flags &= ~Is8Bit;
    213     }
    214 
    215     ALWAYS_INLINE void setLength(unsigned length)
    216     {
    217         ASSERT(length <= MaxLength);
    218         m_length = length;
    219     }
     238
     239    mutable uintptr_t m_fiber;
    220240
    221241private:
    222     // A string is represented either by a String or a rope of fibers.
    223     unsigned m_length { 0 };
    224     mutable uint16_t m_flags { 0 };
    225     mutable String m_value;
    226 
    227242    friend class LLIntOffsetsExtractor;
    228243
    229244    static JSValue toThis(JSCell*, ExecState*, ECMAMode);
    230245
    231     String& string() { ASSERT(!isRope()); return m_value; }
    232246    StringView unsafeView(ExecState*) const;
    233247
     248    friend JSString* jsString(VM*, const String&);
    234249    friend JSString* jsString(ExecState*, JSString*, JSString*);
    235     friend JSString* jsSubstring(ExecState*, JSString*, unsigned offset, unsigned length);
     250    friend JSString* jsString(ExecState*, const String&, JSString*);
     251    friend JSString* jsString(ExecState*, JSString*, const String&);
     252    friend JSString* jsString(ExecState*, const String&, const String&);
     253    friend JSString* jsString(ExecState*, JSString*, JSString*, JSString*);
     254    friend JSString* jsString(ExecState*, const String&, const String&, const String&);
     255    friend JSString* jsSingleCharacterString(VM*, UChar);
     256    friend JSString* jsNontrivialString(VM*, const String&);
     257    friend JSString* jsNontrivialString(VM*, String&&);
     258    friend JSString* jsSubstring(VM*, const String&, unsigned, unsigned);
     259    friend JSString* jsSubstring(VM&, ExecState*, JSString*, unsigned, unsigned);
     260    friend JSString* jsSubstringOfResolved(VM&, GCDeferralContext*, JSString*, unsigned, unsigned);
     261    friend JSString* jsOwnedString(VM*, const String&);
    236262};
    237263
    … …  
    240266class JSRopeString final : public JSString {
    241267    friend class JSString;
    242 
    243     friend JSRopeString* jsStringBuilder(VM*);
    244 
    245268public:
     269#if CPU(ADDRESS64)
     270    static_assert(sizeof(uintptr_t) == sizeof(uint64_t), "");
     271    static constexpr uintptr_t flagMask = 0xffff000000000000ULL;
     272    static constexpr uintptr_t stringMask = ~(flagMask | isRopeInPointer);
     273    static_assert(StringImpl::flagIs8Bit() == 0b100, "");
     274    static constexpr uintptr_t is8BitInPointer = static_cast<uintptr_t>(StringImpl::flagIs8Bit()) << 48;
     275
     276    class CompactFibers {
     277    public:
     278        JSString* fiber1() const
     279        {
     280            return bitwise_cast<JSString*>(static_cast<uintptr_t>(m_fiber1Lower) | (static_cast<uintptr_t>(m_fiber1Upper) << 32));
     281        }
     282
     283        void initializeFiber1(JSString* fiber)
     284        {
     285            uintptr_t pointer = bitwise_cast<uintptr_t>(fiber);
     286            m_fiber1Lower = static_cast<uint32_t>(pointer);
     287            m_fiber1Upper = static_cast<uint16_t>(pointer >> 32);
     288        }
     289
     290        JSString* fiber2() const
     291        {
     292            return bitwise_cast<JSString*>(static_cast<uintptr_t>(m_fiber2Lower) | (static_cast<uintptr_t>(m_fiber2Upper) << 32));
     293        }
     294        void initializeFiber2(JSString* fiber)
     295        {
     296            uintptr_t pointer = bitwise_cast<uintptr_t>(fiber);
     297            m_fiber2Lower = static_cast<uint32_t>(pointer);
     298            m_fiber2Upper = static_cast<uint16_t>(pointer >> 32);
     299        }
     300
     301        unsigned length() const { return m_length; }
     302        void initializeLength(unsigned length)
     303        {
     304            m_length = length;
     305        }
     306
     307        static ptrdiff_t offsetOfLength() { return OBJECT_OFFSETOF(CompactFibers, m_length); }
     308        static ptrdiff_t offsetOfFiber1Lower() { return OBJECT_OFFSETOF(CompactFibers, m_fiber1Lower); }
     309        static ptrdiff_t offsetOfFiber1Upper() { return OBJECT_OFFSETOF(CompactFibers, m_fiber1Upper); }
     310        static ptrdiff_t offsetOfFiber2Lower() { return OBJECT_OFFSETOF(CompactFibers, m_fiber2Lower); }
     311        static ptrdiff_t offsetOfFiber2Upper() { return OBJECT_OFFSETOF(CompactFibers, m_fiber2Upper); }
     312
     313    private:
     314        uint32_t m_length { 0 };
     315        uint32_t m_fiber1Lower { 0 };
     316        uint16_t m_fiber1Upper { 0 };
     317        uint16_t m_fiber2Upper { 0 };
     318        uint32_t m_fiber2Lower { 0 };
     319    };
     320    static_assert(sizeof(CompactFibers) == sizeof(void*) * 2, "");
     321#else
     322    static constexpr uintptr_t stringMask = ~(isRopeInPointer);
     323
     324    class CompactFibers {
     325    public:
     326        JSString* fiber1() const
     327        {
     328            return m_fiber1;
     329        }
     330        void initializeFiber1(JSString* fiber)
     331        {
     332            m_fiber1 = fiber;
     333        }
     334
     335        JSString* fiber2() const
     336        {
     337            return m_fiber2;
     338        }
     339        void initializeFiber2(JSString* fiber)
     340        {
     341            m_fiber2 = fiber;
     342        }
     343
     344        unsigned length() const { return m_length; }
     345        void initializeLength(unsigned length)
     346        {
     347            m_length = length;
     348        }
     349
     350        void initializeIs8Bit(bool flag)
     351        {
     352            if (flag)
     353                m_flags |= static_cast<uintptr_t>(StringImpl::flagIs8Bit());
     354            else
     355                m_flags &= ~static_cast<uintptr_t>(StringImpl::flagIs8Bit());
     356        }
     357
     358        bool is8Bit()
     359        {
     360            return m_flags & static_cast<uintptr_t>(StringImpl::flagIs8Bit());
     361        }
     362
     363        static ptrdiff_t offsetOfLength() { return OBJECT_OFFSETOF(CompactFibers, m_length); }
     364
     365    private:
     366        uint32_t m_length { 0 };
     367        uint32_t m_flags { 0 };
     368        JSString* m_fiber1 { nullptr };
     369        JSString* m_fiber2 { nullptr };
     370    };
     371#endif
     372
    246373    template <class OverflowHandler = CrashOnOverflow>
    247374    class RopeBuilder : public OverflowHandler {
     375        WTF_FORBID_HEAP_ALLOCATION;
    248376    public:
    249377        RopeBuilder(VM& vm)
    250378            : m_vm(vm)
    251             , m_jsString(jsStringBuilder(&vm))
    252             , m_index(0)
    253379        {
    254380        }
    … …  
    258384            if (UNLIKELY(this->hasOverflowed()))
    259385                return false;
    260             if (m_index == JSRopeString::s_maxInternalRopeLength)
     386            if (!jsString->length())
     387                return true;
     388            if (m_strings.size() == JSRopeString::s_maxInternalRopeLength)
    261389                expand();
    262390
    263391            static_assert(JSString::MaxLength == std::numeric_limits<int32_t>::max(), "");
    264             auto sum = checkedSum<int32_t>(m_jsString->length(), jsString->length());
     392            auto sum = checkedSum<int32_t>(m_length, jsString->length());
    265393            if (sum.hasOverflowed()) {
    266394                this->overflowed();
    … …  
    268396            }
    269397            ASSERT(static_cast<unsigned>(sum.unsafeGet()) <= MaxLength);
    270             m_jsString->append(m_vm, m_index++, jsString);
     398            m_strings.append(jsString);
     399            m_length = static_cast<unsigned>(sum.unsafeGet());
    271400            return true;
    272401        }
    273402
    274         JSRopeString* release()
     403        JSString* release()
    275404        {
    276405            RELEASE_ASSERT(!this->hasOverflowed());
    277             JSRopeString* tmp = m_jsString;
    278             m_jsString = nullptr;
    279             return tmp;
     406            JSString* result = nullptr;
     407            switch (m_strings.size()) {
     408            case 0: {
     409                ASSERT(!m_length);
     410                result = jsEmptyString(&m_vm);
     411                break;
     412            }
     413            case 1: {
     414                result = asString(m_strings.at(0));
     415                break;
     416            }
     417            case 2: {
     418                result = JSRopeString::create(m_vm, asString(m_strings.at(0)), asString(m_strings.at(1)));
     419                break;
     420            }
     421            case 3: {
     422                result = JSRopeString::create(m_vm, asString(m_strings.at(0)), asString(m_strings.at(1)), asString(m_strings.at(2)));
     423                break;
     424            }
     425            default:
     426                ASSERT_NOT_REACHED();
     427                break;
     428            }
     429            ASSERT(result->length() == m_length);
     430            m_strings.clear();
     431            m_length = 0;
     432            return result;
    280433        }
    281434
    … …  
    283436        {
    284437            ASSERT(!this->hasOverflowed());
    285             return m_jsString->length();
     438            return m_length;
    286439        }
    287440
    … …  
    290443
    291444        VM& m_vm;
    292         JSRopeString* m_jsString;
    293         size_t m_index;
     445        MarkedArgumentBuffer m_strings;
     446        unsigned m_length { 0 };
    294447    };
    295448
     449    inline unsigned length() const
     450    {
     451        return m_compactFibers.length();
     452    }
     453
    296454private:
    297     ALWAYS_INLINE JSRopeString(VM& vm)
     455    void convertToNonRope(String&&) const;
     456
     457    void initializeIs8Bit(bool flag) const
     458    {
     459#if CPU(ADDRESS64)
     460        if (flag)
     461            m_fiber |= is8BitInPointer;
     462        else
     463            m_fiber &= ~is8BitInPointer;
     464#else
     465        m_compactFibers.initializeIs8Bit(flag);
     466#endif
     467    }
     468
     469    ALWAYS_INLINE void initializeLength(unsigned length)
     470    {
     471        ASSERT(length <= MaxLength);
     472        m_compactFibers.initializeLength(length);
     473    }
     474
     475    JSRopeString(VM& vm, JSString* s1, JSString* s2)
    298476        : JSString(vm)
    299477    {
    300     }
    301 
    302     void finishCreation(VM& vm, JSString* s1, JSString* s2)
    303     {
    304         Base::finishCreation(vm);
    305478        ASSERT(!sumOverflows<int32_t>(s1->length(), s2->length()));
    306         setLength(s1->length() + s2->length());
    307         setIs8Bit(s1->is8Bit() && s2->is8Bit());
    308         setIsSubstring(false);
    309         fiber(0).set(vm, this, s1);
    310         fiber(1).set(vm, this, s2);
    311         fiber(2).clear();
    312     }
    313 
    314     void finishCreation(VM& vm, JSString* s1, JSString* s2, JSString* s3)
    315     {
    316         Base::finishCreation(vm);
     479        initializeIsSubstring(false);
     480        initializeLength(s1->length() + s2->length());
     481        initializeIs8Bit(s1->is8Bit() && s2->is8Bit());
     482        initializeFiber0(s1);
     483        initializeFiber1(s2);
     484        initializeFiber2(nullptr);
     485        ASSERT((s1->length() + s2->length()) == length());
     486    }
     487
     488    JSRopeString(VM& vm, JSString* s1, JSString* s2, JSString* s3)
     489        : JSString(vm)
     490    {
    317491        ASSERT(!sumOverflows<int32_t>(s1->length(), s2->length(), s3->length()));
    318         setLength(s1->length() + s2->length() + s3->length());
    319         setIs8Bit(s1->is8Bit() && s2->is8Bit() &&  s3->is8Bit());
    320         setIsSubstring(false);
    321         fiber(0).set(vm, this, s1);
    322         fiber(1).set(vm, this, s2);
    323         fiber(2).set(vm, this, s3);
    324     }
    325 
    326     void finishCreation(VM& vm, ExecState* exec, JSString* base, unsigned offset, unsigned length)
    327     {
    328         Base::finishCreation(vm);
     492        initializeIsSubstring(false);
     493        initializeLength(s1->length() + s2->length() + s3->length());
     494        initializeIs8Bit(s1->is8Bit() && s2->is8Bit() &&  s3->is8Bit());
     495        initializeFiber0(s1);
     496        initializeFiber1(s2);
     497        initializeFiber2(s3);
     498        ASSERT((s1->length() + s2->length() + s3->length()) == length());
     499    }
     500
     501    JSRopeString(VM& vm, JSString* base, unsigned offset, unsigned length)
     502        : JSString(vm)
     503    {
    329504        RELEASE_ASSERT(!sumOverflows<int32_t>(offset, length));
    330505        RELEASE_ASSERT(offset + length <= base->length());
    331         setLength(length);
    332         setIs8Bit(base->is8Bit());
    333         setIsSubstring(true);
     506        initializeIsSubstring(true);
     507        initializeLength(length);
     508        initializeIs8Bit(base->is8Bit());
    334509        if (base->isSubstring()) {
    335510            JSRopeString* baseRope = jsCast<JSRopeString*>(base);
    336             substringBase().set(vm, this, baseRope->substringBase().get());
    337             substringOffset() = baseRope->substringOffset() + offset;
     511            initializeSubstringBase(baseRope->substringBase());
     512            initializeSubstringOffset(baseRope->substringOffset() + offset);
    338513        } else {
    339             substringBase().set(vm, this, base);
    340             substringOffset() = offset;
    341 
    342             // For now, let's not allow substrings with a rope base.
    343             // Resolve non-substring rope bases so we don't have to deal with it.
    344             // FIXME: Evaluate if this would be worth adding more branches.
    345             if (base->isRope())
    346                 jsCast<JSRopeString*>(base)->resolveRope(exec);
    347         }
    348     }
    349 
    350     ALWAYS_INLINE void finishCreationSubstringOfResolved(VM& vm, JSString* base, unsigned offset, unsigned length)
    351     {
    352         Base::finishCreation(vm);
     514            initializeSubstringBase(base);
     515            initializeSubstringOffset(offset);
     516        }
     517        ASSERT(length == this->length());
     518    }
     519
     520    enum SubstringOfResolvedTag { SubstringOfResolved };
     521    JSRopeString(SubstringOfResolvedTag, VM& vm, JSString* base, unsigned offset, unsigned length)
     522        : JSString(vm)
     523    {
    353524        RELEASE_ASSERT(!sumOverflows<int32_t>(offset, length));
    354525        RELEASE_ASSERT(offset + length <= base->length());
    355         setLength(length);
    356         setIs8Bit(base->is8Bit());
    357         setIsSubstring(true);
    358         substringBase().set(vm, this, base);
    359         substringOffset() = offset;
    360     }
    361 
    362     void finishCreation(VM& vm)
    363     {
    364         JSString::finishCreation(vm);
    365         setIsSubstring(false);
    366         fiber(0).clear();
    367         fiber(1).clear();
    368         fiber(2).clear();
    369     }
    370 
    371     void append(VM& vm, size_t index, JSString* jsString)
    372     {
    373         fiber(index).set(vm, this, jsString);
    374         setLength(length() + jsString->length());
    375         setIs8Bit(is8Bit() && jsString->is8Bit());
    376     }
    377 
    378     static JSRopeString* createNull(VM& vm)
    379     {
    380         JSRopeString* newString = new (NotNull, allocateCell<JSRopeString>(vm.heap)) JSRopeString(vm);
     526        initializeIsSubstring(true);
     527        initializeLength(length);
     528        initializeIs8Bit(base->is8Bit());
     529        initializeSubstringBase(base);
     530        initializeSubstringOffset(offset);
     531        ASSERT(length == this->length());
     532    }
     533
     534    ALWAYS_INLINE void finishCreationSubstring(VM& vm, ExecState* exec)
     535    {
     536        Base::finishCreation(vm);
     537        JSString* updatedBase = substringBase();
     538        // For now, let's not allow substrings with a rope base.
     539        // Resolve non-substring rope bases so we don't have to deal with it.
     540        // FIXME: Evaluate if this would be worth adding more branches.
     541        if (updatedBase->isRope())
     542            jsCast<JSRopeString*>(updatedBase)->resolveRope(exec);
     543    }
     544
     545    ALWAYS_INLINE void finishCreationSubstringOfResolved(VM& vm)
     546    {
     547        Base::finishCreation(vm);
     548    }
     549
     550public:
     551    static ptrdiff_t offsetOfLength() { return OBJECT_OFFSETOF(JSRopeString, m_compactFibers) + CompactFibers::offsetOfLength(); } // 32byte width.
     552#if CPU(ADDRESS64)
     553    static ptrdiff_t offsetOfFlags() { return offsetOfValue() + sizeof(uint16_t) * 3; } // 16byte width.
     554    static ptrdiff_t offsetOfFiber0() { return offsetOfValue(); }
     555    static ptrdiff_t offsetOfFiber1Lower() { return OBJECT_OFFSETOF(JSRopeString, m_compactFibers) + CompactFibers::offsetOfFiber1Lower(); } // 32byte width.
     556    static ptrdiff_t offsetOfFiber1Upper() { return OBJECT_OFFSETOF(JSRopeString, m_compactFibers) + CompactFibers::offsetOfFiber1Upper(); } // 16byte width.
     557    static ptrdiff_t offsetOfFiber2Lower() { return OBJECT_OFFSETOF(JSRopeString, m_compactFibers) + CompactFibers::offsetOfFiber2Lower(); } // 32byte width.
     558    static ptrdiff_t offsetOfFiber2Upper() { return OBJECT_OFFSETOF(JSRopeString, m_compactFibers) + CompactFibers::offsetOfFiber2Upper(); } // 16byte width.
     559#endif
     560
     561    static constexpr unsigned s_maxInternalRopeLength = 3;
     562
     563private:
     564    static JSRopeString* create(VM& vm, JSString* s1, JSString* s2)
     565    {
     566        JSRopeString* newString = new (NotNull, allocateCell<JSRopeString>(vm.heap)) JSRopeString(vm, s1, s2);
    381567        newString->finishCreation(vm);
     568        ASSERT(newString->length());
    382569        return newString;
    383570    }
    384 
    385 public:
    386     static JSString* create(VM& vm, ExecState* exec, JSString* base, unsigned offset, unsigned length)
    387     {
    388         JSRopeString* newString = new (NotNull, allocateCell<JSRopeString>(vm.heap)) JSRopeString(vm);
    389         newString->finishCreation(vm, exec, base, offset, length);
     571    static JSRopeString* create(VM& vm, JSString* s1, JSString* s2, JSString* s3)
     572    {
     573        JSRopeString* newString = new (NotNull, allocateCell<JSRopeString>(vm.heap)) JSRopeString(vm, s1, s2, s3);
     574        newString->finishCreation(vm);
     575        ASSERT(newString->length());
    390576        return newString;
    391577    }
    392578
    393     ALWAYS_INLINE static JSString* createSubstringOfResolved(VM& vm, GCDeferralContext* deferralContext, JSString* base, unsigned offset, unsigned length)
    394     {
    395         JSRopeString* newString = new (NotNull, allocateCell<JSRopeString>(vm.heap, deferralContext)) JSRopeString(vm);
    396         newString->finishCreationSubstringOfResolved(vm, base, offset, length);
     579    static JSRopeString* create(VM& vm, ExecState* exec, JSString* base, unsigned offset, unsigned length)
     580    {
     581        JSRopeString* newString = new (NotNull, allocateCell<JSRopeString>(vm.heap)) JSRopeString(vm, base, offset, length);
     582        newString->finishCreationSubstring(vm, exec);
     583        ASSERT(newString->length());
    397584        return newString;
    398585    }
    399586
    400     ALWAYS_INLINE static JSString* createSubstringOfResolved(VM& vm, JSString* base, unsigned offset, unsigned length)
    401     {
    402         return createSubstringOfResolved(vm, nullptr, base, offset, length);
    403     }
    404 
    405     void visitFibers(SlotVisitor&);
    406 
    407     static ptrdiff_t offsetOfFibers() { return OBJECT_OFFSETOF(JSRopeString, u); }
    408 
    409     static const unsigned s_maxInternalRopeLength = 3;
    410 
    411 private:
    412     static JSString* create(VM& vm, JSString* s1, JSString* s2)
    413     {
    414         JSRopeString* newString = new (NotNull, allocateCell<JSRopeString>(vm.heap)) JSRopeString(vm);
    415         newString->finishCreation(vm, s1, s2);
    416         return newString;
    417     }
    418     static JSString* create(VM& vm, JSString* s1, JSString* s2, JSString* s3)
    419     {
    420         JSRopeString* newString = new (NotNull, allocateCell<JSRopeString>(vm.heap)) JSRopeString(vm);
    421         newString->finishCreation(vm, s1, s2, s3);
     587    ALWAYS_INLINE static JSRopeString* createSubstringOfResolved(VM& vm, GCDeferralContext* deferralContext, JSString* base, unsigned offset, unsigned length)
     588    {
     589        JSRopeString* newString = new (NotNull, allocateCell<JSRopeString>(vm.heap, deferralContext)) JSRopeString(SubstringOfResolved, vm, base, offset, length);
     590        newString->finishCreationSubstringOfResolved(vm);
     591        ASSERT(newString->length());
    422592        return newString;
    423593    }
    … …  
    428598    // If nullOrExecForOOM is null, resolveRope() will be do nothing in the event of an OOM error.
    429599    // The rope value will remain a null string in that case.
    430     JS_EXPORT_PRIVATE void resolveRope(ExecState* nullOrExecForOOM) const;
    431     JS_EXPORT_PRIVATE void resolveRopeToAtomicString(ExecState*) const;
     600    JS_EXPORT_PRIVATE const String& resolveRope(ExecState* nullOrExecForOOM) const;
     601    template<typename Function> const String& resolveRopeWithFunction(ExecState* nullOrExecForOOM, Function&&) const;
     602    JS_EXPORT_PRIVATE AtomicString resolveRopeToAtomicString(ExecState*) const;
    432603    JS_EXPORT_PRIVATE RefPtr<AtomicStringImpl> resolveRopeToExistingAtomicString(ExecState*) const;
    433604    void resolveRopeSlowCase8(LChar*) const;
    … …  
    438609    void resolveRopeInternal16(UChar*) const;
    439610    void resolveRopeInternal16NoSubstring(UChar*) const;
    440     void clearFibers() const;
    441611    StringView unsafeView(ExecState*) const;
    442612    StringViewWithUnderlyingString viewWithUnderlyingString(ExecState*) const;
    443613
    444     WriteBarrierBase<JSString>& fiber(unsigned i) const
     614    JSString* fiber0() const
     615    {
     616        return bitwise_cast<JSString*>(m_fiber & stringMask);
     617    }
     618
     619    JSString* fiber1() const
     620    {
     621        return m_compactFibers.fiber1();
     622    }
     623
     624    JSString* fiber2() const
     625    {
     626        return m_compactFibers.fiber2();
     627    }
     628
     629    JSString* fiber(unsigned i) const
    445630    {
    446631        ASSERT(!isSubstring());
    447632        ASSERT(i < s_maxInternalRopeLength);
    448         return u[i].string;
    449     }
    450 
    451     WriteBarrierBase<JSString>& substringBase() const
    452     {
    453         return u[1].string;
    454     }
    455 
    456     uintptr_t& substringOffset() const
    457     {
    458         return u[2].number;
    459     }
    460 
    461     static uintptr_t notSubstringSentinel()
     633        switch (i) {
     634        case 0:
     635            return fiber0();
     636        case 1:
     637            return fiber1();
     638        case 2:
     639            return fiber2();
     640        }
     641        ASSERT_NOT_REACHED();
     642        return nullptr;
     643    }
     644
     645    void initializeFiber0(JSString* fiber)
     646    {
     647        uintptr_t pointer = bitwise_cast<uintptr_t>(fiber);
     648        ASSERT(!(pointer & ~stringMask));
     649        m_fiber = (pointer | (m_fiber & ~stringMask));
     650    }
     651
     652    void initializeFiber1(JSString* fiber)
     653    {
     654        m_compactFibers.initializeFiber1(fiber);
     655    }
     656
     657    void initializeFiber2(JSString* fiber)
     658    {
     659        m_compactFibers.initializeFiber2(fiber);
     660    }
     661
     662    void initializeSubstringBase(JSString* fiber)
     663    {
     664        initializeFiber1(fiber);
     665    }
     666
     667    JSString* substringBase() const { return fiber1(); }
     668
     669    void initializeSubstringOffset(unsigned offset)
     670    {
     671        m_compactFibers.initializeFiber2(bitwise_cast<JSString*>(static_cast<uintptr_t>(offset)));
     672    }
     673
     674    unsigned substringOffset() const
     675    {
     676        return static_cast<unsigned>(bitwise_cast<uintptr_t>(fiber2()));
     677    }
     678
     679    static constexpr uintptr_t notSubstringSentinel()
    462680    {
    463681        return 0;
    464682    }
    465683
    466     static uintptr_t substringSentinel()
    467     {
    468         return 1;
     684    static constexpr uintptr_t substringSentinel()
     685    {
     686        return 2;
    469687    }
    470688
    471689    bool isSubstring() const
    472690    {
    473         return u[0].number == substringSentinel();
    474     }
    475 
    476     void setIsSubstring(bool isSubstring)
    477     {
    478         u[0].number = isSubstring ? substringSentinel() : notSubstringSentinel();
    479     }
    480 
    481     mutable union {
    482         uintptr_t number;
    483         WriteBarrierBase<JSString> string;
    484     } u[s_maxInternalRopeLength];
    485 
     691        return (m_fiber & stringMask) == substringSentinel();
     692    }
     693
     694    void initializeIsSubstring(bool isSubstring)
     695    {
     696        m_fiber |= (isSubstring ? substringSentinel() : notSubstringSentinel());
     697    }
     698
     699    static_assert(s_maxInternalRopeLength >= 2, "");
     700    mutable CompactFibers m_compactFibers;
    486701
    487702    friend JSString* jsString(ExecState*, JSString*, JSString*);
    488703    friend JSString* jsString(ExecState*, const String&, JSString*);
    489704    friend JSString* jsString(ExecState*, JSString*, const String&);
     705    friend JSString* jsString(ExecState*, const String&, const String&);
    490706    friend JSString* jsString(ExecState*, JSString*, JSString*, JSString*);
    491     friend JSString* jsString(ExecState*, const String&, const String&);
    492707    friend JSString* jsString(ExecState*, const String&, const String&, const String&);
     708    friend JSString* jsSubstringOfResolved(VM&, GCDeferralContext*, JSString*, unsigned, unsigned);
     709    friend JSString* jsSubstring(VM&, ExecState*, JSString*, unsigned, unsigned);
    493710};
    494711
    495712JS_EXPORT_PRIVATE JSString* jsStringWithCacheSlowCase(VM&, StringImpl&);
    496713
     714// JSString::is8Bit is safe to be called concurrently. Concurrent threads can access is8Bit even if the main thread
     715// is in the middle of converting JSRopeString to JSString.
     716ALWAYS_INLINE bool JSString::is8Bit() const
     717{
     718    uintptr_t pointer = m_fiber;
     719    if (pointer & isRopeInPointer) {
     720#if CPU(ADDRESS64)
     721        // Do not load m_fiber twice. We should use the information in pointer.
     722        // Otherwise, JSRopeString may be converted to JSString between the first and second accesses.
     723        return pointer & JSRopeString::is8BitInPointer;
     724#else
     725        // It is OK to load flag since even if JSRopeString is converted to JSString, this flag still exists.
     726        return jsCast<const JSRopeString*>(this)->m_compactFibers.is8Bit();
     727#endif
     728    }
     729    return bitwise_cast<StringImpl*>(pointer)->is8Bit();
     730}
     731
     732// JSString::length is safe to be called concurrently. Concurrent threads can access length even if the main thread
     733// is in the middle of converting JSRopeString to JSString. This is OK because we never override the length bits
     734// when we resolve a JSRopeString.
     735ALWAYS_INLINE unsigned JSString::length() const
     736{
     737    uintptr_t pointer = m_fiber;
     738    if (pointer & isRopeInPointer)
     739        return jsCast<const JSRopeString*>(this)->length();
     740    return bitwise_cast<StringImpl*>(pointer)->length();
     741}
     742
    497743inline const StringImpl* JSString::tryGetValueImpl() const
    498744{
    499     return m_value.impl();
     745    uintptr_t pointer = m_fiber;
     746    if (pointer & isRopeInPointer)
     747        return nullptr;
     748    return bitwise_cast<StringImpl*>(pointer);
    500749}
    501750
    … …  
    543792        RELEASE_ASSERT(vm()->heap.expectDoesGC());
    544793    if (isRope())
    545         static_cast<const JSRopeString*>(this)->resolveRopeToAtomicString(exec);
    546     return AtomicString(m_value);
     794        return static_cast<const JSRopeString*>(this)->resolveRopeToAtomicString(exec);
     795    return AtomicString(valueInternal());
    547796}
    548797
    … …  
    553802    if (isRope())
    554803        return static_cast<const JSRopeString*>(this)->resolveRopeToExistingAtomicString(exec);
    555     if (m_value.impl()->isAtomic())
    556         return static_cast<AtomicStringImpl*>(m_value.impl());
    557     return AtomicStringImpl::lookUp(m_value.impl());
     804    if (valueInternal().impl()->isAtomic())
     805        return static_cast<AtomicStringImpl*>(valueInternal().impl());
     806    return AtomicStringImpl::lookUp(valueInternal().impl());
    558807}
    559808
    … …  
    563812        RELEASE_ASSERT(vm()->heap.expectDoesGC());
    564813    if (isRope())
    565         static_cast<const JSRopeString*>(this)->resolveRope(exec);
    566     return m_value;
     814        return static_cast<const JSRopeString*>(this)->resolveRope(exec);
     815    return valueInternal();
    567816}
    568817
    … …  
    574823        if (isRope()) {
    575824            // Pass nullptr for the ExecState so that resolveRope does not throw in the event of an OOM error.
    576             static_cast<const JSRopeString*>(this)->resolveRope(nullptr);
     825            return static_cast<const JSRopeString*>(this)->resolveRope(nullptr);
    577826        }
    578827    } else
    579828        RELEASE_ASSERT(!isRope());
    580     return m_value;
     829    return valueInternal();
    581830}
    582831
    … …  
    669918}
    670919
    671 inline JSRopeString* jsStringBuilder(VM* vm)
    672 {
    673     return JSRopeString::createNull(*vm);
    674 }
    675 
    676920inline JSString* jsEmptyString(ExecState* exec) { return jsEmptyString(&exec->vm()); }
    677921inline JSString* jsString(ExecState* exec, const String& s) { return jsString(&exec->vm(), s); }
    … …  
    754998        RELEASE_ASSERT(vm()->heap.expectDoesGC());
    755999    if (isSubstring()) {
    756         if (is8Bit())
    757             return StringView(substringBase()->m_value.characters8() + substringOffset(), length());
    758         return StringView(substringBase()->m_value.characters16() + substringOffset(), length());
    759     }
    760     resolveRope(exec);
    761     return m_value;
     1000        auto& base = substringBase()->valueInternal();
     1001        if (base.is8Bit())
     1002            return StringView(base.characters8() + substringOffset(), length());
     1003        return StringView(base.characters16() + substringOffset(), length());
     1004    }
     1005    return resolveRope(exec);
    7621006}
    7631007
    … …  
    7671011        RELEASE_ASSERT(vm()->heap.expectDoesGC());
    7681012    if (isSubstring()) {
    769         auto& base = substringBase()->m_value;
    770         if (is8Bit())
     1013        auto& base = substringBase()->valueInternal();
     1014        if (base.is8Bit())
    7711015            return { { base.characters8() + substringOffset(), length() }, base };
    7721016        return { { base.characters16() + substringOffset(), length() }, base };
    7731017    }
    774     resolveRope(exec);
    775     return { m_value, m_value };
     1018    auto& string = resolveRope(exec);
     1019    return { string, string };
    7761020}
    7771021
    … …  
    7821026    if (isRope())
    7831027        return static_cast<const JSRopeString*>(this)->unsafeView(exec);
    784     return m_value;
     1028    return valueInternal();
    7851029}
    7861030
    … …  
    7891033    if (isRope())
    7901034        return static_cast<const JSRopeString&>(*this).viewWithUnderlyingString(exec);
    791     return { m_value, m_value };
     1035    return { valueInternal(), valueInternal() };
    7921036}
    7931037

  • trunk/Source/JavaScriptCore/runtime/JSStringInlines.h

    r242081 r242252  
    3030namespace JSC {
    3131   
     32inline JSString::~JSString()
     33{
     34    if (isRope())
     35        return;
     36    valueInternal().~String();
     37}
     38
    3239bool JSString::equal(ExecState* exec, JSString* other) const
    3340{
    3441    if (isRope() || other->isRope())
    3542        return equalSlowCase(exec, other);
    36     return WTF::equal(*m_value.impl(), *other->m_value.impl());
     43    return WTF::equal(*valueInternal().impl(), *other->valueInternal().impl());
    3744}
    3845

  • trunk/Source/JavaScriptCore/runtime/ObjectPrototype.cpp

    r240327 r242252  
    333333            RETURN_IF_EXCEPTION(scope, { });
    334334            if (stringTag.isString()) {
    335                 JSRopeString::RopeBuilder<RecordOverflow> ropeBuilder(vm);
    336                 ropeBuilder.append(vm.smallStrings.objectStringStart());
    337                 ropeBuilder.append(asString(stringTag));
    338                 ropeBuilder.append(vm.smallStrings.singleCharacterString(']'));
    339                 if (ropeBuilder.hasOverflowed())
    340                     return throwOutOfMemoryError(exec, scope);
    341 
    342                 JSString* result = ropeBuilder.release();
     335                JSString* result = jsString(exec, vm.smallStrings.objectStringStart(), asString(stringTag), vm.smallStrings.singleCharacterString(']'));
     336                RETURN_IF_EXCEPTION(scope, { });
    343337                thisObject->structure(vm)->setObjectToStringValue(exec, vm, result, toStringTagSlot);
    344338                return result;

  • trunk/Source/JavaScriptCore/runtime/RegExpMatchesArray.h

    r241927 r242252  
    121121            JSValue value;
    122122            if (start >= 0)
    123                 value = JSRopeString::createSubstringOfResolved(vm, &deferralContext, input, start, subpatternResults[2 * i + 1] - start);
     123                value = jsSubstringOfResolved(vm, &deferralContext, input, start, subpatternResults[2 * i + 1] - start);
    124124            else
    125125                value = jsUndefined();
    … …  
    145145            JSValue value;
    146146            if (start >= 0)
    147                 value = JSRopeString::createSubstringOfResolved(vm, &deferralContext, input, start, subpatternResults[2 * i + 1] - start);
     147                value = jsSubstringOfResolved(vm, &deferralContext, input, start, subpatternResults[2 * i + 1] - start);
    148148            else
    149149                value = jsUndefined();

  • trunk/Source/JavaScriptCore/runtime/RegExpObjectInlines.h

    r240593 r242252  
    163163        size_t end = result.end;
    164164        size_t length = end - result.start;
    165         array->putDirectIndex(exec, arrayIndex++, JSRopeString::createSubstringOfResolved(vm, string, result.start, length));
     165        array->putDirectIndex(exec, arrayIndex++, jsSubstringOfResolved(vm, string, result.start, length));
    166166        if (UNLIKELY(scope.exception())) {
    167167            hasException = true;

  • trunk/Source/JavaScriptCore/runtime/RegExpPrototype.cpp

    r240593 r242252  
    654654        },
    655655        [&] (bool isDefined, unsigned start, unsigned length) -> SplitControl {
    656             result->putDirectIndex(exec, resultLength++, isDefined ? JSRopeString::createSubstringOfResolved(vm, inputString, start, length) : jsUndefined());
     656            result->putDirectIndex(exec, resultLength++, isDefined ? jsSubstringOfResolved(vm, inputString, start, length) : jsUndefined());
    657657            RETURN_IF_EXCEPTION(scope, AbortSplit);
    658658            if (resultLength >= limit)
    … …  
    668668        // 21. Perform ! CreateDataProperty(A, ! ToString(lengthA), T).
    669669        scope.release();
    670         result->putDirectIndex(exec, resultLength, JSRopeString::createSubstringOfResolved(vm, inputString, position, inputSize - position));
     670        result->putDirectIndex(exec, resultLength, jsSubstringOfResolved(vm, inputString, position, inputSize - position));
    671671       
    672672        // 22. Return A.
    … …  
    707707        },
    708708        [&] (bool isDefined, unsigned start, unsigned length) -> SplitControl {
    709             result->putDirectIndex(exec, resultLength++, isDefined ? JSRopeString::createSubstringOfResolved(vm, inputString, start, length) : jsUndefined());
     709            result->putDirectIndex(exec, resultLength++, isDefined ? jsSubstringOfResolved(vm, inputString, start, length) : jsUndefined());
    710710            RETURN_IF_EXCEPTION(scope, AbortSplit);
    711711            if (resultLength >= limit)
    … …  
    721721    // 21. Perform ! CreateDataProperty(A, ! ToString(lengthA), T).
    722722    scope.release();
    723     result->putDirectIndex(exec, resultLength, JSRopeString::createSubstringOfResolved(vm, inputString, position, inputSize - position));
     723    result->putDirectIndex(exec, resultLength, jsSubstringOfResolved(vm, inputString, position, inputSize - position));
    724724    // 22. Return A.
    725725    return JSValue::encode(result);

  • trunk/Source/JavaScriptCore/runtime/SmallStrings.cpp

    r241955 r242252  
    4545void SmallStrings::initializeCommonStrings(VM& vm)
    4646{
    47     createEmptyString(&vm);
     47    ASSERT(!m_emptyString);
     48    m_emptyString = JSString::createEmptyString(vm);
     49    ASSERT(m_needsToBeVisited);
    4850
    4951    for (unsigned i = 0; i < singleCharacterStringCount; ++i) {
    … …  
    8284}
    8385
    84 void SmallStrings::createEmptyString(VM* vm)
    85 {
    86     ASSERT(!m_emptyString);
    87     m_emptyString = JSString::createHasOtherOwner(*vm, *StringImpl::empty());
    88     ASSERT(m_needsToBeVisited);
    89 }
    90 
    9186Ref<StringImpl> SmallStrings::singleCharacterStringRep(unsigned char character)
    9287{

  • trunk/Source/JavaScriptCore/runtime/SmallStrings.h

    r241954 r242252  
    128128    static const unsigned singleCharacterStringCount = maxSingleCharacterString + 1;
    129129
    130     void createEmptyString(VM*);
    131 
    132130    void initialize(VM*, JSString*&, const char* value);
    133131

  • trunk/Source/JavaScriptCore/runtime/StringPrototype.cpp

    r242081 r242252  
    4646#include "RegExpConstructor.h"
    4747#include "RegExpGlobalDataInlines.h"
     48#include "StringPrototypeInlines.h"
    4849#include "SuperSampler.h"
    4950#include <algorithm>
    … …  
    11421143    RETURN_IF_EXCEPTION(scope, encodedJSValue());
    11431144
     1145    JSValue a0 = exec->argument(0);
     1146    JSValue a1 = exec->argument(1);
     1147
    11441148    int len = s.length();
    11451149    RELEASE_ASSERT(len >= 0);
    11461150
    1147     JSValue a0 = exec->argument(0);
    1148     JSValue a1 = exec->argument(1);
    1149 
    11501151    // The arg processing is very much like ArrayProtoFunc::Slice
    11511152    double start = a0.toInteger(exec);
    … …  
    11531154    double end = a1.isUndefined() ? len : a1.toInteger(exec);
    11541155    RETURN_IF_EXCEPTION(scope, encodedJSValue());
    1155     double from = start < 0 ? len + start : start;
    1156     double to = end < 0 ? len + end : end;
    1157     if (to > from && to > 0 && from < len) {
    1158         if (from < 0)
    1159             from = 0;
    1160         if (to > len)
    1161             to = len;
    1162         return JSValue::encode(jsSubstring(exec, s, static_cast<unsigned>(from), static_cast<unsigned>(to) - static_cast<unsigned>(from)));
    1163     }
    1164 
    1165     return JSValue::encode(jsEmptyString(exec));
     1156    scope.release();
     1157    return JSValue::encode(stringSlice(exec, WTFMove(s), start, end));
    11661158}
    11671159

  • trunk/Source/WTF/ChangeLog

    r242235 r242252  
     12019-02-28  Yusuke Suzuki  <ysuzuki@apple.com>
     2
     3        [JSC] sizeof(JSString) should be 16
     4        https://bugs.webkit.org/show_bug.cgi?id=194375
     5
     6        Reviewed by Saam Barati.
     7
     8        * wtf/text/StringImpl.h:
     9        (WTF::StringImpl::flagIs8Bit):
     10        (WTF::StringImpl::flagIsAtomic):
     11        (WTF::StringImpl::flagIsSymbol):
     12        (WTF::StringImpl::maskStringKind):
     13        * wtf/text/WTFString.cpp:
     14        (WTF::nullString):
     15        * wtf/text/WTFString.h:
     16
    1172019-02-28  Mark Lam  <mark.lam@apple.com>
    218

  • trunk/Source/WTF/wtf/text/StringImpl.h

    r241493 r242252  
    201201    static constexpr const unsigned s_hashFlagStringKindIsSymbol = 1u << (s_flagStringKindCount + 1);
    202202    static constexpr const unsigned s_hashMaskStringKind = s_hashFlagStringKindIsAtomic | s_hashFlagStringKindIsSymbol;
    203     static constexpr const unsigned s_hashFlag8BitBuffer = 1u << 3;
    204     static constexpr const unsigned s_hashFlagDidReportCost = 1u << 2;
     203    static constexpr const unsigned s_hashFlagDidReportCost = 1u << 3;
     204    static constexpr const unsigned s_hashFlag8BitBuffer = 1u << 2;
    205205    static constexpr const unsigned s_hashMaskBufferOwnership = (1u << 0) | (1u << 1);
    206206
    … …  
    265265
    266266    static unsigned flagsOffset() { return OBJECT_OFFSETOF(StringImpl, m_hashAndFlags); }
    267     static unsigned flagIs8Bit() { return s_hashFlag8BitBuffer; }
    268     static unsigned flagIsAtomic() { return s_hashFlagStringKindIsAtomic; }
    269     static unsigned flagIsSymbol() { return s_hashFlagStringKindIsSymbol; }
    270     static unsigned maskStringKind() { return s_hashMaskStringKind; }
     267    static constexpr unsigned flagIs8Bit() { return s_hashFlag8BitBuffer; }
     268    static constexpr unsigned flagIsAtomic() { return s_hashFlagStringKindIsAtomic; }
     269    static constexpr unsigned flagIsSymbol() { return s_hashFlagStringKindIsSymbol; }
     270    static constexpr unsigned maskStringKind() { return s_hashMaskStringKind; }
    271271    static unsigned dataOffset() { return OBJECT_OFFSETOF(StringImpl, m_data8); }
    272272

  • trunk/Source/WTF/wtf/text/WTFString.cpp

    r242189 r242252  
    11221122}
    11231123
     1124const String& nullString()
     1125{
     1126    static NeverDestroyed<String> nullString;
     1127    return nullString;
     1128}
     1129
    11241130} // namespace WTF
    11251131

  • trunk/Source/WTF/wtf/text/WTFString.h

    r242189 r242252  
    424424template<typename CharacterType> void appendNumber(Vector<CharacterType>&, unsigned char number);
    425425
    426 // Shared global empty string.
     426// Shared global empty and null string.
    427427WTF_EXPORT_PRIVATE const String& emptyString();
     428WTF_EXPORT_PRIVATE const String& nullString();
    428429
    429430template<typename> struct DefaultHash;
    … …  
    659660using WTF::charactersToUIntStrict;
    660661using WTF::emptyString;
     662using WTF::nullString;
    661663using WTF::equal;
    662664using WTF::find;
Note: See TracChangeset for help on using the changeset viewer.