⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 211319 in webkit


Ignore:
Timestamp:
Jan 27, 2017, 7:09:12 PM (10 years ago)
Author:
Yusuke Suzuki
Message:

Lift template escape sequence restrictions in tagged templates
​https://bugs.webkit.org/show_bug.cgi?id=166871

Reviewed by Saam Barati.

JSTests:

Update the error messages and add new tests.

  • ChakraCore/test/es6/unicode_6_identifier_Blue524737.baseline-jsc:
  • stress/lift-template-literal.js: Added.

(dump):
(testTag.return.tag):
(testTag):

  • stress/template-literal-syntax.js:

Source/JavaScriptCore:

This patch implements stage 3 Lifting Template Literal Restriction[1].
Prior to this patch, template literal becomes syntax error if it contains
invalid escape sequences. But it is too restricted; Template literal
can have cooked and raw representations and only cooked representation
can escape sequences. So even if invalid escape sequences are included,
the raw representation can be valid.

Lifting Template Literal Restriction relaxes the above restriction.
When invalid escape sequence is included, if target template literals
are used as tagged templates, we make the result of the template including
the invalid escape sequence undefined instead of making it SyntaxError
immediately. It allows us to accept the templates including invalid
escape sequences in the raw representations in tagged templates.

On the other hand, the raw representation is only used in tagged templates.
So if invalid escape sequences are included in the usual template literals,
we just make it SyntaxError as before.

[1]: ​https://github.com/tc39/proposal-template-literal-revision

  • bytecompiler/BytecodeGenerator.cpp:

(JSC::BytecodeGenerator::emitGetTemplateObject):

  • bytecompiler/NodesCodegen.cpp:

(JSC::TemplateStringNode::emitBytecode):
(JSC::TemplateLiteralNode::emitBytecode):

  • parser/ASTBuilder.h:

(JSC::ASTBuilder::createTemplateString):

  • parser/Lexer.cpp:

(JSC::Lexer<CharacterType>::parseUnicodeEscape):
(JSC::Lexer<T>::parseTemplateLiteral):
(JSC::Lexer<T>::lex):
(JSC::Lexer<T>::scanTemplateString):
(JSC::Lexer<T>::scanTrailingTemplateString): Deleted.

  • parser/Lexer.h:
  • parser/NodeConstructors.h:

(JSC::TemplateStringNode::TemplateStringNode):

  • parser/Nodes.h:

(JSC::TemplateStringNode::cooked):
(JSC::TemplateStringNode::raw):

  • parser/Parser.cpp:

(JSC::Parser<LexerType>::parseAssignmentElement):
(JSC::Parser<LexerType>::parseTemplateString):
(JSC::Parser<LexerType>::parseTemplateLiteral):
(JSC::Parser<LexerType>::parsePrimaryExpression):
(JSC::Parser<LexerType>::parseMemberExpression):

  • parser/ParserTokens.h:
  • parser/SyntaxChecker.h:

(JSC::SyntaxChecker::createTemplateString):

  • runtime/TemplateRegistry.cpp:

(JSC::TemplateRegistry::getTemplateObject):

  • runtime/TemplateRegistryKey.h:

(JSC::TemplateRegistryKey::cookedStrings):
(JSC::TemplateRegistryKey::create):
(JSC::TemplateRegistryKey::TemplateRegistryKey):

  • runtime/TemplateRegistryKeyTable.cpp:

(JSC::TemplateRegistryKeyTable::createKey):

  • runtime/TemplateRegistryKeyTable.h:

LayoutTests:

Update the error messages.

  • inspector/runtime/parse-expected.txt:
  • js/unicode-escape-sequences-expected.txt:
Location:
trunk
Files:
1 added
21 edited

Legend:

Unmodified
Added
Removed

  • trunk/JSTests/ChakraCore/test/es6/unicode_6_identifier_Blue524737.baseline-jsc

    r205387 r211319  
    1 Exception: SyntaxError: Invalid unicode escape in identifier: '\u{13407'
     1Exception: SyntaxError: Invalid unicode escape in identifier: '\u{134071'
    22at unicode_6_identifier_Blue524737.js:6

  • trunk/JSTests/ChangeLog

    r211246 r211319  
     12017-01-27  Yusuke Suzuki  <utatane.tea@gmail.com>
     2
     3        Lift template escape sequence restrictions in tagged templates
     4        https://bugs.webkit.org/show_bug.cgi?id=166871
     5
     6        Reviewed by Saam Barati.
     7
     8        Update the error messages and add new tests.
     9
     10        * ChakraCore/test/es6/unicode_6_identifier_Blue524737.baseline-jsc:
     11        * stress/lift-template-literal.js: Added.
     12        (dump):
     13        (testTag.return.tag):
     14        (testTag):
     15        * stress/template-literal-syntax.js:
     16
    1172017-01-26  Mark Lam  <mark.lam@apple.com>
    218

  • trunk/JSTests/stress/template-literal-syntax.js

    r183559 r211319  
    4747testSyntax("`\\\r\n`");
    4848testSyntax("`\\\r`");
     49testSyntax("Hello`bad escape sequence: \\unicode`");
     50testSyntax("Hello`\\00`");
     51testSyntax("Hello`\\01`");
     52testSyntax("Hello`\\1`");
     53testSyntax("Hello`\\xo`");
     54testSyntax("Hello`\\x0o`");
     55testSyntax("Hello`\\uo`");
     56testSyntax("Hello`\\u0o`");
     57testSyntax("Hello`\\u00o`");
     58testSyntax("Hello`\\u000o`");
     59testSyntax("Hello`\\u{o`");
     60testSyntax("Hello`\\u{0o`");
     61testSyntax("Hello`\\u{110000o`");
    4962
    5063testSyntaxError("`Hello", "SyntaxError: Unexpected EOF");
    … …  
    8295testSyntaxError("`\\u202`", "SyntaxError: \\u can only be followed by a Unicode character sequence");
    8396testSyntaxError("`${expr}\\u202`", "SyntaxError: \\u can only be followed by a Unicode character sequence");
     97
     98testSyntaxError("`bad escape sequence: \\unicode`", "SyntaxError: \\u can only be followed by a Unicode character sequence");
     99
     100testSyntaxError("`\\00`", "SyntaxError: The only valid numeric escape in strict mode is '\\0'");
     101testSyntaxError("`\\01`", "SyntaxError: The only valid numeric escape in strict mode is '\\0'");
     102testSyntaxError("`\\1`", "SyntaxError: The only valid numeric escape in strict mode is '\\0'");
     103testSyntaxError("`\\xo`", "SyntaxError: \\x can only be followed by a hex character sequence");
     104testSyntaxError("`\\x0o`", "SyntaxError: \\x can only be followed by a hex character sequence");
     105testSyntaxError("`\\uo`", "SyntaxError: \\u can only be followed by a Unicode character sequence");
     106testSyntaxError("`\\u0o`", "SyntaxError: \\u can only be followed by a Unicode character sequence");
     107testSyntaxError("`\\u00o`", "SyntaxError: \\u can only be followed by a Unicode character sequence");
     108testSyntaxError("`\\u000o`", "SyntaxError: \\u can only be followed by a Unicode character sequence");
     109testSyntaxError("`\\u{o`", "SyntaxError: \\u can only be followed by a Unicode character sequence");
     110testSyntaxError("`\\u{0o`", "SyntaxError: \\u can only be followed by a Unicode character sequence");
     111testSyntaxError("`\\u{110000o`", "SyntaxError: \\u can only be followed by a Unicode character sequence");

  • trunk/LayoutTests/ChangeLog

    r211314 r211319  
     12017-01-27  Yusuke Suzuki  <utatane.tea@gmail.com>
     2
     3        Lift template escape sequence restrictions in tagged templates
     4        https://bugs.webkit.org/show_bug.cgi?id=166871
     5
     6        Reviewed by Saam Barati.
     7
     8        Update the error messages.
     9
     10        * inspector/runtime/parse-expected.txt:
     11        * js/unicode-escape-sequences-expected.txt:
     12
    1132017-01-27  Yusuke Suzuki  <utatane.tea@gmail.com>
    214

  • trunk/LayoutTests/inspector/runtime/parse-expected.txt

    r200429 r211319  
    6868PASS: Should be SyntaxErrorType UnterminatedLiteral.
    6969Source: var \u007
    70             ^~
    71 Error Message: Incomplete unicode escape in identifier: '\u'
    72 Range: {"startOffset":4,"endOffset":6}
     70            ^~~~~
     71Error Message: Incomplete unicode escape in identifier: '\u007'
     72Range: {"startOffset":4,"endOffset":9}
    7373
    7474

  • trunk/LayoutTests/js/unicode-escape-sequences-expected.txt

    r183552 r211319  
    5959PASS codeUnits(function \u{G}(){}.name) threw exception SyntaxError: Invalid unicode escape in identifier: '\u{'.
    6060PASS codeUnits(function \u{1G}(){}.name) threw exception SyntaxError: Invalid unicode escape in identifier: '\u{1'.
    61 PASS codeUnits(function \u{110000}(){}.name) threw exception SyntaxError: Invalid unicode escape in identifier: '\u{11000'.
    62 PASS codeUnits(function \u{1000000}(){}.name) threw exception SyntaxError: Invalid unicode escape in identifier: '\u{100000'.
    63 PASS codeUnits(function \u{100000000000000000000000}(){}.name) threw exception SyntaxError: Invalid unicode escape in identifier: '\u{100000'.
     61PASS codeUnits(function \u{110000}(){}.name) threw exception SyntaxError: Invalid unicode escape in identifier: '\u{110000'.
     62PASS codeUnits(function \u{1000000}(){}.name) threw exception SyntaxError: Invalid unicode escape in identifier: '\u{1000000'.
     63PASS codeUnits(function \u{100000000000000000000000}(){}.name) threw exception SyntaxError: Invalid unicode escape in identifier: '\u{100000000000000000000000'.
    6464PASS codeUnits(function x\u{41}(){}.name.substring(1)) is "0041"
    6565PASS codeUnits(function x\u{10000}(){}.name.substring(1)) is "D800,DC00"
    … …  
    8888PASS codeUnits(function x\u{G}(){}.name.substring(1)) threw exception SyntaxError: Invalid unicode escape in identifier: 'x\u{'.
    8989PASS codeUnits(function x\u{1G}(){}.name.substring(1)) threw exception SyntaxError: Invalid unicode escape in identifier: 'x\u{1'.
    90 PASS codeUnits(function x\u{110000}(){}.name.substring(1)) threw exception SyntaxError: Invalid unicode escape in identifier: 'x\u{11000'.
    91 PASS codeUnits(function x\u{1000000}(){}.name.substring(1)) threw exception SyntaxError: Invalid unicode escape in identifier: 'x\u{100000'.
    92 PASS codeUnits(function x\u{100000000000000000000000}(){}.name.substring(1)) threw exception SyntaxError: Invalid unicode escape in identifier: 'x\u{100000'.
     90PASS codeUnits(function x\u{110000}(){}.name.substring(1)) threw exception SyntaxError: Invalid unicode escape in identifier: 'x\u{110000'.
     91PASS codeUnits(function x\u{1000000}(){}.name.substring(1)) threw exception SyntaxError: Invalid unicode escape in identifier: 'x\u{1000000'.
     92PASS codeUnits(function x\u{100000000000000000000000}(){}.name.substring(1)) threw exception SyntaxError: Invalid unicode escape in identifier: 'x\u{100000000000000000000000'.
    9393PASS successfullyParsed is true
    9494

  • trunk/Source/JavaScriptCore/ChangeLog

    r211316 r211319  
     12017-01-27  Yusuke Suzuki  <utatane.tea@gmail.com>
     2
     3        Lift template escape sequence restrictions in tagged templates
     4        https://bugs.webkit.org/show_bug.cgi?id=166871
     5
     6        Reviewed by Saam Barati.
     7
     8        This patch implements stage 3 Lifting Template Literal Restriction[1].
     9        Prior to this patch, template literal becomes syntax error if it contains
     10        invalid escape sequences. But it is too restricted; Template literal
     11        can have cooked and raw representations and only cooked representation
     12        can escape sequences. So even if invalid escape sequences are included,
     13        the raw representation can be valid.
     14
     15        Lifting Template Literal Restriction relaxes the above restriction.
     16        When invalid escape sequence is included, if target template literals
     17        are used as tagged templates, we make the result of the template including
     18        the invalid escape sequence `undefined` instead of making it SyntaxError
     19        immediately. It allows us to accept the templates including invalid
     20        escape sequences in the raw representations in tagged templates.
     21
     22        On the other hand, the raw representation is only used in tagged templates.
     23        So if invalid escape sequences are included in the usual template literals,
     24        we just make it SyntaxError as before.
     25
     26        [1]: https://github.com/tc39/proposal-template-literal-revision
     27
     28        * bytecompiler/BytecodeGenerator.cpp:
     29        (JSC::BytecodeGenerator::emitGetTemplateObject):
     30        * bytecompiler/NodesCodegen.cpp:
     31        (JSC::TemplateStringNode::emitBytecode):
     32        (JSC::TemplateLiteralNode::emitBytecode):
     33        * parser/ASTBuilder.h:
     34        (JSC::ASTBuilder::createTemplateString):
     35        * parser/Lexer.cpp:
     36        (JSC::Lexer<CharacterType>::parseUnicodeEscape):
     37        (JSC::Lexer<T>::parseTemplateLiteral):
     38        (JSC::Lexer<T>::lex):
     39        (JSC::Lexer<T>::scanTemplateString):
     40        (JSC::Lexer<T>::scanTrailingTemplateString): Deleted.
     41        * parser/Lexer.h:
     42        * parser/NodeConstructors.h:
     43        (JSC::TemplateStringNode::TemplateStringNode):
     44        * parser/Nodes.h:
     45        (JSC::TemplateStringNode::cooked):
     46        (JSC::TemplateStringNode::raw):
     47        * parser/Parser.cpp:
     48        (JSC::Parser<LexerType>::parseAssignmentElement):
     49        (JSC::Parser<LexerType>::parseTemplateString):
     50        (JSC::Parser<LexerType>::parseTemplateLiteral):
     51        (JSC::Parser<LexerType>::parsePrimaryExpression):
     52        (JSC::Parser<LexerType>::parseMemberExpression):
     53        * parser/ParserTokens.h:
     54        * parser/SyntaxChecker.h:
     55        (JSC::SyntaxChecker::createTemplateString):
     56        * runtime/TemplateRegistry.cpp:
     57        (JSC::TemplateRegistry::getTemplateObject):
     58        * runtime/TemplateRegistryKey.h:
     59        (JSC::TemplateRegistryKey::cookedStrings):
     60        (JSC::TemplateRegistryKey::create):
     61        (JSC::TemplateRegistryKey::TemplateRegistryKey):
     62        * runtime/TemplateRegistryKeyTable.cpp:
     63        (JSC::TemplateRegistryKeyTable::createKey):
     64        * runtime/TemplateRegistryKeyTable.h:
     65
    1662017-01-27  Saam Barati  <sbarati@apple.com>
    267

  • trunk/Source/JavaScriptCore/bytecompiler/BytecodeGenerator.cpp

    r210522 r211319  
    42544254{
    42554255    TemplateRegistryKey::StringVector rawStrings;
    4256     TemplateRegistryKey::StringVector cookedStrings;
     4256    TemplateRegistryKey::OptionalStringVector cookedStrings;
    42574257
    42584258    TemplateStringListNode* templateString = taggedTemplate->templateLiteral()->templateStrings();
    42594259    for (; templateString; templateString = templateString->next()) {
    4260         rawStrings.append(templateString->value()->raw().impl());
    4261         cookedStrings.append(templateString->value()->cooked().impl());
     4260        auto* string = templateString->value();
     4261        ASSERT(string->raw());
     4262        rawStrings.append(string->raw()->impl());
     4263        if (!string->cooked())
     4264            cookedStrings.append(std::nullopt);
     4265        else
     4266            cookedStrings.append(string->cooked()->impl());
    42624267    }
    42634268
    42644269    RefPtr<RegisterID> getTemplateObject = emitGetGlobalPrivate(newTemporary(), propertyNames().builtinNames().getTemplateObjectPrivateName());
    42654270    CallArguments arguments(*this, nullptr);
    4266     emitLoad(arguments.thisRegister(), JSValue(addTemplateRegistryKeyConstant(m_vm->templateRegistryKeyTable().createKey(rawStrings, cookedStrings))));
     4271    emitLoad(arguments.thisRegister(), JSValue(addTemplateRegistryKeyConstant(m_vm->templateRegistryKeyTable().createKey(WTFMove(rawStrings), WTFMove(cookedStrings)))));
    42674272    return emitCall(dst, getTemplateObject.get(), NoExpectedFunction, arguments, taggedTemplate->divot(), taggedTemplate->divotStart(), taggedTemplate->divotEnd(), DebuggableCall::No);
    42684273}

  • trunk/Source/JavaScriptCore/bytecompiler/NodesCodegen.cpp

    r210958 r211319  
    247247    if (dst == generator.ignoredResult())
    248248        return nullptr;
    249     return generator.emitLoad(dst, JSValue(generator.addStringConstant(cooked())));
     249    ASSERT(cooked());
     250    return generator.emitLoad(dst, JSValue(generator.addStringConstant(*cooked())));
    250251}
    251252
    … …  
    266267    for (; templateExpression; templateExpression = templateExpression->next(), templateString = templateString->next()) {
    267268        // Evaluate TemplateString.
    268         if (!templateString->value()->cooked().isEmpty()) {
     269        ASSERT(templateString->value()->cooked());
     270        if (!templateString->value()->cooked()->isEmpty()) {
    269271            temporaryRegisters.append(generator.newTemporary());
    270272            generator.emitNode(temporaryRegisters.last().get(), templateString->value());
    … …  
    278280
    279281    // Evaluate tail TemplateString.
    280     if (!templateString->value()->cooked().isEmpty()) {
     282    ASSERT(templateString->value()->cooked());
     283    if (!templateString->value()->cooked()->isEmpty()) {
    281284        temporaryRegisters.append(generator.newTemporary());
    282285        generator.emitNode(temporaryRegisters.last().get(), templateString->value());

  • trunk/Source/JavaScriptCore/parser/ASTBuilder.h

    r210522 r211319  
    276276    }
    277277
    278     TemplateStringNode* createTemplateString(const JSTokenLocation& location, const Identifier& cooked, const Identifier& raw)
     278    TemplateStringNode* createTemplateString(const JSTokenLocation& location, const Identifier* cooked, const Identifier* raw)
    279279    {
    280280        return new (m_parserArena) TemplateStringNode(location, cooked, raw);

  • trunk/Source/JavaScriptCore/parser/Lexer.cpp

    r209632 r211319  
    634634};
    635635
    636 template<typename CharacterType> ParsedUnicodeEscapeValue Lexer<CharacterType>::parseUnicodeEscape()
     636template<typename CharacterType>
     637ParsedUnicodeEscapeValue Lexer<CharacterType>::parseUnicodeEscape()
    637638{
    638639    if (m_current == '{') {
    … …  
    643644                return m_current ? ParsedUnicodeEscapeValue::Invalid : ParsedUnicodeEscapeValue::Incomplete;
    644645            codePoint = (codePoint << 4) | toASCIIHexValue(m_current);
    645             if (codePoint > UCHAR_MAX_VALUE)
    646                 return ParsedUnicodeEscapeValue::Invalid;
     646            if (codePoint > UCHAR_MAX_VALUE) {
     647                // For raw template literal syntax, we consume `NotEscapeSequence`.
     648                // Here, we consume NotCodePoint's HexDigits.
     649                //
     650                // NotEscapeSequence ::
     651                //     u { [lookahread not one of HexDigit]
     652                //     u { NotCodePoint
     653                //     u { CodePoint [lookahead != }]
     654                //
     655                // NotCodePoint ::
     656                //     HexDigits but not if MV of HexDigits <= 0x10FFFF
     657                //
     658                // CodePoint ::
     659                //     HexDigits but not if MV of HexDigits > 0x10FFFF
     660                shift();
     661                while (isASCIIHexDigit(m_current))
     662                    shift();
     663
     664                return atEnd() ? ParsedUnicodeEscapeValue::Incomplete : ParsedUnicodeEscapeValue::Invalid;
     665            }
    647666            shift();
    648667        } while (m_current != '}');
    … …  
    654673    auto character3 = peek(2);
    655674    auto character4 = peek(3);
    656     if (UNLIKELY(!isASCIIHexDigit(m_current) || !isASCIIHexDigit(character2) || !isASCIIHexDigit(character3) || !isASCIIHexDigit(character4)))
    657         return (m_code + 4) >= m_codeEnd ? ParsedUnicodeEscapeValue::Incomplete : ParsedUnicodeEscapeValue::Invalid;
     675    if (UNLIKELY(!isASCIIHexDigit(m_current) || !isASCIIHexDigit(character2) || !isASCIIHexDigit(character3) || !isASCIIHexDigit(character4))) {
     676        auto result = (m_code + 4) >= m_codeEnd ? ParsedUnicodeEscapeValue::Incomplete : ParsedUnicodeEscapeValue::Invalid;
     677
     678        // For raw template literal syntax, we consume `NotEscapeSequence`.
     679        //
     680        // NotEscapeSequence ::
     681        //     u [lookahead not one of HexDigit][lookahead != {]
     682        //     u HexDigit [lookahead not one of HexDigit]
     683        //     u HexDigit HexDigit [lookahead not one of HexDigit]
     684        //     u HexDigit HexDigit HexDigit [lookahead not one of HexDigit]
     685        while (isASCIIHexDigit(m_current))
     686            shift();
     687
     688        return result;
     689    }
     690
    658691    auto result = convertUnicode(m_current, character2, character3, character4);
    659692    shift();
    … …  
    11821215
    11831216template <typename T>
    1184 template <bool shouldBuildStrings> ALWAYS_INLINE auto Lexer<T>::parseComplexEscape(EscapeParseMode escapeParseMode, bool strictMode, T stringQuoteCharacter) -> StringParseResult
     1217template <bool shouldBuildStrings, LexerEscapeParseMode escapeParseMode> ALWAYS_INLINE auto Lexer<T>::parseComplexEscape(bool strictMode, T stringQuoteCharacter) -> StringParseResult
    11851218{
    11861219    if (m_current == 'x') {
    11871220        shift();
    11881221        if (!isASCIIHexDigit(m_current) || !isASCIIHexDigit(peek(1))) {
     1222            // For raw template literal syntax, we consume `NotEscapeSequence`.
     1223            //
     1224            // NotEscapeSequence ::
     1225            //     x [lookahread not one of HexDigit]
     1226            //     x HexDigit [lookahread not one of HexDigit]
     1227            if (isASCIIHexDigit(m_current))
     1228                shift();
     1229            ASSERT(!isASCIIHexDigit(m_current));
     1230
    11891231            m_lexErrorMessage = ASCIILiteral("\\x can only be followed by a hex character sequence");
    1190             return StringCannotBeParsed;
    1191         }
     1232            return atEnd() ? StringUnterminated : StringCannotBeParsed;
     1233        }
     1234
    11921235        T prev = m_current;
    11931236        shift();
    … …  
    11951238            record16(convertHex(prev, m_current));
    11961239        shift();
     1240
    11971241        return StringParsedSuccessfully;
    11981242    }
    … …  
    12011245        shift();
    12021246
    1203         if (escapeParseMode == EscapeParseMode::String && m_current == stringQuoteCharacter) {
     1247        if (escapeParseMode == LexerEscapeParseMode::String && m_current == stringQuoteCharacter) {
    12041248            if (shouldBuildStrings)
    12051249                record16('u');
    … …  
    12151259
    12161260        m_lexErrorMessage = ASCIILiteral("\\u can only be followed by a Unicode character sequence");
    1217         return character.isIncomplete() ? StringUnterminated : StringCannotBeParsed;
     1261        return atEnd() ? StringUnterminated : StringCannotBeParsed;
    12181262    }
    12191263
    … …  
    12241268            shift();
    12251269            if (character1 != '0' || isASCIIDigit(m_current)) {
     1270                // For raw template literal syntax, we consume `NotEscapeSequence`.
     1271                //
     1272                // NotEscapeSequence ::
     1273                //     0 DecimalDigit
     1274                //     DecimalDigit but not 0
     1275                if (character1 == '0')
     1276                    shift();
     1277
    12261278                m_lexErrorMessage = ASCIILiteral("The only valid numeric escape in strict mode is '\\0'");
    1227                 return StringCannotBeParsed;
     1279                return atEnd() ? StringUnterminated : StringCannotBeParsed;
    12281280            }
    12291281            if (shouldBuildStrings)
    … …  
    12911343                shiftLineTerminator();
    12921344            else {
    1293                 StringParseResult result = parseComplexEscape<shouldBuildStrings>(EscapeParseMode::String, strictMode, stringQuoteCharacter);
     1345                StringParseResult result = parseComplexEscape<shouldBuildStrings, LexerEscapeParseMode::String>(strictMode, stringQuoteCharacter);
    12941346                if (result != StringParsedSuccessfully)
    12951347                    return result;
    … …  
    13731425
    13741426template <typename T>
    1375 template <bool shouldBuildStrings> typename Lexer<T>::StringParseResult Lexer<T>::parseTemplateLiteral(JSTokenData* tokenData, RawStringsBuildMode rawStringsBuildMode)
    1376 {
     1427typename Lexer<T>::StringParseResult Lexer<T>::parseTemplateLiteral(JSTokenData* tokenData, RawStringsBuildMode rawStringsBuildMode)
     1428{
     1429    bool parseCookedFailed = false;
    13771430    const T* stringStart = currentSourcePtr();
    13781431    const T* rawStringStart = currentSourcePtr();
    … …  
    13831436        if (UNLIKELY(m_current == '\\')) {
    13841437            lineNumberAdder.clear();
    1385             if (stringStart != currentSourcePtr() && shouldBuildStrings)
     1438            if (stringStart != currentSourcePtr())
    13861439                append16(stringStart, currentSourcePtr() - stringStart);
    13871440            shift();
    … …  
    13911444            // Most common escape sequences first.
    13921445            if (escape) {
    1393                 if (shouldBuildStrings)
    1394                     record16(escape);
     1446                record16(escape);
    13951447                shift();
    13961448            } else if (UNLIKELY(isLineTerminator(m_current))) {
    13971449                // Normalize <CR>, <CR><LF> to <LF>.
    13981450                if (m_current == '\r') {
    1399                     if (shouldBuildStrings) {
    1400                         ASSERT_WITH_MESSAGE(rawStringStart != currentSourcePtr(), "We should have at least shifted the escape.");
    1401 
    1402                         if (rawStringsBuildMode == RawStringsBuildMode::BuildRawStrings) {
    1403                             m_bufferForRawTemplateString16.append(rawStringStart, currentSourcePtr() - rawStringStart);
    1404                             m_bufferForRawTemplateString16.append('\n');
    1405                         }
     1451                    ASSERT_WITH_MESSAGE(rawStringStart != currentSourcePtr(), "We should have at least shifted the escape.");
     1452
     1453                    if (rawStringsBuildMode == RawStringsBuildMode::BuildRawStrings) {
     1454                        m_bufferForRawTemplateString16.append(rawStringStart, currentSourcePtr() - rawStringStart);
     1455                        m_bufferForRawTemplateString16.append('\n');
    14061456                    }
    14071457
    … …  
    14201470            } else {
    14211471                bool strictMode = true;
    1422                 StringParseResult result = parseComplexEscape<shouldBuildStrings>(EscapeParseMode::Template, strictMode, '`');
    1423                 if (result != StringParsedSuccessfully)
    1424                     return result;
     1472                StringParseResult result = parseComplexEscape<true, LexerEscapeParseMode::Template>(strictMode, '`');
     1473                if (result != StringParsedSuccessfully) {
     1474                    if (rawStringsBuildMode == RawStringsBuildMode::BuildRawStrings && result == StringCannotBeParsed)
     1475                        parseCookedFailed = true;
     1476                    else
     1477                        return result;
     1478                }
    14251479            }
    14261480
    … …  
    14461500                if (m_current == '\r') {
    14471501                    // Normalize <CR>, <CR><LF> to <LF>.
    1448                     if (shouldBuildStrings) {
    1449                         if (stringStart != currentSourcePtr())
    1450                             append16(stringStart, currentSourcePtr() - stringStart);
    1451                         if (rawStringStart != currentSourcePtr() && rawStringsBuildMode == RawStringsBuildMode::BuildRawStrings)
    1452                             m_bufferForRawTemplateString16.append(rawStringStart, currentSourcePtr() - rawStringStart);
    1453 
    1454                         record16('\n');
    1455                         if (rawStringsBuildMode == RawStringsBuildMode::BuildRawStrings)
    1456                             m_bufferForRawTemplateString16.append('\n');
    1457                     }
     1502                    if (stringStart != currentSourcePtr())
     1503                        append16(stringStart, currentSourcePtr() - stringStart);
     1504                    if (rawStringStart != currentSourcePtr() && rawStringsBuildMode == RawStringsBuildMode::BuildRawStrings)
     1505                        m_bufferForRawTemplateString16.append(rawStringStart, currentSourcePtr() - rawStringStart);
     1506
     1507                    record16('\n');
     1508                    if (rawStringsBuildMode == RawStringsBuildMode::BuildRawStrings)
     1509                        m_bufferForRawTemplateString16.append('\n');
    14581510                    lineNumberAdder.add(m_current);
    14591511                    shift();
    … …  
    14791531    bool isTail = m_current == '`';
    14801532
    1481     if (shouldBuildStrings) {
    1482         if (currentSourcePtr() != stringStart)
    1483             append16(stringStart, currentSourcePtr() - stringStart);
    1484         if (rawStringStart != currentSourcePtr() && rawStringsBuildMode == RawStringsBuildMode::BuildRawStrings)
    1485             m_bufferForRawTemplateString16.append(rawStringStart, currentSourcePtr() - rawStringStart);
    1486     }
    1487 
    1488     if (shouldBuildStrings) {
     1533    if (currentSourcePtr() != stringStart)
     1534        append16(stringStart, currentSourcePtr() - stringStart);
     1535    if (rawStringStart != currentSourcePtr() && rawStringsBuildMode == RawStringsBuildMode::BuildRawStrings)
     1536        m_bufferForRawTemplateString16.append(rawStringStart, currentSourcePtr() - rawStringStart);
     1537
     1538    if (!parseCookedFailed)
    14891539        tokenData->cooked = makeIdentifier(m_buffer16.data(), m_buffer16.size());
    1490         // Line terminator normalization (e.g. <CR> => <LF>) should be applied to both the raw and cooked representations.
    1491         if (rawStringsBuildMode == RawStringsBuildMode::BuildRawStrings)
    1492             tokenData->raw = makeIdentifier(m_bufferForRawTemplateString16.data(), m_bufferForRawTemplateString16.size());
    1493         else
    1494             tokenData->raw = makeEmptyIdentifier();
    1495     } else {
    1496         tokenData->cooked = makeEmptyIdentifier();
    1497         tokenData->raw = makeEmptyIdentifier();
    1498     }
     1540    else
     1541        tokenData->cooked = nullptr;
     1542
     1543    // Line terminator normalization (e.g. <CR> => <LF>) should be applied to both the raw and cooked representations.
     1544    if (rawStringsBuildMode == RawStringsBuildMode::BuildRawStrings)
     1545        tokenData->raw = makeIdentifier(m_bufferForRawTemplateString16.data(), m_bufferForRawTemplateString16.size());
     1546    else
     1547        tokenData->raw = nullptr;
     1548
    14991549    tokenData->isTail = isTail;
    15001550
    … …  
    20822132        shift();
    20832133        token = SEMICOLON;
     2134        break;
     2135    case CharacterBackQuote:
     2136        shift();
     2137        token = BACKQUOTE;
    20842138        break;
    20852139    case CharacterOpenBrace:
    … …  
    22352289        break;
    22362290        }
    2237     case CharacterBackQuote: {
    2238         // Skip backquote.
    2239         shift();
    2240         StringParseResult result = StringCannotBeParsed;
    2241         if (lexerFlags & LexerFlagsDontBuildStrings)
    2242             result = parseTemplateLiteral<false>(tokenData, RawStringsBuildMode::BuildRawStrings);
    2243         else
    2244             result = parseTemplateLiteral<true>(tokenData, RawStringsBuildMode::BuildRawStrings);
    2245 
    2246         if (UNLIKELY(result != StringParsedSuccessfully)) {
    2247             token = result == StringUnterminated ? UNTERMINATED_TEMPLATE_LITERAL_ERRORTOK : INVALID_TEMPLATE_LITERAL_ERRORTOK;
    2248             goto returnError;
    2249         }
    2250         token = TEMPLATE;
    2251         break;
    2252         }
    22532291    case CharacterIdentifierStart:
    22542292        ASSERT(isIdentStart(m_current));
    … …  
    24212459
    24222460template <typename T>
    2423 JSTokenType Lexer<T>::scanTrailingTemplateString(JSToken* tokenRecord, RawStringsBuildMode rawStringsBuildMode)
     2461JSTokenType Lexer<T>::scanTemplateString(JSToken* tokenRecord, RawStringsBuildMode rawStringsBuildMode)
    24242462{
    24252463    JSTokenData* tokenData = &tokenRecord->m_data;
    … …  
    24272465    ASSERT(m_buffer16.isEmpty());
    24282466
    2429     // Leading closing brace } is already shifted in the previous token scan.
     2467    // Leading backquote ` (for template head) or closing brace } (for template trailing) are already shifted in the previous token scan.
    24302468    // So in this re-scan phase, shift() is not needed here.
    2431     StringParseResult result = parseTemplateLiteral<true>(tokenData, rawStringsBuildMode);
     2469    StringParseResult result = parseTemplateLiteral(tokenData, rawStringsBuildMode);
    24322470    JSTokenType token = ERRORTOK;
    24332471    if (UNLIKELY(result != StringParsedSuccessfully)) {

  • trunk/Source/JavaScriptCore/parser/Lexer.h

    r207798 r211319  
    3838};
    3939
     40enum class LexerEscapeParseMode { Template, String };
     41
    4042struct ParsedUnicodeEscapeValue;
    4143
    … …  
    7880    JSTokenType scanRegExp(JSToken*, UChar patternPrefix = 0);
    7981    enum class RawStringsBuildMode { BuildRawStrings, DontBuildRawStrings };
    80     JSTokenType scanTrailingTemplateString(JSToken*, RawStringsBuildMode);
     82    JSTokenType scanTemplateString(JSToken*, RawStringsBuildMode);
    8183
    8284    // Functions for use after parsing.
    … …  
    171173    template <bool shouldBuildStrings> NEVER_INLINE StringParseResult parseStringSlowCase(JSTokenData*, bool strictMode);
    172174
    173     enum class EscapeParseMode { Template, String };
    174     template <bool shouldBuildStrings> ALWAYS_INLINE StringParseResult parseComplexEscape(EscapeParseMode, bool strictMode, T stringQuoteCharacter);
    175     template <bool shouldBuildStrings> ALWAYS_INLINE StringParseResult parseTemplateLiteral(JSTokenData*, RawStringsBuildMode);
     175    template <bool shouldBuildStrings, LexerEscapeParseMode escapeParseMode> ALWAYS_INLINE StringParseResult parseComplexEscape(bool strictMode, T stringQuoteCharacter);
     176    ALWAYS_INLINE StringParseResult parseTemplateLiteral(JSTokenData*, RawStringsBuildMode);
    176177    ALWAYS_INLINE void parseHex(double& returnValue);
    177178    ALWAYS_INLINE bool parseBinary(double& returnValue);

  • trunk/Source/JavaScriptCore/parser/NodeConstructors.h

    r210522 r211319  
    112112    }
    113113
    114     inline TemplateStringNode::TemplateStringNode(const JSTokenLocation& location, const Identifier& cooked, const Identifier& raw)
     114    inline TemplateStringNode::TemplateStringNode(const JSTokenLocation& location, const Identifier* cooked, const Identifier* raw)
    115115        : ExpressionNode(location)
    116116        , m_cooked(cooked)

  • trunk/Source/JavaScriptCore/parser/Nodes.h

    r210522 r211319  
    491491    class TemplateStringNode : public ExpressionNode {
    492492    public:
    493         TemplateStringNode(const JSTokenLocation&, const Identifier& cooked, const Identifier& raw);
    494 
    495         const Identifier& cooked() { return m_cooked; }
    496         const Identifier& raw() { return m_raw; }
    497 
    498     private:
    499         RegisterID* emitBytecode(BytecodeGenerator&, RegisterID* = 0) override;
    500 
    501         const Identifier& m_cooked;
    502         const Identifier& m_raw;
     493        TemplateStringNode(const JSTokenLocation&, const Identifier* cooked, const Identifier* raw);
     494
     495        const Identifier* cooked() { return m_cooked; }
     496        const Identifier* raw() { return m_raw; }
     497
     498    private:
     499        RegisterID* emitBytecode(BytecodeGenerator&, RegisterID* = 0) override;
     500
     501        const Identifier* m_cooked;
     502        const Identifier* m_raw;
    503503    };
    504504

  • trunk/Source/JavaScriptCore/parser/Parser.cpp

    r211017 r211319  
    905905        SavePoint savePoint = createSavePoint();
    906906        assignmentTarget = parseDestructuringPattern(context, kind, exportType, duplicateIdentifier, hasDestructuringPattern, bindingContext, depth);
    907         if (assignmentTarget && !match(DOT) && !match(OPENBRACKET) && !match(OPENPAREN) && !match(TEMPLATE))
     907        if (assignmentTarget && !match(DOT) && !match(OPENBRACKET) && !match(OPENPAREN) && !match(BACKQUOTE))
    908908            return assignmentTarget;
    909909        restoreSavePoint(savePoint);
    … …  
    40904090template <class TreeBuilder> typename TreeBuilder::TemplateString Parser<LexerType>::parseTemplateString(TreeBuilder& context, bool isTemplateHead, typename LexerType::RawStringsBuildMode rawStringsBuildMode, bool& elementIsTail)
    40914091{
    4092     if (!isTemplateHead) {
     4092    if (isTemplateHead)
     4093        ASSERT(match(BACKQUOTE));
     4094    else
    40934095        matchOrFail(CLOSEBRACE, "Expected a closing '}' following an expression in template literal");
    4094         // Re-scan the token to recognize it as Template Element.
    4095         m_token.m_type = m_lexer->scanTrailingTemplateString(&m_token, rawStringsBuildMode);
    4096     }
     4096
     4097    // Re-scan the token to recognize it as Template Element.
     4098    m_token.m_type = m_lexer->scanTemplateString(&m_token, rawStringsBuildMode);
    40974099    matchOrFail(TEMPLATE, "Expected an template element");
    40984100    const Identifier* cooked = m_token.m_data.cooked;
    … …  
    41014103    JSTokenLocation location(tokenLocation());
    41024104    next();
    4103     return context.createTemplateString(location, *cooked, *raw);
     4105    return context.createTemplateString(location, cooked, raw);
    41044106}
    41054107
    … …  
    41074109template <class TreeBuilder> typename TreeBuilder::TemplateLiteral Parser<LexerType>::parseTemplateLiteral(TreeBuilder& context, typename LexerType::RawStringsBuildMode rawStringsBuildMode)
    41084110{
     4111    ASSERT(match(BACKQUOTE));
    41094112    JSTokenLocation location(tokenLocation());
    41104113    bool elementIsTail = false;
    … …  
    42694272        return re;
    42704273    }
    4271     case TEMPLATE:
     4274    case BACKQUOTE:
    42724275        return parseTemplateLiteral(context, LexerType::RawStringsBuildMode::DontBuildRawStrings);
    42734276    case YIELD:
    … …  
    45064509            break;
    45074510        }
    4508         case TEMPLATE: {
     4511        case BACKQUOTE: {
    45094512            semanticFailIfTrue(baseIsSuper, "Cannot use super as tag for tagged templates");
    45104513            JSTextPosition expressionEnd = lastTokenEndPosition();

  • trunk/Source/JavaScriptCore/parser/ParserTokens.h

    r209350 r211319  
    110110    COMMA,
    111111    QUESTION,
     112    BACKQUOTE,
    112113    INTEGER,
    113114    DOUBLE,

  • trunk/Source/JavaScriptCore/parser/SyntaxChecker.h

    r210522 r211319  
    195195    int createArguments(int) { return ArgumentsResult; }
    196196    ExpressionType createSpreadExpression(const JSTokenLocation&, ExpressionType, int, int, int) { return SpreadExpr; }
    197     TemplateString createTemplateString(const JSTokenLocation&, const Identifier&, const Identifier&) { return TemplateStringResult; }
     197    TemplateString createTemplateString(const JSTokenLocation&, const Identifier*, const Identifier*) { return TemplateStringResult; }
    198198    TemplateStringList createTemplateStringList(TemplateString) { return TemplateStringListResult; }
    199199    TemplateStringList createTemplateStringList(TemplateStringList, TemplateString) { return TemplateStringListResult; }

  • trunk/Source/JavaScriptCore/runtime/TemplateRegistry.cpp

    r209018 r211319  
    5959
    6060    for (unsigned index = 0; index < count; ++index) {
    61         templateObject->putDirectIndex(exec, index, jsString(exec, templateKey.cookedStrings()[index]), ReadOnly | DontDelete, PutDirectIndexLikePutDirect);
     61        auto cooked = templateKey.cookedStrings()[index];
     62        if (cooked)
     63            templateObject->putDirectIndex(exec, index, jsString(exec, cooked.value()), ReadOnly | DontDelete, PutDirectIndexLikePutDirect);
     64        else
     65            templateObject->putDirectIndex(exec, index, jsUndefined(), ReadOnly | DontDelete, PutDirectIndexLikePutDirect);
    6266        RETURN_IF_EXCEPTION(scope, nullptr);
     67
    6368        rawObject->putDirectIndex(exec, index, jsString(exec, templateKey.rawStrings()[index]), ReadOnly | DontDelete, PutDirectIndexLikePutDirect);
    6469        RETURN_IF_EXCEPTION(scope, nullptr);

  • trunk/Source/JavaScriptCore/runtime/TemplateRegistryKey.h

    r208953 r211319  
    3939    friend class TemplateRegistryKeyTable;
    4040    typedef Vector<String, 4> StringVector;
     41    typedef Vector<std::optional<String>, 4> OptionalStringVector;
    4142
    4243    enum DeletedValueTag { DeletedValue };
    … …  
    5253
    5354    const StringVector& rawStrings() const { return m_rawStrings; }
    54     const StringVector& cookedStrings() const { return m_cookedStrings; }
     55    const OptionalStringVector& cookedStrings() const { return m_cookedStrings; }
    5556
    5657    bool operator==(const TemplateRegistryKey& other) const { return m_hash == other.m_hash && m_rawStrings == other.m_rawStrings; }
    … …  
    6768
    6869private:
    69     static Ref<TemplateRegistryKey> create(const StringVector& rawStrings, const StringVector& cookedStrings)
     70    static Ref<TemplateRegistryKey> create(StringVector&& rawStrings, OptionalStringVector&& cookedStrings)
    7071    {
    71         return adoptRef(*new TemplateRegistryKey(rawStrings, cookedStrings));
     72        return adoptRef(*new TemplateRegistryKey(WTFMove(rawStrings), WTFMove(cookedStrings)));
    7273    }
    7374
    74     TemplateRegistryKey(const StringVector& rawStrings, const StringVector& cookedStrings);
     75    TemplateRegistryKey(StringVector&& rawStrings, OptionalStringVector&& cookedStrings);
    7576
    7677    TemplateRegistryKeyTable* m_table { nullptr };
    7778    StringVector m_rawStrings;
    78     StringVector m_cookedStrings;
     79    OptionalStringVector m_cookedStrings;
    7980    unsigned m_hash { 0 };
    8081};
    8182
    82 inline TemplateRegistryKey::TemplateRegistryKey(const StringVector& rawStrings, const StringVector& cookedStrings)
    83     : m_rawStrings(rawStrings)
    84     , m_cookedStrings(cookedStrings)
     83inline TemplateRegistryKey::TemplateRegistryKey(StringVector&& rawStrings, OptionalStringVector&& cookedStrings)
     84    : m_rawStrings(WTFMove(rawStrings))
     85    , m_cookedStrings(WTFMove(cookedStrings))
    8586    , m_hash(calculateHash(rawStrings))
    8687{

  • trunk/Source/JavaScriptCore/runtime/TemplateRegistryKeyTable.cpp

    r208953 r211319  
    4343}
    4444
    45 Ref<TemplateRegistryKey> TemplateRegistryKeyTable::createKey(const TemplateRegistryKey::StringVector& rawStrings, const TemplateRegistryKey::StringVector& cookedStrings)
     45Ref<TemplateRegistryKey> TemplateRegistryKeyTable::createKey(TemplateRegistryKey::StringVector&& rawStrings, TemplateRegistryKey::OptionalStringVector&& cookedStrings)
    4646{
    47     auto key = TemplateRegistryKey::create(rawStrings, cookedStrings);
     47    auto key = TemplateRegistryKey::create(WTFMove(rawStrings), WTFMove(cookedStrings));
    4848    auto addResult = m_atomicTable.add<TemplateRegistryKeyTranslator>(key.ptr());
    4949    if (addResult.isNewEntry)

  • trunk/Source/JavaScriptCore/runtime/TemplateRegistryKeyTable.h

    r208953 r211319  
    3737public:
    3838    using StringVector = Vector<String, 4>;
     39    using OptionalStringVector = Vector<std::optional<String>, 4>;
    3940
    4041    TemplateRegistryKeyTable() = default;
    4142
    42     Ref<TemplateRegistryKey> createKey(const StringVector& rawStrings, const StringVector& cookedStrings);
     43    Ref<TemplateRegistryKey> createKey(StringVector&& rawStrings, OptionalStringVector&& cookedStrings);
    4344
    4445    void unregister(TemplateRegistryKey&);
Note: See TracChangeset for help on using the changeset viewer.