⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 245348 in webkit


Ignore:
Timestamp:
May 15, 2019, 2:44:43 PM (7 years ago)
Author:
Alan Coon
Message:

Cherry-pick r243948. rdar://problem/50753934

SIGSEGV in JSC::BytecodeGenerator::addStringConstant
​https://bugs.webkit.org/show_bug.cgi?id=196486

Reviewed by Saam Barati.

JSTests:

  • stress/arrow-function-and-use-strict-directive.js: Added.
  • stress/arrow-function-syntax.js: Added. Checking EOF token handling. (checkSyntax): (checkSyntaxError): Currently not using it. But it is useful for testing more things related to arrow function syntax.

Source/JavaScriptCore:

When parsing a FunctionExpression / FunctionDeclaration etc., we use SyntaxChecker for the body of the function because we do not have any interest on the nodes of the body at that time.
The nodes will be parsed with the ASTBuilder when the function itself is parsed for code generation. This works well previously because all the function ends with "}" previously.
SyntaxChecker lexes this "}" token, and parser restores the context back to ASTBuilder and continues parsing.

But now, we have ArrowFunctionExpression without braces arrow => expr. Let's consider the following code.

arrow => expr
"string!"

We parse arrow function's body with SyntaxChecker. At that time, we lex "string!" token under the SyntaxChecker context. But this means that we may not build string content for this token
since SyntaxChecker may not have interest on string content itself in certain case. After the parser is back to ASTBuilder, we parse "string!" as ExpressionStatement with string constant,
generate StringNode with non-built identifier (nullptr), and we accidentally create StringNode with nullptr.

This patch fixes this problem. The root cause of this problem is that the last token lexed in the previous context is used. We add lexCurrentTokenAgainUnderCurrentContext which will re-lex
the current token under the current context (may be ASTBuilder). This should be done only when the caller's context is different from SyntaxChecker, which avoids unnecessary lexing.
We leverage existing SavePoint mechanism to implement lexCurrentTokenAgainUnderCurrentContext cleanly.

And we also fix the bug in the existing SavePoint mechanism, which is shown in the attached test script. When we save LexerState, we do not save line terminator status. This patch also introduces
lexWithoutClearingLineTerminator, which lex the token without clearing line terminator status.

  • parser/ASTBuilder.h: (JSC::ASTBuilder::createString):
  • parser/Lexer.cpp: (JSC::Lexer<T>::parseMultilineComment): (JSC::Lexer<T>::lexWithoutClearingLineTerminator): EOF token also should record offset information. This offset information is correctly handled in Lexer::setOffset too. (JSC::Lexer<T>::lex): Deleted.
  • parser/Lexer.h: (JSC::Lexer::hasLineTerminatorBeforeToken const): (JSC::Lexer::setHasLineTerminatorBeforeToken): (JSC::Lexer<T>::lex): (JSC::Lexer::prevTerminator const): Deleted. (JSC::Lexer::setTerminator): Deleted.
  • parser/Parser.cpp: (JSC::Parser<LexerType>::allowAutomaticSemicolon): (JSC::Parser<LexerType>::parseSingleFunction): (JSC::Parser<LexerType>::parseStatementListItem): (JSC::Parser<LexerType>::maybeParseAsyncFunctionDeclarationStatement): (JSC::Parser<LexerType>::parseFunctionInfo): (JSC::Parser<LexerType>::parseClass): (JSC::Parser<LexerType>::parseExportDeclaration): (JSC::Parser<LexerType>::parseAssignmentExpression): (JSC::Parser<LexerType>::parseYieldExpression): (JSC::Parser<LexerType>::parseProperty): (JSC::Parser<LexerType>::parsePrimaryExpression): (JSC::Parser<LexerType>::parseMemberExpression):
  • parser/Parser.h: (JSC::Parser::nextWithoutClearingLineTerminator): (JSC::Parser::lexCurrentTokenAgainUnderCurrentContext): (JSC::Parser::internalSaveLexerState): (JSC::Parser::restoreLexerState):

git-svn-id: ​https://svn.webkit.org/repository/webkit/trunk@243948 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Location:
branches/safari-607-branch
Files:
2 added
7 edited

Legend:

Unmodified
Added
Removed

  • branches/safari-607-branch/JSTests/ChangeLog

    r245146 r245348  
     12019-05-14  Kocsen Chung  <kocsen_chung@apple.com>
     2
     3        Cherry-pick r243948. rdar://problem/50753934
     4
     5    SIGSEGV in JSC::BytecodeGenerator::addStringConstant
     6    https://bugs.webkit.org/show_bug.cgi?id=196486
     7   
     8    Reviewed by Saam Barati.
     9   
     10    JSTests:
     11   
     12    * stress/arrow-function-and-use-strict-directive.js: Added.
     13    * stress/arrow-function-syntax.js: Added. Checking EOF token handling.
     14    (checkSyntax):
     15    (checkSyntaxError): Currently not using it. But it is useful for testing more things related to arrow function syntax.
     16   
     17    Source/JavaScriptCore:
     18   
     19    When parsing a FunctionExpression / FunctionDeclaration etc., we use SyntaxChecker for the body of the function because we do not have any interest on the nodes of the body at that time.
     20    The nodes will be parsed with the ASTBuilder when the function itself is parsed for code generation. This works well previously because all the function ends with "}" previously.
     21    SyntaxChecker lexes this "}" token, and parser restores the context back to ASTBuilder and continues parsing.
     22   
     23    But now, we have ArrowFunctionExpression without braces `arrow => expr`. Let's consider the following code.
     24   
     25            arrow => expr
     26            "string!"
     27   
     28    We parse arrow function's body with SyntaxChecker. At that time, we lex "string!" token under the SyntaxChecker context. But this means that we may not build string content for this token
     29    since SyntaxChecker may not have interest on string content itself in certain case. After the parser is back to ASTBuilder, we parse "string!" as ExpressionStatement with string constant,
     30    generate StringNode with non-built identifier (nullptr), and we accidentally create StringNode with nullptr.
     31   
     32    This patch fixes this problem. The root cause of this problem is that the last token lexed in the previous context is used. We add lexCurrentTokenAgainUnderCurrentContext which will re-lex
     33    the current token under the current context (may be ASTBuilder). This should be done only when the caller's context is different from SyntaxChecker, which avoids unnecessary lexing.
     34    We leverage existing SavePoint mechanism to implement lexCurrentTokenAgainUnderCurrentContext cleanly.
     35   
     36    And we also fix the bug in the existing SavePoint mechanism, which is shown in the attached test script. When we save LexerState, we do not save line terminator status. This patch also introduces
     37    lexWithoutClearingLineTerminator, which lex the token without clearing line terminator status.
     38   
     39    * parser/ASTBuilder.h:
     40    (JSC::ASTBuilder::createString):
     41    * parser/Lexer.cpp:
     42    (JSC::Lexer<T>::parseMultilineComment):
     43    (JSC::Lexer<T>::lexWithoutClearingLineTerminator): EOF token also should record offset information. This offset information is correctly handled in Lexer::setOffset too.
     44    (JSC::Lexer<T>::lex): Deleted.
     45    * parser/Lexer.h:
     46    (JSC::Lexer::hasLineTerminatorBeforeToken const):
     47    (JSC::Lexer::setHasLineTerminatorBeforeToken):
     48    (JSC::Lexer<T>::lex):
     49    (JSC::Lexer::prevTerminator const): Deleted.
     50    (JSC::Lexer::setTerminator): Deleted.
     51    * parser/Parser.cpp:
     52    (JSC::Parser<LexerType>::allowAutomaticSemicolon):
     53    (JSC::Parser<LexerType>::parseSingleFunction):
     54    (JSC::Parser<LexerType>::parseStatementListItem):
     55    (JSC::Parser<LexerType>::maybeParseAsyncFunctionDeclarationStatement):
     56    (JSC::Parser<LexerType>::parseFunctionInfo):
     57    (JSC::Parser<LexerType>::parseClass):
     58    (JSC::Parser<LexerType>::parseExportDeclaration):
     59    (JSC::Parser<LexerType>::parseAssignmentExpression):
     60    (JSC::Parser<LexerType>::parseYieldExpression):
     61    (JSC::Parser<LexerType>::parseProperty):
     62    (JSC::Parser<LexerType>::parsePrimaryExpression):
     63    (JSC::Parser<LexerType>::parseMemberExpression):
     64    * parser/Parser.h:
     65    (JSC::Parser::nextWithoutClearingLineTerminator):
     66    (JSC::Parser::lexCurrentTokenAgainUnderCurrentContext):
     67    (JSC::Parser::internalSaveLexerState):
     68    (JSC::Parser::restoreLexerState):
     69   
     70    git-svn-id: https://svn.webkit.org/repository/webkit/trunk@243948 268f45cc-cd09-0410-ab3c-d52691b4dbfc
     71
     72    2019-04-05  Yusuke Suzuki  <ysuzuki@apple.com>
     73
     74            SIGSEGV in JSC::BytecodeGenerator::addStringConstant
     75            https://bugs.webkit.org/show_bug.cgi?id=196486
     76
     77            Reviewed by Saam Barati.
     78
     79            * stress/arrow-function-and-use-strict-directive.js: Added.
     80            * stress/arrow-function-syntax.js: Added. Checking EOF token handling.
     81            (checkSyntax):
     82            (checkSyntaxError): Currently not using it. But it is useful for testing more things related to arrow function syntax.
     83
    1842019-05-09  Ryan Haddad  <ryanhaddad@apple.com>
    285

  • branches/safari-607-branch/Source/JavaScriptCore/ChangeLog

    r244798 r245348  
     12019-05-14  Kocsen Chung  <kocsen_chung@apple.com>
     2
     3        Cherry-pick r243948. rdar://problem/50753934
     4
     5    SIGSEGV in JSC::BytecodeGenerator::addStringConstant
     6    https://bugs.webkit.org/show_bug.cgi?id=196486
     7   
     8    Reviewed by Saam Barati.
     9   
     10    JSTests:
     11   
     12    * stress/arrow-function-and-use-strict-directive.js: Added.
     13    * stress/arrow-function-syntax.js: Added. Checking EOF token handling.
     14    (checkSyntax):
     15    (checkSyntaxError): Currently not using it. But it is useful for testing more things related to arrow function syntax.
     16   
     17    Source/JavaScriptCore:
     18   
     19    When parsing a FunctionExpression / FunctionDeclaration etc., we use SyntaxChecker for the body of the function because we do not have any interest on the nodes of the body at that time.
     20    The nodes will be parsed with the ASTBuilder when the function itself is parsed for code generation. This works well previously because all the function ends with "}" previously.
     21    SyntaxChecker lexes this "}" token, and parser restores the context back to ASTBuilder and continues parsing.
     22   
     23    But now, we have ArrowFunctionExpression without braces `arrow => expr`. Let's consider the following code.
     24   
     25            arrow => expr
     26            "string!"
     27   
     28    We parse arrow function's body with SyntaxChecker. At that time, we lex "string!" token under the SyntaxChecker context. But this means that we may not build string content for this token
     29    since SyntaxChecker may not have interest on string content itself in certain case. After the parser is back to ASTBuilder, we parse "string!" as ExpressionStatement with string constant,
     30    generate StringNode with non-built identifier (nullptr), and we accidentally create StringNode with nullptr.
     31   
     32    This patch fixes this problem. The root cause of this problem is that the last token lexed in the previous context is used. We add lexCurrentTokenAgainUnderCurrentContext which will re-lex
     33    the current token under the current context (may be ASTBuilder). This should be done only when the caller's context is different from SyntaxChecker, which avoids unnecessary lexing.
     34    We leverage existing SavePoint mechanism to implement lexCurrentTokenAgainUnderCurrentContext cleanly.
     35   
     36    And we also fix the bug in the existing SavePoint mechanism, which is shown in the attached test script. When we save LexerState, we do not save line terminator status. This patch also introduces
     37    lexWithoutClearingLineTerminator, which lex the token without clearing line terminator status.
     38   
     39    * parser/ASTBuilder.h:
     40    (JSC::ASTBuilder::createString):
     41    * parser/Lexer.cpp:
     42    (JSC::Lexer<T>::parseMultilineComment):
     43    (JSC::Lexer<T>::lexWithoutClearingLineTerminator): EOF token also should record offset information. This offset information is correctly handled in Lexer::setOffset too.
     44    (JSC::Lexer<T>::lex): Deleted.
     45    * parser/Lexer.h:
     46    (JSC::Lexer::hasLineTerminatorBeforeToken const):
     47    (JSC::Lexer::setHasLineTerminatorBeforeToken):
     48    (JSC::Lexer<T>::lex):
     49    (JSC::Lexer::prevTerminator const): Deleted.
     50    (JSC::Lexer::setTerminator): Deleted.
     51    * parser/Parser.cpp:
     52    (JSC::Parser<LexerType>::allowAutomaticSemicolon):
     53    (JSC::Parser<LexerType>::parseSingleFunction):
     54    (JSC::Parser<LexerType>::parseStatementListItem):
     55    (JSC::Parser<LexerType>::maybeParseAsyncFunctionDeclarationStatement):
     56    (JSC::Parser<LexerType>::parseFunctionInfo):
     57    (JSC::Parser<LexerType>::parseClass):
     58    (JSC::Parser<LexerType>::parseExportDeclaration):
     59    (JSC::Parser<LexerType>::parseAssignmentExpression):
     60    (JSC::Parser<LexerType>::parseYieldExpression):
     61    (JSC::Parser<LexerType>::parseProperty):
     62    (JSC::Parser<LexerType>::parsePrimaryExpression):
     63    (JSC::Parser<LexerType>::parseMemberExpression):
     64    * parser/Parser.h:
     65    (JSC::Parser::nextWithoutClearingLineTerminator):
     66    (JSC::Parser::lexCurrentTokenAgainUnderCurrentContext):
     67    (JSC::Parser::internalSaveLexerState):
     68    (JSC::Parser::restoreLexerState):
     69   
     70    git-svn-id: https://svn.webkit.org/repository/webkit/trunk@243948 268f45cc-cd09-0410-ab3c-d52691b4dbfc
     71
     72    2019-04-05  Yusuke Suzuki  <ysuzuki@apple.com>
     73
     74            SIGSEGV in JSC::BytecodeGenerator::addStringConstant
     75            https://bugs.webkit.org/show_bug.cgi?id=196486
     76
     77            Reviewed by Saam Barati.
     78
     79            When parsing a FunctionExpression / FunctionDeclaration etc., we use SyntaxChecker for the body of the function because we do not have any interest on the nodes of the body at that time.
     80            The nodes will be parsed with the ASTBuilder when the function itself is parsed for code generation. This works well previously because all the function ends with "}" previously.
     81            SyntaxChecker lexes this "}" token, and parser restores the context back to ASTBuilder and continues parsing.
     82
     83            But now, we have ArrowFunctionExpression without braces `arrow => expr`. Let's consider the following code.
     84
     85                    arrow => expr
     86                    "string!"
     87
     88            We parse arrow function's body with SyntaxChecker. At that time, we lex "string!" token under the SyntaxChecker context. But this means that we may not build string content for this token
     89            since SyntaxChecker may not have interest on string content itself in certain case. After the parser is back to ASTBuilder, we parse "string!" as ExpressionStatement with string constant,
     90            generate StringNode with non-built identifier (nullptr), and we accidentally create StringNode with nullptr.
     91
     92            This patch fixes this problem. The root cause of this problem is that the last token lexed in the previous context is used. We add lexCurrentTokenAgainUnderCurrentContext which will re-lex
     93            the current token under the current context (may be ASTBuilder). This should be done only when the caller's context is different from SyntaxChecker, which avoids unnecessary lexing.
     94            We leverage existing SavePoint mechanism to implement lexCurrentTokenAgainUnderCurrentContext cleanly.
     95
     96            And we also fix the bug in the existing SavePoint mechanism, which is shown in the attached test script. When we save LexerState, we do not save line terminator status. This patch also introduces
     97            lexWithoutClearingLineTerminator, which lex the token without clearing line terminator status.
     98
     99            * parser/ASTBuilder.h:
     100            (JSC::ASTBuilder::createString):
     101            * parser/Lexer.cpp:
     102            (JSC::Lexer<T>::parseMultilineComment):
     103            (JSC::Lexer<T>::lexWithoutClearingLineTerminator): EOF token also should record offset information. This offset information is correctly handled in Lexer::setOffset too.
     104            (JSC::Lexer<T>::lex): Deleted.
     105            * parser/Lexer.h:
     106            (JSC::Lexer::hasLineTerminatorBeforeToken const):
     107            (JSC::Lexer::setHasLineTerminatorBeforeToken):
     108            (JSC::Lexer<T>::lex):
     109            (JSC::Lexer::prevTerminator const): Deleted.
     110            (JSC::Lexer::setTerminator): Deleted.
     111            * parser/Parser.cpp:
     112            (JSC::Parser<LexerType>::allowAutomaticSemicolon):
     113            (JSC::Parser<LexerType>::parseSingleFunction):
     114            (JSC::Parser<LexerType>::parseStatementListItem):
     115            (JSC::Parser<LexerType>::maybeParseAsyncFunctionDeclarationStatement):
     116            (JSC::Parser<LexerType>::parseFunctionInfo):
     117            (JSC::Parser<LexerType>::parseClass):
     118            (JSC::Parser<LexerType>::parseExportDeclaration):
     119            (JSC::Parser<LexerType>::parseAssignmentExpression):
     120            (JSC::Parser<LexerType>::parseYieldExpression):
     121            (JSC::Parser<LexerType>::parseProperty):
     122            (JSC::Parser<LexerType>::parsePrimaryExpression):
     123            (JSC::Parser<LexerType>::parseMemberExpression):
     124            * parser/Parser.h:
     125            (JSC::Parser::nextWithoutClearingLineTerminator):
     126            (JSC::Parser::lexCurrentTokenAgainUnderCurrentContext):
     127            (JSC::Parser::internalSaveLexerState):
     128            (JSC::Parser::restoreLexerState):
     129
    11302019-04-24  Alan Coon  <alancoon@apple.com>
    2131

  • branches/safari-607-branch/Source/JavaScriptCore/parser/ASTBuilder.h

    r237241 r245348  
    242242    ExpressionNode* createString(const JSTokenLocation& location, const Identifier* string)
    243243    {
     244        ASSERT(string);
    244245        incConstants();
    245246        return new (m_parserArena) StringNode(location, *string);

  • branches/safari-607-branch/Source/JavaScriptCore/parser/Lexer.cpp

    r239559 r245348  
    17061706        if (isLineTerminator(m_current)) {
    17071707            shiftLineTerminator();
    1708             m_terminator = true;
     1708            m_hasLineTerminatorBeforeToken = true;
    17091709        } else
    17101710            shift();
    … …  
    17851785
    17861786template <typename T>
    1787 JSTokenType Lexer<T>::lex(JSToken* tokenRecord, unsigned lexerFlags, bool strictMode)
     1787JSTokenType Lexer<T>::lexWithoutClearingLineTerminator(JSToken* tokenRecord, unsigned lexerFlags, bool strictMode)
    17881788{
    17891789    JSTokenData* tokenData = &tokenRecord->m_data;
    … …  
    17961796
    17971797    JSTokenType token = ERRORTOK;
    1798     m_terminator = false;
    17991798
    18001799start:
    18011800    skipWhitespace();
    18021801
    1803     if (atEnd())
    1804         return EOFTOK;
    1805    
    18061802    tokenLocation->startOffset = currentOffset();
    18071803    ASSERT(currentOffset() >= currentLineStartOffset());
    18081804    tokenRecord->m_startPosition = currentPosition();
     1805
     1806    if (atEnd()) {
     1807        token = EOFTOK;
     1808        goto returnToken;
     1809    }
    18091810
    18101811    CharacterType type;
    … …  
    19171918        if (m_current == '+') {
    19181919            shift();
    1919             token = (!m_terminator) ? PLUSPLUS : AUTOPLUSPLUS;
     1920            token = (!m_hasLineTerminatorBeforeToken) ? PLUSPLUS : AUTOPLUSPLUS;
    19201921            break;
    19211922        }
    … …  
    19311932        if (m_current == '-') {
    19321933            shift();
    1933             if ((m_atLineStart || m_terminator) && m_current == '>') {
     1934            if ((m_atLineStart || m_hasLineTerminatorBeforeToken) && m_current == '>') {
    19341935                if (m_scriptMode == JSParserScriptMode::Classic) {
    19351936                    shift();
    … …  
    19371938                }
    19381939            }
    1939             token = (!m_terminator) ? MINUSMINUS : AUTOMINUSMINUS;
     1940            token = (!m_hasLineTerminatorBeforeToken) ? MINUSMINUS : AUTOMINUSMINUS;
    19401941            break;
    19411942        }
    … …  
    23082309        shiftLineTerminator();
    23092310        m_atLineStart = true;
    2310         m_terminator = true;
     2311        m_hasLineTerminatorBeforeToken = true;
    23112312        m_lineStart = m_code;
    23122313        goto start;
    … …  
    23482349
    23492350        while (!isLineTerminator(m_current)) {
    2350             if (atEnd())
    2351                 return EOFTOK;
     2351            if (atEnd()) {
     2352                token = EOFTOK;
     2353                fillTokenInfo(tokenRecord, token, lineNumber, endOffset, lineStartOffset, endPosition);
     2354                return token;
     2355            }
    23522356            shift();
    23532357        }
    23542358        shiftLineTerminator();
    23552359        m_atLineStart = true;
    2356         m_terminator = true;
     2360        m_hasLineTerminatorBeforeToken = true;
    23572361        m_lineStart = m_code;
    23582362        if (!lastTokenWasRestrKeyword())

  • branches/safari-607-branch/Source/JavaScriptCore/parser/Lexer.h

    r239427 r245348  
    6666
    6767    JSTokenType lex(JSToken*, unsigned, bool strictMode);
     68    JSTokenType lexWithoutClearingLineTerminator(JSToken*, unsigned, bool strictMode);
    6869    bool nextTokenIsColon();
    6970    int lineNumber() const { return m_lineNumber; }
    … …  
    7879    void setLastLineNumber(int lastLineNumber) { m_lastLineNumber = lastLineNumber; }
    7980    int lastLineNumber() const { return m_lastLineNumber; }
    80     bool prevTerminator() const { return m_terminator; }
     81    bool hasLineTerminatorBeforeToken() const { return m_hasLineTerminatorBeforeToken; }
    8182    JSTokenType scanRegExp(JSToken*, UChar patternPrefix = 0);
    8283    enum class RawStringsBuildMode { BuildRawStrings, DontBuildRawStrings };
    … …  
    111112        m_lineNumber = line;
    112113    }
    113     void setTerminator(bool terminator)
     114    void setHasLineTerminatorBeforeToken(bool terminator)
    114115    {
    115         m_terminator = terminator;
     116        m_hasLineTerminatorBeforeToken = terminator;
    116117    }
    117118
    … …  
    203204    Vector<UChar> m_buffer16;
    204205    Vector<UChar> m_bufferForRawTemplateString16;
    205     bool m_terminator;
     206    bool m_hasLineTerminatorBeforeToken;
    206207    int m_lastToken;
    207208
    … …  
    404405}
    405406
     407template <typename T>
     408ALWAYS_INLINE JSTokenType Lexer<T>::lex(JSToken* tokenRecord, unsigned lexerFlags, bool strictMode)
     409{
     410    m_hasLineTerminatorBeforeToken = false;
     411    return lexWithoutClearingLineTerminator(tokenRecord, lexerFlags, strictMode);
     412}
     413
    406414} // namespace JSC

  • branches/safari-607-branch/Source/JavaScriptCore/parser/Parser.cpp

    r240371 r245348  
    346346bool Parser<LexerType>::allowAutomaticSemicolon()
    347347{
    348     return match(CLOSEBRACE) || match(EOFTOK) || m_lexer->prevTerminator();
     348    return match(CLOSEBRACE) || match(EOFTOK) || m_lexer->hasLineTerminatorBeforeToken();
    349349}
    350350
    … …  
    626626        if (*m_token.m_data.ident == m_vm->propertyNames->async && !m_token.m_data.escaped) {
    627627            next();
    628             failIfFalse(match(FUNCTION) && !m_lexer->prevTerminator(), "Cannot parse the async function");
     628            failIfFalse(match(FUNCTION) && !m_lexer->hasLineTerminatorBeforeToken(), "Cannot parse the async function");
    629629            statement = parseAsyncFunctionDeclaration(context, ExportType::NotExported, DeclarationDefaultContext::Standard, functionConstructorParametersEndPosition);
    630630            break;
    … …  
    697697            SavePoint savePoint = createSavePoint();
    698698            next();
    699             if (UNLIKELY(match(FUNCTION) && !m_lexer->prevTerminator())) {
     699            if (UNLIKELY(match(FUNCTION) && !m_lexer->hasLineTerminatorBeforeToken())) {
    700700                result = parseAsyncFunctionDeclaration(context);
    701701                break;
    … …  
    20272027    SavePoint savePoint = createSavePoint();
    20282028    next();
    2029     if (match(FUNCTION) && !m_lexer->prevTerminator()) {
     2029    if (match(FUNCTION) && !m_lexer->hasLineTerminatorBeforeToken()) {
    20302030        const bool isAsync = true;
    20312031        result = parseFunctionDeclarationStatement(context, isAsync, parentAllowsFunctionDeclarationAsStatement);
    … …  
    24222422        matchOrFail(ARROWFUNCTION, "Expected a '=>' after arrow function parameter declaration");
    24232423
    2424         if (m_lexer->prevTerminator())
     2424        if (m_lexer->hasLineTerminatorBeforeToken())
    24252425            failDueToUnexpectedToken();
    24262426
    … …  
    26142614        newInfo = SourceProviderCacheItem::create(parameters);
    26152615    }
     2616
     2617    bool functionScopeWasStrictMode = functionScope->strictMode();
    26162618   
    26172619    popScope(functionScope, TreeBuilder::NeedsFreeVariableInfo);
    … …  
    26202622        matchOrFail(CLOSEBRACE, "Expected a closing '}' after a ", stringForFunctionMode(mode), " body");
    26212623        next();
     2624    } else {
     2625        // We need to lex the last token again because the last token is lexed under the different context because of the following possibilities.
     2626        // 1. which may have different strict mode.
     2627        // 2. which may not build strings for tokens.
     2628        // But (1) is not possible because we do not recognize the string literal in ArrowFunctionBodyExpression as directive and this is correct in terms of the spec (`value => "use strict"`).
     2629        // So we only check TreeBuilder's type here.
     2630        ASSERT_UNUSED(functionScopeWasStrictMode, functionScopeWasStrictMode == currentScope()->strictMode());
     2631        if (!std::is_same<TreeBuilder, SyntaxChecker>::value)
     2632            lexCurrentTokenAgainUnderCurrentContext();
    26222633    }
    26232634
    … …  
    28782889                    ident = m_token.m_data.ident;
    28792890                    next();
    2880                     if (match(OPENPAREN) || match(COLON) || match(EQUAL) || m_lexer->prevTerminator())
     2891                    if (match(OPENPAREN) || match(COLON) || match(EQUAL) || m_lexer->hasLineTerminatorBeforeToken())
    28812892                        break;
    28822893                    if (UNLIKELY(consume(TIMES)))
    … …  
    34033414            SavePoint savePoint = createSavePoint();
    34043415            next();
    3405             if (match(FUNCTION) && !m_lexer->prevTerminator()) {
     3416            if (match(FUNCTION) && !m_lexer->hasLineTerminatorBeforeToken()) {
    34063417                next();
    34073418                if (match(IDENT))
    … …  
    35483559            if (*m_token.m_data.ident == m_vm->propertyNames->async && !m_token.m_data.escaped) {
    35493560                next();
    3550                 semanticFailIfFalse(match(FUNCTION) && !m_lexer->prevTerminator(), "Expected 'function' keyword following 'async' keyword with no preceding line terminator");
     3561                semanticFailIfFalse(match(FUNCTION) && !m_lexer->hasLineTerminatorBeforeToken(), "Expected 'function' keyword following 'async' keyword with no preceding line terminator");
    35513562                DepthManager statementDepth(&m_statementDepth);
    35523563                m_statementDepth = 1;
    … …  
    36663677                if (matchContextualKeyword(m_vm->propertyNames->async)) {
    36673678                    next();
    3668                     isAsyncArrow = !m_lexer->prevTerminator();
     3679                    isAsyncArrow = !m_lexer->hasLineTerminatorBeforeToken();
    36693680                }
    36703681            }
    … …  
    37833794    SavePoint savePoint = createSavePoint();
    37843795    next();
    3785     if (m_lexer->prevTerminator())
     3796    if (m_lexer->hasLineTerminatorBeforeToken())
    37863797        return context.createYield(location);
    37873798
    … …  
    39433954                }
    39443955
    3945                 failIfTrue(m_lexer->prevTerminator(), "Expected a property name following keyword 'async'");
     3956                failIfTrue(m_lexer->hasLineTerminatorBeforeToken(), "Expected a property name following keyword 'async'");
    39463957                if (UNLIKELY(consume(TIMES)))
    39473958                    parseMode = SourceParseMode::AsyncGeneratorWrapperMethodMode;
    … …  
    44954506            JSTokenLocation location(tokenLocation());
    44964507            next();
    4497             if (match(FUNCTION) && !m_lexer->prevTerminator())
     4508            if (match(FUNCTION) && !m_lexer->hasLineTerminatorBeforeToken())
    44984509                return parseAsyncFunctionExpression(context);
    44994510
    … …  
    47604771        base = parsePrimaryExpression(context);
    47614772        failIfFalse(base, "Cannot parse base expression");
    4762         if (UNLIKELY(isAsync && context.isResolve(base) && !m_lexer->prevTerminator())) {
     4773        if (UNLIKELY(isAsync && context.isResolve(base) && !m_lexer->hasLineTerminatorBeforeToken())) {
    47634774            if (matchSpecIdentifier()) {
    47644775                // AsyncArrowFunction

  • branches/safari-607-branch/Source/JavaScriptCore/parser/Parser.h

    r239427 r245348  
    13651365    }
    13661366
     1367    ALWAYS_INLINE void nextWithoutClearingLineTerminator(unsigned lexerFlags = 0)
     1368    {
     1369        int lastLine = m_token.m_location.line;
     1370        int lastTokenEnd = m_token.m_location.endOffset;
     1371        int lastTokenLineStart = m_token.m_location.lineStartOffset;
     1372        m_lastTokenEndPosition = JSTextPosition(lastLine, lastTokenEnd, lastTokenLineStart);
     1373        m_lexer->setLastLineNumber(lastLine);
     1374        m_token.m_type = m_lexer->lexWithoutClearingLineTerminator(&m_token, lexerFlags, strictMode());
     1375    }
     1376
    13671377    ALWAYS_INLINE void nextExpectIdentifier(unsigned lexerFlags = 0)
    13681378    {
    … …  
    13731383        m_lexer->setLastLineNumber(lastLine);
    13741384        m_token.m_type = m_lexer->lexExpectIdentifier(&m_token, lexerFlags, strictMode());
     1385    }
     1386
     1387    ALWAYS_INLINE void lexCurrentTokenAgainUnderCurrentContext()
     1388    {
     1389        auto savePoint = createSavePoint();
     1390        restoreSavePoint(savePoint);
    13751391    }
    13761392
    … …  
    17641780        unsigned oldLastLineNumber;
    17651781        unsigned oldLineNumber;
     1782        bool hasLineTerminatorBeforeToken;
    17661783    };
    17671784
    … …  
    17771794        result.oldLastLineNumber = m_lexer->lastLineNumber();
    17781795        result.oldLineNumber = m_lexer->lineNumber();
     1796        result.hasLineTerminatorBeforeToken = m_lexer->hasLineTerminatorBeforeToken();
    17791797        ASSERT(static_cast<unsigned>(result.startOffset) >= result.oldLineStartOffset);
    17801798        return result;
    … …  
    17861804        m_lexer->setOffset(lexerState.startOffset, lexerState.oldLineStartOffset);
    17871805        m_lexer->setLineNumber(lexerState.oldLineNumber);
    1788         next();
     1806        m_lexer->setHasLineTerminatorBeforeToken(lexerState.hasLineTerminatorBeforeToken);
     1807        nextWithoutClearingLineTerminator();
    17891808        m_lexer->setLastLineNumber(lexerState.oldLastLineNumber);
    17901809    }
Note: See TracChangeset for help on using the changeset viewer.