Changeset 247171 in webkit


Ignore:
Timestamp:
Jul 5, 2019 1:10:36 PM (5 years ago)
Author:
rmorisset@apple.com
Message:

[WHLSL] Optimize the lexer
​https://bugs.webkit.org/show_bug.cgi?id=199520

Reviewed by Myles Maxfield.

This patch includes a number of small optimizations to the lexer:

  • By creating two new types of Token (Invalid and EOF), we can avoid having to manipulate Optional<Token> everywhere
  • By combining the lexing of identifiers, keywords and operator names, we can avoid re-reading them several times
  • By storing mere offsets in the tokens instead of string views, we save space in the common case where the string view is unused (e.g. for keywords)
  • By storing mere offsets in the tokens instead of line numbers, we can simplify the whitespace and comment lexing; and we can easily recompute the line number on demand
  • By using templates we can avoid constructing and destructing vectors at runtime every time we call tryTypes, consumeTypes or peekTypes; as well as avoiding a call to find().
  • I also hoisted the length check out of the loop in Lexer::string()

All of this combined brought the time spent in the parser when running LayoutTests/webgpu/whlsl-test-harness-test.html 5 times on my machine from 612ms to 462ms.

I also removed the 'anyCharacter' method since it was dead code.
I also removed Lexer::position(), but it is directly equivalent now to peek().startOffset.

No new tests as there is no intended functional change.

  • Modules/webgpu/WHLSL/WHLSLLexer.cpp:

(WebCore::WHLSL::Lexer::Token::typeName):
(WebCore::WHLSL::Lexer::consumeTokenFromStream):
(WebCore::WHLSL::Lexer::lineNumberFromOffset):
(WebCore::WHLSL::isWhitespace):
(WebCore::WHLSL::isNewline):
(WebCore::WHLSL::Lexer::skipWhitespaceAndComments):
(WebCore::WHLSL::Lexer::digitStar const):
(WebCore::WHLSL::Lexer::completeOperatorName const):

  • Modules/webgpu/WHLSL/WHLSLLexer.h:

(WebCore::WHLSL::Lexer::Lexer):
(WebCore::WHLSL::Lexer::Token::stringView):
(WebCore::WHLSL::Lexer::consumeToken):
(WebCore::WHLSL::Lexer::peek const):
(WebCore::WHLSL::Lexer::peekFurther const):
(WebCore::WHLSL::Lexer::state const):
(WebCore::WHLSL::Lexer::setState):
(WebCore::WHLSL::Lexer::isFullyConsumed const):
(WebCore::WHLSL::Lexer::errorString):
(WebCore::WHLSL::Lexer::string const):

  • Modules/webgpu/WHLSL/WHLSLNameResolver.h:
  • Modules/webgpu/WHLSL/WHLSLParser.cpp:

(WebCore::WHLSL::Parser::parse):
(WebCore::WHLSL::Parser::peek):
(WebCore::WHLSL::Parser::peekFurther):
(WebCore::WHLSL::Types::includes):
(WebCore::WHLSL::Types::appendNameTo):
(WebCore::WHLSL::Types<t>::includes):
(WebCore::WHLSL::Types<t>::appendNameTo):
(WebCore::WHLSL::Parser::peekTypes):
(WebCore::WHLSL::Parser::tryType):
(WebCore::WHLSL::Parser::tryTypes):
(WebCore::WHLSL::Parser::consumeType):
(WebCore::WHLSL::Parser::consumeTypes):
(WebCore::WHLSL::Parser::consumeIntegralLiteral):
(WebCore::WHLSL::Parser::parseConstantExpression):
(WebCore::WHLSL::Parser::parseTypeArgument):
(WebCore::WHLSL::Parser::parseTypeSuffixAbbreviated):
(WebCore::WHLSL::Parser::parseTypeSuffixNonAbbreviated):
(WebCore::WHLSL::Parser::parseType):
(WebCore::WHLSL::Parser::parseTypeDefinition):
(WebCore::WHLSL::Parser::parseBuiltInSemantic):
(WebCore::WHLSL::Parser::parseResourceSemantic):
(WebCore::WHLSL::Parser::parseQualifiers):
(WebCore::WHLSL::Parser::parseStructureElement):
(WebCore::WHLSL::Parser::parseStructureDefinition):
(WebCore::WHLSL::Parser::parseEnumerationDefinition):
(WebCore::WHLSL::Parser::parseEnumerationMember):
(WebCore::WHLSL::Parser::parseNativeTypeDeclaration):
(WebCore::WHLSL::Parser::parseParameter):
(WebCore::WHLSL::Parser::parseComputeFunctionDeclaration):
(WebCore::WHLSL::Parser::parseVertexOrFragmentFunctionDeclaration):
(WebCore::WHLSL::Parser::parseRegularFunctionDeclaration):
(WebCore::WHLSL::Parser::parseBlockBody):
(WebCore::WHLSL::Parser::parseSwitchCase):
(WebCore::WHLSL::Parser::parseVariableDeclaration):
(WebCore::WHLSL::Parser::parseStatement):
(WebCore::WHLSL::Parser::parseEffectfulAssignment):
(WebCore::WHLSL::Parser::parseLimitedSuffixOperator):
(WebCore::WHLSL::Parser::parseSuffixOperator):
(WebCore::WHLSL::Parser::completeAssignment):
(WebCore::WHLSL::Parser::parsePossibleTernaryConditional):
(WebCore::WHLSL::Parser::completePossibleLogicalBinaryOperation):
(WebCore::WHLSL::Parser::completePossibleRelationalBinaryOperation):
(WebCore::WHLSL::Parser::completePossibleShift):
(WebCore::WHLSL::Parser::completePossibleAdd):
(WebCore::WHLSL::Parser::completePossibleMultiply):
(WebCore::WHLSL::Parser::parsePossiblePrefix):
(WebCore::WHLSL::Parser::parseCallExpression):
(WebCore::WHLSL::Parser::parseTerm):

  • Modules/webgpu/WHLSL/WHLSLParser.h:
  • Modules/webgpu/WHLSL/WHLSLPreserveVariableLifetimes.cpp:

(WebCore::WHLSL::anonymousToken):

Location:
trunk/Source/WebCore
Files:
7 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/WebCore/ChangeLog

    r247170 r247171  
     12019-07-05  Robin Morisset  <rmorisset@apple.com>
     2
     3        [WHLSL] Optimize the lexer
     4        https://bugs.webkit.org/show_bug.cgi?id=199520
     5
     6        Reviewed by Myles Maxfield.
     7
     8        This patch includes a number of small optimizations to the lexer:
     9        - By creating two new types of Token (Invalid and EOF), we can avoid having to manipulate Optional<Token> everywhere
     10        - By combining the lexing of identifiers, keywords and operator names, we can avoid re-reading them several times
     11        - By storing mere offsets in the tokens instead of string views, we save space in the common case where the string view is unused (e.g. for keywords)
     12        - By storing mere offsets in the tokens instead of line numbers, we can simplify the whitespace and comment lexing; and we can easily recompute the line number on demand
     13        - By using templates we can avoid constructing and destructing vectors at runtime every time we call tryTypes, consumeTypes or peekTypes; as well as avoiding a call to find().
     14        - I also hoisted the length check out of the loop in Lexer::string()
     15        All of this combined brought the time spent in the parser when running LayoutTests/webgpu/whlsl-test-harness-test.html 5 times on my machine from 612ms to 462ms.
     16
     17        I also removed the 'anyCharacter' method since it was dead code.
     18        I also removed Lexer::position(), but it is directly equivalent now to peek().startOffset.
     19
     20        No new tests as there is no intended functional change.
     21
     22        * Modules/webgpu/WHLSL/WHLSLLexer.cpp:
     23        (WebCore::WHLSL::Lexer::Token::typeName):
     24        (WebCore::WHLSL::Lexer::consumeTokenFromStream):
     25        (WebCore::WHLSL::Lexer::lineNumberFromOffset):
     26        (WebCore::WHLSL::isWhitespace):
     27        (WebCore::WHLSL::isNewline):
     28        (WebCore::WHLSL::Lexer::skipWhitespaceAndComments):
     29        (WebCore::WHLSL::Lexer::digitStar const):
     30        (WebCore::WHLSL::Lexer::completeOperatorName const):
     31        * Modules/webgpu/WHLSL/WHLSLLexer.h:
     32        (WebCore::WHLSL::Lexer::Lexer):
     33        (WebCore::WHLSL::Lexer::Token::stringView):
     34        (WebCore::WHLSL::Lexer::consumeToken):
     35        (WebCore::WHLSL::Lexer::peek const):
     36        (WebCore::WHLSL::Lexer::peekFurther const):
     37        (WebCore::WHLSL::Lexer::state const):
     38        (WebCore::WHLSL::Lexer::setState):
     39        (WebCore::WHLSL::Lexer::isFullyConsumed const):
     40        (WebCore::WHLSL::Lexer::errorString):
     41        (WebCore::WHLSL::Lexer::string const):
     42        * Modules/webgpu/WHLSL/WHLSLNameResolver.h:
     43        * Modules/webgpu/WHLSL/WHLSLParser.cpp:
     44        (WebCore::WHLSL::Parser::parse):
     45        (WebCore::WHLSL::Parser::peek):
     46        (WebCore::WHLSL::Parser::peekFurther):
     47        (WebCore::WHLSL::Types::includes):
     48        (WebCore::WHLSL::Types::appendNameTo):
     49        (WebCore::WHLSL::Types<t>::includes):
     50        (WebCore::WHLSL::Types<t>::appendNameTo):
     51        (WebCore::WHLSL::Parser::peekTypes):
     52        (WebCore::WHLSL::Parser::tryType):
     53        (WebCore::WHLSL::Parser::tryTypes):
     54        (WebCore::WHLSL::Parser::consumeType):
     55        (WebCore::WHLSL::Parser::consumeTypes):
     56        (WebCore::WHLSL::Parser::consumeIntegralLiteral):
     57        (WebCore::WHLSL::Parser::parseConstantExpression):
     58        (WebCore::WHLSL::Parser::parseTypeArgument):
     59        (WebCore::WHLSL::Parser::parseTypeSuffixAbbreviated):
     60        (WebCore::WHLSL::Parser::parseTypeSuffixNonAbbreviated):
     61        (WebCore::WHLSL::Parser::parseType):
     62        (WebCore::WHLSL::Parser::parseTypeDefinition):
     63        (WebCore::WHLSL::Parser::parseBuiltInSemantic):
     64        (WebCore::WHLSL::Parser::parseResourceSemantic):
     65        (WebCore::WHLSL::Parser::parseQualifiers):
     66        (WebCore::WHLSL::Parser::parseStructureElement):
     67        (WebCore::WHLSL::Parser::parseStructureDefinition):
     68        (WebCore::WHLSL::Parser::parseEnumerationDefinition):
     69        (WebCore::WHLSL::Parser::parseEnumerationMember):
     70        (WebCore::WHLSL::Parser::parseNativeTypeDeclaration):
     71        (WebCore::WHLSL::Parser::parseParameter):
     72        (WebCore::WHLSL::Parser::parseComputeFunctionDeclaration):
     73        (WebCore::WHLSL::Parser::parseVertexOrFragmentFunctionDeclaration):
     74        (WebCore::WHLSL::Parser::parseRegularFunctionDeclaration):
     75        (WebCore::WHLSL::Parser::parseBlockBody):
     76        (WebCore::WHLSL::Parser::parseSwitchCase):
     77        (WebCore::WHLSL::Parser::parseVariableDeclaration):
     78        (WebCore::WHLSL::Parser::parseStatement):
     79        (WebCore::WHLSL::Parser::parseEffectfulAssignment):
     80        (WebCore::WHLSL::Parser::parseLimitedSuffixOperator):
     81        (WebCore::WHLSL::Parser::parseSuffixOperator):
     82        (WebCore::WHLSL::Parser::completeAssignment):
     83        (WebCore::WHLSL::Parser::parsePossibleTernaryConditional):
     84        (WebCore::WHLSL::Parser::completePossibleLogicalBinaryOperation):
     85        (WebCore::WHLSL::Parser::completePossibleRelationalBinaryOperation):
     86        (WebCore::WHLSL::Parser::completePossibleShift):
     87        (WebCore::WHLSL::Parser::completePossibleAdd):
     88        (WebCore::WHLSL::Parser::completePossibleMultiply):
     89        (WebCore::WHLSL::Parser::parsePossiblePrefix):
     90        (WebCore::WHLSL::Parser::parseCallExpression):
     91        (WebCore::WHLSL::Parser::parseTerm):
     92        * Modules/webgpu/WHLSL/WHLSLParser.h:
     93        * Modules/webgpu/WHLSL/WHLSLPreserveVariableLifetimes.cpp:
     94        (WebCore::WHLSL::anonymousToken):
     95
    1962019-07-05  Robin Morisset  <rmorisset@apple.com>
    297

  • trunk/Source/WebCore/Modules/webgpu/WHLSL/WHLSLLexer.cpp

    r243091 r247171  
    248248    case Type::At:
    249249        return "@";
     250    case Type::EndOfFile:
     251        return "EOF";
     252    case Type::Invalid:
     253        return "LEXING_ERROR";
    250254    }
    251255}
    … …  
    373377}
    374378
    375 auto Lexer::consumeTokenFromStream() -> Optional<Token>
    376 {
    377     auto prepare = [&](unsigned newOffset, Token::Type type) -> Optional<Token> {
     379auto Lexer::consumeTokenFromStream() -> Token
     380{
     381    auto prepare = [&](unsigned newOffset, Token::Type type) -> Token {
    378382        auto oldOffset = m_offset;
    379383        m_offset = newOffset;
    380384        skipWhitespaceAndComments();
    381         return {{ m_stringView.substring(oldOffset, newOffset - oldOffset), m_lineNumber, type }};
     385        return { oldOffset, newOffset, type };
    382386    };
    383387
     388    if (auto newOffset = identifier(m_offset)) {
     389        if (auto result = recognizeKeyword(*newOffset)) {
     390            if (*result == Token::Type::Operator) {
     391                if (auto newerOffset = completeOperatorName(*newOffset))
     392                    return prepare(*newerOffset, Token::Type::OperatorName);
     393            }
     394            return prepare(*newOffset, *result);
     395        }
     396        return prepare(*newOffset, Token::Type::Identifier);
     397    }
    384398    if (auto newOffset = floatLiteral(m_offset))
    385399        return prepare(*newOffset, Token::Type::FloatLiteral);
    … …  
    388402    if (auto newOffset = intLiteral(m_offset))
    389403        return prepare(*newOffset, Token::Type::IntLiteral);
    390     if (auto newOffset = operatorName(m_offset))
    391         return prepare(*newOffset, Token::Type::OperatorName);
    392     if (auto newOffset = identifier(m_offset)) {
    393         if (auto result = recognizeKeyword(*newOffset))
    394             return prepare(*newOffset, *result);
    395         return prepare(*newOffset, Token::Type::Identifier);
    396     }
    397404    // Sorted by length, so longer matches are preferable to shorter matches.
    398405    if (auto newOffset = string(">>=", m_offset))
    … …  
    491498        return prepare(*newOffset, Token::Type::At);
    492499
    493     return WTF::nullopt;
    494 }
    495 
    496 void Lexer::skipWhitespaceAndComments()
    497 {
    498     unsigned savedOffset;
    499     do {
    500         savedOffset = m_offset;
    501         skipWhitespace();
    502         skipLineComment();
    503         skipLongComment();
    504     } while (savedOffset != m_offset);
    505 }
     500    if (m_offset == m_stringView.length())
     501        return prepare(m_offset, Token::Type::EndOfFile);
     502    return prepare(m_offset, Token::Type::Invalid);
     503}
     504
     505unsigned Lexer::lineNumberFromOffset(unsigned targetOffset)
     506{
     507    // Counting from 1 to match most text editors.
     508    unsigned lineNumber = 1;
     509    for (unsigned offset = 0; offset < targetOffset; ++offset) {
     510        if (m_stringView[offset] == '\n')
     511            ++lineNumber;
     512    }
     513    return lineNumber;
     514}
     515
     516// We can take advantage of two properties of Unicode:
     517// 1. The consitutent UTF-16 code units for all non-BMP code points are surrogates,
     518// which means we'll never see a false match. If we see a BMP code unit, we
     519// really have a BMP code point.
     520// 2. Everything we're looking for is in BMP
    506521
    507522static inline bool isWhitespace(UChar codeUnit)
    … …  
    529544}
    530545
    531 // We can take advantage of two properties of Unicode:
    532 // 1. The consitutent UTF-16 code units for all non-BMP code points are surrogates,
    533 //        which means we'll never see a false match. If we see a BMP code unit, we
    534 //        really have a BMP code point.
    535 // 2. Everything we're looking for is in BMP
    536 
    537 void Lexer::skipWhitespace()
    538 {
    539     for ( ; m_offset < m_stringView.length() && isWhitespace(m_stringView[m_offset]); ++m_offset) {
    540         if (m_stringView[m_offset] == '\r' && m_offset + 1 < m_stringView.length() && m_stringView[m_offset + 1] == '\n') {
     546void Lexer::skipWhitespaceAndComments()
     547{
     548    while (m_offset < m_stringView.length()) {
     549        if (isWhitespace(m_stringView[m_offset]))
    541550            ++m_offset;
    542             ++m_lineNumber;
    543         } else if (isNewline(m_stringView[m_offset]))
    544             ++m_lineNumber;
    545     }
    546 }
    547 
    548 void Lexer::skipLineComment()
    549 {
    550     if (m_offset + 1 >= m_stringView.length() || m_stringView[m_offset] != '/' || m_stringView[m_offset + 1] != '/')
    551         return;
    552 
    553     m_offset += 2;
    554     for ( ; m_offset < m_stringView.length() && !isNewline(m_stringView[m_offset]); ++m_offset) { }
    555 }
    556 
    557 void Lexer::skipLongComment()
    558 {
    559     if (m_offset + 1 >= m_stringView.length() || m_stringView[m_offset] != '/' || m_stringView[m_offset + 1] != '*')
    560         return;
    561 
    562     m_offset += 2;
    563     do {
    564         for ( ; m_offset < m_stringView.length() && m_stringView[m_offset] != '*'; ++m_offset) {
    565             if (m_stringView[m_offset] == '\r' && m_offset + 1 < m_stringView.length() && m_stringView[m_offset + 1] == '\n') {
    566                 ++m_offset;
    567                 ++m_lineNumber;
    568             } else if (isNewline(m_stringView[m_offset]))
    569                 ++m_lineNumber;
    570         }
    571         if (m_offset < m_stringView.length())
    572             ++m_offset;
    573         if (m_offset < m_stringView.length() && m_stringView[m_offset] == '/') {
    574             ++m_offset;
     551        else if (m_stringView[m_offset] == '/' && m_offset + 1 < m_stringView.length()) {
     552            if (m_stringView[m_offset + 1] == '/') {
     553                // Line comment
     554                m_offset += 2;
     555                // Note that in the case of \r\n this makes the comment end on the \r. It is fine, as the \n after that is simple whitespace.
     556                for ( ; m_offset < m_stringView.length() && !isNewline(m_stringView[m_offset]); ++m_offset) { }
     557            } else if (m_stringView[m_offset + 1] == '*') {
     558                // Long comment
     559                for ( ; m_offset < m_stringView.length() ; ++m_offset) {
     560                    if (m_stringView[m_offset] == '*' && m_offset + 1 < m_stringView.length() && m_stringView[m_offset + 1] == '/') {
     561                        m_offset += 2;
     562                        break;
     563                    }
     564                }
     565            }
     566        } else
    575567            break;
    576         }
    577     } while (m_offset < m_stringView.length());
     568    }
    578569}
    579570
    … …  
    677668unsigned Lexer::digitStar(unsigned offset) const
    678669{
    679     while (true) {
    680         auto result = digit(offset);
    681         if (!result)
    682             return offset;
     670    while (auto result = digit(offset))
    683671        offset = *result;
    684     }
     672    return offset;
    685673}
    686674
    … …  
    755743}
    756744
    757 Optional<unsigned> Lexer::operatorName(unsigned offset) const
    758 {
    759     if (auto result = string("operator&.", offset))
     745Optional<unsigned> Lexer::completeOperatorName(unsigned offset) const
     746{
     747    // Sorted by length, so longer matches are preferable to shorter matches.
     748    if (auto result = string("&[]", offset))
     749        return result;
     750    if (auto result = string("[]=", offset))
     751        return result;
     752    if (auto result = string(">>", offset))
     753        return result;
     754    if (auto result = string("<<", offset))
     755        return result;
     756    if (auto result = string("++", offset))
     757        return result;
     758    if (auto result = string("--", offset))
     759        return result;
     760    if (auto result = string("&&", offset))
     761        return result;
     762    if (auto result = string("||", offset))
     763        return result;
     764    if (auto result = string(">=", offset))
     765        return result;
     766    if (auto result = string("<=", offset))
     767        return result;
     768    if (auto result = string("==", offset))
     769        return result;
     770    if (auto result = string("[]", offset))
     771        return result;
     772    if (auto result = string("&.", offset))
    760773        return validIdentifier(*result);
    761     if (auto result = string("operator.", offset)) {
    762         if ((result = validIdentifier(*result))) {
    763             if (auto result2 = character('=', *result))
    764                 return result2;
    765             return *result;
     774    if (auto result = character('+', offset))
     775        return result;
     776    if (auto result = character('-', offset))
     777        return result;
     778    if (auto result = character('*', offset))
     779        return result;
     780    if (auto result = character('/', offset))
     781        return result;
     782    if (auto result = character('%', offset))
     783        return result;
     784    if (auto result = character('<', offset))
     785        return result;
     786    if (auto result = character('>', offset))
     787        return result;
     788    if (auto result = character('!', offset))
     789        return result;
     790    if (auto result = character('~', offset))
     791        return result;
     792    if (auto result = character('&', offset))
     793        return result;
     794    if (auto result = character('^', offset))
     795        return result;
     796    if (auto result = character('|', offset))
     797        return result;
     798    if (auto result = character('.', offset)) {
     799        if (auto result2 = validIdentifier(*result)) {
     800            if (auto result3 = character('=', *result2))
     801                return result3;
     802            return *result2;
    766803        }
    767804    }
    768     if (auto result = string("operator", offset)) {
    769         // Sorted by length, so longer matches are preferable to shorter matches.
    770         if (auto result2 = string("&[]", *result))
    771             return result2;
    772         if (auto result2 = string("[]=", *result))
    773             return result2;
    774         if (auto result2 = string(">>", *result))
    775             return result2;
    776         if (auto result2 = string("<<", *result))
    777             return result2;
    778         if (auto result2 = string("++", *result))
    779             return result2;
    780         if (auto result2 = string("--", *result))
    781             return result2;
    782         if (auto result2 = string("&&", *result))
    783             return result2;
    784         if (auto result2 = string("||", *result))
    785             return result2;
    786         if (auto result2 = string(">=", *result))
    787             return result2;
    788         if (auto result2 = string("<=", *result))
    789             return result2;
    790         if (auto result2 = string("==", *result))
    791             return result2;
    792         if (auto result2 = string("[]", *result))
    793             return result2;
    794         if (auto result2 = character('+', *result))
    795             return result2;
    796         if (auto result2 = character('-', *result))
    797             return result2;
    798         if (auto result2 = character('*', *result))
    799             return result2;
    800         if (auto result2 = character('/', *result))
    801             return result2;
    802         if (auto result2 = character('%', *result))
    803             return result2;
    804         if (auto result2 = character('<', *result))
    805             return result2;
    806         if (auto result2 = character('>', *result))
    807             return result2;
    808         if (auto result2 = character('!', *result))
    809             return result2;
    810         if (auto result2 = character('~', *result))
    811             return result2;
    812         if (auto result2 = character('&', *result))
    813             return result2;
    814         if (auto result2 = character('^', *result))
    815             return result2;
    816         if (auto result2 = character('|', *result))
    817             return result2;
    818     }
    819805    return WTF::nullopt;
    820806}

  • trunk/Source/WebCore/Modules/webgpu/WHLSL/WHLSLLexer.h

    r247164 r247171  
    5858
    5959    struct Token {
    60         Token() = delete;
    61         Token(const Token&) = default;
    62         Token(Token&&) = default;
    63         Token& operator=(const Token&) = default;
    64         Token& operator=(Token&&) = default;
    65 
    66         StringView stringView;
    67         unsigned lineNumber;
    68         enum class Type {
     60        unsigned startOffset;
     61        unsigned endOffset;
     62        enum class Type : uint8_t {
    6963            IntLiteral,
    7064            UintLiteral,
    … …  
    173167            ExclamationPoint,
    174168            At,
    175         } type;
     169            EndOfFile,
     170            Invalid
     171        } type {Type::Invalid};
    176172
    177173        static const char* typeName(Type);
     174
     175        const StringView stringView(const Lexer& lexer)
     176        {
     177            return lexer.m_stringView.substring(startOffset, endOffset - startOffset);
     178        }
    178179    };
    179180
    180     Optional<Token> consumeToken()
     181    Token consumeToken()
    181182    {
    182183        auto result = m_ringBuffer[m_ringBufferIndex];
    … …  
    186187    }
    187188
    188     Optional<Token> peek()
     189    Token peek() const
    189190    {
    190191        return m_ringBuffer[m_ringBufferIndex];
    191192    }
    192193
    193     Optional<Token> peekFurther()
     194    Token peekFurther() const
    194195    {
    195196        return m_ringBuffer[(m_ringBufferIndex + 1) % 2];
    … …  
    199200    // https://bugs.webkit.org/show_bug.cgi?id=198357
    200201    struct State {
    201         Optional<Token> ringBuffer[2];
     202        Token ringBuffer[2];
    202203        unsigned ringBufferIndex;
    203204        unsigned offset;
    204         unsigned lineNumber;
    205205    };
    206206
    … …  
    212212        state.ringBufferIndex = m_ringBufferIndex;
    213213        state.offset = m_offset;
    214         state.lineNumber = m_lineNumber;
    215214        return state;
    216215    }
    … …  
    222221        m_ringBufferIndex = state.ringBufferIndex;
    223222        m_offset = state.offset;
    224         m_lineNumber = state.lineNumber;
    225223
    226224    }
    … …  
    232230        m_ringBufferIndex = WTFMove(state.ringBufferIndex);
    233231        m_offset = WTFMove(state.offset);
    234         m_lineNumber = WTFMove(state.lineNumber);
    235232    }
    236233
    237234    bool isFullyConsumed() const
    238235    {
    239         return m_offset == m_stringView.length();
     236        return peek().type == Token::Type::EndOfFile;
    240237    }
    241238
    242239    String errorString(const Token& token, const String& message)
    243240    {
    244         return makeString("Parse error at line ", token.lineNumber, ": ", message);
     241        return makeString("Parse error at line ", lineNumberFromOffset(token.startOffset), ": ", message);
    245242    }
    246243
    247244private:
    248     Optional<Token> consumeTokenFromStream();
     245    Token consumeTokenFromStream();
    249246
    250247    void skipWhitespaceAndComments();
    251     void skipWhitespace();
    252     void skipLineComment();
    253     void skipLongComment();
     248
     249    unsigned lineNumberFromOffset(unsigned offset);
    254250
    255251    Optional<Token::Type> recognizeKeyword(unsigned end);
    … …  
    266262    unsigned digitStar(unsigned) const;
    267263    Optional<unsigned> character(char, unsigned) const;
    268     template<unsigned length> Optional<unsigned> anyCharacter(const char (&string)[length], unsigned) const;
    269264    Optional<unsigned> coreFloatLiteralType1(unsigned) const;
    270265    Optional<unsigned> coreFloatLiteral(unsigned) const;
    … …  
    273268    Optional<unsigned> validIdentifier(unsigned) const;
    274269    Optional<unsigned> identifier(unsigned) const;
    275     Optional<unsigned> operatorName(unsigned) const;
     270    Optional<unsigned> completeOperatorName(unsigned) const;
    276271
    277272    StringView m_stringView;
    278     Optional<Token> m_ringBuffer[2];
     273    Token m_ringBuffer[2];
    279274    unsigned m_ringBufferIndex { 0 };
    280275    unsigned m_offset { 0 };
    281     unsigned m_lineNumber { 0 };
    282276};
    283277
    284278template<unsigned length> Optional<unsigned> Lexer::string(const char (&string)[length], unsigned offset) const
    285279{
     280    if (offset + length > m_stringView.length())
     281        return WTF::nullopt;
    286282    for (unsigned i = 0; i < length - 1; ++i) {
    287         if (offset + i >= m_stringView.length() || m_stringView[offset + i] != string[i])
     283        if (m_stringView[offset + i] != string[i])
    288284            return WTF::nullopt;
    289285    }
    … …  
    291287}
    292288
    293 template<unsigned length> Optional<unsigned> Lexer::anyCharacter(const char (&string)[length], unsigned offset) const
    294 {
    295     if (offset >= m_stringView.length())
    296         return WTF::nullopt;
    297     for (unsigned i = 0; i < length - 1; ++i) {
    298         if (m_stringView[offset] == string[i])
    299             return offset + 1;
    300     }
    301     return WTF::nullopt;
    302 }
    303 
    304289} // namespace WHLSL
    305290

  • trunk/Source/WebCore/Modules/webgpu/WHLSL/WHLSLNameResolver.h

    r247170 r247171  
    4545    virtual ~NameResolver();
    4646
    47     void visit(AST::FunctionDefinition&) override;
    48 
    4947    void setCurrentFunctionDefinition(AST::FunctionDefinition* functionDefinition)
    5048    {
    … …  
    5351
    5452private:
     53    void visit(AST::FunctionDefinition&) override;
    5554    void visit(AST::NativeFunctionDeclaration&) override;
    5655    void visit(AST::TypeReference&) override;

  • trunk/Source/WebCore/Modules/webgpu/WHLSL/WHLSLParser.cpp

    r247110 r247171  
    6767    while (!m_lexer.isFullyConsumed()) {
    6868        auto token = m_lexer.peek();
    69         if (!token)
    70             break;
    71         switch (token->type) {
     69        switch (token.type) {
     70        case Lexer::Token::Type::Invalid:
     71            return WTF::nullopt;
    7272        case Lexer::Token::Type::Semicolon:
    7373            m_lexer.consumeToken();
    … …  
    136136auto Parser::peek() -> Expected<Lexer::Token, Error>
    137137{
    138     if (auto token = m_lexer.peek()) {
    139         return *token;
    140     }
     138    auto token = m_lexer.peek();
     139    if (token.type != Lexer::Token::Type::Invalid && token.type != Lexer::Token::Type::EndOfFile)
     140        return { token };
    141141    return fail("Cannot consume token"_str, TryToPeek::No);
    142142}
    … …  
    144144auto Parser::peekFurther() -> Expected<Lexer::Token, Error>
    145145{
    146     if (auto token = m_lexer.peekFurther()) {
    147         return *token;
    148     }
     146    auto token = m_lexer.peekFurther();
     147    if (token.type != Lexer::Token::Type::Invalid && token.type != Lexer::Token::Type::EndOfFile)
     148        return { token };
    149149    return fail("Cannot consume two tokens"_str, TryToPeek::No);
    150150}
    151151
    152 bool Parser::peekTypes(const Vector<Lexer::Token::Type>& types)
    153 {
    154     if (auto token = m_lexer.peek())
    155         return std::find(types.begin(), types.end(), token->type) != types.end();
    156     return false;
     152template <Lexer::Token::Type t, Lexer::Token::Type... ts>
     153struct Types {
     154    static bool includes(Lexer::Token::Type type)
     155    {
     156        return t == type || Types<ts...>::includes(type);
     157    }
     158
     159    static void appendNameTo(StringBuilder& builder)
     160    {
     161        builder.append(Lexer::Token::typeName(t));
     162        builder.append(", ");
     163        Types<ts...>::appendNameTo(builder);
     164    }
     165};
     166template <Lexer::Token::Type t>
     167struct Types<t> {
     168    static bool includes(Lexer::Token::Type type)
     169    {
     170        return t == type;
     171    }
     172
     173    static void appendNameTo(StringBuilder& builder)
     174    {
     175        builder.append(Lexer::Token::typeName(t));
     176    }
     177};
     178
     179template <Lexer::Token::Type... types>
     180bool Parser::peekTypes()
     181{
     182    auto token = m_lexer.peek();
     183    return Types<types...>::includes(token.type);
    157184}
    158185
    159186Optional<Lexer::Token> Parser::tryType(Lexer::Token::Type type)
    160187{
    161     if (auto token = m_lexer.peek()) {
    162         if (token->type == type)
    163             return m_lexer.consumeToken();
    164     }
     188    auto token = m_lexer.peek();
     189    if (token.type == type)
     190        return { m_lexer.consumeToken() };
    165191    return WTF::nullopt;
    166192}
    167193
    168 Optional<Lexer::Token> Parser::tryTypes(const Vector<Lexer::Token::Type>& types)
    169 {
    170     if (auto token = m_lexer.peek()) {
    171         if (std::find(types.begin(), types.end(), token->type) != types.end())
    172             return m_lexer.consumeToken();
    173     }
     194template <Lexer::Token::Type... types>
     195Optional<Lexer::Token> Parser::tryTypes()
     196{
     197    auto token = m_lexer.peek();
     198    if (Types<types...>::includes(token.type))
     199        return { m_lexer.consumeToken() };
    174200    return WTF::nullopt;
    175201}
    … …  
    177203auto Parser::consumeType(Lexer::Token::Type type) -> Expected<Lexer::Token, Error>
    178204{
    179     if (auto token = m_lexer.consumeToken()) {
    180         if (token->type == type)
    181             return *token;
    182         return fail(makeString("Unexpected token (expected ", Lexer::Token::typeName(type), " got ", Lexer::Token::typeName(token->type), ")"));
    183     }
    184     return fail(makeString("Cannot consume token (expected ", Lexer::Token::typeName(type), ")"));
    185 }
    186 
    187 auto Parser::consumeTypes(const Vector<Lexer::Token::Type>& types) -> Expected<Lexer::Token, Error>
     205    auto token = m_lexer.consumeToken();
     206    if (token.type == type)
     207        return { token };
     208    return fail(makeString("Unexpected token (expected ", Lexer::Token::typeName(type), " got ", Lexer::Token::typeName(token.type), ")"));
     209}
     210
     211template <Lexer::Token::Type... types>
     212auto Parser::consumeTypes() -> Expected<Lexer::Token, Error>
    188213{
    189214    auto buildExpectedString = [&]() -> String {
    190215        StringBuilder builder;
    191216        builder.append("[");
    192         for (unsigned i = 0; i < types.size(); ++i) {
    193             if (i > 0)
    194                 builder.append(", ");
    195             builder.append(Lexer::Token::typeName(types[i]));
    196         }
     217        Types<types...>::appendNameTo(builder);
    197218        builder.append("]");
    198219        return builder.toString();
    199220    };
    200221
    201     if (auto token = m_lexer.consumeToken()) {
    202         if (std::find(types.begin(), types.end(), token->type) != types.end())
    203             return *token;
    204         return fail(makeString("Unexpected token (expected one of ", buildExpectedString(), " got ", Lexer::Token::typeName(token->type), ")"));
    205     }
    206     return fail(makeString("Cannot consume token (expected ", buildExpectedString(), ")"));
     222    auto token = m_lexer.consumeToken();
     223    if (Types<types...>::includes(token.type))
     224        return { token };
     225    return fail(makeString("Unexpected token (expected one of ", buildExpectedString(), " got ", Lexer::Token::typeName(token.type), ")"));
    207226}
    208227
    … …  
    280299auto Parser::consumeIntegralLiteral() -> Expected<Variant<int, unsigned>, Error>
    281300{
    282     auto integralLiteralToken = consumeTypes({ Lexer::Token::Type::IntLiteral, Lexer::Token::Type::UintLiteral });
     301    auto integralLiteralToken = consumeTypes<Lexer::Token::Type::IntLiteral, Lexer::Token::Type::UintLiteral>();
    283302    if (!integralLiteralToken)
    284303        return Unexpected<Error>(integralLiteralToken.error());
    … …  
    286305    switch (integralLiteralToken->type) {
    287306    case Lexer::Token::Type::IntLiteral: {
    288         auto result = intLiteralToInt(integralLiteralToken->stringView);
     307        auto result = intLiteralToInt(integralLiteralToken->stringView(m_lexer));
    289308        if (result)
    290309            return {{ *result }};
    … …  
    293312    default: {
    294313        ASSERT(integralLiteralToken->type == Lexer::Token::Type::UintLiteral);
    295         auto result = uintLiteralToUint(integralLiteralToken->stringView);
     314        auto result = uintLiteralToUint(integralLiteralToken->stringView(m_lexer));
    296315        if (result)
    297316            return {{ *result }};
    … …  
    336355auto Parser::parseConstantExpression() -> Expected<AST::ConstantExpression, Error>
    337356{
    338     auto type = consumeTypes({
     357    auto type = consumeTypes<
    339358        Lexer::Token::Type::IntLiteral,
    340359        Lexer::Token::Type::UintLiteral,
    … …  
    343362        Lexer::Token::Type::True,
    344363        Lexer::Token::Type::False,
    345         Lexer::Token::Type::Identifier,
    346     });
     364        Lexer::Token::Type::Identifier>();
    347365    if (!type)
    348366        return Unexpected<Error>(type.error());
    … …  
    350368    switch (type->type) {
    351369    case Lexer::Token::Type::IntLiteral: {
    352         auto value = intLiteralToInt(type->stringView);
     370        auto value = intLiteralToInt(type->stringView(m_lexer));
    353371        if (!value)
    354372            return Unexpected<Error>(value.error());
    … …  
    356374    }
    357375    case Lexer::Token::Type::UintLiteral: {
    358         auto value = uintLiteralToUint(type->stringView);
     376        auto value = uintLiteralToUint(type->stringView(m_lexer));
    359377        if (!value)
    360378            return Unexpected<Error>(value.error());
    … …  
    362380    }
    363381    case Lexer::Token::Type::FloatLiteral: {
    364         auto value = floatLiteralToFloat(type->stringView);
     382        auto value = floatLiteralToFloat(type->stringView(m_lexer));
    365383        if (!value)
    366384            return Unexpected<Error>(value.error());
    … …  
    377395        CONSUME_TYPE(origin, FullStop);
    378396        CONSUME_TYPE(next, Identifier);
    379         return { AST::EnumerationMemberLiteral(WTFMove(*origin), type->stringView.toString(), next->stringView.toString()) };
     397        return { AST::EnumerationMemberLiteral(WTFMove(*origin), type->stringView(m_lexer).toString(), next->stringView(m_lexer).toString()) };
    380398    }
    381399    }
    … …  
    391409    }
    392410    CONSUME_TYPE(result, Identifier);
    393     return AST::TypeArgument(makeUniqueRef<AST::TypeReference>(Lexer::Token(*result), result->stringView.toString(), AST::TypeArguments()));
     411    return AST::TypeArgument(makeUniqueRef<AST::TypeReference>(Lexer::Token(*result), result->stringView(m_lexer).toString(), AST::TypeArguments()));
    394412}
    395413
    … …  
    423441auto Parser::parseTypeSuffixAbbreviated() -> Expected<TypeSuffixAbbreviated, Error>
    424442{
    425     auto token = consumeTypes({ Lexer::Token::Type::Star, Lexer::Token::Type::SquareBracketPair, Lexer::Token::Type::LeftSquareBracket });
     443    auto token = consumeTypes<
     444        Lexer::Token::Type::Star,
     445        Lexer::Token::Type::SquareBracketPair,
     446        Lexer::Token::Type::LeftSquareBracket>();
    426447    if (!token)
    427448        return Unexpected<Error>(token.error());
    … …  
    438459auto Parser::parseTypeSuffixNonAbbreviated() -> Expected<TypeSuffixNonAbbreviated, Error>
    439460{
    440     auto token = consumeTypes({ Lexer::Token::Type::Star, Lexer::Token::Type::SquareBracketPair, Lexer::Token::Type::LeftSquareBracket });
     461    auto token = consumeTypes<
     462        Lexer::Token::Type::Star,
     463        Lexer::Token::Type::SquareBracketPair,
     464        Lexer::Token::Type::LeftSquareBracket>();
    441465    if (!token)
    442466        return Unexpected<Error>(token.error());
    … …  
    448472        return {{ *token, WTF::nullopt, *numElements }};
    449473    }
    450     auto addressSpaceToken = consumeTypes({ Lexer::Token::Type::Constant, Lexer::Token::Type::Device, Lexer::Token::Type::Threadgroup, Lexer::Token::Type::Thread});
     474    auto addressSpaceToken = consumeTypes<
     475        Lexer::Token::Type::Constant,
     476        Lexer::Token::Type::Device,
     477        Lexer::Token::Type::Threadgroup,
     478        Lexer::Token::Type::Thread>();
    451479    if (!addressSpaceToken)
    452480        return Unexpected<Error>(addressSpaceToken.error());
    … …  
    472500auto Parser::parseType() -> Expected<UniqueRef<AST::UnnamedType>, Error>
    473501{
    474     auto addressSpaceToken = tryTypes({ Lexer::Token::Type::Constant, Lexer::Token::Type::Device, Lexer::Token::Type::Threadgroup, Lexer::Token::Type::Thread});
     502    auto addressSpaceToken = tryTypes<
     503        Lexer::Token::Type::Constant,
     504        Lexer::Token::Type::Device,
     505        Lexer::Token::Type::Threadgroup,
     506        Lexer::Token::Type::Thread>();
    475507
    476508    CONSUME_TYPE(name, Identifier);
    … …  
    506538        };
    507539        PARSE(firstTypeSuffixAbbreviated, TypeSuffixAbbreviated);
    508         UniqueRef<AST::UnnamedType> result = makeUniqueRef<AST::TypeReference>(WTFMove(*addressSpaceToken), name->stringView.toString(), WTFMove(*typeArguments));
     540        UniqueRef<AST::UnnamedType> result = makeUniqueRef<AST::TypeReference>(WTFMove(*addressSpaceToken), name->stringView(m_lexer).toString(), WTFMove(*typeArguments));
    509541        auto next = constructTypeFromSuffixAbbreviated(*firstTypeSuffixAbbreviated, WTFMove(result));
    510542        result = WTFMove(next);
    … …  
    535567        }
    536568    };
    537     UniqueRef<AST::UnnamedType> result = makeUniqueRef<AST::TypeReference>(WTFMove(*name), name->stringView.toString(), WTFMove(*typeArguments));
     569    UniqueRef<AST::UnnamedType> result = makeUniqueRef<AST::TypeReference>(WTFMove(*name), name->stringView(m_lexer).toString(), WTFMove(*typeArguments));
    538570    while (true) {
    539571        PEEK(nextToken);
    … …  
    558590    PARSE(type, Type);
    559591    CONSUME_TYPE(semicolon, Semicolon);
    560     return AST::TypeDefinition(WTFMove(*origin), name->stringView.toString(), WTFMove(*type));
     592    return AST::TypeDefinition(WTFMove(*origin), name->stringView(m_lexer).toString(), WTFMove(*type));
    561593}
    562594
    563595auto Parser::parseBuiltInSemantic() -> Expected<AST::BuiltInSemantic, Error>
    564596{
    565     auto origin = consumeTypes({
     597    auto origin = consumeTypes<
    566598        Lexer::Token::Type::SVInstanceID,
    567599        Lexer::Token::Type::SVVertexID,
    … …  
    577609        Lexer::Token::Type::SVGroupID,
    578610        Lexer::Token::Type::SVGroupIndex,
    579         Lexer::Token::Type::SVGroupThreadID});
     611        Lexer::Token::Type::SVGroupThreadID>();
    580612    if (!origin)
    581613        return Unexpected<Error>(origin.error());
    … …  
    624656
    625657    CONSUME_TYPE(info, Identifier);
    626     if (info->stringView.length() < 2 || (info->stringView[0] != 'u'
    627         && info->stringView[0] != 't'
    628         && info->stringView[0] != 'b'
    629         && info->stringView[0] != 's'))
    630         return Unexpected<Error>(Error(makeString(info->stringView.substring(0, 1), " is not a known resource type ('u', 't', 'b', or 's')")));
     658    auto infoStringView = info->stringView(m_lexer);
     659    if (infoStringView.length() < 2 || (infoStringView[0] != 'u'
     660        && infoStringView[0] != 't'
     661        && infoStringView[0] != 'b'
     662        && infoStringView[0] != 's'))
     663        return Unexpected<Error>(Error(makeString(infoStringView.substring(0, 1), " is not a known resource type ('u', 't', 'b', or 's')")));
    631664
    632665    AST::ResourceSemantic::Mode mode;
    633     switch (info->stringView[0]) {
     666    switch (infoStringView[0]) {
    634667    case 'u':
    635668        mode = AST::ResourceSemantic::Mode::UnorderedAccessView;
    … …  
    646679    }
    647680
    648     auto index = recognizeSimpleUnsignedInteger(info->stringView.substring(1));
     681    auto index = recognizeSimpleUnsignedInteger(infoStringView.substring(1));
    649682    if (!index)
    650683        return Unexpected<Error>(index.error());
    … …  
    653686    if (tryType(Lexer::Token::Type::Comma)) {
    654687        CONSUME_TYPE(spaceToken, Identifier);
     688        auto spaceTokenStringView = spaceToken->stringView(m_lexer);
    655689        auto prefix = "space"_str;
    656         if (!spaceToken->stringView.startsWith(StringView(prefix)))
    657             return Unexpected<Error>(Error(makeString("Second argument to resource semantic ", spaceToken->stringView, " needs be of the form 'space0'")));
    658         if (spaceToken->stringView.length() <= prefix.length())
    659             return Unexpected<Error>(Error(makeString("Second argument to resource semantic ", spaceToken->stringView, " needs be of the form 'space0'")));
    660         auto spaceValue = recognizeSimpleUnsignedInteger(spaceToken->stringView.substring(prefix.length()));
     690        if (!spaceTokenStringView.startsWith(StringView(prefix)))
     691            return Unexpected<Error>(Error(makeString("Second argument to resource semantic ", spaceTokenStringView, " needs be of the form 'space0'")));
     692        if (spaceTokenStringView.length() <= prefix.length())
     693            return Unexpected<Error>(Error(makeString("Second argument to resource semantic ", spaceTokenStringView, " needs be of the form 'space0'")));
     694        auto spaceValue = recognizeSimpleUnsignedInteger(spaceTokenStringView.substring(prefix.length()));
    661695        if (!spaceValue)
    662696            return Unexpected<Error>(spaceValue.error());
    … …  
    718752    AST::Qualifiers qualifiers;
    719753    while (auto next = tryType(Lexer::Token::Type::Qualifier)) {
    720         if ("nointerpolation" == next->stringView)
     754        auto nextStringView = next->stringView(m_lexer);
     755        if ("nointerpolation" == nextStringView)
    721756            qualifiers.append(AST::Qualifier::Nointerpolation);
    722         else if ("noperspective" == next->stringView)
     757        else if ("noperspective" == nextStringView)
    723758            qualifiers.append(AST::Qualifier::Noperspective);
    724         else if ("uniform" == next->stringView)
     759        else if ("uniform" == nextStringView)
    725760            qualifiers.append(AST::Qualifier::Uniform);
    726         else if ("centroid" == next->stringView)
     761        else if ("centroid" == nextStringView)
    727762            qualifiers.append(AST::Qualifier::Centroid);
    728763        else {
    729             ASSERT("sample" == next->stringView);
     764            ASSERT("sample" == nextStringView);
    730765            qualifiers.append(AST::Qualifier::Sample);
    731766        }
    … …  
    745780    CONSUME_TYPE(semicolon, Semicolon);
    746781
    747     return AST::StructureElement(WTFMove(*origin), WTFMove(qualifiers), WTFMove(*type), name->stringView.toString(), WTFMove(*semantic));
     782    return AST::StructureElement(WTFMove(*origin), WTFMove(qualifiers), WTFMove(*type), name->stringView(m_lexer).toString(), WTFMove(*semantic));
    748783}
    749784
    … …  
    760795    }
    761796
    762     return AST::StructureDefinition(WTFMove(*origin), name->stringView.toString(), WTFMove(structureElements));
     797    return AST::StructureDefinition(WTFMove(*origin), name->stringView(m_lexer).toString(), WTFMove(structureElements));
    763798}
    764799
    … …  
    782817    PARSE(firstEnumerationMember, EnumerationMember);
    783818
    784     AST::EnumerationDefinition result(WTFMove(*origin), name->stringView.toString(), WTFMove(*type));
     819    AST::EnumerationDefinition result(WTFMove(*origin), name->stringView(m_lexer).toString(), WTFMove(*type));
    785820    auto success = result.add(WTFMove(*firstEnumerationMember));
    786821    if (!success)
    … …  
    802837{
    803838    CONSUME_TYPE(identifier, Identifier);
    804     auto name = identifier->stringView.toString();
     839    auto name = identifier->stringView(m_lexer).toString();
    805840
    806841    if (tryType(Lexer::Token::Type::EqualsSign)) {
    … …  
    819854    CONSUME_TYPE(semicolon, Semicolon);
    820855
    821     return AST::NativeTypeDeclaration(WTFMove(*origin), name->stringView.toString(), WTFMove(*typeArguments));
     856    return AST::NativeTypeDeclaration(WTFMove(*origin), name->stringView(m_lexer).toString(), WTFMove(*typeArguments));
    822857}
    823858
    … …  
    874909    String name;
    875910    if (auto token = tryType(Lexer::Token::Type::Identifier))
    876         name = token->stringView.toString();
     911        name = token->stringView(m_lexer).toString();
    877912
    878913    PARSE(semantic, Semantic);
    … …  
    921956    PARSE(semantic, Semantic);
    922957    bool isOperator = false;
    923     return AST::FunctionDeclaration(WTFMove(*origin), WTFMove(*attributeBlock), AST::EntryPointType::Compute, WTFMove(*type), name->stringView.toString(), WTFMove(*parameters), WTFMove(*semantic), isOperator);
     958    return AST::FunctionDeclaration(WTFMove(*origin), WTFMove(*attributeBlock), AST::EntryPointType::Compute, WTFMove(*type), name->stringView(m_lexer).toString(), WTFMove(*parameters), WTFMove(*semantic), isOperator);
    924959}
    925960
    926961auto Parser::parseVertexOrFragmentFunctionDeclaration() -> Expected<AST::FunctionDeclaration, Error>
    927962{
    928     auto entryPoint = consumeTypes({ Lexer::Token::Type::Vertex, Lexer::Token::Type::Fragment });
     963    auto entryPoint = consumeTypes<Lexer::Token::Type::Vertex, Lexer::Token::Type::Fragment>();
    929964    if (!entryPoint)
    930965        return Unexpected<Error>(entryPoint.error());
    … …  
    937972
    938973    bool isOperator = false;
    939     return AST::FunctionDeclaration(WTFMove(*entryPoint), { }, entryPointType, WTFMove(*type), name->stringView.toString(), WTFMove(*parameters), WTFMove(*semantic), isOperator);
     974    return AST::FunctionDeclaration(WTFMove(*entryPoint), { }, entryPointType, WTFMove(*type), name->stringView(m_lexer).toString(), WTFMove(*parameters), WTFMove(*semantic), isOperator);
    940975}
    941976
    … …  
    946981    PARSE(type, Type);
    947982
    948     auto name = consumeTypes({ Lexer::Token::Type::Identifier, Lexer::Token::Type::OperatorName });
     983    auto name = consumeTypes<Lexer::Token::Type::Identifier, Lexer::Token::Type::OperatorName>();
    949984    if (!name)
    950985        return Unexpected<Error>(name.error());
    … …  
    954989    PARSE(semantic, Semantic);
    955990
    956     return AST::FunctionDeclaration(WTFMove(*origin), { }, WTF::nullopt, WTFMove(*type), name->stringView.toString(), WTFMove(*parameters), WTFMove(*semantic), isOperator);
     991    return AST::FunctionDeclaration(WTFMove(*origin), { }, WTF::nullopt, WTFMove(*type), name->stringView(m_lexer).toString(), WTFMove(*parameters), WTFMove(*semantic), isOperator);
    957992}
    958993
    … …  
    10041039{
    10051040    AST::Statements statements;
    1006     while (!peekTypes({Lexer::Token::Type::RightCurlyBracket, Lexer::Token::Type::Case, Lexer::Token::Type::Default})) {
     1041    while (!peekTypes<Lexer::Token::Type::RightCurlyBracket, Lexer::Token::Type::Case, Lexer::Token::Type::Default>()) {
    10071042        PARSE(statement, Statement);
    10081043        statements.append(WTFMove(*statement));
    … …  
    10551090auto Parser::parseSwitchCase() -> Expected<AST::SwitchCase, Error>
    10561091{
    1057     auto origin = consumeTypes({ Lexer::Token::Type::Case, Lexer::Token::Type::Default });
     1092    auto origin = consumeTypes<Lexer::Token::Type::Case, Lexer::Token::Type::Default>();
    10581093    if (!origin)
    10591094        return Unexpected<Error>(origin.error());
    … …  
    11621197    }
    11631198
    1164     return AST::VariableDeclaration(WTFMove(*origin), WTFMove(qualifiers), { WTFMove(type) }, name->stringView.toString(), WTFMove(*semantic), WTFMove(initializer));
     1199    return AST::VariableDeclaration(WTFMove(*origin), WTFMove(qualifiers), { WTFMove(type) }, name->stringView(m_lexer).toString(), WTFMove(*semantic), WTFMove(initializer));
    11651200}
    11661201
    … …  
    12191254        auto breakToken = m_lexer.consumeToken();
    12201255        CONSUME_TYPE(semicolon, Semicolon);
    1221         auto breakObject = AST::Break(WTFMove(*breakToken));
     1256        auto breakObject = AST::Break(WTFMove(breakToken));
    12221257        return { makeUniqueRef<AST::Break>(WTFMove(breakObject)) };
    12231258    }
    … …  
    12251260        auto continueToken = m_lexer.consumeToken();
    12261261        CONSUME_TYPE(semicolon, Semicolon);
    1227         auto continueObject = AST::Continue(WTFMove(*continueToken));
     1262        auto continueObject = AST::Continue(WTFMove(continueToken));
    12281263        return { makeUniqueRef<AST::Continue>(WTFMove(continueObject)) };
    12291264    }
    … …  
    12311266        auto fallthroughToken = m_lexer.consumeToken();
    12321267        CONSUME_TYPE(semicolon, Semicolon);
    1233         auto fallthroughObject = AST::Fallthrough(WTFMove(*fallthroughToken));
     1268        auto fallthroughObject = AST::Fallthrough(WTFMove(fallthroughToken));
    12341269        return { makeUniqueRef<AST::Fallthrough>(WTFMove(fallthroughObject)) };
    12351270    }
    … …  
    12371272        auto trapToken = m_lexer.consumeToken();
    12381273        CONSUME_TYPE(semicolon, Semicolon);
    1239         auto trapObject = AST::Trap(WTFMove(*trapToken));
     1274        auto trapObject = AST::Trap(WTFMove(trapToken));
    12401275        return { makeUniqueRef<AST::Trap>(WTFMove(trapObject)) };
    12411276    }
    … …  
    12431278        auto returnToken = m_lexer.consumeToken();
    12441279        if (auto semicolon = tryType(Lexer::Token::Type::Semicolon)) {
    1245             auto returnObject = AST::Return(WTFMove(*returnToken), WTF::nullopt);
     1280            auto returnObject = AST::Return(WTFMove(returnToken), WTF::nullopt);
    12461281            return { makeUniqueRef<AST::Return>(WTFMove(returnObject)) };
    12471282        }
    12481283        PARSE(expression, Expression);
    12491284        CONSUME_TYPE(finalSemicolon, Semicolon);
    1250         auto returnObject = AST::Return(WTFMove(*returnToken), { WTFMove(*expression) });
     1285        auto returnObject = AST::Return(WTFMove(returnToken), { WTFMove(*expression) });
    12511286        return { makeUniqueRef<AST::Return>(WTFMove(returnObject)) };
    12521287    }
    … …  
    13221357    PARSE(expression, PossiblePrefix, &isEffectful);
    13231358
    1324     if (!isEffectful || peekTypes({
     1359    if (!isEffectful || peekTypes<
    13251360        Lexer::Token::Type::EqualsSign,
    13261361        Lexer::Token::Type::PlusEquals,
    … …  
    13341369        Lexer::Token::Type::RightShiftEquals,
    13351370        Lexer::Token::Type::LeftShiftEquals
    1336     })) {
     1371    >()) {
    13371372        return completeAssignment(WTFMove(*origin), WTFMove(*expression));
    13381373    }
    … …  
    13431378auto Parser::parseLimitedSuffixOperator(UniqueRef<AST::Expression>&& previous) -> SuffixExpression
    13441379{
    1345     auto type = consumeTypes({ Lexer::Token::Type::FullStop, Lexer::Token::Type::Arrow, Lexer::Token::Type::LeftSquareBracket });
     1380    auto type = consumeTypes<
     1381        Lexer::Token::Type::FullStop,
     1382        Lexer::Token::Type::Arrow,
     1383        Lexer::Token::Type::LeftSquareBracket>();
    13461384    if (!type)
    13471385        return SuffixExpression(WTFMove(previous), false);
    … …  
    13521390        if (!identifier)
    13531391            return SuffixExpression(WTFMove(previous), false);
    1354         return SuffixExpression(makeUniqueRef<AST::DotExpression>(WTFMove(*type), WTFMove(previous), identifier->stringView.toString()), true);
     1392        return SuffixExpression(makeUniqueRef<AST::DotExpression>(WTFMove(*type), WTFMove(previous), identifier->stringView(m_lexer).toString()), true);
    13551393    }
    13561394    case Lexer::Token::Type::Arrow: {
    … …  
    13581396        if (!identifier)
    13591397            return SuffixExpression(WTFMove(previous), false);
    1360         return SuffixExpression(makeUniqueRef<AST::DotExpression>(Lexer::Token(*type), makeUniqueRef<AST::DereferenceExpression>(WTFMove(*type), WTFMove(previous)), identifier->stringView.toString()), true);
     1398        return SuffixExpression(makeUniqueRef<AST::DotExpression>(Lexer::Token(*type), makeUniqueRef<AST::DereferenceExpression>(WTFMove(*type), WTFMove(previous)), identifier->stringView(m_lexer).toString()), true);
    13611399    }
    13621400    default: {
    … …  
    13741412auto Parser::parseSuffixOperator(UniqueRef<AST::Expression>&& previous) -> SuffixExpression
    13751413{
    1376     auto suffix = consumeTypes({ Lexer::Token::Type::FullStop, Lexer::Token::Type::Arrow, Lexer::Token::Type::LeftSquareBracket, Lexer::Token::Type::PlusPlus, Lexer::Token::Type::MinusMinus });
     1414    auto suffix = consumeTypes<
     1415        Lexer::Token::Type::FullStop,
     1416        Lexer::Token::Type::Arrow,
     1417        Lexer::Token::Type::LeftSquareBracket,
     1418        Lexer::Token::Type::PlusPlus,
     1419        Lexer::Token::Type::MinusMinus>();
    13771420    if (!suffix)
    13781421        return SuffixExpression(WTFMove(previous), false);
    … …  
    13831426        if (!identifier)
    13841427            return SuffixExpression(WTFMove(previous), false);
    1385         return SuffixExpression(makeUniqueRef<AST::DotExpression>(WTFMove(*suffix), WTFMove(previous), identifier->stringView.toString()), true);
     1428        return SuffixExpression(makeUniqueRef<AST::DotExpression>(WTFMove(*suffix), WTFMove(previous), identifier->stringView(m_lexer).toString()), true);
    13861429    }
    13871430    case Lexer::Token::Type::Arrow: {
    … …  
    13891432        if (!identifier)
    13901433            return SuffixExpression(WTFMove(previous), false);
    1391         return SuffixExpression(makeUniqueRef<AST::DotExpression>(Lexer::Token(*suffix), makeUniqueRef<AST::DereferenceExpression>(WTFMove(*suffix), WTFMove(previous)), identifier->stringView.toString()), true);
     1434        return SuffixExpression(makeUniqueRef<AST::DotExpression>(Lexer::Token(*suffix), makeUniqueRef<AST::DereferenceExpression>(WTFMove(*suffix), WTFMove(previous)), identifier->stringView(m_lexer).toString()), true);
    13921435    }
    13931436    case Lexer::Token::Type::LeftSquareBracket: {
    … …  
    14521495auto Parser::completeAssignment(Lexer::Token&& origin, UniqueRef<AST::Expression>&& left) -> Expected<UniqueRef<AST::Expression>, Error>
    14531496{
    1454     auto assignmentOperator = consumeTypes({
     1497    auto assignmentOperator = consumeTypes<
    14551498        Lexer::Token::Type::EqualsSign,
    14561499        Lexer::Token::Type::PlusEquals,
    … …  
    14631506        Lexer::Token::Type::OrEquals,
    14641507        Lexer::Token::Type::RightShiftEquals,
    1465         Lexer::Token::Type::LeftShiftEquals
    1466     });
     1508        Lexer::Token::Type::LeftShiftEquals>();
    14671509    if (!assignmentOperator)
    14681510        return Unexpected<Error>(assignmentOperator.error());
    … …  
    15231565    PARSE(expression, PossiblePrefix);
    15241566
    1525     if (peekTypes({Lexer::Token::Type::EqualsSign,
     1567    if (peekTypes<Lexer::Token::Type::EqualsSign,
    15261568        Lexer::Token::Type::PlusEquals,
    15271569        Lexer::Token::Type::MinusEquals,
    … …  
    15331575        Lexer::Token::Type::OrEquals,
    15341576        Lexer::Token::Type::RightShiftEquals,
    1535         Lexer::Token::Type::LeftShiftEquals})) {
     1577        Lexer::Token::Type::LeftShiftEquals>()) {
    15361578        return completeAssignment(WTFMove(*origin), WTFMove(*expression));
    15371579    }
    … …  
    15571599auto Parser::completePossibleLogicalBinaryOperation(UniqueRef<AST::Expression>&& previous) -> Expected<UniqueRef<AST::Expression>, Error>
    15581600{
    1559     while (auto logicalBinaryOperation = tryTypes({
     1601    while (auto logicalBinaryOperation = tryTypes<
    15601602        Lexer::Token::Type::OrOr,
    15611603        Lexer::Token::Type::AndAnd,
    … …  
    15631605        Lexer::Token::Type::Xor,
    15641606        Lexer::Token::Type::And
    1565         })) {
     1607        >()) {
    15661608        PARSE(next, PossibleRelationalBinaryOperation);
    15671609
    … …  
    16091651auto Parser::completePossibleRelationalBinaryOperation(UniqueRef<AST::Expression>&& previous) -> Expected<UniqueRef<AST::Expression>, Error>
    16101652{
    1611     while (auto relationalBinaryOperation = tryTypes({
     1653    while (auto relationalBinaryOperation = tryTypes<
    16121654        Lexer::Token::Type::LessThanSign,
    16131655        Lexer::Token::Type::GreaterThanSign,
    … …  
    16161658        Lexer::Token::Type::EqualComparison,
    16171659        Lexer::Token::Type::NotEqual
    1618         })) {
     1660        >()) {
    16191661        PARSE(next, PossibleShift);
    16201662
    … …  
    16641706auto Parser::completePossibleShift(UniqueRef<AST::Expression>&& previous) -> Expected<UniqueRef<AST::Expression>, Error>
    16651707{
    1666     while (auto shift = tryTypes({
     1708    while (auto shift = tryTypes<
    16671709        Lexer::Token::Type::LeftShift,
    16681710        Lexer::Token::Type::RightShift
    1669         })) {
     1711        >()) {
    16701712        PARSE(next, PossibleAdd);
    16711713
    … …  
    16981740auto Parser::completePossibleAdd(UniqueRef<AST::Expression>&& previous) -> Expected<UniqueRef<AST::Expression>, Error>
    16991741{
    1700     while (auto add = tryTypes({
     1742    while (auto add = tryTypes<
    17011743        Lexer::Token::Type::Plus,
    17021744        Lexer::Token::Type::Minus
    1703         })) {
     1745        >()) {
    17041746        PARSE(next, PossibleMultiply);
    17051747
    … …  
    17321774auto Parser::completePossibleMultiply(UniqueRef<AST::Expression>&& previous) -> Expected<UniqueRef<AST::Expression>, Error>
    17331775{
    1734     while (auto multiply = tryTypes({
     1776    while (auto multiply = tryTypes<
    17351777        Lexer::Token::Type::Star,
    17361778        Lexer::Token::Type::Divide,
    17371779        Lexer::Token::Type::Mod
    1738         })) {
     1780        >()) {
    17391781        PARSE(next, PossiblePrefix);
    17401782
    … …  
    17651807auto Parser::parsePossiblePrefix(bool *isEffectful) -> Expected<UniqueRef<AST::Expression>, Error>
    17661808{
    1767     if (auto prefix = tryTypes({
     1809    if (auto prefix = tryTypes<
    17681810        Lexer::Token::Type::PlusPlus,
    17691811        Lexer::Token::Type::MinusMinus,
    … …  
    17751817        Lexer::Token::Type::At,
    17761818        Lexer::Token::Type::Star
    1777     })) {
     1819    >()) {
    17781820        PARSE(next, PossiblePrefix);
    17791821
    … …  
    18781920{
    18791921    CONSUME_TYPE(name, Identifier);
    1880     auto callName = name->stringView.toString();
     1922    auto callName = name->stringView(m_lexer).toString();
    18811923
    18821924    CONSUME_TYPE(leftParenthesis, LeftParenthesis);
    … …  
    19001942auto Parser::parseTerm() -> Expected<UniqueRef<AST::Expression>, Error>
    19011943{
    1902     auto type = consumeTypes({
     1944    auto type = consumeTypes<
    19031945        Lexer::Token::Type::IntLiteral,
    19041946        Lexer::Token::Type::UintLiteral,
    … …  
    19081950        Lexer::Token::Type::False,
    19091951        Lexer::Token::Type::Identifier,
    1910         Lexer::Token::Type::LeftParenthesis
    1911     });
     1952        Lexer::Token::Type::LeftParenthesis>();
    19121953    if (!type)
    19131954        return Unexpected<Error>(type.error());
    … …  
    19151956    switch (type->type) {
    19161957    case Lexer::Token::Type::IntLiteral: {
    1917         auto value = intLiteralToInt(type->stringView);
     1958        auto value = intLiteralToInt(type->stringView(m_lexer));
    19181959        if (!value)
    19191960            return Unexpected<Error>(value.error());
    … …  
    19211962    }
    19221963    case Lexer::Token::Type::UintLiteral: {
    1923         auto value = uintLiteralToUint(type->stringView);
     1964        auto value = uintLiteralToUint(type->stringView(m_lexer));
    19241965        if (!value)
    19251966            return Unexpected<Error>(value.error());
    … …  
    19271968    }
    19281969    case Lexer::Token::Type::FloatLiteral: {
    1929         auto value = floatLiteralToFloat(type->stringView);
     1970        auto value = floatLiteralToFloat(type->stringView(m_lexer));
    19301971        if (!value)
    19311972            return Unexpected<Error>(value.error());
    … …  
    19391980        return { makeUniqueRef<AST::BooleanLiteral>(WTFMove(*type), false) };
    19401981    case Lexer::Token::Type::Identifier: {
    1941         auto name = type->stringView.toString();
     1982        auto name = type->stringView(m_lexer).toString();
    19421983        return { makeUniqueRef<AST::VariableReference>(WTFMove(*type), WTFMove(name)) };
    19431984    }

  • trunk/Source/WebCore/Modules/webgpu/WHLSL/WHLSLParser.h

    r247124 r247171  
    144144    Expected<Lexer::Token, Error> peek();
    145145    Expected<Lexer::Token, Error> peekFurther();
    146     bool peekTypes(const Vector<Lexer::Token::Type>&);
     146    template <Lexer::Token::Type... types>
     147    bool peekTypes();
    147148    Optional<Lexer::Token> tryType(Lexer::Token::Type);
    148     Optional<Lexer::Token> tryTypes(const Vector<Lexer::Token::Type>&);
     149    template <Lexer::Token::Type... types>
     150    Optional<Lexer::Token> tryTypes();
    149151    Expected<Lexer::Token, Error> consumeType(Lexer::Token::Type);
    150     Expected<Lexer::Token, Error> consumeTypes(const Vector<Lexer::Token::Type>&);
     152    template <Lexer::Token::Type... types>
     153    Expected<Lexer::Token, Error> consumeTypes();
    151154
    152155    Expected<Variant<int, unsigned>, Error> consumeIntegralLiteral();

  • trunk/Source/WebCore/Modules/webgpu/WHLSL/WHLSLPreserveVariableLifetimes.cpp

    r247164 r247171  
    9494static ALWAYS_INLINE Lexer::Token anonymousToken(Lexer::Token::Type type)
    9595{
    96     return Lexer::Token { { }, 0, type };
     96    return Lexer::Token { 0, 0, type };
    9797}
    9898
Note: See TracChangeset for help on using the changeset viewer.