Lexer Implementation
Lexer Implementation
- protected function getRawRules()
Lexer Rules
Lexer Rules
- protected function getRules() — Process, normalize, and validate the raw lexer rules.
Lexer Tokens
Lexer Tokens
- public function getTokens($input, $initial_state) — Lex an input string into tokens.
Other Methods
Other Methods
- public function mergeTokens($tokens) — Merge adjacent tokens of the same type. For example, if a comment is tokenized as <"//", "comment">, this method will merge the two tokens into a single combined token.
- public function getLexerState()
- public function getNiceTokens($input)