Lexer Implementation
Lexer Implementation
- protected function getRawRules()
 
Lexer Rules
Lexer Rules
- protected function getRules() — Process, normalize, and validate the raw lexer rules.
 
Lexer Tokens
Lexer Tokens
- public function getTokens($input, $initial_state) — Lex an input string into tokens.
 
Other Methods
Other Methods
- public function mergeTokens($tokens) — Merge adjacent tokens of the same type. For example, if a comment is tokenized as <"//", "comment">, this method will merge the two tokens into a single combined token.
 - public function getLexerState()
 - public function getNiceTokens($input)