Package org.ek9lang.compiler.tokenizer

package org.ek9lang.compiler.tokenizer

Focuses on aspects of Lexing and some parts of Parser configuration.

  • Class
    Description
    DelegatingLexer
    A Lexer, but one that delegates to the actual lexer that in plugged in.
    Ek9Lexer
    Original good for python but too pythonesque for EK9 - needed reworking to remove # comment and tabs for indenting.
    Ek9LexerForInput
    Just wraps up the creation of the EK9Lexer from an inputStream.
    Ek9Token
    Models a token that we need to create as a synthetic token.
    IToken
    The internal model of a parse token - typically created.
    LexerPlugin
    Due to way Antlr does is class generation we need to make an interface to decouple to enable debugging and alternatives.
    ParserCreator
    Creates the lexer and parser and plugs in the listners.
    ParserSpec
    To be used to construct a parser, with a lexer.
    TokenConsumptionListener
    Listen as Tokens are consumed as they are pulled from the Lexer into the parser.
    TokenResult
    Typically used for locating the nearest token to some position in a source code file.
    TokenStreamAssessment
    But also to aid in listing out tokens so that when building grammars out of tokens it is possible to see what is being produced.