Package org.ek9lang.compiler.tokenizer

Class Ek9Lexer

java.lang.Object

org.antlr.v4.runtime.Recognizer<Integer,org.antlr.v4.runtime.atn.LexerATNSimulator>

org.antlr.v4.runtime.Lexer

org.ek9lang.antlr.EK9LexerRules

org.ek9lang.compiler.tokenizer.Ek9Lexer

All Implemented Interfaces:

org.antlr.v4.runtime.TokenSource, LexerPlugin

public class Ek9Lexer
extends EK9LexerRules
implements LexerPlugin

Original good for python but too pythonesque for EK9 - needed reworking to remove # comment and tabs for indenting. EK9 only allows spaces for indents.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	TEXT_INSERTED_INDENT

Fields inherited from class org.ek9lang.antlr.EK9LexerRules

_ATN, _decisionToDFA, _serializedATN, _sharedContextCache, ABS, ABSTRACT, ADD, ADD_ASSIGN, ALLOW, AND, ANY, APPLICATION, AS, ASPECT, ASSERT, ASSIGN, ASSIGN_UNSET, ASSIGN2, ASYNC, AT, BANG, BinaryLiteral, BLOCK_COMMENT1, BLOCK_COMMENT2, BooleanLiteral, BY, CALL, CARET, CASE, CAT, CATCH, channelNames, CharacterLiteral, CHECK, CLASS, CLOSE, CLOSE_STRING, CMP, COALESCE_GE, COALESCE_GT, COALESCE_LE, COALESCE_LT, CODE_COMMENT, COLLECT, COLON, COLONCOLON, ColourLiteral, COMMA, COMPONENT, CONST, CONSTANT, CONSTRAIN, CONTAINS, COPY, DateLiteral, DateTimeLiteral, DEC, DecorationDimensionLiteral, DecorationResolutionLiteral, DEFAULT, DEFINES, DISPATCHER, DIV, DIV_ASSIGN, DO, DOLLAR, DOT, DurationLiteral, ELLIPSIS, ELSE, ELVIS, EMPTY, ENTER_EXPR_INTERPOLATION, EQUAL, EXTENDS, EXTERN, FILTER, FINAL, FINALLY, FLATTEN, FloatingPointLiteral, FOR, FUNCTION, FUZ, GE, GIVEN, GROUP, GT, GUARD, HANDLE, HASH, HASHCODE, HEAD, HTTP_CONTENT, HTTP_CONTEXT, HTTP_DELETE, HTTP_GET, HTTP_HEAD, HTTP_HEADER, HTTP_OPTIONS, HTTP_PATCH, HTTP_PATH, HTTP_POST, HTTP_PUT, HTTP_QUERY, HTTP_REQUEST, Identifier, IF, IN, INC, INDENT, IntegerLiteral, IS, ISOLATED, JOIN, LBRACE, LBRACK, LE, LEFT_ARROW, LENGTH, LINE_COMMENT, LPAREN, LT, MAP, MATCHES, MERGE, MillisecondLiteral, MOD, modeNames, MODULE, MoneyLiteral, MUL, MUL_ASSIGN, NL, NOT, NOTEQUAL, NOTEQUAL2, OF, ONLY, OPEN, OPEN_STRING, OPERATOR, OR, OVERRIDE, PACKAGE, PATH, PathLiteral, PIPE, PREFIX, PRIVATE, PROGRAM, PROMOTE, PROTECTED, PUBLIC, PURE, QUESTION, RANGE, RBRACE, RBRACK, RECORD, REFERENCES, RegExLiteral, REGISTER, REM, REPLACE, RIGHT_ARROW, RPAREN, ruleNames, SELECT, SEMI, SERVICE, SHEBANG, SHFTL, SHFTR, SKIPPING, SORT, SPLIT, SQRT, STRING_MODE, STRING_TEXT, StringLiteral, SUB, SUB_ASSIGN, SUFFIX, SUPER, SWITCH, TAB, TAIL, TEE, TEXT, TEXT_CHAR, THEN, THIS, THROW, TILDE, TimeLiteral, TOJSON, tokenNames, TRAIT, TRY, TYPE, UNIQ, URI, Uriproto, VersionNumberLiteral, VOCABULARY, WHEN, WHILE, WITH, WS, XOR

Fields inherited from class org.antlr.v4.runtime.Lexer

_channel, _factory, _hitEOF, _input, _mode, _modeStack, _text, _token, _tokenFactorySourcePair, _tokenStartCharIndex, _tokenStartCharPositionInLine, _tokenStartLine, _type, DEFAULT_MODE, DEFAULT_TOKEN_CHANNEL, HIDDEN, MAX_CHAR_VALUE, MIN_CHAR_VALUE, MORE, SKIP

Fields inherited from class org.antlr.v4.runtime.Recognizer

_interp, EOF

Constructor Summary

Constructors

Constructor	Description
Ek9Lexer(org.antlr.v4.runtime.CharStream input, int indentToken, int dedentToken)	Create a lexer for an input with specific indent and dedent tokens.

Method Summary

Modifier and Type	Method	Description
int	getDedentToken()
int	getIndentToken()
String	getSourceName()
String	getSymbolicName(int tokenType)
protected boolean	isRegexPossible()
org.antlr.v4.runtime.Token	nextToken()
int	popMode()
Ek9Lexer	setPrintTokensAsSupplied(boolean printTokensAsSupplied)
Ek9Lexer	setSourceName(String sourceName)
void	setTokenListener(TokenConsumptionListener tokenListener)

Methods inherited from class org.ek9lang.antlr.EK9LexerRules

getATN, getChannelNames, getGrammarFileName, getModeNames, getRuleNames, getSerializedATN, getTokenNames, getVocabulary, sempred

Methods inherited from class org.antlr.v4.runtime.Lexer

emit, emit, emitEOF, getAllTokens, getChannel, getCharErrorDisplay, getCharIndex, getCharPositionInLine, getErrorDisplay, getErrorDisplay, getInputStream, getLine, getText, getToken, getTokenFactory, getType, mode, more, notifyListeners, pushMode, recover, recover, reset, setChannel, setCharPositionInLine, setInputStream, setLine, setText, setToken, setTokenFactory, setType, skip

Methods inherited from class org.antlr.v4.runtime.Recognizer

action, addErrorListener, getErrorHeader, getErrorListenerDispatch, getErrorListeners, getInterpreter, getParseInfo, getRuleIndexMap, getState, getTokenErrorDisplay, getTokenType, getTokenTypeMap, precpred, removeErrorListener, removeErrorListeners, setInterpreter, setState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.ek9lang.compiler.tokenizer.LexerPlugin

addErrorListener, removeErrorListeners

Methods inherited from interface org.antlr.v4.runtime.TokenSource

getCharPositionInLine, getInputStream, getLine, getTokenFactory, setTokenFactory

Field Details

TEXT_INSERTED_INDENT

public static final String TEXT_INSERTED_INDENT

See Also:

• Constant Field Values

Constructor Details

Ek9Lexer

public Ek9Lexer(org.antlr.v4.runtime.CharStream input,
 int indentToken,
 int dedentToken)

Create a lexer for an input with specific indent and dedent tokens.

Method Details

getSourceName

public String getSourceName()

Specified by:

getSourceName in interface org.antlr.v4.runtime.TokenSource

Overrides:

getSourceName in class org.antlr.v4.runtime.Lexer

setSourceName

public Ek9Lexer setSourceName(String sourceName)

setTokenListener

public void setTokenListener(TokenConsumptionListener tokenListener)

getIndentToken

public int getIndentToken()

Specified by:

getIndentToken in interface LexerPlugin

getDedentToken

public int getDedentToken()

Specified by:

getDedentToken in interface LexerPlugin

getSymbolicName

public String getSymbolicName(int tokenType)

Specified by:

getSymbolicName in interface LexerPlugin

nextToken

public org.antlr.v4.runtime.Token nextToken()

Specified by:

nextToken in interface org.antlr.v4.runtime.TokenSource

Overrides:

nextToken in class org.antlr.v4.runtime.Lexer

setPrintTokensAsSupplied

public Ek9Lexer setPrintTokensAsSupplied(boolean printTokensAsSupplied)

popMode

public int popMode()

Overrides:

popMode in class org.antlr.v4.runtime.Lexer

isRegexPossible

protected boolean isRegexPossible()

Overrides:

isRegexPossible in class EK9LexerRules