Class Ek9Lexer

java.lang.Object

org.antlr.v4.runtime.Recognizer<Integer, org.antlr.v4.runtime.atn.LexerATNSimulator>

org.antlr.v4.runtime.Lexer

org.ek9lang.antlr.EK9LexerRules

org.ek9lang.compiler.tokenizer.Ek9Lexer

All Implemented Interfaces:

org.antlr.v4.runtime.TokenSource, LexerPlugin

public class Ek9Lexer
extends EK9LexerRules
implements LexerPlugin

Original good for python but too pythonesque for EK9 - needed reworking to remove # comment and tabs for indenting. EK9 only allows spaces for indents.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	TEXT_INSERTED_INDENT

Fields inherited from class EK9LexerRules

_ATN, _decisionToDFA, _serializedATN, _sharedContextCache, ABS, ABSTRACT, ADD, ADD_ASSIGN, ALLOW, AND, ANY, APPLICATION, AS, ASPECT, ASSERT, ASSERT_DOES_NOT_THROW, ASSERT_PANIC, ASSERT_THROWS, ASSIGN, ASSIGN_UNSET, ASSIGN2, ASYNC, AT, BANG, BinaryLiteral, BLOCK_COMMENT1, BLOCK_COMMENT2, BooleanLiteral, BREAK, BY, BytesLiteral, CALL, CARET, CASE, CAT, CATCH, channelNames, CharacterLiteral, CHECK, CLASS, CLOSE, CLOSE_STRING, CMP, COALESCE_GE, COALESCE_GT, COALESCE_LE, COALESCE_LT, CODE_COMMENT, COLLECT, COLON, COLONCOLON, ColourLiteral, COMMA, COMPONENT, CONST, CONSTANT, CONSTRAIN, CONTAINS, CONTINUE, COPY, DateLiteral, DateTimeLiteral, DEC, DecorationDimensionLiteral, DecorationResolutionLiteral, DEF, DEFAULT, DEFINES, DISPATCHER, DIV, DIV_ASSIGN, DO, DOLLAR, DOT, DurationLiteral, ELIF, ELLIPSIS, ELSE, ELVIS, EMPTY, ENTER_EXPR_INTERPOLATION, EQUAL, EXTENDS, EXTERN, FILTER, FINAL, FINALLY, FLATTEN, FloatingPointLiteral, FOR, FUNCTION, FUZ, GE, GIVEN, GOTO, GROUP, GT, GUARD, HANDLE, HASH, HASHCODE, HEAD, HTTP_CONTENT, HTTP_CONTEXT, HTTP_DELETE, HTTP_GET, HTTP_HEAD, HTTP_HEADER, HTTP_OPTIONS, HTTP_PATCH, HTTP_PATH, HTTP_POST, HTTP_PUT, HTTP_QUERY, HTTP_REQUEST, Identifier, IF, IN, INC, INDENT, IntegerLiteral, IS, JOIN, LBRACE, LBRACK, LE, LEFT_ARROW, LENGTH, LINE_COMMENT, LPAREN, LT, MAP, MATCHES, MERGE, MillisecondLiteral, MOD, modeNames, MODULE, MoneyLiteral, MUL, MUL_ASSIGN, NEW, NL, NONE, NOT, NOTEQUAL, NOTEQUAL2, NULL, OF, ONLY, OPEN, OPEN_STRING, OPERATOR, OR, OVERRIDE, PACKAGE, PATH, PathLiteral, PIPE, PREFIX, PRIVATE, PROGRAM, PROMOTE, PROTECTED, PUBLIC, PURE, QUESTION, RANGE, RBRACE, RBRACK, RECORD, REFERENCES, RegExLiteral, REGISTER, REJECT, REM, REPLACE, REQUIRE, RETURN, RIGHT_ARROW, RPAREN, ruleNames, SANITIZED, SELF, SEMI, SERVICE, SHEBANG, SHFTL, SHFTR, SKIPPING, SORT, SPLIT, SQRT, STRING_MODE, STRING_TEXT, StringLiteral, SUB, SUB_ASSIGN, SUFFIX, SUPER, SWITCH, TAB, TAIL, TEE, TEXT, TEXT_CHAR, THEN, THIS, THROW, TILDE, TimeLiteral, TOJSON, tokenNames, TRAIT, TRY, TYPE, UNIQ, Uriproto, VersionNumberLiteral, VOCABULARY, WHEN, WHILE, WITH, WS, XOR

Fields inherited from class org.antlr.v4.runtime.Lexer

_channel, _factory, _hitEOF, _input, _mode, _modeStack, _text, _token, _tokenFactorySourcePair, _tokenStartCharIndex, _tokenStartCharPositionInLine, _tokenStartLine, _type, DEFAULT_MODE, DEFAULT_TOKEN_CHANNEL, HIDDEN, MAX_CHAR_VALUE, MIN_CHAR_VALUE, MORE, SKIP

Fields inherited from class org.antlr.v4.runtime.Recognizer

_interp, EOF

Constructor Summary

Constructors

Constructor	Description
Ek9Lexer(org.antlr.v4.runtime.CharStream input, int indentToken, int dedentToken)	Create a lexer for an input with specific indent and dedent tokens.

Method Summary

Modifier and Type	Method	Description
int	getDedentToken()
int	getIndentToken()
String	getSourceName()
String	getSymbolicName(int tokenType)
boolean	isLimitExceeded()	True once this source exceeded one of the shape limits the lexer enforces.
protected boolean	isRegexPossible()
org.antlr.v4.runtime.Token	nextToken()
int	popMode()
Ek9Lexer	setLimitsEnforced(boolean limitsEnforced)	The shape limits are on by default, because they protect the parser.
Ek9Lexer	setPrintTokensAsSupplied(boolean printTokensAsSupplied)
Ek9Lexer	setSourceName(String sourceName)
void	setTokenListener(TokenConsumptionListener tokenListener)

Methods inherited from class EK9LexerRules

getATN, getChannelNames, getGrammarFileName, getModeNames, getRuleNames, getSerializedATN, getTokenNames, getVocabulary, sempred

Methods inherited from class org.antlr.v4.runtime.Lexer

emit, emit, emitEOF, getAllTokens, getChannel, getCharErrorDisplay, getCharIndex, getCharPositionInLine, getErrorDisplay, getErrorDisplay, getInputStream, getLine, getText, getToken, getTokenFactory, getType, mode, more, notifyListeners, pushMode, recover, recover, reset, setChannel, setCharPositionInLine, setInputStream, setLine, setText, setToken, setTokenFactory, setType, skip

Methods inherited from class org.antlr.v4.runtime.Recognizer

action, addErrorListener, getErrorHeader, getErrorListenerDispatch, getErrorListeners, getInterpreter, getParseInfo, getRuleIndexMap, getState, getTokenErrorDisplay, getTokenType, getTokenTypeMap, precpred, removeErrorListener, removeErrorListeners, setInterpreter, setState

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface LexerPlugin

addErrorListener, removeErrorListeners

Methods inherited from interface org.antlr.v4.runtime.TokenSource

getCharPositionInLine, getInputStream, getLine, getTokenFactory, setTokenFactory

Field Details

TEXT_INSERTED_INDENT

public static final String TEXT_INSERTED_INDENT

See Also:

• Constant Field Values

Constructor Details

Ek9Lexer

public Ek9Lexer(org.antlr.v4.runtime.CharStream input,
 int indentToken,
 int dedentToken)

Create a lexer for an input with specific indent and dedent tokens.

Method Details

getSourceName

public String getSourceName()

Specified by:

getSourceName in interface org.antlr.v4.runtime.TokenSource

Overrides:

getSourceName in class org.antlr.v4.runtime.Lexer

setSourceName

public Ek9Lexer setSourceName(String sourceName)

setTokenListener

public void setTokenListener(TokenConsumptionListener tokenListener)

getIndentToken

public int getIndentToken()

Specified by:

getIndentToken in interface LexerPlugin

getDedentToken

public int getDedentToken()

Specified by:

getDedentToken in interface LexerPlugin

getSymbolicName

public String getSymbolicName(int tokenType)

Specified by:

getSymbolicName in interface LexerPlugin

nextToken

public org.antlr.v4.runtime.Token nextToken()

Specified by:

nextToken in interface org.antlr.v4.runtime.TokenSource

Overrides:

nextToken in class org.antlr.v4.runtime.Lexer

setPrintTokensAsSupplied

public Ek9Lexer setPrintTokensAsSupplied(boolean printTokensAsSupplied)

setLimitsEnforced

public Ek9Lexer setLimitsEnforced(boolean limitsEnforced)

The shape limits are on by default, because they protect the parser. A token scanner that never parses - an editor locating comments or identifiers, the syntax highlighter - turns them off: it must see every token, and a stream cut at a limit would hide whatever follows the cut (a reference used only there would read as unused).

popMode

public int popMode()

Overrides:

popMode in class org.antlr.v4.runtime.Lexer

isRegexPossible

protected boolean isRegexPossible()

Overrides:

isRegexPossible in class EK9LexerRules

isLimitExceeded

public boolean isLimitExceeded()

True once this source exceeded one of the shape limits the lexer enforces.