Class TokenStreamAssessment

java.lang.Object
org.ek9lang.compiler.tokenizer.TokenStreamAssessment
public class TokenStreamAssessment extends Object
Drives ReadabilityAssessor over a whole lexer, for standalone and debug use - and to list out tokens so that when building grammars out of tokens it is possible to see what is being produced.

Note the use of the LexerPlugin, this is so we can alter the underlying Lexer to get the right stream of tokens.

The compiler itself does NOT use this on the compile path: lexing a source a second time just to score it costs more than the parse. CompilableSource feeds the assessor from tokenConsumed during the parser's own lex instead. The counting rules live in ReadabilityAssessor so both routes cannot drift apart.

  • Constructor Details

    • TokenStreamAssessment

      public TokenStreamAssessment()

  • Method Details

    • assess

      public String assess(LexerPlugin lexer, boolean printTokens)
      Assess the reading complexity of the tokens from the lexer. This consumes the lexer.