Geek asks and answers

How to use antlr 4 TokenStream as an iterable stream?

I created lexer using antlr 4 to tokenize Turkish texts, I need to make a token stream that I can get tokens one by one. CommonTokenStream returns a list if I use it as follows:

ANTLRInputStream inputStream = new ANTLRInputStream(input);
TurkishLexer lexer = new TurkishLexer(inputStream);
CommonTokenStream tokenStream = new CommonTokenStream(lexer);
List<Token> tokens = tokenStream.fill();
for (Token token : token) ...

However, I do not want to create a list of tokens, since my input can be huge, I just want something like:

for (Token token: tokenStream.next()) ...

I would try before getting the EOF marker.

Is there a token stream that allows me to iterate over tokens?

+5
source share
1 answer

Instead of using, CommonTokenStreamyou can just use Lexer.nextToken.

for (Token token = lexer.nextToken();
     token.getType() != Token.EOF;
     token = lexer.nextToken())
{
    ...
+10
source

More articles:


All Articles