Geek asks and answers

ANTLRv3 does not read parameters

I am very new to ANTLR and trying to understand how Lexer and Parser rules work. I am having problems with the grammar I wrote, which seems to be related to lexer tokens with multiple characters, which are considered “matches”, even if only the first few characters actually match. To demonstrate this, I wrote a simple ANTLR 3 grammar:

grammar test;
options {
    k=3;
}

@lexer::header { package test;}
@header {package test;}

sentence    :   (CHARACTER)*;

CHARACTER   :   'a'..'z'|' ';
SPECIAL     :   'special';

I use AntlrWorks to parse the following test input:

apple basic say sponsor speeds speckled specific wonder

The output I get is:

apple basic say nsor ds led ic wonder

, LEXER k = 1 SPECIAL , "sp". "sp", SPECIAL, - ( ), , :

  line 1:18 mismatched chracter 'o' expecting 'e'

, . , ('special') - , . , , /, ,

:

  • antlr 3 (, k = 2 k = 3 ..)? , , , , .
  • , , , ?
+3
1

k options { ... } , .

,

CHARACTER   :   'a'..'z'|' ';
SPECIAL     :   'special';

: 'special' 7 'a'..'z'. :

grammar Test;

sentence : (special | word | space)+ EOF;
special  : SPECIAL;
word     : WORD;
space    : SPACE;

SPECIAL  : 'special';
WORD     : 'a'..'z'+;
SPACE    : ' ';

:

specia special specials

:

enter image description here

.. ( ) LL (1) " ". , , , Definitive ANTLR Reference ( , ...). , , .

AFAIK, char -tokens , char -tokens, - , , , "". :

grammar test;

tokens {
  LETTER;
}

@lexer::members {
  // manual look ahead
  private boolean ahead(String text) {
    for(int i = 0; i < text.length(); i++) {
      if(input.LA(i+1) != text.charAt(i)) {
        return false;
      }
    }
    return true;
  }
}

sentence
  :  (t=. {System.out.printf("\%-7s :: '\%s'\n", tokenNames[$t.type], $t.text);})+ EOF
  ;

SPECIAL 
  :  {ahead("special")}?=> 'special'
  |  {ahead("keyword")}?=> 'keyword'
  |  'a'..'z' {$type = LETTER;} // Last option and no keyword is found: 
                                // change the type of this token
  ;

SPACE
  :  ' '
  ;

, , :

import org.antlr.runtime.*;

public class Main {
    public static void main(String[] args) throws Exception {
        ANTLRStringStream in = new ANTLRStringStream("apple basic special speckled keyword keywor");
        testLexer lexer = new testLexer(in);
        CommonTokenStream tokens = new CommonTokenStream(lexer);
        testParser parser = new testParser(tokens);
        parser.sentence();
    }
}

, :

apple basic special speckled keyword keywor

:

LETTER  :: 'a'
LETTER  :: 'p'
LETTER  :: 'p'
LETTER  :: 'l'
LETTER  :: 'e'
SPACE   :: ' '
LETTER  :: 'b'
LETTER  :: 'a'
LETTER  :: 's'
LETTER  :: 'i'
LETTER  :: 'c'
SPACE   :: ' '
SPECIAL :: 'special'
SPACE   :: ' '
LETTER  :: 's'
LETTER  :: 'p'
LETTER  :: 'e'
LETTER  :: 'c'
LETTER  :: 'k'
LETTER  :: 'l'
LETTER  :: 'e'
LETTER  :: 'd'
SPACE   :: ' '
SPECIAL :: 'keyword'
SPACE   :: ' '
LETTER  :: 'k'
LETTER  :: 'e'
LETTER  :: 'y'
LETTER  :: 'w'
LETTER  :: 'o'
LETTER  :: 'r'

. Q & A " " ANTLR?, ANTLR.

+4

More articles:


All Articles