Geek asks and answers

Is this a lexer job?

Let's say I used the lexing ruby ​​method definition:

def print_greeting(greeting = "hi")  
end

Is this the lexer's task to maintain state and emit relevant tokens, or should it be relatively dumb? Note that in the above example, the parameter greetinghas a default value "hi". In another context greeting = "hi", it is a variable assignment that sets greetingto "hi". Should the lexer generate common tokens, such as IDENTIFIER EQUALS STRING, or should it be context-sensitive and allocate something like PARAM_NAME EQUALS STRING?

+3
source share
4 answers

, , , , IDENTIFIER EQUALS STRING. ( ..) , . , , () . .

+4

, lexer "" - : DEF IDENTIFIER OPEN_PARENTHESIS IDENTIFIER EQUALS STRING CLOSE_PARENTHESIS END. - .

+2

ruby, .

, , ( " " " " ).

"" . .

lexer/scanners , , "" , , , , . lexer , .

, , "+" "-" .

+1

The distinction between lexical analysis and parsing is arbitrary. In many cases, you do not need a separate step. However, since performance is usually the most important issue (otherwise parsing would be basically a trivial task), then you need to decide and maybe measure whether additional processing is justified during lexical analysis or not. There is no general answer.

+1
source

More articles:


All Articles