Geek asks and answers

Make lex read UTF-8 not working

I wrote an xml parser that parses ASCII files, but now I need to be able to read UTF-8 encoded files. I have the following regex in lex , but they do not conform to UTF-8. I'm not sure what I'm doing wrong:

utf_8       [\x00-\xff]*
bom         [\xEF\xBB\xBF]

then

bom             { fprintf( stderr, "OMG I SAW A BOM"); return BOM;}
utf_8           { fprintf( stderr, "OMG I SAW A UTF CHAR", yytext[0] ); return UTF_8;}

I also have the following grammar rules:

program 
: UTF8 '<' '?'ID attribute_list '?''>' 
root ...

where UTF8:

Utf8

: BOM           {printf("i saw a bom\n");}
| UTF_8         {printf("i saw a utf\n");}
|               {printf("i didn't see anything.'\n");} 
;

It always comes with i didn't see anything, my parser works for ASCII files, that is, when I copy the UTF-8 XML file to an empty document.

Any help would be greatly appreciated.

EDIT:

Here is the edited .l file for reference:

%{
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "y.tab.h"
int lines = 1;
%}

utf_8       [\x0000-\xffff]*
bom         [\xEF\xBB\xBF]
whitespace  [ \t]
ev          (.|{bom})
ev1         (.|{utf_8})
%%
{whitespace}    { fprintf( stderr, "%s", yytext );}
\n              { fprintf( stderr, "%s%d ", yytext, lines++ );}
.               { fprintf( stderr, "{TOKEN:%c}", yytext[0] ); return yytext[0];}
bom             { fprintf( stderr, "OMG I SAW A BOM"); return BOM;}
utf_8           { fprintf( stderr, "OMG I SAW A UTF CHAR", yytext[0] ); return UTF_8;}
%%

void error( char *message )
{
    fprintf( stderr, "Error: %s\n", message );
    exit(1);
}
+3
source share
1 answer

Ok, this is your problem:

utf_8       [\x0000-\xffff]*
bom         [\xEF\xBB\xBF]

. -, Flex Unicode. . , , UTF-8. http://keithdevens.com/weblog/archive/2004/Jun/29/UTF-8.regex , , Flex (. ). -, EF, BB BF, EB BB BF, .

(, UTF-8 , ).

Flex, , :

%{
#include <stdio.h>
%}

bom     \xEF\xBB\xBF
white   [ \t]

u2a     [\xC2-\xDF][\x80-\xBF]
u2b     \xE0[\xA0-\xBF][\x80-\xBF]
u3a     [\xE1-\xEC\xEE\xEF][\x80-\xBF]{2}
u3b     \xED[\x80-\x9F][\x80-\xBF]
u4a     \xF0[\x90-\xBF][\x80-\xBF]{2}
u4b     [\xF1-\xF3][\x80-\xBF]{3}
u4c     \xF4[\x80-\x8F][\x80-\xBF]{2}

utf_8   {u2a}|{u2b}|{u3a}|{u3b}|{u4a}|{u4b}|{u4c}

%%

{white}     { putchar(' ');  }
\n          { putchar('\n'); }
{bom}       { putchar('B');  }
{utf_8}     { putchar('u');  }
[\x21-\x7e] { putchar('.');  }
.           { putchar('^');  }
+5

More articles:


All Articles