Let's stop parser Hell

[Jacob Carlborg]

On 2012-08-01 00:38, Jonathan M Davis wrote:

> I don't have the code with me at the moment, but I believe that the token type
> looks something like
>
> struct Token
> {
>   TokenType type;
>   string str;
>   LiteralValue value;
>   SourcePos pos;
> }
>
> struct SourcePos
> {
>   size_t line;
>   size_ col;
>   size_t tabWidth = 8;
> }
>

What about the end/length of a token? Token.str.length would give the 
number of bytes (code units?) instead of the number of characters (code 
points?). I'm not entirely sure what's needed when, for example, doing 
syntax highlighting. I assume you would know the length in characters of 
a given token internally inside the lexer?

-- 
/Jacob Carlborg