Let's stop parser Hell

[Jonathan M Davis]

On Wednesday, August 01, 2012 14:44:29 Philippe Sigaud wrote:
> Everytime I think I understand D strings, you prove me wrong. So, I
> *still* don't get how that works:
> 
> say I have
> 
> auto s = " - some greek or chinese chars, mathematical symbols, whatever -
> "d;
> 
> Then, the "..." part is lexed as a string literal. How can the string
> field in the Token magically contain UTF32 characters?

It contains unicode. The lexer is lexing whatever encoding the source is in, 
which has _nothing_ to do with the d on the end. It could be UTF-8, or UTF-16, 
or UTF-32. If we supported other encodings in ranges, it could be one of 
those. Which of those it is is irrelevant. As far as the value of the literal 
goes, these two strings are identical:

"ウェブサイト"
"\u30A6\u30A7\u30D6\u30B5\u30A4\u30C8"

The encoding of the source file is irrelevant. By tacking a d on the end

"ウェブサイト"d
"\u30A6\u30A7\u30D6\u30B5\u30A4\u30C8"d

you're just telling the compiler that you want the value that it generates to 
be in UTF-32. The source code could be in any of the supported encodings, and 
the string could be held in any encoding until the object code is actually 
generated.

> So, wouldn't it make sense to at least provide an option on the lexer
> to specifically store identifier lexemes and comments as a dstring?

You mean make it so that Token is 

struct Token(R)
{
    TokenType    type;
    R       str;
    LiteralValue value
    SourcePos    pos;
}

instead of

struct Token
{
    TokenType    type;
    string       str;
    LiteralValue value
    SourcePos    pos;
}

or do you mean something else? I may do something like that, but I would point 
out that if R doesn't have slicing, then that doesn't work. So, str can't 
always be the same type as the original range. For ranges with no slicing, it 
would have to be something else (probably either string or 
typeof(takeExactly(range))). However, making str R _does_ come at the cost of 
complicating code using the lexer, since instead of just using Token, you have 
to worry about whether it's a Token!string, Token!dstring, etc, and whether 
it's worth that complication is debatable. By far the most common use case is 
to lex string, and if str is string, and R is not, then you incur the penalty 
of converting R to string. So, the common use case is fast, and the uncommon 
use case still works but is slower, and the user of the lexer doesn't have to 
care what the original range type was.

It could go either way. I used string on first pass, but as I said, I could 
change it to R later if that makes more sense. I'm not particularly hung up on 
that little detail at this point, and that's probably one of the things that 
can be changed reasonably easily later.

- Jonathan M Davis