std.d.lexer : voting thread

[Martin Nowak]

On 10/02/2013 04:41 PM, Dicebot wrote:
> After brief discussion with Brian and gathering data from the review
> thread, I have decided to start voting for `std.d.lexer` inclusion into
> Phobos.

I also have to vote with no for now.

My biggest concern is that the lexer incorporates a string pool,
something that isn't strictly part of lexing.
IMO this is a major design flaw and possible performance/memory issue.
It is buried into the API because byToken takes const(byte)[], i.e. 
mutable data, but each Token carries a string value, so it always 
requires a copy.
For stream oriented lexing, e.g. token highlighting, no string pool is 
required at all.
Instead the value type of Token should be something like 
take(input.save, lengthOfToken).

Why was the Tok!">>=", Tok!"default" idea turned down. This leaves us 
with undesirable names like Tok.shiftRightAssign, Tok.default_.

There are a few smaller issues that haven't yet been addressed, but of 
course this can be done during the merge code review.

Adding it as experimental module would be a good idea.