std.d.lexer: pre-voting review / discussion

[Martin Nowak]

On 09/12/2013 03:30 AM, deadalnix wrote:
>>
>> That's correct, but that implies re-lexing the tokens, which has
>> negative performance implications.
>
> Indeed. What solution do you have in mind ?

Buffering the tokens would work. There are some ways promote input 
ranges to forward ranges. But there are also some pitfalls like the 
implicit save on copy.

I have two prototypes for a generic input range buffer.

https://gist.github.com/dawgfoto/2187220 - uses growing ring buffer
https://gist.github.com/dawgfoto/1257196 - uses ref counted lookahead 
buffers in a singly linked list

The lexer itself has a ringbuffer for input ranges.
https://github.com/Hackerpilot/phobos/blob/master/std/d/lexer.d#L2278