std.d.lexer: pre-voting review / discussion

[deadalnix]

On Thursday, 12 September 2013 at 01:39:52 UTC, Walter Bright 
wrote:
> On 9/11/2013 6:30 PM, deadalnix wrote:
>> Indeed. What solution do you have in mind ?
>
> The solution dmd uses is to put in an intermediary layer that 
> saves the lookahead tokens in a linked list.

But then, you have an extra step when looking up every tokens + 
memory management overhead. How big is the performance 
improvement ?