std.d.lexer: pre-voting review / discussion

[qznc]

On Thursday, 12 September 2013 at 01:39:52 UTC, Walter Bright 
wrote:
> On 9/11/2013 6:30 PM, deadalnix wrote:
>> Indeed. What solution do you have in mind ?
>
> The solution dmd uses is to put in an intermediary layer that 
> saves the lookahead tokens in a linked list.

I think this is the right approach. It can probably be another 
function, we can put into std.range and reuse for other 
lexer/parsers. The lexer or the parser should not be made more 
complex for this.