std.d.lexer requirements

[Tobias Pankrath]

On Friday, 3 August 2012 at 14:49:55 UTC, 
travert at phare.normalesup.org
>
> If I may add, there are several possitilities here:
>  1- a real slice of the input range
>  2- a slice of the input range created with .save and 
> takeExactly
>  3- a slice allocated in GC memory by the lexer
>  4- a slice of memory owned by the lexer, which is reused for 
> the next
> token (thus, the next call to popFront invalidates the token).
>  5- a slice of memory from a lookup table.

#6 None. Just provide information of token type and location 
(i.e. for syntax highlighting).