std.d.lexer requirements

[Jakob Ovrum]

On Thursday, 2 August 2012 at 00:11:15 UTC, Walter Bright wrote:
> 3. tokens should be values, not classes

I agree with everything but this point. Tokens become quite large 
when they have all kinds of string and position information on 
them, much larger than the typical recommended sizes for 
pass-by-value. It's still possible to provide an interface for 
them that doesn't require as much copying (ref returns and 
getters and whatnot), but it's fiddly and way too easy to copy 
these huge Token structs around, especially if the output is a 
range of Token structs.