std.d.lexer requirements

[Walter Bright]

On 8/1/2012 5:21 PM, Jakob Ovrum wrote:
> On Thursday, 2 August 2012 at 00:11:15 UTC, Walter Bright wrote:
>> 3. tokens should be values, not classes
>
> I agree with everything but this point. Tokens become quite large when they have
> all kinds of string and position information on them, much larger than the
> typical recommended sizes for pass-by-value. It's still possible to provide an
> interface for them that doesn't require as much copying (ref returns and getters
> and whatnot), but it's fiddly and way too easy to copy these huge Token structs
> around, especially if the output is a range of Token structs.

Doing a class requires a heap allocation per token, which will have disastrous 
performance consequences.