std.d.lexer requirements

[Walter Bright]

On 8/1/2012 10:53 PM, Jonathan M Davis wrote:
> What I was expecting there to  be was a type which was a range of tokens.  You
> passed the source string to a function which returned that range, and you
> iterated over it to process each token. What you appear to have been arguing
> for is another type which you get the range from which holds additional state
> (e.g. the indentifier table). So, you're saying now that you only meant that
> the token type needs to be separate from the range type (which I would have
> thought would be the only way to do it), or are you indeed saying that there
> should be another type that the range comes from and which holds additional
> state?

You could do it either way.