std.d.lexer requirements

[Walter Bright]

On 8/1/2012 10:34 PM, Jonathan M Davis wrote:
> On Wednesday, August 01, 2012 22:09:05 Walter Bright wrote:
>> On 8/1/2012 9:54 PM, Jonathan M Davis wrote:
>>> Then just pass the same identifier table to the function which creates the
>>> token range. That doesn't require another type.
>>
>> You're still going to require another type, otherwise you'll have to
>> duplicate the state in every token allocation, with resultant heavy memory
>> and initialization costs.
>
> Why would you be duplicating any state in the token? The range would have the
> state, not the token. The token would only have the data specific to the token,
> not the range as a whole.

I meant two types: the Lexer and the Token. The Lexer can present a range interface.