std.d.lexer requirements

[Jonathan M Davis]

On Wednesday, August 01, 2012 22:41:01 Walter Bright wrote:
> On 8/1/2012 10:34 PM, Jonathan M Davis wrote:
> > On Wednesday, August 01, 2012 22:09:05 Walter Bright wrote:
> >> On 8/1/2012 9:54 PM, Jonathan M Davis wrote:
> >>> Then just pass the same identifier table to the function which creates
> >>> the
> >>> token range. That doesn't require another type.
> >> 
> >> You're still going to require another type, otherwise you'll have to
> >> duplicate the state in every token allocation, with resultant heavy
> >> memory
> >> and initialization costs.
> > 
> > Why would you be duplicating any state in the token? The range would have
> > the state, not the token. The token would only have the data specific to
> > the token, not the range as a whole.
> 
> I meant two types: the Lexer and the Token. The Lexer can present a range
> interface.

What I was expecting there to  be was a type which was a range of tokens.  You 
passed the source string to a function which returned that range, and you 
iterated over it to process each token. What you appear to have been arguing 
for is another type which you get the range from which holds additional state 
(e.g. the indentifier table). So, you're saying now that you only meant that 
the token type needs to be separate from the range type (which I would have 
thought would be the only way to do it), or are you indeed saying that there 
should be another type that the range comes from and which holds additional 
state?

- Jonathan M Davis