std.d.lexer requirements

[Tobias Pankrath]

>> Correct, that's the whole point of using a range - it can come 
>> from anything.
>
> For example, let's suppose we want to do D syntax highlighting 
> in our IDE. It is highly unlikely that the text editor's data 
> structure is a simple string. It's likely to be an array of 
> lines, or something like that. It's simple to provide a range 
> interface to that, where it presents the data structure as a 
> sequence of chars.
>

Would this be an argument for putting the computation of source 
locations (i.e. line + offset or similar) into the range / into 
an template argument / policy, so that it's done in the most 
effective way for the client?

Kate for example has a "range"-type that marks a span in the text 
buffer. This way the lexer can return token with the correct 
"textrange" attached and you don't need to recompute the text 
ranges from line/col numbers.