std.d.lexer requirements

[Jonathan M Davis]

On Thursday, August 02, 2012 11:08:23 Walter Bright wrote:
> The tokens are not kept, correct. But the identifier strings, and the string
> literals, are kept, and if they are slices into the input buffer, then
> everything I said applies.

String literals often _can't_ be slices unless you leave them in their 
original state rather than giving the version that they translate to (e.g. 
leaving \© in the string rather than replacing it with its actual, 
unicode value). And since you're not going to be able to create the literal 
using whatever type the range is unless it's a string of some variety, that 
means that the literals often can't be slices, which - depending on the 
implementation - would make it so that that they can't _ever_ be slices.

Identifiers are a different story, since they don't have to be translated at 
all, but regardless of whether keeping a slice would be better than creating a 
new string, the identifier table will be far superior, since then you only need 
one copy of each identifier. So, it ultimately doesn't make sense to use slices 
in either case even without considering issues like them being spread across 
memory.

The only place that I'd expect a slice in a token is in the string which 
represents the text which was lexed, and that won't normally be kept around.

- Jonathan M Davis