std.d.lexer requirements

[Christophe Travert]

Dmitry Olshansky , dans le message (digitalmars.D:174214), a écrit :
> Most likely - since you re-read the same memory twice to do it.

You're probably right, but if you do this right after the token is 
generated, the memory should still be near the processor. And the 
operation on the first read should be very basic: just check nothing 
illegal appears, and check for the end of the token. The cost is not 
negligible, but what you do with litteral tokens can vary much, and what 
the lexer will propose may not be what the user want, so the user may 
suffer the cost of the litteral decoding (including allocation of the 
decoded string, the copy of the caracters, etc), that he doesn't want, 
or will have to re-do his own way...

-- 
Christophe