std.d.lexer requirements

[Dmitry Olshansky]

On 04-Aug-12 14:55, Christophe Travert wrote:
> Dmitry Olshansky , dans le message (digitalmars.D:174214), a écrit :
>> Most likely - since you re-read the same memory twice to do it.
>
> You're probably right, but if you do this right after the token is
> generated, the memory should still be near the processor. And the
> operation on the first read should be very basic: just check nothing
> illegal appears, and check for the end of the token.

q{ .. }
"\x13\x27 ...\u1212"

In most cases it takes around the same time to check correctness and 
output it as simply pass it by. (see also re-decoding unicode in 
identifiers, though that's rare to see unicode chars in identifier)


> The cost is not
> negligible, but what you do with litteral tokens can vary much, and what
> the lexer will propose may not be what the user want, so the user may
> suffer the cost of the litteral decoding (including allocation of the
> decoded string, the copy of the caracters, etc), that he doesn't want,
> or will have to re-do his own way...
>
I see it as a compile-time policy, that will fit nicely and solve both 
issues. Just provide a templates with a few hooks, and add a Noop policy 
that does nothing.

-- 
Dmitry Olshansky