std.d.lexer : voting thread

[dennis luehring]

Am 05.10.2013 02:24, schrieb Andrei Alexandrescu:
> Instead of associating token types with small integers, we associate
> them with string addresses. (For efficiency we may use pointers to
> zero-terminated strings, but I don't think that's necessary).

would it be also more efficent to generate a big string out of the token 
list containing all tokes concatenated and use a generated string-slice 
for the associated string accesses?

imutable string generated_flat_token_stream = "...publicprivateclass..."

"public" = generated_flat_token_stream[3..9]

or would that kill caching on todays machines?