std.d.lexer requirements

[Walter Bright]

On 8/3/2012 6:18 AM, deadalnix wrote:
> lexer can have a parameter that tell if it should build a table of token or
> slice the input. The second is important, for instance for an IDE : lexing will
> occur often, and you prefer slicing here because you already have the source
> file in memory anyway.

A string may span multiple lines - IDEs do not store the text as one string.

> If the lexer allocate chunks, it will reuse the same memory location for the
> same string. Considering the following mecanism to compare slice, this will
> require 2 comparaisons for identifier lexed with that method :
>
> if(a.length != b.length) return false;
> if(a.ptr == b.ptr) return true;
> // Regular char by char comparison.
>
> Is that a suitable option ?

You're talking about doing for strings what is done for identifiers - returning 
a unique handle for each. I don't think this works very well for string 
literals, as there seem to be few duplicates.