Request for comments: std.d.lexer

[FG]

On 2013-01-30 17:50, Dmitry Olshansky wrote:
> Instead I suggest to try and allocate a big block of fixed size (say about
> 16-64K) upfront and copy identifiers one by one there. When it fills just
> allocate another one and move on.

Yeah, similar to what I suggested, except that probably it would be good to also 
have a look-up structure for identifiers, so that only unique strings go into 
those blocks.

I wonder what would be a practical data structure for such look-ups:
A trie is great except for the implementation hurdle to keep it also in one or a 
few memory blocks to prevent frequent allocations.
A simpler approach would be to make it an array of (hash, string_slice) pairs, 
sorted by hash (of the identifier) - lots of string scanning though.

What do you think?