std.d.lexer requirements

[Jason House]

On Thursday, 2 August 2012 at 04:48:56 UTC, Walter Bright wrote:
> On 8/1/2012 9:41 PM, H. S. Teoh wrote:
>> Whether it's part of the range type or a separate lexer type,
>> *definitely* make it possible to have multiple instances. One 
>> of the
>> biggest flaws of otherwise-good lexer generators like lex and 
>> flex
>> (C/C++) is that the core code assumes a single instance, and
>> multi-instances were glued on after the fact, making it a 
>> royal pain to
>> work with anything that needs lexing multiple things at the 
>> same time.
>
> Yup. I keep trying to think of a way to lex multiple files at 
> the same time in separate threads, but the problem is 
> serializing access to the identifier table will likely kill off 
> any perf gain.

The following is an incredibly fast multithreaded hash table. It 
is both lock-free and fence-free. Would something like that solve 
your problem?

http://www.azulsystems.com/events/javaone_2007/2007_LockFreeHash.pdf