Lexer in D

[Zach the Mystic]

On Saturday, 2 March 2013 at 11:59:39 UTC, Dmitry Olshansky wrote:
> 02-Mar-2013 15:04, David пишет:
>> Am 02.03.2013 10:33, schrieb Namespace:
>>> For one of our courses this year we have a very cool topic: 
>>> Lexer and
>>> Compiler.
>>> In the past I have already tried some experiments in this 
>>> direction with
>>> Romulus and Remus, but I've never measured my Lexing time.
>>> That's why I wanted to start again to write a lexer (hitherto 
>>> purely for
>>> the learning effect).
>>> I looked at the code of some of the other lexers, that are 
>>> also written
>>> in D, to get some inspiration from, but currently my Lexer is 
>>> a little
>>> bit slow, it requires for example ~200 msecs for std.datetime.
>>> So I wanted to ask whether there are nice people who help me 
>>> to improve
>>> our lexer.
>>>
>>> Code: https://github.com/Dgame/Puzzle/blob/master/lexer.d
>>>
>>> Thanks in advance. :)
>>
>> Making `types` and `keywords` an array might improve the speed 
>> a little bit:
>>
>> if (id in types) -> if (types.canFind(id))
>> (same with keywords)
>
> That would be slower as canFind is linear search.
> Simpler way is use first letter and length to select a bucket 
> of keywords.
>
> But otherwise, yes, the first advice is never use built-in AA 
> as these take quite some time to allocate. Plus they not that 
> fast to lookup as they used mod-prime not mod-power-of-2 to 
> pick a bucket.

And it doesn't even need to:
http://d.puremagic.com/issues/show_bug.cgi?id=9522