Request for comments: std.d.lexer

[Timon Gehr]

On 01/27/2013 11:42 AM, Brian Schott wrote:
> On Sunday, 27 January 2013 at 10:17:48 UTC, Philippe Sigaud wrote:
>> * Having a range interface is good. Any reason why you made byToken a
>> class and not a struct? Most (like, 99%) of range in Phobos are
>> structs. Do you need reference semantics?
>
> It implements the InputRange interface from std.range so that users have
> a choice of using template constraints or the OO model in their code.
> ...

The lexer range must be a struct.

> ...
>> * A rough estimate of number of tokens/s would be good (I know it'll
>> vary). Walter seems to think if a lexer is not able to vomit thousands
>> of tokens a seconds, then it's not good. On a related note, does your
>> lexer have any problem with 10k+-lines files?
>
> $ time dscanner --sloc ../phobos/std/datetime.d
> 14950
>
> real    0m0.319s
> user    0m0.313s
> sys    0m0.006s
>
> $ time dmd -c ../phobos/std/datetime.d
>
> real    0m0.354s
> user    0m0.318s
> sys    0m0.036s
>
> Yes, I know that "time" is a terrible benchmarking tool, but they're
> fairly close for whatever that's worth.
>

You are measuring lexing speed against compilation speed. A reasonably 
well performing lexer is around one order of magnitude faster on 
std.datetime. Maybe you should profile a little?