Request for comments: std.d.lexer

[Johannes Pfau]

Am Sun, 27 Jan 2013 12:38:33 +0100
schrieb Timon Gehr <timon.gehr at gmx.ch>:

> On 01/27/2013 11:42 AM, Brian Schott wrote:
> > On Sunday, 27 January 2013 at 10:17:48 UTC, Philippe Sigaud wrote:
> >> * Having a range interface is good. Any reason why you made
> >> byToken a class and not a struct? Most (like, 99%) of range in
> >> Phobos are structs. Do you need reference semantics?
> >
> > It implements the InputRange interface from std.range so that users
> > have a choice of using template constraints or the OO model in
> > their code. ...
> 
> The lexer range must be a struct.
> 
> > ...
> >> * A rough estimate of number of tokens/s would be good (I know
> >> it'll vary). Walter seems to think if a lexer is not able to vomit
> >> thousands of tokens a seconds, then it's not good. On a related
> >> note, does your lexer have any problem with 10k+-lines files?
> >
> > $ time dscanner --sloc ../phobos/std/datetime.d
> > 14950
> >
> > real    0m0.319s
> > user    0m0.313s
> > sys    0m0.006s
> >
> > $ time dmd -c ../phobos/std/datetime.d
> >
> > real    0m0.354s
> > user    0m0.318s
> > sys    0m0.036s
> >
> > Yes, I know that "time" is a terrible benchmarking tool, but they're
> > fairly close for whatever that's worth.
> >
> 
> You are measuring lexing speed against compilation speed. A
> reasonably well performing lexer is around one order of magnitude
> faster on std.datetime. Maybe you should profile a little?
> 

Profiling is always a good idea, but to be fair: His dmd was probably
compiled with gcc and -O2 if it's a normal release build.
So to compare that he should use gdc, -O2 -release -fno-bounds-check
and probably more flags to compile the d code.