Request for comments: std.d.lexer

Jacob Carlborg doob at me.com
Thu Jan 31 04:15:40 PST 2013


On 2013-01-30 10:49, Brian Schott wrote:

> Results:
>
> $ avgtime -q -r 200 ./dscanner --tokenCount ../phobos/std/datetime.d
>
> ------------------------
> Total time (ms): 13861.8
> Repetitions    : 200
> Sample mode    : 69 (90 ocurrences)
> Median time    : 69.0745
> Avg time       : 69.3088
> Std dev.       : 0.670203
> Minimum        : 68.613
> Maximum        : 72.635
> 95% conf.int.  : [67.9952, 70.6223]  e = 1.31357
> 99% conf.int.  : [67.5824, 71.0351]  e = 1.72633
> EstimatedAvg95%: [69.2159, 69.4016]  e = 0.0928836
> EstimatedAvg99%: [69.1867, 69.4308]  e = 0.12207
>
> If my math is right, that means it's getting 4.9 million tokens/second
> now. According to Valgrind the only way to really improve things now is
> to require that the input to the lexer support slicing. (Remember the
> secret of Tango's XML parser...) The bottleneck is now on the calls to
> .idup to construct the token strings from slices of the buffer.

How many tokens would that be in total?

-- 
/Jacob Carlborg


More information about the Digitalmars-d mailing list