Request for comments: std.d.lexer
Jacob Carlborg
doob at me.com
Thu Jan 31 04:15:40 PST 2013
On 2013-01-30 10:49, Brian Schott wrote:
> Results:
>
> $ avgtime -q -r 200 ./dscanner --tokenCount ../phobos/std/datetime.d
>
> ------------------------
> Total time (ms): 13861.8
> Repetitions : 200
> Sample mode : 69 (90 ocurrences)
> Median time : 69.0745
> Avg time : 69.3088
> Std dev. : 0.670203
> Minimum : 68.613
> Maximum : 72.635
> 95% conf.int. : [67.9952, 70.6223] e = 1.31357
> 99% conf.int. : [67.5824, 71.0351] e = 1.72633
> EstimatedAvg95%: [69.2159, 69.4016] e = 0.0928836
> EstimatedAvg99%: [69.1867, 69.4308] e = 0.12207
>
> If my math is right, that means it's getting 4.9 million tokens/second
> now. According to Valgrind the only way to really improve things now is
> to require that the input to the lexer support slicing. (Remember the
> secret of Tango's XML parser...) The bottleneck is now on the calls to
> .idup to construct the token strings from slices of the buffer.
How many tokens would that be in total?
--
/Jacob Carlborg
More information about the Digitalmars-d
mailing list