std.d.lexer requirements

[Ed McCardell]

On 08/02/2012 04:41 AM, Walter Bright wrote:
> On 8/2/2012 1:21 AM, Jonathan M Davis wrote:
>> How would we measure that? dmd's lexer is tied to dmd, so how would we
>> test
>> the speed of only its lexer?
>
> Easy. Just make a special version of dmd that lexes only, and time it.

I made a lexing-only version of dmd at

https://github.com/edmccard/dmd/tree/lexonly

by stripping non-lexer-related code from mars.c, and adding a lexModule 
function that is called instead of Module::parse.. There's no 
benchmarking code yet; it basically just does

   while (token.value != TOKeof) nextToken();

for each D source file passed on the command line.

--Ed