std.d.lexer requirements

[dennis luehring]

Am 03.08.2012 09:56, schrieb Ed McCardell:
> On 08/02/2012 04:41 AM, Walter Bright wrote:
>> On 8/2/2012 1:21 AM, Jonathan M Davis wrote:
>>> How would we measure that? dmd's lexer is tied to dmd, so how would we
>>> test
>>> the speed of only its lexer?
>>
>> Easy. Just make a special version of dmd that lexes only, and time it.
>
> I made a lexing-only version of dmd at
>
> https://github.com/edmccard/dmd/tree/lexonly
>
> by stripping non-lexer-related code from mars.c, and adding a lexModule
> function that is called instead of Module::parse.. There's no
> benchmarking code yet; it basically just does
>
>     while (token.value != TOKeof) nextToken();
>
> for each D source file passed on the command line.
>
> --Ed
>

Walter mentioned an easier way - without "forking" dmd - should be 
better an integral part of the ongoing development maybe under a tools 
or benchmark section?

news://news.digitalmars.com:119/jvfv14$1ri0$1@digitalmars.com
->Look in doc.c at highlightCode2() for how to call the lexer by itself.