Request for comments: std.d.lexer

[FG]

On 2013-02-04 01:41, Brian Schott wrote:
>> What are you comparing here? How do you time DMD's lexing stage?
>
> A simple hack of module.c that prints out the file's token count and then calls
> exit(0).

Ah, fine. Then it's not that bad. :)
Have you already made it use slices of the whole source,
without copying the string values of tokens?
What kind of optimizations have you made?