std.d.lexer requirements

[Brad Roberts]

To help with performance comparisons I ripped dmd's lexer out and got it building as a few .d files.  It's very crude.
It's got tons of casts (more than the original c++ version).  I attempted no cleanup or any other change than the
minimum I could to get it to build and run.  Obviously there's tons of room for cleanup, but that's not the point...
it's just useful as a baseline.

The branch:
    https://github.com/braddr/phobos/tree/dmd_lexer

The commit with the changes:
    https://github.com/braddr/phobos/commit/040540ef3baa38997b15a56be3e9cd9c4bfa51ab

On my desktop (far from idle, it's running 2 of the auto testers), it consistently takes 0.187s to lex all of the .d
files in phobos.

Later,
Brad

On 8/1/2012 5:10 PM, Walter Bright wrote:
> Given the various proposals for a lexer module for Phobos, I thought I'd share some characteristics it ought to have.
> 
> First of all, it should be suitable for, at a minimum:
> 
> 1. compilers
> 
> 2. syntax highlighting editors
> 
> 3. source code formatters
> 
> 4. html creation
> 
> To that end:
> 
> 1. It should accept as input an input range of UTF8. I feel it is a mistake to templatize it for UTF16 and UTF32. Anyone
> desiring to feed it UTF16 should use an 'adapter' range to convert the input to UTF8. (This is what component
> programming is all about.)
> 
> 2. It should output an input range of tokens
> 
> 3. tokens should be values, not classes
> 
> 4. It should avoid memory allocation as much as possible
> 
> 5. It should read or write any mutable global state outside of its "Lexer"
> instance
> 
> 6. A single "Lexer" instance should be able to serially accept input ranges, sharing and updating one identifier table
> 
> 7. It should accept a callback delegate for errors. That delegate should decide whether to:
>    1. ignore the error (and "Lexer" will try to recover and continue)
>    2. print an error message (and "Lexer" will try to recover and continue)
>    3. throw an exception, "Lexer" is done with that input range
> 
> 8. Lexer should be configurable as to whether it should collect information about comments and ddoc comments or not
> 
> 9. Comments and ddoc comments should be attached to the next following token, they should not themselves be tokens
> 
> 10. High speed matters a lot
> 
> 11. Tokens should have begin/end line/column markers, though most of the time this can be implicitly determined
> 
> 12. It should come with unittests that, using -cov, show 100% coverage
> 
> 
> Basically, I don't want anyone to be motivated to do a separate one after seeing this one.