std.d.lexer : voting thread

[Joseph Rushton Wakeling]

On 06/10/13 18:07, Andrei Alexandrescu wrote:
> I'm working on related code, and got all the way there in one day (Friday) with
> a C++ tokenizer for linting purposes (doesn't open #includes or expand #defines
> etc; it wasn't meant to).
>
> The core generated fragment that does the matching is at https://dpaste.de/GZY3.
>
> The surrounding switch statement (also in library code) handles whitespace and
> line counting. The client code needs to handle by hand things like parsing
> numbers (note how the matcher stops upon the first digit), identifiers, comments
> (matcher stops upon detecting "//" or "/*") etc. Such things can be achieved
> with hand-written code (as I do), other similar tokenizers, DFAs, etc. The point
> is that the core loop that looks at every character looking for a lexeme is fast.

What I'm getting at is that I'd be prepared to give a vote "no to std, yes to 
etc" for Brian's d.lexer, _if_ I was reasonably certain that we'd see an 
alternative lexer module submitted to Phobos within the next month :-)