Let's stop parser Hell

[Philippe Sigaud]

On Tue, Jul 31, 2012 at 11:20 PM, Jonathan M Davis <jmdavisProg at gmx.com> wrote:
> On Tuesday, July 31, 2012 23:10:37 Philippe Sigaud wrote:
>> Having std.lexer in Phobos would be quite good. With a pre-compiled lexer
>> for D.
>
> I'm actually quite far along with one now - one which is specifically written
> and optimized for lexing D. I'll probably be done with it not too long after
> the 2.060 release (though we'll see).

That was quick! Cool!

>Writing it has been going surprisingly
> quickly actually, and I've already found some bugs in the spec as a result
> (some of which have been fixed, some of which I still need to create pull
> requests for). So, regardless of what happens with my lexer, at least the spec
> will be more accurate.

Could you please describe the kind of token it produces?
Can it build a symbol table?
Does it recognize all kind of strings (including q{ } ones)?
How does it deal with comments, particularly nested ones?
Does it automatically discard whitespaces or produce them as a token?
I'd favor this approach, if only because wrapping the lexer in a
filter!noWS(tokenRange) is easy.
Does it produce a lazy range btw?