Looking for champion - std.lang.d.lex

[Nick Sabalausky]

"Walter Bright" <newshound2 at digitalmars.com> wrote in message 
news:ia5j41$2bnk$1 at digitalmars.com...
>
> To specifically answer your question, yes, in the lexers I make, you know 
> you're parsing a string, so you process it as you parse it.
>
>...
>
> I don't know. I'd have to study the issue for a while. I suggest taking a 
> look at dmd's lexer and compare. I'm not sure what Spirit's approach to 
> this is.
>

I've taken a deeper look at Spirit's docs:

In the older Spirit 1.x, the lexing is handled as part of the parsing. The 
structure of it definitely suggests it should be easy for it to do all 
token-conversion right as the string is being lexed, although I couldn't 
tell whether or not it actually did so (I'd have to look at the source). 
But, since Spirit 1.x doesn't handle lexing separately from parsing, I 
*think* backtracking (it *is* a backtracking parser) results in re-lexing, 
even for terminals that never get special processing, such as keywords (But 
I'm not completely certain because I don't have much experience with LL).

In Spirit 2.x, standard usage involves having the lexing separate from 
parsing. I didn't see anything at all in the docs for Spirit 2.x that seemed 
to suggest even the possibility of it processing tokens as they're lexed. 
However, Spirit is designed with heavy policy-based customizability in mind, 
so such a thing might still possible in Spirit 2.x...But if so, it's 
definitely an advanced feature (or just really poorly documented).

I have thought of another way to get such an ability into Goldie, and it 
would be very easy-to-use, but it would also be a fairly non-trivial to 
implement. And really, I'm starting to question again how important it would 
*really* be, at least initially. When I think of typical code, usually only 
a small amount of it is made up of the the sorts of terminals that would 
need extra processing.

I have to admit, I still have no idea whether or not it would be worth it to 
get Goldie into Phobos. Maybe, maybe not, I dunno. I think popular opinion 
would probably be the best gauge of that. It seems like we're the only ones 
still in this thread, though...maybe that's a bad sign? ;) I do still think 
that if your primary goal is to provide parsing of D code through Phobos, 
then adapting DDMD would be the best best. Goldie would be more appropriate 
if customized lexing/parsing is the goal.