std.d.lexer requirements

[Jonathan M Davis]

On Tuesday, August 07, 2012 14:30:51 Walter Bright wrote:
> On 8/7/2012 2:14 PM, Jonathan M Davis wrote:
> > I expect that the configuration stuff is going to have to be adjusted
> > after I'm done, since I'm not sure that it's entirely clear what's worth
> > configuring or not.
> 
> "When in doubt, leave it out."
> 
> If experience later shows it is really needed, it is easier to add it in
> than to have a blizzard of useless configuration options that cannot be
> removed.

On the whole, I agree, and I'm definitely leaving some of that stuff out at this 
point. A simplified version of what I have at the moment is

struct LexConfig { /+ configuration options +/}

struct Lexer(alias config)
 if(is(typeof(config) == LexConfig))
{
 auto lex(R)(R range, SourcePos pos = SourcePos.init) {}
}

The main problem that I have is that if I end up with any delegates which need 
to take the range being lexed, then either I can't put them in the Lexer like 
I am with the error handler delegate (forcing you to pass it to the lex 
function every time that you lex a new range), or I have to templatize Lexer 
on the range type, which is also undesirable.

If it turns out later that we want to add such delegates and we didn't 
templatize Lexer on the range type, then we'll be forced to make it so that 
they're arguments to the lex function. But templatizing the Lexer on the range 
type is ugly enough that it's not really something that I want to do if I can 
reasonbly avoid it. Neither solution appeals to me. If we don't ever need to 
add such delegates, then it doesn't really matter, but I'd prefer to have a 
solution which is appropriately flexible if we _do_ need to add that sort of 
thing later.

I guess that I'll have to keep thinking about it to see if I can come up with 
a reasonable way to do it without hampering the API by either making it ugly 
through unneeded flexibility or by making it too simple to be able to 
reasonably add more functionality later.

- Jonathan M Davis