std.d.lexer requirements

[Jonathan M Davis]

On Tuesday, August 07, 2012 08:00:24 Christophe Travert wrote:
> Walter Bright , dans le message (digitalmars.D:174360), a écrit :

> > That's why I suggested supplying a callback delegate to decide what to do
> > with errors (ignore, throw exception, or quit) and have the delegate
> > itself do that. That way, there is no customization of the Lexer
> > required.
> 
> It may be easier to take into accound few cases (return error token and
> throwing is enough, so that is a basic static if), than to define a way
> to integrate a delegate (what should be the delegate's signature, what
> value to return to query for stopping, how to provide ways to recovers,
> etc).

For the moment at least, I'm doing this

bool delegate(string errorMsg, SourcePos pos) errorHandler;

where SourcePos is a struct which holds the line and col of the place in the 
source code where the bad token starts. It it returns true, the token is 
skipped. If it returns false, an exception is thrown - and of course the 
delegate can throw its own exception if it wants to.

But you can also configure the lexer to return an error token instead of using 
the delegate if that's what you prefer. But Walter is right in that if you 
have to check every token for whether it's an error, that will incur overhead. 
So, depending on your use case, that could be unacceptable.

- Jonathan M Davis