std.d.lexer requirements

Christophe Travert travert at phare.normalesup.org
Tue Aug 7 04:43:46 PDT 2012


Walter Bright , dans le message (digitalmars.D:174394), a écrit :
> On 8/7/2012 1:14 AM, Jonathan M Davis wrote:
>> But you can also configure the lexer to return an error token instead of using
>> the delegate if that's what you prefer. But Walter is right in that if you
>> have to check every token for whether it's an error, that will incur overhead.
>> So, depending on your use case, that could be unacceptable.
> 
> It's not just overhead - it's just plain ugly to constantly check for error 
> tokens. It's also tedious and error prone to insert those checks.

It's not necessarily ugly, because of the powerful range design. You can 
branch the lexer to a range adapter that just ignore error tokens, or 
throw when it meats an error token.

For example, just use:
auto tokens = data.lexer.throwOnErrorToken;

I don't think this is more ugly than:
auto tokens = data.lexer!(complex signature) { throw LexException; };

But yes, there is overhead, so I understand returning error tokens is 
not satisfactory for everyone.

> I don't see any advantage to it.

Storing the error somewhere can be of use.
For example, you may want to lex a whole file into an array of tokens, 
and then deal with you errors as you analyse the array of tokens. 
Of course, you can alway make a delegate to store the error somewhere, 
but it is easier if this somewhere is in your token pile.

What I don't see any advantage is using a delegate that can only return 
or throw. A policy makes the job:
auto tokens = data.lexer!ExceptionPolicy.throwException;
That's clean too.

If you want the delegate to be of any use, then it must have 
data to process. That's why I said we have to worry about the 
signature of the delegate.

-- 
Christophe



More information about the Digitalmars-d mailing list