std.d.lexer requirements

[Jonathan M Davis]

On Tuesday, August 07, 2012 02:54:42 Walter Bright wrote:
> On 8/7/2012 1:14 AM, Jonathan M Davis wrote:
> > But you can also configure the lexer to return an error token instead of
> > using the delegate if that's what you prefer. But Walter is right in that
> > if you have to check every token for whether it's an error, that will
> > incur overhead. So, depending on your use case, that could be
> > unacceptable.
> 
> It's not just overhead - it's just plain ugly to constantly check for error
> tokens. It's also tedious and error prone to insert those checks.
> 
> I don't see any advantage to it.

It's easier to see where in the range of tokens the errors occur. A delegate 
is disconnected from the point where the range is being consumed, whereas if 
tokens are used for errors, then the function consuming the range can see 
exactly where in the range of tokens the error is (and potentially handle it 
differently based on that information).

Regardless, I was asked to keep that option in there by at least one person 
(Philippe Sigaud IIRC), which is why I didn't just switch over to the delegate 
entirely.

- Jonathan M Davis