std.d.lexer requirements

[deadalnix]

>> 7. It should accept a callback delegate for errors. That delegate should
>> decide whether to:
>>      1. ignore the error (and "Lexer" will try to recover and continue)
>>      2. print an error message (and "Lexer" will try to recover and continue)
>> 3. throw an exception, "Lexer" is done with that input range
>
> I'm currently treating errors as tokens. It then becomes easy for the code
> using the lexer to just ignore the errors, to process them immediately, or to
> put off dealing with them until the lexing is complete. So, the code using the
> lexer can handle errors however and whenever it likes without having to worry
> about delegates or exceptions. Since tokens are lexed lazily, the fact that an
> error is reported as a token doesn't require that the lexing continue, but it
> also makes it _easy_ to continue lexing, ignoring or saving the error. I'm
> inclined to think that that's a superior approach to using delegates and
> exceptions.
>

Really nice idea. It is still easy to wrap the Range in another Range 
that process errors in a custom way.