std.d.lexer requirements

[Philippe Sigaud]

On Tue, Aug 7, 2012 at 12:06 PM, Jonathan M Davis <jmdavisProg at gmx.com> wrote:

> Regardless, I was asked to keep that option in there by at least one person
> (Philippe Sigaud IIRC), which is why I didn't just switch over to the delegate
> entirely.

IIRC, I was not the only one, as people here interested in coding an
IDE asked for it too. A lexer is useful for more than 'just' parsing D
afterwards: an IDE could easily color tokens according to their type
and an error token is just was is needed to highlight errors.

Also, what I proposed was a *static* decision: with SkipErrors { no,
yes }. With a static if inside its guts, the lexer could change its
behavior accordingly. Make skipError.yes the default and Walter get
its speed. It's just that an IDE or another parser could use

auto lex = std.lexer.Lexer!(SkipError.no)(input);


Walter, with all due respect, you sometimes give the impression to
forget we are talking about D and go back to deeply entrenched C-isms.
Compile-time decisions can be used to avoid any overhead as long as
you have a clear idea of what the two code paths should look like.

And, as Christophe said, ranges are a powerful API. In another thread
Simen and me did some comparison between C-like code and code using
only ranges upon ranges upon ranges. A (limited!) difference in speed
appeared only for very long calculations.