DCT: D compiler as a collection of libraries

[Roman D. Boiko]

On Monday, 14 May 2012 at 19:04:20 UTC, Tove wrote:
> On Monday, 14 May 2012 at 16:58:42 UTC, Roman D. Boiko wrote:
>>> You are over engineering the whole stuff.
>> I'm trying to solve this and other tradeoffs. I'd like to
>> simplify but satisfy my design goals.
>
> What if there were two different lex:er modes... with different 
> struct:s.
>
> 1. For an IDE with on the fly lexing:
>   Assumption, the error rate is high.(need to keep much info)
>
> 2. For the compiler
> Assumption, the error rate is non existent, and if there is an 
> error it really doesn't matter if it's slow.
>
> So... when choosing the "compiler mode"... and there actually 
> is an error, then just lex it again, to produce a pretty error 
> message ;)
>
> try
> {
>   lex(mode.compiler);
> }
> catch
> {
>   lex(mode.ide); // calculates column etc. what ever info it 
> needs.
> }
So far it doesn't seem expensive to tolerate errors and proceed.
The only thing I miss is some sort of specification when to stop
including characters into token spelling and start a new token. I
don't think I'll use backtracking for that in the nearest future.
If I did, I would really separate part of lexer and provide two
implementations for that part. Given this, accepting errors and
moving on simply requires some finite set of rules about
boundaries of invalid tokens. I also think structural code
editing concepts will help here, but I didn't do any research on
this topic yet.

The problem with multiple lexer implementations is that it might
become much more difficult to maintain them.