A lexical change (a breaking change, but trivial to fix)

H. S. Teoh hsteoh at quickfur.ath.cx
Mon Jul 9 10:07:58 PDT 2012


On Sun, Jul 08, 2012 at 09:59:38AM +0200, Mehrdad wrote:
[...]
> Right, it's trivial to fix with an extra state variable like
> 'consumedtrailingdot'.

This is eventually what I did in my own D lexer.

Well, actually, I kinda blasted an ant with an M16... I had a queue of
backlogged tokens which getNext will return if non-empty, and when I
recognized things like 1..4, I would push 2 or 3 tokens onto the backlog
queue, so no extra state is needed (although the backlog queue itself is
really just another form of extra state).


> The point was, it requires an extra lookahead character, which I was
> trying to avoid (mainly for fun).
> 
> In this case, it doesn't really make a difference in practice -- but
> in general I don't like lookaheads, because depending on future data
> makes it hard for e.g. the user to enter data via the console.

In my case, the extra lookahead only happens when the lexer sees string
prefixes like "4..", which doesn't usually happen at the end of a line.
In all other cases, no lookahead is actually necessary, so except for
the very rare case, entering data via console actually works just fine.


T

-- 
I am a consultant. My job is to make your job redundant. -- Mr Tom


More information about the Digitalmars-d mailing list