std.d.lexer: pre-voting review / discussion

[Manfred Nowak]

Jonathan M Davis wrote:

> You have to look ahead to figure out whether it's .. or a floating
> point literal.

This lookahead is introduced by using a petty grammar.

Please reconsider, that lexing searches for the leftmost longest pattern 
of the rest of the input. This means that introducing a pattern like
 `<int>\.\.'    	return TokenType.INTDOTDOT;
would eliminate the lookahead in the lexer.

In the parser an additional rule then has to be added:
  <range> ::= INT DOTDOT INT
           |  INTDOTDOT INT

-manfred