Request for comments: std.d.lexer

[deadalnix]

On Wednesday, 6 February 2013 at 03:51:33 UTC, Andrei 
Alexandrescu wrote:
> I think it would be reasonable for a lexer to require a range 
> of ubyte as input, and carry its own decoding. In the first 
> approximation it may even require a random-access range of 
> ubyte.
>

Playing around that, I discovered a bug in std.utf : slice and 
other range are not threated the same way by decodeFront, which 
is rather problematic. Jonathan also hit that bug : 
http://d.puremagic.com/issues/show_bug.cgi?id=9456

That make the lexer hard to write with a consistent behavior for 
InputRanges.

The bug probably exists in everything that rely on decodeFront at 
some point.