Request for comments: std.d.lexer

[Jonathan M Davis]

On Saturday, February 09, 2013 08:19:33 deadalnix wrote:
> So, std.utf.decodeFront pop or not the utf character. And in case
> it does not, you ends up having to do extra hanky panky (and
> duplicate logic in std.utf to know if it does pop or not, which
> is likely to be very bug prone).

Well, with the pull request that I have, it'll always pop for input ranges and 
never for anything else. I'm disinclined to have it pop off elements at all, 
because I'd like it to function as much like decode as possible, but there's 
no choice with pure input ranges, and it may be better to just make it always 
pop the elements off. I'll have to think about it. I hate pure input ranges in 
general. They're just so frustrating to deal with, and I believe that you can 
always have a forward range if you're willing to put a bit more effort into it.

The lexer that I'm writing doesn't even support pure input ranges, because it 
has to be able to save to do what it does, and the only reason that 
decodeFront exists is because I wrote it to support that use case. And needing 
to operate on ranges of code units is likely to be quite rare, and the 
function is quite new, so I don't expect that much code is affected by any bugs 
in decodeFront or that much would be broken were it to be changed.

- Jonathan M Davis