Request for comments: std.d.lexer

[deadalnix]

On Friday, 8 February 2013 at 20:54:32 UTC, Jonathan M Davis 
wrote:
> On Friday, February 08, 2013 23:50:13 Dmitry Olshansky wrote:
>> Complication - yes, slight performance cost is what I doubt it 
>> in RA
>> case. Seems like a compiler/optimizer issue.
>
> The main problem isn't RA ranges. It's quite probable that the 
> optimizer takes
> care of any minor additional overhead that's incurred there. 
> The issue is
> really pure forward ranges, because you're forced to count the 
> code units as
> they're processed (or even save the range as you go along, 
> given how expensive
> it would be to go find the index even if you have it). But we 
> really have no
> way of knowing how bad it is without hard data. It would 
> certainly be trivial
> to end up doing other stuff in the implementation which cost 
> far more.
>

For pure InputRanges, that is pretty bad as the decoding of UTF 
chars basically have to be done 2 times, and each codepoint 
popped individually, or the lexer have to embed its own homebrew 
version of std.utf .