Range of chars (narrow string ranges)

[Chris via Digitalmars-d]

On Wednesday, 29 April 2015 at 15:13:15 UTC, Jonathan M Davis 
wrote:
> On Wednesday, 29 April 2015 at 10:02:09 UTC, Chris wrote:
>> This sounds like a good starting point for a transition plan. 
>> One important thing, though, would be to do some benchmarking 
>> with and without autodecoding, to see if it really boosts 
>> performance in a way that would justify the transition.
>
> Well, personally, I think that it's worth it even if the 
> performance is identical (and it's a guarantee that it's going 
> to be better without autodecoding - it's just a question of how 
> much better - since it's going to have less work to do without 
> autodecoding). Simply operating at the code point level like we 
> do now is the worst of all worlds in terms of flexibility and 
> correctness. As long as the Unicode is normalized, operating at 
> the code unit level is the most efficient, and decoding is 
> often unnecessary for correctness, and if you need to decode, 
> then you really need to go up to the grapheme level in order to 
> be operating on the full character, meaning that operating on 
> code points really has the same problems as operating on code 
> units as far as correctness goes. So, it's less performant 
> without actually being correct. It just gives the illusion of 
> correctness.
>
> By treating strings as ranges of code units, you don't take a 
> performance hit when you don't need to, and it forces you to 
> actually consider something like byDchar or byGrapheme if you 
> want to operate on full, Unicode characters. It's similar to 
> how operating on UTF-16 code units as if they were characters 
> (as Java and C# generally do) frequently gives the incorrect 
> impression that you're handling Unicode correctly, because you 
> have to work harder at coming up with characters that can't fit 
> in a single code unit, whereas with UTF-8, anything but ASCII 
> is screwed if you treat code units as code points. Treating 
> code points as if they were full characters like we're doing 
> now in Phobos with ranges just makes it that much harder to 
> notice that you're not handling Unicode correctly.
>
> Also, treating strings as ranges of code units makes it so that 
> they're not so special and actually are treated like every 
> other type of array, which eliminates a lot of the special 
> casing that we're forced to do right now, and it eliminates all 
> of the confusion that folks keep running into when string 
> doesn't work with many functions, because it's not a 
> random-access range or doesn't have length, or because the 
> resulting range isn't the same type (copy would be a prime 
> example of a function that doesn't work with char[] when it 
> should). By leaving in autodecoding, we're basically leaving in 
> technical debt in D permanently. We'll forever have to be 
> explaining it to folks and forever have to be working around it 
> in order to achieve either performance or correctness.
>
> What we have now isn't performant, correct, or flexible, and 
> we'll be forever paying for that if we don't get rid of 
> autodecoding.
>
> I don't criticize Andrei in the least for coming up with it, 
> since if you don't take graphemes into account (and he didn't 
> know about them at the time), it seems like a great idea and 
> allows us to be correct by default and performant if we put 
> some effort into, but after having seen how it's worked out, 
> how much code has to be special-cased, how much confusion there 
> is over it, and how it's not actually correct anyway, I think 
> that it's quite clear that autodecoding was a mistake. And at 
> this point, it's mainly a question of how we can get rid of it 
> without being too disruptive and whether we can convince Andrei 
> that it makes sense to make the change, since he seems to still 
> think that autodecoding is fine in spite of the fact that it's 
> neither performant nor correct.
>
> It may be that the decision will be that it's too disruptive to 
> remove autodecoding, but I think that that's really a question 
> of whether we can find a way to do it that doesn't break tons 
> of code rather than whether it's worth the performance or 
> correctness gain.
>
> - Jonathan M Davis

Ok, I see. Well, if we don't want to repeat C++'s mistakes, we 
should fix it before it's too late. Since I'm dealing a lot with 
strings (non ASCII) and depend on Unicode (and correctness!), I 
would be more than happy to test any changes to Phobos with my 
programs to see if it screws up anything.