"tuple unpacking" with zip?

[Jonathan M Davis via Digitalmars-d]

On Wednesday, 21 October 2015 at 14:13:43 UTC, Shriramana Sharma 
wrote:
> John Colvin wrote:
>
>>> But this is false, no? Since ElementType!string is char and 
>>> not dchar?
>> 
>> No. char[], wchar[] and dchar[] all have ElementType dchar. 
>> Strings are special for ranges. It's a bad mistake, but it is 
>> what it is and apparently won't be changed.
>
> Why is it a mistake? That seems a very sane thing, although 
> somewhat quirky. Since ElementType is a Range primitive, and 
> apparently iterating through a string as a range will produce 
> each semantically meaningful Unicode character rather than each 
> UTF-8 or UTF-16 codepoint, it does make sense to do this.

LOL. This could open up a huge discussion if you're not careful. 
A code point is not necessarily a full character. Operating on 
individual code units is generally wrong, because you frequently 
need multiple code units to get a full code point. Similarly, to 
get a full character - what's called a grapheme - you sometimes 
need multiple code points. To make matters even worse, the same 
grapheme can often be represented by different combinations of 
code points (e.g. an accented e can be represented as a single 
code point or it could be represented with the code point for e 
and the code point for the accent - and depending on the unicode 
normalization form being used, the order of those code points 
could differ).

So, operating at the code point level does _not_ actually make 
your program correct. It gets you closer, but you're still 
operating on pieces of characters - and it's arguably more 
pernicious, because more of the common characters "just work" 
while still not ensuring that all of them work, making it harder 
to catch when you screw it up.

However, operating at the grapheme level is incredibly expensive. 
In fact, operating at the code point level is often unnecessarily 
expensive. So, if you care about efficiency, you want to be 
operating at the code unit level as much as possible. And because 
most string code doesn't actually need to operate on individual 
characters, operating at the code unit level is actually 
frequently plenty (especially if your strings have had their code 
points normalized so that the same characters will always result 
in the same sequence of code units).

So, what we have with Phobos is neither fast nor correct. It's 
constantly decoding code points when it's completely unnecessary 
(Phobos has to special case its algorithms for strings all over 
the place to avoid unnecessary decoding). And because ranges deal 
at the code point level by default, they're not correct. Really, 
code should either be operating at the code unit level or the 
grapheme level. You're getting the worst of both worlds when 
operating at the code point level.

Rather, what's really needed is for the programmer to know enough 
about Unicode to know when they should be operating on code 
points or graphemes (or occasionally code points) and then 
explicitly do that - which is why we have 
std.utf.byCodeUnit/byChar/byWchar/byDchar and 
std.uni.byCodePoint/byGrapheme. But as soon as you use those, you 
lose out on the specializations that operate on arrays as well as 
any other code that specifically operates on arrays - even when 
you just want to operate on a char[] as a range of char.

The reality of the matter is that _most_ algorithms would work 
just fine with treating char[] as a range of char so long as they 
do explicit decoding when necessary (and it often wouldn't be 
necessary), but instead, we're constantly autodecoding, because 
that's what front and popFront do for arrays of char or wchar.

When Andrei came up with the current scheme, he didn't know about 
graphemes. He thought that code points were always full 
characters. And if that were the case, the way Phobos works would 
make sense. It might be slower by default, but it would be 
correct, and you could special-case on strings to operate on them 
more efficiently if you needed the extra efficiency. However, 
because code points are _not_ necessarily full characters, we're 
taking an efficiency hit without getting full correctness. 
Instead, we're getting the illusion of correctness. It's like how 
Andrei explained in TDPL that UTF-16 is worse than UTF-8, because 
it's harder to catch when you screw up and chop a character in 
half. Only, it turns out that that applies to UTF-32 as well.

- Jonathan M Davis