The Case Against Autodecode

[Joakim via Digitalmars-d]

On Monday, 30 May 2016 at 17:35:36 UTC, Chris wrote:
> On Monday, 30 May 2016 at 16:03:03 UTC, Marco Leise wrote:
>
>> *** http://site.icu-project.org/home#TOC-What-is-ICU-
>
> I was actually talking about ICU with a colleague today. Could 
> it be that Unicode itself is broken? I've often heard criticism 
> of Unicode but never looked into it.

Part of it is the complexity of written language, part of it is 
bad technical decisions.  Building the default string type in D 
around the horrible UTF-8 encoding was a fundamental mistake, 
both in terms of efficiency and complexity.  I noted this in one 
of my first threads in this forum, and as Andrei said at the 
time, nobody agreed with me, with a lot of hand-waving about how 
efficiency wasn't an issue or that UTF-8 arrays were fine.  
Fast-forward years later and exactly the issues I raised are now 
causing pain.

UTF-8 is an antiquated hack that needs to be eradicated.  It 
forces all other languages than English to be twice as long, for 
no good reason, have fun with that when you're downloading text 
on a 2G connection in the developing world.  It is unnecessarily 
inefficient, which is precisely why auto-decoding is a problem.  
It is only a matter of time till UTF-8 is ditched.

D devs should lead the way in getting rid of the UTF-8 encoding, 
not bickering about how to make it more palatable.  I suggested a 
single-byte encoding for most languages, with double-byte for the 
ones which wouldn't fit in a byte.  Use some kind of header or 
other metadata to combine strings of different languages, _rather 
than encoding the language into every character!_

The common string-handling use case, by far, is strings with only 
one language, with a distant second some substrings in a second 
language, yet here we are putting the overhead into every 
character to allow inserting characters from an arbitrary 
language!  This is madness.

Yes, the complexity of diacritics and combining characters will 
remain, but that is complexity that is inherent to the variety of 
written language.  UTF-8 is not: it is just a bad technical 
decision, likely chosen for ASCII compatibility and some 
misguided notion that being able to combine arbitrary language 
strings with no other metadata was worthwhile.  It is not.