Of possible interest: fast UTF8 validation

[Patrick Schluter]

On Thursday, 17 May 2018 at 05:01:54 UTC, Joakim wrote:
> On Wednesday, 16 May 2018 at 20:11:35 UTC, Andrei Alexandrescu 
> wrote:
>> On 5/16/18 1:18 PM, Joakim wrote:
>>> On Wednesday, 16 May 2018 at 16:48:28 UTC, Dmitry Olshansky 
>>> wrote:
>>>> On Wednesday, 16 May 2018 at 15:48:09 UTC, Joakim wrote:
>>>>> On Wednesday, 16 May 2018 at 11:18:54 UTC, Andrei 
>>>>> Alexandrescu wrote:
>>>>>> https://www.reddit.com/r/programming/comments/8js69n/validating_utf8_strings_using_as_little_as_07/
>>>>>>
>>>>>
>>>>> Sigh, this reminds me of the old quote about people 
>>>>> spending a bunch of time making more efficient what 
>>>>> shouldn't be done at all.
>>>>
>>>> Validating UTF-8 is super common, most text protocols and 
>>>> files these days would use it, other would have an option to 
>>>> do so.
>>>>
>>>> I’d like our validateUtf to be fast, since right now we do 
>>>> validation every time we decode string. And THAT is slow. 
>>>> Trying to not validate on decode means most things should be 
>>>> validated on input...
>>> 
>>> I think you know what I'm referring to, which is that UTF-8 
>>> is a badly designed format, not that input validation 
>>> shouldn't be done.
>>
>> I find this an interesting minority opinion, at least from the 
>> perspective of the circles I frequent, where UTF8 is 
>> unanimously heralded as a great design. Only a couple of weeks 
>> ago I saw Dylan Beattie give a very entertaining talk on 
>> exactly this topic: 
>> https://dotnext-piter.ru/en/2018/spb/talks/2rioyakmuakcak0euk0ww8/
>
> Thanks for the link, skipped to the part about text encodings, 
> should be fun to read the rest later.
>
>> If you could share some details on why you think UTF8 is badly 
>> designed and how you believe it could be/have been better, I'd 
>> be in your debt!
>
> Unicode was a standardization of all the existing code pages 
> and then added these new transfer formats, but I have long 
> thought that they'd have been better off going with a 
> header-based format that kept most languages in a single-byte 
> scheme,

This is not practical, sorry. What happens when your message 
loses the header? Exactly, the rest of the message is garbled. 
That's exactly what happened with code page based texts when you 
don't know in which code page it is encoded. It has the 
supplemental inconvenience that mixing languages becomes 
impossible or at least very cumbersome.
UTF-8 has several properties that are difficult to have with 
other schemes.
- It is state-less, means any byte in a stream always means the 
same thing. Its meaning  does not depend on external or a 
previous byte.
- It can mix any language in the same stream without acrobatics 
and if one thinks that mixing languages doesn't happen often 
should get his head extracted from his rear, because it is very 
common (check wikipedia's front page for example).
- The multi byte nature of other alphabets is not as bad as 
people think because texts in computer do not live on their own, 
meaning that they are generally embedded inside file formats, 
which more often than not are extremely bloated (xml, html, 
xliff, akoma ntoso, rtf etc.). The few bytes more in the text do 
not weigh that much.

I'm in charge at the European Commission of the biggest 
translation memory in the world. It handles currently 30 
languages and without UTF-8 and UTF-16 it would be unmanageable. 
I still remember when I started there in 2002 when we handled 
only 11 languages of which only 1 was of another alphabet 
(Greek). Everything was based on RTF with codepages and it was a 
braindead mess. My first job in 2003 was to extend the system to 
handle the 8 newcomer languages and with ASCII based encodings it 
was completely unmanageable because every document processed 
mixes languages and alphabets freely (addresses and names are 
often written in their original form for instance).
2 years ago we implemented also support for Chinese. The nice 
thing was that we didn't have to change much to do that thanks to 
Unicode. The second surprise was with the file sizes, Chinese 
documents were generally smaller than their European 
counterparts. Yes CJK requires 3 bytes for each ideogram, but 
generally 1 ideogram replaces many letters. The ideogram 亿 
replaces "One hundred million" for example, which of them take 
more bytes? So if CJK indeed requires more bytes to encode, it is 
firstly because they NEED many more bits in the first place 
(there are around 30000 CJK codepoints in the BMP alone, add to 
it the 60000 that are in the SIP and we have a need of 17 bits 
only to encode them.


> as they mostly were except for obviously the Asian CJK 
> languages. That way, you optimize for the common string, ie one 
> that contains a single language or at least no CJK, rather than 
> pessimizing every non-ASCII language by doubling its character 
> width, as UTF-8 does. This UTF-8 issue is one of the first 
> topics I raised in this forum, but as you noted at the time 
> nobody agreed and I don't want to dredge that all up again.
>
> I have been researching this a bit since then, and the stated 
> goals for UTF-8 at inception were that it _could not overlap 
> with ASCII anywhere for other languages_, to avoid issues with 
> legacy software wrongly processing other languages as ASCII, 
> and to allow seeking from an arbitrary location within a byte 
> stream:
>
> https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt
>
> I have no dispute with these priorities at the time, as they 
> were optimizing for the institutional and tech realities of 
> 1992 as Dylan also notes, and UTF-8 is actually a nice hack 
> given those constraints. What I question is that those 
> priorities are at all relevant today, when billions of 
> smartphone users are regularly not using ASCII, and these tech 
> companies are the largest private organizations on the planet, 
> ie they have the resources to design a new transfer format. I 
> see basically no relevance for the streaming requirement today, 
> as I noted in this forum years ago, but I can see why it might 
> have been considered important in the early '90s, before 
> packet-based networking protocols had won.
>
> I think a header-based scheme would be _much_ better today and 
> the reason I know Dmitry knows that is that I have discussed 
> privately with him over email that I plan to prototype a format 
> like that in D. Even if UTF-8 is already fairly widespread, 
> something like that could be useful as a better intermediate 
> format for string processing, and maybe someday could replace 
> UTF-8 too.