First Impressions!

[Patrick Schluter]

On Thursday, 30 November 2017 at 19:37:47 UTC, Steven 
Schveighoffer wrote:
> On 11/30/17 1:20 PM, Patrick Schluter wrote:
>> On Thursday, 30 November 2017 at 17:40:08 UTC, Jonathan M 
>> Davis wrote:
>>> English and thus don't as easily hit the cases where their 
>>> code is wrong. For better or worse, UTF-16 hides it better 
>>> than UTF-8, but the problem exists in both.
>>>
>> 
>> To give just an example of what can go wrong with UTF-16. 
>> Reading a file in UTF-16 and converting it tosomething else 
>> like UTF-8 or UTF-32. Reading block by block and hitting 
>> exactly a SMP codepoint at the buffer limit, high surrogate at 
>> the end of the first buffer, low surrogate at the start of the 
>> next. If you don't think about it => 2 invalid characters 
>> instead of your nice poop 💩 emoji character (emojis are in the 
>> SMP and they are more and more frequent).
>
> iopipe handles this: 
> http://schveiguy.github.io/iopipe/iopipe/textpipe/ensureDecodeable.html
>

It was only to give an example. With UTF-8 people who implement 
the low level code in general think about the multiple codeunits 
at the buffer boundary. With UTF-16 it's often forgotten. In 
UTF-16 there are also 2 other common pitfalls, that exist also in 
UTF-8 but are less consciously acknowledged, overlong encoding 
and isolated codepoints. So UTF-16 has the same issues as UTF-8, 
plus some more, endianness and size.