The Case Against Autodecode

[tsbockman via Digitalmars-d]

On Friday, 27 May 2016 at 20:42:13 UTC, Andrei Alexandrescu wrote:
> On 05/27/2016 03:39 PM, Dmitry Olshansky wrote:
>> No, this is not the point of normalization.
>
> What is? -- Andrei

1) A grapheme may include several combining characters (such as 
diacritics) whose order is not supposed to be semantically 
significant. Normalization sorts them in a standardized way so 
that string comparisons return the expected result for graphemes 
which differ only by the internal order of their constituent 
combining code points.

2) Some graphemes (like accented latin letters) can be 
represented by a single code point OR a letter followed by a 
combining diacritic. Normalization either splits them all apart 
(NFD), or combines them whenever possible (NFC). Again, this is 
primarily intended to make things like string comparisons work as 
expected, and perhaps to simplify low-level tasks like graphical 
rendering of text.

(Disclaimer: This is an oversimplification, because nothing about 
Unicode is ever simple.)