Unicode handling comparison

[Jakob Ovrum]

On Wednesday, 27 November 2013 at 16:18:34 UTC, Wyatt wrote:
> I agree with the assertion that people SHOULD know how unicode 
> works if they want to work with it, but the way our docs are 
> now is off-putting enough that most probably won't learn 
> anything.  If they know, they know; if they don't, the wall of 
> jargon is intimidating and hard to grasp (more examples up 
> front of more things that you'd actually use std.uni for).  
> Even though I'm decently familiar with Unicode, I was having 
> trouble following all that (e.g. Isn't "noe\u0308l" a grapheme 
> cluster according to std.uni?).  On the flip side, std.utf has 
> a serious dearth of examples and the relationship between the 
> two isn't clear.

I thought it was nice that std.uni had a proper terminology 
section, complete with links to Unicode documents to kick-start 
beginners to Unicode. It mentions its relationship with std.utf 
right at the top.

Maybe the first paragraph is just too thin, and it's hard to see 
the big picture. Maybe it should include a small leading 
paragraph detailing the three levels of Unicode granularity that 
D/Phobos chooses; arrays of code units -> ranges of code points 
-> std.uni for graphemes and algorithms.

> Yes, please.  While operations on single codepoints and 
> characters seem pretty robust (i.e. you can do lots of things 
> with and to them), it feels like it just falls apart when you 
> try to work with strings.  It honestly surprised me how many 
> things in std.uni don't seem to work on ranges.
>
> -Wyatt

Most string code is Unicode-correct as long as it works on code 
points and all inputs are of the same normalization format; 
explicit grapheme-awareness is rarely a necessity. By that I mean 
the most common string operations, such as searching, getting a 
substring etc. will work without any special grapheme decoding 
(beyond normalization).

The hiccups appear when code points are shuffled around, or the 
order is changed. Apart from these rare string manipulation 
cases, grapheme awareness is necessary for rendering code.