Unicode Normalization (and graphemes and locales)

[Jonathan M Davis via Digitalmars-d]

On Thursday, June 02, 2016 17:14:13 Walter Bright via Digitalmars-d wrote:
> On 6/2/2016 4:29 PM, Jonathan M Davis via Digitalmars-d wrote:
>  > How do you suggest that we handle the normalization issue? Should we just
>  > assume NFC like std.uni.normalize does and provide an optional template
>  > argument to indicate a different normalization (like normalize does)?
>  > Since
>  > without providing a way to deal with the normalization, we're not
>  > actually
>  > making the code fully correct, just faster.
>
> The short answer is, we don't.

I generally agree. The main problem that I was concerned about were the
cases like find where we're talking about encoding the needle to match the
haystack so that we can compare with code units, and I was thinking that
we'd be forced to pick a normalization scheme with that, and if that didn't
match the normalization of the haystack, we'd be in trouble (hence the
concern about being able to specify a normalization scheme). However,
thinking about it further, that's not actually a problem. If the needle is a
dchar, then code point normalization isn't an issue, because it's only ever
one code point, and if the needle uses a different encoding (e.g. UTF-16
instead of UTF-8), and we re-encode it with the encoding of the haystack,
that doesn't change the normalization of the needle. Even if the code units
have changed, the code points that they represent are the same. So, it
doesn't even potentially make sense to try and doing anything with the
normalization when re-encoding the needle.

So, it looks like my concern was born from not thinking the issue through
thoroughly enough.

- Jonathan M Davis