Proposal for fixing dchar ranges

[Dmitry Olshansky]

18-Mar-2014 10:21, Marco Leise пишет:
> The Unicode standard is too complex for general purpose
> algorithms to do useful things on D strings. We don't see that
> however, since our writing systems are sufficiently well
> supported.

> As an inspiration I'll leave a string here that contains
> combined characters in Korean
> (http://decodeunicode.org/hangul_syllables)
> and Latin as well as full width characters that span 2
> characters in e.g. Latin, Greek or Cyrillic scripts
> (http://en.wikipedia.org/wiki/Halfwidth_and_fullwidth_forms):
>
> Halfwidth / Ｆｕｌｌｗｉｄｔｈ, ᆨᆨᆨᆚᆚᅱᅱᅱᅡᅓᅲᄄᆒᄋᆮ, a͢b 9͚ c̹̊
>
> (I used the "unfonts" package for the Hangul part)
>
> What I want to say is that for correct Unicode handling we
> should either use existing libraries or get a feeling for
> what the Unicode standard provides, then form use cases out of it.

There is ICU and very few other things, like support in OSX frameworks 
(NSString). Industry in general kinda sucks on this point but 
desperately wants to improve.

>
> For example when we talk about the length of a string we are
> actually talking about 4 different things:
>
>    - number of code units
>    - number of code points
>    - number of user perceived characters
>    - display width using a monospace font
>
> The same distinction applies for slicing, depending on use case.
>
> Related:
>    - What normalization do D strings use. Both Linux and
>      MacOS X use UTF-8, but the binary representation of non-ASCII
>      file names is different.

There is no single normalization to fix on.
D programs may be written for Linux only, for Mac-only or for both.

IMO we should just provide ways to normalize strings.
(std.uni.normalize has 'normalize' for starters).

>    - How do we handle sorting strings?

Unicode collation algorithm and provide ways to tweak the default one.

> The topic matter is complex, but not difficult (as in rocket science).
> If we really want to find a solution, we should form an expert group
> and stop talking until we read the latest Unicode specs.

Well, I did. You seem motivated, would you like to join the group?

> They are a
> moving target. Don't expect to ever be "done" with full Unicode
> support in D.

The 6.x standard line seems pretty stable to me. There is a point in 
provding support that worth approaching. After that ROI is drooping 
steadily as the amount of work to specialize for each specific culture 
rises. At some point we can only talk about opening up ways to specialize.

D (or any library for that matter) won't ever have all possible 
tinkering that Unicode standard permits. So I expect D to be "done" with 
Unicode one day simply by reaching a point of having all universally 
applicable stuff (and stated defaults) plus having a toolbox to craft 
your own versions of algorithms. This is the goal of new std.uni.


-- 
Dmitry Olshansky