Major performance problem with std.array.front()

[Dmitry Olshansky]

09-Mar-2014 21:16, Andrei Alexandrescu пишет:
> On 3/9/14, 4:34 AM, Peter Alexander wrote:
>> I think this is the main confusion: the belief that iterating by code
>> point has utility.
>>
>> If you care about normalization then neither by code unit, by code
>> point, nor by grapheme are correct (except in certain language subsets).
>
> I suspect that code point iteration is the worst as it works only with
> ASCII and perchance with ASCII single-byte extensions. Then we have code
> unit iteration that works with a larger spectrum of languages.

Was clearly meant to be: code point <--> code unit

> One
> question would be how large that spectrum it is. If it's larger than
> English, then that would be nice because we would've made progress.
>

Code points help only in so far that many (~all) high-level algorithms 
in Unicode are described in terms of code points. Code points have 
properties, code unit do not have anything. Code points with assigned 
semantic value are "abstract characters".

It's up to programmer to implement a particular algorithm to make it "as 
if" decoding really happened, working directly on code units or do 
decoding and work with code points which is simpler.

Current std.uni offering mostly work on code points and decodes, crucial 
building block to work directly on code units is in review:

https://github.com/D-Programming-Language/phobos/pull/1685

> I don't know about normalization beyond discussions in this group, but
> as far as I understand from
> http://www.unicode.org/faq/normalization.html, normalization would be a
> one-step process, after which code point iteration would cover still
> more human languages. No? I'm pretty sure it's more complicated than
> that, so please illuminate me :o).

Technically most apps just assume say "input comes in UTF-8 that is in 
normalization form C". Other such as browsers strive to get uniform 
representation on any input, do normalization of any input (often times 
normalization turns out to be just a no-op).


>> If you don't care about normalization then by code unit is just as good
>> as by code point, but you don't need to specialise everywhere in Phobos.
>>
>> AFAIK, there is only one exception, stuff like s.all!(c => c == 'é'),
>> but as Vladimir correctly points out: (a) by code point, this is still
>> broken in the face of normalization, and (b) are there any real
>> applications that search a string for a specific non-ASCII character?
>
> What happened to counting characters and such?

Counting chars is dubious. But, for instance, collation is defined in 
terms of code points. Regex pattern matching is _defined_ in terms of 
codepoints (even the mystical level 3 Unicode support of it). So there 
is certain merit to work at that level. But hacking it to be this way 
isn't the way to go.

The least intrusive change would be to generalize the current choice 
w.r.t. to RA ranges of char/wchar.

-- 
Dmitry Olshansky