Major performance problem with std.array.front()

[Vladimir Panteleev]

On Saturday, 8 March 2014 at 23:59:15 UTC, Andrei Alexandrescu 
wrote:
> My only claim is that recognizing and iterating strings by code 
> point is better than doing things by the octet.

Considering or disregarding the disadvantages of this choice?

>> 1. Eliminating dangerous constructs, such as s.countUntil and 
>> s.indexOf
>> both returning integers, yet possibly having different values 
>> in
>> circumstances that the developer may not foresee.
>
> I disagree there's any danger. They deal in code points, end of 
> story.

Perhaps I did not explain clearly enough.

auto pos = s.countUntil(sub);
writeln(s[pos..$]);

This will compile, and work for English text. For someone without 
complete knowledge of Phobos functions and how D handles Unicode, 
it is not obvious that this code is actually wrong. In certain 
situations, this can have devastating effects: consider, for 
example, if this code is extracting a slice from a string that 
elsewhere contains sensitive data (e.g. a configuration file 
containing, among other data, a password). An attacker could 
supply an Unicode string where the developer did not expect it, 
thus causing "pos" to have a smaller value than the corresponding 
indexOf result, thus revealing a slice of "s" which was not 
intended to be visible. Thus, a developer currently needs to 
tread very carefully wherever he is slicing strings, so as to not 
accidentally use indices obtained from functions that count code 
points.

>> 2. Very high complexity of implementations (the 
>> ElementEncodingType
>> problem previously mentioned).
>
> I disagree with "very high".

I'm quite sure that std.range and std.algorithm will lose a LOT 
of weight if they were fixed to not treat strings specially.

> Besides if you want to do Unicode you gotta crack some eggs.

No, I can't see how this justifies the choice. An explicit 
decoding range would have simplified things greatly while 
offering much of the same advantages. Whether the fact that it is 
there "by default" an advantage of the current approach at all is 
debatable.

>> 3. Hidden, difficult-to-detect performance problems. The 
>> reason why this
>> thread was started. I've had to deal with them in several 
>> places myself.
>
> I disagree with "hidden, difficult to detect".

Why? You can only find out that an algorithm is slower than it 
needs to be via either profiling (at which point you're wondering 
why the @#$% the thing is so slow), or feeding it invalid UTF. If 
you had made a different choice for Unicode in D, this problem 
would not exist altogether.

> Also I'd add that I'd rather not have hidden, difficult to 
> detect correctness problems.

Except we already do. Arguments have already been presented in 
this thread that demonstrate correctness problems with the 
current approach. I don't think that these can stand up to the 
problems that the simpler by-char iteration approach would have.

>> 4. Encourage D programmers to write Unicode-capable code that 
>> is correct
>> in the full sense of the word.
>
> I disagree we are presently discouraging them.

I did not say we are. The problem is that we aren't encouraging 
them either - we are instead setting an example of how to do it 
in a wrong (incomplete) way.

> I do agree a change would make certain things clearer.

I have an issue with all the counter-arguments presented in this 
thread being shoved behind the one word "clearer".

> But not enough to nearly make up for the breakage.

I would still like to go ahead with my suggestion to attempt some 
possible changes without releasing them. I'm going to try them 
with my own programs first to see how much it will break. I 
believe that you are too eagerly dismissing all proposals without 
even evaluating them.

>> I think the above list has enough weight to merit at least 
>> considering
>> *some* breaking changes.
>
> I think a better approach is to figure what to add.

This is obvious:
- more Unicode algorithms (normalization, segmentation, etc.)
- better documentation