Inconsitency

[nickles]

> This will _not_ return a trailing surrogate of a Cyrillic 
> letter. It will return the second code unit of the "ä" 
> character (U+00E4).

True. It's UTF-8, not UTF-16.

> However, it could also yield the first code unit of the umlaut 
> diacritic, depending on how the string is represented.

This is not true for UTF-8, which is not subject to "endianism".

> If the string were in UTF-32, [2] could yield either the 
> Cyrillic character, or the umlaut diacritic.
> The .length of the UTF-32 string could be either 3 or 4.

Both are not true for UTF-32. There is no interpretation (except 
for the "endianism", which could be taken care of in a 
library/the core) for the code point.

> There are multiple reasons why .length and index access is 
> based on code units rather than code points or any higher level 
> representation, but one is that the complexity would suddenly 
> be O(n) instead of O(1).

see my last statement below

> In-place modifications of char[] arrays also wouldn't be 
> possible anymore

They would be, but

> as the size of the underlying array might have to change.

Well that's a point; on the other hand, D is constantly creating 
and throwing away new strings, so this isn't quite an argument. 
The current solution puts the programmer in charge of dealing 
with UTF-x, where a more consistent implementation would put the 
burden on the implementors of the libraries/core, i.e. the ones 
who usually have a better understanding of Unicode than the 
average programmer.

Also, implementing such a semantics would not per se abandon a 
byte-wise access, would it?

So, how do you guys handle UTF-8 strings in D? What are your 
solutions to the problems described? Does it all come down to 
converting "string"s and "wstring"s to "dstring"s, manipulating 
them, and re-convert them to "string"s? Btw, what would this mean 
in terms of speed?

These is no irony in my questions. I'm really looking for 
solutions...