The Case Against Autodecode

[Jonathan M Davis via Digitalmars-d]

On Sunday, May 29, 2016 13:47:32 H. S. Teoh via Digitalmars-d wrote:
> On Sun, May 29, 2016 at 03:55:22PM -0400, Andrei Alexandrescu via 
Digitalmars-d wrote:
> > So now code points are good? -- Andrei
>
> It depends on what you're trying to accomplish. That's the point we're
> trying to get at.  For some operations, working with code points makes
> the most sense. But for other operations, it does not.  There is no one
> representation that is best for all situations; it needs to be decided
> on a case-by-case basis.  Which is why forcing everything to decode to
> code points eventually leads to problems.

Exactly. And even a given function can't necessarily always be defined to
use a specific level of Unicode, because whether that's correct or not
depends on what the programmer is actually trying to do with the function.
And then there are cases where the programmer knows enough about the data
that they're dealing with that they're able to operate at a different level
of Unicode than would normally be correct. The most obvious example of that
is when you know that your strings are pure ASCII, but it's not the only
case.

We should strive to make Phobos operate correctly on strings by default
where we can, but there are cases where the programmer needs to know enough
to specify the behavior that they want, and deciding for them is just going
to lead to behavior that happens to be right some of the time while making
it hard for code using Phobos to have the correct behavior the rest of the
time. And the default behavior that we currently have is inefficient to
boot.

- Jonathan M Davis