UTF-8 Unicode operators vs digraphs

[Jonathan M Davis]

On Thursday, August 13, 2026 7:17:03 PM Mountain Daylight Time H. S. Teoh via Digitalmars-d wrote:
> tl;dr: the issues involved are much more complex than what an English
> speaker may imagine.  Unicode isn't without its silly points and
> arguably backwards decisions, but given the complexity of the issue,
> it's doing not too bad.

Yeah...

There are definitely a number of bad decisions in the Unicode standard, and
some of the idiocy comes from a variety of external requirements that make
things _way_ messier than they'd be if you could design Unicode in a vacuum
(e.g. requiring that it be possible to convert from a different character
set to Unicode and back again with the result being identical forces Unicode
to include graphemes with the quirks of a bunch of different character sets
whose design philosophies go against what Unicode is trying to do, which is
part of why we have multiple ways to represent the same grapheme).

However, a _lot_ of the stupidy of Unicode comes from the insanity of
natural languages. English has its problems, but when it comes to encoding
characters, it's downright trivial in comparison to a number of other
languages and writing systems (e.g. Cyrillic and Arabic as you mentioned).
Unicode has a number of goals along the lines of what Walter is looking for
which got completely screwed over when dealing with real languages (e.g.
characters being treated differently depending on where in the word or
sentence they are). Even correctly determining whether code points are part
of a single grapheme or not can actually be language-dependent! It's
absolutely insane.

So, on the whole, I'd say that the main problems with Unicode generally fall
into two categories:

1. Natural languages are messy, making handling them in a way that's sane
for a computer essentially impossible. It may be possible to do a better job
than Unicode has, but to a degree, the problem simply cannot be solved in a
sane fashion.

2. Requirements that were placed on Unicode in order to be compatible with
older encodings make following even the sane rules which would otherwise
have been followable impossible to follow.

Both of those mean that pretty much every rule that Unicode has for itself
gets broken somewhere in order to work in some special case. I honestly
question whether there's a single piece of software on the planet which
fully handles Unicode correctly. It's just too much of a disgusting mess.
It's still arguably better than what we had before, but it's by no means
good.

- Jonathan M Davis