UTF-8 Unicode operators vs digraphs

[H. S. Teoh]

On Thu, Aug 13, 2026 at 05:17:36PM -0700, Walter Bright via Digitalmars-d wrote:
> On 8/13/2026 12:20 AM, Dlighted wrote:
> > It is unavoidable for letters from different scripts to look the
> > same in some fonts when supporting many scripts.
> 
> What Unicode has done wrong is attach semantic meaning to code points.
> This is all wrong. When you read a printed page, the semantic meaning
> depends on the context, not the glyph.
[...]
> Bottom line: you should be able to print a page written in Unicode,
> then OCR it and get back the same Unicode text (at least for the
> non-blank points).  Instead, we have Unicode that means one thing to
> the human reader and something totally different to a computer reading
> it.

The problem isn't as simple as you imagine.  Some scripts have the
*same* logical glyph look differently in different contexts, a *subset*
of which looks like a subset of a different glyph in another language.

Take for example, the Cyrillic т.  The cursive form looks like this: m.
To a native speaker, they are EXACTLY THE SAME LETTER, just differently
styled.  The same way an English speaker sees the same letter whether
the glyph is an upright M or a cursive M.  This is a variation that a
different font face would give you.

Here's the problem: suppose Unicode does as you propose: represent the
glyph m as a single codepoint.  Now you have a problem: т and m now must
be represented as two different codepoints, even though they are EXACTLY
THE SAME LETTER.  Because in English, t and m are different letters with
different glyphs, so you can't have just a single codepoint for т and m.
It makes no sense for an English word like "mama" to be represented the
same way as Cyrillic "тата".  The only sane solution is to code the
Cyrillic т/m and the English M/m differently.  Otherwise you still have
the same problem that multiple codepoints map to the same logical
character.

Another example is the Cyrillic п which in some fonts look like n.  You
can't seriously be proposing that we represent the Cyrillic п with the
same codepoint as the English n.  In Cyrillic, that's a stylistic
difference, in English, that's a completely different glyph.  If you
insist that the same glyph shapes must share the same codepoint, then
the Greek π should be represented as the English n too, since п and π
are equivalent, and in some fonts look identical.  If you argue that
since п and n are visually distinct shapes, they must be separate
codepoints, then I could also argue that English upright letters should
be coded differently from italics, since they *are* visually distinct,
and in many fonts involve actually different glyph shapes.

This doesn't even begin to scratch the surface of the issues that arise
when you're dealing with languages like Arabic, where the same letter
can appear in completely different shapes depending on where it occurs
in a word, and in what context it's used.

tl;dr: the issues involved are much more complex than what an English
speaker may imagine.  Unicode isn't without its silly points and
arguably backwards decisions, but given the complexity of the issue,
it's doing not too bad.


T

-- 
BREAKFAST.COM halted...Cereal Port Not Responding. -- YHL