Why can't D store all UTF-8 code units in char type? (not really understanding explanation)

[H. S. Teoh]

On Fri, Dec 02, 2022 at 09:18:44PM +0000, thebluepandabear via Digitalmars-d-learn wrote:
> Hello (noob question),
> 
> I am reading a book about D by Ali, and he talks about the different
> char types: char, wchar, and dchar. He says that char stores a UTF-8
> code unit, wchar stores a UTF-16 code unit, and dchar stores a UTF-32
> code unit, this makes sense.
> 
> He then goes on to say that:
> 
> "Contrary to some other programming languages, characters in D may
> consist of different numbers of bytes. For example, because 'Ğ' must
> be represented by at least 2 bytes in Unicode, it doesn't fit in a
> variable of type char. On the other hand, because dchar consists of 4
> bytes, it can hold any Unicode character."
> 
> It's his explanation as to why this code doesn't compile even though Ğ
> is a UTF-8 code unit:
> 
> ```D
> char utf8 = 'Ğ';
> ```
> 
> But I don't really understand this? What does it mean that it 'must be
> represented by at least 2 bytes'? If I do `char.sizeof` it's 2 bytes
> so I am confused why it doesn't fit, I don't think it was explained
> well in the book.

That's wrong, char.sizeof should be exactly 1 byte, no more, no less.

First, before we talk about Unicode, we need to get the terminology
straight:

Code unit = unit of storage in a particular representation (encoding) of
Unicode. E.g., a UTF-8 string consists of a stream of 1-byte code units,
a UTF-16 string consists of a stream of 2-byte code units, etc.. Do NOT
confuse this with "code point", or worse, "character".

Code point = the abstract Unicode entity that occupies a single slot in
the Unicode tables.  Usually written as U+xxx where xxx is some
hexadecimal number.

	IMPORTANT NOTE: do NOT confuse a code point with what a normal
	human being thinks of as a "character".  Even though in many
	cases a code point happens to represent a single "character",
	this isn't always true.  It's safer to understand a code point
	as a single slot in one of the Unicode tables.

	NOTE: a code point may be represented by multiple code units,
	depending on the encoding. For example, in UTF-8, some code
	points require multiple code units (multiple bytes) to
	represent. This varies depending on the character; the code
	point `A` needs only a single code unit, but the code point `Ш`
	needs 3 bytes, and the code point `😀` requires 4 bytes. In
	UTF-16, `A` and `Ш` occupy only 1 code unit (2 bytes, because in
	UTF-16, one code unit == 2 bytes), but `😀` needs 2 code units
	(4 bytes).

Note that neither code unit nor code point correspond directly with what
we normally think of as a "character".  The Unicode terminology for that
is:

Grapheme = one or more code points that combine together to produce a
single visual representation.  For example, the 2-code-point sequence
U+006D U+030A produce the *single* grapheme `m̊`, and the 3-code-point
sequence U+03C0 U+0306 U+032f produces the grapheme `π̯̆`.  Note that each
code point in these sequences may require multiple code units, depending
on which encoding you're using.  This email is encoded in UTF-8, so the
first sequence occupies 3 bytes (1 byte for the 1st code point, 2 bytes
for the second), and the second sequence occupies 6 bytes (2 bytes per
code point).

//

OK, now let's talk about D.  In D, we have 3 "character" types (I'm
putting "character" in quotes because they are actually code units, do
NOT confuse them with visual characters): char, wchar, dchar, which are
1, 2, and 4 bytes, respectively.

To find out whether something fits into a char, first you have to find
out how many code points it occupies, and second, how many code units
are required to represent those code points.  For example, the character
`À` can be represented by the single code point U+00C0. However, it
requires *two* UTF-8 code units to represent (this is a consequence of
how UTF-8 represents code points), in spite of being a value that's less
than 256.  So U+00C0 would not fit into a single char; you need (at
least) 2 chars to hold it.

If we were to use UTF-16 instead, U+00C0 would easily fit into a single
code unit.  Each code unit in UTF-16, however, is 2 bytes, so for some
code points (such as 'a', U+0061), the UTF-8 encoding would be smaller.

A dchar always fits any Unicode code point, because code points can only
go up to 0x10FFFF (max 3 bytes).  HOWEVER, using dchar does NOT
guarantee that it will hold a complete visual character, because Unicode
graphemes can be arbitrarily long.  For example, the `π̯̆` grapheme above
requires at least 3 code points to represent, which means it requires at
least 3 dchars (== 12 bytes) to represent. In UTF-8 encoding, however,
it occupies only 6 bytes (still the same 3 code points, just encoded
differently).

//

I hope this is clear (as mud :P -- Unicode is a complex beast). Or at
least clear*er*, anyway.


T

-- 
People say I'm indecisive, but I'm not sure about that. -- YHL, CONLANG