UTF-8 Unicode operators vs digraphs

[Timon Gehr]

On 8/12/26 14:30, Dlighted wrote:
> On Wednesday, 12 August 2026 at 11:52:44 UTC, Timon Gehr wrote:
>> Instead of discussing what is interesting about the language, a good 
>> chunk of the comments are just low-effort anti-Unicode-syntax posts. 
>> It's not considered sufficiently "normal" and people like writing 
>> snarky comments, so you always get a lot of spam whenever you are 
>> trying to share something.
> 
> Read this post and reflect.
> ...

This is entirely redundant (how else do you think I engage with the 
forums) and also somewhat disrespectful.

I was answering your only question, hoping that the insight might be 
helpful. Recall that you wrote:
 > The only question I have is why we haven't done it yet.

(This is responding to what you mean assuming I am reading your mind 
correctly, feel free to clarify by simply spelling it out.)

>> I think `∧` and `∨` I think are not great choices for `&&` and `||`, 
>> because `&&` and `||` are short-circuiting operators. Then `and` and 
>> `or` don't match either, for the same reason (though I guess Python 
>> does this anyway). One alternative I have seen that actually makes 
>> sense is `and then` and `or else`, but it's a bit verbose.
> 
> There's a long list of languages that use "and" and "or".

There is an even longer list of languages without Unicode operator 
symbols. ;)

> Pascal, Ada, Fortran, Lua, Python, PHP, R and probably others.
> Ruby and Perl have them but with different precedence.
> ...

Anyway, let's take a cursory glance at your list:

Pascal: Inconsistent, there are literally compiler directives to make it 
evaluate both operands in Free Pascal.
https://www.freepascal.org/docs-html/prog/progsu4.html

Ada: `and` and `or` evaluate both operands, `and then` and `or else` are 
the short-circuiting versions.
https://docs.adacore.com/live/wave/arm12/html/arm12/arm12-4-5-1.html

Fortran: AFAIU it's actually spelled .AND. and .OR.
Also, the Fortran standard actually does not require them to 
short-circuit at all:
https://gcc.gnu.org/onlinedocs/gcc-15.3.0/gfortran/Evaluation-of-logical-expressions.html

Furthermore, they are even allowed to short-circuit backwards.

Lua: Yes.

PHP: Yes. However, please do understand that citing PHP as a precedent 
is the opposite of persuasive.

R: No? I think it is spelled `&&`/`||` in R.
https://cran.r-project.org/doc/manuals/r-release/R-intro.html#Conditional-execution_003a-if-statements

Ruby/Perl: Yes. (They have both `&&`/`||` and `and`/`or`, but `and`/`or` 
have much lower precedence.)

>> OpenAI ignores 𝔹 when reading HTML pages, so GPT is actually not able 
>> to read documentation if it is written with Unicode.
> 
> Gemini says its a problem with the letter you chose and not with Unicode 
> in general:
> ...

Gemini is hallucinating. I guess this was not the flagship model.

This was quite obviously neither a problem with the letter I chose nor 
with Unicode in general, nor with tokenization. This was strictly an 
issue with the agentic harness that ChatGPT was using for online chats. 
However, because ChatGPT is more popular than Unicode syntax, it did 
become an issue for us, and we could have avoided it by not adopting 
Unicode in our language.

Anyway, I had reported the bug to OpenAI, and testing it again now, it 
appears they actually did recently fix the issue (after having remained 
open for a very long time). However, had we not used Unicode, we would 
never have had this problem in the first place.

It's not that this is my personal preference, it's just that there are 
external factors. Personally I much prefer formatting my code with 
Unicode symbols. (Much to the chagrin of some of the people I work with.)

It is an uphill battle. Another anecdote: see how my co-author chose to 
interface with my Unicode identifiers in D:
https://github.com/silq-lang/silq/blob/6e0c42bf78a5c924912d5adf3c471bd988d5320b/hqir.d#L34-L60

It would be easy to dismiss his instinct as entirely irrational, but I 
think it is not the case. There is just a lot of software out there that 
won't deal with Unicode correctly, and there is often an added 
complexity cost with a nontrivial additional bug risk when you do want 
to handle it correctly.

The ChatGPT harness was just another example of poorly written software 
that I don't control and that can't deal with Unicode correctly, and 
this is the sense in which the anecdote is meaningful in this context.


I also used to use a big file with rules like these in a LaTeX header:

\usepackage[utf8]{inputenc}
...
\DeclareUnicodeCharacter{2115}{\mathbb N}
\DeclareUnicodeCharacter{2124}{\mathbb Z}
\DeclareUnicodeCharacter{211A}{\mathbb Q}
\DeclareUnicodeCharacter{211D}{\mathbb R}
\DeclareUnicodeCharacter{2102}{\mathbb C}
\DeclareUnicodeCharacter{2119}{\mathbb P}
...
\DeclareUnicodeCharacter{2080}{_0}
\DeclareUnicodeCharacter{2081}{_1}
\DeclareUnicodeCharacter{2082}{_2}
\DeclareUnicodeCharacter{2083}{_3}
\DeclareUnicodeCharacter{2084}{_4}
\DeclareUnicodeCharacter{2085}{_5}
\DeclareUnicodeCharacter{2086}{_6}
\DeclareUnicodeCharacter{2087}{_7}
\DeclareUnicodeCharacter{2088}{_8}
\DeclareUnicodeCharacter{2089}{_9}
\DeclareUnicodeCharacter{208A}{_{+}}
\DeclareUnicodeCharacter{208B}{_{-}}
\DeclareUnicodeCharacter{208C}{_{=}}
...

This allowed me to write more readable LaTeX files with content like:

ℕ₀ (turns into \mathbb{N}_0)

Why did I have to abandon it? My boss was using a crappy editor on 
Windows and it did not support utf-8 properly, so he was unable to 
actually see, let alone edit, my formulas.

And this was all while pretending that there are no issues whatsoever 
with Unicode itself. (It does not even have subscripted versions of all 
small Latin letters; very frustrating, but this is because it was not 
actually even made to be useful for typesetting formulas in a plaintext 
file.)

> Modern AI models (like ChatGPT) can read both symbols, but they process 
> them through a tokenizer that breaks text down into bytes or smaller 
> subword chunks.
> - ∨ (Logical OR / Down Tack): This is a standard mathematical/logical 
> symbol present in Unicode (U+2228). Because it is frequently used in 
> math, logic, and programming datasets online, tokenizers often handle it 
> gracefully, sometimes mapping it directly or splitting it efficiently.
> - 𝔹 (Blackboard Bold Capital B): This belongs to the Unicode 
> Mathematical Alphanumeric Symbols block (U+1D539). These symbols are 
> notorious for causing tokenization inefficiencies. Because they look 
> visually similar to a standard Latin "B" to a human, but have completely 
> different underlying byte codes, tokenizers often fragment them into 
> multiple awkward tokens.

As I indicated above, this diagnosis is incorrect, it literally was not 
seeing the tokens _only when grabbing the HTML from an online search in 
the in-browser chat_. There was no problem at all when pasting the 
symbol directly into e.g. the chat window (or when using the API with a 
custom harness). There is no issue with tokenization. The model just got 
really adamant that the syntax with the token stripped is correct, 
because it assumed that it retrieved it that way from the authoritative 
source.