The Future of D Runtime

[Adam Wilson]

History and Problem
-------------------

Now that the Editions DIP is up public comment it is time to have 
a frank discussion about where the D Runtime (DRT for the 
remainder) fits into the puzzle that Editions introduces into the 
D ecosystem. The reason I bring this up is that it appears that, 
at present, most people treat DRT as a particularly vexing 
extension of the compiler that nobody wants to touch. As a 
result, the presumption seems to be that, with the advent of 
editions, we should just freeze DRT wherever it is when the first 
edition drops and never touch it again because doing so might 
break something across editions. While this position is certainly 
expedient, I would argue that this position is going to severely 
limit the future of Phobos and D.

Furthermore, I would argue that this position makes a presumption 
that all DRT should ever be is a compiler extension library. I 
propose that this view is both incorrect and a significant 
limitation on what we can do with Phobos 3 and even future 
editions.

I want to call back to DConf 2023 where I spent much of 
conference working to undo the deprecation of the ODBC bindings 
in Phobos because they were being replaced with `version 
(Windows)` bindings in DRT, which is utter nonsense, ODBC can be 
used on POSIX quite easily (and I do every day). The explanation 
I was given as to why this was done is that the bindings have 
nothing to do with system resources, but because the bindings 
were provided by DRT they should be removed from Phobos because 
Phobos is not in the business of providing library bindings.

This creates a rule for Phobos. Phobos does not expose system 
resources or bindings. IMO this is a good rule. Phobos is 
bespoke, and that is the proper situation. The standard library 
really should not be in the business of exposing system level 
constructs.

Then why not use the DRT bindings? Because the bindings in DRT 
are automatically generated and are fundamentally broken because 
of it. At the time it was noted to me that those bindings really 
should not be in DRT *either* because DRT isn’t supposed to 
provide access to non-System resources like ODBC and they were 
only there by accident because the ODBC headers got swept into 
the auto-generation process.

This creates a rule for DRT. DRT is only to expose system 
resources, nothing more. And indeed, this is a good rule as well. 
The purpose of a runtime is to expose system resources in a 
uniform manner, and to the extent that DRT does this, it does a 
fantastic job.

Currently we call DRT a runtime, but when you catalog what it 
does, it’s really a “compiler support library that provides some 
runtime services.” There is no argument from me that the GC is a 
runtime service, the same for threads, but the rest of the listed 
items that DRT provides are very definitely aimed at enabling the 
compiler to do things like AA’s for example. AA’s are a feature 
that is only accessible from the language hence “compiler 
support”.

The most often given reason for keeping DRT as-is is that “we 
only need to port DRT to a new platform, Phobos is platform 
independent.” I submit that this is factually incorrect. There 
are currently 31 files in Phobos with `version (WinXXXX)` 
statements in them. 14 with `version (linux)`. And IIRC everybody 
who has tried a port D to a new platform recently has just not 
bothered to port Phobos at all. To be sure, some of these version 
statements are benign, but take a look at `std.stdio`, I wish 
whoever wants to port that much luck in their endeavors. It has 
32 instances `version (WinXXXX)` alone. And that is our basic I/O 
module.

Runtimes are not compiler support libraries, they are 
application-system interfaces. The .NET runtime does much more 
than threads and the GC. The Java runtime does much more than 
threads and the GC. The C runtime does much more than memory 
management. I could go on, but you get the point.

The purpose of a runtime is to provide a system resource access 
layer for the standard library. Consider the C Runtime and the C 
Standard Library. The C Runtime provides the system specific 
implementation, and the Standard Library provides the 
standardized application interface. No matter what system you are 
on, calling `printf` will print a string to the terminal via the 
runtime. If you get a `FILE*` you will work with a file on disk 
no matter what filesystem the system is using. These are not 
compiler supports. They are a universal system-application 
interface.

An example of why this matters is found in `std.stdio`. We 
currently use the C Runtime to do basic I/O operations. While 
that works and is expedient, it limits us. For example, doing 
colored terminal output is so difficult nobody does it, or, we 
are limited to the C file interface, etc. To gain access to more 
advanced capabilities, we need to use the system API’s, but these 
are diverse, and putting them in Phobos would significantly 
expand the code required and make maintenance a nightmare. What 
is interesting about this is that, in fact, D has two runtimes. 
The C Runtime, which we use as a universal system interface, and 
the DRT which is a bunch of compiler support tools that the CRT 
doesn’t have.

Proposed Design
---------------

My proposal is that we expand DRT into a universal system 
interface for the standard library. DRT would contain low-level 
interfaces for Terminal and File I/O, threads, event-loops, 
sockets, cryptography, anything that would normally access a 
system API. Why go to all this effort? Simple, so that we can 
move beyond the limitations imposed by the C Runtime. Using the 
CRT was certainly expedient in the early days of D, but we’ve 
grown beyond that and it’s time to upgrade our capabilities.

However, this will significantly expand the scope of DRT and will 
make porting harder. I think the answer to this problem came out 
of left field in a conversation I had last Thursday with an 
ex-Google engineer who has no prior knowledge of D. What he said 
was: “you need to shard your runtime.” This statement caused 
several things to “click” and got me started writing this missive.

Before I get into the design of the DRT I want to propose rules 
that will allow us to continue to evolve DRT in the future 
without breaking past editions. From an ABI standpoint, there are 
only three actions that can be taken: Add, Rename, Remove.
1. Remove. Banned. We cannot remove symbols from the DRT API. But 
we can rename them provided we follow the rename rules.
2. Rename. Renames include changing the parameters in any way. We 
can support this by offering either a shim that redirects to the 
new method or leaving the original method alone and building a 
new one next to it.
3. Add. The is probably the easiest. New symbols can be added at 
any time.
If we follow these rules, then we can safely use the *latest* DRT 
version with all prior editions.

For DRT I propose a sharded design with a split between the 
compiler support modules and the universal system interface 
shards. What this means is that there will be a “Core-DRT” that 
only contains things that the compiler needs to function. Then, 
for each subsystem in Phobos, there would be a corresponding DRT 
subsystem that can be built and linked (statically or 
dynamically). This means that the only mandatory component that 
would have to be ported is the “Core-DRT” component for the 
compiler. The porting engineer would then be free to choose to 
port as much or as little of what remains of DRT and know that 
the corresponding Phobos subsystem is guaranteed to work once the 
port is complete. Another to view these components is as an 
“onion” where each one builds on the one above it. Core must 
exist for System, System exists for Cryptography, etc. Below is 
a, likely incomplete, list of potential runtime shards.

Core:
1. Compiler Hooks
2. GC
3. Threads
4. Event Loop (in later iterations)

System:
1. Console/Terminal I/O
2. File I/O

Cryptography:
1. System Cryptography Primitives
2. Optional OpenSSL extended Cryptography Primitives (to support 
what the OS does not).

Network:
1. Sockets
2. SSL/TLS

I’m not married to the above so please destroy. One thing that I 
think would be beneficial for ease-of-porting is researching if 
it would be possible to move Threads and the Event Loop out of 
Core and into the System component then provide a mechanism for 
notifying the GC of thread creation/deletion without actually 
requiring the Threads (and thus the Event Loop) implementation to 
be in the Core DRT so that people porting to systems that don’t 
have Threads (or simply don’t want to support them) don’t have to 
port them to get basic D code running.

The goal of Core is to be the most reduced component of the 
runtime that can still produce a functional program in D. The 
larger goal of this DRT expansion project is to eventually drop 
the requirement for the C Runtime. By doing so we free ourselves 
from the limitations of the CRT, such as being unable to handle 
Console Colors on systems that support them, and we remove a 
layer of abstraction by calling directly into the system API’s. 
This does mean more work for porting in terms of needing to call 
the system API’s directly as opposed to using the CRT, however, 
one option could be to provide an opt-in CRT-enabled build of DRT 
that uses the CRT for a limited set of features.

I want to point out that these are ideas intended to address the 
limitations of the current situation and the proposal to make 
changes is a result of looking at the work we’re going to have to 
do with Phobos 3. I fully understand that I can be wrong on 
things here, but the present situation with DRT is not going to 
remain tenable as we move forward with expanding Phobos.

Another angle that might be worth pursuing is building DRT up 
into a Runtime that can be used by many languages, similar to how 
the CRT is used. It would also increase the appeal of D to folks 
looking for a platform to start building a language on, like how 
an ARM backend will help the hobbyist language community. I 
understand that LLVM and GCC are there, but they are not simple 
to work with. If we can provide an ecosystem that is simple to 
work with for language experimentation and development, we can 
capture people looking for a tool to start designing languages 
quickly and still provide them a path to integrate with LLVM or 
GCC via our existing integrations with those tools.

These are ideas, not directions, so please destroy.