A big update on 2.110

Thu Mar 6 13:30:51 UTC 2025

Hello everyone,

As you all know, the official DMD releases got severely delayed. 
I think you're owed an explanation of why it took so long.

Iain Buclaw, our release manager, was working on streamlining the 
release process to make it more automated and reduce the '[bus 
factor](https://en.wikipedia.org/wiki/Bus_factor)'.

Then he got 'hit by a bus'.

Development on dmd and automatic 
[nightly](https://github.com/dlang/dmd/releases/tag/nightly) 
builds continued, but the official releases for 2.110 and beyond 
got delayed and delayed, as nobody was around with the know-how 
and access to make a proper release.
Luckily, as the [November 
update](https://forum.dlang.org/post/xlsnoajvuphefxlestwu@forum.dlang.org) alluded to, Iain was going to slowly return and on January 8, he uploaded the release candidate for 2.110. While doing that, he walked me through the process in a Slack huddle / tmate session, so I could take over the task for the time being. I was optimistic that things were going to be back on track now, and updated the release schedule with 2.110.0 due for February 1st 2025. However, once that day passed, it turned out we were not in the clear yet. At first I thought there was just a small wait for [the new changelog generator after our migration from Bugzilla to GitHub issues](https://github.com/dlang/tools/pull/466), but no: 2.110.0 still needed to be built and uploaded, so this week I made it my priority to get everything sorted out.

### Long story
Feel free to skip to the next paragraph, this is just a bunch of 
technical details for your amusement / understanding of why it 
was so hard to complete a release for me.

So like I said, Iain and I scheduled an online meeting to go over 
the release process on January 8. Right from the start, Slack 
huddles don't work in Firefox, so I downloaded the client, which 
crashed really badly when I started sharing my screen through it, 
requiring a reboot of my computer to fix my audio. That's not 
relevant, but it set the tone wonderfully for everyone that was 
about to come.

So the release orchestration is largely handled by one big 
[release.sh](https://gist.github.com/ibuclaw/35d19c5ff64ecea25e4a7c7ce6a79eac) bash script with about 20 sequential steps, but it can't just be run on any computer: it has a lot of dependencies on software being installed, configuration, authentication to various servers etc. So Iain started by giving me an account on his server from which he builds releases, that I can ssh into. Then we walked through each step, one by one. Because I had some trouble getting my git configuration/authentication/keys etc. working on the server, I ended up `rsync`'ing my user folder on the server to my own desktop so I could continue locally. That folder is 18 GB because it contains 4 different virtual machines for building the Linux, Windows, OSX and FreeBSD releases, but once the download completes, I should be able to build the release, right?

Each step uncovered new missing dependencies / setup on my part. 
I installed `vagrant` and `virtualbox`, had to `modprobe 
vboxdrv`, but that required installing `linux-rt-headers`, wait 
it still doesn't work, oh the right package is `linux-headers` 
yadayadayada... Running the VM also requires enabling 
virtualization in your BIOS, a setting which I couldn't find at 
first because it's labeled 'SVM mode' on this particular 
motherboard. The biggest issue was that I have an AMD processor, 
and OSX refuses to boot on anything but an Intel processor. 
Apparently it's easy to fool the cpuid on libvirt, but not on 
virtualbox.
That's how far we came in that huddle: I made progress, but 
ultimately couldn't build myself yet.

When it became clear this week that I had to figure out a way to 
build myself, I started to look if I had a different computer 
laying around. Since my old desktop with an Intel processor has a 
broken motherboard, I dug up my old 2015 student laptop with 
Windows 10, and installed linux on a new partition. The USB drive 
from which I previously installed NixOS on a mini laptop suddenly 
failed to start the display manager. It looks like some bits got 
corrupted, because a decompression program reported bad file 
contents. I always suspected that particular USB stick to be 
unreliable, but the 2 other drives I tried wouldn't even get 
picked up by my Laptop's boot loader, so I copied a fresh NixOS 
installer on the sketchy one and hoped it would work. It did. I 
can't seem to boot it directly though, I have to boot Windows 10 
first and then shift + restart => use device => Linux Boot 
Loader. Oh well.

Now I had to configure everything on my new installation. Some 
ChatGPT assistance, [Stack 
Overflow](https://stackoverflow.com/questions/29933918/ssh-key-permissions-0644-for-id-rsa-pub-are-too-open-on-mac), and an [MVP GitHub comment](https://github.com/NixOS/nixpkgs/issues/76108#issuecomment-592195955) later, I could finally boot the virtual boxes! Then it exceeded my 8GB of RAM and the OOM killer crashed the whole PC.

I rebooted and set up an 8GB swap file. I retried, but suddenly 
the Virtual Boxes wouldn't boot anymore, now reporting a new 
error. Huh? When I retried, it failed even before getting to the 
command to start the VM. Can't write to `/tmp/XXX`? Oh, I see. I 
ran out of disk space. I only had room for a 58 GB partition, 
which I thought was enough for the 20 GB of VMs, but not when 
`~/VirtualBox VMs` and `/tmp` get filled up by running vagrant so 
many times. Some disk cleaning and 2 hours of anxiously watching 
my task manager showing 95% RAM usage later, it actually did it. 
It built releases for all platforms! That felt real good, but my 
troubleshooting hadn't come to an end just yet...

### Long story short

To run the script that handles the release process, [to quote 
Atila](https://forum.dlang.org/post/fiyfbeczwihhsbiudtaw@forum.dlang.org), you need a special ~~Windows~~ Linux box commisioned by the Vatican and blessed by the Pope himself. But after a ton of troubleshooting, my old laptop became such a blessed box. It's probably the third one on the planet, and I'm storing it in a safe in a nuclear bunker. :)

### GitHub actions outage

Every release is accompanied with a PGP signature in the form a 
.sig file, which the `install.sh` script and the setup-dlang 
GitHub action uses to `gpg --verify` the executables with. Since 
I'm a new signer, I added my public key to the 
[gpg_keys](https://dlang.org/gpg_keys.html) page. However, I 
wasn't aware that I also had to manually update the binary 
`d-keyring.pgp` file, which is the source of public keys 
installation scripts actually use. So yesterday (March 5) you 
might have noticed the following error in your CI:

```
Run dlang-community/setup-dlang at v1
Enabling dmd-latest
Downloading 
https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.osx.tar.xz
Verifying the download with GPG
gpg: Signature made Wed Mar  5 12:46:50 2025 UTC
gpg:                using RSA key 
F3F896F3274BBD9BBBA59058710592E7FB7AF6CA
gpg: Can't check signature: No public key
```

I fixed that this morning (Mar 6 11:00 UTC) with assistance from 
Andrei Horodniceanu on Slack (thanks!). I'm sorry for the 
inconvenience. If you encounter this error while manually 
installing/verifying, ensure you are using the latest keyring.

### What's happening now

My current plans are:

- Fix the changelog (either by trying Robert's new tool or 
writing something manually)
- Formally announce the 2.110 release on Friday March 7, and 
publish the download links on the website
- Update the [release archive 
index](https://downloads.dlang.org/releases/2.x/) (currently I 
hit a snag where 
[d-requests](https://code.dlang.org/packages/requests) can't find 
the right libssl.so)
- Update the documentation archives
- Document everything I learned about the release process for 
future maintainers
- Do the releases for 2.111 onwards until Iain or someone else 
takes over again

If you want to download 2.110 already, you can do so at:

[libphobos2-109_2.110.0-0_i386.deb.sig](https://downloads.dlang.org/releases/2.x/2.110.0/libphobos2-109_2.110.0-0_i386.deb.sig)
[libphobos2-109_2.110.0-0_i386.deb](https://downloads.dlang.org/releases/2.x/2.110.0/libphobos2-109_2.110.0-0_i386.deb)
[libphobos2-109_2.110.0-0_amd64.deb.sig](https://downloads.dlang.org/releases/2.x/2.110.0/libphobos2-109_2.110.0-0_amd64.deb.sig)
[libphobos2-109_2.110.0-0_amd64.deb](https://downloads.dlang.org/releases/2.x/2.110.0/libphobos2-109_2.110.0-0_amd64.deb)
[dmd_2.110.0-0_i386.deb.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd_2.110.0-0_i386.deb.sig)
[dmd_2.110.0-0_i386.deb](https://downloads.dlang.org/releases/2.x/2.110.0/dmd_2.110.0-0_i386.deb)
[dmd_2.110.0-0_amd64.deb.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd_2.110.0-0_amd64.deb.sig)
[dmd_2.110.0-0_amd64.deb](https://downloads.dlang.org/releases/2.x/2.110.0/dmd_2.110.0-0_amd64.deb)
[dmd.2.110.0.windows.zip.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.windows.zip.sig)
[dmd.2.110.0.windows.zip](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.windows.zip)
[dmd.2.110.0.windows.7z.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.windows.7z.sig)
[dmd.2.110.0.windows.7z](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.windows.7z)
[dmd.2.110.0.osx.zip.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.osx.zip.sig)
[dmd.2.110.0.osx.zip](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.osx.zip)
[dmd.2.110.0.osx.tar.xz.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.osx.tar.xz.sig)
[dmd.2.110.0.osx.tar.xz](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.osx.tar.xz)
[dmd.2.110.0.linux.zip.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.linux.zip.sig)
[dmd.2.110.0.linux.zip](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.linux.zip)
[dmd.2.110.0.linux.tar.xz.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.linux.tar.xz.sig)
[dmd.2.110.0.linux.tar.xz](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.linux.tar.xz)
[dmd.2.110.0.freebsd-64.zip.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.freebsd-64.zip.sig)
[dmd.2.110.0.freebsd-64.zip](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.freebsd-64.zip)
[dmd.2.110.0.freebsd-64.tar.xz.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.freebsd-64.tar.xz.sig)
[dmd.2.110.0.freebsd-64.tar.xz](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.freebsd-64.tar.xz)
[dmd.2.110.0.dmg.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.dmg.sig)
[dmd.2.110.0.dmg](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.dmg)
[dmd-2.110.0.exe.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0.exe.sig)
[dmd-2.110.0.exe](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0.exe)
[dmd-2.110.0-0.openSUSE.x86_64.rpm.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0-0.openSUSE.x86_64.rpm.sig)
[dmd-2.110.0-0.openSUSE.x86_64.rpm](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0-0.openSUSE.x86_64.rpm)
[dmd-2.110.0-0.openSUSE.i386.rpm.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0-0.openSUSE.i386.rpm.sig)
[dmd-2.110.0-0.openSUSE.i386.rpm](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0-0.openSUSE.i386.rpm)
[dmd-2.110.0-0.fedora.x86_64.rpm.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0-0.fedora.x86_64.rpm.sig)
[dmd-2.110.0-0.fedora.x86_64.rpm](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0-0.fedora.x86_64.rpm)
[dmd-2.110.0-0.fedora.i386.rpm.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0-0.fedora.i386.rpm.sig)
[dmd-2.110.0-0.fedora.i386.rpm](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0-0.fedora.i386.rpm)

### Future release schedule

Note that 2.110 only contains bug fixes implemented in the last 7 
months, not the new features of the last 7 months. Those are 
scheduled for 2.111, which brings up a question: Iain and I had 
the idea of skipping 2.110.1 and going straight to 2.111.0, since 
we're so far behind. There's also been talk of slowing down the 
amount of releases in general. What do you all think?

### Final words

I know the whole situation is very unprofessional. Personally, 
I'm working part time as Pull Request and Issue Manager for the D 
Language Foundation, and part time as D Programmer for 
[SARC](https://www.sarc.nl/), a Dutch maritime software company 
with a roughly similar head count as the DLF 'core team'. Every 
week, I go to SARC's office where there's a weekly meeting with 
everyone. You can see what everyone is working on, knowledge is 
easily shared, and issues can be solved together. While I think 
it's amazing that DLF operates completely remote, with volunteer 
contributions coming from all around the world, it is awful for 
handling situations like this 2.110 release debacle. But we are 
learning and improving overall I think.

I want to thank Iain for doing the thankless job of being release 
manager, and helping us out recently when he really shouldn't 
have had to.

Have a good day everyone.

~Dennis Korpel