D Language Foundation April 2024 Quarterly Meeting Summary

[Mike Parker]

The D Language Foundation's quarterly meeting for April 2024 took 
place on Friday the 5th at 15:00 UTC. It lasted for about 50 
minutes.

Our quarterly meetings are where representatives from businesses 
big and small can bring us their most pressing D issues, status 
reports on their use of D, and so on.

## The Attendees

The following people attended the meeting:

* Mathis Beer (Funkwerk)
* Walter Bright (DLF)
* Johan Engelen (Weka)
* Dennis Korpel (DLF/SARC)
* Mathias Lang (DLF/Symmetry)
* Razvan Nitu (DLF)
* Vladimir Panteleev (Symmetry)
* Mike Parker (DLF)
* Robert Schadek (DLF/Symmetry)
* Bastiaan Veelo (SARC)

## The Summary

### Vladimir

Vladimir brought three issues to us that he said were not a 
priority, but had been a pain in his side. He had now encountered 
them in the code base at work, so he wanted to ensure they were 
at least on someone's radar. He stressed they were not super 
urgent. Two of them were codegen bugs, and one of those two was a 
regression. The third one he said was just weird.

* [`this` not captured by lazy 
expression](https://issues.dlang.org/show_bug.cgi?id=24050)
* [[REG2.101.0] Wrong code with temporary and tuple range 
foreach](https://issues.dlang.org/show_bug.cgi?id=23923)
* [Type list (tuple) not expanded in delegate during 
IFTI](https://issues.dlang.org/show_bug.cgi?id=21804)

### Bastiaan

Bastiaan began by telling us he was very happy that Dennis had 
started working with SARC. He then gave us an update on the 
conversion from Pascal to D.

He reminded us that in our October 2023 meeting, [he'd reported 
on a thread contention 
problem](https://forum.dlang.org/post/jucppgelbdvxvoubxzkr@forum.dlang.org) he'd encountered with allocations (which [he subsequently reported had been solved](https://forum.dlang.org/post/gguwkjyiduyxjilyluvf@forum.dlang.org)). Since then, he'd had a problem with his implementation of Pascal sets. Those were bit arrays, and the implementation had been a bit sloppy regarding allocations, but that was solved now, too. He said the `@nogc` attribute had helped track down where the allocations were happening.

He said that overall, prospects were good. But one thing he'd 
noticed was that when doing function overloads, you could have 
`inout` to account for `const`, `immutable`, and mutable function 
parameters vs return types, there was nothing comparable for the 
presence or absence of `@nogc` on parameters which were 
delegates. Especially in `opApply`.

Dennis said this was a longstanding issue. He thought Mathias 
Lang had a DIP for attribute polymorphism on delegates, and there 
had been other proposals. But they were kind of complex and so 
far hadn't gained traction.

Mathias said that, yes, [he had a DIP for 
it](https://github.com/dlang/DIPs/pull/198), but he wasn't 100% 
confident about it due to some corner cases. He hadn't had a 
reason to make the time to pursue it further. If anyone was 
willing to pick it up, he'd be happy to discuss it with them, but 
he didn't see himself getting back to it anytime soon.

Bastiaan said it wasn't a big problem, but especially if you 
wanted to overload `opApply`, you had to do it twice to account 
for `@nogc` delegates and those that weren't.

Dennis said that sometimes with templates you could have a 
generic template type for your return type. He didn't know if 
`opApply` supported that, but he thought it did. Mathias said 
templates were nice until you needed to use classes or virtual 
functions, then things would just break.

Walter asked Bastiaan to file a Bugzilla issue on this, and he 
said he would.

### Mathis Beer

Mathis said he'd been trying to push incremental compilation at 
work as everyone was switching to 32GB laptops. A problem he'd 
encountered is that once you pass the compiler a D file along 
with some other files, you'd then not be able to rebuild that 
file in isolation. The compiler would insert some symbols into it 
which would then be missing in another module, or the other way 
around.

They'd been working around this by tracking which D files they 
were passing to which command line. Then when they needed to 
rebuild that file in isolation, they'd go ahead and rebuild the 
other files it had been built with. So they were kind of taking 
the files as a set, then breaking it into two different sets and 
building both of them. The first time around, there was no 
speedup, but they were seeing an improvement the second time 
around.

The main problem they'd had with that was memory usage. Even with 
their laptops as large as they were, it was getting critical. 
Especially if they broke it up too far and had something like 
eight instances of DMD running or passing the same modules in 
parallel. They'd been optimizing stuff to have the right amount 
of template usage for a single build and take good advantage of 
the object and module caches. If they were passing the same 
modules eight times, it may be faster, but it wasn't better for 
memory usage.

He pasted [a Bugzilla 
issue](https://issues.dlang.org/show_bug.cgi?id=24459) in the 
chat and said we should treat it as low priority. He said the 
reported issue was what happened when you tried to rebuild a 
single object that was part of a larger build set earlier. He 
anticipated that once this was fixed, there would be three more 
issues waiting in the wings behind it. What they had right now 
did seem to be working.

As an addendum, he noted that [the mold 
linker](https://github.com/rui314/mold) was absolutely amazing. 
He said that anyone on Linux needed to try it. He thought it 
should be part of the distribution, as it was very fast.

Dennis agreed that mold was fast. He added that ryuukk_ had 
[published a forum 
post](https://forum.dlang.org/post/zmknwhsidfigzsiqcibs@forum.dlang.org) saying how you could speed up build times by using mimalloc rather than the glibc malloc. Dennis said he hadn't looked into it yet.

Johan asked if Mathis could share the build system he'd 
described. Mathis said that if it worked out, he'd do a write-up 
on the forums about how it worked. Right now, it was just a heap 
of scripts that he was the only person actively maintaining. He 
wouldn't want to impose them on the world. He said that if you 
had an estimate of how much memory you were going to use, it was 
the sort of thing that could probably included in dub fairly 
straightforwardly.

Robert asked if they had tried the combination of reggae and 
ninja. He said ultimately, they'd run into the same problem, but 
instead of doing it all themselves there might be something there 
that was good enough for them. Mathis said that the big problem 
with the build systems he'd considered with this, or at least as 
far as he was aware, was that when you were using DMD in this 
way, then you'd have a build rule that reacted to change in one 
of five source files by rebuilding five object files, only some 
of which would be outdated. He said that was generally hard to 
express.

Robert said he was using reggae with ninja on a pretty big 
project and it was working well. Mathis provided more background 
and explained that they'd run into an issue when passing `-op` to 
DMD to maintain the source paths on generated object files so 
that they'd end up with a directory tree in the build folder that 
matched the source tree. This worked pretty well most of the 
time, and without it, you sometimes ran into name collisions with 
the object files. The problem came in when using it with dub when 
you passed files with absolute paths. You might see the object 
files ending up in a folder where you didn't want them. He 
intended to file an issue about it but hadn't done so yet. That 
was why they were doing a lot of stuff manually at the moment.

### Johan

This was Johan's first time joining the quarterly, so he'd 
intended to just come in and listen and hadn't brought anything 
specific to discuss. He reported that Weka was still at LDC 1.30, 
which was the equivalent of DMD 2.100. He said it was common that 
they skipped multiple versions because the updates were too 
frequent, but they'd recently started upgrading to LDC 1.37 or 
maybe 1.38. They'd run into the usual problems, but fewer than 
he'd expected, so that was positive.

As a result of the upgrade process, he said we may have recently 
noticed a few extra bug reports on Bugzilla. These were for 
issues for which there were no easy workarounds. Overall, he 
thought it was working out pretty well. There'd been nothing 
completely unsolvable or very crazy yet, but so far they were 
still at the stage of fixing compilation errors and hadn't gotten 
to the point of running anything. He said that in the past, 
compiler upgrades would break the binary, but that hadn't 
happened the past couple of times.

He said that Weka had so much templated introspective code that 
they seemed to hit every corner case of the language. With every 
compiler update, a corner case may have changed. They were very 
sensitive to that.

As an example, he said they had a struct with an `opAssign` 
override for `const`. Then somewhere in Phobos, it broke. There 
was a range iterator testing for `opAssign`, then afterward it 
didn't perform a move, but instead did a `memcpy` on the object, 
and then complained that it couldn't copy into constant memory. 
So it had this check for the assignment, and it passed because 
this `opAssign` override on a `const` object was allowed.

He said that Jonathan had already come up with a fix for it, so 
he was happy that it had been noticed and acted upon so quickly. 
But it showed how they had all these funny things in their code 
base. So he had the idea that he should make a test file that 
collected all these "structures from hell" where they did strange 
things. Then if we were looking for an annoying structure to test 
against, we could look at this file where they had a bunch of 
them. And the list would link to the bugs that caused each 
structure to be included in it.

Walter said that would be great, and that's what the test suite 
was for. Johan said the difficulty was often in figuring out 
whether it was valid code or not. Should the language support it, 
or was it just too crazy?

Mathis Beer asked what they were doing with that `opAssign` 
overload, and Johan explained it. It was just one place in the 
code base where they were essentially using a mixin to fix up the 
structure to basically have tail const. He said that very often 
in these cases, he didn't even know how the whole thing worked. 
He'd just update the compiler and then encounter compilation 
errors across this huge code base with so much template stuff 
going on that it was usually kind of difficult to figure out what 
the error actually was.

He said that understanding what the code was doing was usually 
his smallest concern. Sometimes he had to, but he usually tried 
to avoid getting into the meaning of the code, because then he 
might try and fix it and then run into problems with changing 
program behavior. So he usually tried to reproduce what the old 
compiler was doing to see if that was correct or if it was 
actually a bug.

He said one thing that was nice to mention was that the compiler 
now complained if you had a static assert on just a string, where 
instead of `static assert(false, "message")` you were just doing 
`static assert("message")`. This had uncovered a bug in their 
code base where someone had done exactly that, so he was really 
happy that the compiler complained about it. But even in that 
case, he had to stop and figure out what was going on, and if he 
could fix it by just adding `false` to the assert. Would that 
break the code? Had people started depending on the current 
behavior? And with all the introspection going on...

Razvan noted that he'd always found the ability to have `const` 
or `immutable` on `opAssign` really odd. He'd raised it in the 
past, but at the time some people had a use case for it. They 
weren't modifying any data, but just logging calls to it. He 
still found it odd and thought that we shouldn't allow 
`immutable` or `const` on there. But people did crazy things so 
now we had to support it.

Razvan then asked Johan if there was anything we could do to make 
things easier for him. Obviously, we couldn't put the Weka code 
base into our testing pipeline. But maybe there would be some way 
that when we had patches we could automatically send them over, 
then if Johan had any errors he could reduce a test case. Maybe 
that would take some of the load off of his shoulders.

Johan said that DMD couldn't compile the Weka code base anyway, 
though he hadn't tried it recently. He'd been trying to minimize 
the delta between Weka's LDC and vanilla LDC, and now it was very 
small. He thought it might no longer be relevant for just testing 
whether compilation worked. So the first step would be to get 
very close to the upstream compiler being able to compile it. But 
not to generate the binaries---there were some LDC-specific 
things in there that DMD couldn't compile, so just being able to 
do semantic analysis on it would be great.

He thought they were getting there. Then someone like him could 
just do a compilation check once a month or maybe once every 
week. We wouldn't want it on every DMD update, for sure. They 
didn't have that even on LDC. But it would be nice to have it 
done earlier. Deciding if it was valid code or not would still 
take time, but it would be better than the current situation. He 
didn't check LDC betas against Weka just yet. That might be a 
good first step.

He said recent template emission changes in both DMD and LDC were 
an improvement for Weka. They used to need to fix or change LDC 
to match whatever the Weka build system expected with template 
emission. That would never work with the upstream tests. It was 
only valid for Weka and was invalid in general. But now because 
of these upstream changes in template emissions, they had changed 
their build system significantly. So maybe now upstream LDC could 
compile Weka's code base. He was more skeptical about testing the 
binaries.

Razvan said that DMD didn't get many patches to the backend. Most 
changes were in the frontend. It would be great if we could test 
against Weka somehow when making changes to the semantic 
analysis. Johan said it was good to keep reminding them to do 
these kinds of tests and not delay them as much as they do now.

Johan added that a big incident they'd had with binaries on a new 
release was when the object layout had changed due to a change in 
class layout. The monitor field had been shifted toward the end, 
and that had an unexpectedly big impact. That kind of thing had 
been so bad in the past that they had tooling to check it. Every 
class and structure is hashed by introspecting the binary layout. 
If anything changes, the build system flags it. At a deeper 
level, it should flag even when the compiler does something like 
changing the padding. He said that kind of thing can happen even 
on an LLVM update, or when changing optimization flags, so it 
wasn't just the frontend they had to think about.

Bastiaan pointed out that the policy now was we didn't want to 
break things as frequently, and there had been some reversions. 
So in incrementally updating, Johan might end up making changes 
that turned out to be unnecessary. Johan agreed. He said that if 
they increased the amount of testing, then instead of fixing the 
code, he'd just report it and then fix it upon a real release in 
case a change of fix got reverted.

But he clarified that in upgrading from LDC 1.30 to 1.37, he 
wasn't testing every release in between. There were some places 
where they were still using 1.24, but in general, the code just 
had to compile with one compiler version, although compiling with 
two was very good. Ideally, he'd prefer to be able to make the 
code changes, test with the old compiler to make sure nothing had 
changed in terms of performance or whatever, and then update to 
the new compiler without fixing any code. Then they'd have a 
little separation between code change and compiler change.

He said that Weka had a script that automatically selected the 
compiler for you based on whatever git hash you were on. This 
allowed them to use several versions but always ensure that a 
given version of the code base was compiled with the version of 
the compiler it was known to work with.

### Addendum

Johan's point about compiler versions reminded Bastiaan of an 
issue with dub dependencies. Sometimes maintainers pulled 
packages off of the dub registry, or they'd go inactive and code 
would stop compiling. He wondered if there was an established way 
to deal with that for commercial apps. He thought it might be 
best to make sure you had forks of all those packages. It was 
nice that you could configure dub to find the dependencies 
elsewhere.

Mathis Beer said that at some point, dub was changed so that you 
could query the server using an API, and it then became 
incompatible with just dumping a copy of the registry in a 
folder. But you could still configure dub to use a registry that 
was just file and folder-based, and he thought that's what they 
were doing at Funkwerk. Then you could write a script that just 
downloaded all the packages and stashed them in a folder 
somewhere so you'd always have a backup. Anything that Funkwerk 
used or installed was backed up to a local server.

Mathias Lang said that they had a monorepo at Symmetry and wanted 
dub to support that better. So there were probably some changes 
coming that would make that possible. The last release had made 
`dub fetch` project aware so that it would fetch all a project's 
dependencies. And then using `--cache=local` with it, it would 
store them in your project. There were more improvements they 
wanted to make.

## The Next Meetings
Our April Monthly meeting took place the following Friday. Our 
next quarterly was on Friday, July 5th.