D Language Foundation October 2024 Quarterly Meeting Summary

[Mike Parker]

The D Language Foundation's quarterly meeting for October 2024 
took place on Friday the 4th at 15:00 UTC. It lasted about 45 
minutes.

Our quarterly meetings are where representatives from businesses 
big and small can bring us their most pressing D issues, status 
reports on their use of D, and so on.

## The Attendees

The following people attended the meeting:

* Walter Bright (DLF)
* Ali Çehreli (DLF/MBRDNA)
* Dennis Korpel (DLF/SARC)
* Martin Kinkelin (DLF/Symmetry)
* Mathias Lang (DLF/Symmetry)
* Átila Neves (DLF/Symmetry)
* Razvan Nitu (DLF)
* Mike Parker (DLF)
* Carsten Rasmussen (Decard)
* Bastiaan Veelo (SARC)

## The Summary

### Carsten

Carsten said they were doing a lot at work, but nothing special. 
They were trying to make the network more distributed and were 
working on consensus rules. He said it might sound boring, but it 
was exciting.

They were also trying to build a new way to do what we might call 
"smart contracts", except it wasn't really smart contracts. It 
was an execution tree with functions.

### Bastiaan

Bastiaan said work was progressing. The day before the meeting 
he'd resolved a bug that had been bugging them for two years. 
Things were getting misplaced in the Windows GUI when using D 
with MDI. It was due to an old Windows SDK.

He said that Dennis had been helping them find bottlenecks in 
compile times. Bastiaan was changing the transpiler because of 
that. In one program they saw a reduction from 28 minutes down to 
14 minutes.

Otherwise, they had no complaints about D.

### Dennis (wearing his SARC hat)

Dennis said they were using DMD at SARC. They couldn't use LDC 
because of a 32-bit stack bug. Debug builds for certain projects 
were 20 seconds, and then the release builds would be 30 minutes. 
They'd found that 25 minutes of that was spent optimizing five 
very large functions. That was because the DMD back-end optimizer 
had some quadratic algorithms that could take a really long time 
if you had, e.g., a complex `for` loop.

He'd been looking for ways to mitigate that and was wondering if 
there could be a threshold at which the back end could determine 
a function was too complex and decide not to optimize. Or maybe 
an attribute to prevent it. LDC had the `@optStrategy` attribute. 
He wanted to know in particular if Walter was down for something 
like that since he knew a lot about the back end.

Walter said there were some cases in which the optimizer could 
get into an infinite loop. They never told you about this in 
optimization books, but some of the optimizations would undo each 
other, then redo, then undo, then redo, and so on. The only way 
he'd found to combat it was to set a limit on passes through the 
compiler so that it would give up and continue on.

He suggested the easiest way to deal with it, since they'd 
identified the functions, was to break them into smaller 
functions. He asked Dennis how big they were.

Dennis said that some of them were translated from the original 
Pascal source. Pascal had some features that were translated into 
meta programming in D. For example, a large struct with a variant 
record would get translated into a 3,000-line D function because 
of template expansions, inlining, and such.

Walter said that Dennis could look at the optimizer and try to 
fiddle with the limit on optimization passes. But it was a 
difficult problem when the compiler was stuck in a loop of doing 
and undoing something. If a function was small enough, it 
wouldn't slow things down. A 3,000-line function was probably too 
much to handle, so breaking it up might be the best option.

Ali agreed with that approach. He suggested the compiler should 
print a message that announced it was giving up and that 
recommended the user split up the function. Walter said that 
could be done.

Walter said that right now it was counting the number of passes, 
but we could put a wall clock in there to tell it to give up on 
optimization after a certain amount of time.

Carsten liked the idea of a UDA telling the compiler not to 
optimize a function. That would be very flexible. Bastiaan noted 
that the user would need to know where to put it. Carsten agreed. 
Walter suggested the compiler could print a message when a 
function took too long to optimize.

Walter said another thing to do was to pass `-b` or `--c` to the 
compiler and look at the output. You might see it flipping 
between two states.

Dennis posted the following minimal reproduction in the chat:

```d
void main()
{
  for (int i = 0; i < 1; i++)
  {
  int j;
  static foreach (k; 0 .. 2000) j++;
  }
}
```

He said you could vary the upper range from `2000` to something 
else and see the quadratic explosion of slowdown. He saw it was 
going through those 2,000 things 2,000 times. He thought it was 
because of the way it was structured as a big comma expression.

Walter asked him to post an issue and he linked to an existing 
one (which is [now on 
GitHub](https://github.com/dlang/dmd/issues/19759), thanks to 
Robert Schadek).

Bastiaan asked what the `--c` switch was. Walter said it printed 
out what the optimizer was doing. Everyone said they'd never 
heard of it. Walter said he'd documented some of the internal 
switches in the past and had received some negative feedback for 
it, so he'd stopped.

### Ali

Ali said they were still using D at work and were happy with it. 
They were on an older compiler version, but that didn't matter.

He told us that Mike Shah had invited him and Steve Schveighoffer 
out to Yale to talk to some students there.

### Razvan

Razvan said he'd made a PR for an issue Manu Evans had posted 
regarding warnings for unused imports inside the compiler. It was 
easy for the simple cases, but there were some hard questions. 
What should be done with imports inside the body of a template? 
What about symbols from imports used in template constraints? 
Things were messy when you put the templates inside static 
conditionals. He'd made the PR and then couldn't even compile the 
runtime with it, since the build for it treated warnings as 
errors.

He knew Walter was against adding warnings to the compiler. He 
wasn't sure if it was worth trying to improve the implementation, 
but it had been a good experience because it showed that putting 
this into the compiler was doable. Implementing it instead via 
DMD-as-a-library was not doable because of the library's current 
interface. He asked if anyone had any thoughts on this.

Walter said that every time he wanted to use `printf`, he found 
that someone had removed the import for it. Every time he wanted 
to dump the AST trees, someone had removed the import for it. 
This was always irritating. He constantly put them back in and 
they were constantly removed. He was not a fan of the idea of 
removing unused imports. If someone did care about that, they 
could do it in their own code, but not in DMD. He kept those 
imports there for debugging.

Razvan said that was fine, but the issue was about having it in 
the compiler for those who wanted it. Walter said that meant 
another switch, and the more switches we had, the more problems 
we had.

Bastiaan suggested that imports used only for debugging should be 
`debug import foo`. Walter said that was a good idea.

Dennis said we didn't have a switch to remove unused imports now, 
but people were removing unused imports anyway. So something that 
indicated an import was unused, whether it removed them, put out 
a message, or added a `debug` prefix, would be useful to have.

Walter said you might have conditionally compiled code in there 
that needed particular imports.

Mathias thought it would be a good thing to implement and it 
would be good to make DMD-as-a-library work, but he wouldn't want 
it in the current development cycle. When you were in 
development, moving things around would get in your way. And what 
about versioned imports? For example, one that was only used 
inside `version(Windows)`.

Walter suggested it might be better suited for a style checker.

Razvan said that, based on his experience, he saw no way to solve 
this for all cases. But having the switch would give you the 
possibility to use it. And if you knew you had imports in static 
conditionals and versions and such, then you could choose not to 
use it.

Walter pondered it briefly, then said he would be okay with 
adding a switch.

Carsten asked if it could be done by DScanner. Razvan said you'd 
need semantic analysis to do it. The way that symbol resolution 
and semantic analysis were currently implemented in DMD made it 
very hard, probably impossible, to create an interface that 
allowed you to implement it outside of the compiler.

Walter said he'd accept the switch and also accept that it 
wouldn't be perfect. In addition to the cases Razvan had 
mentioned, it was also possible to generate an import with a 
mixin. So we could just have the switch, accept that it served an 
advisory role, and just not worry about it.

Razvan said we could implement it only for normal functions first 
just to get it in. We could add solutions for more complicated 
cases down the road if we discovered any.

### Dennis (wearing his DLF hat)

Dennis said an old issue had recently [popped up in the 
forums](https://forum.dlang.org/post/dlaomdfxdzhxtpbsoixl@forum.dlang.org). It was about [instantiating a template through a constructor call](https://github.com/dlang/dmd/issues/18343). He'd put together [a prototype implementation](https://github.com/dlang/dmd/pull/16910) to see if it was feasible to do it without throwing or completely rewriting the template system. It seemed doable.

He wondered if this was worth pursuing, or if it were dead on 
arrival or had been rejected in the past.

Walter said his initial thought was, "Why?" Dennis said it was to 
reduce the number of trampoline functions in Phobos, like `tuple` 
forwarding to `Tuple`. So we could, e.g., get rid of `auto 
tuple(T...)(T args) => Tuple!T(args)` and just use `Tuple(1, 2, 
3)`. He noted that C++ had this.

Átila said it had been very popular in the C++ world. Walter said 
if C++ had it, maybe it wouldn't be a big deal for us to do it. 
And he couldn't believe he was saying that.

Átila said that a consequence of this was that you could write 
`vector v = {1, 2, 3}` and it would infer `int`, so you didn't 
have the template parameters anywhere.

Walter said if Dennis could make it work without significant 
disruption, then he thought it would be okay. He asked if it 
needed a DIP. Átila said it probably didn't. Walter said he'd at 
least like to see a description of it that fully described what 
it was so we could put it in the spec. Dennis said there was [an 
old DIP for it](https://wiki.dlang.org/DIP40) on the Wiki.

I said that in that case Dennis should just take that DIP and 
shape it up. He could put it directly into the DIP Development 
forum for feedback. Then after a bit, I'd assign a new number to 
it and we could move it on through.

Dennis said he wasn't sure how comprehensive of a feature it 
should be. DIP 40 had a whole set of rules to combine struct 
templates and constructor template arguments into one so you 
could have template constructors. Átila said it would be up to 
Dennis, but he advised against it.

Dennis said he had a simple implementation now that didn't do 
extra stuff like partial evaluation or anything advanced. It 
worked for the `Tuple` case. The DIP described a more expressive 
variant that allowed for more complicated versions. Was the 
simple version expressive enough?

Walter said he thought it was. If there was no use for those 
other complicated cases now, we should stick with the simple one. 
I suggested we could do the simple one first and add more complex 
cases later if needed. Átila agreed.

### Razvan

Razvan brought up an incident that had happened recently. Some 
PRs had been merged that were aimed at organizing some of DMD's 
source modules into packages. He said this had broken some of 
Walter's PRs, and Walter didn't understand why it was being done. 
It had also caused issues for Razvan's student who was working on 
DMD-as-a-library. The student had to update to the latest to fix 
the breakage and would then have to go back and rebase everything 
if the upstream PRs were to be reverted.

Razvan asked if we could decide on something here and say that it 
wouldn't change for the next 10 years. People had been arguing 
about packaging for a while.

Walter said that packaging didn't really help because everything 
was importing everything else. He felt packaging was better 
restricted to cases where module imports weren't going sideways, 
or deeper into the package heirarchy. For example, he tried to 
have the root package not call anything in the common package, 
and nothing in the back-end package calling anything in the front 
end. That kind of thing forced you to think more about 
encapsulation than just shoving things willy-nilly into 
subdirectories.

Martin asked if the student had to track master closely or if he 
could stick with stable. With LDC, he was able to go with stable 
bumps every two or three months so he didn't have to track master 
all the time.

Razvan said that at the start they'd been making a lot of PRs to 
DMD because they were modifying the interface. Because of that, 
they'd needed to track master. They hadn't needed to submit DMD 
PRs for a while, but still wanted to create a better interface, 
so still needed to track master.

Walter said there was an effort to divide DMD into modules that 
were not mutually dependent. That had been going on for a while 
and had made good progress. Packaging things would be better done 
once that was completed. For example, he had spent a lot of 
effort on making the parser independent of everything else, so 
the parser and the lexer could go into separate packages now. He 
didn't think he'd gotten that far with the rest of the stuff.

Dennis said that the packaging PRs hadn't just been shuffling 
stuff. Some of it was about reduced visibility. Some `public` 
things had been changed to `package`, so the packages were 
already somewhat separated. Mathias agreed and noted that in some 
cases, from what he could see in the PRs, some symbols had been 
moved to where they were used and their imports removed. That 
sort of thing was still useful.

Walter said before we packagized things, we needed to do the 
groundwork to make sure we had actual packages and not window 
dressing.

Razvan thought that discussions about visibility weren't really 
relevant to DMD's code base. He thought `private` was important 
when there was an interface through which you wanted to prevent 
access to certain things, but sometimes in DMD it was annoying 
when things were private. You might really need it in another 
module, so you'd just make it public. He didn't see that kind of 
refactoring as a big gain.

He thought one thing packaging really helped with was when you 
had 150 modules in a single directory. For someone just getting 
started on modifying the compiler, it was hard to know where 
things were. Otherwise, he didn't really care about it. He didn't 
see any big advantages either way. All he wanted was for us to 
decide on one approach and stick with it.

Walter asked how many packages the PRs had created. Dennis said 
there were three: one related to scanning object files, one for 
the visitors, and one for C++ mangling. Walter suggested we stop 
there for now. Dennis thought that was where it stopped anyway. 
Those were the most obvious and natural. Everything else was 
pretty entwined.

Martin said he was only interested in a single package: a glue 
layer package to clearly delineate which part of the front end 
was shared by all the compilers and which was DMD-specific so he 
could get rid of it. Walter said that was a fair point. Dennis 
and Razvan agreed.

To close this out, Razvan wanted to verify that we weren't 
reverting the packaging PRs. Walter confirmed.

## The Next Meetings
Our October Monthly meeting took place the following Friday, 
October 11th. Our next quarterly was on Friday, January 3rd, 2025.

If you are running or working for a business using D, large or 
small, and would like to join our quarterly meetings periodically 
or regularly to share your problems or experiences, please let me 
know.