D Language Foundation Monthly Meeting Summary for May 2023
Mike Parker
aldacron at gmail.com
Fri Jun 23 14:32:51 UTC 2023
The monthly meeting for May 2023 took place on Friday the 5th at
14:00 UTC. It lasted about an hour and a half. This was the last
meeting before we started our new planning sessions.
The following people attended:
* Walter Bright
* Iain Buclaw
* Ali Çehreli
* Martin Kinkelin
* Dennis Korpel
* Mathias Lang
* Razvan Nitu
* Mike Parker
* Robert Schadek
## The summary
### Me
I started with an update on DConf submissions, reporting the
numbers and some of the details on who and what. At that point,
we had a total of 11 submissions from 8 people. I suggested we
could put a panel in one of the open slots if we needed to and
asked everyone to think of some ideas. (We ended up with 39
submissions from 21 people, or 24 if you count the additional
three members of the one team that submitted, and had no room at
all for a panel).
Next, I told everyone we had secured some [sponsorship funding
from Ahrefs](https://ahrefs.com/) and noted that I planned to use
it for BeerConf. I hoped to rent the same space we rented last
year at The Fox, so I detailed how I was going to work out paying
for it via our event planner. Last year, Funkwerk provided the
funding to the DLF, Átila and Walter each paid one or more nights
on their credit cards, and the DLF reimbursed them.
(Unfortunately, pub hire rates this year have massively increased
over last year, putting them well beyond our budget. We've fallen
back to the old way of designating a hotel as the BeerConf
location, this time Travelodge Central City Road. The sponsorship
funding will go toward our speaker reimbursement budget.)
Next, I gave an update on registrations. At that point, only one
person had registered. I then talked about some email exchanges
I'd had with some D shops about possible sponsorships. (Our
current headcount as of June 23 is 43. I'm anticipating a final
number in the neighborhood of 60-70, which would be in the same
ballpark as last year.)
Walter brought up [Sebastiaan Koppe's presentation from last
year](https://youtu.be/hJhNhIeq29U) on structured concurrency. He
said he'd like to see Sebastiaan there this year for an update on
the project, preferably as a follow-up presentation. He was
excited about the project and felt it was a big deal in general
and a big deal for D. Even if Sebastiaan decided not to move the
project forward, Walter felt we should find someone to take it
over. He said the ideas behind it were big and would be a nice
thing for us. (We subsequently confirmed with Sebastiaan that he
didn't have enough material yet for a follow-up talk).
I summed up by saying that DConf planning was on track and there
were no issues so far.
Later, I remembered to mention that I had Robert's GitHub to
Bugzilla migration script in hand. Now it was just waiting on me
to make time to test it out, so I was the blocker now rather than
Robert. He warned me about a command line option I should leave
off while testing. (You can read about the progress on that [in
the General
forum](https://forum.dlang.org/thread/pfpoeaoihsqpohumjvsi@forum.dlang.org) and track our progress [on our GitHub projects page](https://github.com/orgs/dlang/projects/23?pane=issue&itemId=29572916.)).)
I asked about some issues that [had been migrated to the tools
repository](https://github.com/dlang/tools/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc) from Bugzilla. Robert said that was not from him. Mathias jumped in to say that he had done it a long time ago when he took a stab at this. He had closed all of the Bugzilla issues he'd migrated, so there would be no duplications when migrating the open ones.
### Iain
Iain opened what turned into a 25-minute segment by saying that
he hadn't gotten much done since the middle of March as he'd
spent several long weekends on holiday. He'd released D 2.103.0
in April and 2.103.1 on May 1. He had also pushed out the first
beta of 2.104.0, and it was on track for release at the end of
the month ([he announced it on June
2](https://forum.dlang.org/thread/waigfuztqsqlhzoujcmz@forum.dlang.org)).
He said that GCC 13.1 had been released on April 26. That
included version 2.103 of the D frontend in GDC.
Walter thanked Iain for handling the releases for us. Martin
asked how long the release process takes nowadays. Iain said it
depends on if he hits an issue building the Windows release. The
Windows box, and *only* the Windows box, seems to hit a file
system sync issue. There's a D program, `copyimports.exe`, that
had replaced several make recipes to copy files during the build.
Sometimes when building the releases, `make` tries to call that
program before the linker has created the executable. When that
happens, it aborts and Iain has to run the release process from
the start again. [He had filed an
issue](https://issues.dlang.org/show_bug.cgi?id=23486) for it.
If everything goes smoothly, it's about a two-hour process. The
biggest part of that is waiting for the builds to complete, and
then waiting for the CI pipelines to finish. He had noticed that
the CI pipelines take longer now. The Mac pipeline had broken a
while back and Max had fixed it. Before the breakage, it took
about 30 minutes to complete, but since the fix, it had been
taking 50 minutes. He didn't know if the slowdown was due to the
fix or because the actual runners had slowed down. Martin didn't
think it had anything to do with the fix.
At a prompt from Martin, Iain went into quite a bit of detail
about the Mac CI pipeline. The slowdown comes when testing the
optimized build of dmd. It's two times slower than testing the
debug build. It's also running on Rosetta, so there could be many
potential reasons for the problem. He said it's a minor
annoyance, but someone with an M1 machine should look into it to
determine the cause.
Martin noted that they still hadn't found the root cause of the
problem that prompted Max's fix. It appears to be due to some
sort of change in Apple's libc. Things worked just fine on other
platforms and had been working fine on Mac for years until that
change.
Iain then talked about Cirrus CI's usage policy and how this
issue affects us there. This was followed by a discussion about
some of the different kinds of pipelines we have, the pros and
cons of different CI providers we've used/are using, and the
possibility of moving some of the pipelines to GitHub Actions.
Mathias noted that we are severely underutilizing GH Actions.
This was followed by a discussion of Apple's libc, other possible
causes of the slowdown, the cost of renting a dedicated M1 Mac,
more details about our CI pipelines, and other related topics.
We wrapped up by agreeing that we need to look into making more
use of GH Actions.
### Ali
Ali reported he was still using D to prototype things at work. He
then told us he had recently taken a trip to Germany and met up
with several D programmers while he was there.
### Martin
Martin said that unlike Iain, for whom everything had worked out
with the most recent GDC release, the LDC side was having some
issues. They had hit several regressions.
He described one example that had been reported where adding `-g`
to the LDC command line caused compilation to take 10 times
longer than it used to. At first, he wasn't sure if it was a
major problem or an outlier, but then Robert had seen a 50-times
slow down on the Symmetry code base. That was Martin's #1
regression to tackle. He said it was related to a change in LLVM
15 that hadn't yet been accounted for in LDC, and it was blocking
a new release. (He was able to resolve it [a few days after the
meeting](https://github.com/ldc-developers/ldc/issues/4354) and
[released the fix in LDC
1.32.2](https://github.com/ldc-developers/ldc/releases/tag/v1.32.2).)
For D 2.103 support, all of the tests were green on Linux except
on Windows. It was apparently related to a batch file generated
either by CMake or Ninja exceeding the maximum command-line
length.
Next, he brought up a topic that led us into a discussion about
how to interact with contributors. The gist of it is that someone
had submitted a PR that was an incorrect fix for a particular
issue. Most likely, the submitter didn't have the big picture and
was unable to see that the root problem was elsewhere. By
submitting the PR, he did get attention to the issue and someone
else solved it with the correct fix, but Martin felt he should
have been told sooner that the fix was wrong and why, instead of
being asked to, e.g., add missing tests.
Razvan explained why he had taken a neutral approach to that
particular PR instead of coming right out and saying it was
wrong. In short, he was trying not to upset the person who had
submitted it. Martin said he understood the intent, but what if
no one else had stepped in and mentioned it, and no one else had
provided a more appropriate fix, until after the submitter had
addressed any issues raised with the PR? He would surely have
been upset in that case. We should always try to be as polite as
possible, but if the approach seems to be wrong in the first
place, stating that up front is better than wasting his time.
Walter thought Razvan had made a good effort at trying to resolve
the issue without creating conflict. He said he often worries
about how to get things on track when someone submits a PR that
he thinks is the wrong approach. He cited an example where
someone had submitted a PR for something that Walter thought was
a good idea that had the wrong implementation. He had completely
failed to get that point across and ended up upsetting the
contributor. This is a tough problem that he knows he needs to do
better with.
Robert said he'd found in conflict situations, text-based
communication is bad. He said he'd had text discussions with
coworkers that seemed negative and weren't working out. In that
situation, just setting up a video call with the person changes
everything. It can be uncomfortable and weird just talking with
people you've never met, especially for techies like us, but
you're doing that in text already, and something is lost in text.
In doing that, he had found that the way he had read a coworker's
words was not the way the coworker had written them.
I noted that Razvan, Dennis, Walter, and I had discussed this not
long before. When a conflict arises with a contributor in a
discussion thread, invite the contributor to have a call so that
you aren't typing past each other. Iain brought up a quote (he
couldn't recall the source): "On the internet, no one can read
you speak subtle."
Robert said the invitation to the call should be done publicly in
the discussion thread rather than privately by email so that the
next person that comes along can see that ultimately there was a
resolution. Otherwise, you might resolve the problem, but then
you close the issue and, for whatever reason, don't leave a note
describing why. If someone sees an accepted invitation to a call
and the issue is subsequently closed, then it's safer to assume
there was a discussion involved. Walter said that was good advice.
(I should take this opportunity to note that improving the
environment around contributions is something we're aiming at as
part of our ongoing organizational development. We want to
encourage increased contributions while reducing the potential
for people to waste their time. One of the early steps is to
outline a policy on PR reviews, [a task you can track in one of
our GH
Projects](https://github.com/orgs/dlang/projects/24?pane=issue&itemId=29573012) once Razvan and Dennis get started on it.)
### Mathias
Mathias said he had written a whole 20 lines of D over the past
month. He still wanted to get back to dub but hadn't had much
time. As such, he had nothing to report.
### Robert
Robert talked about the 50-times slow down he'd run into at work,
and thanked Martin for helping him track the problem down.
Next, he said he'd made the work project quicker this week by
doing quite a bit of refactoring of [his graphqld
library](https://github.com/burner/graphqld). He said it's still
too slow, so the compiler is too slow. Martin disagreed that the
compiler was at fault, noting how Robert's project takes longer
to compile than a full repl (an internal Symmetry project) that
takes 70 seconds. He doubts it's the compiler that's off. Robert
said that he had seen a 100% speedup on his machine after
refactoring, getting down to a one-minute compile from two. An
incremental rebuild with Ninja was still 12 or 13 seconds.
I asked if he had run LDC's time trace on the project. He said he
had. The most time was spent in Phobos, but they're calls that
make sense. If he were to do those things himself, he'd just be
bloating his code and recreating Phobos. The project is 70K lines
of D at this point, and it's mostly templates. He feels D would
be much more competitive in the language space if such a project
took less than a second to incrementally compile. He wouldn't
want to do something like this with Rust because it would take
forever to compile. Go would be much faster, but then you
wouldn't get the nice templates, and you'd have a much larger
project.
Walter noted that some particular Phobos templates had been
reported as taking too much time, and he'd been able to speed
some of those up with new implementations. Sometimes when you
profile, it may just be one template, or a handful of them,
giving you speed problems.
Robert explained that graphqld takes so much compile time because
it introspects all of the types it's given and creates an
internal AST that's then used to generate the reflection API and
a schema file that he needs for the front end. His refactoring
got that down from happening three times to one time. That in
itself is taking quite a while. He has to do it at least once,
and unless he writes it all out by hand as the graphql libraries
in other languages do, he doesn't see how he can make it faster.
That's the lure of D's introspection: you just give it the entry
point and it gets all of the links in the graph, and all the
types, and gives you something else.
As a user of the language, he shouldn't have to care about the
compile-time performance of a feature of the language. If he
wants to write a 10,000-line template function, it shouldn't be
much different from writing a 10,000-line normal function.
On the topic of compile times, Walter said that in trying to
resolve a compiler bug, he'd discovered that CTFE was being
called three times for each use of CTFE. He hadn't yet been able
to track down why it was doing that, but we could potentially be
looking at a situation where he's able to triple the speed of
CTFE if he can resolve it.
Finally, Robert thanked Dennis [for his awesome
video](https://youtu.be/b8wZqU5t9vs), saying he learned something
from it.
### Razvan
Razvan had been focused on fixing regressions and compiler ICEs
while managing the PR queues and going through Bugzilla issues.
He had been looking at statistics on the issues and found the
total number of open issues was on a clear downward trend. At the
time of the meeting, we had gotten down to a total we last saw in
2018.
He said the Phobos PR queue was pretty quiet, with only two or
three PRs submitted per week. There was much more activity in the
compiler repository, with around 20 or 30 PRs per week. Most of
the dmd PRs in the past month had come from Razvan, Dennis, and
Walter, and only a few from other people. But in the past week,
there had been a sudden spike in activity from other people. He
wondered who we could assign tasks to outside of the people in
this meeting once we set up our task lists.
### Dennis
__Broken nightlies__
Dennis reported that the nightlies were broken because FreeBSD
switched from LLVM 9 straight to 15 and the old package was no
longer installed. Martin had been working on that for LDC, and
Dennis asked if Iain had run into that problem with the last DMD
release. Iain had not. The only thing he'd noticed was that LLVM
1.32.1 was missing, so he'd had to use the .0 release to build
DMD.
As for building on FreeBSD, Iain said he builds on his own box.
He [linked the line from the release
builder](https://github.com/dlang/installer/blob/d761b194c513fa9ab2814f1e4c84b1c56e7be380/create_dmd_release/build_all.d#L19-L23). Martin noted the comments show that the VM this uses is a version of FreeBSD from 2020, which is why it still works. Iain checked and found the latest configuration available on Vagrant was from FreeBSD 12.4 from January of this year, so he could switch to using that for building releases. Or he could jump to 13. If he updates, maybe he'll hit the same issue that was killing the nightlies.
Martin said it's always nice if the CI scripts use echoing. It
comes in handy precisely in this kind of situation. Concerning
FreeBSD, he had no idea what they'd changed. He didn't use it
himself outside of LDC CI. Somehow they changed *extremely*, even
on FreeBSD 12, a years-old version. They didn't just bump the LDC
package from 9 to 15, but they also removed the LLVM 9 package.
10+ are all available, but 9 is gone. There also was a major bump
in the GCC package. And they changed the "official" LDC version
to 1.32.0. He'd already encountered a problem with some sort of
change they made in the GNU tools that forced him to flip the
order of some switches for a trivial command. His experience with
backward compatibility on FreeBSD was horrible. He reiterated
something he'd said in the past: CI is a constant battle. Apple
suddenly made changes in something that had worked for years, and
FreeBSD has made changes in something that had worked for
years... it's always like this.
Dennis had opened [a PR to update the
installer](https://github.com/dlang/installer/pull/562). With
that, building now succeeded, but he was getting an error from a
later step in the process (see the discussion thread on the
linked PR). He had no clue how to debug that and could use some
assistance. Martin spent a little time looking at it but saw
nothing obvious.
I asked if we should take the FreeBSD builds out of the
nightlies. Dennis said he'd try that and see how it goes.
__More CI woes__
Dennis had been working on refactoring the backend, something for
which Walter had given his approval in our April meeting. He'd
been replacing C-style function prototypes with imports, and that
had revealed some missing `const` and `@nogc` attributes and how
incremental compilation is broken. He'd made nice progress with
that. Unfortunately, replacing a prototype with an import
sometimes causes the Azure CI to complain about being out of
memory. Either it's at the brink of its resource limits, or
replacing a prototype with an import somehow spikes memory usage.
There's barely an impact on memory usage locally. The CI for
Ubuntu sometimes times out.
I asked if we need to upgrade the Azure account. Dennis had no
idea what the current resource limits were. Martin said it's
mostly the same as GitHub Actions. They're built on the same
infrastructure and resource limits are the same. It used to allow
10 concurrent jobs at a time, but they doubled that to 20 last
year. They're using the same extremely slow Azure VMs.
Dennis said he was surprised by the OOM. Martin said the VM
probably had at least 8GB. He asked for [a link to the PR that
triggered the OOM](https://github.com/dlang/dmd/pull/15188).
Walter suggested switching to separate compilation and that
should cut down on the memory usage. The backend was originally
set up to be separately compiled. Dennis said currently it's all
compiled in one go. The front end is compiled separately. He
suspected that it wasn't a resource problem. Phobos should take
more memory to compile, but never hits an OOM. Something weird
was going on, but he didn't know what. Walter said the backend
had significantly larger functions than Phobos. I reminded
everyone that Dennis had said it was only happening on Azure and
not locally.
As we were talking, Martin had been investigating. He said it
wasn't only Azure, but also Cirrus, and only when using DMD as
the host compiler. With LDC, they were passing. It appeared to be
a DMD problem, but it made no sense to him at all based on the
diffs. It was astonishing that replacing a prototype with an
import could lead to this behavior.
No one had any ideas, so Dennis said he would work on something
else and defer this to later.
(Apparently, it was fixed, but I don't know the details.)
### Walter
Following on from his update in our April meeting, Walter
reported that his efforts at Twitter outreach had shown some
moderate success. Instead of tweeting once every few months as he
had before, he'd been trying to tweet something every day about
D, programming in general, and maybe some engineering anecdotes.
Next, he said he'd been working on ImportC. One frustrating thing
was that he had no idea how successful it was among users. He had
[posted in the
forums](https://forum.dlang.org/thread/u2el6n$1tee$1@digitalmars.com) asking people to try it and report any issues. That had resulted in a bunch of bug reports that he was fixing. That was kind of an endless whack-a-mole. There's always somebody using some silly C extension in their header files that's best removed, but he can't remove it so he has to find a way to make it work.
He'd recently done a PR to implement C vector types, which had
turned out to be a surprisingly small amount of code. That was
nice. But he doesn't have a feel for where we are in the utility
of ImportC.
He encouraged people to give ImportC a try and tell him how it
breaks, then he would try to fix it. He'd added the vector
instructions because of an issue Steven Schveighoffer ran into
when he tried it. He wants people to report even the small bugs.
Those are like a small rock in your shoe: it's easy to get out,
but annoying while it's there. He wants to get ImportC polished
even at that level.
Otherwise, he was just continuing the process of working on bug
fixes rather than enhancements. That may be reflected in the
statistics Razvan reported of the bug count going down.
Walter then brought up [the rejected enum inference
DIP](https://forum.dlang.org/thread/fdxkixkcgzrxveayybel@forum.dlang.org). He thinks an implicit `with` on case statements when the switch variable is an `enum` is a better alternative. He'd been thinking a lot about it, and it was still in the back of his mind. The problem with it was, where do you stop? If you have an expression, `a === b`, where `a` is an enum expression and `b` is an enum value, does that automatically imply a `with` on the `b`? I noted that Java stopped with the case. Walter thinks it's a good idea, he just worries that if we do it we'll be opening the door to all kinds of whacky things.
There had been a lot of trouble with using `format` with BetterC
and CTFE. It boils down to `format` allocating memory from the
GC. So he'd been going through the source of `format` to find out
where it's allocating memory and if it really needs to allocate.
He thought he could make some progress on that. For example,
converting an integer to a string always allocates, but there's
no reason for it to allocate because the amount of memory it
needs is always bounded.
Other than that, he had no momentous issues. He was just grinding
along, fixing problems one by one. He hoped to get ImportC's
problems into the background. As far as implementing C, it's
doing very well. It's just all those "nutburger" extensions
people use that cause problems.
Martin said he'd welcome getting rid of the C initializers in the
AST. He and Walter had talked before about converting them to D
initializers in the AST. Walter had said he'd do it, but we were
still waiting on it. This got Walter going on a rant about how
complex C initializers had become in C23. The spec on them had
blown up into pages and pages of descriptions and, even as a
language lawyer who's spent his whole career reading C specs, he
can't figure out what it's saying. These days, it's a fraud to
regard C as a simple language. And it's unnecessary complexity.
## The next meetings
We started our new planning sessions the following week, the
results of which I've already posted in the General forum. Our
next monthly took place on June 2 at 15:00 UTC and was the first
with expanded membership. I'll have the summary for that as soon
as I can.
As always, if you have an issue or idea you'd like to discuss
with us, please let me know and I'll bring you into the first
monthly I can. I want to emphasize that these are *not* formal
meetings. Our conversations are free, open, and casual. We're
eager for input from diverse sources, so if you feel like you
have something that can improve the D language, ecosystem, or
community, please don't hesitate to bring it to us.
More information about the Digitalmars-d-announce
mailing list