D Language Foundation April 2023 Monthly Meeting Summary
Mike Parker
aldacron at gmail.com
Sun May 14 12:47:59 UTC 2023
We held the monthly meeting for April 2023, on Friday the 14th at
14:00 UTC. It lasted just short of an hour and a half. We'd [had
a quarterly meeting the previous
week](https://forum.dlang.org/thread/yuewobigwhvrtfxlzxpk@forum.dlang.org). This was the first time wev'e held a monthly meeting in the month of a quarterly. Previously, we'd combined them, but [decided in our March meeting](https://forum.dlang.org/thread/aaebiekxqbwgxzkkynef@forum.dlang.org) to split them up.
The following people attended:
* Walter Bright
* Martin Kinkelin
* Dennis Korpel
* Mathias Lang
* Átila Neves
* Razvan Nitu
* Mike Parker
* Robert Schadek
## The summary
### Me
I opened the meeting with an update on DConf preparations. Most
of it was boring details. Other than that, the venue contract had
been signed and a down payment sent a couple of weeks before, and
Symmetry had reached an agreement with our event planner on the
terms and conditions of their contract. I was waiting on word
that it was signed (I have since received it).
I noted that Sönke Ludwig had contacted me before I left for my
trip in late March. He was doing some maintenance work on
codemirror.dlang.org and asked if I wanted to redirect the URL to
his new VPS or if he should move it onto a DLF server. Since this
is one of the services we need to move over eventually anyway, I
asked him to go ahead and do it. Vladimir Panteleev had already
set up the documentation tester on a DLF server, so I reached out
to him to see if we could move codemirror there or if we needed a
new one. After discussing it with Sönke, he said we had enough
resources for both services. So I set up Sönke with an account,
then he and Vladimir got it up and running while I was on my trip.
### Dennis
__CI failures__
Dennis started by saying that the CI was randomly failing again.
He didn't have a Mac, so he'd been unable to chase down the
problem. Random CI failures are a recurring problem. There are so
many checks, and he doesn't know who created them or who knows
exactly what the checks are doing. He wishes the tests had
someone responsible for them who he could turn to when they fail.
Walter asked who had previously been in charge of the tests.
Razvan said he didn't recall if one person was ever in charge of
them. At some point, someone decided it was a good idea to have a
particular test and it got added to the pipeline.
Dennis asked if we should only keep tests that have a maintainer.
Martin and Mathias quickly rejected that. Martin said the tests
are good. CI failures are usually caused by CI image bumps or a
PR. CI image changes are a PITA for LDC's tests, and PR-related
failures may not be easy to resolve, but failures are hardly ever
the fault of a test. And there's never been any specific person
responsible for any of DMD's CI systems. They just grew
organically. Then someone who knew the details moved on and no
one else knows them... it's a constant maintenance burden, but
it's worth the effort.
There was a bit more discussion about the maintenance burden,
after which I noted that this is the story of our ecosystem.
We're responsible now for things none of us set up, and we need
to get a handle on it all. Dennis agreed and added that the CI is
in a special position. When one of them is outdated, it doesn't
just sit there out of the way, it becomes an annoyance to
development.
(NOTE: This is one of the many aspects of our ecosystem that
we'll be working to improve [under our new
workflow](https://forum.dlang.org/thread/avvmlvjmvdniwwxemcqu@forum.dlang.org).)
__The -J compiler switch__
Next, Dennis said that, as he understood it, `-J` was added as a
measure against [string
imports](https://dlang.org/spec/expression.html#import_expressions) being used as a malware vector. As he sees it, string imports are no more risky than normal imports. Why can't they be handled the same way? Walter said the concern was that they could be used to read any file from the filesystem and bake it into the binary, and that could be used to steal data from the user's system. With normal imports, the compiler only reads D source files. It can't import arbitrary data. He then talked about some examples. What it comes down to is that there are so many clever ways to implement malware, he just doesn't want to open that door wide to the possibility.
Martin talked a bit about how Symmetry uses `-J` in their
codebase, and Átila told us how he used it a lot with reggae.
Dennis proposed that any directory given to `-J` be treated as a
root, and its subdirectories added to the search path. Martin and
Átila both said that the compiler already allows string imports
from subdirectories of any `-J` directory (when the full path is
included in the import expression).
__The DMD backend__
Dennis said that because the DMD backend was shared with the
Digital Mars C and C++ compiler (DMC), there was a rule in place
that changes to the backend couldn't break DMC, but there were no
checks for it. Now that ImportC is a thing, do we still need to
maintain DMD's backend with an eye toward not breaking DMC, or
can we say that DMD's backend is only for DMD and just leave DMC
frozen in time?
Walter said he had given up on maintaining DMC. The DMD backend
had already diverged, and he saw no further point in maintaining
DMC. Dennis said that will make maintaining the backend easier,
and we could remove the notes in the source about being careful
because of DMC. Walter agreed.
Dennis said that when building DMD, the backend is using some
translated header files where it's using extern declarations
instead of D imports. Those should be changed to more modern D
instead of this archaic header business. Walter agreed. He said
there's a lot of old-fashioned stuff in there. He hadn't taken
advantage of the separation to do any big refactoring but had
stopped backporting changes to the DMC backend a while ago.
Dennis next noted there were a bunch of `version(Mars)` and
`version(scpp)` blocks, and asked if they could be changed to
`version(all)` and `version(none)`, and the latter then removed.
Walter said he's not in a rush to do a big refactoring, but yes,
that's fine. He hadn't worked on the backend for a while other
than for bug fixes and had gradually changed some things to a D
style in the process. He gave a recent example involving some
changes he made when he fixed an inliner bug.
Walter said ImportC will probably never replace DMC, mainly
because DMC also compiles C++ code. Dennis added that DMC also
still has 16-bit support for the niche programmer.
(UPDATE: Dennis has since submitted several PRs removing function
prototypes from the backend, [starting with
pt1](https://github.com/dlang/dmd/pull/15111).)
### Mathias
Mathias said he had nothing to report. Walter asked what he had
been working on. He said he hadn't been working on any D stuff
lately. He still [had an open
PR](https://github.com/dlang/dmd/pull/14999) that fixes [a CTFE
bug](https://issues.dlang.org/show_bug.cgi?id=23785) related to
`-preiview=in`. In the PR thread, Martin had pointed out an issue
with it regarding the evaluation order of argument expressions,
and then the two had discussed it further outside of the PR
thread, where Martin explained how the order of destruction in
the presence of exceptions needs to be taken care of. That led
Mathias down a rabbit hole, as it seems there are cases where
destructors aren't called in CTFE, so the PR was stalled (and
it's still open as I write this). He plans to finish it up, as
his goal is to make it possible for `-preview=in` to be enabled
by default.
### Robert
Robert said he was going to ping me the following week about
finding some time to get the Bugzilla to GitHub migration. Other
than that, the compiler was still too slow.
(NOTE: He eventually emailed me everything I need to get the
Bugzilla to GitHub migration done. Now I'm the bottleneck. I'm
behind on everything right now, but I'm slowly catching up. When
I work my way to this task, I'm first going to test it out on a
private repo to make sure I've got everything in hand. Then I'll
make an announcement here in the forums with a date on which I
plan to start. When that date comes around, I'll open a forum
thread with periodic updates, e.g., "starting the migration of
tools issues now", and so on. We're almost there!)
### Razvan
__Fixing issues__
Razvan reported that he had mostly been fixing segfaults in the
compiler and had found that there were some ICEs in the C++
mangler. He'd submitted some PRs to fix those. There was [one in
particular](https://github.com/dlang/dmd/pull/15099) for which he
wasn't sure if the fix was the right one. He asked Mathias to
take a look at it as the original author.
Razvan had also been fixing some nosharedaccess issues. He had
seen a regression that Átila had reported, but other than that
there were no other open nosharedaccess issues. He expected there
would be more bugs out there, and Átila expected he'd find some.
__BetterC linker errors__
Next, Razvan said he'd been looking into BetterC errors. He then
described [an issue related to template instantiations in
CTFE](https://issues.dlang.org/show_bug.cgi?id=23799). When a
template includes runtime hooks, it's no problem in BetterC if
it's only used for CTFE and there's no code generation, but
sometimes the same instantiation gets used at runtime, the front
end doesn't catch it, and it results in linker errors. The
problem was uncovered [by a fix from
Walter](https://github.com/dlang/dmd/pull/14791) for [an issue
related to `@nogc` functions in
BetterC](https://issues.dlang.org/show_bug.cgi?id=23606), and
that PR also [introduced a
regression](https://issues.dlang.org/show_bug.cgi?id=23803).
Razvan then described a related BetterC problem. There are some
cases where the compiler tries to semantically analyze the body
of a generated function and marks the function as disabled if
something fails. He then described [a chain of consequences that
can arise in
BetterC](https://issues.dlang.org/show_bug.cgi?id=23806) in which
the generated function ends up instantiating a template that uses
a DRuntime hook, and that instantiation ends up causing linker
errors even though the generated function that instantiated it
was disabled.
He said the fundamental problem is that the compiler currently
has no way to reanalyze template instances once they've been
cached. He's not sure how best to solve this. He said he doesn't
like the way the compiler is going with special cases: if you're
in ImportC do this, if you're in BetterC do that, if you're in
CTFE do this other thing. Átila and Martin nodded in agreement.
Martin said that's his main problem with BetterC. It's full of
hacks in the front end and the glue layer.
Martin said he'd seen many missing symbol errors in BetterC. The
majority of them were related to template culling, which is a
compiler optimization. He described how it works, and noted that
it doesn't happen when using the `-allinst` switch. That's the
first thing he suggests trying when linker errors arise in
BetterC. When `-allinst` works, then the problem is probably just
template culling.
Regarding reanalyzing template instances, Martin said the
compiler does try to do it. That's the cause for a funny issue
where you have `static if(!__traits(compiles, something) { // do
the something that failed the compiles test }` and it only works
with the `static if(!_traits(compiles, ...)`. When analysis of
template instantiation fails and the compiler comes across a
second instantiation, it tries again and again, and sometimes it
works due to forward referencing issues (which he calls the root
of all evil and says is the main problem we have with the
frontend).
Átila agreed that the special case stuff is something we
shouldn't be doing. He thinks we should get to the point where
BetterC is a no-op. If your code doesn't require the runtime,
then it just shouldn't be linked. You shouldn't have to do
anything special to get that behavior. The only thing the
`-betterC` switch should do is tell the compiler to analyze your
code and tell you if something won't work in BetterC. Nothing
else would change.
Walter said the root of the problem is trying to use CTFE in
BetterC. CTFE uses the GC to allocate and is crippled without it.
Átila said that should be possible in BetterC, and Razvan agreed.
Walter said the problem there is: how can the compiler know that
the function is only used in CTFE?
This took the discussion down into the weeds, and I'm not going
to attempt to summarize all of it. Eventually, Martin said the
root problem is that the BetterC checks (for `TypeInfo`,
`ModuleInfo`, etc.) are done in the front end rather than in the
glue layer, and that's a problem for CTFE. If they were done in
the glue layer, it would be a PITA in that each compiler
implementation would have to do it then, but at least then you'd
only get issues if that thing is going to be codegenned.
Dennis asked if we should have a separate BetterC pass that does
the checks between Semantic3 and codegen. Martin said that's an
option at least. Átila reiterated BetterC should be a no-op.
Martin agreed that's a great long-term goal, but before we get
there we need to get rid of the module and type infos.
Átila took us back to the idea of indicating a function is
CTFE-only. Rather than add a new UDA, why not just use an
`in(__ctfe)` contract? Then it's easier for the compiler to
analyze, as there's no need to look at the body of the function.
Walter said he'd used `if(__ctfe} { // code } else assert(0)`.
Átila thinks the contract is a lot prettier than that, and Walter
agreed, and it does avoid the need for new syntax.
Átila said it also makes sense in that it declares the function
contract as being "I only work in CTFE, so don't bother calling
me otherwise". It's not even a hack. Dennis thought it is a bit
of a hack because you can still take the address of it. You have
to treat it as a static condition, but `__ctfe` is a runtime
variable. Regardless, Walter thought it was a very good idea
worth exploring.
Martin suggested using `if(__ctfe)` as a template constraint
instead of an `in` contract. Dennis liked that idea, but Walter
said he didn't want to start applying template constraints to
ordinary functions. That would be new syntax, and that's what the
`in(__ctfe)` was avoiding. Átila said it implies a template
function and probably isn't what we want.
__More on BetterC__
Razvan next brought up Teodor Dutu's work on [converting DRuntime
hooks to templates](https://youtu.be/dsa8GWL6TUo). An intended
side effect of this effort is that it should make things more
usable in BetterC, but sometimes, getting real work done in one
of these templates means calling into the C library (`memset`,
`memcpy`, etc). This is an issue when there's no standard C
library.
Walter said BetterC was never intended to be used without a
standard C library, so relying on the presence of those functions
is fine.
This prompted more discussion about linker issues with BetterC
and how to alleviate them. At the end of which Martin said
BetterC is a sledgehammer, and that's exactly why it has the
special case code Razvan mentioned before. We need to be working
to cut that down.
Walter said we should look into redesigning features so that they
don't need `TypeInfo`. Átila thinks `TypeInfo` should be opt-in
anyway. There are ways to use compile-time reflection to generate
all of that for us. Martin said the new templated hooks help in
that regard (though he later admitted that they were a bit of a
PITA causing some extra work for the GC-to-stack optimization in
LDC, which he went into some detail about). The big thing
hindering it is associative arrays. They heavily use `TypeInfo`.
He mentioned [Steven Schveighoffer's
implementation](https://github.com/schveiguy/newaa) which, as far
as he knows, is free of `TypeInfo`. Maybe eventually we can get
to use that.
### Martin
__LDC__
Martin said he was swamped with LDC-related work, with no time to
work on the front end. He still had a lot of catching up to do
with LLVM versions and the front-end version. LDC wasn't yet at D
2.103. The tests were looking quite good, but the previous
release had some problems. LLVM 15 brought some nasty surprises
that had only been caught after the final release of LDC 1.32, so
he needed to put out a point release (which [he did a few days
later](https://forum.dlang.org/thread/kcunuyrsogkhlljmyink@forum.dlang.org)).
He had noticed that the LDC beta releases didn't seem to be used
as much as they used to be. They used to see 1000 downloads of
beta releases in the first week, but with the most recent they'd
had only 300 or 400 in the first two weeks. It could be because
we have more frequent LDC releases now, so people just don't
bother with the beta anymore. He was thinking about doing some
announcements about how maintainers of dub packages can
contribute to compiler stability. For example, in GitHub actions,
there's the "latest beta" tag for LDC (he says it's a misnomer,
as it always refers to the most recent release, whether it's a
final or a beta). If people set up scheduled builds using that
tag every week or so and report issues when they happen, that
could improve the situation.
__DIP1000 issues__
Martin said they were still using D 2.100 at Symmetry because the
2.101 release came with lots of scope-related deprecations, which
are enabled even if you aren't compiling with `-preview=dip1000`
(they're errors with the preview, and [without it they're
deprecations](https://dlang.org/changelog/2.101.0.html#dmd.dip1000_deprecation_warnings)). In the beginning, there were about 1000 deprecations in their main project, which consists of around 150 dub packages overall. At the time of the meeting, they reduced the count to 200 by hunting them down and changing the code. Mostly, the deprecations were in third-party packages.
They had also encountered an issue related to using
`std.algorithm.all` with something else (he couldn't remember).
It prevents them from looping over an array of structs that have
`opEquals` in `@safe` code, and there's nothing they can do about
it on the user side. He said it might be a Phobos bug, but he
suspects it's a compiler issue regarding attribute inference.
He'd not yet had time to reduce it and file an issue.
Also, he had worked on fixing deprecations in [the concurrency
dub
package](https://github.com/symmetryinvestments/concurrency/pull/77). He got those fixed for a library using that package, but then trying to compile the concurrency package's unit tests segfaulted the compiler. This happened in both the D 2.101 and 2.102 versions of DMD and LDC. Again, he'd not yet had time to reduce it to file an issue. Dennis offered to take a look at it.
Átila said the main takeaway here is that dealing with
deprecations is time-consuming, especially when they're in
third-party code. And in this case, trying to find out why the
deprecated code would be inferred as `@system` instead of `@safe`
isn't always trivial. Maybe we should rethink DIP1000. Symmetry,
at least, is devoting resources to it, but most people would just
give up.
Dennis said he preferred using `return scope` as little as
possible and just letting the compiler infer it. It's annoying
opening BuildKite projects to fix related issues when someone got
`return scope` wrong because the compiler was wrong.
(There was more discussion about the value of `@safe`, and how
the GC and tools like ASAN can mitigate the need for it.)
__DIP1008 issue__
Next, one of Symmetry's projects uses DIP1008. It makes sense
there, as it's a project that interacts with Dot Net code, which
also has a GC, and you're not able to reliably suspend one GC
when the other one is active, so they're trying to avoid the D GC
in most cases. They still need to throw exceptions, so that's why
they're using `-preview=dip1008`. Now they want to statically
link that project into the main project. The problem then is that
all of the dub projects pulled in (around 300) will be compiled
with it. That leads to garbled exception messages, as
non-DIP1008-aware code doesn't prevent the exceptions from
escaping their scope.
Martin proposed coming up with an alternative DIP1008
implementation where the only change is that `nogc` checks are
disabled for `throw` expressions. He said the team told him
they'd be okay with leaving garbage lying around resulting from
exceptions. He thought it would be pretty easy to get that done.
He suggested implementing it as an experiment, then upstreaming
it if it works with their codebase. It would mean not relying on
`malloc` and reference counting, just allocating from the GC and
never freeing it. The GC is disabled in that project anyway.
Walter said he's kind of sorry that D has exceptions at all.
They're an unending source of problems. Using option types would
have been a better design choice in hindsight, but at the time he
was just following what C++ and Java did. One thing we might do
is to remove exceptions from Phobos so that you can reliably use
it without exceptions at all. Another thing is to continue
minimizing the use of the GC in Phobos.
Dennis said that there are places in Phobos where exceptions are
just ingrained in the API. He cited ranges specifically, where
you can just decode text and get a `UTFException`, or parsing
numbers could throw a `ConvException`. To change that would
require all users to unwrap the integers, unwrap the codepoints,
etc. Átila said we could use an `alias this` that behaves as it
does now and if the user forgets to unwrap, you throw an
exception then. Dennis said that was an interesting idea.
Walter gave the example of using `std.algorithm.chain` in
`std.path` to avoid allocating memory when concatenating paths
and returning the chain thing. He's wondering if there are other
opportunities like that in Phobos that can help us avoid memory
allocations and exceptions, but he hasn't gotten that far in
looking into how to eliminate exceptions. Átila said he'd been
thinking about copying Herb Sutter's idea for zero-overhead C++
exceptions. He said the basic idea is that you get rid of
exceptions while keeping the syntax. (Here's [a link to the
PDF](https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p0709r4.pdf)).
### Átila
Átila had to leave right after the discussion above on exceptions.
### Walter
Walter had been continuing his pause on implementing new features
and had been going through the ImportC bug list. The biggest
problem he'd run into is the pervasiveness of compiler extensions
in system header files, and it's almost always unnecessary. He
then went on an entertaining rant about it (a header will compile
on FreeBSD but not macOS, then a different header succeeds on
macOS and not FreeBSD; Microsoft is the worst offender; etc.).
ImportC was intended to be a standard C compiler, but you can't
compile any system headers with a standard C compiler. He ended
up implementing a lot of extensions to get things to work.
On the plus side, he'd made a lot of headway with it. For
example, he'd figured out how to implement statement expressions
with just a few lines of code. He was shocked it was possible
with only a few lines and was happy with the results. So he's
making good progress with getting ImportC to compile all these
random `.h` files. That and documenting this stuff was mostly
what he'd been working on. He hoped to get the ImportC bug list
down soon and turn to other bugs. (He posted about this in the
forums a couple of weeks later with [a call for help finding
problematic C
headers](https://forum.dlang.org/post/u2el6n$1tee$1@digitalmars.com).)
He then reiterated he'd like to reduce GC and exceptions in
Phobos and eliminate dependencies such as `TypeInfo`. The further
we can go with that the better.
Next, he said he'd stepped up his use of Twitter. He encouraged
us, and anyone with a Twitter account, to tweet more about what
we're doing with D. He'd looked into common hashtags with
programming posts and started using them more, and he also signed
up for the blue checkmark. He has since noticed an increase in
followers. If Twitter's not your thing, then consider posting on
whatever social media platform you prefer. This is our most
effective means of marketing. He and Andrei had tried various
approaches to publishing ads via Google, Facebook, and other
platforms in the past with no meaningful results whatsoever. What
he had found effective was writing articles and just posting on
social media about what you're doing. Just talk about the work
you do with D.
I noted we had discussed before the idea of Walter either doing
an occasional live-coding session or just recording himself
working throughout the week and editing together a video summary
of the work he'd done. Walter brought up what John Carmack did
back in the day with his `.plan` files to keep gamers informed of
the work he was doing. As Walter sees it, that was an early
version of what Twitter has become. It's a great way to let your
customers know what you're doing, and because they make a
conscious decision to follow you, that's not considered spam. He
decided to keep his tweets strictly related to programming
topics. He isn't going to tweet about politics or what he had for
breakfast. He'd followed several programmers because he wanted
their opinions about programming, only to be put off when they
tweet about everything else. He cited John Carmack again as an
example of someone who tweets mostly about programming topics. He
said that if you wanted to use Twitter for outreach, just keep it
on topics related to programming.
I mentioned that I'd like to record some more conversations like
the two I did with Walter last year ([Part
One](https://youtu.be/-kkMYJN3MnA) and [Part
Two](https://youtu.be/G6b62HmsO6M)). I have a list of people I
plan to contact for that.
Razvan doesn't use Twitter, but he's interested in what Walter
has to say about what he's working on. He suggested that Walter
post updates on the forums for people like him. Walter said he
could do more of that and that's a good idea.
Walter finished by saying that he has so much work to do and it
keeps growing all the time. He said he appreciated Razvan and
Dennis helping out with his pull requests. They'd been a big help.
## The next meeting and more
The next meeting was a monthly that we held on May 5 at 14:00 UTC
just before [our final IVY program
session](https://forum.dlang.org/thread/avvmlvjmvdniwwxemcqu@forum.dlang.org) with UCora. I'll try to have the summary for that ready before the end of this month. Then on May 12, we held the first of what will be weekly planning meetings. I'll write more about that in a separate post.
More information about the Digitalmars-d-announce
mailing list