Phobos 3 Discussion Notes - 02-01-2024

[Adam Wilson]

On Saturday, 3 February 2024 at 19:08:34 UTC, Lance Bachmeier 
wrote:
> It doesn't need to if only the most recent version as of the 
> compiler release is supported.

This makes a number of assumptions about both the package itself 
and is inherently limiting. Problems:

1. Updating a package for security/bugs/etc. now requires a 
compiler update.
2. Chicken-egg problems. Compiler makes breaking change that 
breaks the package, but the package maintainer cannot update the 
package until after the compiler has shipped. Therefore the 
package, and all of it's users, are broken for an entire release 
cycle. This could theoretically be solved with extremely tight 
coordination with package maintainers, but nothing in D's history 
suggests that there is even a remote possibility of this 
occurring on even a semi-repeatable basis. The package 
maintainers are volunteers too, if they miss the deadline, broken 
code ships and people yell at us for shipping broken code.
3. More chicken-egg problems. The syntax you propose implies the 
execution of DUB in the background. OK, but DUB is built with 
DMD, so by this design, you need DUB to pull a Phobos package to 
build DUB. Yes, there are bootstrapping designs, but ask Iain or 
Kinke about the bootstrapping rules for GCC or LLVM some time, 
they are highly restrictive.
4. Enforces the use of DUB, no other tool/package system can be 
built or substituted without changing the compiler syntax. D has 
a relatively diverse build tool ecosystem, are we telling them to 
just go pound sand?
5. Not all packages conform to Phobos' standards. Many 
gc-allocate heavily, or use exceptions, etc. By doing this we are 
explicitly telling entire groups of the D userspace that chunks 
of Phobos aren't for them, but that we can't easily tell them 
which chunks those are because we don't control them, so now they 
have to go lobby individual maintainers to support them. The 
screaming about Phobos' problems are loud enough without inviting 
even more trouble by pulling in packages that were never intended 
or designed to be held to Phobos' standards.

And this doesn't address the fact that the first thing the 
community will ask for is the ability to specify package versions.

I can hear Atila (Mr. Build-Speed himself) warming up his fingers 
from here for an explanation as to why any one of the above four 
problems make this idea untenable. But all five make it a 
complete non-starter with everybody who maintains the compiler. I 
submit that there are more useful ways to expend energy than 
pining for something that is simply never going to happen.

> This is actually a non-issue for many packages on 
> code.dlang.org. I've been downloading the D source files and 
> putting them in my repos for years. I specified that this would 
> be for some of the popular packages, not arbitrary packages.

This implies that there are design constraints on the package 
then (no dependencies?), correct?

> I actually listed multiple good reasons in my original comment, 
> all of which you ignored, so I gave another.

You have correctly pointed out that I did not do a point-by-point 
answer to your claims so here you go. (Relevant snippet below 
with responses interspersed)

> There are good reasons to not add that stuff to Phobos.

I disagree, expanding Phobos expands the range of software that 
can be built without needing DUB, thus expanding the capabilities 
of software that can be built where DUB is unavailable or 
untenable, while avoiding the problems listed above.

> The standard to get something in Phobos is high enough that it 
> leads to a lot of wasted time for many people,

It is high, and there is a good reason for that. Phobos has to 
work everywhere, it has to meet a broad variety of use case, and 
user environments.

I do agree that there are adjustments that could be made. For 
example, we tend to spend a lot of time on unnecessary 
bikeshedding and in some cases bikeshedding is used to 
intentionally stall out PRs. So one change than can be made is to 
focus on more qualitative reviews and not allow bikeshedding to 
stall a PR. Another, more controversial, change that I've been 
considering proposing, but know is going to receive considerable 
resistance, is allowing PR's that do not fulfill all the 
environmental rules (like not using the GC or using exceptions) 
to pass and then inviting others to modify the code to fit those 
rules. This is a deeply controversial idea and the debates I've 
had that only nibble at the edges of it have been ... emotional.

> it doesn't bloat the compiler download size,

Just how much does the library code add to the download though? 
Most of the download size in the binaries, not the library code. 
Assuming an atrocious 5:1 text-compression ratio we would have to 
add 10MB of code to add just 2MB of download size. For reference, 
Phobos V 2.106 is 12.6MB on my NTFS partition. We could triple 
the capability of the standard library for the cost of an 
additional 4MB on a 32MB installer. In this modern era of 
Cable/DSL/Fiber I fail to see how an additional megabyte or two 
of download size is a serious concern, especially since 10-20MB 
of library code is a serious amount of additional utility that 
would be available OOTB.

> it takes advantage of work that's already been done,

This is true, but is the work that's been done of the quality we 
want to ship as a top-line package? Does it follow the standards 
and idioms that the users of Phobos are used too?

> and it could be done quickly, rather than the years it takes to 
> get stuff done otherwise.

This is likely untrue. First, even using DUB as an external 
executable, you still need to teach DMD how to execute DUB and 
parse the `dub -v` output. That is a non-trivial amount of work. 
And we would have to figure out how this interacts with 
traditional `dub` project files (if two different versions are 
specified, what happens? The interactions are non-trivial and 
will require significant amount of code and testing of the 
compiler. Then you have to teach the library which packages are 
acceptable and manually maintain lists of versions match which 
compilers.

> This...doesn't make sense. There was a complaint that library 
> writers have to support too many compiler releases. Well, 
> that's an easy problem to solve, which the open source world 
> has used for decades. Just say "we only support releases X and 
> Y". I even gave examples. Wouldn't take any work at all, just a 
> simple management decision, done.
>
> Now you're telling me I should just shut up because I'm not the 
> one doing any of the work that doesn't need to be done?
>
> I give up.

As I pointed out above, there is in fact significant work that 
would need to be done.