Why Phobos is cool

[Petar]

On Thursday, 25 June 2020 at 15:33:18 UTC, Paolo Invernizzi wrote:
> [snip]
>
> I understand your point of view and what you are trying to say. 
> In my opinion it's a chicken and egg problem, IMHO.

Yes, that's a fair assessment.

> My question is: why 90% of the code you need is not in Phobos,

I can go in more detail later, but let's just say that Phobos is 
a generic library aiming to solve general use-cases of most/all 
language users. In many domains, such as mine that's less than 5% 
of what I need, the rest is really domain-specific. And obviously 
there are more domain-specific libraries in other more popular 
languages.
Also, Phobos is very poor in terms of general application-level 
programming. Unlike e.g. Node.js or Flutter, it's neither 
sufficient for frontend (web/mobile/desktop) nor backend 
programming.

> and why is Phobos stagnating?

Several related reasons from the top of my head:
1. People are simply not interested in contributing (anymore).
A few hypothetical examples (I/me == random contributor):
1.a. As an individual developer I write code for fun, so I don't 
find value in going through the difficult (as it should be) 
process of proposing my code for std and passing through code 
reviews.
1.b. As an organization, we like to contribute to the community, 
and publishing on GitHub/GitLab/etc. + code.dlang.org is mutually 
beneficial: Other community members or companies can find value 
in what we do and can reports bugs and contribute improvements. 
However, contributing to Phobos takes just too long and also 
limits the potential number of contributors. Also we don't 
control the code anymore. So in general it's hard to justify from 
a business point of view.
1.c. (I guess that would be a fair approximation of the ndslice 
case): I'm a highly motivated individual who wants to contribute 
a foundational piece to the standard library, the positive 
effects of which would permeate throughout the ecosystem. Few 
problems:
- I want to support older/newer GDC/LDC versions (a few years ago 
that wasn't at all possible for upstream phobos)
- It's a nice start, but actually the reason I made this 
contribution is so I can advance in my domain (e.g. scientific 
computing, fintech, whatever), so now I need to to write domain 
specific libraries. Well actually, to support those libraries I 
need to make changes to the API. Well I can't do that easily 
anymore as I don't own the code and breaking changes are not well 
supported in Phobos (no SemVer). I also want betterC support, so 
I need to vendor (fork and rewrite) parts of druntime. The 
changes are good enough for me, but they can't be accepted 
upstream. And so my library now has non-Phobos dependencies, and 
so obviously it can't be part of Phobos.
1.d. I work on a security-sensitive library (e.g. crypto). I need 
to follow a responsible security vulnerability disclosure policy. 
Also when I have fixed the issue, and the changes are audited, I 
need to push a new release ASAP. A new compiler release (even 
point/bugfix one) typically has a latency of at least 1 month. It 
can contain also other unrelated breaking changes.

2. Failure to cultivate and *****retain***** community talent. By 
community talent I mean contributors who are *not* paid. E.g. 
Dicebot, JinShil, berni, Kenji, Joakim, and so on. Yeah, each 
case has its specifics, but if people with contributions of their 
caliber were to leave my company, I would be seriously worried 
and losing sleep (and selling stonks :D)
For example, not until recently (let's say 2017/2018), if a 
newcomer were to look at this page: 
https://github.com/dlang/dmd/graphs/contributors they would 
conclude that dmd was Kenji and Walter's project (and not Walter 
+ contributors), as Kenji had substantially bigger number of 
commits than Walter at that point of time.


> In the just released beta of DMD, there are zero Phobos 
> references in the changelog ...

Well, this varies with each release, but outside of bug fixes, 
refactoring or otherwise improvements to existing code and 
addition of fundamental vocabulary types (e.g. SumType), what 
exactly do you expect to see? Can you list exactly what kinds of 
contributions would you expect to see from non-paid volunteers? 
And I don't mean to say "non-paid" in a bad way regarding DLF. I 
mean that people who work for free are usually even more 
self-motivated then paid contributors. Why would anyone want to 
put new modules in Phobos, where they can develop them much 
faster in the open?

Also, contributing to the compiler and runtime has a much higher 
value than Phobos, especially given how easy and beneficial in 
many cases it is to replace Phobos with a code.dlang.org package.

Additionally, most of the code in dmd/druntime is internal and 
it's much easier to change than Phobos, which for the most part 
is comprised of public functions. In the compiler and the runtime 
you can make huge changes without breaking backwards 
compatibility at all.

>>> Anyway, if we can, we try to stick with Phobos, as long as we 
>>> don't have a particular problem to solve that needs an 
>>> external library: recent example, Martin std.io or SumType 
>>> instead of std.variant ...
>>
>> See, some of the problems in Phobos already have a good 
>> solution. The problem that we need to address is the trust.
>> The more high-quality and well-maintained libraries there are, 
>> the less cautious one would need to be before reaching to 
>> code.dlang.org.
>
> It's not only a problem of trust, it's a problem of integration 
> and maintenance also: for example, some days ago I tried to use 
> Martin and Steven std.io with std.concurrency, and I faced a 
> wall of problems related to non copiable struct, variants, 
> reference counting implementation, and so on ...

By trust, I also meant compatibility and general dependability.
For example, in the frontend world, even though Redux and React 
are developed by different teams (there's overlap but still), I 
feel like an illuminati apocalypse would be a more likely event 
than Redux somehow breaking React.js compatibility or vice versa 
:D

In the case of Steven and Martin's libraries, they're both pretty 
cool guys, whom I've met and chatted with at DConf, so I trust 
them as people, but they're both pretty busy these days (just my 
casual observation from GitHub and the forums), so I can't 
"trust" these libraries for critical stuff, as they don't have a 
high enough number of contributors who could take the lead in 
case Martin or Steven are not there.

I really hope we can improve as a community in this regard.

> I know that there're advantages in flexibility, but there are 
> costs also, for example, not everyone scouting for DLang as a 
> candidate for a task has your (great!) experience in managing 
> all the sort of problems that arises when trying to combine 
> different packages, or has the experience (or the time?) to 
> choose the "golden" one.
>
> Anyway, you are right, it's not a binary choice.
>
> /P

Yeah, flexibility almost always brings complexity (e.g. generic 
code can be easy to write, but sometimes it can be complex to 
debug). Really, what I mean is that we need to create a carefully 
curated collection of packages that work together and are 
maintained by a good number of qualified people. That's kind of 
the idea of dlang-community, but with a single large project like 
DCD in it (and its dependencies), it kind of twists the balance 
of contributions towards DCD. There's also the vibe-d and libmir 
organizations. However, we lack a broader collaboration and 
cooperation between them.

> managing all the sort of problems that arises when trying to 
> combine different packages

P.S. That is actually full-time job in the npm world :D
But it actually makes a lot of sense from business point of view. 
Nowadays, no company, no matter how huge could build in-house all 
that already exists (and is of high-quality) in the open-source 
world. It's cheaper to spend one week choosing 1 library from 
100+ then spending a few months to create another one that solves 
the same problem. Especially if your in-house developers need to 
move on a different project and then no one could maintain your 
in-house creation.

----

These are all really difficult problems to solve, but solving 
them can potentially create the biggest value of the community. 
Just like the Nodejs codebase represents <<<< 0.01% of the code 
of the whole npmjs ecosystem, the D language spec, dmd, druntime, 
phobos and the tools are absolutely foundational and critical, 
but would still represent a small fraction of the total value 
that our community can create.
We now need to focus on scaling the bigger part of the pie.