Why Phobos is cool

[H. S. Teoh]

On Thu, Jun 25, 2020 at 06:52:05AM +0000, Petar via Digitalmars-d wrote:
[...]
> I suggest checking http://mir-algorithm.libmir.org/.
> Ilya has been doing a truly amazing job at both developing,
> maintaining and optimizing this library.
> Also he has put a hard focus on memory allocators, @nogc and betterC
> (for exposing libmir to C and C++ users and also for scripting
> languages like Python or Julia).

I'm aware of Ilya's work, and have been meaning to look it up, but
haven't had the time. And also I'm working with different constraints: I
primarily work with batch-oriented computations, so @nogc is not really
on my radar (though I do try to minimize GC use for obvious reasons),
and betterC isn't really under consideration for what I need to do. So
these factors in Mir don't really stand out to me as much as they may to
others.


[...]
> Meanwhile, if you look at std.numeric, for past several years,
> basically all contributions but a few have been just around basic
> stuff like code style, imports and fixing a few compile errors in
> generic code.

I'm not sure what this has to do with anything. If an algorithm is
solidly implemented, why would there need to be a continuous stream of
changes?  And just because something is changing quickly, doesn't
necessarily equate to higher quality.


> Phobos is dead. There haven't been any new modules since ~2016.
> Actually the last one (IIRC) was std.experimental.ndslice, which Ilya
> contributed and then removed (1-2 releases after) from Phobos to start
> libmir.
> 
> Since std.experimental.allocator was added to Phobos circa 2015, there
> hasn't been a single Phobos or Druntime module using it (ndslice and
> checkedint don't count). Many packages on code.dlang.org
> enthusiastically embraced std.experimental.allocator only to get
> burned a few dmd releases after as Phobos doesn't follow SemVer.

Um... it's std.*experimental* for a reason?  Why are people surprised
when it changes?

OTOH, though, you do have a valid point about the distribution model.
Phobos is locked to compiler releases, and while that does have its
advantages in some areas, it also comes with disadvantages.


> The solution? Many of the most popular OSS D apps libraries (I can
> only comment on those) transitioned to using the Dub fork:
> https://github.com/dlang-community/stdx-allocator. How does it help?
> Well, OSS D libraries can't control the compiler that their users are
> using. For example, vibe.d tries to support the last 5 major versions
> of dmd/ldc, and Dub the last 15+ major versions. This means they can't
> depend on std.experimental.* as there's no telling what kind changes
> can happen for such a large timespan. Instead they can stick a
> particular version of https://code.dlang.org/packages/stdx-allocator
> in their dub.{json,sdl} file and be sure that it won't suddenly break
> their code.

This is a good case to make for prospective new Phobos modules to start
out as Dub packages, gain some real-world usage and traction, then
promote to Phobos.  Introducing packages directly into Phobos (via
std.experimental or otherwise) has proven to be high-risk, and in some
cases disastrous.


> The sooner we realize this and start collectively working on improving
> Dub and code.dlang.org, the better. Yes, Dub is not a great build
> tool. And so what? Most great build tools were not so great in their
> infancy. I doubt Scons was much better in 1999. If people continue
> avoiding Dub for whatever reason it won't get better by itself.

In 1999, SCons was not yet; there was its predecessor Cons, which has a
hard-to-maintain Perl script.  But the basic concepts that make it
awesome were already there.  This is a key point: it started out on the
right principles, and even though the implementation left some things to
be desired, it had the right footing and therefore had the foundation to
move on to be even better.  Dub, IMO, started out with the wrong
principles, or should I say, different principles from what I expect a
build tool to have, and therefore unless it undergoes a fundamental,
breaking rewrite from ground up, I doubt it will ever reach my
expectations of a build tool.

There are many advantages to integrating a build tool with a package
manager, I suppose, but TBH the way Dub does it is rather disappointing.
IMO, a package manager ought to be distinct from a build tool, even if
they do build(!) off each other.  Dub is actually not bad as a package
manager, but as a build tool I simply cannot vouch for it.

If you were to ask me, the first step to improving Dub would be to
separate the package manager from the build tool. As in, completely
extricate one from another so that they are usable as independent
pieces. (Note: this does not mean the user-facing driver cannot depend
on both at the same time.)

The next step, IMO, would be to separate the package manager code from
its unfriendly user-interface and make it dub-as-a-library.  That would
be something I can work with.


> There's no reason why Dub can't expose a lower-level DSL for precise
> target dependency graph building. It's just a matter of people making
> a sustained effort in contributing one.

TBH, I would much rather have dub-as-a-library than to try to
monkey-patch yet another hack on top of a foundation not designed to
support this sort of usage.


> The Rust community has realized that a slim standard library + a solid
> package manager is the way to go.
[...]

TBH, lately I've become skeptical of this approach.

More than a few times, I have been bitten by (non-D) libraries I depend
on (in my non-D projects) that have either become too obsolete and
doesn't compile anymore, or the new version has breaking changes that
cannot be easily worked around, or the network resource serving that
library has mysteriously vanished into the ether, never to be found
again, and also by external tools whose interfaces have changed, or
worst of all, whose behaviour has subtly changed in fundamentally
incompatible ways that cannot be remedied.  Also lately, shared
libraries that broke ABI compatibility just because I upgraded an OS
package, causing half of my programs to instantly become unrunnable.

As a result, I've become highly skeptical about the whole code reuse
mantra.  Where's the actual code reuse when it entails installing 5
different versions of the same library just because different programs
(or worse, different components of the same program) depend on different
versions?  This is dependency hell reincarnated.  What's the use of
depending on external programs that vanish into the ether, or get
changed in incompatible ways so that the only recourse is to keep a copy
of the old code around forever? (Fortunately, I only use open source
tools for this purpose; at least I keep a copy of the code around as
long as it remains compilable; imagine the disaster if it was a
proprietary tool that's no longer maintained, vanished into the ether,
or plain just doesn't run anymore on the latest OS.)

That's why I generally avoid libraries that have deep dependencies --
imagine all of the above multiplied recursively per dependent package.
Truly it's dependency hell reincarnated.  Instead, I look for libraries
like Adam Ruppe's arsd library, where most of the code has minimal
dependencies and can be used by literally just copying the danged file
into your source tree, where it will always be available, and is
guaranteed to be exactly the same code you last left it as and won't
subtly change and break stuff because some 3rd party network resource
somewhere out there in the ether decided to change it.  And where
calling a single lousy function does not pull in 2,345 irrelevant
dependencies that slow your build time down to an unusable crawl.

(And TBH, any library that requires pulling in that many recursive
dependencies makes me skeptical as to the quality of its design. IMNSHO
a properly-designed library ought to be as orthogonal as possible to
anything else, and to focus solely on providing that special
functionality that is its raison d'etre, and nothing else. It isn't as
though this is C, where composability is limited and a lot of external
things have to be baked into the library API just for it to be generally
applicable; in D we have metaprogramming, delegates, and all of that
good stuff that can be employed to eliminate all unnecessary library
dependencies and provide a powerful composable API that's completely
orthogonal to whatever's outside of its purview.  If a library requires
234 dependencies in order to do its job, perhaps it should get a new
job! :-P)


T

-- 
Живёшь только однажды.