I like dlang but i don't like dub

[H. S. Teoh]

On Fri, Mar 18, 2022 at 11:16:51AM -0700, Ali Çehreli via Digitalmars-d-learn wrote:
> tldr; I am talking on a soap box with a big question mind hovering
> over on my head: Why can't I accept pulling in dependencies
> automatically?

Because it's a bad idea for your code to depend on some external
resource owned by some anonymous personality somewhere out there on the
'Net that isn't under your control.


> On 3/18/22 07:48, H. S. Teoh wrote:
> 
> > As a package manager, dub is OK, it does its job.
> 
> As a long-time part of the D community, I am ashamed to admit that I
> don't use dub. I am ashamed because there is no particular reason, or
> my reasons may not be rational.

I have only used dub once -- for an experimental vibe.d project -- and
that only using a dummy empty project the sole purpose of which was to
pull in vibe.d (the real code is compiled by a different build system).

And I'm not even ashamed to admit it. :-P


> > As a build system
> 
> I have seen and used a number of build systems that were started after
> make's shortcomings and they ended up with their own shortcomings.
> Some of them were actually program code that teams would write to
> build their system. As in steps "compile these, then do these". What?
> My mind must have been tainted by the beauty of make that writing
> build steps in a build tool strikes me as unbelievable... But it
> happened. I don't remember its name but it was in Python. You would
> modify Python code to build your programs. (?)

Maybe you're referring to SCons?  I love SCons... not because it's
Python, but because it's mostly declarative (the Python calls don't
actually build anything immediately -- they register build actions with
the build engine and are executed later by an opaque scheduler). The
procedural part is really for things like creating lists of files and
such (though for the most common tasks there are already
basically-declarative functions available for use), or for those
occasions where the system simply doesn't have the means to express what
you want to do, and you need to invent your own build recipe and plug it
in.


> I am well aware of make's many shortcomings but love it's declarative
> style where things happen automatically. That's one smart program
> there. A colleague loves Bazel and is playing with it. Fingers
> crossed...

Make in its most basic incarnation was on the right path.  What came
after, however, was a gigantic mess. The macro system, for example,
which leads to spaghetti code of the C #ifdef-hell kind.  Just look at
dmd/druntime/phobos' makefiles sometime, and see if you can figure out
what exactly it's trying to do, and how.

There's also implementational issues, the worst of which is
non-reproducibility: running `make` after making some changes has ZERO
guarantees about the consistency of what happens afterwards. It *may*
just work, or it may silently link in stale binaries from previous
builds that silently replace some symbols with obsolete versions,
leading to heisenbugs that exist in your executable but do not exist in
your code.  (I'm not making this up; I have seen this with my own eyes
in my day job on multiple occasions.)

The usual bludgeon-solution to this is `make clean; make`, which defeats
the whole purpose of having a build system in the first place (just
write a shell script to recompile everything from scratch, every time).
Not to mention that `clean` isn't a built-in rule, and I've encountered
far too many projects where `make clean` doesn't *really* clean
everything thoroughly. Lately I've been resorting to `git clean -dfx` as
a nuke-an-ant solution to this persistent problem. (Warning: do NOT run
the above git command unless you know what you're doing. :-P)


> > I much rather prefer Adam's arsd libs[1], where you can literally
> > just copy the module into your own workspace (they are almost all
> > standalone single-file modules
> 
> That sounds great but aren't there common needs of those modules to
> share code from common modules?

Yes and no. The dependencies aren't zero, to be sure.  But Adam also
doesn't take code reuse to the extreme, in that if some utility function
can be written in 2-3 lines, there's really no harm repeating it across
modules.  Introducing a new module just to reuse 2-3 lines of code is
the kind of emperor's-clothes philosophy that leads to Dependency Hell.

Unfortunately, since the late 70's/early 80's code reuse has become the
sacred cow of computer science curriculums, and just about everybody has
been so indoctrinated that they would not dare copy-n-paste a 2-3 line
function for fear that the Reuse Cops would come knocking on their door
at night.


> It is ironic that packages being as small as possible reduces the
> chance of dependencies of those modules and at the same time it
> increases the total number of dependencies.

IMNSHO, when the global dependency graph becomes non-trivial (e.g.,
NP-complete Dependency Hell), that's a sign that you've partitioned your
code wrong.  Dependencies should be simple, i.e., more-or-less like a
tree, without diamond dependencies or conflicting dependencies of the
kind that makes resolving dependencies NP-complete.

The one-module-per-dependency thing about Adam's arsd is an ideal that
isn't always attainable. But the point is that one ought to strive in
the direction of less recursive dependencies rather than more.  When
importing a single Go or Python module triggers the recursive
installation of 50+ modules, 45 of which I've no idea why they're
needed, is a sign that something has gone horribly, horribly wrong with
the whole thing; we're losing sight of the forest for the trees. That
way be NP-complete dragons.


> > The dependency graph of a project should not be more than 2 levels
> > deep (preferably just 1).
> 
> I am fortunate that my programs are commond line tools and libraries
> that so far depended only on system libraries. The only outside
> dependency is cmake-d to plug into our build system. (I don't
> understand or agree with all of cmake-d but things are in an
> acceptable balance at the moment.) The only system tool I lately
> started using is ssh. (It's a topic for another time but my program
> copies itself to the remote host over ssh to work as a pair of client
> and server.)

I live and breathe ssh. :-D  I cannot imagine getting anything done at
all without ssh.  Incidentally, this is why I prefer a vim-compatible
programming environment over some heavy-weight IDE any day. Running an
IDE over ssh is out of the question.


> > You shouldn't have to download half the world
> 
> The first time I learned about pulling in dependencies terrified me.

This is far from the first time I encountered this concept, and it
*still* terrifies me. :-D


> (This is the part I realize I am very different from most other
> programmers.)

I love being different! ;-)


> I am still terrified that my dependency system will pull in a tree of
> code that I have no idea doing. Has it been modified to be malicious
> overnight? I thought it was possible. The following story is an
> example of what I was exactly terrified about:
> 
> https://medium.com/hackernoon/im-harvesting-credit-card-numbers-and-passwords-from-your-site-here-s-how-9a8cb347c5b5

EXACTLY!!!  This is the sort of thing that gives nightmares to people
working in network security.  Cf. also the Ken Thompson compiler hack.


> Despite such risks many projects just pull in code. (?) What am I
> missing?

IMNSHO, it's because of the indoctrination of code reuse. "Why write
code when you can reuse something somebody else has already written?"
Sounds good, but there are a lot of unintended consequences:

1) You become dependent on code of unknown provenance written by authors
of unknown motivation; how do you know you aren't pulling in malicious
code?  (Review the code you say? Ha! If you were that diligent, you'd
have written the code yourself in the first place.  Not likely.)  This
problem gets compounded with every recursive dependency (it's perhaps
imaginable if you carefully reviewed library L before using it -- but L
depends on 5 other libraries, each of which in turn depends on 8 others,
ad nauseaum. Are you seriously going to review ALL of them?)

2) You become dependent on an external resource, the availability of
which may not be under your control. E.g., what happens if you're on the
road without an internet connection, your local cache has expired, and
you really *really* need to recompile something?  Or what if one day,
the server on which this dependency was hosted suddenly upped and
vanished itself into the ether?  Don't tell me "but it's hosted on XYZ
network run by Reputable Company ABC, they'll make sure their servers
never go down!" -- try saying that 10 years later when you suddenly
really badly need to recompile your old code.  Oops, it doesn't compile
anymore, because a critical dependency doesn't exist anymore and nobody
has a copy of the last ancient version the code compiled with.

3) The external resource is liable to change any time, without notice
(the authors don't even know you exist, let alone who you are and why
changing some API will seriously break your code). Wake up the day of
your important release, and suddenly your project doesn't compile
anymore 'cos upstream committed an incompatible change. Try explaining
that one to your irate customers. :-P


> I heard about a team at a very high-profile company actually reviewing
> such dependencies before accepting them to the code base. But
> reviewing them only at acceptance time! Once the dependency is
> accepted, the projects would automatically pull in all unreviewed
> changes and run potentially malicious code on your computer.

Worse yet, at review time library L depended on external packages X, Y,
Z.  Let's grant that X, Y, Z were reviewed as well (giving the benefit
of the doubt here).  But are the reviewers seriously going to continue
reviewing X, Y, Z on an ongoing basis?  Perhaps X, Y, Z depended upon P,
Q, R as well; is *anyone* who uses L going to even notice when R's
maintainer turned rogue and committed some nasty backdoor into his code?


> I am still trying to understand where I went wrong. I simply cannot
> understand this. (I want to believe they changed their policy and they
> don't pull in automatically anymore.)

If said company is anything like the bureaucratic nightmare I have to
deal with every day, I'd bet that nobody cares about this 'cos it's not
their department.  Such menial tasks are owned by the department of
ItDoesntGetDone, and nobody ever knows what goes on there -- we're just
glad they haven't bothered us about show-stopping security flaws yet.
;-)


> When I (had to) used Go for a year about 4 years ago, it was the same:
> The project failed to build one morning because tere was an API change
> on one of the dependencies. O... K... They fixed it in a couple of
> hours but still...  Yes, the project should probably have depended on
> a particular version but then weren't we interested in bug fixes or
> added functionality? Why should we have decided to hold on to version
> 1.2.3 instead of 1.3.4? Should teams follow their many dependencies
> before updating? Maybe that's the part I am missing...

See, this is the fundamental problem I have with today's philosophy of
"put it all in `the cloud', that's the hip thing to do". I do *not*
trust that code from some external server somewhere out there isn't
going to just vanish into the ether suddenly, or keel over and die the
day after, or just plain get hacked (very common these days) and had
trojan code inserted into the resource I depend on.  Or the server just
plain becomes unreachable because I'm on the road, or my ISP is acting
up (again), or the network it's on got sanctioned overnight and now I'm
breaking the law just by downloading it.

I also do *not* trust that upstream isn't going to commit some
incompatible change that will fundamentally break my code in ways that
are very costly to fix.  I mean, they have every right to do so, why
should they stop just because some anonymous user out here depended on
their code?  I want to *manually* verify every upgrade to make sure that
it hasn't broken anything, before I commit to the next version of the
dependency. AND I want to keep a copy of the last working version on my
*local harddrive* until I'm 100% sure I don't need it anymore.  I do NOT
trust some automated package manager to do this for me correctly (I
mean, software can't possibly ever fail, right?).

And you know what?  I've worked with some projects that have lasted for
over a decade or two, and on that time scale, the oft-touted advantages
of code reuse have taken on a whole new perspective that people these
days don't often think about.  I've seen several times how as time goes
on, external dependencies become more of a liability than an asset. In
the short term, yeah it lets you get off the ground faster, saves you
the effort of reinventing the wheel, blah blah blah.  In the long term,
however, these advantages don't seem so advantageous anymore:

- You don't *really* understand the code you depend on, which means if
  upstream moves in an incompatible direction, or just plain abandons
  the project (the older the project the more likely this happens), you
  would not have the know-how to replicate the original functionality
  required by your own code.

- Sometimes the upstream breakage is a subtle one -- it works most of
  the time, but in this one setting with this one particular customer
  the behaviour changed. Now your customer is angry and you don't have
  the know-how to fix it (and upstream isn't going to do it 'cos the old
  behaviour was a bug).

- You may end up with an irreplacable dependency on abandoned old code,
  but since it isn't your code you don't have the know-how to maintain
  it (e.g., fix bugs, security holes, etc.). This can mean you're stuck
  with a security flaw that will be very expensive to fix.

- Upstream may not have broken anything, but the performance
  characteristics may have changed (for the worse). I'm not making this
  up -- I've seen an actual project where compiling with the newer
  library causes a 2x reduction in runtime performance. Many months
  after, it was somewhat improved, but still inferior to the original
  *unoptimized* library. And complaining upstream didn't help -- they
  insisted their code wasn't intended to be used this way, so the
  performance issues are the user's fault, not theirs.

- Upstream licensing terms may change, leaving you stuck up the creek
  without a paddle.

Writing the code yourself may have required more up-front investment
(and provoke the ire of the Code Reuse police), but you have the
advantage that you own the code, have a copy of it always available,
won't have licensing troubles, and understand the code well enough to
maintain it over the long term.  You become independent of the network
availability, immune to outages and unwanted breaking changes.

The code reuse emperor has no clothes, but his cronies brand me as
heretic scum worthy only to be spat out like a gnat. Such is life. ;-)


> Thanks for listening... Boo hoo... Why am I like this? :)
[...]

'cos you're the smart one. ;-)  Most people don't even think about these
issues, and then years later it comes back and bites them in the behind.


T

-- 
It is of the new things that men tire --- of fashions and proposals and improvements and change. It is the old things that startle and intoxicate. It is the old things that are young. -- G.K. Chesterton