Dub, Cargo, Go, Gradle, Maven

[H. S. Teoh]

On Mon, Feb 12, 2018 at 10:35:06AM +0000, Russel Winder via Digitalmars-d wrote:
> In all the discussion of Dub to date, it hasn't been pointed out that
> JVM building merged dependency management and build a long time ago.
> Historically:
> 
>   Make → Ant → Maven → Gradle
> 
> and Gradle can handle C++ as well as JVM language builds.
> 
> So the integration of package management and build as seen in Go,
> Cargo, and Dub is not a group of outliers. Could it be then that it is
> the right thing to do. After all package management is a dependency
> management activity and build is a dependency management activity, so
> why separate them, just have a single ADG to describe the whole thing.

I have no problem with using a single ADG/DAG to describe the whole
thing.  However, a naïve implementation of this raises a few issues:

If a dependent node requires network access, it forces network access
every time the DAG is updated.  This is slow, and also unreliable: the
shape of the DAG could, in theory, change arbitrarily at any time
outside the control of the user.  If I'm debugging a program, the very
last thing I want to happen is that the act of building the software
also pulls in new library versions that cause the location of the bug to
shift, thereby ruining any progress I may have made on narrowing down
its locus. It would be nice to locally cache such network-dependent
nodes so that they are only refreshed on demand.

Furthermore, a malicious external entity can introduce arbitrary changes
into the DAG, e.g., hijack an intermediate DNS server so that network
lookups get redirected to a malicious server which then adds
dependencies on malware to your DAG.  The next time you update: boom,
your software now contains a trojan horse. (Even better if you have
integrated package dependencies with builds all the way to deployment:
now all your customers have a copy of the trojan deployed on their
machines, too.)  To mitigate this, some kind of security model would be
required (e.g., verifiable server certificates, cryptographically signed
package payloads).  Which adds to the cost of refreshing network nodes,
and hence is another big reason why this should be done on-demand, NOT
automatically every time you ask for a new software build.

Also, if the machine I'm working on happens to be offline, it would
totally suck to be unable to build my project just because of that.
The whole point of having a DAG is reliable builds, and having the graph
depend on remote resources over an inherently unreliable network defeats
the purpose.  That is why caching is basically mandatory, as is control
over when the network is accessed.

And furthermore, one always has to be mindful of the occasional need to
rollback.  Generally, source code control is used for the local source
code component -- if you need to revert a change, just checkout an
earlier revision from your repo.  But if a network resource that used to
provide library X v1.0 now has moved on to X v2.0, and has dropped all
support for v1.0 so that it is no longer downloadable from the server,
then rollback is no longer possible.  You are now unable to reproduce a
build you made 2 years ago.  (Which you might need to, if a customer
environment is still running the old version and you need to debug it.)
IOW, the network is inherently unreliable.  Some form of local caching /
cache revision control is required.


[...]
> Then, is a DevOps world, there is deployment, which is usually a
> dependency management task. Is a totally new tool doing ADG
> manipulation really needed for this?

My answer is: the ADG/DAG manipulation should be a *library*, a reusable
component that can be integrated into diverse systems that require it.
Multiple systems that implement functionality X is not necessarily a
valid reason to argue for merging said systems into a single monolithic
monster.  Rather, what it *does* suggest is to factor out functionality
X so that it can be reused across said systems.


[...]
> Merging ideas from Dub, Gradle, and Reggae, into a project management
> tool for D (with C) projects is relatively straightforward of plan
> albeit really quite a complicated project. Creating the core ADG
> processing is the first requirement. It has to deal with external
> dependencies, project build dependencies, and deployment dependencies.

Your last sentence already shows that such a project is ill-advised,
because while all of them in an abstract sense reduce to nothing but DAG
manipulation, that is not an argument for integrating all systems that
happen to use DAGs as a core algorithm into a single monolithic system.
Rather, it's an indication that DAG manipulation code ought to be a
common library that's reused across systems that require such
functionality, i.e., external dependencies, build dependencies, and
deployment dependencies.

It's really very simple.  If your code has function X and function Y,
and X and Y have a lot of code in common, it does not mean you should
write function Z that can perform the role of both X and Y.  Rather, it
means you should factor out the common parts into function W, and reuse
W from X and Y.  (Alas, the former is seen all too often in large
"enterprise" software, where functions start out being straightforward
with a clean API, and end up being a monstrous chimera with 50
non-orthogonal, sometimes mutually-contradictory parameters, that can
nevertheless do everything you want -- if you can only figure out what
exactly each parameter means and which subset of parameters are actually
relevant to what you want.)

Similarly, if you have systems P, Q, and R, and they all have DAG
manipulation as a common functionality, that is an argument for
factoring out said DAG manipulation as a reusable component. It is not
an argument for making a new system S that includes everything that P,
Q, and R can do. (Unless S can also provide new functionality that P, Q,
and R could not have been able to achieve without such integration.)


[...]
> (*) The O(N) vs. O(n), SCons vs. Tup thing that T raised in another
> thread is important, but actually it is an implementation thing of how
> do you detect change, it isn't an algorithmic issue at a system design
> level. But it is important.

The O(N) vs. O(n) issue is actually very important once you generalize
beyond the specifics of build dependencies, esp. if you start talking
about network-dependent DAGs.  If a task has a DAG that depends on, say,
100 network nodes, then I absolutely do NOT want the dependency
resolution tool to be querying all 100 nodes every time I ask for a
refresh.  That's just ridiculously inefficient.  Rather, the tool should
subscribe for updates from the network servers so that they inform it
when their part of the DAG changes.  IOW, the amount of network traffic
should be proportional to the number of *changes* in the remote nodes,
NOT the *total* number of nodes.

Similarly, for deployment management, if my project has 100 installation
targets (remote customer machines), each of which has 1000 entities
(let's say files, like data files and executables), then I really do NOT
want to have to scan all 1000 entities on all 100 installation targets,
just to decide that only 50 files on 2 installation targets have
changed.  I should be able to push out only the files that have changed,
and not everything else. IOW, the size of the update should be
proportional to the size of the change, NOT the total size of the entire
deployment.  Otherwise it is simply not scalable and will quickly become
impractical as project sizes grow.

If such considerations are not integrated into the system design at the
top level, you can be sure that there will be inherent design flaws that
preclude efficient implementation later on. IOW, DAG updates must be
proportional to the size of the DAG change. Nowhere must there be any
algorithm that requires scanning the entire DAG (unless the changeset
covers the entire DAG).


T

-- 
Guns don't kill people. Bullets do.