DUB - call to arms

[H. S. Teoh]

On Wed, Apr 17, 2019 at 04:51:57PM +0000, Dragos Carp via Digitalmars-d wrote:
> On Wednesday, 17 April 2019 at 16:03:24 UTC, H. S. Teoh wrote:
> > The idea behind this is that in order to avoid baking in support for
> > a multitude of build systems (make, scons, meson, gradle, whatever)
> > into dub, which will inevitably be unmaintained and out-of-date 'cos
> > is too much work to keep up with everything, we delegate the job to
> > individual packages.  There would be one package per external build
> > system, e.g., a 'make' package for make support, an 'scons' package
> > for scons support, and so on.  These packages don't have to do very
> > much, all they need is to (1) define a way of installing said build
> > system, in a way that can be used by dub via a standard interface,
> > and (2) export various command-line templates that dub will use for
> > invoking the build system.
> > [...]
> 
> Invoking the external tools is probably doable, the problem is
> consuming artifacts generated by these tools.

It's not really a problem, as far as dub is concerned.  All we need is
some way to export the list of generated artifacts.  E.g., if it's a
library package, all you really need is to export (1) the import paths
and (2) the path(s) to the generated .so / .a / .lib / .dll files, and
dub should be able to figure out how to make use of this library without
actually knowing what the build command does.

In other words, we hide the complexity of the build system under one or
more canned invocation commands (e.g., 'make', 'make clean', 'make
docs', 'make test', for a standard set of actions you might commonly
want to do), and have the wrapper package tell us what the build
artifacts are.


> Then there are the specialties of each tool: some distinguish between
> configuration (done once for a fresh build) and build, some are very
> good at tracking the dependencies and incremental builds, some
> rediscover the world on each invocation, some support cross-compiling,
> some use command line parameter for customizing the build, other use
> configuration files, some support debug builds, some support listing
> the build targets, some support a server mode, some require a server
> mode, some deprecate features over time, etc., etc. Finally, it's a
> mess.

I don't think we need to support all of those cases. As far as
dub-as-a-package-manager is concerned, all it really cares about is that
each dependency is some black box amorphous blob of files, with an
associated set of arbitrary shell commands for each standard action:
build debug, build release, test, query import paths, query library
paths, etc..  These actions should be standard across all codebases, and
it doesn't matter how they are implemented underneath. They can be
arbitrarily complex shell commands. The dirty details are abstracted
away by the build system wrapper packages.

As for cross-compilation, IMO this should have built-in support as an
optional feature. I.e., there should be a standard command for
"cross-compile to architecture XYZ", and packages can either support
this or not. If one or more dependencies don't support
cross-compilation, then you cannot cross-compile (with dub).


> There is an effort in Scala world to do something in the same
> direction (they have a lot of build systems):
> https://github.com/scalacenter/bsp/blob/master/docs/bsp.md. But this
> effort resumes itself to just IDE integration.
> 
> What I'm missing is the tool for building D projects, do you suggest
> that each project should use what it seams fit, or just stick with
> dub?

If you're asking me about the *current* state of dub, I have to say I
don't use it except to pull in code.dlang.org dependencies.  I.e., I
find it only useful as a package manager and nothing else.  As a build
system it does not support critical use cases that I need,[*] so I find
it completely unusable.

[*] Specifically, these are not supported by the current version of dub
AFAIK:

(1) Arbitrary codegen (i.e., build a subset of D files into helper
program H1, then invoke H1 on some data files to generate D code, then
compile the resulting D code into the final executable);

(2) Cross-compilation (e.g., Linux x86 host -> Android/ARM);

(3) Arbitrary preprocessing of data files (build a subset of D files
into helper program H2, then invoke H2 on some data input files to
produce final data files -- e.g., compress HTML/Javascript, rescale
images, etc.);

(4) Building of deliverables (e.g., pack build products X,Y,Z into a
.zip file, or build/sign an APK, or build a Debian .deb package, etc.).

These are all non-trivial tasks, and I do not expect that dub will ever
support all of them natively (though it would be nice if it allowed me
to specify custom commands to accomplish such tasks!).  That's why I'm
pushing for separating dub as a package manager from dub as a build
tool.  The package manager's responsibility is to resolve dependencies,
resolve version constraints, and download dependencies.  How these
dependencies are built should be handled by a *real* build system that
can handle (1) to (4) above.

That's why I'm proposing to encapsulate the task of building to a
wrapper dub package that abstracts away the complexity.  At the end of
the day, it just boils down to (1) you have a bunch of input files, (2)
there's some command that transforms these input files into output
files, and (3) these output files reside somewhere in the filesystem.
As long as you're told where the output files in (3) are, you know
enough to use this package.  What goes on inside (2) is irrelevant; said
command can be arbitrarily complex -- it can be an entire build system,
for example, that consists of many subcommmands -- but a package manager
doesn't need to (and shouldn't) care about such details.


> > It would help if somebody can provide a concrete example of a
> > complex dependency graph that requires anything more than a
> > straightforward topological walk.
> 
> I don't know if I understand you right, but probably having some
> generated code, or calling shell scripts with side-effects already
> complicates the problem enough.

This is only a problem if you're conflating package management with a
build system.  If you tackle solely the problem of package management,
then these are not problems at all.  As far as a package manager is
concerned, you have a root package P that has dependencies on some
packages A, B, C, that in turn depend on other packages D, E, F.  Your
job as package manager is to figure out which version(s) of A, B, C, D,
E, F to download.  Each package is an arbitrary collection of files,
with an associated command that transforms these files into build
products.  All you need to know is a string containing the command that
performs this transformation, and a list of paths to the build
product(s).  You don't care what files are in the package, really.  For
all you care they could be a bunch of URIs that point to remote objects,
or a bunch of encrypted data files, or a .zip file of an entire OS.
None of that is relevant.  All that matters is that there's a command
(specified within the package itself) that tells you how to transform
said files into the build products.  You just run this command and let
it do its job -- how, you don't care.  As long as the command produces
the build products, that's good enough.  Once it's done, you just query
for the path(s) to the build products, and use that in the next node up
the dependency tree.

It's not the package manager's job to stick its grubby hands into the
dirty details of the actual build process.


T

-- 
ASCII stupid question, getty stupid ANSI.