A brief survey of build tools, focused on D

[H. S. Teoh]

On Wed, Dec 12, 2018 at 10:38:55AM +0100, Sönke Ludwig via Digitalmars-d-announce wrote:
> Am 11.12.2018 um 20:46 schrieb H. S. Teoh:
> > [...]
> > Wait, what does --parallel do if it doesn't compile multiple files
> > at once?
> 
> It currently only works when building with `--build-mode=singleFile`,
> so compiling individual files in parallel instead of compiling chunks
> of files in parallel, which would be the ideal.

Ah, I see.  But that should be relatively easy to fix, right?


[...]
> There are the three directives sourcePaths, sourceFiles and
> excludedSourceFiles (the latter two supporting wildcard expressions)
> to control the list of files. Once an explicit sourcePaths directive
> is given, the folder that is possibly detected by default
> ("source"/"src") is also skipped. They are documented in the package
> format specs ([1], [2]).

Thanks for the info.


> > Also, you refer to "the output binary". Does that mean I cannot
> > generate multiple executables? 'cos that's a showstopper for me.
> 
> Compiling multiple executables currently either requires multiple
> invocations (e.g. with different configurations or sub packages
> specified), or a targetType "none" package that has one dependency per
> executable - the same configuration/architecture applies to all of
> them in that case. If they are actually build dependencies, a possible
> approach is to invoke dub recursively inside of a preBuildCommand.

Unfortunately, that is not a practical solution for me.  Many of my
projects have source files that are generated by utilities that are
themselves D code that needs to be compiled (and run) as part of the
build.  I suppose in theory I could separate them into subpackages, and
factor out the common code shared between these utilities and the main
executable(s), but that is far too much work for something that IMO
ought to be very simple -- since most of the utilities are single-file
drivers with a small number of imports of some shared modules. Creating
entire subpackages for each of them just seems excessive, esp. during
development where the set of utilities / generated files may change a
lot.  Creating/deleting a subpackage every time is just too much work
for little benefit.

Also, does dub correctly support the case where some .d files are
generated by said utilities (which would be dub subpackages, if we
hypothetically went with that setup), but the output may change
depending on the contents of some input data/config files? I.e., if I
change a data file and run dub again, it ought to re-run the codegen
tool and then recompile the main executable that contains the changed
code.  This is a pretty important use-case for me, since it's kinda the
whole point of having a codegen tool.

Compiling the same set of sources for multiple archs (with each arch
possibly entailing a separate list of source files) is kinda a special
case for my current Android project; generally I don't really need
support for this. But solid support for codegen that properly percolates
changes from input data down to recompiling executables is must-have for
me.  Not being able to do this in the most efficient way possible would
greatly hamper my productivity.


> But what I meant is that there is for example currently no way to
> customize the output binary base name ("targetName") and directory
> ("targetPath") depending on the build type.

But this shouldn't be difficult to support, right?  Though I don't
particularly need this feature -- for the time being.


[...]
> > Does dub support the following scenario?
[...]
> This will currently realistically require invoking an external tool
> such as make through a pre/post-build command (although it may
> actually be possible to hack this together using sub packages, build
> commands, and string import paths for the file dependencies). Most
> notably, there is a directive missing to specify arbitrary files as
> build dependencies.

I see.  I think this is a basic limitation of dub's design -- it assumes
a certain (common) compilation model of sources to (single) executable,
and everything else is only expressible in terms of larger abstractions
like subpackages.  It doesn't really match the way I work, which I guess
explains my continuing frustration with using it.  I think of my build
processes as a general graph of arbitrary input files being converted by
arbitrary operations (not just compilation) into arbitrary output files.
When I'm unable to express this in a simple way in my build spec, or
when I'm forced to use tedious workarounds to express what in my mind
ought to be something very simple, it distracts me from my focusing on
my problem domain, and results in a lot of lost time/energy and
frustration.


[...]
> BTW, my plan for the Android part of this was to add support for
> plugins (fetchable from the registry, see [3] for a draft) that handle
> the details in a centralized manner instead of having to put that
> knowledge into the build recipe of each individual project.
> Unfortunately I still don't really have time to do substantial work on
> DUB.
[...]

This would be nice, though without support for codegen tools and general
data transformation actions, and having a core design that, AFAICT, is
ill-suited to this sort of use, I'm afraid I can't see myself using dub
in any serious way in the foreseeable future.  The most I can see myself
using dub for, unfortunately, is just to fetch code.dlang.org
dependencies via an empty dummy project, like I've been doing for my
vibe.d project.

Though, on the brighter side, after updating dub to the latest version
yesterday I did note that there has been a noticeable performance
improvement.  I didn't measure exactly how much or how it compares to
alternative build tools that I use, but as far as dub-compatible builds
are concerned, things seem to be improving, and that's a good thing.
Even if I won't be able to reap the benefits myself.


T

-- 
All problems are easy in retrospect.