A brief survey of build tools, focused on D

[Sönke Ludwig]

Am 11.12.2018 um 20:46 schrieb H. S. Teoh:
> On Tue, Dec 11, 2018 at 11:26:45AM +0100, Sönke Ludwig via Digitalmars-d-announce wrote:
> [...]
> 
>> The main open point right now AFAICS is to make --parallel work with
>> the multiple-files-at-once build modes for machines that have enough
>> RAM. This is rather simple, but someone has to do it. But apart from
>> that, I think that the current state is relatively fine form a
>> performance point of view.
> 
> Wait, what does --parallel do if it doesn't compile multiple files at
> once?

It currently only works when building with `--build-mode=singleFile`, so 
compiling individual files in parallel instead of compiling chunks of 
files in parallel, which would be the ideal.
>>> Then it requires a specific source layout, with incomplete /
>>> non-existent configuration options for alternatives.  Which makes it
>>> unusable for existing code bases.  Unacceptable.
>>
>> You can define arbitrary import/source directories and list (or
>> delist) source files individually if you want. There are restrictions
>> on the naming of the output binary, though, is that what you mean?
> 
> Is this documented? I couldn't find any info on it the last time I
> looked.

There are the three directives sourcePaths, sourceFiles and 
excludedSourceFiles (the latter two supporting wildcard expressions) to 
control the list of files. Once an explicit sourcePaths directive is 
given, the folder that is possibly detected by default ("source"/"src") 
is also skipped. They are documented in the package format specs ([1], [2]).

> 
> Also, you refer to "the output binary". Does that mean I cannot
> generate multiple executables? 'cos that's a showstopper for me.

Compiling multiple executables currently either requires multiple 
invocations (e.g. with different configurations or sub packages 
specified), or a targetType "none" package that has one dependency per 
executable - the same configuration/architecture applies to all of them 
in that case. If they are actually build dependencies, a possible 
approach is to invoke dub recursively inside of a preBuildCommand.

But what I meant is that there is for example currently no way to 
customize the output binary base name ("targetName") and directory 
("targetPath") depending on the build type.

>>> Worst of all, it does not support custom build actions, which is a
>>> requirement for many of my projects.  It does not support polyglot
>>> projects. It either does not support explicit control over exact
>>> build commands, or any such support is so poorly documented it might
>>> as well not exist.  This is not only unacceptable, it is a
>>> show-stopper.
>>
>> Do you mean modifying the compiler invocations that DUB generates or
>> adding custom commands (aka pre/post build/generate commands)?
> 
> Does dub support the following scenario?
> 
> - There's a bunch of .java files that have to be compiled with javac.
>     - But some of the .java files are generated by an external tool, that
>       must be run first, before the .java files are compiled.
> - There's a bunch of .d files in two directories.
>     - The second directory contains .d files that need to be compiled
>       into multiple executables, and they must be compiled with a local
>       (i.e., non-cross) compiler.
>     - Some of the resulting executables must be run first in order to
>       generate a few .d files in the first directory (in addition to
>       what's already there).
>     - After the .d files are generated, the first directory needs to be
>       compiled TWICE: once with a cross-compiler (LDC, targetting
>       Arm/Android), once with the local D compiler. The first compilation
>       must link with cross-compilation Android runtime libraries, and the
>       second compilation must link with local X11 libraries.
>        - (And obviously, the build products must be put in separate
>          subdirectories to prevent stomping over each other.)
> - After the .java and .d files are compiled, a series of tools must be
>    invoked to generate an .apk file, which also includes a bunch of
>    non-code files in resource subdirectories.  Then, another tool must be
>    run to align and sign the .apk file.
> 
> And here's a critical requirement: any time a file is changed (it can be
> a .java file, a .d file, or one of the resources that they depend on),
> all affected build products must be correctly updated. This must be done
> as efficiently as possible, because it's part of my code-compile-test
> cycle, and if it requires more than a few seconds or recompiling the
> entire codebase, it's a no-go.
> 
> If dub can handle this, then I'm suitably impressed, and retract most of
> my criticisms against it. ;-)

This will currently realistically require invoking an external tool such 
as make through a pre/post-build command (although it may actually be 
possible to hack this together using sub packages, build commands, and 
string import paths for the file dependencies). Most notably, there is a 
directive missing to specify arbitrary files as build dependencies.

Another feature that should be there to get this to a fully 
self-contained state is the ability to build `targetType "none"` 
packages that include custom build commands. Currently this will just 
result in a build error.

BTW, my plan for the Android part of this was to add support for plugins 
(fetchable from the registry, see [3] for a draft) that handle the 
details in a centralized manner instead of having to put that knowledge 
into the build recipe of each individual project. Unfortunately I still 
don't really have time to do substantial work on DUB.

So there is definitely still a enough valid criticism, especially as far 
as language-heterogeneous projects are concerned (see also [4]).

[1] https://dub.pm/package-format-sdl.html#build-settings
[2] https://dub.pm/package-format-json.html#build-settings
[3] https://github.com/dlang/dub/wiki/DEP3
[4] https://github.com/dlang/dub/wiki/DEP5