Dub, Cargo, Go, Gradle, Maven

[Russel Winder]

On Wed, 2018-02-21 at 09:30 -0800, H. S. Teoh via Digitalmars-d wrote:
> 
> […]

> I think the ideal situation straddles the divide between declarative
> build specs and a full-fledged general programming language.  You
> don't
> want it to get too general, lest you end up with the build equivalent
> of
> spaghetti code where the build script becomes unreadable and
> unmaintainable.  OTOH a purely declarative approach is limited by how
> well the DSL is designed.  An insufficiently-expressive declarative
> language leads to much frustration when you find yourself unable to
> express something that you need to do with your build.

Gradle with Groovy or Kotlin scripts, and SCons with Python scripts are
at the extreme, one I actually like. You have all the openness and
flexibility of a programming language to describe the project and the
build. But yes you have all the dangers of a programming language. But
with Python and Groovy, there is a philosophy of trusting the
programmer to do things in the simplest and most declarative way.
Obviously there are a lot of untrustworthy programmers out there, but
that can be solved with training and mentoring.

Dub, Cargo, and to a great extent Maven, give you tools to specify
declaratively the project and nothing else. All actions are pre-defined 
and you have to work within the convention or not at all. The tool
allows no trust of the people working on the project to do the right
thing.

Meson (I'm not sure about CMake, I do not like its DSL) uses a Python-
like fixed DSL written in Python, so not an internal DSL, but really an
external DSL. Fortunately it does allow escaping to Python, but I fear
they may close this incredibly useful loophole.

Reggae allows for a number of different programming language to be used
but I am not sure if it is SCons-style DSL or Meson style DSL, I need
to do more work with it to find out.

Dub chose to use JSON (personal comment: yuk) and then SDL (personal
comment: why not TOML) so as to have only key:value declaration of
project properties. OK so there is a Maven-esque project lifecycle with
hooks, but unlike Maven there appears no way of providing new
lifecycles. This plugin feature is the only way Maven survives. With
Gradle having Groovy (and Kotlin) specification obviates the need for
most plugins and yet there is also a plugin system. Gradle is a superb
example in my view of how to get it right. A declarative DSL for
describing all the project properties, which is all you need for a
project working with the standard convention.

A core question is should D be the language of specification. Reggae
certainly allows it and shows a way forward. But is it arguably better
to use Python or JavaScript (personal opinion: please no, use Python)
exactly because it is a dynamic language.

Taking all the lessons of Dub and Reggae is clearly important, but then
is the strategy to amend Dub and merge Reggae or start a new codebase? 

> Ultimately, at the bottom level, we're just dealing with DAGs, so at
> some level the build spec should simply be a bunch of node specs,
> where
> each node consists of some set of inputs, some set of outputs, and a
> computational black box (the build action) to get from the former to
> the
> latter. This black box can be anything at all, and should not be
> unnecessarily constrained.
> 
> On a higher level, though, specifying individual nodes may not always
> be
> desirable (e.g., you want to be able to express things like "all .d
> files in this directory", or "all recursive import dependencies of
> source file F").  Typically you'd want a slightly higher-level
> language
> for generating DAG node specs.  Here is where some serious thought
> needs
> to be put into designing a language that's both straightforward for
> common tasks, yet expressive enough to handle unusual tasks.  To
> abuse a
> metaphor, DAG node specs are the assembly language of build systems,
> and
> generally you want to "program" your builds in a higher-level
> language,
> but you could either end up with the build equivalent of C with the
> build analogues of unchecked array bounds, uncontrolled pointer
> arithmetic, and hundreds of gotchas completely non-obvious to the
> uninitiated, resulting in ugly, unmaintainable build scripts, or the
> build equivalent of D with its expressive power yet packaged in a
> clean
> syntax, resulting in clean build scripts that are comfortable to
> write
> and maintain.

I think SCons, Gradle, Reggae, and likely Dub, actually have all these
issues covered. Important to make them explicit, but not anything that
should be problematic.

> So far I've not yet decided which approach is better: using a
> full-fledged programming language as the high-level API, like SCons
> does, or a more constrained but more easily controlled (and
> implemented!) declarative DSL that is not necessarily Turing-
> complete.
> Either way, I prefer baking in as little special behaviour as
> possible,
> in the sense that any special capabilities like build-in source code
> dependency scanning or linker flag control ought to be built on top
> of
> primitives that are exposed to user build scripts, so that at least
> in
> theory, user build scripts could also attain to equivalent capability
> without needing to hack the build tool.  Generally, I dislike any
> disparity between what built-in constructs can do "magically" vs.
> what
> user code (build scripts) can achieve, because that usually means
> that
> things will work well as long as you remain within the parameters the
> author has conceived, but should you ever encounter a situation the
> author didn't anticipate, you're stuck up the creek without a paddle.

I still find Meson constraining for anything not fitting that standard
project model. Indeed it was the Meson developers who stated that for
one of my projects I had to use the "drop down to Python", there was no
other way of handling things.

Whilst I will use Meson for Debian related builds when I can (Meson is
approved, SCons is not) I am still an advocate for internal DSL in a
programming language, and make sure that good build people do the most
declarative, least complex thing.

> 
> T
> 
-- 
Russel.
===========================================
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20180222/1949a305/attachment.sig>