D build and SCons [was Answers needed from those using D for Web Development, Web APIs and Services]

[H. S. Teoh]

On Sun, Dec 24, 2017 at 12:09:56PM +0000, Russel Winder wrote:
> On Fri, 2017-12-22 at 10:39 -0800, H. S. Teoh via Digitalmars-d wrote:
[...]
> > In the long term, I think an approach similar to tup will have to be
> > adopted. O(n) dependency scanning just doesn't cut it anymore for
> > code the size of today's large software projects. And with dynamic
> > dependencies (e.g. CTFE-dependent imports) that are bound to happen
> > in D code with heavy metaprogramming, there's really no sane way to
> > manage dependencies explicitly; you really need to just instrument
> > the compiler and record all input files it reads the way tup does.
> > I shouldn't be needing to write custom scanners just to accomodate
> > CTFE-generated imports that may change again after a few more
> > commits. It's SSOT (single source of truth) all over again: the
> > compiler is the ultimate authority that determines which file
> > depends on what, and having to repeat this information in your build
> > script (or independently derive it via scanners) introduces
> > fragility / incompleteness into your build system.
> 
> Again unless we do something nothing will change.
> 
> I am not sure you can get away from some element of O(n) behaviour if
> a build system is to detect what is to be rebuilt in a compile then
> link system. Obviously there are ways of minimising cf. Tup and Ninja
> vs.  Make and to some extent SCons. Tup still has a form of scan it is
> just very fast due to the use of the file system tools it uses.
> 
> So if SCons is to be abandoned for D builds let's agree that and got
> on with the tool that SCons and Dub are not.

OK, I may have worded things poorly here.  What I meant was that with
"traditional" build systems like make or SCons, whenever you needed to
rebuild the source tree, the tool has to scan the *entire* source tree
in order to discover what needs to be rebuilt. I.e., it's O(N) where N
is the size of the source tree.  Whereas with tup, it uses the Linux
kernel's inotify mechanism to learn about which file(s) being monitored
have been changed since the last invocation, so that it can scan the
changed files in O(n) time where n is the number of changed files, and
in the usual case, n is much smaller than N. It's still linear in terms
of the size of the change, but sublinear in terms of the size of the
entire source tree.

I think it should be obvious that an approach whose complexity is
proportional to the size of the changeset is preferable to an approach
whose complexity is proportional to the size of the entire source tree,
esp.  given the large sizes of today's typical software projects.  If I
modify 1 file in a project of 10,000 source files, rebuilding should not
be orders of magnitude slower than if I modify 1 file in a project of
100 files.

In this sense, while SCons is far superior to make in terms of usability
and reliability, its core algorithm is still inferior to tools like tup.
Now, I've not actually used tup myself other than a cursory glance at
how it works, so there may be other areas in which it's inferior to
SCons.  But the important thing is that it gets us away from the O(N) of
traditional build systems that requires scanning the entire source tree,
to the O(n) that's proportional to the size of the changeset. The former
approach is clearly not scalable. We ought to be able to update the
dependency graph in proportion to how many nodes have changed; it should
not require rebuilding the entire graph every time you invoke the build.


> […]
> > One thing I found annoying with dub was the sheer amount of time it
> > spent at startup to scan all dependencies and packages and possibly
> > downloading a whole bunch of stuff. The network latency really kills
> > the compile-test-debug cycle time.  I know there's a switch to
> > suppress this behaviour, but the initial dependency scanning is
> > still pretty slow even in spite of that.  When a 1-line change
> > requires waiting 15-20 seconds just to recompile, that really breaks
> > my workflow.
> 
> I have been dithering with replacing the use of Dub itself, with a
> SCons tool to replace Dub and work directly with the repository. Dub's
> build structure really isn't useful for anything other than using Dub
> as a built system.
> 
> Having two modes: update each time vs. only update when the developer
> requires it is important. Unless a version glob is used, checking
> dependencies should never take long.

Preferably, checking dependencies ought not to be done at all unless the
developer calls for it. Network access is slow, and I find it
intolerable when it's not even necessary in the first place.  Why should
it need to access the network just because I changed 1 line of code and
need to rebuild?


> > Plus, sometimes I *don't* want anything updated -- when debugging a
> > program, the last thing I want is for dub or the build script or
> > whatever to decide to link in a slightly different version of a
> > library, and suddenly I'm no longer sure if the new crash is caused
> > by the library or my own code, or the bug may now be masked by the
> > slightly different behaviour of an upgraded library.
> 
> Isn't this consequent on the Dub version specification? If a specific
> version is required this behaviour should not happen.

The documentation does not help in this respect. The only thing I could
find was a scanty description of how to invoke dub in its most basic
forms, with little or no information (or hard-to-find information) on
how to configure it more precisely.  Also, why should I need to hardcode
a specific version of a dependent library just to suppress network
access when rebuilding?! Sometimes I *do* want to have the latest
libraries pulled in -- *when* I ask for it -- just not every single time
I build.


[...]
> I am not a fan of Dub as a build system, but it appears to be the
> accepted standard, or in my view sub-standard. (Trying to develop GtkD
> code with Dub is a pain in the ####.)

AFAIK, the only standard that Dub is, is a packaging system for D.  I
find it quite weak as a build tool.  That's the problem, it tries to do
too much.  It would have been nice if it stuck to just dealing with
packaging, rather than trying to do builds too, and doing it IMO rather
poorly.


> Should the community push to ditch Make, CMake, SCons, Dub and use
> Reggae (and hence Tup or Ninja)?
> 
> Not a simple question. For example CLion requires CMake. CMake-D
> appears not to work so we can do D in CLion. Work on D in IntelliJ
> IDEA is progressing but is relatively slow due to relying on
> volunteers.  Compare Rust which is now officially supported by
> JetBrains. This makes a huge difference.
> 
> The develoment environment is almost as important as the programming
> language.
[...]

Honestly, I don't care to have a "standard" build system for D. A
library should be able to produce a .so or .a, and have an import path,
and I couldn't care less how that happens; the library could be built by
a hardcoded shell script for all I care. All I should need to do in my
code is to link to that .so or .a and specify -I with the right import
path(s). Why should upstream libraries dictate how my code is built?!

To this end, a standard way of exporting import paths in a D library (it
can be as simple as a text file in the code repo, or some script or tool
akin to llvm-config or sdl-config that spits out a list of paths /
libraries / etc) would go much further than trying to shoehorn
everything into a single build system.


T

-- 
No! I'm not in denial!