Common makefile (gasp) stuff

[Andrei Alexandrescu via Digitalmars-d]

On 1/13/15 3:55 PM, H. S. Teoh via Digitalmars-d wrote:
> On Tue, Jan 13, 2015 at 03:20:12PM -0800, Andrei Alexandrescu via Digitalmars-d wrote:
>> Hey folks,
>>
>>
>> Over the time a number of stuff has become quite duplicated across our
>> makefiles for dmd, druntime, and phobos.
>>
>> These include fetching OS and model but (newer) general-purpose macros for
>> e.g. rebasing repos, see https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.
>>
>> I think it's time to reduce clutter and duplication by migrating such
>> common stuff into one common.mak file. Question is, where should that
>> file sit? One candidate is tools/ but it's not really a tool. Another
>> candidate is dmd/src/ because that would be the first thing anyone
>> depends on.
>>
>> Ideas?
> [...]
>
> Actually, my first reaction is to go the opposite direction. Currently,
> we have way too many complicated interdependencies between different
> repos. For example, the 'html' target of phobos/posix.mak depends on
> files in the dlang.org repos, and the website build scripts of dlang.org
> in turn depends on files generated by the Phobos ddoc targets.

Yah but when you look closer it's not that bad; just an attribution 
tradeoff. Building the phobos docs belongs with the phobos makefile 
because that's where the module informations resides; however, part of 
the build requires macro definitions that are at best consolidated in 
the website directory.

Building phobos's docs without dlang.org around is not sensible, just 
the same as building dlang.org without phobos. So I'm not too worried 
about that.

If someone builds phobos without caring about the docs, that's fine as 
long as they don't try to build html.

One simplification would be to not set STDDOC in 
https://github.com/D-Programming-Language/phobos/blob/master/posix.mak#L95, 
and refuse to build html if STDDOC is not set from the outside. But 
again I see this as just minor progress.

> This may
> be fine for the core devs and hard-core bleeding-edge people like myself
> who don't mind running off git HEAD and dealing with breakages, but it's
> bad for anybody else:
>
> 1) These interdependencies are currently expressed by hard-coded
> filesystem paths, which presumes a specific directory layout for
> checking out dmd, dlang.org, phobos, druntime. This means the build will
> break if somebody doesn't know the correct directory structure to use
> when checking out these repos.

Yah, that should be in a wiki. I keep on thinking about producing a 
10-minutes video "Your First Contribution to D" in which I explain step 
by step how to get a pull request in starting from a scratch.

We also have tools/update.sh which should probably be more prominently 
advertised.

But in the grand scheme of things I don't see a foolproof solution to 
that - you need dmd for building druntime, druntime for building phobos, 
etc. If you got dmd, druntime, and phobos in the wrong places then 
You're Gonna Have A Bad Time.

> 2) This makes the build fragile, because any changes to makefiles that
> might affect these dependencies have to be merged in sync between
> multiple repos, e.g., a PR that changes the layout of dlang.org needs to
> have a corresponding PR that updates the Phobos makefiles as well. This
> is kinda what we already have to do with compiler-dependent changes to
> druntime/phobos anyway, but in the case of documentation, any oversight
> is not caught by the autotester and may wind up on the public-facing
> dlang.org website before the problem is noticed.

I don't understand how this is a problem, how my suggestion makes it any 
worse, what a solution to it all would be.

> 3) Due to these interdependencies, you always have to checkout all 4
> repos (in a specific directory layout), otherwise the build will fail
> one way or another.

This is expected, no? What is an alternative? How would one e.g. build 
druntime without dmd?

> Again, not a problem for core devs, but a gratuitous
> obstacle for the casual user who wants to try things out. (Actually, to
> this day I'm almost certain I *still* don't have the exact layout
> assumed by the build scripts, because dlang.org appears to deposit
> generated html files in dlang.org/web, whereas phobos/posix.mak's 'html'
> target generates them in ../web/, which in my setup is one directory
> level off. I hacked around the problem with a symlink from web to
> dlang.org/web, but I'm pretty sure many users who wanted to experiment
> with Phobos docs gave up because they didn't have the patience to figure
> out how to get the thing to work properly.)

That's an inconsistency we should fix. Right now phobos is best build 
starting from the dlang.org build. We should probably disable phobos' 
own docs build.

> Because of this, my inclination right now is to *reduce*
> interdependencies between these repos, rather than introduce yet more.
>
> Either that, or we should just forget about trying to manually keep 3-4
> disparate repos in sync, and just use a single repo for dmd, druntime,
> phobos, dlang.org, etc.. Except that some people will then complain that
> they just want the bare compiler and nothing else, so why should they
> have to clone excess baggage like druntime/phobos. *shrug* Some days you
> just can't win.

Yah, that's why I think a set of simple tactical tools a la 
tools/update.sh are the proportional response here.

> (Of course, the *ideal* solution is that the makefiles will tell you if
> something is missing and where to get it and how to set it up correctly.
> But I'm skeptical whether this will ever materialize.  I'm not holding
> my breath.)

dlang.org does something of that kind already - "make rebase".


Andrei