Ready for review: new std.uni

[David Nadlinger]

On Sunday, 13 January 2013 at 23:57:48 UTC, Walter Bright wrote:
> On 1/13/2013 3:10 PM, David Nadlinger wrote:
>> To elaborate – and again, please don't take this as a personal 
>> attack –, every
>> time this discussion comes up you seem to resort to the 
>> »diversity is good«
>> cliché
>
> Being a cliche doesn't make it invalid.

You conveniently ended your citation before the »without ever 
actually responding to the arguments other people bring up« part.

>> You want to stick to the DMC backend? Fine, that's your good 
>> right, but wouldn't
>> it only be fair to share your reasons with the other people 
>> who are investing
>> considerable amounts of work and time in »your« language?
>
> It's long ceased to be my language. It's a group creation.
>
> Anyhow, I like using the dmc backend because:

Let me reply specifically focusing on LLVM:

> 1. I can completely control the user experience.

You can do this with LLVM as well, as you can modify its source 
at will. Let me also note that this means that *only you* can fix 
certain parts of the user experience (see for example the Optlink 
issues).

And besides, to be brutally honest, there has not been too much 
good feedback about the DMD toolchain so far, apart from 
compilation speeds, has there? The reason for this might of 
course be that users just expect a D compiler to work just like 
every other C/C++ compiler, but I can't see how having a 
potentially bigger degree of control would have helped much so 
far.

> 2. I can fix a problem where it should be fixed, rather than 
> working around it.

You can do this with LLVM as well – after all, that's precisely 
the point of open source software. And additionally, it has the 
big advantage that other people can fix problems in it too, and 
in many cases, the LLVM people will even do the work for you.

> 3. I understand it completely, and so I can reasonably quickly 
> add in things that some other back end may not support.

LLVM is reasonably well documented and has many useful debugging 
and visualization features, so I'm sure you would find your way 
around it quickly if you wanted to.

> 4. It's fast.

No objection there. But let me note that its weak(er) optimizer 
might get it disqualified as far as release builds go before this 
argument even counts.

> 5. I don't run any risk of "you stole X's ideas when you worked 
> on X's back end."

You can use any code from LLVM under a BSD-like license. Would 
this be too restrictive? I am not a lawyer, but I'm reasonably 
certain it would even allow you to just take the whole LLVM code 
base and sell it as a closed-source package.

> 6. I enjoy working on it.

Fair enough, and all too understandable given that it's your 
creation! But to put it bluntly, it's a hassle for almost 
everybody else working on the compiler, so a project you want to 
attract other contributors to might not be the best environment 
to pursue this »hobby«.

>> So, in short, even if the DMC backend was technically superior 
>> to every other
>> solution out there right now, I would be hard-pressed to make 
>> an argument for
>> preferring it over a more widely used solution like LLVM 
>> because of this. I
>> cannot see how putting work into a custom backend could 
>> realistically help to
>> make D more attractive compared to other languages, so the 
>> current situation
>> seems like a case of making things unnecessarily hard for 
>> ourselves.
>
> This assumes that the existence of dmd takes away from ldc. I 
> don't believe it does. Ldc doesn't need me.

Why are you even bringing up LDC here? My last post must have 
been very unclear in this regard? But of course, LDC does need 
you – after all, it uses the DMD frontend!

My entire point is that *D* needs you and your expertise: Having 
a veteran compiler writer on board is just about the best thing 
that can happen to a language effort; just because of your 
experience you are infinitely more productive in some areas than 
most of us other contributors are. What I'm trying to say is that 
if you want D to succeed, which I assume you do, spending (part 
of) your time on writing a custom backend might not be the best 
idea: First, you could have spent that time working on things 
that help to sell the language, and second, doing so actually 
makes life *harder* for other contributors.

>> To summarize: Although there is quite a lot of promising 
>> development going on in
>> the language space right now, I still think D has an excellent 
>> offer to make,
>> otherwise I would left the camp long ago. But it doesn't seem 
>> wise to
>> intentionally weaken our position in the arena by spending 
>> considerable amounts
>> of manpower on solving a problem from the ground up for which 
>> there is already a
>> free off-the-shelf solution that would get us at least 95% 
>> there – and the fact
>> that our resources are limited is precisely what you seem to 
>> neglect when
>> reciting the »diversity is good« mantra.
>
> I'm not convinced that the effort to keep up with llvm in order 
> to work with it is actually less time.

Less time than what? Developing a code generator that matches 
theirs? You are certainly a very capable compiler engineer, but 
LLVM has quite a few bright minds working on it as well, 
full-time. I honestly don't see how »keeping up with LLVM« could 
be more time-consuming than replicating all of their work. Maybe 
you can elaborate on what exactly you meant?

But for what it's worth, Kai and I have had no problems with 
keeping LDC up to date with the latest LLVM version for the last 
few upstream releases, and we both can't spend too much time on 
LDC, so this likely wouldn't be a big issue for you either.

>>> For example, LDC doesn't work with VS, and so would have cost
>>> us the possible design win at Remedy.
>>
>> In view of the above, I'm not sure how much weight this 
>> argument could carry,
>> but I don't think it is valid in the first place: If our 
>> reference compiler was
>> backed by LLVM, you could have just added CV8 support to it 
>> like you did for
>> DMD.
>
> The thing is, I don't see how that would have saved time. It 
> would have been more time - I already had CV4 support in dmd, 
> so I had a head start on CV8. There were also a number of wacky 
> things about Mscoff64 - how to integrate them into LLVM would 
> have required a thorough knowledge of it on my part, and *then* 
> I'd still have to implement it. There were also the MS64 ABI 
> changes, which consumed a considerable amount of time. If that 
> is not in LLVM, I would have been in for a world of effort and 
> would have had to do some considerable surgery on it.

If I'm not mistaken, most of the related work for LLVM was done 
by the end of 2011.

>> By the way, the other thing missing from LLVM regarding VS 
>> compatibility is
>> SEH support, but DMD doesn't implement that either (LLVM 
>> actually supports EH on
>> MinGW right now, so the implementation could likely have been 
>> re-used as a
>> workaround, just like you did for DMD).
>
> I think that underestimates what all is necessary to be ABI and 
> format compatible with VC64. SEH is only a small part of it.
>
> For example,
>
> 1. Win64 ABI, which is fundamentally different from any other 
> platform
> 2. Making it compatible with and use VC++'s runtime
> 3. Generating MsCoff 64.
> 4. Adding the wacky pdata and xdata sections

I was not just blindly guessing in my last post. LLVM supports 
directly emitting 64 bit COFF, and as far as I know the code for 
writing out the RUNTIME_FUNCTION/UNWIND_INFO instances is 
functional.

> 5. Learning the completely undocumented CV8 format, then 
> implementing it
>
> Doing this all requires a fairly comprehensive understanding of 
> the innards of the back end. If LLVM is lacking one or more of 
> 1..5, then I wouldn't be surprised a bit if it took me 
> considerably *longer* to do it for LLVM, and then I'd have to 
> sell my changes to the LLVM group. (If they don't buy it, then 
> I'm stuck with a fork I have to maintain myself. Ugh.)

Given the modular design of LLVM, maintaining a fork probably 
wouldn't even be too bad; many companies using it e.g. have their 
own private backends. That being said, the LLVM folks are 
generally very open with accepting contributions into the main 
tree (except for 32 bit SEH support, where the Apple people are 
afraid of issues regarding that notorious Borland patent).

But note that I didn't even want to convince you that 
implementing a given new feature (such as CV8) for LLVM would 
take you _less_ time than for your own backend. I'm just saying 
that I would expect it to take a comparable amount of time, and 
even if it might take you some time to get familiar with the LLVM 
internals at first, I'm sure that initial cost would quickly be 
offset by the fact that you don't have to write *everything* on 
your own. I wonder if there is another production-grade backend 
out there that is maintained by a single person?

David