[D-runtime] Why does druntime us .di files instead of .d?

[Jonathan M Davis]

On 2011-07-14 14:40, Andrew Wiley wrote:
> On Thu, Jul 14, 2011 at 4:01 AM, Steve Schveighoffer 
<schveiguy at yahoo.com>wrote:
> > Right. I would think the most important reason to use .di files is to
> > hide private imports. The GC is a good example, you don't want to
> > import the GC implementation when importing core.memory. But in fact,
> > core.memory doesn't import anything from rt, it links to the GC
> > implementation via extern(C) functions, so the implementation of
> > core.memory is not a good example.
> > 
> > I think to a point, using .di files has merit. To be certain, anything
> > that imports anything from rt should be hidden (I'm not sure if there are
> > any cases of that). But we should consider the cost of disabling CTFE on
> > druntime functions, especially those in core that really aren't managing
> > the actual runtime of the compiler, but are just tools used by both
> > druntime and phobos (core.time and core.math come to mind).
> > 
> > -Steve
> 
> It seems to me that a way to fix all this and guarantee CTFE-ability (which
> seems to be the end goal) would be to add some sort of modifier or
> annotation to designate that a function is CTFE-able. This could be
> verified by the compiler at build-time and used as a flag to emit full
> function source when generating a di file. Things get more complicated
> because it would have to be transitive, so it wouldn't be a simple
> addition, and might be heavier than what we're looking for.
> On the other hand, it would provide a strong guarantee about a D feature
> that's somewhat nebulous right now.
> Thoughts?

It's not altogether a bad idea, but at least in theory, one of the main 
features of CTFE is that it's supposed to just work without treating a 
function specially. Some functions just can't be run at compile time due to 
their nature, but in theory, pretty much anything is supposed to be CTFEable. 
So, on some level, CTFE is a bit of a failure if a function requires an 
attribute to be CTFEable. On the other hand, having an attribute whose purpose 
was purely to make it so that the function has its full source put in a .di 
file (which would ensure that it was CTFEable) but which was otherwise ignored 
and completely unnecessary for CTFE wouldn't be as bad. It might even have 
some value beyond CTFE. For instance, wouldn't the lack of source in a .di 
file make a function un-inlinable? As such, there are a lot of functions which 
you would want the full source for in a .di file. In the inline case at least, 
I can see the compiler being smart enough to realize that it's probably going 
to want to inline a function, so it then puts the source for that function in 
the .di file. But CTFE functions can be arbitrarily complex, so that wouldn't 
work for them.

I don't know. In C/C++, at least it's totally up to the programmer to separate 
the header and source files as they deem appropriate. In D, the .di is likely 
to be generated, so while you could edit it by hand after generation, you then 
have problems keeping it in sync with the .d file (or you have to regenerate 
the .di file every time and patch it). And since the .d file contains 
everything in it while the .di file has a subset, it's not quite the same 
situation as you get with a header and source file in C/C++.

As long as you can generate a .di file from a .d file without losing any of 
the source that you actually need (for inlining or CTFE or whatever), then 
you're fine. But beyond that, it seems like you either give up those 
capabilities, or you have to hand edit the .di file every time that a change 
is made to the .d file which would affect the .di file. Neither option seems 
particularly pleasant to me.

If modules were organized such that those which could afford to not be inlined 
or CTFEed at all were .di files, and the rest were .d files, then that 
essentially solves the problem, but depending on the modules, that could 
negatively affect module organization. I suppose that you could effectively 
organize modules as either internal or external and make the internal ones 
have the .di files and the external ones just .d files. But that's not 
necessarily reasonable. Pretty soon, you're effectively trying to have .d 
files which are akin to C/C++ source files, and .d files which are akin to 
C/++ header files and have .di files. Overall, it just seems simpler to use .d 
files, but we don't necessarily want to lose the compilation gains from .di 
files either. So, I don't know what the best way to handle this is. But I'd be 
very concerned about using .di files if it negatively affects CTFE and 
inlinability. If it doesn't, then they're fine. But from the looks of it, it 
does.

- Jonathan M Davis