AST files instead of DI interface files for faster compilation and easier distribution

[Don Clugston]

On 14/06/12 10:10, Jonathan M Davis wrote:
> On Thursday, June 14, 2012 10:03:05 Don Clugston wrote:
>> On 13/06/12 16:29, Walter Bright wrote:
>>> On 6/13/2012 1:07 AM, Don Clugston wrote:
>>>> On 12/06/12 18:46, Walter Bright wrote:
>>>>> On 6/12/2012 2:07 AM, timotheecour wrote:
>>>>>> There's a current pull request to improve di file generation
>>>>>> (https://github.com/D-Programming-Language/dmd/pull/945); I'd like to
>>>>>> suggest
>>>>>> further ideas.
>>>>>> As far as I understand, di interface files try to achieve these
>>>>>> conflicting goals:
>>>>>>
>>>>>> 1) speed up compilation by avoiding having to reparse large files over
>>>>>> and over.
>>>>>> 2) hide implementation details for proprietary reasons
>>>>>> 3) still maintain source code in some form to allow inlining and CTFE
>>>>>> 4) be human readable
>>>>>
>>>>> (4) was not a goal.
>>>>>
>>>>> A .di file could very well be a binary file, but making it look like D
>>>>> source enabled them to be loaded with no additional implementation work
>>>>> in the compiler.
>>>>
>>>> I don't understand (1) actually.
>>>>
>>>> For two reasons:
>>>> (a) Is lexing + parsing really a significant part of the compilation
>>>> time? Has
>>>> anyone done some solid profiling?
>>>
>>> It is for debug builds.
>>
>> Iain's data indicates that it's only a few % of the time taken on
>> semantic1().
>> Do you have data that shows otherwise?
>>
>> It seems to me, that slow parsing is a C++ problem which D already solved.
>
> If this is the case, is there any value at all to using .di files in druntime
> or Phobos other than in cases where we're specifically trying to hide
> implementation (e.g. with the GC)? Or do we still end up paying the semantic
> cost for importing the .d files such that using .di files would still help with
> compilation times?
>
> - Jonathan M Davis

I don't think Phobos should use .di files at all. I don't think there 
are any cases where we want to conceal code.

The performance benefit you would get is completely negligible. It 
doesn't even reduce the number of files that need to be loaded, just the 
length of each one.

I think that, for example, improving the way that array literals are 
dealt with would have at least as much impact on compilation time.
For the DMD backend, fixing up the treatment of comma expressions would 
have a much bigger impact than getting lexing and parsing time to zero.

And we're well set up for parallel compilation. There's no shortage of 
things we can do to improve compilation time.

Using di files for speed seems a bit like jettisoning the cargo to keep 
the ship afloat. It works but you only do it when you've got no other 
options.