AST files instead of DI interface files for faster compilation and easier distribution

[Martin Nowak]

On Mon, 18 Jun 2012 19:53:43 +0200, Walter Bright  
<newshound2 at digitalmars.com> wrote:

> On 6/18/2012 6:07 AM, Don Clugston wrote:
>> On 17/06/12 00:37, Walter Bright wrote:
>>> On 6/14/2012 1:03 AM, Don Clugston wrote:
>>>>> It is for debug builds.
>>>> Iain's data indicates that it's only a few % of the time taken on
>>>> semantic1().
>>>> Do you have data that shows otherwise?
>>>
>>> Nothing recent, it's mostly from my C++ compiler testing.
>>
>> But you argued in your blog that C++ parsing is inherently slow, and  
>> you've
>> fixed those problems in the design of D.
>> And as far as I can tell, you were extremely successful!
>> Parsing in D is very, very fast.
>
> Yeah, but I can't escape that lingering feeling that lexing is slow.
>
> I was fairly disappointed that asynchronously reading the source files  
> didn't have a measurable effect most of the time.

Lexing is definitely taking a big part of debug compilation time.
I haven't profiled the compiler for some time now but here are some  
thoughts.

- speeding up the identifier hash table
   there was always a profile spike at StringTable::lookup, though it  
reduced
   since you increased the bucket count

- memory mapping the source file saves a copy for UTF-8 sources
   this is by far the fastest way to read a source file

- parallel reading/parsing doesn't help much if most of the source files  
are
   read during import semantic

I'm regularly hitting other bottle necks so I don't think that lexing is  
#1.
When compiling std.range with unittests for example more that 50% of the  
compile time
is spend to check for existing template instantiations using O(N^2)/2  
compares of template arguments.
If we managed to fix http://d.puremagic.com/issues/show_bug.cgi?id=7469 we  
could efficiently use
the mangled name as key.