DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Sat, 24 Dec 2016 04:34:03 -0500, Andrei Alexandrescu wrote:
> Upon more investigation, I see a large discrepancy between the findings
> of DIP1005 and yours.

There's no discrepancy.

In part, you are misinterpreting most of what I said.

In part, you are assuming that imports on non-template declarations will 
be handled lazily, even though that is not part of this DIP, even though 
that is likewise possible with static and selective imports.

In part, you are using lines of code as a proxy for compile time.

In part, you dispute that this only affects template constraints, but:
* An import used only in the body of a template can be made a local 
import today.
* An import used in the declaration of a templated type or function can 
be addressed by using explicit template syntax, offering a place to 
insert your imports.
* An import used anywhere else must still be processed, even assuming 
this DIP is implemented.
* If, in a future DIP, we make it so that `with(import)` is handled 
lazily, we can also make it so that static and selective imports are 
handled lazily.

> The findings of DIP1005 are the following:
> 
> * Importing a single std module also imports on average 10.5 other
> modules.

Seems reasonable. Between 2 and 3.5 direct dependencies, by my count, and 
you're counting transitive dependencies.

We're concerned with the effects of DIP1005, though, which only affects 
template constraints.

> * Importing a single std module costs on average 64.6 ms.

55-ish for your hardware, you reported elsewhere. 47-ish for mine.

> * (Not stated in the DIP) A majority of std templates would acquire
> inline imports.

Again, that wouldn't impact compile times because these aren't template 
constraints.

You can make a separate DIP to make imports lazy. That can impact static, 
selective, and `with` imports equally well. But it's not part of what 
we're discussing today.

> According to the DIP, one may estimate that the proposed feature would
> reduce additional imports to 0 and the average time to import a single
> module by a factor of 10 to under 10 ms.

"The proposed feature" must be lazy semantic analysis, especially of 
imports. That isn't part of DIP1005.

You won't get to zero additional imports. You might get to zero 
*extraneous* imports -- that is, only the set of imports required to 
create a custom *.di file containing only the parts of the module that 
your application uses.

> By your estimates:
> 
> * 26 templates in std need inline imports.

I said that 26 templates *could possibly benefit from* your new style of 
imports. There's a difference between possibly benefitting from a change 
and needing that change.

> * Importing a single std module today would only imports 1-3 other
> modules most of the time (one or more of std.traits, std.meta, and
> std.range.primitives).

No, that's not what I said at all. I said that the only modules you would 
sometimes *stop* processing because of DIP1005 are std.traits, std.meta, 
and std.range.primitives. That's because those modules contain templates 
used in other modules as template constraints.

In order to get any additional improvements, you need lazy imports, which 
can also apply to static or selective imports without any syntax changes.

> * These additional imports cost in aggregate under 10ms, bringing the
> average cost of importing a module itself to 54.6 ms.

~10ms is the upper bound of the added cost if you import just one module 
in std that has a template constraint you don't use.

The way you state it implies that every module brings in std.traits, 
std.meta, and std.range.primitives unnecessarily, instead of 26 templates 
across at most 26 modules importing them for a reason.

> * It follows that the average module takes 5.46 more times to import
> alone than the sum of std.traits, std.meta, and std.range.primitives
> (which have a total of 11263 lines, 5x more than the average Phobos
> module).

More like 4.7 on my hardware, but yeah. 11k lines that have to be parsed 
and 0 lines that require semantic analysis. Not terribly surprising.

> I don't see how your claims can be simultaneously true with the findings
> of DIP1005.

You found that the average cost of importing a std module is 54ms or 
thereabouts. std.traits, std.meta, and std.range.primitives are well 
below average. No conflict there. They aren't even the cheapest modules 
in the standard library.

The modules in question are mostly unittests. The compiler doesn't run 
semantic on unittests in a module that wasn't included in the command 
line. (Even if you pass -unittest. Try it out -- you can even have a 
unittest that says `static assert(false);` and it does nothing.)

The parts of the modules that are not unittests are templates. The 
compiler doesn't run semantic analysis on templates until you use them.

So it should be pretty obvious why these modules are so cheap to import 
and not use.

> The scripts that compute those numbers are available with
> the DIP. Were you able to reproduce them?

The times it reported on my hardware:

Min: 5ms
Max: 300ms
Median: 21ms
Average: 47ms

The minimum isn't terribly useful because it gets to the point of testing 
the process scheduler and IO more than the compiler. If we want numbers 
that we can trust on the low end, we'll need to put timing information 
into the compiler, maybe control for IO by using a ramfs, that sort of 
thing.

You also reproduced my test, so this isn't a quirk of my installation.