DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Joakim via Digitalmars-d]

On Tuesday, 20 December 2016 at 20:51:54 UTC, Andrei Alexandrescu 
wrote:
> I've asked Joakim about this via email just now, likely other 
> folks also know the answer:
>
> 1. I found these PRs related to local imports:
>
> https://github.com/dlang/phobos/pull/4361
> https://github.com/dlang/phobos/pull/4365
> https://github.com/dlang/phobos/pull/4370
> https://github.com/dlang/phobos/pull/4373
> https://github.com/dlang/phobos/pull/4379
> https://github.com/dlang/phobos/pull/4392
> https://github.com/dlang/phobos/pull/4467
>
> Are there more? Is there a central point for them (such as a 
> bugzilla issue)?

Ilya lists a lot more above, he did most of the work.  No central 
point that I know of.

> 2. I see you've done a bunch of work in the area. Where, in your
> estimate, are we on the spectrum of making imports local 
> without a
> major reorganization? Any low-hanging fruit to look at, or have 
> Joakim,
> Ilya, Jack and others made a pass through most/all modules 
> already?

There is more to be done, but my guess would be 80-90% done for 
Phobos. Ilya scoped a lot, but usually left druntime imports 
alone.  Top-level module imports mostly don't use selective 
imports yet, because Martin wanted to hold off till he was sure 
the symbol leak bug was fixed.

On Tuesday, 20 December 2016 at 22:08:38 UTC, Andrei Alexandrescu 
wrote:
> On 12/20/2016 03:46 AM, Joakim wrote:
>> I didn't just say "eh:" I gave evidence for why I think the 
>> problem is
>> minimal and asked why it's so important to scope those last 
>> 3-4 imported
>> modules too, which you didn't answer.
>
> You have asked for a smoking gun, and one has been found.
>
> I have just uploaded a major update that carefully analyzes the 
> improvements brought about by switching to local imports in the 
> D Standard Library. Please refer to the section "Workaround: 
> Are Local Imports Good Enough?" and Appendix A:
>
> https://github.com/dlang/DIPs/pull/51/files
>
> https://github.com/dlang/DIPs/blob/91baecedcfe7cb75ac22e66478722ec797ebb901/DIPs/DIP1005.md

Thanks for this analysis of the remaining dependency graph, it is 
worth looking at.  Allow me to poke some holes in it.

To begin with, the amount of scoping that has been done is 
overstated, if you simply count scoped imports and compare it to 
module-level imports.  Each module-level import has to be 
replicated multiple times for each local scope, especially in 
unittest blocks.  A better number is more like 20-30%, as I 
pointed out 4 out of 13 modules remain at top-level in std.array. 
Using that metric, a 3-4X reduction in top-level imports has led 
to at least a 2.2x improvement in imported files, so the effort 
has been more meaningful than you conclude.

Second, as I noted above, most top-level imports have not been 
made selective yet, because of the symbol leak bug that was 
recently fixed by Martin.  You will see in my PRs that I only 
list those symbols as a comment, because I could not turn those 
into selective imports yet.  If the compiler is doing its job 
right, selective imports should greatly reduce the cost of 
importing a module, even if your metric would still show the 
module being imported.

Third, checking some of the output from the commands you ran in 
your script shows that up to half of the imported modules are 
from druntime.  I noted earlier that Ilya usually didn't bother 
scoping top-level druntime imports, because he perceived their 
cost to be low (I scoped them too in the handful I cleaned up, 
just for completeness).  As far as I know, nobody has bothered to 
spend any time scoping druntime, so it would be better if you 
filtered out druntime imports from your analysis.

Finally, while it's nice to know the extent of the dependency 
graph, what really matters is the _cost_ of each link of the 
graph, which is what I keep hammering on.  If the cost of links 
is small, it doesn't matter how entangled it is.  If minimizing 
the dependency graph through scoping alone, ie without 
implementing this DIP, removes most of the cost, that's all I 
care about.

I have noted one example above, where _a single DCD in phobos_, 
ie a scoped, selective import, had gigantic costs in terms of 
executable size, where entire modules were included because of 
it.  If that's the case more generally, then _no_ amount of 
dependency disentangling will matter, because the cost of single 
DCDs is still huge.  Perhaps that's just an isolated issue 
however, it needs to be investigated.

My point is that the dependency graph matters, but now that we're 
getting down to the last entanglements, we need to know the cost 
of those last links.  Your dependency analysis gives us some 
quantitative idea of the size of the remaining graph, but tells 
us nothing about the cost of those links. That's what I'm looking 
for.

I will spend some time now investigating those costs with sample 
code.  My request all along has been that you give us some idea 
of those costs, if you know the answer already.