DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/20/2016 11:32 PM, Joakim wrote:
> On Tuesday, 20 December 2016 at 20:51:54 UTC, Andrei Alexandrescu wrote:
> Thanks for this analysis of the remaining dependency graph, it is worth
> looking at.  Allow me to poke some holes in it.
>
> To begin with, the amount of scoping that has been done is overstated,
> if you simply count scoped imports and compare it to module-level
> imports.  Each module-level import has to be replicated multiple times
> for each local scope, especially in unittest blocks.  A better number is
> more like 20-30%, as I pointed out 4 out of 13 modules remain at
> top-level in std.array. Using that metric, a 3-4X reduction in top-level
> imports has led to at least a 2.2x improvement in imported files, so the
> effort has been more meaningful than you conclude.

Fixed. I also made a note about the need to duplicate imports as they 
are pushed down.

> Second, as I noted above, most top-level imports have not been made
> selective yet, because of the symbol leak bug that was recently fixed by
> Martin.  You will see in my PRs that I only list those symbols as a
> comment, because I could not turn those into selective imports yet.  If
> the compiler is doing its job right, selective imports should greatly
> reduce the cost of importing a module, even if your metric would still
> show the module being imported.

That is not relevant to this section, which discusses the effectiveness 
of using local imports with the current compilation technology. Per the 
section's opening sentence:

> A legitimate question to ask is whether consistent use of local
> imports wherever possible would be an appropriate approximation of
> the Dependency-Carrying Declarations goal with no change in the
> language at all.

The section "Alternative: Lazy Imports" discusses how static or local 
imports could be used in conjunction with new compilation technologies. 
If there are improvements to be made there, please advise.

> Third, checking some of the output from the commands you ran in your
> script shows that up to half of the imported modules are from druntime.
> I noted earlier that Ilya usually didn't bother scoping top-level
> druntime imports, because he perceived their cost to be low (I scoped
> them too in the handful I cleaned up, just for completeness).  As far as
> I know, nobody has bothered to spend any time scoping druntime, so it
> would be better if you filtered out druntime imports from your analysis.

Fixed to only count imports from std.

> Finally, while it's nice to know the extent of the dependency graph,
> what really matters is the _cost_ of each link of the graph, which is
> what I keep hammering on.  If the cost of links is small, it doesn't
> matter how entangled it is.  If minimizing the dependency graph through
> scoping alone, ie without implementing this DIP, removes most of the
> cost, that's all I care about.

In first approximation, whether a file gets opened or not makes a 
difference (filesystem operations (possibly including networking), 
necessity to rebuild if dependent code is changed. The analysis shows 
there is significant overhead remaining, on average 10.5 additional 
files per unused import.

If the current document could be clearer in explaining costs, please let 
me know.

> I have noted one example above, where _a single DCD in phobos_, ie a
> scoped, selective import, had gigantic costs in terms of executable
> size, where entire modules were included because of it.  If that's the
> case more generally, then _no_ amount of dependency disentangling will
> matter, because the cost of single DCDs is still huge.  Perhaps that's
> just an isolated issue however, it needs to be investigated.

That seems an unrelated matter. Yes, you could pull a large dependency 
in one shot with old or new technology.

> My point is that the dependency graph matters, but now that we're
> getting down to the last entanglements, we need to know the cost of
> those last links.  Your dependency analysis gives us some quantitative
> idea of the size of the remaining graph, but tells us nothing about the
> cost of those links. That's what I'm looking for.
>
> I will spend some time now investigating those costs with sample code.
> My request all along has been that you give us some idea of those costs,
> if you know the answer already.

I don't know how to make matters much clearer than the current document. 
Any suggestions are welcome. The section "Workaround: Are Local Imports 
Good Enough?" discusses the material cost in terms of extra files that 
need to be opened and parsed (some unnecessarily) in order to complete a 
compilation. The "Rationale" part of the document discusses the costs in 
terms of maintainability, clarity, and documentation.


Thanks,

Andrei