DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Joakim via Digitalmars-d]

On Sunday, 18 December 2016 at 18:42:36 UTC, Andrei Alexandrescu 
wrote:
> On 12/18/16 1:03 PM, Joakim wrote:
>> I largely agree with Dmitry. Ilya refactored several Phobos 
>> modules to
>> use scoped, selective imports much more, and I pitched in for 
>> some
>> remaining imports in the largest modules, so that only these
>> module-level imports remain, ie those necessary for symbols 
>> imported in
>> template constraints:
>>
>> std.datetime - https://github.com/dlang/phobos/pull/4373/files
>> std.uni - https://github.com/dlang/phobos/pull/4365/files
>> std.string and std.traits - 
>> https://github.com/dlang/phobos/pull/4370/files
>
> Yah, there's been a lot of good work (Jack Stouffer did a lot 
> as well IIRC) on pushing imports inside. The following searches 
> should be relevant:
>
> git grep '^\(private \)\?import' | wc -l
>
> yields about 426 top-level import declarations. The number of 
> indented imports is 4605:
>
> git grep '     *import\W' | wc -l
>
> So we're looking at 10% of imports being problematic. Sadly, 
> they turn out to make things quite difficult for Phobos. Last 
> time I looked at the module dependency graph it wasn't a lot 
> better than it used to before scoped imports. (I don't have it 
> handy... could anyone please produce it?)

Why do you care _so_ much about the module dependency graph?  To 
make this question concrete, let's look at an example, the 
std.array module you keep mentioning.  This is what it looked 
like before Ilya scoped as many imports as he could:

https://github.com/dlang/phobos/commit/3fcf723aa498b96de165361b5abb9d3450fdc069#diff-54cf8402b22024ae667d4048a5126f0e

That was a mess, similar to opaque C/C++ code, 13 modules 
imported at module-scope were whittled down to 4.  You just made 
those more specific in a commit related to this DIP, by listing 
the actual symbols selectively imported from those four modules:

https://github.com/dlang/phobos/commit/e064d5664f92c4b2f0866c08f6d0290ba66825ed#diff-54cf8402b22024ae667d4048a5126f0e

If I'm looking at the template constraints for any particular 
function and see a couple symbols I don't recognize, I don't 
think it's a big deal to find the symbols in that list at the top.

In other words, D already allows you to scope most imports.  I 
don't consider the dozen or two remaining symbols from templaint 
constraints and function arguments to provide much overhead.  
Rather, I consider the weight of this additional syntax, ie the 
cognitive overhead from having to remember and parse more syntax 
in my head, to be worse than the remaining dependency reasoning 
problem you're trying to solve: the cost outweights the benefit.  
Perhaps that's subjective and others may disagree.

Now, there's also the question of purely technical benefits, like 
compilation speed or executable bloat.  I looked at the latter a 
little last summer, after Ilya had cleaned up a lot of the 
standard library:

http://forum.dlang.org/thread/gmjqfjoemwtvgqrtdsdr@forum.dlang.org

I found that commenting out a single scoped, selective import of 
"std.string: format" in std.utf led to a 5% decrease in 
executable size for "hello world."  This is a problem with how 
dmd compiles or appends these module dependencies and would 
presumably still be there after this DIP, as you would not remove 
the dependency.

I think scoped, selective imports have been great at hacking away 
at the module dependency graph, as you lay out.  It is not clear 
what technical costs you see from the remaining few dependencies 
and if this DIP is the best way to remove them.  I think you 
should explain why you want to untangle the remaining dependency 
graph, and consider if this DIP is really doing that much.

>> When I first saw this DIP, like Dmitry I was happy that we 
>> could get rid
>> of those too, but the more I see these horrible syntax 
>> suggestions for
>> what is really a minor convenience, I changed my mind.  
>> std.datetime,
>> the 35k line (17 kloc according to Dscanner) beast of phobos, 
>> only needs
>> 20 or so symbols at module-scope. std.uni- 10k lines, 4.2 
>> kloc- only
>> needs 17 symbols, all from the three modules Dmitry mentioned.
>>  I don't
>> think his workaround of splitting up modules is even needed 
>> for such a
>> low amount of module-level imports.
>
> This paragraph is a good example of a couple of 
> counterarguments that I think point directly to flaws in the 
> DIP:
>
> (1) The DIP uses Phobos as an example, so it is applicable 
> mostly to Phobos. In fact, Phobos is among the systems that 
> would benefit least from the DIP: it has only druntime as 
> dependency, and is distributed in its entirety. Many projects 
> out there list multiple dependencies and may have various 
> building and distribution policies.

It is not clear how those alternate dependency, building and 
distribution policies change the picture.  Perhaps you should 
cite one of those as an example.

> The converse is to believe that working around a problem in 
> Phobos would render the DIP less useful in general.

The argument is not that Phobos has "worked around" the problem, 
but that once you scope everything you can other than template 
constraints (which you have said should be standard operating 
procedure in the DIP), the problem is minimal.

> (2) "I don't like the syntax, hence I don't like the feature." 
> I see this as a good opportunity for tasteful design, not a 
> downside of the feature.

The syntax is bad, but even if were beautiful, I don't consider 
the addition itself to be worth it.  Again, could be subjective.

>> Maybe there are other issues having to do with symbol 
>> resolution and
>> dependency encapsulation that are addressed by this DIP, ie the
>> technical performance of the compiler rather than refactoring 
>> or code
>> clarity, that I don't fully grasp, but from the first two 
>> points of the
>> claimed benefits of DCDs, ie ease of reasoning about 
>> dependencies and
>> refactoring code, I don't think this feature will come 
>> anywhere close to
>> carrying its own weight.
>
> Does the refactoring in 
> https://github.com/dlang/phobos/pull/4962 make dependencies 
> clearer?

I'm fine with your selective import refactoring, which I linked 
above:

https://github.com/dlang/phobos/pull/4963

> Are you e.g. clear on what you need in order to use Appender?

It is easy to see that it requires isDynamicArray and that is 
found in std.traits.  It may internally rely on other symbols 
that are currently imported from module scope, ie symbols from 
template constraints inside the struct Appender that happen to 
also be used outside that struct and so are imported at the top 
of the module.  If you really wanted to make that clear, you 
could scope those imports inside the struct too.

> Would you want to take this (or another) experiment to another 
> module and see how it improves its dependency structure?

Ilya and I have already done this, in the PRs I linked above.  I 
was mildly disappointed back then that I could not get rid of the 
remaining module-level imports, because of template constraints.  
I was predisposed to favor doing something about it, but I don't 
think this DIP is worth it.

>> As for the third benefit having to do with scalable template 
>> libraries,
>> I'm not sure I completely understand all the details there, 
>> but I wonder
>> if those problems aren't an artifact of the way dmd works now 
>> rather
>> than something that can't be fixed without this DIP.
>
> The DIP now dedicates an entire section to the pluses and 
> minuses of lazy imports, and concludes that lazy imports would 
> address scalability if carefully used.

Yes, there might be other ways to lower the technical costs of 
the dependency graph too, such as the size issue I mentioned 
above.