Reducing the inter-dependencies (in Phobos and at large)

[Dmitry Olshansky]

Recently I've struggled again to integrate a module in Phobos and the 
amount of curious forward reference bugs made think about a related but 
not equivalent problem.

Intro

Basically an import graph of Phobos is a rat's nest of mutual imports.
With the most of modules being template "toolkits" you shouldn't pay for 
what you don't use. Yet it isn't true in general as the module may drag 
in other modules and with that all of static constructors and/or globals 
related.

Adding even more fun to this stuff - to get a "constraint checker" you 
have to import all other stuff from that module (and transitively from 
all imported by it modules, see std.range example below).

One motivating example (though I believe there are much more beyond this 
one):

As some might know regular expression engine can be used for both 
generating sequences for a known pattern and matching/finding pieces 
that do match it in some data.

In fact I have such a functionality in std.regex hidden from public 
interface (not yet complete, wasn't sure of the kind of API etc.).

Now the *key* fact:

auto generate(RegEx, rng) (RegEx re, Random rng)
	if(isRegex!RegEx && isUniformRNG!Random)
{
...
}

Now given that innocent signature you have dependency on the whole of 
std.random *including* seeding the default random number generator at 
the program start!

And recall that generating strings while neat is arguably more rare need 
then pattern matching.

Same thing with std.datetime - to get an ability to accept any kind of 
Date as a template parameter (in your API) you have to get the whole 
std.datetime *even if the user never ever calls this function* of your 
module API.

And everyone and their granny depends on full version of std.range
that in turn depends on std.algorithm that depends on std.conv that 
depends on std.format(!) and that incidentally on std.uni.
BTW std.uni turns out to be a kind of sink everybody imports sooner or 
later if only for unittests, sadly it's mostly imported unconditionally.

And that skipping a full rat's nest to preserve the brains of the reader.

After a couple of frustrating evenings dealing with it (multiplied by 
the bogus dmd) I've come up with the idea below.


Solution

First of all no compiler magic required (phew-ew!) :)

Not to mention the 2 obvious facts - smaller modules might help, as 
would guarding by version(unittest) imports used only for unit tests.

What we need is to re-arrange the module hierarchy (and we need that 
anyway) so that we split off the "concept" part of modules to a separate 
package.

That would allow modules that need this to use these Duck-typed entities 
(IFF the user ever passes such an entity) can stick with importing only 
the concept part.

Applying that to the current layout would look like:
std.concept.range
std.concept.random
std.concept.* //every other module with any useful isXYZ constraint
std.* // stays as is

Any module that has "concept" part then looks like this:

module std.xyz;
import std.concept.xyz;
... //the rest

And then other weakly-dependent modules (i.e. these that are satisfied 
with traits and duck-typed interfaces) can safely import std.concept.xyz 
instead of std.xyz. E.g. std.regex would import std.concept.random to 
get isUniformRNG and rely on duck typing thusly described to use it 
correctly.

The change is backwards compatible and introduces no breakage.
Only clean sugar-free interdependence of modules in Phobos.

Later people (mostly library writers) can use e.g. std.concept.range
to avoid pulling full dependency tree in case only constraints are 
needed. The technique can be touted as coding guideline for template and 
duck-type heavy libraries.

Thoughts? Other ideas?

-- 
Dmitry Olshansky