Templating everything? One module per function/struct/class/etc, grouped by package?

[Jonathan M Davis via Digitalmars-d-learn]

On Mon, 12 May 2014 08:37:42 +0000
JR via Digitalmars-d-learn <digitalmars-d-learn at puremagic.com> wrote:

> Given that...
>
> 1. importing a module makes it compile the entirety of it, as
> well as whatever it may be importing in turn
> 2. templates are only compiled if instantiated
> 3. the new package.d functionality
>
> ...is there a reason *not* to make every single
> function/struct/class separate submodules in a package, and make
> *all* of those templates? Unnused functionality would never be
> imported nor instantiated, and as such never be compiled, so my
> binary would only include what it actually uses.
>
>
> std/stdio/package.d:
>      module std.stdio;
>      // still allows for importing the entirety of std.stdio
>
>      public import std.stdio.foo;
>      public import std.stdio.writefln;
>      __EOF__
>
>
> std/stdio/foo.d:
>      module std.stdio.foo;
>
>      void fooify(Args...)(Args args)
>      if (Args.length > 0)
>      {
>          // ...
>      }
>
>      void fooify()()
>      {
>          // don't need this, won't compile this
>      }
>      __EOF__
>
>
> std/stdio/writefln.d;
>      module std.stdio.writefln;
>
>      // nevermind the incompatible signature
>      auto writefln(string pattern, Args...)(Args args)
>      if (!pattern.canFind(PatternIdentifier.json))
>      {
>          // code that doesn't need std.json --> it is never
> imported
>      }
>      __EOF__
>
>
> What am I missing?

Well, that would be a lot of extraneous files, which would be very messy IMHO.
It also makes it much harder to share private functionality, because
everything is scattered across modules - you'd be force to use the package for
that. It also wouldn't surprise me if it cost more to compile the code that
way if you were actually using most of it (though it may very well save
compilation time if you're using a relatively small number of the functions
and types). So, from a purely organization perspective, I think that it's a
very bad idea, though others may think that it's a good one. And since
package.d imports all of those modules anyway, separating them out into
separate files didn't even help you any.

Also, templates cost more to compile, so while you may avoid having to compile
some functions, becasue you're not using them, everything that _does_ get
compiled will take longer to compile. And if you templatize them in a way that
would result in more template instantiations (e.g. you actually templatize the
parameters rather than just giving the function an empty template parameter
list), then not only will the functions have to be compiled more frequently
(due to multiple instantiations), but they'll take up more space in the final
binary. Also, while D does a _much_ better job with template errors than C++
does, template-related errors in D are still far worse than with functions
that aren't templated, so you're likely going to cost yourself more time
debugging template-related compilation errors than you ever would gain in
reduced compilation times.

In addition, if a function is templatized, it's harder to use it with function
prototypes, which can be a definite problem for some code. It's also a
horrible idea for libraries to have functions templatized just to be
templatized, because that means that a function has to be compiled _every_
time that a program uses it rather than having it compiled once when the
library is compiled (the function would still have to be parsed unless it just
had its signature in a .di file, but that's still faster than full
compilation - and if a .di file is used, then all that has to be parsed is the
signature). So, while it's often valuable to templatize functions,
templatizing them to save compilation times is questionable at best.

D already does a _very_ good job at compiling quickly. Often, the linking step
costs more than the actualy compilation does (though obviosuly, as programs
grow larger, the compilation time does definitely exceed the link time).
Unless you're running into problems with compilation speed, I'd strongly
advise against trying to work around the compiler to speed up code
compilation. Templatize functions when it makes sense to do so, but don't
templatize them just in an attempt to avoid having them be compiled.

If you're looking to speed up compilation times, it makes far more sense to
look at doing things like reducing how much CTFE you use and how many
templates you use. CTFE in particular is ridiculously expensive thanks to it
effectively just being hacked into the compiler originally. Don has been doing
work to improve that, and I expect it to improve over time, but I don't know
how far along he is, and I don't know that it'll ever be exactly cheap to use
CTFE.

Keep in mind that lexing and parsing are the _cheap_ part of the compiler. So,
importing stuff really doesn't cost you much. Already, the compiler won't
fully compile all of the symbols within a module except when it's compiling
that module. Simply importing it just causes it to process the module as much
as required to fully compile the module that's importing it - which generally
means getting function signatures (though in the case of templates, it could
mean fully compiling the functions which are used rather than just pulling in
their signatures, since the template is not fully compiled with its own module
but by the module that instantiates it).

So, that explanation is probably too long, but essentially what you're
suggesting is likely to be _more_ expensive, not less, and IMHO, it makes code
organization much messier (though that part is obviously subjective). I've
never seen the insanely large number of files required in Java to be a benefit
of Java, and you're essentially suggesting that we take that to the extreme
and have _everything_ have its own module rather than just each class.

- Jonathan M Davis