Some thoughts on dub housekeeping tasks for future design work

[Rikki Cattermole]

Hello again!

After debugging an already fixed in ~master bug, I had a thought 
kinda stuck in my mind about how dub is working internally 
although extremely poorly. It's related to how the metadata is 
actually pretty simple in its behavior, but not abstracted 
properly for the build manager. The metadata has one of three 
behaviors (although it can be in more than one category). It can 
go up, down, or nowhere (only that build).

A lot of the metadata is already abstracted into a single struct, 
BuildSettings, that's not an issue. What we want to do is move 
into having three instances per (sub)package being built. By 
doing this we can remove a significant amount of busy work that 
goes on within dub in duplication of BuildSettings generation and 
allow a much simpler process when building. Just pass in an array 
of BuildSettings and it'll merge them as part of the build. The 
responsibility of picking which BuildSettings apply to a 
particular build is the responsibility of the package manager, 
not the build manager.

Right now a lot of this logic is all intermixed with the package 
manager itself and done very badly. It's going to be a lot of 
work to untangle this and could easily break people's builds. So 
before anything structural like this can be done, anything that 
can be split out like leaf modules needs to be done. I've 
identified some housekeeping tasks that if done would make this 
process a lot easier or have quite significant benefits both 
currently and after such work is complete.

dub:

- ``getBestPackage`` simplify down to one request instead of two 
(move logic to dub-registry)
- Introduce caching mechanism of downloaded artifacts in file 
system, must be class and swapped out at runtime
- Registered non-registry package sources must be able to be 
compiled into a JSON blob full of package versions ext. info
- Able to consume compressed (zip) copies of package information 
(one per file in zip) in lieu of a registry
- Decouple and split out into own sub package compilers, 
packagesuppliers, cache, platform, dependencyresolver, semver 
packages/modules

dub-registry:

- Rewrite cache to use dub's (new) cache mechanism
- Move the repositories package to dub and rewrite as required to 
fit both purposes
- Use dub's new repositories that were moved here, minus registry 
(must be configurable at runtime)
- Able to produce compressed (zip) copy of all package version 
information (one file in zip per package)

If we can do these things, and split up BuildSettings in 
preparation for directionality (up/down/nowhere) with arrays 
support; we might have a way to do invasive structural changes in 
the package management side to break up the behavior with clearer 
divisions of build/package manager. But it's going to be slow 
going, and it's going to have to be bottom-up from the leaf 
modules.

One of the reasons we have to start bottom up is because dub has 
two abstractions for metadata. Recipes are what the dub files are 
represented by, and BuildSettings for the build itself. Ideally, 
the package manager would not know about the build manager's 
metadata (although it would tell the build manager to load its 
data), and the build manager wouldn't know about the package 
manager's metadata, which isn't the case right now.

Thoughts?