DUB - call to arms

[H. S. Teoh]

On Mon, Apr 15, 2019 at 07:52:07PM -0400, Nick Sabalausky (Abscissa) via Digitalmars-d wrote:
[...]
> [...]  Basically, a package manager's config should need exactly three
> things from your project:
> 
> 1. What dependencies.
> 2. The command(s) to (build, test, etc.)
> 3. Name(s)/Location(s) of the build's output.
> 
> Then, with that information, a package manager provides services such
> as (but not necessarily limited to):
> 
> 1. A simple, standardized way for you and your users to obtain/build
> the dependencies.
> 
> 2. A simple, standardized way for buildscripts/buildsystems to obtain
> the information needed to include the dependencies in their own build
> (such as -I... include directories, paths to the now-already-built
> lib/exec binaries, etc.)

I'd add:

3. A standard query interface for querying a remote repository for
packages (matching some names / patterns) and version numbers.


> From this, each project can naturally either just roll its own
> buildscripts, or depend on another package providing a builsystem.

That's a good idea.  Completely decouple package management from
building.  Let the package manager do what it does best: managing
packages, and leave the compilation to another tool more suited for the
job.  Of course, the default build command can be set to `dub build` to
keep existing users happy.

I was thinking the build/test/etc. command can itself be defined by a
dependency.  For example, if your project uses make for builds, there
could be a `make` dub package that defines command-line templates used
for generating platform-specific build commands. Your package would
simply specify:

	"build-system": "make"

and the "make" package would define the exact command(s) for invoking
make (e.g., if gmake is picked up as a resolution, it could define the
executable name as `gmake`, or it could define different CLI syntax for
Windows vs. Linux, etc.).  The package manager would then recognize
"make" as a dependency, and would download and install that package
(which presumably would install an appropriate version of 'make') if it
hasn't already been installed, then invoke it to build the project.

The "make" package itself would be presumably some kind of wrapper
around a system utility, or an installer of some sort that installs an
appropriate version of make on the system.  It could be implemented as a
normal dub package whose "build" command is something that runs an
attached installer.


> Some of the *details* can be quite nontrivial...like dependency
> resolution algorithms, or designing the interactions between package
> manager and buildsystem to be simple, yet effective enough to suit all
> parties needs.  But ultimately, it boils down conceptually to be very
> simple.
[...]

If done correctly, dependency resolution is just a topological walk on a
DAG.  Get this right, and everything else falls into place.

To generate the DAG, there would need to be some kind of list of
repositories, which defaults to code.dlang.org (but which can be
customized if necessary -- e.g., add local filesystem repos for local
packages, or a URL to a private additional repository).  There would
also need to be some kind of version resolution scheme.  It shouldn't be
too hard to do: all you need is for repositories to support three
queries:

1) Fetch a list of packages matching the given name(s);

2) Filter said list by a given OS/platform identifier.

3) Filter said list by zero or more given version specs (either an exact
match, or a <= or >= match, or whatever else you may want to filter
version strings by.).

Repositories ought to support such queries efficiently, i.e., the
package manager shouldn't have to download the entire list of 1000
packages each with 50 different version identifiers, and then perform an
O(n) search on the list to find the necessary matching package.  It
should behave like a database: you send it an OS identifier, a list of
desired package names, each with one or more version filters, and it
returns a list of matching package descriptions, including URLs from
which you may download the package(s).  It should take only 1 network
roundtrip to get this information.

Then once the package manager receives the URLs, it can use whatever
method necessary to download said URLs (or just cache it somewhere if
it's a pathname on the local filesystem), and read the package
description(s) to find any additional dependencies that it may need.


T

-- 
A program should be written to model the concepts of the task it performs rather than the physical world or a process because this maximizes the potential for it to be applied to tasks that are conceptually similar and, more important, to tasks that have not yet been conceived. -- Michael B. Allen