envy for "Writing Go Packages"

[Graham Fawcett]

Hi BCS,

On Fri, 07 May 2010 01:54:17 +0000, BCS wrote:
n> 
> I haven't watched the video (not enough bandwidth) but I'd like a well
> done package tool as well. Thinking in the direction of a spec, what do
> you like about the Go system?

I'll summarize the video (which, by the way, contains everything I
know about the Go package system!). The screencast shows a developer
writing a short application with a reusable function in it. He factors
the function out into a separate module in a subdirectory, then (with
the aid of a copied Makefile) installs the new module into a shared
location on the host, so it can be reused by other apps. Next, he
pushes the new module into a github repository, and reinstalls the
module from github like so:

goinstall github.com/nf/mytools   # (github user nf, package mytools)

The source is checked out of github, compiled and installed in the
central Go 'pkg' directory (both sources and objects are stored
there). He commits some new changes to github, and reinstalls the
updated module using the same goinstall command.

Finally, he returns to his original application, and replaces the Go
equivalent of "import mytools;" with "import github.com/nf/mytools;"
and the application now refers to the version of mytools that was
installed from Github.

> Some things I'd like:
> 
> -It must work offline (via sneeker net) and from trivial local static
> mirrrors/archives -It should support executing bash commands, D
> executables or something that does most of what people use that kind of
> things for. (Some kind of sandbox/security feature might be needed)

Stepping away from Go specifics, I like the idea of a system that does
some of the following things. (just a brain-dump, no particular
order):

(by the way, I'm overloading the term "package" here, at times to mean
a package in the D sense, but usually to mean a downloadable and
installable unit.)

* It knows how to build and install packages from an arbitrary source,
  specified by a URL: a git/hg/svn repository, a tarball on the
  filesystem, a zipfile on a Web server. Exotic sources/formats could
  possibly be added with plugins.

* It keeps the ceremony of sharing code to a minimum. It should be
  easy to publish a simple library package, without reading an excess
  of documentation or writing an excess of package metadata.

* It separates the concerns of installing packages and managing a
  central repository of packages. As an analogy, consider 'dpkg' vs.
  'apt-get' on Debian systems: one knows about the mechanics of
  package installation and removal, the other can search a central
  catalogue, automatically install dependencies, etc. While 'd-get'
  would be very nice, 'd-dpkg' is the essential building block. This
  is important: the core package tool should not try to do more than
  absolutely necessary to simply fetch and install code.

* It enables (but does not provide) apt-get-like use by providing ways
  to (optionally) declare package dependencies (platform, D-version,
  other needed packages), and include metadata about the package
  (author, website, etc.) but does not require a complex package
  description for simple packages. (An important usability question
  is, what is the minimum package description that should be required
  of a package's author? Can we work with no metadata at all?)

* It should be easy to have both user and global package directories.
  It shouldn't be necessary to 'have root' to install a package.

* It doesn't depend on Make or another build tool, at least not for
  basic library packages. (From the DSSS docs, it seems that DSSS was
  able to manage dependency graphs, and do the right thing with a
  simple 'dsss compile and install' command.) Having said that, a
  package with complex build requirements ought to be buildable with a
  Makefile, perhaps requiring a 'd-build' target in the Makefile that
  builds the package into a state that meets our package system's
  installation requirements.

* It makes it easy for library consumers to integrate the code. After
  I've installed some third-party libaries, I'd like a tool that can
  manage (or support) the build of an application that uses those
  libraries -- for example, by figuring out which object files need to
  be linked in, based on the modules I've imported. This could be a
  tool that just calculates the necessary compiler and linker flags,
  and/or a tool that runs the compiler/linker for me with those flags
  in place, and/or a tool that builds a suitable Makefile for me,
  based on my project's dependencies. (Arguably, this feature is
  outside the scope of a simple fetch-and-install system; but if it's
  not easy to use those fetched packages, then the overall value of
  the system is weakened.)

* Nice to have: a way to allow multiple versions of the same package
  to be installed locally, and a corresponding way to specify which
  version of the package my application is using.

* Nice to have: a way to specify dependencies between third-party
  packages, and dependencies on external libraries (not necessarily a
  way to install those external dependencies, just a declaration).

* Nice to have: some support for running unit tests, especially a
  basic 'pre-install' unit test to confirm that all dependencies are
  in place (if possible).

* Nice to have: a way to publish to a central feed of "known D
  packages". Less formal than dsource.org: not a user site, but a
  machine-readable one, and possibly distributed/mirrorable. As an
  example, the Clojure community has a "clojars.org" site which lets
  anyone publish Clojure packages there. Clojars has very limited
  support for searching/browsing, it's really more of a key/value
  store: it is intended for use by automated tools (it's a Maven
  repository, as I understand it). Clojars actually maintains an
  archive of those packages (as jar files), this might/might not be a
  good idea. I think the "feed" is the key thing.

* Centralizing the feed-of-packages would also enable another useful
  feature: a way of mapping a canonical package name (and optionally,
  version) to the URL of the source for that package. (Possibly
  multiple URLs for the same source.) This is headed into 'apt-get'
  territory, but minimizing the responsibility of the central
  repository to "here's where to find package x.y.z" without any other
  assurances (e.g., that x.y.z will build properly on your system).
  Conventions about package naming (should my DSP package be called
  "dsp", or "fawcett.dsp", "algorithms.signals.dsp", etc.) and
  package-name dispute resolution are orthogonal and should be left
  out of the scope of the system.

These next thoughts are outside the scope; but a good design of the
core tool won't get in the way of future progress in these areas:

* Thinking ahead: ensure the system can be used by continuous
  integration systems, so that e.g. a buildbot could read the feed for
  new "known D packages", and attempt to install them on various
  D-version and OS permutations, reporting the results. (I'm not
  suggesting that the buildbot is part of the package management
  system, just that it should be easy for buildbot maintainers to be
  aware of new package versions so they can be tested.) The same
  facility could be used by developers who want to keep a "mirror" of
  some or all known D packages (even though those packages might
  reside upstream on several different hosts).

* Thinking ahead: eventually we hope for a nice site where people can
  browse, review, grade, and comment upon packages. An opt-in system
  where consumers agree to publish their installation activity (like
  Debian's 'popularity contest').

* Thinking ahead: easy message-digests and digital signatures for
  packages, enabling a truly distributed package-publishing system.
  Anyone can host a d-libraries mirror, and a signed "central feed"
  document can publish the known hashes for those packages to avoid
  man-in-the-middle attacks.

* Thinking ahead: I really like the way github.com shows a 52-week
  activity graph on projects that are hosted there. I don't think we
  can do exactly that, but it would be nice to be able to generate a
  report for all packages, showing how frequently they are updated and
  maintained.

* I personally favour a system that works with D2 only, but I am new
  here. :) Otherwise, the implications of D1/D2 and Phobos/Tango need
  to be considered by more seasoned heads than mine.

And for now, I'm all thought out. :) Your thoughts?

Best,
Graham