Article discussing Go, could well be D

[Johannes Pfau]

Jacob Carlborg wrote:
>On 2011-06-18 21:00, Johannes Pfau wrote:
>> Jacob Carlborg wrote:
>>> On 2011-06-16 23:27, Andrei Alexandrescu wrote:
>>>> On 6/16/11 4:19 PM, Jacob Carlborg wrote:
>>>>> I'm already working on a package management tool for D.
>>>>
>>>> Excellent. Suggestion: at the risk of getting flooded with
>>>> suggestions, post your design early and often.
>>>>
>>>> Andrei
>>>
>>> Posting my ideas here as well:
>>> https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D
>>>
>>
>> Some comments to start the suggestion flood ;-)
>>
>> It seems like building orb packages would only work with one specific
>> build system, Dake. I understand that we need a standard way to build
>> packages to allow automated package builds, but I think it should
>> also work with other build systems (waf, make, autotools...).
>> The solution most linux package managers use is to let the 'source
>> package' provide 'build' and 'package' methods. Those methods are
>> then required to store the files in a specific temporary folder and
>> all files from that folder will form the package. Having the config
>> files in Ruby seems like a perfect fit for this approach.
>
>The default build system is Dake. Other build systems are supported as 
>well, like make, dsss, cmake and others. I been thinking about either 
>having methods for different events, like before/after 
>building/installing. Or having a "shell" build method available. For 
>example, if autotools is not an available build method you could run 
>that any way. The tool would basically just execute a string in the
>shell.
>
>You can also read at the top: "These tools can work individually on 
>their own but work much better together.".

Sorry, I must have missed that sentence.
>Also note that this is kind of like an overview of my ideas. I can go 
>into much more detail if necessary. But I though it was too much
>detail (for now) if I wrote down basically the complete orbspec
>specification.

I understand that. I just wanted to post my thoughts about some things
I consider important to package management.
>
>> Also: why does a orbspec have to specify its imports? I think it
>> should rather specify the packages it uses. With some more work it
>> could be possible to even let dmd find all needed imports and guess
>> the needed packages from these imports.
>
>I didn't think straight when I wrote that at first. I was thinking
>about having pre-built libraries in the package and that listed the
>import files needed by the library. But I have already changed that to
>just "files", meaning all files necessary to build the package.

I still don't understand that completely. So does it list the files
which will be contained in the package later, or file dependencies
contained in other packages?
(I'm asking because I'm not familiar
with file-dependencies in package management systems. Most package
management systems make a package depend on other packages, but not on
the files in the packages)
>> Another detail: I wouldn't use .orb for package extensions. We might
>> want to change the compression type later (tar+lzma for example),
>> so .orb.zip would be better. Then we could just use .orb.tar.xz with
>> the new compression. (This is also how archlinux works, for
>> example .pkg.tar.xz)
>
>Ok, I can do that. I chose zip because that is available in Tango and 
>Phobos.

Yes, right now, zip is seems to be the best choice, but at some point
the small size difference between zip and lzma could matter.

>> It seems like C libraries would also be packaged with orb (the sqlite
>> example). This might be needed, but it will be a major pita for linux
>> packagers, as it'll likely cause conflicts. I think it should be
>> possible for those linux packages to hook into orb. Orb should
>> recognize something like 'orb --external libsqlite:library
>> --version 3.7.0' and then just assume that sqlite is installed (but
>> it should not assume that sqlites dependencies are installed - those
>> would have to be registered with --external again). This approach
>> should work well for D packages (so a D package is in Orb first, but
>> some distribution decides to package it. In this case they can add
>> the orb hooks to their packages). It's unlikely that a distribution
>> will change all C packages though. Probably at some time orb should
>> interact with 'pkg-config' to look for already installed C packages,
>> I'm not sure what's a good solution for this problem.
>
>With the sqlite example, I was actually thinking about bindings. But
>as you say, it would be good to be able to specify external
>dependencies, like C libraries.

I totally forgot about bindings! I'm quite uncertain about packaging C
libs though: If we don't package C libs windows users
will have to acquire all C libs manually. But if we do provide C
libraries we have to decide if we also ship the C headers, etc. and it
will be some more maintenance work.
>
>> 'type :library' in the orbspec suggests that there'll be different
>> package types. I think this is a good idea so we don't have to use
>> package name hacks like 'libsqlite' 'libsqlite-dev' (debian)
>> Package types which make sense:
>> :doc -->  documentation. Later possibly in a specific format?
>> :lib -->  shared libraries (.so/.dll) when available
>> :slib -->  static lib (.a/.lib)
>> :dev -->  header files (.di)
>> :src -->  source package used to build other packages
>
>Yes, exactly.

Sound great. Orb could be the first package management system to get
that right. One more question is where .h headers and .di files would
go for C packages. Both in :dev or .h headers into additional :cdev
packages, or something like that.
>
>> We should also think about how the versioning scheme would interact
>> with git/hg/svn whatever snapshots and alpha/beta/rc releases. The
>> debian package system doesn't have explicit support for this which
>> leads to strange version numbers. Archlinux even uses different
>> packages for git versions (libsqlite-git) which also isn't a good
>> solution.
>
>If you have any ideas I'm listening.

OK, my proposal follows, but be warned, it's a little longer than I
first thought :-)

Regarding alpha/beta/rc's a simple scheme could help:
all those releases are pre-releases, so consider a fictional libjson
as an example.

libjson 0.0.1 is released (a final release)
libjson 0.0.2 alpha1 is released --> prerelease 1
libjson 0.0.2 alpha2 is released --> prerelease 2
libjson 0.0.2 beta1 is released --> prerelease 3
libjson 0.0.2 beta2 is released --> prerelease 4
libjson 0.0.2 rc1 is released --> prerelease 5
libjson 0.0.2 is releases (final release)

So in this case 0.0.1 < 0.0.2 pre1 < 0.0.2 pre2 < 0.0.2 pre3 < 0.0.2
pre4 < 0.0.2 pre5 < 0.0.2 pre[X] < 0.0.2

The end user should specify for which packages he'd like to use
prereleases. A standard upgrade for libjson would look like this:
0.0.1 --> 0.0.2 so no prereleases should be installed be default.
If prereleases were enabled for that package, prereleases should be
upgraded automatically:
0.0.1 --> 0.0.2 pre1 --> 0.0.2 pre2 ... --> 0.0.2

It's also possible that a version 0.0.1.1 is released somewhere in
between:
libjson 0.0.1 is released (a final release)
libjson 0.0.2 alpha1 is released --> prerelease 1
libjson 0.0.1.1 rc1 is released --> prerelease 1
libjson 0.0.1.1 is released (a final release)
libjson 0.0.2 alpha2 is released --> prerelease 2
...

in this case: 0.0.1 < 0.0.1.1 pre 1 < 0.0.1.1 < 0.0.2 pre1 < 0.0.2 pre2
< ... < 0.0.2

It should also be possible to skip 0.0.2 (people actually do such
things ;-))
libjson 0.0.1 is released (a final release)
libjson 0.0.2 alpha1 is released --> prerelease 1
libjson 0.0.2 alpha2 is released --> prerelease 2
libjson 0.0.2 beta1 is released --> prerelease 3
libjson 0.0.2 beta2 is released --> prerelease 4
libjson 0.0.2 rc1 is released --> prerelease 5
libjson 0.0.3 rc1 is released -->prerelease 1
libjson 0.0.3 is released --> final release

so: 0.0.1 < 0.0.2 pre1 < 0.0.2 pre2 < ... < 0.0.3 pre1 < 0.0.3
a user not wanting to use prereleases would just skip all preX in the
above examples.

Now regarding snapshot versions: First we have to simplify the problem:
I think we should only support a linear system, so we assume there's
only one master repository and one branch where packages are created
from.
Now we still have the problem that git/hg etc revision numbers cannot
be easily compared (is 5363aed42ff7f2edd796 more recent than
882cc02a58797a313a62 ?).

So I suppose the following: A git/hg/... snapshot always has a 'base'
release. This is the release the snapshot is based on. A snapshot is
always more recent than it's base release:

release1 0.0.1
snapshot1 5363aed42ff7f2edd796 base:0.0.1

so release 1 < snapshot1
snapshots can be based on pre-relases

release1 0.0.1
pre-release1 0.0.2-pre1
snapshot1 5363aed42ff7f2edd796 base:0.0.2-pre1
pre-release2 0.0.2-pre2

so release1 < pre-release1 < snapshot1 < pre-release2

A git snapshot always only replaces it's base release! If there's a
newer base release, the git snapshot is considered to be 'old'.

No how do we sort multiple snapshots based on the same base release?
I think a date based approach makes sense(with second or minute
resolution?)

So we now have this final (and complex) example:

libjson 0.0.1 is released (a final release)         #1
libjson snapshot 2011.06.19.18.45 base:0.0.1        #2
libjson snapshot 2011.06.19.18.46 base:0.0.1        #3
libjson 0.0.2 alpha1 is released --> prerelease 1   #4
libjson snapshot 2011.06.19.18.45 base:0.0.2-pre1   #5
libjson snapshot 2011.06.19.18.50 base:0.0.2-pre1   #6
libjson 0.0.2 alpha2 is released --> prerelease 2   #7
libjson 0.0.1.1 rc1 is released --> prerelease 1    #8
libjson snapshot 2011.06.19.18.45 base:0.0.1.1-pre1 #9
libjson snapshot 2011.06.19.18.50 base:0.0.1.1-pre1 #10
libjson 0.0.1.1 is released (a final release)       #11
libjson 0.0.2 beta1 is released --> prerelease 3    #12
libjson 0.0.2 beta2 is released --> prerelease 4    #13
libjson 0.0.2 rc1 is released --> prerelease 5      #14
libjson 0.0.3 rc1 is released -->prerelease 1       #15
libjson 0.0.3 is released --> final release         #16

0.0.1 < snapshot #2 < snapshot #3 < 0.0.1.1-pre1 < snapshot #9 <
snapshot #10 < 0.0.1.1 < 0.0.2-pre1 < snapshot #5 < snapshot #6 <
0.0.2-pre2 < 0.0.2-pre3 < 0.0.2-pre4 < 0.0.2-pre5 < 0.0.3-pre1 < 0.0.3

Someone with snapshots enabled will get updates like presented in the
above chain, with one exception: "0.0.2-pre1" is released
immediately after "snapshot #3" so the releases 'between' "snapshot #3"
and "0.0.2-pre1" will be skipped!

With snapshots disabled the update path looks like this:
0.0.1 < 0.0.1.1 < 0.0.3

>
>> Also here's a list of variables a source package can set in
>> Archlinux: https://wiki.archlinux.org/index.php/PKGBUILD
>> It might be a good idea to have a look at this and take some
>> inspiration from it. Most of these variables are also useful to
>> orbit.
>>
>> Here's an example Archlinux PKGBUILD:
>> http://pastebin.com/MeXiLDV9
>> The archlinux package system has the easiest source package syntax I
>> know.
>
>I'll have a look.
>

-- 
Johannes Pfau