Article discussing Go, could well be D

[Jacob Carlborg]

On 2011-06-19 19:02, Johannes Pfau wrote:
> I still don't understand that completely. So does it list the files
> which will be contained in the package later, or file dependencies
> contained in other packages?
> (I'm asking because I'm not familiar
> with file-dependencies in package management systems. Most package
> management systems make a package depend on other packages, but not on
> the files in the packages)

Ok, let me explain. When developing a package management system the 
first thing one has do decide is if the package should contain pre-built 
binaries/libraries, we can call these binary packages, or the necessary 
files to build the package when installing, we can call these source 
package (not to be confused with the source type you've mentioned 
below). As a third option, one could have a mixed package system 
containing both binary and source packages. Maybe even mixed packages 
could be possible.

When I first started thinking about Orbit I decided for source packages. 
The reason for this is that the developer only have to create one 
package or doesn't have to build the app/lib for all supported platforms 
when releasing a new version of the package (although it would be good 
to know that it works on all supported platforms).

When I first wrote down my ideas about Orbit on the wiki I was, 
incorrectly, thinking it should have binary packages, hence the 
confusion (even I'm confused). This was when the orbspec examples 
contained the "imports" method. I then corrected this error and changed 
"imports" to "files" to reflect that the packages are source packages.

Now, "files" contains all the necessary files to build the package when 
installing it. These files most somehow be referenced in the orbspec 
because the tool needs to know what files to put in the package. This is 
for a package with no dependencies on other packages.

Note that usually you don't need to explicitly specify all the necessary 
files, thanks to Ruby's fleshed out standard library. For example, you 
could do like this:

imports Dir["**/*.d"]

Which will, recursively, include all *.d files in the current directory. 
If you use Orbit together with Dake you don't have to specify any files 
at all since Dake will know what files to include (this is noted in the 
wiki).

If the package depends on other package this needs to be listed as well 
in the orbspec. This is not showed in any of the examples on the wiki.

I hope this explains most of the things and I'm sorry for any confusion 
I may have caused.

>>> Another detail: I wouldn't use .orb for package extensions. We might
>>> want to change the compression type later (tar+lzma for example),
>>> so .orb.zip would be better. Then we could just use .orb.tar.xz with
>>> the new compression. (This is also how archlinux works, for
>>> example .pkg.tar.xz)
>>
>> Ok, I can do that. I chose zip because that is available in Tango and
>> Phobos.
>
> Yes, right now, zip is seems to be the best choice, but at some point
> the small size difference between zip and lzma could matter.
>
>>> It seems like C libraries would also be packaged with orb (the sqlite
>>> example). This might be needed, but it will be a major pita for linux
>>> packagers, as it'll likely cause conflicts. I think it should be
>>> possible for those linux packages to hook into orb. Orb should
>>> recognize something like 'orb --external libsqlite:library
>>> --version 3.7.0' and then just assume that sqlite is installed (but
>>> it should not assume that sqlites dependencies are installed - those
>>> would have to be registered with --external again). This approach
>>> should work well for D packages (so a D package is in Orb first, but
>>> some distribution decides to package it. In this case they can add
>>> the orb hooks to their packages). It's unlikely that a distribution
>>> will change all C packages though. Probably at some time orb should
>>> interact with 'pkg-config' to look for already installed C packages,
>>> I'm not sure what's a good solution for this problem.
>>
>> With the sqlite example, I was actually thinking about bindings. But
>> as you say, it would be good to be able to specify external
>> dependencies, like C libraries.
>
> I totally forgot about bindings! I'm quite uncertain about packaging C
> libs though: If we don't package C libs windows users
> will have to acquire all C libs manually. But if we do provide C
> libraries we have to decide if we also ship the C headers, etc. and it
> will be some more maintenance work.

Yeah, I don't know exactly what to do with external dependencies. The 
easiest would be to have it just as information for the user of the package.

>>
>>> 'type :library' in the orbspec suggests that there'll be different
>>> package types. I think this is a good idea so we don't have to use
>>> package name hacks like 'libsqlite' 'libsqlite-dev' (debian)
>>> Package types which make sense:
>>> :doc -->   documentation. Later possibly in a specific format?
>>> :lib -->   shared libraries (.so/.dll) when available
>>> :slib -->   static lib (.a/.lib)
>>> :dev -->   header files (.di)
>>> :src -->   source package used to build other packages
>>
>> Yes, exactly.
>
> Sound great. Orb could be the first package management system to get
> that right. One more question is where .h headers and .di files would
> go for C packages. Both in :dev or .h headers into additional :cdev
> packages, or something like that.

I haven't though that far. The first step have to be to get regular, D 
only, packages to work first.

>>
>>> We should also think about how the versioning scheme would interact
>>> with git/hg/svn whatever snapshots and alpha/beta/rc releases. The
>>> debian package system doesn't have explicit support for this which
>>> leads to strange version numbers. Archlinux even uses different
>>> packages for git versions (libsqlite-git) which also isn't a good
>>> solution.
>>
>> If you have any ideas I'm listening.
>
> OK, my proposal follows, but be warned, it's a little longer than I
> first thought :-)
>
> Regarding alpha/beta/rc's a simple scheme could help:
> all those releases are pre-releases, so consider a fictional libjson
> as an example.
>
> libjson 0.0.1 is released (a final release)
> libjson 0.0.2 alpha1 is released -->  prerelease 1
> libjson 0.0.2 alpha2 is released -->  prerelease 2
> libjson 0.0.2 beta1 is released -->  prerelease 3
> libjson 0.0.2 beta2 is released -->  prerelease 4
> libjson 0.0.2 rc1 is released -->  prerelease 5
> libjson 0.0.2 is releases (final release)
>
> So in this case 0.0.1<  0.0.2 pre1<  0.0.2 pre2<  0.0.2 pre3<  0.0.2
> pre4<  0.0.2 pre5<  0.0.2 pre[X]<  0.0.2
>
> The end user should specify for which packages he'd like to use
> prereleases. A standard upgrade for libjson would look like this:
> 0.0.1 -->  0.0.2 so no prereleases should be installed be default.
> If prereleases were enabled for that package, prereleases should be
> upgraded automatically:
> 0.0.1 -->  0.0.2 pre1 -->  0.0.2 pre2 ... -->  0.0.2
>
> It's also possible that a version 0.0.1.1 is released somewhere in
> between:
> libjson 0.0.1 is released (a final release)
> libjson 0.0.2 alpha1 is released -->  prerelease 1
> libjson 0.0.1.1 rc1 is released -->  prerelease 1
> libjson 0.0.1.1 is released (a final release)
> libjson 0.0.2 alpha2 is released -->  prerelease 2
> ...
>
> in this case: 0.0.1<  0.0.1.1 pre 1<  0.0.1.1<  0.0.2 pre1<  0.0.2 pre2
> <  ...<  0.0.2
>
> It should also be possible to skip 0.0.2 (people actually do such
> things ;-))
> libjson 0.0.1 is released (a final release)
> libjson 0.0.2 alpha1 is released -->  prerelease 1
> libjson 0.0.2 alpha2 is released -->  prerelease 2
> libjson 0.0.2 beta1 is released -->  prerelease 3
> libjson 0.0.2 beta2 is released -->  prerelease 4
> libjson 0.0.2 rc1 is released -->  prerelease 5
> libjson 0.0.3 rc1 is released -->prerelease 1
> libjson 0.0.3 is released -->  final release
>
> so: 0.0.1<  0.0.2 pre1<  0.0.2 pre2<  ...<  0.0.3 pre1<  0.0.3
> a user not wanting to use prereleases would just skip all preX in the
> above examples.

Ok, I think I understand so far. I was thinking something similar. But 
is a four digit version really necessary?

> Now regarding snapshot versions: First we have to simplify the problem:
> I think we should only support a linear system, so we assume there's
> only one master repository and one branch where packages are created
> from.
> Now we still have the problem that git/hg etc revision numbers cannot
> be easily compared (is 5363aed42ff7f2edd796 more recent than
> 882cc02a58797a313a62 ?).
>
> So I suppose the following: A git/hg/... snapshot always has a 'base'
> release. This is the release the snapshot is based on. A snapshot is
> always more recent than it's base release:
>
> release1 0.0.1
> snapshot1 5363aed42ff7f2edd796 base:0.0.1
>
> so release 1<  snapshot1
> snapshots can be based on pre-relases
>
> release1 0.0.1
> pre-release1 0.0.2-pre1
> snapshot1 5363aed42ff7f2edd796 base:0.0.2-pre1
> pre-release2 0.0.2-pre2
>
> so release1<  pre-release1<  snapshot1<  pre-release2
>
> A git snapshot always only replaces it's base release! If there's a
> newer base release, the git snapshot is considered to be 'old'.
>
> No how do we sort multiple snapshots based on the same base release?
> I think a date based approach makes sense(with second or minute
> resolution?)
>
> So we now have this final (and complex) example:
>
> libjson 0.0.1 is released (a final release)         #1
> libjson snapshot 2011.06.19.18.45 base:0.0.1        #2
> libjson snapshot 2011.06.19.18.46 base:0.0.1        #3
> libjson 0.0.2 alpha1 is released -->  prerelease 1   #4
> libjson snapshot 2011.06.19.18.45 base:0.0.2-pre1   #5
> libjson snapshot 2011.06.19.18.50 base:0.0.2-pre1   #6
> libjson 0.0.2 alpha2 is released -->  prerelease 2   #7
> libjson 0.0.1.1 rc1 is released -->  prerelease 1    #8
> libjson snapshot 2011.06.19.18.45 base:0.0.1.1-pre1 #9
> libjson snapshot 2011.06.19.18.50 base:0.0.1.1-pre1 #10
> libjson 0.0.1.1 is released (a final release)       #11
> libjson 0.0.2 beta1 is released -->  prerelease 3    #12
> libjson 0.0.2 beta2 is released -->  prerelease 4    #13
> libjson 0.0.2 rc1 is released -->  prerelease 5      #14
> libjson 0.0.3 rc1 is released -->prerelease 1       #15
> libjson 0.0.3 is released -->  final release         #16
>
> 0.0.1<  snapshot #2<  snapshot #3<  0.0.1.1-pre1<  snapshot #9<
> snapshot #10<  0.0.1.1<  0.0.2-pre1<  snapshot #5<  snapshot #6<
> 0.0.2-pre2<  0.0.2-pre3<  0.0.2-pre4<  0.0.2-pre5<  0.0.3-pre1<  0.0.3
>
> Someone with snapshots enabled will get updates like presented in the
> above chain, with one exception: "0.0.2-pre1" is released
> immediately after "snapshot #3" so the releases 'between' "snapshot #3"
> and "0.0.2-pre1" will be skipped!
>
> With snapshots disabled the update path looks like this:
> 0.0.1<  0.0.1.1<  0.0.3

This got quite complex. When I was thinking about SCM integration I was 
thinking about you only specify the address to the repository, which 
will mean the latest commit on the main branch. Then you could also 
specify tags, branches and perhaps specific commits. But you could never 
specify, for example, a release (or commit) newer then another commit. 
This wouldn't work:

orb "dwt", "~> 0.3.4", :git => "git://github.com/jacob-carlborg/libjson.git"

I see now that I've specified a version in the git example on the wiki. 
This was a mistake, I removed the version now.

-- 
/Jacob Carlborg