Orbit - Package Manager - Specification/ideas

Jacob Carlborg doob at me.com
Fri Jul 15 07:50:37 PDT 2011


On 2011-07-15 09:14, Nick Sabalausky wrote:
> "Jacob Carlborg"<doob at me.com>  wrote in message
> news:ivkrdj$ci4$1 at digitalmars.com...
>> I've written a more formal specification of my ideas for a package manager
>> for D.
>>
>> https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D
>>
>> Note that I am exploring the possibility of using D as the language for
>> all the files mentioned in the link above.
>>
>> The current status is that building packages and installing them works,
>> but quite limited. No dependency tracking or central repository so far.
>>
>> Please comment and suggest.
>>
>
> Good start :)  Here are my random thoughts on it (sorry if you've already
> answered some of them, I haven't read the rest of this thread yet):
>
> - I find the "orb" vs "orbit" distinction a little confusing and
> unnecessary. Why not just call it all "orb"? (Or call everything "orbit"?)

This is how I'm thinking: Orbit is the name of the package manager, 
"orb" is what you type on the command line when interacting with the 
package manager. A package is also called an "orb".

> - I think the files "orbfile", "{name}.orbspec" and "metadata" (the one
> inside the packages) should all just be one file. Frankly, the existence of
> all three of them confuses the hell out of me.

Ok, this is how it works:

"orbfile" is a completely optional file listing all packages a project 
depends on, it shouldn't contain anything else. It allows you to run 
"orb install" in the project directory to install all needed packages. 
Just because a project uses packages doesn't mean it self needs to be a 
package. The orbspec file requires more than just listing dependencies. 
I guess the tool could look for an .orbspec file and install all its 
dependencies. But what happends if it finds several .orbspec files in a 
directory. This is just like the Gemfile for those how have used 
bundler: http://gembundler.com/

"orbspec" is a specification of how a package looks like and what it 
contains. It's intended for creating a package out of a project.

"metadata" is basically the orbspec copied into the archive. I was first 
thinking about "compiling" down the Ruby code into YAML or JSON but for 
now the Ruby code is included in the archive.

> - Are the subtypes a "pick one" or "pick all that are included" deal? I
> think that latter would make more sense.

I'm thinking this can be inferred from other fields like "executables" 
and "libraries". A package could contain several types.

> - Instead of "~>  0.3.4", what about ">  0.3+.4+"? Or ""~>  0.3+.4+"? Or
> something vaguely like that. That would be more flexible.

So you mean I can have a version like this: "> 0.3.4+" meaning any 
version from "0.3.4" to "0.3.9"? It might be a good idea.

> - What happens if someone tries to upload a newer version with the same old
> version number? Or if they forcefully do it?

I haven't thought about that. It probably shouldn't be possible.

> - Are version numbers allowed to have more or less than three parts? I think
> they should. Do version comparisons still work on version numbers with an
> arbitrary number of parts? Again, I think they should.

I was hoping to only have version numbers with three parts. If fewer 
parts are used it would probably easiest to infer a 0 for the missing 
parts, i.e. "1" == "1.0.0".

Is there a need for more parts than three? The whole idea of having 
three version parts is to be able to use "~>  0.3.4". But if "> 0.3.4+" 
would be allowed then arbitrary number of parts could be allowed.

> - Is version "2.10" after "2.1" or are they the same? What about "2.01" vs
> "2.1"? I would vote for "2.10>  2.1 == 2.01", because I see version "parts"
> as distinct numbers separated by a period, rather than fractional numbers.

I haven't thought about that. I see version parts as distinct numbers as 
well.

> - It should allow boolean operators and parens for the version selections.
> For instance: "(>= 2.1&&  <= 2.6&&  != 2.4) ||>= 3.4" (Ie, "Any version
> from 2.1 through 2.6, but 2.4 has critical bugs, and 3.4+ contains a 2.x
> compatibility layer.")

Hehe. Now this is getting quite complicate, but it would be nice to have 
yes. Not something I will aim for in the first release.

> - There should be a "list" command and a "list {package name}" command to
> see what's installed. Maybe even "list {package name} {version expression}"?
> And maybe something too see what's available but not yet installed? They
> should all list the versions in guaranteed-sorted order so you can see which
> are the newest and oldest installed versions by looking at the first and
> last.

Absolutely, I've completely forgot about the "list" command.

> - At some point, 7z should be supported (and tarballs, of course).

If someone is willing to create D module of create bindings for any 
available libraries. Any libraries the forces a specific license is out 
of the question.

> - How should platforms be handled WRT packages? Ie, Do all platforms need to
> be in the same orb package? Do they all need to be in separate packages?
> Either way? If they're not all required to be in the same package, how does
> orb find the package that had the right platform?

I good questions. I haven't given binary packages that much thought. I 
was first going for a source only package manager that requires all 
packages to be built before installed.

I see three options:

* One package for all platforms
* Include the platform in the package name and in the orbspec
* Have a sub path (on the server) for every platform, i.e.:

dorbit.org/orbs/linux/dwt-1.3.2.orb.zip

> - Is it really necessary to have separate "build_dependencies" and
> "runtime_dependencies"? And why have both "runtime_dependencies" and "orbs"
> instead of just picking one name and sticking with it?

There is no runtime dependency on something that is statically linked. 
Therefore it would be unnecessary to do a permanent installation on 
those dependencies. The user could get an option to either permanently 
installed these dependencies or to temporarily install them.

The other way around would be possible as well. A package can depend on 
a dynamically linked library and use it only through function pointers. 
Then the package would only have a runtime dependency on the library, 
i.e. it wouldn't be needed when building.

> - Would it be a good idea to have and additional field "extra" for
> non-standard expandability? So people could add extra fields they felt would
> be useful, like "extra.foo" and "extra.bar", etc. And popular ones could
> eventually be formally added to the specification as just simply "foo" and
> "bar".

I guess there could be an "extra" field accepting a hash. How would the 
field be used, by other tools?

> - What's the point of the fields "files", "libraries" and "executables"?
> Seems like extra work for no real benefit.

"files" is basically all files that should be put in the package. 
"libraries" and "executables" are all libraries and executables that 
should be installed (regardless if they are pre-compiled or built during 
installation).

> - This supports having multiple versions of the same package installed at
> the same time, right? If not it should.

Yes, that's the whole point of having version, as I see it.

> - I see there's an "upgrade" callback, but I didn't see an "upgrade"
> command. Is upgrading in or out? I think that there should be an "upgrade"
> command that upgrades the installed versions of packages as far as it can
> *without* breaking any other installed packages that depend on it. Ex, if
> Foo requires Bar v2.6 or earlier, and SuperFoo requires Bar v2.7 or earlier,
> and Bar v2.3 is installed, but the latest Bar is v2.9, then "upgrade bar"
> would upgrade Bar v2.3 to v2.6 and display a message that says "Bar upgraded
> from 2.3 to 2.6, but the newest is 2.9, run "orb install Bar" to install the
> newest Bar, too." (Or maybe it should install both 2.6 and 2.7? Or one/both
> of those and 2.9?) For upgrading, we should also think about how to do
> upgrades without clobbering any of it's settings.

I guess I didn't think this through. I think it will require some thought.

> - For POSTing a package to a repository, how does authentication work? All
> repos don't have to provide unrestricted upload access do they?

I haven't thought about this more than there will be some kind of 
authentication. Probably HTTP basic authentication.

> - I'm not sure I understand how the "source" command works. Can it be
> provided more than once? And then it just picks the first one that actually
> has the package?

The "source" command specifies a path to a repository where to fetch 
packages from. I haven't thought about if the it can be provided more 
than once. It might be a good idea.

> - The "central repositories" don't necessarily sound all that central, so
> they probably should just be called "repositories".

Ok.

> - What about default repositories? It should support that. (Kinda makes
> sense, otherwise how would "orb install xxx" know where to look?) And there
> should be simple commands to add/update/remove/list (and reorder?) the
> default repositories. If a package A specifies a dependency B and a
> repository for that dependency B, then which one has priority for
> downloading B: The default repositories or the repository specified by
> package A?

Yes there will be a default repository. As someone else suggested there 
could be an "orbfile" in the users home directory that can contain 
default settings like for "source". Or a more general config file for orbit.

> - Here's a problem with using an actual programming language for the
> orbfile/name.orbspec/metadata file: Suppose Orb version X uses Ruby version
> Y. Then, Orb X+1 comes out and which has Ruby upgraded to Y+1. Now, someone
> creates PackageA with an orbfile/name.orbspec/whatever that relies on Ruby
> Y+1. Someone else still has Orb version X and tries to get PackageA. Kaboom!

Yeah, that is a problem. But I wonder how much this will be a problem in 
practice. I don't think this will so big problem in practice using Ruby 
as the language. On the other hand, using D, will be a big problem. D 
breaks something in every release.

> Therefore, the orbfile/name.orbspec/whatever needs to specify which version
> of Orb (or Ruby) it requires. But now we have a chicken-and-the-egg problem:
> How can Orb X figure out that PackageA requires Orb X+1 if Orb X can't
> properly read PackageA's orbfile/name.orbspec/whatever?

That is a problem. If the metadata is "compiled" YAML/JSON then we can 
get around this.

> - If I install D library "libfoo", then I should be able write myapp.d with
> "include foo.blah;" and then do "dmd myapp.d" *without* manually
> specifying -Ipath_to_libfoo. It should just work. How will Orb handle that?

It can't. The solution to this is a build tool, as I see it. The build 
tool knows about the package manager and let you specify dependencies on 
package. Think about Drake, it could look like this:

target("myapp.d", {
     orb("libfoo"); // automatically links with "libfoo" and includes 
its header path.
});

-Ipath_to_libfoo needs somehow be passed to the compiler, and linking 
with the library as well. Maybe it would be possible to manipulate the 
dmd.conf/sc.ini but this seems very complicated.

> And how will that interact with DVM? Ie, if I do "dvm use 2.051", then "orb
> install libfoo", then "dvm use 2.054", then I should still have access to
> libfoo without needing to specify -Ipath_to_libfoo.

If you do "dvm use 2.051", then "orb install libfoo" then "libfoo" will 
only be installed for dmd 2.051, that's the whole point. You would have 
to run "orb install libfoo" again after switching compiler. Orbit could 
of course share the same package if possible.

> - Where does everything get installed?

For now, on Posix, in /usr/local/orbit. If used through DVM it will be 
installed in somewhere in ~/.dvm.

> - In many ways this sounds a lot like a generalized DVM. Maybe Orb should
> eventually take over DVM's duties by making a DMD orb package.

No, I don't think so. DVM is quite specialized in what it does. 
Manipulating the PATH variable (or the registy on Windows) to be able to 
do what it does. I don't what to mix DVM and Orbit.

-- 
/Jacob Carlborg


More information about the Digitalmars-d mailing list