Orbit - Package Manager - Specification/ideas

[Jacob Carlborg]

On 2011-07-18 00:26, Nick Sabalausky wrote:
> "Jacob Carlborg"<doob at me.com>  wrote in message
> But why should the name of the project and the name of the tool be
> different? It would be less confusing to have a packager manager named Orb
> that gets invoked via "orb". (Or a package manager named Orbit that gets
> invoked via "orbit".

I don't know. It's also easy to separate the library code in one package 
named "orbit" and the application code in on package named "orb".

> I think the "metadata" would make sence if, and *only* if, it were
> "compiled" down to a pure-data format.

Ok, fair enough. That was the initial idea, to have the "metadata" file 
in a pure-data format.

> As for orbfile vs orbspec: I still think these could and should be merged. I
> don't need a separate buildscript for different actions. *Everything* the
> buildsystem does is configured in one centralized script (and whatever other
> files it may or may not choose to delegate out to). I see no reason why the
> same can't or shouldn't be done with package management.

I guess I can do that.

> If your "orbblah" file doesn't contain the parts needed for orb to
> auto-create a package, then you just can't have orb auto-create your
> package. Simple.

Ok.

> Also, I don't see why the name of "orbspec" needs to be prefixed with the
> package's name. Just like an orbfile, or makefile, or rakefile, etc., you
> already know what package it's for from what directory/package it's inside
> of.

The package needs a name. One option is to specify that in the orbspec 
file. Another option is to specify it in the name of the orbspec file. I 
just thought that it was convenient.

Is it a good idea to take the name directory where the orbspec is located?

Note that many of my choices are based on RubyGems.

> Well, any version that's>= 0.3.4 and<  0.4.0

Ok.

> That would make sense. But, it could always be forcefully done by doing it
> manually without going through orb. Don't know if that's worth worrying
> about though.

Do you mean if you have direct access to the server where the repository 
is located? I don't think that's something worth worrying about. Isn't 
that the same with all servers you don't own yourself? If you have a 
website on a hosted server they can, if they want to, just remove it.


> I agree with inferring 0 for any parts not provided.

Good.

> I think it should be up to the project developer to choose how many parts if
> appropriate for their project. And I don't see any problems with doing that.

Ok. But then I think we most allow: ">  0.3.4+". Or else you cannot know 
if a given version will break the API.

> I think that might be a bit limiting. And keep in mind, just because
> something is source-only doesn't mean it'll actually work on more than one
> platform.

As you can read in the specification I do intend to have binary 
packages. But as I said, that was not my original thought and that's why 
I haven't given it much thought.

Of course it won't work on all platform just because it's source only 
but if there are source only packages then the problem below doesn't exist.

>> I see three options:
>>
>> * One package for all platforms
>> * Include the platform in the package name and in the orbspec
>> * Have a sub path (on the server) for every platform, i.e.:
>>
>> dorbit.org/orbs/linux/dwt-1.3.2.orb.zip
>>
>
> I think the first option makes the most sense, and the second option sounds
> like it would be a sensible workaround when someone really wants things
> separate.

There's always this issue with downloading files you don't need.

> I'm not sure I like the third option because that wouldn't work for
> repositories that aren't explicity chosen unless you made it a standard
> naming system, but then that creates extra requirements for the server, and
> then how do you know whether to GET from the platform-specific directory or
> the platform-independent/cross-platform directory, etc. And what about
> having a windows package and a combination OSX/linux package? It's workable,
> but I'm not sure it's worth it in light of the first two options.

I don't understand what you mean.

Why would you have one package for Windows and one for osx/linux? Or 
have I misunderstood something.

> Although, on second thought, if a good system for the third option could be
> worked out, then maybe that would be best. (But maybe just not for the
> initial release?)

An other issue with one package for all platforms is when building the 
package. The developer first needs to build the package on one platform, 
then move the package to all other platforms and rebuild the package. 
The tool needs to open the package and add files for the other platforms.

>>> - Is it really necessary to have separate "build_dependencies" and
>>> "runtime_dependencies"? And why have both "runtime_dependencies" and
>>> "orbs"
>>> instead of just picking one name and sticking with it?
>>
>> There is no runtime dependency on something that is statically linked.
>> Therefore it would be unnecessary to do a permanent installation on those
>> dependencies. The user could get an option to either permanently installed
>> these dependencies or to temporarily install them.
>>
>
> So you mean a statically-linked dependency then? "Runtime dependency" makes
> me think it's needed at runtime and thus *would* need to be installed.
> Actually, what you're describing sounds just like a build dependency to me.

Statically-linked dependency is basically the same as build dependency. 
If you have a look at the DVM project page you can see the list of build 
dependencies. The only thing you need to actually run the application is 
zlib (on Poisx), because it's dynamically linked, aka a runtime dependency.

> Is there really all that much benefit to doing a temporary install of things
> temporarally needed anyway? And is it that common of a need? I'm wondering
> if it's worth the extra complexity.

The package would need access to the dependencies when its build. How 
should it else be done?

>> The other way around would be possible as well. A package can depend on a
>> dynamically linked library and use it only through function pointers. Then
>> the package would only have a runtime dependency on the library, i.e. it
>> wouldn't be needed when building.
>>
>
> Hmm I think I still don't understand the whole runtime dep vs build dep
> matter.

If you use, on Posix, dlopen and friends to open a dynamic library you 
don't need to link to it when building. You call the functions via 
function pointers, that's how Derelict works, if you have used it.

> That would be up to them. The idea is just that the package metadata is
> extendable for whatever uses may arise.

Ok, that won't be any problem including.

>>> - What's the point of the fields "files", "libraries" and "executables"?
>>> Seems like extra work for no real benefit.
>>
>> "files" is basically all files that should be put in the package.
>> "libraries" and "executables" are all libraries and executables that
>> should be installed (regardless if they are pre-compiled or built during
>> installation).
>>
>
> So it's purely for creating a package?

Yes, the "files" field is just when creating the package.

> If not, then I'm afraid I still don't understand.
>
> If so, then I still don't see the use of "libraries" and "executables". Also
> I think they should all be optional (if they aren't already) because all
> that info is already going to exist in the buildscript, and so a lot of
> people (like me) are going to want to generate the package via their build
> system rather than duplicating all that work and information just so that
> the package manager can do it "automatically".

Both the "libraries" and "executables" fields are optional. "summary" 
and "version" are the only required fields (so far), as you can see on 
the wiki. As I said before, "libraries" and "executables" are what 
actually should be installed.

This is what I'm thinking and is planning for Orbit and my build tool 
Dake. Dake will be able to generate an orbspec file, because, as you 
said, the build tool will have all the necessary information. Then you 
don't have to duplicate any information. But, I'm not forcing anyone to 
use my build tool if they want to use Orbit, therefore the fields 
"libraries" and "executables" are available.

> Other things to think about are: How does the user actually specify the
> login info? Wouldn't want them in the orbfile/orbspec/metadata/etc.

No, absolutely not. Passing user name and password using HTTP basic 
authentication on the command line?

> Ouch. I have to say, I don't like that at all. The way I see it, this is one
> of the two primary responsibilities of a package manager for a language (the
> other being automatic dependency handling). Without it, we're not much ahead
> of where we are right now.

If you have suggestions I'm listening.

> That doesn't work: If orb is managing the packages, then how is
> dake/drake/etc supposed to know what path to include for -I? Orb knows where
> it's sticking packages. The build systems don't know that.

The build tool invokes Orbit and asks about the include path.

> This also reminds me of another issue: For D libraries, the
> orbfile/orbspec/metadata/whatever needs to give the relative path for the
> base include directory.

Something like that. Or if given full path the tool will assume the 
current directory is the base directory and remove that from the path. 
For example:

$ cd ~/projects/d/foo

Contains:

main.d
bar/foobar.d

# foo.orbspec
files << ["~/projects/d/foo/main.d", "~/projects/d/foo/bar/foobar.d"]

$ orb build foo

Since I run the "orb build" command in the "~/projects/d/foo" directory 
the tool will just remove that from the paths when including the files 
in the package.

> I think what we need is to have DVM handle dmd.conf/sc.ini separately from
> the compiler version (which kind of goes along with DVM Issue #9). Orb and
> DVM need to work together to manage dmd.conf/sc.ini. It's complicated, but I
> think it's necessary.

I was hoping I didn't need to drag in DVM in this. And I don't won't to 
force anyone using DVM if they want to use Orbit.

> This makes me think of another, too: How does the buildscript know what
> version of a lib orb has chosen? Since orb packages (rightfully) are allowed
> to specify a range of versions instead of just one specific version, I think
> that will be needed.

It doesn't need to know what exact version is chosen, it just needs to 
know the include paths and libraries to link with. The rest is up to 
Orbit to handle and decide. The build tool just links with what it 
received from Orbit.

> I think that's very, very bad. I should be able to compile something with a
> different version of DMD without reinstalling everything. "dvm use xxx"
> should *just work*, just as it already does now.

The whole point is to be able to have different packages installed with 
different compilers. There could be a command for moving over packages 
from one DMD installation to another.

Say I'm using DMD 2.053, then installing package "foo". Then I'm 
installing DMD 2.054 and switching to it. Say also that package "foo" 
doesn't work with DMD 2.054, what happens when you switch to 2.054?

> Orb manages different versions of arbitrary packages. DVM manages different
> versions of one specific package. If they're so different despite all that,
> then either Orbit isn't generalized enough, or DVM is too specialized.

DVM is very specialized, yes. The installation and switching of D 
compiler, using DVM, is very specialized. DMD requires wrappers, 
manipulating the PATH or registry. None of the Orbit packages require this.

Orbit would require loads of special cases to have DVM built-in. On your 
description DVM and Orbit sound very similar but how they actually work 
when installing and switch compilers are very different. You should know 
this, you ported DVM to Windows.

The current compiler decides what packages are available/installed, that 
wouldn't work if the compiler package would just be a regular orb package.

What would be possible it to create a tool that is a front end for DVM 
and Orbit. I'm currently building Orbit as a library with a thin wrapper 
that is the tool the user will use. I'm planning to rewrite DVM in the 
same way. This also allows to create GUIs and integrate the tools into IDEs.

-- 
/Jacob Carlborg