Orbit - Package Manager - Specification/ideas

[Nick Sabalausky]

"Jacob Carlborg" <doob at me.com> wrote in message 
news:ivpkdp$hq4$1 at digitalmars.com...
> On 2011-07-15 09:14, Nick Sabalausky wrote:
>> "Jacob Carlborg"<doob at me.com>  wrote in message
>> news:ivkrdj$ci4$1 at digitalmars.com...
>>> I've written a more formal specification of my ideas for a package 
>>> manager
>>> for D.
>>>
>>> https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D
>>>
>>> Note that I am exploring the possibility of using D as the language for
>>> all the files mentioned in the link above.
>>>
>>> The current status is that building packages and installing them works,
>>> but quite limited. No dependency tracking or central repository so far.
>>>
>>> Please comment and suggest.
>>>
>>
>> Good start :)  Here are my random thoughts on it (sorry if you've already
>> answered some of them, I haven't read the rest of this thread yet):
>>
>> - I find the "orb" vs "orbit" distinction a little confusing and
>> unnecessary. Why not just call it all "orb"? (Or call everything 
>> "orbit"?)
>
> This is how I'm thinking: Orbit is the name of the package manager, "orb" 
> is what you type on the command line when interacting with the package 
> manager. A package is also called an "orb".
>

But why should the name of the project and the name of the tool be 
different? It would be less confusing to have a packager manager named Orb 
that gets invoked via "orb". (Or a package manager named Orbit that gets 
invoked via "orbit".

>> - I think the files "orbfile", "{name}.orbspec" and "metadata" (the one
>> inside the packages) should all just be one file. Frankly, the existence 
>> of
>> all three of them confuses the hell out of me.
>
> Ok, this is how it works:
>
> "orbfile" is a completely optional file listing all packages a project 
> depends on, it shouldn't contain anything else. It allows you to run "orb 
> install" in the project directory to install all needed packages. Just 
> because a project uses packages doesn't mean it self needs to be a 
> package. The orbspec file requires more than just listing dependencies. I 
> guess the tool could look for an .orbspec file and install all its 
> dependencies. But what happends if it finds several .orbspec files in a 
> directory. This is just like the Gemfile for those how have used bundler: 
> http://gembundler.com/
>
> "orbspec" is a specification of how a package looks like and what it 
> contains. It's intended for creating a package out of a project.
>
> "metadata" is basically the orbspec copied into the archive. I was first 
> thinking about "compiling" down the Ruby code into YAML or JSON but for 
> now the Ruby code is included in the archive.
>

I think the "metadata" would make sence if, and *only* if, it were 
"compiled" down to a pure-data format.

As for orbfile vs orbspec: I still think these could and should be merged. I 
don't need a separate buildscript for different actions. *Everything* the 
buildsystem does is configured in one centralized script (and whatever other 
files it may or may not choose to delegate out to). I see no reason why the 
same can't or shouldn't be done with package management.

If your "orbblah" file doesn't contain the parts needed for orb to 
auto-create a package, then you just can't have orb auto-create your 
package. Simple.

Also, I don't see why the name of "orbspec" needs to be prefixed with the 
package's name. Just like an orbfile, or makefile, or rakefile, etc., you 
already know what package it's for from what directory/package it's inside 
of.


>> - Instead of "~>  0.3.4", what about ">  0.3+.4+"? Or ""~>  0.3+.4+"? Or
>> something vaguely like that. That would be more flexible.
>
> So you mean I can have a version like this: "> 0.3.4+" meaning any version 
> from "0.3.4" to "0.3.9"? It might be a good idea.
>

Well, any version that's >= 0.3.4 and < 0.4.0


>> - What happens if someone tries to upload a newer version with the same 
>> old
>> version number? Or if they forcefully do it?
>
> I haven't thought about that. It probably shouldn't be possible.
>

That would make sense. But, it could always be forcefully done by doing it 
manually without going through orb. Don't know if that's worth worrying 
about though.


>> - Are version numbers allowed to have more or less than three parts? I 
>> think
>> they should. Do version comparisons still work on version numbers with an
>> arbitrary number of parts? Again, I think they should.
>
> I was hoping to only have version numbers with three parts. If fewer parts 
> are used it would probably easiest to infer a 0 for the missing parts, 
> i.e. "1" == "1.0.0".
>

I agree with inferring 0 for any parts not provided.


> Is there a need for more parts than three? The whole idea of having three 
> version parts is to be able to use "~>  0.3.4". But if "> 0.3.4+" would be 
> allowed then arbitrary number of parts could be allowed.
>

I think it should be up to the project developer to choose how many parts if 
appropriate for their project. And I don't see any problems with doing that.


>> - It should allow boolean operators and parens for the version 
>> selections.
>> For instance: "(>= 2.1&&  <= 2.6&&  != 2.4) ||>= 3.4" (Ie, "Any version
>> from 2.1 through 2.6, but 2.4 has critical bugs, and 3.4+ contains a 2.x
>> compatibility layer.")
>
> Hehe. Now this is getting quite complicate, but it would be nice to have 
> yes. Not something I will aim for in the first release.
>

Fair enough. :)


>> - How should platforms be handled WRT packages? Ie, Do all platforms need 
>> to
>> be in the same orb package? Do they all need to be in separate packages?
>> Either way? If they're not all required to be in the same package, how 
>> does
>> orb find the package that had the right platform?
>
> I good questions. I haven't given binary packages that much thought. I was 
> first going for a source only package manager that requires all packages 
> to be built before installed.
>

I think that might be a bit limiting. And keep in mind, just because 
something is source-only doesn't mean it'll actually work on more than one 
platform.


> I see three options:
>
> * One package for all platforms
> * Include the platform in the package name and in the orbspec
> * Have a sub path (on the server) for every platform, i.e.:
>
> dorbit.org/orbs/linux/dwt-1.3.2.orb.zip
>

I think the first option makes the most sense, and the second option sounds 
like it would be a sensible workaround when someone really wants things 
separate.

I'm not sure I like the third option because that wouldn't work for 
repositories that aren't explicity chosen unless you made it a standard 
naming system, but then that creates extra requirements for the server, and 
then how do you know whether to GET from the platform-specific directory or 
the platform-independent/cross-platform directory, etc. And what about 
having a windows package and a combination OSX/linux package? It's workable, 
but I'm not sure it's worth it in light of the first two options.

Although, on second thought, if a good system for the third option could be 
worked out, then maybe that would be best. (But maybe just not for the 
initial release?)


>> - Is it really necessary to have separate "build_dependencies" and
>> "runtime_dependencies"? And why have both "runtime_dependencies" and 
>> "orbs"
>> instead of just picking one name and sticking with it?
>
> There is no runtime dependency on something that is statically linked. 
> Therefore it would be unnecessary to do a permanent installation on those 
> dependencies. The user could get an option to either permanently installed 
> these dependencies or to temporarily install them.
>

So you mean a statically-linked dependency then? "Runtime dependency" makes 
me think it's needed at runtime and thus *would* need to be installed. 
Actually, what you're describing sounds just like a build dependency to me.

Is there really all that much benefit to doing a temporary install of things 
temporarally needed anyway? And is it that common of a need? I'm wondering 
if it's worth the extra complexity.


> The other way around would be possible as well. A package can depend on a 
> dynamically linked library and use it only through function pointers. Then 
> the package would only have a runtime dependency on the library, i.e. it 
> wouldn't be needed when building.
>

Hmm I think I still don't understand the whole runtime dep vs build dep 
matter.


>> - Would it be a good idea to have and additional field "extra" for
>> non-standard expandability? So people could add extra fields they felt 
>> would
>> be useful, like "extra.foo" and "extra.bar", etc. And popular ones could
>> eventually be formally added to the specification as just simply "foo" 
>> and
>> "bar".
>
> I guess there could be an "extra" field accepting a hash. How would the 
> field be used, by other tools?
>

That would be up to them. The idea is just that the package metadata is 
extendable for whatever uses may arise.


>> - What's the point of the fields "files", "libraries" and "executables"?
>> Seems like extra work for no real benefit.
>
> "files" is basically all files that should be put in the package. 
> "libraries" and "executables" are all libraries and executables that 
> should be installed (regardless if they are pre-compiled or built during 
> installation).
>

So it's purely for creating a package?

If not, then I'm afraid I still don't understand.

If so, then I still don't see the use of "libraries" and "executables". Also 
I think they should all be optional (if they aren't already) because all 
that info is already going to exist in the buildscript, and so a lot of 
people (like me) are going to want to generate the package via their build 
system rather than duplicating all that work and information just so that 
the package manager can do it "automatically".


>> - For POSTing a package to a repository, how does authentication work? 
>> All
>> repos don't have to provide unrestricted upload access do they?
>
> I haven't thought about this more than there will be some kind of 
> authentication. Probably HTTP basic authentication.
>

Other things to think about are: How does the user actually specify the 
login info? Wouldn't want them in the orbfile/orbspec/metadata/etc.


>> - If I install D library "libfoo", then I should be able write myapp.d 
>> with
>> "include foo.blah;" and then do "dmd myapp.d" *without* manually
>> specifying -Ipath_to_libfoo. It should just work. How will Orb handle 
>> that?
>
> It can't. The solution to this is a build tool, as I see it.

Ouch. I have to say, I don't like that at all. The way I see it, this is one 
of the two primary responsibilities of a package manager for a language (the 
other being automatic dependency handling). Without it, we're not much ahead 
of where we are right now.


> The build tool knows about the package manager and let you specify 
> dependencies on package. Think about Drake, it could look like this:
>
> target("myapp.d", {
>     orb("libfoo"); // automatically links with "libfoo" and includes its 
> header path.
> });
>

That doesn't work: If orb is managing the packages, then how is 
dake/drake/etc supposed to know what path to include for -I? Orb knows where 
it's sticking packages. The build systems don't know that.

This also reminds me of another issue: For D libraries, the 
orbfile/orbspec/metadata/whatever needs to give the relative path for the 
base include directory.


> -Ipath_to_libfoo needs somehow be passed to the compiler, and linking with 
> the library as well. Maybe it would be possible to manipulate the 
> dmd.conf/sc.ini but this seems very complicated.
>

I think what we need is to have DVM handle dmd.conf/sc.ini separately from 
the compiler version (which kind of goes along with DVM Issue #9). Orb and 
DVM need to work together to manage dmd.conf/sc.ini. It's complicated, but I 
think it's necessary.

This makes me think of another, too: How does the buildscript know what 
version of a lib orb has chosen? Since orb packages (rightfully) are allowed 
to specify a range of versions instead of just one specific version, I think 
that will be needed.


>> And how will that interact with DVM? Ie, if I do "dvm use 2.051", then 
>> "orb
>> install libfoo", then "dvm use 2.054", then I should still have access to
>> libfoo without needing to specify -Ipath_to_libfoo.
>
> If you do "dvm use 2.051", then "orb install libfoo" then "libfoo" will 
> only be installed for dmd 2.051, that's the whole point. You would have to 
> run "orb install libfoo" again after switching compiler.

I think that's very, very bad. I should be able to compile something with a 
different version of DMD without reinstalling everything. "dvm use xxx" 
should *just work*, just as it already does now.


>> - In many ways this sounds a lot like a generalized DVM. Maybe Orb should
>> eventually take over DVM's duties by making a DMD orb package.
>
> No, I don't think so. DVM is quite specialized in what it does. 
> Manipulating the PATH variable (or the registy on Windows) to be able to 
> do what it does. I don't what to mix DVM and Orbit.
>

Orb manages different versions of arbitrary packages. DVM manages different 
versions of one specific package. If they're so different despite all that, 
then either Orbit isn't generalized enough, or DVM is too specialized.