Orbit - Package Manager - Specification/ideas

[Johannes Pfau]

Jacob Carlborg wrote:
>On 2011-07-15 09:14, Nick Sabalausky wrote:
>> "Jacob Carlborg"<doob at me.com>  wrote in message
>> news:ivkrdj$ci4$1 at digitalmars.com...
>>> I've written a more formal specification of my ideas for a package
>>> manager for D.
>>>
>>> https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D
>>>
>>> Note that I am exploring the possibility of using D as the language
>>> for all the files mentioned in the link above.
>>>
>>> The current status is that building packages and installing them
>>> works, but quite limited. No dependency tracking or central
>>> repository so far.
>>>
>>> Please comment and suggest.
>>>
>>
>> Good start :)  Here are my random thoughts on it (sorry if you've
>> already answered some of them, I haven't read the rest of this
>> thread yet):
>>
>> - I find the "orb" vs "orbit" distinction a little confusing and
>> unnecessary. Why not just call it all "orb"? (Or call everything
>> "orbit"?)
>
>This is how I'm thinking: Orbit is the name of the package manager, 
>"orb" is what you type on the command line when interacting with the 
>package manager. A package is also called an "orb".
>
>> - I think the files "orbfile", "{name}.orbspec" and "metadata" (the
>> one inside the packages) should all just be one file. Frankly, the
>> existence of all three of them confuses the hell out of me.
>
>Ok, this is how it works:
>
>"orbfile" is a completely optional file listing all packages a project 
>depends on, it shouldn't contain anything else. It allows you to run 
>"orb install" in the project directory to install all needed packages. 
>Just because a project uses packages doesn't mean it self needs to be
>a package. The orbspec file requires more than just listing
>dependencies. I guess the tool could look for an .orbspec file and
>install all its dependencies. But what happends if it finds
>several .orbspec files in a directory. This is just like the Gemfile
>for those how have used bundler: http://gembundler.com/
>
>"orbspec" is a specification of how a package looks like and what it 
>contains. It's intended for creating a package out of a project.
>
>"metadata" is basically the orbspec copied into the archive. I was
>first thinking about "compiling" down the Ruby code into YAML or JSON
>but for now the Ruby code is included in the archive.
>
>> - Are the subtypes a "pick one" or "pick all that are included"
>> deal? I think that latter would make more sense.
>
>I'm thinking this can be inferred from other fields like "executables" 
>and "libraries". A package could contain several types.
>
>> - Instead of "~>  0.3.4", what about ">  0.3+.4+"? Or ""~>
>> 0.3+.4+"? Or something vaguely like that. That would be more
>> flexible.
>
>So you mean I can have a version like this: "> 0.3.4+" meaning any 
>version from "0.3.4" to "0.3.9"? It might be a good idea.
>
>> - What happens if someone tries to upload a newer version with the
>> same old version number? Or if they forcefully do it?
>
>I haven't thought about that. It probably shouldn't be possible.
>
>> - Are version numbers allowed to have more or less than three parts?
>> I think they should. Do version comparisons still work on version
>> numbers with an arbitrary number of parts? Again, I think they
>> should.
>
>I was hoping to only have version numbers with three parts. If fewer 
>parts are used it would probably easiest to infer a 0 for the missing 
>parts, i.e. "1" == "1.0.0".
>
>Is there a need for more parts than three? The whole idea of having 
>three version parts is to be able to use "~>  0.3.4". But if ">
>0.3.4+" would be allowed then arbitrary number of parts could be
>allowed.
>
>> - Is version "2.10" after "2.1" or are they the same? What about
>> "2.01" vs "2.1"? I would vote for "2.10>  2.1 == 2.01", because I
>> see version "parts" as distinct numbers separated by a period,
>> rather than fractional numbers.
>
>I haven't thought about that. I see version parts as distinct numbers
>as well.
>
>> - It should allow boolean operators and parens for the version
>> selections. For instance: "(>= 2.1&&  <= 2.6&&  != 2.4) ||>=
>> 3.4" (Ie, "Any version from 2.1 through 2.6, but 2.4 has critical
>> bugs, and 3.4+ contains a 2.x compatibility layer.")
>
>Hehe. Now this is getting quite complicate, but it would be nice to
>have yes. Not something I will aim for in the first release.
>
>> - There should be a "list" command and a "list {package name}"
>> command to see what's installed. Maybe even "list {package name}
>> {version expression}"? And maybe something too see what's available
>> but not yet installed? They should all list the versions in
>> guaranteed-sorted order so you can see which are the newest and
>> oldest installed versions by looking at the first and last.
>
>Absolutely, I've completely forgot about the "list" command.
>
>> - At some point, 7z should be supported (and tarballs, of course).
>
>If someone is willing to create D module of create bindings for any 
>available libraries. Any libraries the forces a specific license is
>out of the question.

The lzma SDK is public domain (lzma is the compression algorithm used
in 7z), xzutils has a library called liblzma which is based on the lzma
sdk, easier to use and also public domain. Tar should be easy enough to
implement in d. Seems like there are more or less recent D
bindings for liblzma:
http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=117344
But someone still has to combine those parts together. LZMA bindings
would also fit nicely into phobos, as we already have zlib in there.

>
>> - How should platforms be handled WRT packages? Ie, Do all platforms
>> need to be in the same orb package? Do they all need to be in
>> separate packages? Either way? If they're not all required to be in
>> the same package, how does orb find the package that had the right
>> platform?
>
>I good questions. I haven't given binary packages that much thought. I 
>was first going for a source only package manager that requires all 
>packages to be built before installed.
>
>I see three options:
>
>* One package for all platforms
>* Include the platform in the package name and in the orbspec
>* Have a sub path (on the server) for every platform, i.e.:
>
>dorbit.org/orbs/linux/dwt-1.3.2.orb.zip
>
>> - Is it really necessary to have separate "build_dependencies" and
>> "runtime_dependencies"? And why have both "runtime_dependencies" and
>> "orbs" instead of just picking one name and sticking with it?
>
>There is no runtime dependency on something that is statically linked. 
>Therefore it would be unnecessary to do a permanent installation on 
>those dependencies. The user could get an option to either permanently 
>installed these dependencies or to temporarily install them.
>
>The other way around would be possible as well. A package can depend
>on a dynamically linked library and use it only through function
>pointers. Then the package would only have a runtime dependency on the
>library, i.e. it wouldn't be needed when building.
>
>> - Would it be a good idea to have and additional field "extra" for
>> non-standard expandability? So people could add extra fields they
>> felt would be useful, like "extra.foo" and "extra.bar", etc. And
>> popular ones could eventually be formally added to the specification
>> as just simply "foo" and "bar".
>
>I guess there could be an "extra" field accepting a hash. How would
>the field be used, by other tools?
>
>> - What's the point of the fields "files", "libraries" and
>> "executables"? Seems like extra work for no real benefit.
>
>"files" is basically all files that should be put in the package. 
>"libraries" and "executables" are all libraries and executables that 
>should be installed (regardless if they are pre-compiled or built
>during installation).
>
>> - This supports having multiple versions of the same package
>> installed at the same time, right? If not it should.
>
>Yes, that's the whole point of having version, as I see it.
>
>> - I see there's an "upgrade" callback, but I didn't see an "upgrade"
>> command. Is upgrading in or out? I think that there should be an
>> "upgrade" command that upgrades the installed versions of packages
>> as far as it can *without* breaking any other installed packages
>> that depend on it. Ex, if Foo requires Bar v2.6 or earlier, and
>> SuperFoo requires Bar v2.7 or earlier, and Bar v2.3 is installed,
>> but the latest Bar is v2.9, then "upgrade bar" would upgrade Bar
>> v2.3 to v2.6 and display a message that says "Bar upgraded from 2.3
>> to 2.6, but the newest is 2.9, run "orb install Bar" to install the
>> newest Bar, too." (Or maybe it should install both 2.6 and 2.7? Or
>> one/both of those and 2.9?) For upgrading, we should also think
>> about how to do upgrades without clobbering any of it's settings.
>
>I guess I didn't think this through. I think it will require some
>thought.
>
>> - For POSTing a package to a repository, how does authentication
>> work? All repos don't have to provide unrestricted upload access do
>> they?
>
>I haven't thought about this more than there will be some kind of 
>authentication. Probably HTTP basic authentication.
>
>> - I'm not sure I understand how the "source" command works. Can it be
>> provided more than once? And then it just picks the first one that
>> actually has the package?
>
>The "source" command specifies a path to a repository where to fetch 
>packages from. I haven't thought about if the it can be provided more 
>than once. It might be a good idea.
>
>> - The "central repositories" don't necessarily sound all that
>> central, so they probably should just be called "repositories".
>
>Ok.
>
>> - What about default repositories? It should support that. (Kinda
>> makes sense, otherwise how would "orb install xxx" know where to
>> look?) And there should be simple commands to add/update/remove/list
>> (and reorder?) the default repositories. If a package A specifies a
>> dependency B and a repository for that dependency B, then which one
>> has priority for downloading B: The default repositories or the
>> repository specified by package A?
>
>Yes there will be a default repository. As someone else suggested
>there could be an "orbfile" in the users home directory that can
>contain default settings like for "source". Or a more general config
>file for orbit.
>
>> - Here's a problem with using an actual programming language for the
>> orbfile/name.orbspec/metadata file: Suppose Orb version X uses Ruby
>> version Y. Then, Orb X+1 comes out and which has Ruby upgraded to
>> Y+1. Now, someone creates PackageA with an
>> orbfile/name.orbspec/whatever that relies on Ruby Y+1. Someone else
>> still has Orb version X and tries to get PackageA. Kaboom!
>
>Yeah, that is a problem. But I wonder how much this will be a problem
>in practice. I don't think this will so big problem in practice using
>Ruby as the language. On the other hand, using D, will be a big
>problem. D breaks something in every release.
>
>> Therefore, the orbfile/name.orbspec/whatever needs to specify which
>> version of Orb (or Ruby) it requires. But now we have a
>> chicken-and-the-egg problem: How can Orb X figure out that PackageA
>> requires Orb X+1 if Orb X can't properly read PackageA's
>> orbfile/name.orbspec/whatever?
>
>That is a problem. If the metadata is "compiled" YAML/JSON then we can 
>get around this.
>
>> - If I install D library "libfoo", then I should be able write
>> myapp.d with "include foo.blah;" and then do "dmd myapp.d" *without*
>> manually specifying -Ipath_to_libfoo. It should just work. How will
>> Orb handle that?
>
>It can't. The solution to this is a build tool, as I see it. The build 
>tool knows about the package manager and let you specify dependencies
>on package. Think about Drake, it could look like this:
>
>target("myapp.d", {
>     orb("libfoo"); // automatically links with "libfoo" and includes 
>its header path.
>});
>
>-Ipath_to_libfoo needs somehow be passed to the compiler, and linking 
>with the library as well. Maybe it would be possible to manipulate the 
>dmd.conf/sc.ini but this seems very complicated.
>
>> And how will that interact with DVM? Ie, if I do "dvm use 2.051",
>> then "orb install libfoo", then "dvm use 2.054", then I should still
>> have access to libfoo without needing to specify -Ipath_to_libfoo.
>
>If you do "dvm use 2.051", then "orb install libfoo" then "libfoo"
>will only be installed for dmd 2.051, that's the whole point. You
>would have to run "orb install libfoo" again after switching compiler.
>Orbit could of course share the same package if possible.
>
>> - Where does everything get installed?
>
>For now, on Posix, in /usr/local/orbit. If used through DVM it will be 
>installed in somewhere in ~/.dvm.
>
>> - In many ways this sounds a lot like a generalized DVM. Maybe Orb
>> should eventually take over DVM's duties by making a DMD orb package.
>
>No, I don't think so. DVM is quite specialized in what it does. 
>Manipulating the PATH variable (or the registy on Windows) to be able
>to do what it does. I don't what to mix DVM and Orbit.
>


-- 
Johannes Pfau