Quo vadis, D2? Thoughts on the D library ecosystem.

[Jacob Carlborg]

On 2011-03-20 00:12, Jonathan M Davis wrote:
> On Saturday 19 March 2011 07:43:37 David Nadlinger wrote:
>> While lying in the bed with fever yesterday (so please excuse any
>> careless mistakes), I was pondering a bit about the current discussions
>> regarding Phobos additions, package management, etc. It occurred to me
>> that there is a central unanswered question, which I think deserves to
>> be broadly discussed right now.
>>
>> But first, let me start out by describing how I see current situation
>> regarding D2. Leaving aside a few minor things like @property
>> enforcement or the recent suggestions about a new alias syntax, the
>> language is fairly stable and critical bugs in DMD 2 are not frequent
>> enough to make it completely unusable for day-to-day development
>> anymore. Of course, there is still a large way to go for the D toolchain
>> (with the ideal result being a rock-solid self-hosting compiler
>> front-end, usable as a library as well), but in a sense, we are more or
>> less at the end of a certain stage of D2 development.
>>
>> I think most of you would agree with me if I say that the main goal for
>> D2 right now should be to build a vibrant library ecosystem around the
>> language, to foster adoption in real-world applications. There has been
>> a number of related discussions recently, but as mentioned above, I
>> think there is a central question:
>>
>> Have we reached the critical mass yet where it makes sense to split the
>> effort in a number of smaller library projects, or are we off better
>> with concentrating on a central, comprehensive standard library
>> (Phobos), considering the current community size?
>>
>> I do not really have an answer to this question, but here are a few
>> thoughts on the topic, which might also help to make clearer what I mean:
>>
>> I think that adopting a Boost-like review process for Phobos has
>> certainly been a clever and valuable move, for more than one reason.
>> First, together with the move to Git, it has helped to reinforce the
>> point that D2 and Phobos are open to contributions from everyone, given
>> that they meet certain quality standards. Second, it certainly boosts
>> code quality of further standard library additions, which had been a
>> problem for some parts in the past (at least from my point of view, no
>> offense intended). Third, and this overlaps with another point below, I
>> think that the quality improvements will also help to reduce bit rot,
>> which has traditionally been a problem with D libraries.
>>
>> But however good a fit this model is for the standard library, I think
>> it is no silver bullet either. There are small, one-off style projects,
>> arising from a central need, where the amount of time needed to get the
>> code through the whole review process is prohibitive – even if the code
>> quality was high enough –, but the result is still usable for the wide
>> public. Common examples for this would be low-level wrappers for C
>> libraries, although they don't really qualify for inclusion into Phobos
>> for other reasons (often, another wrapper layer is needed to be usable
>> with common D idioms). Also, people new to the language might be scared
>> away by the mere thought of contributing to a standard library. How to
>> make sure that these libraries are not forgotten? Maybe a central
>> package system with SCM (Git, …) integration can help here?
>>
>> And, which brings me to the next point, how to fight the unfavorable
>> outcome of having a huge inscrutable pile of half-finished bit-rotten
>> code, a problem that DSource is currently experiencing? A central,
>> well-maintained standard library effort with a wider scope could
>> certainly help to reduce this problem, at least from the (D) user side,
>> but on the other hand, larger amounts of code de facto becoming
>> unmaintained would be a problem for it as well.
>>
>> Should we build something like a staging area, an incubator for
>> community contributions not taken yet through formal review, but of
>> interest for a wider audience? What about the etc.* package – would it
>> be an option to expand it into such an incubation area? If not, what
>> should it evolve into – a collection of C-level library bindings (see
>> the recent discussion on SQLite bindings started by David Simcha)? Who
>> will take care of the maintenance duties?
>>
>> Looking forward to a stimulating discussion,
>> David
>
> There has been some discussion in the past of creating an incubator project of
> sorts where code which may or may not make it into Phobos can be put so that it
> can develop and evolve with people actually using it - maybe even using it
> heavily - before it tried to get into Phobos. Once a library was considered
> mature enough, it could go through the Phobos review process and attempt to get
> into the standard library. If it succeeded, then, in theory, we'd have a well-
> used and well-tested library added to Phobos. If it failed, it would still be
> around for people to use, and it could continue to be used and evolve - either
> to make a later attempt at inclusion in Phobos or to simply live as a 3rd party
> library that folks find useful but isn't appropriate for inclusion in the
> standard library for one reason or another.
>
> Exactly what the best way to do all this would be, I don't know, but it would
> need to be far better manage than the current situation with dsource. The
> current activity level of a project would need to clear, as well as how complete
> and stable it's considered. DSource is probably the best place to host such a
> project, but obviously dsource would have to be cleaned up first, and it would
> need someone who was willing to spearhead the effort and manage it. Otherwise,
> we'll just end up with a situation similar to what dsource is now.
>
> Really, the problem is that someone needs to take the initiative on this. They
> need to work on setting it up and supporting the ecosystem which would result in
> a group of such projects. Good ideas tend to be presented around here and then
> go nowhere, because no one actually takes the initiative to do them. The
> "wouldn't this be a good idea?" tactic doesn't tend to get very far, even if
> everyone agrees, simply because someone has to put in the time and effort to do
> it, and while people may think that it's a good idea, there are only so many
> people working on Phobos and other D-related stuff, and there's a lot to be done,
> and everyone has something that they'd like to see done, and _that_ is what
> they're generally working on.
>
> So, I definitely think that an incubation project is a great idea. It has been
> proposed before. However, unless someone steps up to the plate and takes on
> setting it up and organizing it, it's not going to happen.
>
> - Jonathan M Davis

I think if we get a good package management tool for D it doesn't matter 
so much if a library is included in Phobos or not.

-- 
/Jacob Carlborg