Quo vadis, D2? Thoughts on the D library ecosystem.

[Jonathan M Davis]

On Saturday 19 March 2011 07:43:37 David Nadlinger wrote:
> While lying in the bed with fever yesterday (so please excuse any
> careless mistakes), I was pondering a bit about the current discussions
> regarding Phobos additions, package management, etc. It occurred to me
> that there is a central unanswered question, which I think deserves to
> be broadly discussed right now.
> 
> But first, let me start out by describing how I see current situation
> regarding D2. Leaving aside a few minor things like @property
> enforcement or the recent suggestions about a new alias syntax, the
> language is fairly stable and critical bugs in DMD 2 are not frequent
> enough to make it completely unusable for day-to-day development
> anymore. Of course, there is still a large way to go for the D toolchain
> (with the ideal result being a rock-solid self-hosting compiler
> front-end, usable as a library as well), but in a sense, we are more or
> less at the end of a certain stage of D2 development.
> 
> I think most of you would agree with me if I say that the main goal for
> D2 right now should be to build a vibrant library ecosystem around the
> language, to foster adoption in real-world applications. There has been
> a number of related discussions recently, but as mentioned above, I
> think there is a central question:
> 
> Have we reached the critical mass yet where it makes sense to split the
> effort in a number of smaller library projects, or are we off better
> with concentrating on a central, comprehensive standard library
> (Phobos), considering the current community size?
> 
> I do not really have an answer to this question, but here are a few
> thoughts on the topic, which might also help to make clearer what I mean:
> 
> I think that adopting a Boost-like review process for Phobos has
> certainly been a clever and valuable move, for more than one reason.
> First, together with the move to Git, it has helped to reinforce the
> point that D2 and Phobos are open to contributions from everyone, given
> that they meet certain quality standards. Second, it certainly boosts
> code quality of further standard library additions, which had been a
> problem for some parts in the past (at least from my point of view, no
> offense intended). Third, and this overlaps with another point below, I
> think that the quality improvements will also help to reduce bit rot,
> which has traditionally been a problem with D libraries.
> 
> But however good a fit this model is for the standard library, I think
> it is no silver bullet either. There are small, one-off style projects,
> arising from a central need, where the amount of time needed to get the
> code through the whole review process is prohibitive – even if the code
> quality was high enough –, but the result is still usable for the wide
> public. Common examples for this would be low-level wrappers for C
> libraries, although they don't really qualify for inclusion into Phobos
> for other reasons (often, another wrapper layer is needed to be usable
> with common D idioms). Also, people new to the language might be scared
> away by the mere thought of contributing to a standard library. How to
> make sure that these libraries are not forgotten? Maybe a central
> package system with SCM (Git, …) integration can help here?
> 
> And, which brings me to the next point, how to fight the unfavorable
> outcome of having a huge inscrutable pile of half-finished bit-rotten
> code, a problem that DSource is currently experiencing? A central,
> well-maintained standard library effort with a wider scope could
> certainly help to reduce this problem, at least from the (D) user side,
> but on the other hand, larger amounts of code de facto becoming
> unmaintained would be a problem for it as well.
> 
> Should we build something like a staging area, an incubator for
> community contributions not taken yet through formal review, but of
> interest for a wider audience? What about the etc.* package – would it
> be an option to expand it into such an incubation area? If not, what
> should it evolve into – a collection of C-level library bindings (see
> the recent discussion on SQLite bindings started by David Simcha)? Who
> will take care of the maintenance duties?
> 
> Looking forward to a stimulating discussion,
> David

There has been some discussion in the past of creating an incubator project of 
sorts where code which may or may not make it into Phobos can be put so that it 
can develop and evolve with people actually using it - maybe even using it 
heavily - before it tried to get into Phobos. Once a library was considered 
mature enough, it could go through the Phobos review process and attempt to get 
into the standard library. If it succeeded, then, in theory, we'd have a well-
used and well-tested library added to Phobos. If it failed, it would still be 
around for people to use, and it could continue to be used and evolve - either 
to make a later attempt at inclusion in Phobos or to simply live as a 3rd party 
library that folks find useful but isn't appropriate for inclusion in the 
standard library for one reason or another.

Exactly what the best way to do all this would be, I don't know, but it would 
need to be far better manage than the current situation with dsource. The 
current activity level of a project would need to clear, as well as how complete 
and stable it's considered. DSource is probably the best place to host such a 
project, but obviously dsource would have to be cleaned up first, and it would 
need someone who was willing to spearhead the effort and manage it. Otherwise, 
we'll just end up with a situation similar to what dsource is now.

Really, the problem is that someone needs to take the initiative on this. They 
need to work on setting it up and supporting the ecosystem which would result in 
a group of such projects. Good ideas tend to be presented around here and then 
go nowhere, because no one actually takes the initiative to do them. The 
"wouldn't this be a good idea?" tactic doesn't tend to get very far, even if 
everyone agrees, simply because someone has to put in the time and effort to do 
it, and while people may think that it's a good idea, there are only so many 
people working on Phobos and other D-related stuff, and there's a lot to be done, 
and everyone has something that they'd like to see done, and _that_ is what 
they're generally working on.

So, I definitely think that an incubation project is a great idea. It has been 
proposed before. However, unless someone steps up to the plate and takes on 
setting it up and organizing it, it's not going to happen.

- Jonathan M Davis