Quo vadis, D2? Thoughts on the D library ecosystem.

[David Nadlinger]

While lying in the bed with fever yesterday (so please excuse any 
careless mistakes), I was pondering a bit about the current discussions 
regarding Phobos additions, package management, etc. It occurred to me 
that there is a central unanswered question, which I think deserves to 
be broadly discussed right now.

But first, let me start out by describing how I see current situation 
regarding D2. Leaving aside a few minor things like @property 
enforcement or the recent suggestions about a new alias syntax, the 
language is fairly stable and critical bugs in DMD 2 are not frequent 
enough to make it completely unusable for day-to-day development 
anymore. Of course, there is still a large way to go for the D toolchain 
(with the ideal result being a rock-solid self-hosting compiler 
front-end, usable as a library as well), but in a sense, we are more or 
less at the end of a certain stage of D2 development.

I think most of you would agree with me if I say that the main goal for 
D2 right now should be to build a vibrant library ecosystem around the 
language, to foster adoption in real-world applications. There has been 
a number of related discussions recently, but as mentioned above, I 
think there is a central question:

Have we reached the critical mass yet where it makes sense to split the 
effort in a number of smaller library projects, or are we off better 
with concentrating on a central, comprehensive standard library 
(Phobos), considering the current community size?

I do not really have an answer to this question, but here are a few 
thoughts on the topic, which might also help to make clearer what I mean:

I think that adopting a Boost-like review process for Phobos has 
certainly been a clever and valuable move, for more than one reason. 
First, together with the move to Git, it has helped to reinforce the 
point that D2 and Phobos are open to contributions from everyone, given 
that they meet certain quality standards. Second, it certainly boosts 
code quality of further standard library additions, which had been a 
problem for some parts in the past (at least from my point of view, no 
offense intended). Third, and this overlaps with another point below, I 
think that the quality improvements will also help to reduce bit rot, 
which has traditionally been a problem with D libraries.

But however good a fit this model is for the standard library, I think 
it is no silver bullet either. There are small, one-off style projects, 
arising from a central need, where the amount of time needed to get the 
code through the whole review process is prohibitive – even if the code 
quality was high enough –, but the result is still usable for the wide 
public. Common examples for this would be low-level wrappers for C 
libraries, although they don't really qualify for inclusion into Phobos 
for other reasons (often, another wrapper layer is needed to be usable 
with common D idioms). Also, people new to the language might be scared 
away by the mere thought of contributing to a standard library. How to 
make sure that these libraries are not forgotten? Maybe a central 
package system with SCM (Git, …) integration can help here?

And, which brings me to the next point, how to fight the unfavorable 
outcome of having a huge inscrutable pile of half-finished bit-rotten 
code, a problem that DSource is currently experiencing? A central, 
well-maintained standard library effort with a wider scope could 
certainly help to reduce this problem, at least from the (D) user side, 
but on the other hand, larger amounts of code de facto becoming 
unmaintained would be a problem for it as well.

Should we build something like a staging area, an incubator for 
community contributions not taken yet through formal review, but of 
interest for a wider audience? What about the etc.* package – would it 
be an option to expand it into such an incubation area? If not, what 
should it evolve into – a collection of C-level library bindings (see 
the recent discussion on SQLite bindings started by David Simcha)? Who 
will take care of the maintenance duties?

Looking forward to a stimulating discussion,
David