Patches, bottlenecks, OpenSource

[Robert Clipsham]

On 10/04/10 13:09, bearophile wrote:
> After the last posts about patches, I can write something myself
> about this topic :-) I am far from being an expert of big software
> projects, but I think I am now able to understand some things of the
>  D project.
>
> I like D, it's an alive project, but from what I've seen D so far is
>  not having a so significant success.

It seems this way to me too, I wonder what needs doing to change this...
Perhaps getting the word out more will help (I believe Walter and Andrei
are giving a 1 day talk at ACCU this week, I wonder if this will have an
impact at all). I also wonder if TDPL will make a difference, I guess
we'll see.

> I think lot of people (especially young programmers) don't want "a
> better C++", they want no stinking C++ at all. They want something
> quite different. (On the other hand now I am not so sure Scala will
> have a widespread success, its type system makes it not easy to
> learn).

Speaking myself as a young programmer (probably one of the youngest D 
users), I'm not sure where I stand on this. On one hand we have to learn 
from our mistakes... If C++ didn't happen D would be as much of a mess 
as C++ is :) On the other hand, if C++ went away I wouldn't feel so 
obliged to learn an obviously inferior language. I made the choice to 
learn D before C or C++, and as a result my motivation to learn them has 
been minimal...

> A C++-class language compiler is a big project that requires quite
> different skill sets: in the D compiler there is work to do on the
> vector operations, lent/uniqueness, multi-core, 64 bit, work for keep
> it well updated on Mac/Linux/Win and well packaged, development of
> Phobos and its data structures, work on the back-end to use the SSE
> registers and future CPUs or GPUs, tuning of the associative arrays,
> future improvements of the core language, improvements in the unit
> test system, large improvement in the D web site and the
> documentation, big improvements needed for the GC to turn it into
> something more modern, and so on. Such things have to be refined,
> debugged, efficient, cared of, polished.

That is, despite the size of the paragraph... A huge list of tasks :D I 
think a lot of effort can be saved on this front by, as you mention 
below, utilizing the likes of llvm, improving and tweaking the code base 
as needed. As for the other features, I guess it's a matter of time, 
some of these things will need to wait for D3 (lent/uniqueness), others 
just need people to put in the effort.

> So as D develops and grows it will get hard for a single person to
> write all patches to the language and other parts. As D grows this
> will become a bottleneck, and from what I've seen it already is. The
>  disadvantage of allowing other people the rights to patch the
> front-end is some lost control, but Walter can review patches after
> they are already applied&  working (in a big project having a
> director is positive). This can speed up the patching process itself.
> The compiler can become more like a patchwork of different
> programming styles, and Walter has to ask to other people how some
> parts not written by him work. This can be doable if the front-end
> becomes more modular. LLVM shows a more modular design that can be
> copied. This is a lot of refractoring work.

I think more and more people are noticing this recently, with about 70 
patches now in bugzilla that need filtering through and pushing into the 
code base. I don't really think a huge refactor is practical right now, 
the time is better spent working out the bugs, and generally polishing 
up the compiler (particularly the front end, which all the current 
working D compilers use). Eventually as D gains momentum this will 
become practical, and we could end up with a clang style front end for D 
(I believe this was started but didn't get too far, Dang on DSource I 
believe :)).

> Increasing openess can increase the influx of devs and their
> willingness to help the project. This is quite positive and its
> importance can't be underestimated.

The truth of this has already been visible, when the DM backend became 
source available a few months ago (but not open source), a huge 
selection of patches appeared, and as a result DMD improved. Moving to a 
more open development model for dmd, and D in general will always be a 
good thing, the evidence is all around us.

> Pyrex (http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/ )
> was forked in Cython (http://www.cython.org/ ) because Pyrex author
> was too much slow in patching. A fork will not happen in D soon
> because there are not enough people yet that care and work for D
> enough to make a fork viable. D is not a project as alive as Pyrex.
> Cython is a tiny rib of the Python language circus.

A side-effect of being completely open, eventually people will fork the 
project, and people have to choose what to use, causing a split in the 
community... Of course there have been many positive effects from forks 
of open source projects in the past, such as new features becoming 
available, and more stable code in general. The best things about forks 
is seeing the hard work paying off and seeing the forks merged back into 
the main codebase, so the advantages of each can be enjoyed by all :)

> I suggest to give patching rights to Don, it can be one more step
> forward toward Open Sourcing D development style :-) In future such
> right can be given to other few good people that have shown to
> deserve enough trust.

I have also suggested this, it would be good to see more people with 
commit access, even if it is just one person and they have a lot of 
restrictions with what can get in etc.

> I am not sure the future of D is in the dmd back-end, llvm looks in
> better shape for this purpose. LLVM is able to support 99.9% of the
> features of D2 (and I presume llvm devs are willing to add the few
> missing things if someone gives them patches; from the
> unladen-swallow project I have seen they are open enough), and it
> supports a large number of good features currently not done/present
> by/in the back-end of dmd. Currently ldc is a compiler much better
> than dmd (try it if you don't believe me!), but its devs are not
> appreciating D2 language yet much. I don't like to see the
> Walter-locomotive so detached from the present nice ldc D1 compiler,
>  and from the future of LDC and llvm. I'd still like ldc to become
> the official D2 compiler, developed and cared of :-)

I don't believe the future is in the DM backend either, I don't see it 
disappearing for a long time though. Walter's worked on that backend for 
a significant proportion of his life, I can see why he would be 
reluctant to give it up too easily. DM will continue to evolve, 
particularly as D2 becomes more stable and work on the 64bit backend 
starts (hopefully Walter will make sure this is done in such a way that 
the new 64 bit code is open source and not under the restrictive backend 
license, we'll see though :)).

I'd like to see more effort put into ldc to get it working with D2, I 
guess that unless the ldc developers find time for it, or other people 
do it won't happen for a good while yet unfortunately. I did try a while 
back, but realised what a huge task it was, and how little time I had 
available for it :/ Maybe someone else will be able to pick up where I 
left off :)

> Bye, bearophile