Start of dmd 2.064 beta program

Andrei Alexandrescu SeeWebsiteForEmail at erdani.org
Fri Oct 18 11:45:27 PDT 2013


This is a good list of things that we could and should improve. Getting 
all minded to perfection would be difficult, but definitely worth living 
into.

I appreciate your enthusiasm for and involvement in D. At the top level 
a few simple realities need to be understood.

First, there is no OSS community that doesn't have its annoyances. I've 
been a direct participant to one other and am lurking on the forums of 
three more. Some are led by juntas of mini-tyrants; some are 
unnecessarily snooty; some are consumed by intestine fighting, politics, 
and horse-trading; and so on.

Second, some of your points revolve around the fact that Walter and 
myself are not as good as we should at certain tasks and roles, such as 
build master, PR person, manager, and operational officer. The general 
argument pattern revolves around "I/we told Walter/Andrei several times 
they need to do X, and they forget/ignore/do it badly." Clearly Walter 
is a self-confessed mediocre build czar at best. But he's doing it 
because, although there is no shortage of suggestions on how he should 
be doing it, nobody has the time and inclination to actually _be_ the 
build czar (which is entirely understandable).

Within a traditional organization where people are paid to work on a 
certain project, the solution would be simple and obvious - Walter would 
be relieved of the roles he doesn't excel in, and left to focus on those 
he's really good at. Other people would fill in duties and 
responsibilities such as preparing betas, sending them out for testing, 
announcing them, etc.

Within our current organizational landscape, where nobody is paid to 
work on D (not _with_ D) except for myself, addressing issues such as 
"we should have a better build master" is much more difficult to address.

Third, for what it's worth, here's what I believe are the top issues 
with D today:

1. Nobody is being paid to work on D (aside from me since recently, and 
on work that would benefit my employer).

2. D is not backed up solidly by a large engineering organization.

3. We are unable to review and accept contributions at the rate they are 
arriving.

I tend to ask myself how various proposals for improving our process 
address these three key points. I believe your list effects mostly (3), 
albeit not directly.

More answers inline.

 > - We do not have any vision or major plans ahead for the language.
> Currently we're stuck in a bug-driven development environment, where
> bugs are arbitrarily picked off of bugzilla and fixed. But there's no
> major plans ahead, e.g. "let's plan to fix these X major bugs for some
> upcoming release". We can't force people to work on X or Y, but if
> they're in a visual place and marked important and scheduled to be
> fixed, this will give an incentive for contributors to work on these
> problems.

Walter and I frequently think of ways to gently steer people toward 
working on specific items. Votes, categorization, discussions are of 
limited impact. On occasion we've emailed major contributors directly 
and asked politely but point blank if they could look into an issue 
(sometimes something they had an expertise in, or even an issue they had 
caused). The rate of success has been very low. People still love 
working on things, just not on the things they're told to.

We've added the "preapproved" tag - take a look: 
http://d.puremagic.com/issues/buglist.cgi?quicksearch=preapproved. A 
couple have opened pull requests, which have not yet received reviews 
yet (which is not my or Walter's fault as much as a larger community 
issue that we need to fix, see (3) above). Most don't. You yourself 
didn't find the time to update a task you volunteered on.

That said, it's entirely possible a formula for success does exist. 
Looking forward to proposals, like improving the visuals of "bugs to be 
fixed". What I think is obvious is that solution involving more work for 
Walter and myself are unlikely to work well, for the simple reason we 
are both impatiently waiting for the sun to rise every day to do more 
work on D.

> - We do not have any defined release timeline. When is it time to
> start prepping for a release? It's up to Walter's arbitrary decision
> for when this happens, he doesn't even talk to the community or
> contributors on whether it's a good time for a beta phase (maybe
> there's a huge or disruptive new pull request that's planning to be
> merged and a beta should be delayed).

I understand how that can be a bother. Walter figures the time is ripe 
for a new release when we have enough compelling features and fixes. 
I'll try to make the appropriate announcements in the future prior to 
deciding on starting a beta.

> - We do not have a defined timeline for the beta testing period. How
> long before we decide that the beta has been tested for long enough
> before a release is made? Again, it's up to Walter's decision.

We are generally aiming for two weeks, but it sometimes gets longer 
because of the impending regressions. One good phenomenon has been that 
we've got an increasing number of people who are testing the beta.

> Having a defined release cycle and a beta testing period will
> ultimately be beneficial for everyone. People who are busy can
> rearrange their schedule to make free time and ensure they have enough
> time to test a beta compiler on their projects, and contributors to D
> can prepare for a cycle of days or weeks where they can rapidly work
> on reducing regressions and polish everything up for the release (e.g.
> writing an elaborate changelog).

I also believe a better cycle would be beneficial, but I don't think it 
would address our core issues.

> - We do not have a proper release system. Nick Sabalausky has been
> working hard on one[1], but Walter seems to have completely ignored
> it. It would have been a terrific opportunity to try and make it work
> for this release. What better way to test it than to test it on a
> release?

We'll look into it, but this harkens back to the simple dynamics 
mentioned above. That is essentially a request for Walter to change the 
way he does releases, and people tend to get set in their ways and have 
difficulty finding the time to stop and change things when there are so 
many other things vying for their attention. The best solution here is 
if Nick (or someone else) would _be_ the build manager using those great 
tools that Nick wrote.

Anyhow, I'll look into that.

> - The betas are still not visible. The homepage makes no notes on a
> new beta being available, the download page does not list the betas
> either, and AFAICT there's no RSS feed either. The social media groups
> are not notified either (neither Andrei's nor Walter's twitter feed
> make a mention of a new beta being out, the same applies to
> https://twitter.com/D_Programming or the #dlang hash tags). To top it
> all off, you cannot post to the dmd-beta newsgroups from the D Forums,
> you have to separately register to this mailing list.

As a simple start, did you consider announcing the beta yourself? Anyone 
can tweet #dlang @D_programming, and in all likelihood we and many 
others would retweet. Also, did you consider submitting a pull request 
for placing the beta announcement on the homepage?

> If we want user feedback on betas we absolutely must make the betas
> visible and give an opportunity for people to post feedback.

I agree. "We" is the right word.

> - Walter is still not tagging the beta releases by the file name (it's
> always dmd2beta.zip). I've complained about this several times and
> IIRC someone else did as well at dconf (maybe I'm remembering wrong
> though). They should at least be named as "dmd2_064_beta1.zip",
> "dmd2_064_beta2.zip". And all of them should always be available for
> download (including visibility on the download page), so people who do
> not use Git or build manually from master can quicky check whether a
> regression was introduced in a specific beta version.

Probably that's a good thing to do, and easy enough. I don't think it 
would push the needle significantly.

> - I still sigh when I hear about Walter and Andrei having private
> phone conversations, or any kind of decision is made behind-the-scenes
> rather than publicly where the community has a say in it. Walter's
> implementation of UDAs directly in master which lead to having a
> deprecated syntax even though nobody used this syntax is what comes to
> mind.

I understand this is well intended but it's the kind of remark that 
bends me out of shape. First, there's no substitute for real 
communication than direct conversation. We can't have a phone 
conversation with the community. Then, your first three points complain 
about a lack of leadership, and in the same breath you complain about 
there being too much of it.

We believe we've done well in making this community a meritocracy where 
competent contributors can make a difference and make their word heard. 
To the extent we're doing it insufficiently, it's because too few people 
assume the responsibility to review and accept contributions.

Walter scrambled to implement UDAs in a rush and breaking protocol in 
order to win a corporate D user, Remedy Games. It was a major, 
exceptional event. Would you have preferred the protocol to have been 
followed at the cost of Remedy?

> - Both Walter and Andrei not following the rules and making novice
> mistakes w.r.t Git and Github. Walter still seems to struggle with
> basic usage of git, where people continually have to re-explain what's
> wrong and how to fix an issue. I'm sorry, but if someone bought the
> Git book years ago and is still struggling with *concepts* then no
> amount of hand-guiding is going to help.

I agree that's a problem. Probably what would help would be more quality 
reviews and pulls by our members with commit rights, of whom you are one.

> And Andrei doing things like
> merging a dozen pull requests at once, with complete disregard to the
> fact that merging to master means other pulls could easily break (and
> so master can be broken). You cannot make so many merges unless you're
> absolutely sure each pull request does not interfere with another.

If a pull request is sensible and meaningful it shall be pulled. That's 
the way we do things at Facebook, too, and it scales. There may be pulls 
that are so closely related they break the build when put in 
conjunction, but I believe that case is rare.

I also want to convey a bit of perspective here. There are 221 open pull 
requests right now. These are 221 instances of good work put in by 
talented people, who associated hopes and wishes to improve things with 
that work. This bothers me. I lose sleep at night because of it. And the 
fact that those contributions don't get looked at is our entire 
community's fault as it is mine.

So when I have a few free minutes, I try to take a look at the extant 
pull requests - really, any would do. I try to pull in those that are 
meaningful and uncontroversial. I agree I could probably spend more time 
on carefully analyzing interactions between pulls, but that's time I 
can't afford.

> - Back to Walter, a few weeks ago he merged a pull request over night,
> without regard to the pull request not being fully tested by the test
> machines. The result? Master was broken **for the entire next day**.
> Nobody knew which commit broke master, so nothing was done until
> Walter came back to Github the next day and started reverting his
> pulls. In the meantime the entire day was wasted because nobody's pull
> request could get green. Luckily it was a sunday, so there weren't too
> many complaints. But I could have easily merged a few pulls that day
> (as it happens I like to do things on a sunday), and as a result we
> would have a smaller pull queue.

I guess such accidents are bound to happen to the best of us. A build 
czar with the appropriate skills would have swiftly undone the damage. 
But there is no build czar.

> There's just many things that are going plain wrong here, and a lot of
> promises are always made but ultimately never delivered (whether it's
> about breaking changes or an improved development process -- again
> think about scheduling bug fixes for the future, we could help people
> prepare for the breaking changes).

Whatever we don't get delivered, it's not for the lack of trying. 
Whatever we do, it doesn't come easily.

> Personally I find D to be a huge part of me right now (probably not
> the other way around, I'm a small part of this compared to the huge
> contributors), and I feel really bummed out when both Andrei and
> Walter take a casual stance when things are broken (whether it's the
> system itself or something specific like master being broken). I
> really think we have a great thing going here with the language, but
> everything else has to improve, and particularly the development and
> release process.
>
> So that's what I'm protesting about.

The individual points are salient, but I fail to grab the most 
significant bit. The logic doesn't follow.

We're in this together; to wit, for many of the things you're asking why 
they don't get done, you could be asked the same thing.

You did a great job on formatting the changelog. It has been an 
unqualified success, it was amazing marketing for D, and it has been 
thoroughly appreciated by everyone. It is great work that we need more 
of. Now you're not doing that work in protest against... not enough work 
getting done.


Andrei



More information about the Digitalmars-d-announce mailing list