[dmd-internals] Jira?

[Andrei Alexandrescu via dmd-internals]

We all share a good bit of your frustration. But it doesn't have us 
leave. The simple point is if you want to improve the professionalism of 
the D team the one way to go about it is to contribute professional 
expertise to it. Long emails aren't it.

We have a vision document at http://wiki.dlang.org/Vision/2015H1, which 
is generally agreed upon among forum participants. We're due to update 
it soon for the second half of this year. Some of these things are 
universally agreed upon and easy to address incrementally, such as 
reducing the use of GC in Phobos. Much of the work is trivial. Yet 
nobody does it, for which reason Walter has had to do it (see his 
trailing pull requests to phobos). My view on this is we simply don't 
have enough people with time to work on things they themselves agree are 
important.

So it seems to me that the community not agreeing with what we think is 
important etc. is simply false. You seem to be asserting it, ignore 
evidence to the contrary, and conclude that if we disagree with you we 
can't or don't want to understand. Your position is always right; is not 
falsifiable so it's not scientific. Frankly it's sensible to accept you 
may as well accept you may be wrong once in a blue moon, and from where 
I stand your assertions about what's going on are wrong right, left and 
center. What you say is happening is simply not what's happening. The 
one way to figure is more participation; drive-by philosophy won't cut it.

Framing your failure to add unittests to dmd as Walter & Comp. being a 
bunch of boneheads is simple and convenient. Another way to frame it was 
that you could not demonstrate that your view and approach in the matter 
was worth it. (Again, we do have a history of folks who come, dump a 
bunch of stuff, and leave letting the few of us pay the piper.)

We plan to add named unittests. There is a library proposal at 
https://github.com/D-Programming-Language/phobos/pull/3207. I think we 
should actually make names part of the language, so they appear in 
stacktraces etc. They have generated names anyway, it makes sense to 
allow users to customize that name.


Andrei

On 5/31/15 1:17 AM, David B. Held wrote:
> When I see what the current fires are, I see them switching from C++
> support to ranges to DLLs to allocators to GC and back and forth. I'm
> never sure why the thing that you and Walter say is the most important
> really is the most important, and I see the community questioning this
> as well. If anything, it is quite clear that there are as many agendas
> as contributors.
>
> What I don't see anywhere is something as simple as a product backlog.
> This doesn't even need fancy software to maintain. It could be a wiki
> page. But the point of it is to make clear how all the agendas interact,
> intersect, and conflict. Obviously, this can become a highly political
> issue, which is likely why it hasn't been attempted.
>
> It should also expose how the priorities shift over time, as well as the
> extent to which the community has influence over what is considered
> important. If there's one bit of process that is almost universally
> valuable, it is transparency. I see D development as more like black
> art. Shining a light on everyone's priorities would lend credibility to
> the process and likely make people more willing to compromise on them.
> What I sense is instead a community that sometimes resents having
> priorities dictated to them.
>
> D has real users, and by that I mean people who are betting their money
> and their jobs on it. The hobbyists are real users too, but don't have
> quite so much skin in the game. These people deserve more than black
> magic when it comes to setting priorities. If I'm wrong and a backlog or
> equivalent does exist and is easily discoverable, and contains rationale
> for the ordering, then just point me to it so I can understand the
> process better.
>
> I'm not married to Jira. If folks prefer Trello, I'm fine with that. But
> the odd thing about the D community is the excitement over the
> possibilities mixed with the frustration over dysfunction. I don't see
> why trying to bring some clarity to the process is a bad thing.
>
> Now, let's talk about why I gave up on contributing to D. To be
> perfectly honest, it's because D development lacks a certain
> professional maturity. I wanted to add unit tests to the compiler (dmd).
> You and Walter said that was a waste of time. The important tests are
> higher level (in some unspecified way). I find it highly ironic that the
> D language added inline unit tests because they are so important all
> code needs them, but the D compiler itself is somehow immune from bugs.
> Frankly, this is bullshit. I can hardly believe a professional software
> engineer could make such a claim, let alone one who has written several
> acclaimed books.
>
> Ok, so I wrote some tests anyway. Everyone complained. Literally every
> comment on the tests were a complaint. No "hey unit tests in dmd...it's
> about time!" Just complaints about the testing library. Complaints about
> the layout. Complaints about the formatting. For all the singing of
> praises of the unittest keyword, the dmd maintainers were almost
> universally hostile to actual unit tests in the compiler. This still
> blows my mind. This is so far from professional software development
> that it's not even funny.
>
> I wanted to add names to unit test blocks. "Why?" Why? Why?!? Have you
> ever RUN unit test before? How about in an IDE? In a build farm? In a
> continuous integration pipeline? The idea that unit tests should have
> names is so blindingly obvious to anyone who has used Junit for 5
> minutes that the best answer I can give is literally "watch my tests run
> in junit for a minute." I could give excruciating detail on why unit
> test names are essential, but really, watching unit tests run in a
> professional environment is far more educational. Having to justify
> something this basic exceeded my patience, and my hope.
>
> I assumed that the dmd community was a bunch of hobbyists who were
> professional coders by day and who would mostly see eye to eye on a lot
> of these basic issues. I still don't know why I was so wrong about this,
> but I've talked to enough people that I believe that most of them are,
> in fact, competent professional software engineers. That's why I like
> the conferences.
>
> Unfortunately, I think much of the friction in the community is due to
> mixing professionals with hobbyists and academics. Each has a very
> different world view. This often drives the divergent priorities, or so
> I speculate. But it also leads to fundamental disagreements and
> misunderstandings which lead to folks talking past and over each other.
>
> Because software engineers have far more confidence than they need to
> get the job done, nobody wants to admit that anyone else is right. It's
> pathological Microsoft politics played out tragically in an OSS
> community. What the community needs is less talking down from on high
> and more listening from the bottom up. I'm actually a little surprised
> that the community hasn't completely imploded already. I think it is a
> testament to how much folks believe in the product, even if they have
> doubts about how it is produced.
>
> People not following orders is called "disagreeing with the dictats in a
> passive-aggressive manner." It's a strong signal that the leadership
> does not have buy-in from the led. It is, unfortunately, a vote of no
> confidence.
>
> I was going to offer to maintain Jira in parallel with BugZilla, but I
> see that the opposition is to the idea of a backlog with history in the
> first place. This is a bad sign. It means that transparency itself is
> what is contentious. I can't fix that. So sad.
>
> Dave
>
>
> -------- Original message --------
> From: Andrei Alexandrescu <andrei at erdani.com>
> Date: 05/30/2015 10:16 PM (GMT-08:00)
> To: David Held <dmd at wyntrmute.com>, Discuss the internals of DMD
> <dmd-internals at puremagic.com>
> Subject: Re: [dmd-internals] Jira?
>
> I'll be curious what Martin thinks of this.
>
> My thoughts - as one who's thought of this for years and tried various
> angles - are generally in disagreement with yours, though not vehemently so.
>
> * This looks like a random change of a tool that works, for benefits
> that seem small and an overhaul that seems large.
>
> * We are trying trello for management. This is the second attempt to
> introduce it, by Martin; first, by me, was a failure. Sadly this one
> also doesn't seem to fare very well.
>
> * Don't take this personally, but we've had a pattern of strong
> up-and-coming folks without a track record and proposing a large change
> with "trust me it'll be better" written all over it. Invariably those
> have been flashes in the pan. If I remember correctly, one of such
> initiatives was from yourself. Again, don't take this personally but it
> is my opinion that if you or anyone wants to impact the community, the
> one way is to consistently get work done on issues that matter.
>
> * I believed D has had a management problem for a good while, until I
> figured it doesn't. We are in need of talent. I understand how being at
> DConf - it's a good conference and those are inspiring - instills one
> with the impression that there's just a lot of folks ready to work but
> are not organized properly. Monitoring the forums may give a similar
> impression. The reality in the trenches is we have problems that are
> very well known, often for years, small and large - often unbelievably
> trivial - and there are very few people out there to work on them. Those
> who do, do make progress (e.g. look at Martin's work on the GC, Unique,
> RefCounted, etc., or Walter's work on weaning Phobos off GC). Walter and
> I have been trying to lead both by example and by simply telling what
> needs to be done, several time, with virtually no success.
>
>
> Andrei
>
> On 5/30/15 6:01 PM, David Held via dmd-internals wrote:
>  > Although BugZilla has served well for many years, I would like to
>  > propose a move to Jira.  I see the following benefits:
>  >
>  > * BugZilla is a bug tracking system; Jira is bug tracking + project
>  > management
>  > * Jira has a more mature UI, due to commercial licensing paying for
>  > development
>  > * Jira is free for OSS!
>  >
>  > So why does D need project management?  I would suggest that work on D
>  > is somewhat disorganized and haphazard.  That is partly just the nature
>  > of OSS/volunteer work, but also partly due to a lack of organizational
>  > structure.  Although language changes are neatly encapsulated by D
>  > Improvement Proposals, the vast majority of work which would benefit D
>  > falls far short of a proposed language change, and is not neatly and
>  > clearly tracked.  From what I can tell, this is where someone would look
>  > to find out where to make a contribution:
>  >
>  > http://wiki.dlang.org/How_You_Can_Help
>  >
>  > This is better than nothing, but just barely.  With a proper project
>  > management tool, we can capture issues which arise on the newsgroups,
>  > informal discussions, conferences, IRC, etc., along with associated
>  > communication and any artifacts produced.  We can order work by
>  > importance and track progress.  We can set goals and gain visibility
>  > into how and where contributions are made.
>  >
>  > None of this means that the current D community workflow needs to
>  > change, other than using Jira instead of BugTracker.  While this does
>  > incur a user cost to switch, I think most people will find that Jira is
>  > generally easy to use and relatively painless to pick up. They should
>  > also find it much easier to run reports and view dashboards.
>  >
>  > If the community agrees to make the switch, I will volunteer to apply
>  > for the Jira license and work with the current BugZilla maintainer(s) to
>  > migrate the DB (Jira has a tool for BugZilla to do this).  I will also
>  > volunteer to capture information from the newsgroups into tasks and
>  > other relevant project-tracking data.
>  >
>  > Although it is possible to track non-bug issues in BugZilla, this is not
>  > the kind of use case for which BugZilla was designed.  Since Atlassian
>  > is making their tools available to us for free, I think we should take
>  > advantage of this offer.  For the record, I think we should stick with
>  > MediaWiki rather than move to Confluence.  And I do not know of anyone
>  > who thinks that Stash is better than GitHub.
>  >
>  > Dave
>  >
>  >
>  > P.S. For those less familiar with the Agile methodology, what I propose
>  > to do is to create a master backlog of work.  The community can
>  > collaborate to define the ordering of the backlog, but the backlog would
>  > represent the priority of work, such that someone who wanted to
>  > contribute would know that the top item on the backlog is considered the
>  > most important task to work on next, and they could just scan from the
>  > top down until they find something they think they can work on.
>  >
>  > We can debate whether there should be a backlog per work type (website
>  > maintenance, compiler maintenance, documentation, etc.) or a global
>  > backlog, but these things only make sense to discuss when we have a tool
>  > that makes maintaining a backlog easy to do.  The backlog can also
>  > capture work which was only partially done.  So someone might pick up a
>  > task, work on it for a while, and then update the work item to explain
>  > how far they got, even if they couldn't close it out.
>  >
>  > The project tracking system can help produce reports on task progress,
>  > bug vs. non-bug work rate, etc.  The idea is that by having easy access
>  > to this reporting information, the community can have greater visibility
>  > into the overall workings and accomplishments of the "D machine" and see
>  > the areas which are deemed important but neglected in a quantified way.
>  > Ideally, this would result in self-directed behavior which would
>  > optimize towards the high-value work rather than just the easy work.
>  > Even if it only does so in, say, 20% of the contributions, this would
>  > still be a significant improvement in return-on-contribution-invested.
>  >
>  > Furthermore, contributors could voluntarily report on status, which
>  > would give a sense of what people were actively working on, and help
>  > answer the question: "Why isn't issue X being addressed?"  Folks could
>  > just look at the capacity dashboard and say: "Oh, I see people are
>  > working on Y and Z, which is much higher in the backlog."  Then they
>  > could either lobby to have the backlog reordered, or they could admit
>  > that the highest value-add was being worked on, and pitch in to the
>  > tasks that they deem important.
>  >
>  > Of course, I have done a lot of hand-waving about "community-ordered
>  > backlog".  In reality, I expect this to be a major point of contention
>  > and source of more than a few flame wars.  But having a single place to
>  > witness the debate over priority unfold should be valuable all by
>  > itself.  As long as justifications are provided as to why the backlog is
>  > ordered the way it is, I think the community can come together and agree
>  > that it is reasonable, even if it doesn't fit every person's exact
>  > preferences (nor could such an ordering be produced).  And, of course,
>  > the backlog doesn't determine what anyone works on.  It just represents
>  > what the community thinks is important, in rank order.  Anyone is still
>  > free to pick something from the bottom and work on it.  However, finding
>  > that a lot of people are working on low-ranked tasks should communicate
>  > something important about the community in and of itself.  That's why
>  > these kinds of tools are important.
>  >
>  > A unified backlog should also make it easier for people to discover
>  > overlapping and/or duplicated work.  This probably doesn't happen often,
>  > but may save a lot of time and effort for folks when it does.  At the
>  > least it should make people aware of each other's work so they can
>  > proceed deliberately rather than in ignorance.  These kinds of things
>  > are done informally now via the newsgroups, but are a kind of waste of
>  > contributor cycles, because humans do not need to be the web servers for
>  > this kind of mundane information.  By offloading these overhead tasks to
>  > the proper software, the forums can be optimized for discussions which
>  > cannot be automated.  That should make everyone happier and more
> efficient.
>  >
>  > _______________________________________________
>  > dmd-internals mailing list
>  > dmd-internals at puremagic.com
>  > http://lists.puremagic.com/mailman/listinfo/dmd-internals
>