What's up with pull request buildbots?

[Adam Wilson]

On Sun, 02 Jun 2013 22:27:12 -0700, Dylan Knutson <tcdknutson at gmail.com>  
wrote:

> Hello,
> I'm a bit confused as to how the DMD buildbot is supposed to work: it  
> seems like 50% of the time (ballparked from the first 15 or so pull  
> requests), the buildbot just doesn't report failures or successes. This  
> bugs (heh) me a little bit because there are tons of months old pull  
> requests just waiting in the pipeline, stuck at "Determining merge  
> status". I don't know if this is part of the review process or caused by  
> the yellow status, but for instance I've opened up a bug report a week  
> or so prior:
> http://d.puremagic.com/issues/show_bug.cgi?id=10113
> and a few days afterwards, a pull request was submitted:
> https://github.com/D-Programming-Language/dmd/pull/2080
> which, turns out, was more or less a dup of another pull request,  
> submitted 6 months ago:
> https://github.com/D-Programming-Language/dmd/pull/1358

I'll see if I can't give you a stab at an explanation. Note that I did not  
write the Auto-Tester, that is the work of Brad Anderson. Here are some  
facts about the AT as I understand them:
- The AT works on a most current pull first basis, so a pull that is older  
than 6 months will naturally be scheduled much later.
- The AT schedules work based on how many testing machines are available,  
so each pull is tested only once per listed platform, and when multiple  
build servers for a given platform are available, different pulls can be  
tested simultaneously on that platform.
- Pull requests are tested by cloning [DMD/DRuntime/Phobos] master and  
merging the pull into it. This ensures a clean pull against master.
- The AT must restart pull request testing every time code is merged into  
DMD/DRuntime/Phobos because of the above testing strategy.
- If many pulls are merged into [DMD/DRuntime/Phobos] master in a given  
period of, for example a day, the AT will restart prior to completing a  
full test pass.
- With the current number of open pulls the AT can complete a full testing  
pass in roughly 8 hours.
- Each pull takes a rough average of 15 minutes to test.

The lack of success/failure reports on older pulls is directly due to the  
above testing strategy, essentially the AT never made it that far before  
someone merged a pull into [DMD/DRuntime/Phobos] master and forced a  
restart.

> However, neither of these pull requests have been merged. And neither of  
> the bugs have been fixed (my report was a dup of  
> http://d.puremagic.com/issues/show_bug.cgi?id=2950, submitted 4 years  
> ago!).
>
> So we're stuck with a buildbot system that leaves valuable pull requests  
> from ever getting merged, and duplicate fixes being written because  
> previous fixes were submitted and never accepted so long ago.

The AT system is intentionally incapable of merging to master, so it  
itself is not leaving pulls behind. However if a pull is out-of-date it is  
likely that it will fail the AT anyways due the codebase delta in the  
target project. This is why pulls are tested in a newest first order.

> On a somewhat related note: Why is the Puremagic bugtracker still used,  
> when Github has bug tracking with (IMO) better repo integration,  
> discussion, and searching (which I think we can all agree on), on a  
> repository *hosted at Github*? I can understand the reluctance to switch  
> over, but what's keeping us from accepting new issues on Github, closing  
> Puremagic down from accepting new requests, and working through the open  
> requests on Puremagic until all can be taken care of have been?

According to the Bug Stats page (http://dlang.org/bugstats.php) there are  
nearly 3000 open issues in the BugZilla and over 7000 closed issues! Now I  
tend to agree that better systems than BugZilla exist. But consider the  
effort required to move all of those bugs to a new system, automated  
tooling or no, it wouldn't be easy or quick. And consider that GitHub  
issues lacks much of meta information capability that BugZilla has, we  
would loose a fantastic amount of information. Yes of course GitHub has  
the best integration with it's own issue tracker, but we've achieved a  
fairly high level of integration between GitHub and BugZilla, so that has  
become almost a non-issue these days.

> Thanks for your time,
> Dylan Knutson

-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/