auto testing (was: Bug Prediction at Google)

[Brad Roberts]

On Thu, 15 Dec 2011, Andrew Wiley wrote:

> On Thu, Dec 15, 2011 at 6:04 PM, Robert Clipsham
> <robert at octarineparrot.com> wrote:
> > I really think github needs built in review tools (more advanced than just
> > pull requests) to allow things like the auto-tester to be run, or algorithms
> > like this to be used for manual review and so on.
> 
> Well, Github does have an API to allow that sort of thing to happen.
> In theory, a bot could examine a pull request, merge it and run tests,
> and post back the results.

I'm 90% done with adding pull testing to the existing auto-tester fleet.  
It's based on the work that Daniel did.

The basic overview:

server:

  every 10 minutes, check github for changes to pull requests
    (if they had notifications for pull changes, I'd use it instead)

  every time a trunk commit notification is received, check github for
    changes to pull requests


client:

  forever {
    check for trunk build
    if yes { build trunk; continue }
    check for pull build
    if yes { build pull; continue }
    sleep 1m
  }


Left to do:

  1) deploy changes to the tester hosts (it's on 2 already)

  2) finish the ui

  3) trigger pull rebuilds when trunk is updated

  4) add back in support for related pull requests (ie, two pulls that
     separately fail but together succeed)

  5) consider adding updating the pull request on github with tester 
     results.  This one needs to be very carefully done to not spam the 
     report every time a build fails or succeeds.

  6) update the auto-tester grease monkey script to integrate the pull 
     tester results with github's ui.

I'll hopefully finish 1 and 2 tonight.  I can do 3 manually until it's 
automated.  I'm not sure about the ordering of 4-6.  They're nice to haves 
rather than must haves.

All these extra builds are going to cost a lot of time.  There's about 100 
open pull requests right now.  The fastest runs are on the order of 10 
minutes.  That's 6 per hour or roughly 17 hours.  The slowest are closer 
to an hour.  So, obviously there's some growing pains to deal with.  I'll 
probably add a way for github committers to prioritize pull requests so 
they build first.

Luckily this stuff is trivial to throw hardware at.. it's super 
parallelizeable.  Also the hardware I have for those long runs is super 
old.  I think the freebsd/32 box is p4 era box.  The win/32 is my asus eee 
atom based netbook.

If anyone wants to volunteer build hardware, particularly for the 
non-linux platforms, please contact me via email (let's not clutter up the 
newsgroup with that chatter).

Requirements:
  I need to be able to access it remotely.

  It needs to be have reliable connectivity (bandwidth is pretty much a 
  non-issue.. it doesn't need much at all).

  It needs to be hardware you're willing to have hammered fairly hard at 
  random times.

I'll almost certainly write some code during the holiday to fire up and 
use ec2 nodes for windows and linux builds.  With the application of just 
a little money, all of those runs could be done fully parallel.. which 
would just be sweet to see.  Ok, I admit it.. I love working at amazon on 
ec2.. and I'm happy to finally have a project that could actually use it.

Later,
Brad