auto testing

[Brad Roberts]

On 12/16/2011 1:29 PM, Brad Anderson wrote:
> On Thu, Dec 15, 2011 at 6:43 PM, Brad Roberts <braddr at puremagic.com <mailto:braddr at puremagic.com>> wrote:
> 
> 
>     Left to do:
> 
>      1) deploy changes to the tester hosts (it's on 2 already)

done

>      2) finish the ui

very ugly but minimally functional: http://d.puremagic.com/test-results/pulls.ghtml

>      3) trigger pull rebuilds when trunk is updated

partly implemented, but not being done yet

>      4) add back in support for related pull requests (ie, two pulls that
>         separately fail but together succeed)
> 
>      5) consider adding updating the pull request on github with tester
>         results.  This one needs to be very carefully done to not spam the
>         report every time a build fails or succeeds.
> 
>      6) update the auto-tester grease monkey script to integrate the pull
>         tester results with github's ui.
> 
>     I'll hopefully finish 1 and 2 tonight.  I can do 3 manually until it's
>     automated.  I'm not sure about the ordering of 4-6.  They're nice to haves
>     rather than must haves.
> 
>     All these extra builds are going to cost a lot of time.  There's about 100
>     open pull requests right now.  The fastest runs are on the order of 10
>     minutes.  That's 6 per hour or roughly 17 hours.  The slowest are closer
>     to an hour.  So, obviously there's some growing pains to deal with.  I'll
>     probably add a way for github committers to prioritize pull requests so
>     they build first.
> 
>     Luckily this stuff is trivial to throw hardware at.. it's super
>     parallelizeable.  Also the hardware I have for those long runs is super
>     old.  I think the freebsd/32 box is p4 era box.  The win/32 is my asus eee
>     atom based netbook.
> 
>     If anyone wants to volunteer build hardware, particularly for the
>     non-linux platforms, please contact me via email (let's not clutter up the
>     newsgroup with that chatter).
> 
>     Requirements:
>      I need to be able to access it remotely.
> 
>      It needs to be have reliable connectivity (bandwidth is pretty much a
>      non-issue.. it doesn't need much at all).
> 
>      It needs to be hardware you're willing to have hammered fairly hard at
>      random times.
> 
>     I'll almost certainly write some code during the holiday to fire up and
>     use ec2 nodes for windows and linux builds.  With the application of just
>     a little money, all of those runs could be done fully parallel.. which
>     would just be sweet to see.  Ok, I admit it.. I love working at amazon on
>     ec2.. and I'm happy to finally have a project that could actually use it.
> 
>     Later,
>     Brad
> 
> 
>> With the application of just a little money
> 
> Out of curiosity, how much is a little?

I'll need to experiment.  It's the kind of thing that the more money thrown at the problem the faster the builds could
be churned through.  The limit of that would be one box per platform per pull request.  A silly (and not possible due to
ec2 not supporting some platforms, such as osx) extreme though.  One c1.medium running 24x7 at the current spot price is
about $29/month.  Spot is perfect because this is a totally interruptable / resumable process.  I'll need to do a little
work to make it resume an in-flight test run, but it's not hard at all to do.

After watching the build's slowly churn through the first round of build requests, it's clear to me that one of the
areas I'll need to invest a good bit more time is in scheduling of pulls.  Right now it's pseudo-random (an artifact of
how the data is stored in a hash table).

Later,
Brad