auto testing (was: Bug Prediction at Google)

Brad Anderson eco at gnuk.net
Fri Dec 16 13:29:02 PST 2011


On Thu, Dec 15, 2011 at 6:43 PM, Brad Roberts <braddr at puremagic.com> wrote:

> On Thu, 15 Dec 2011, Andrew Wiley wrote:
>
> > On Thu, Dec 15, 2011 at 6:04 PM, Robert Clipsham
> > <robert at octarineparrot.com> wrote:
> > > I really think github needs built in review tools (more advanced than
> just
> > > pull requests) to allow things like the auto-tester to be run, or
> algorithms
> > > like this to be used for manual review and so on.
> >
> > Well, Github does have an API to allow that sort of thing to happen.
> > In theory, a bot could examine a pull request, merge it and run tests,
> > and post back the results.
>
> I'm 90% done with adding pull testing to the existing auto-tester fleet.
> It's based on the work that Daniel did.
>
> The basic overview:
>
> server:
>
>  every 10 minutes, check github for changes to pull requests
>    (if they had notifications for pull changes, I'd use it instead)
>
>  every time a trunk commit notification is received, check github for
>    changes to pull requests
>
>
> client:
>
>  forever {
>    check for trunk build
>    if yes { build trunk; continue }
>    check for pull build
>    if yes { build pull; continue }
>    sleep 1m
>  }
>
>
> Left to do:
>
>  1) deploy changes to the tester hosts (it's on 2 already)
>
>  2) finish the ui
>
>  3) trigger pull rebuilds when trunk is updated
>
>  4) add back in support for related pull requests (ie, two pulls that
>     separately fail but together succeed)
>
>  5) consider adding updating the pull request on github with tester
>     results.  This one needs to be very carefully done to not spam the
>     report every time a build fails or succeeds.
>
>  6) update the auto-tester grease monkey script to integrate the pull
>     tester results with github's ui.
>
> I'll hopefully finish 1 and 2 tonight.  I can do 3 manually until it's
> automated.  I'm not sure about the ordering of 4-6.  They're nice to haves
> rather than must haves.
>
> All these extra builds are going to cost a lot of time.  There's about 100
> open pull requests right now.  The fastest runs are on the order of 10
> minutes.  That's 6 per hour or roughly 17 hours.  The slowest are closer
> to an hour.  So, obviously there's some growing pains to deal with.  I'll
> probably add a way for github committers to prioritize pull requests so
> they build first.
>
> Luckily this stuff is trivial to throw hardware at.. it's super
> parallelizeable.  Also the hardware I have for those long runs is super
> old.  I think the freebsd/32 box is p4 era box.  The win/32 is my asus eee
> atom based netbook.
>
> If anyone wants to volunteer build hardware, particularly for the
> non-linux platforms, please contact me via email (let's not clutter up the
> newsgroup with that chatter).
>
> Requirements:
>  I need to be able to access it remotely.
>
>  It needs to be have reliable connectivity (bandwidth is pretty much a
>  non-issue.. it doesn't need much at all).
>
>  It needs to be hardware you're willing to have hammered fairly hard at
>  random times.
>
> I'll almost certainly write some code during the holiday to fire up and
> use ec2 nodes for windows and linux builds.  With the application of just
> a little money, all of those runs could be done fully parallel.. which
> would just be sweet to see.  Ok, I admit it.. I love working at amazon on
> ec2.. and I'm happy to finally have a project that could actually use it.
>
> Later,
> Brad
>

> With the application of just a little money

Out of curiosity, how much is a little?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20111216/33113e5a/attachment.html>


More information about the Digitalmars-d mailing list