auto testing (was: Bug Prediction at Google)
Brad Anderson
eco at gnuk.net
Fri Dec 16 13:29:02 PST 2011
On Thu, Dec 15, 2011 at 6:43 PM, Brad Roberts <braddr at puremagic.com> wrote:
> On Thu, 15 Dec 2011, Andrew Wiley wrote:
>
> > On Thu, Dec 15, 2011 at 6:04 PM, Robert Clipsham
> > <robert at octarineparrot.com> wrote:
> > > I really think github needs built in review tools (more advanced than
> just
> > > pull requests) to allow things like the auto-tester to be run, or
> algorithms
> > > like this to be used for manual review and so on.
> >
> > Well, Github does have an API to allow that sort of thing to happen.
> > In theory, a bot could examine a pull request, merge it and run tests,
> > and post back the results.
>
> I'm 90% done with adding pull testing to the existing auto-tester fleet.
> It's based on the work that Daniel did.
>
> The basic overview:
>
> server:
>
> every 10 minutes, check github for changes to pull requests
> (if they had notifications for pull changes, I'd use it instead)
>
> every time a trunk commit notification is received, check github for
> changes to pull requests
>
>
> client:
>
> forever {
> check for trunk build
> if yes { build trunk; continue }
> check for pull build
> if yes { build pull; continue }
> sleep 1m
> }
>
>
> Left to do:
>
> 1) deploy changes to the tester hosts (it's on 2 already)
>
> 2) finish the ui
>
> 3) trigger pull rebuilds when trunk is updated
>
> 4) add back in support for related pull requests (ie, two pulls that
> separately fail but together succeed)
>
> 5) consider adding updating the pull request on github with tester
> results. This one needs to be very carefully done to not spam the
> report every time a build fails or succeeds.
>
> 6) update the auto-tester grease monkey script to integrate the pull
> tester results with github's ui.
>
> I'll hopefully finish 1 and 2 tonight. I can do 3 manually until it's
> automated. I'm not sure about the ordering of 4-6. They're nice to haves
> rather than must haves.
>
> All these extra builds are going to cost a lot of time. There's about 100
> open pull requests right now. The fastest runs are on the order of 10
> minutes. That's 6 per hour or roughly 17 hours. The slowest are closer
> to an hour. So, obviously there's some growing pains to deal with. I'll
> probably add a way for github committers to prioritize pull requests so
> they build first.
>
> Luckily this stuff is trivial to throw hardware at.. it's super
> parallelizeable. Also the hardware I have for those long runs is super
> old. I think the freebsd/32 box is p4 era box. The win/32 is my asus eee
> atom based netbook.
>
> If anyone wants to volunteer build hardware, particularly for the
> non-linux platforms, please contact me via email (let's not clutter up the
> newsgroup with that chatter).
>
> Requirements:
> I need to be able to access it remotely.
>
> It needs to be have reliable connectivity (bandwidth is pretty much a
> non-issue.. it doesn't need much at all).
>
> It needs to be hardware you're willing to have hammered fairly hard at
> random times.
>
> I'll almost certainly write some code during the holiday to fire up and
> use ec2 nodes for windows and linux builds. With the application of just
> a little money, all of those runs could be done fully parallel.. which
> would just be sweet to see. Ok, I admit it.. I love working at amazon on
> ec2.. and I'm happy to finally have a project that could actually use it.
>
> Later,
> Brad
>
> With the application of just a little money
Out of curiosity, how much is a little?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20111216/33113e5a/attachment.html>
More information about the Digitalmars-d
mailing list