auto testing (was: Bug Prediction at Google)
Brad Roberts
braddr at puremagic.com
Thu Dec 15 17:43:37 PST 2011
On Thu, 15 Dec 2011, Andrew Wiley wrote:
> On Thu, Dec 15, 2011 at 6:04 PM, Robert Clipsham
> <robert at octarineparrot.com> wrote:
> > I really think github needs built in review tools (more advanced than just
> > pull requests) to allow things like the auto-tester to be run, or algorithms
> > like this to be used for manual review and so on.
>
> Well, Github does have an API to allow that sort of thing to happen.
> In theory, a bot could examine a pull request, merge it and run tests,
> and post back the results.
I'm 90% done with adding pull testing to the existing auto-tester fleet.
It's based on the work that Daniel did.
The basic overview:
server:
every 10 minutes, check github for changes to pull requests
(if they had notifications for pull changes, I'd use it instead)
every time a trunk commit notification is received, check github for
changes to pull requests
client:
forever {
check for trunk build
if yes { build trunk; continue }
check for pull build
if yes { build pull; continue }
sleep 1m
}
Left to do:
1) deploy changes to the tester hosts (it's on 2 already)
2) finish the ui
3) trigger pull rebuilds when trunk is updated
4) add back in support for related pull requests (ie, two pulls that
separately fail but together succeed)
5) consider adding updating the pull request on github with tester
results. This one needs to be very carefully done to not spam the
report every time a build fails or succeeds.
6) update the auto-tester grease monkey script to integrate the pull
tester results with github's ui.
I'll hopefully finish 1 and 2 tonight. I can do 3 manually until it's
automated. I'm not sure about the ordering of 4-6. They're nice to haves
rather than must haves.
All these extra builds are going to cost a lot of time. There's about 100
open pull requests right now. The fastest runs are on the order of 10
minutes. That's 6 per hour or roughly 17 hours. The slowest are closer
to an hour. So, obviously there's some growing pains to deal with. I'll
probably add a way for github committers to prioritize pull requests so
they build first.
Luckily this stuff is trivial to throw hardware at.. it's super
parallelizeable. Also the hardware I have for those long runs is super
old. I think the freebsd/32 box is p4 era box. The win/32 is my asus eee
atom based netbook.
If anyone wants to volunteer build hardware, particularly for the
non-linux platforms, please contact me via email (let's not clutter up the
newsgroup with that chatter).
Requirements:
I need to be able to access it remotely.
It needs to be have reliable connectivity (bandwidth is pretty much a
non-issue.. it doesn't need much at all).
It needs to be hardware you're willing to have hammered fairly hard at
random times.
I'll almost certainly write some code during the holiday to fire up and
use ec2 nodes for windows and linux builds. With the application of just
a little money, all of those runs could be done fully parallel.. which
would just be sweet to see. Ok, I admit it.. I love working at amazon on
ec2.. and I'm happy to finally have a project that could actually use it.
Later,
Brad
More information about the Digitalmars-d
mailing list