auto testing

[Brad Anderson]

On Fri, Dec 16, 2011 at 11:40 PM, Brad Roberts <braddr at puremagic.com> wrote:

> On 12/16/2011 1:29 PM, Brad Anderson wrote:
> > On Thu, Dec 15, 2011 at 6:43 PM, Brad Roberts <braddr at puremagic.com<mailto:
> braddr at puremagic.com>> wrote:
> >
> >
> >     Left to do:
> >
> >      1) deploy changes to the tester hosts (it's on 2 already)
>
> done
>
> >      2) finish the ui
>
> very ugly but minimally functional:
> http://d.puremagic.com/test-results/pulls.ghtml
>
> >      3) trigger pull rebuilds when trunk is updated
>
> partly implemented, but not being done yet
>
> >      4) add back in support for related pull requests (ie, two pulls that
> >         separately fail but together succeed)
> >
> >      5) consider adding updating the pull request on github with tester
> >         results.  This one needs to be very carefully done to not spam
> the
> >         report every time a build fails or succeeds.
> >
> >      6) update the auto-tester grease monkey script to integrate the pull
> >         tester results with github's ui.
> >
> >     I'll hopefully finish 1 and 2 tonight.  I can do 3 manually until
> it's
> >     automated.  I'm not sure about the ordering of 4-6.  They're nice to
> haves
> >     rather than must haves.
> >
> >     All these extra builds are going to cost a lot of time.  There's
> about 100
> >     open pull requests right now.  The fastest runs are on the order of
> 10
> >     minutes.  That's 6 per hour or roughly 17 hours.  The slowest are
> closer
> >     to an hour.  So, obviously there's some growing pains to deal with.
>  I'll
> >     probably add a way for github committers to prioritize pull requests
> so
> >     they build first.
> >
> >     Luckily this stuff is trivial to throw hardware at.. it's super
> >     parallelizeable.  Also the hardware I have for those long runs is
> super
> >     old.  I think the freebsd/32 box is p4 era box.  The win/32 is my
> asus eee
> >     atom based netbook.
> >
> >     If anyone wants to volunteer build hardware, particularly for the
> >     non-linux platforms, please contact me via email (let's not clutter
> up the
> >     newsgroup with that chatter).
> >
> >     Requirements:
> >      I need to be able to access it remotely.
> >
> >      It needs to be have reliable connectivity (bandwidth is pretty much
> a
> >      non-issue.. it doesn't need much at all).
> >
> >      It needs to be hardware you're willing to have hammered fairly hard
> at
> >      random times.
> >
> >     I'll almost certainly write some code during the holiday to fire up
> and
> >     use ec2 nodes for windows and linux builds.  With the application of
> just
> >     a little money, all of those runs could be done fully parallel..
> which
> >     would just be sweet to see.  Ok, I admit it.. I love working at
> amazon on
> >     ec2.. and I'm happy to finally have a project that could actually
> use it.
> >
> >     Later,
> >     Brad
> >
> >
> >> With the application of just a little money
> >
> > Out of curiosity, how much is a little?
>
> I'll need to experiment.  It's the kind of thing that the more money
> thrown at the problem the faster the builds could
> be churned through.  The limit of that would be one box per platform per
> pull request.  A silly (and not possible due to
> ec2 not supporting some platforms, such as osx) extreme though.  One
> c1.medium running 24x7 at the current spot price is
> about $29/month.  Spot is perfect because this is a totally interruptable
> / resumable process.  I'll need to do a little
> work to make it resume an in-flight test run, but it's not hard at all to
> do.
>
> After watching the build's slowly churn through the first round of build
> requests, it's clear to me that one of the
> areas I'll need to invest a good bit more time is in scheduling of pulls.
>  Right now it's pseudo-random (an artifact of
> how the data is stored in a hash table).
>
> Later,
> Brad
>


That seems very reasonable.  It sounds like this autotester will help
immensely with processing the pull request backlog.  More free time for
Walter and everyone else who works on the project is a great thing.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20111217/d0e67721/attachment.html>