Auto-testing of GitHub pull requests

[Brad Roberts]

On 9/25/13 2:58 PM, Joseph Rushton Wakeling wrote:
> On 24/09/13 18:25, Brad Roberts wrote:
>> I'd use a standard tool (or would have, silly to switch when this one does the
>> job) but none of them support the combination of features used.  Primarily:
>> github, testing on multiple platforms, testing pulls as merges into the target
>> branch (rather than just the pull itself), multiple repositories in the same
>> build.  There's probably more requirements, it's been a while since I researched
>> the state of the ecosystem and exactly what the gaps are.
>>
>> But now that it's written, it's just fundamentally not that much code, nor is it
>> actually difficult code.
>
> Thanks very much for pointing me to that.  I'm looking into auto-testing stuff that would work with
> a project hosted on Gitorious (GitHub is unsatisfactory for political reasons) -- I'm working on the
> assumption that one of the standard testing frameworks will play nice with that (I've seen a writeup
> on Gerrit + Gitorious), but assuming anything custom is required, this could be a very useful template.

The only parts that really hook into github are:

1) serverd/update_pulls.d -- That's a cronjob that polls github ever few minutes for the current set 
of pull requests.  It's pretty ghetto due to evolution and lack of time investment.  It really ought 
to be replaced with a webhook that github posts to and just left around as a backup in case a hook 
call gets lost.

2) server/add/github-post.ghtml -- That's called by a github web hook when the master branches are 
checked into.  Ignore the language it's written in.  The gist is easy to understand: grab what's 
been updated, flush old state in the db.

Unless I'm forgetting something, all the rest of the code is the tester itself:

a) client -- the code that executes on the clients

b) server -- the web ui, written in an ancient scripting language that I still use from a previous 
job.  Don't use as is unless you enjoy reverse engineering interpreters.

c) serverd -- the apis that the client code calls to ask about work and report status.

And there's a massive lack of ui surrounding the configuration tables.  I keep putting off working 
on them since it's awfully rare that any of them need to be updated, but I always hate running sql 
directly against a live data store.


In case stats are of interesting to anyone (they are to me):

# of builds/tests executed so far
    741000 pull          (first one on Dec 14, 2011)
     51000 master branch (first one on Sept 5th, 2010 -- just over 3 years ago)

Users with 100 or more closed pull requests (note, data doesn't cover a small number pull requests 
that pre-dates the tester)
+---------+----------------+----------+
| user_id | username       | count(*) |
+---------+----------------+----------+
|  254394 | 9rnsr          |     1068 |
|  288976 | dawgfoto       |      328 |
|  580100 | yebblies       |      306 |
|  567837 | donc           |      283 |
|  568298 | WalterBright   |      229 |
|  360179 | AndrejMitrovic |      226 |
|  579956 | jmdavis        |      167 |
|   44076 | alexrp         |      152 |
| 1911406 | monarchdodra   |      129 |
|   40242 | braddr         |      123 |
+---------+----------------+----------+