Autotesting dub packages with dmd nightly

[Seb via Digitalmars-d-announce]

On Monday, 22 August 2016 at 20:44:05 UTC, Sebastiaan Koppe wrote:
> On Wednesday, 10 August 2016 at 18:35:03 UTC, Sebastiaan Koppe 
> wrote:
>> So true. Then I will do PR's first.
>
> I finally got around implementing running dmd/druntime/phobos 
> pull requests against all dub packages. Thank you digger, for 
> making it so easy.
>
> I did a run of master + druntime#1602. Then I ran a comparison 
> with a batch from 2.071.2-b2. 108 packages had a different 
> build result. I have no nice stats or pictures, but a quick 
> glance over the raw data:
>
> 50 of them went from green unittests to a dmd exit code 1.
> 16 went from unknown build results to dmd exit code 1.
> 10 went from dmd exit code 255 to 1
> 9 of them are now green.
> 8 of them went from linker errors to a dmd exit code 1.
> 6 of then went from non-zero exit code during unittest run to a 
> dmd exit code 1.
> 3 previous ran out of memory but now resulted in dmd exit code 
> 1.
> etc.
>
> All in all I think +/- 96 package are affected. A little over 
> 11%.

That's awesome to know!

How difficult would it be to integrate it with the dlang GitHub 
PR workflow?
I am just shooting an idea that popped into my head:

We already use CircleCi and Travis for the dlang repos, so if we 
lock the packages to a fixed version (to prevent failures caused 
by the package authors), we might be able create a simple file 
like:

```
vibe.d==0.7.29
mir==0.16.3
...
```

We could select a subset (e.g. 50-100), s.t. the runtime doesn't 
get exorbitant.
Following we could then do enable the checking in CircleCi with 
sth. similar to:

```
wget 
https://raw.githubusercontent.com/dlang/community-list/master/dlang-packages.master # just an example, idea is to host the file somewhere on GitHub s.t. editing it is easy
dub fetch your-fancy-tool --version="x.y.y"
dub run your-fancy-tool --config dlang-stable.packages
```

Of course CircleCi doesn't have the access rights to post back to 
the hook API, but you could send a notification to dlang-bot [1] 
which has the permissions or let the CI error / fail.

Otherwise you could of course look into setting up your own job 
queue (or hack with the code from the auto-tester [2]), which 
might be fun too.

[1] https://github.com/MartinNowak/dlang-bot
[2] https://github.com/braddr/d-tester