Adding more projects to the Project Tester

[Seb]

On Friday, 6 July 2018 at 21:25:28 UTC, tide wrote:
> On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote:
>> So learning from the recent Vibe.d regression fiasco (we 
>> temporarily disabled a subconfiguration in Vibe.d and promptly 
>> got a regression in 2.081), I think we should try to add more 
>> projects to the Project Tester.
>>
>> The current list is here:
>> https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy#L443
>>
>> Any suggestions?
>>
>> Why should I add my project to the Project Tester?
>> --------------------------------------------------
>>
>> Once a project is added to the Project Tester, DMD can't 
>> regress on it anymore as for every PR at dmd, druntime, 
>> phobos, tools and dub the testsuite of the added projects are 
>> run.
>>
>> How does the Project Tester work?
>> ---------------------------------
>>
>> - By default, it will run the same commands as Travis would 
>> do. Although, if necessary, custom commands can be used too.
>> - It will checkout the latest, stable git tag
>>
>> Requirements
>> ------------
>>
>> - moderately popular or was prone to regressions in the past
>> - rather easy to build (i.e. you don't need to download and 
>> recompile clang)
>> - no flaky testsuite (random errors in the testsuite due to 
>> network connectivity shouldn't happen. Though there's 
>> `DETERMINISTIC_HINT=1` set, s.t. you could disable such parts 
>> of your testsuite)
>> - reachable author or development (if there's ever a case 
>> where we need to push changes via a trivial PR to the repo, it 
>> shouldn't sit in the queue for weeks)
>
> Include Windows as part of the testing done.

Just to avoid confusion, testing of DMD/Druntime/Phobos/etc. is 
done on Windows:

https://auto-tester.puremagic.com/platform-history.ghtml?projectid=1&os=Win_32
https://auto-tester.puremagic.com/platform-history.ghtml?projectid=1&os=Win_32_64
https://ci.appveyor.com/project/greenify/dmd
...

In my experience are the huge majority of all regressions 
originating in the D frontend where architecture actually doesn't 
matter so much.

That being said, we are currently investigating to move the 
Project Tester from Jenkins to Buildkite 
(https://github.com/dlang/ci/pull/225) and one advantage of this 
move is that it'll be a lot easier to add new machines to the 
build cloud (so you could then hook up that Windows machine in 
your garage with the Project Tester).
However, AFAICT many of the dub projects do have dependencies 
that are harder to install on Windows, so this might not happen 
in the near future.