Please be more careful when changing the language behavior

[Mathias LANG]

On Tuesday, 2 June 2020 at 08:44:56 UTC, Jon Degenhardt wrote:
> On Tuesday, 2 June 2020 at 08:22:03 UTC, Seb wrote:
>> On Tuesday, 2 June 2020 at 06:17:50 UTC, mw wrote:
>>> E.g. https://code.dlang.org/ has total 1808 packages. Given 
>>> some packages are no longer maintained, let's add the top 500 
>>> (or 1000) most recently updated packages to the compiler test 
>>> suite: make sure at least these top 500 can be compiled (weak 
>>> guarantee), and their `dub test` all pass (strong guarantee), 
>>> before making new compiler releases.
>>
>> We're already doing the strong guarantee (dub test) 
>> continuously ;-)
>> Check the buildkite.com/dlang or a buildkite link on any of 
>> the main D repos.
>> At the moment it's about 50 projects, but that's already a 
>> huge pain to maintain as some of these projects are hard to 
>> build, relied on network to test, had spurious failures due to 
>> randomness in the testsuite, rely on very specific OS features 
>> (or versions) and every now and then a new version is released 
>> which either introduces some of the above or just has an 
>> outright problem with the testsuite.
>
> Package owners can help out by setting up weekly or nightly 
> cron jobs that run the package test suite on different platform 
> and compiler combinations, including nightly and beta builds. 
> For some packages this can generate a broader set of tests than 
> are run in the project tester.

The main problem is replicating Travis' infrastructure. Anything 
that doesn't rely on the CI of the project is bound to get 
outdated. And for the moment we rely on the parsing the testsuite 
from the `.travis.yml`. But the runners have their own 
environment (provisioned) and there's no way for a project to 
install its dependencies / specific tools (e.g. the lack of meson 
have led to countless issues). You also have to pin package 
versions to a single one.

Personally I'd like to have support for Github CI and explore 
having the test-suite run cross-platform when possible (for the 
moment we only test Linux).