Please be more careful when changing the language behavior

[kinke]

On Tuesday, 2 June 2020 at 09:56:41 UTC, Mathias LANG wrote:
> On Tuesday, 2 June 2020 at 08:44:56 UTC, Jon Degenhardt wrote:
>> Package owners can help out by setting up weekly or nightly 
>> cron jobs that run the package test suite on different 
>> platform and compiler combinations, including nightly and beta 
>> builds. For some packages this can generate a broader set of 
>> tests than are run in the project tester.
>
> The main problem is replicating Travis' infrastructure. 
> Anything that doesn't rely on the CI of the project is bound to 
> get outdated. And for the moment we rely on the parsing the 
> testsuite from the `.travis.yml`. But the runners have their 
> own environment (provisioned) and there's no way for a project 
> to install its dependencies / specific tools (e.g. the lack of 
> meson have led to countless issues). You also have to pin 
> package versions to a single one.

BuildKite certainly improved the overall situation, but it's 
obviously not scalable and can already be a problem if compiler 
changes require a new version of one of the tested projects. 
E.g., I've been waiting for months for a new Ocean release to be 
finally published with a tiny adaptation, and now that 
Sociomantic isn't anymore, I'm not sure I'll see that release 
happen this year at all.

Anyway, projects contributing themselves to regression testing 
via regularly triggered builds using the latest beta would be 
very much appreciated. But that doesn't work for DMD, where the 
final release usually happens 1-2 weeks after the 1st beta - that 
won't be enough for people to adapt their projects to 
language/library changes and investigate any apparent regressions.