Thinktank: CI's, compiler lists, and project automation

[Nick Sabalausky (Abscissa)]

On 03/03/2018 10:22 AM, Jacob Carlborg wrote:
> On 2018-03-03 07:39, Nick Sabalausky (Abscissa) wrote:
>> - Maybe there's a simple setting I've overlooked, but when a build job 
>> fails on travis, the author does not get proactively notified. The 
>> author only finds out next time they go into travis. (I've been 
>> surprised many times to discover failed builds that had occured 
>> several days ago, or more.)
> 
> You should get an email notification about failed builds [1] [2],

Weird. Never worked for me. Will have to check into that.

>> - A project author will still need to: 1. Actively notice new compiler 
>> releases and 2. Manually update the .travis.yml files for each of 
>> their projects. Certainly there's room for more automation here.
> 
> No. If you specify "dmd" (without any version) in the list of compilers 
> [5], it will build the latest version. For example, this build [3], 
> which ran 23 hours ago uses DMD 2.078.3, while this one [4], 29 days 
> ago, uses 2.078.2. As you can see on the commit in Travis CI, it's the 
> same commit.
> 

No. That is only sufficient *temporarily*, that is, until the next time 
the label "dmd" is updated to another new release once again. At *some* 
point, .travis.yml will still need to be manually updated...

Scenario 1-2-3:

1. Suppose a library supports DMD v2.071.0 and up. Because of this, 
.travis.yml includes:

   - dmd  # Ie, dmd-2.079.0, ATM
   - dmd-2.079.0
   - dmd-2.078.2
   - dmd-2.078.1
   - dmd-2.078.0
   - dmd-2.077.1
   - dmd-2.077.0

Summary: CI now tests 2.076.1 through 2.079.0.

2. Release of DMD 2.080.0 occurs. "dmd" now points to 2.080.0 (and 
forget reproducibility: if 2.080.0 contains an unexpected regression 
then using travis's "Restart Job" on the "dmd" job will break overall 
reproducibility.)

Summary: CI now tests 2.076.1 through 2.080.0.

So far so good, right? Until...

3. Release of DMD 2.081.0 occurs. "dmd" now points to 2.081.0. Unless 
.travis.yml has been manually updated, it still contains:

   - dmd  # Ie, dmd-2.081.0, now
   # WTF?!?! dmd-2.080.0 is no longer being tested!!!!!!!!
   - dmd-2.079.0
   - dmd-2.078.2
   - dmd-2.078.1
   - dmd-2.078.0
   - dmd-2.077.1
   - dmd-2.077.0

Summary: CI *stops* testing dmd-2.080.0.

FAIL.