6-weeks release cycle

[Jonathan M Davis via Digitalmars-d]

On Friday, 5 June 2015 at 06:42:06 UTC, Rikki Cattermole wrote:
> And shouldn't the CI be doing regression testing already?

The autotester runs the unit tests that are in druntime, Phobos, 
and dmd. It catches a lot of stuff and generally prevents us from 
merging bad code. But it can't possibly catch everything. It 
catches the stuff that we thought to specifically test for and 
bugs that were fixed (since unit tests are usually added for bug 
fixes). But it's not that infrequent for something _new_ to break 
that's never broken before and is subtle enough that it doesn't 
get past all of the tests - especially when you're dealing with 
changes to the compiler.

To catch all of that stuff before it goes out the door in a 
release, we need to test a _much_ larger code base than just the 
standard stuff - which is part of why we have betas. We want 
folks to try out their projects with the betas so that we can 
catch the stuff that we missed before it gets released in an 
official release. Simply grabbing an arbitrary commit and 
declaring it a release just because it's at about the time that 
we want to do a release would be a disaster. Too much gets 
through as it is simply because not enough folks test the betas 
and report what they find. _All_ of that would get through if we 
just picked a random commit and declared it to be a release.

_Maybe_ someday our test suites will catch such a large portion 
of the regressions that we won't actually end up with any 
regressions getting out, but I doubt it. Even large, heavily used 
projects like gcc or KDE end up with regressions getting out, 
much as they try to avoid it. But like them, we need to do our 
best to have releases which have been tested well enough via 
betas and whatnot rather than just releasing stuff simply because 
it's a certain date.

- Jonathan M Davis