Serious Problems with the Test Suite

H. S. Teoh hsteoh at quickfur.ath.cx
Thu Jun 18 02:34:42 UTC 2020


On Thu, Jun 18, 2020 at 01:59:39AM +0000, Avrina via Digitalmars-d wrote:
> On Wednesday, 17 June 2020 at 23:59:52 UTC, Walter Bright wrote:
> > 11. Every time we run into "oh, that's just a heisenbug, try
> > re-running the test" that is a BUG in the test suite and needs to be
> > fixed. Those are gigantic time and resource wasting problems.
> 
> I've run into these problems with, for example, optlink. When trying
> to get optlink removed, you prevent it. These heisenbugs exist
> because, a lot of the time, you aren't willing to chop off dead
> weight.

Whoa, holey miss the point batman!  Optlink may have its own share of
issues, but the problem here isn't with this or that piece of software,
it's with the structure of the testsuite.

Tests that are non-deterministic or depend on external state, strictly
speaking, shouldn't be in the test suite. This includes tests that
involve downloading some remote resource over the network, tests that
assume things about the host OS and filesystem, etc..  There are a
couple of these in the test suite, and they put you at the mercy of
external state which is beyond your control. (I remember one time there
was a heisenbug that had to do with random number generators, meaning,
its probability of arbitrary, totally coincidental failure was non-zero.
Sigh.)

These tests ought to be removed, or at least disabled in CI.  Any time
you depend on external state, it really does not belong in the test
suite, or at least, it does not belong in the autotester, because it
just leads to tons of wasted time trying to track down exactly what it
is that failed, which most of the time isn't even relevant to the PR
you're trying to push through.


T

-- 
MASM = Mana Ada Sistem, Man!


More information about the Digitalmars-d mailing list