[dmd-internals] dmd commit, revision 657

[Jason House]

On Sep 2, 2010, at 2:07 PM, Brad Roberts <braddr at puremagic.com> wrote:

> I agree.  However, in this case, disabling the tests and turning on an extra
> level of tightness in error checking of the test suite was a net improvement.



> At the same time I disabled those three tests, I made it so that any
> 'fail_compilation' test that causes the compiler to crash (segv, abort, etc..
> any signal) to fail the test.

Detecting seg faults as failures is definitely a good change.


> The three tests that caused the compiler to crash
> all have bugs filed for them and all have patches pending.

I'd argue that is a good reason to leave the failures in the test suite results. It's a reminder to go apply the patch!


> Anyway.. I considered adding something to do more formal disabling, but I hate
> disable tests more than that would indicate.  I don't want to even have the mechanism.

Here's how I'd rank things by level of dislike:
 • A crippled test suite where failing tests are deleted
 • A crippled test suite with failing tests are commented out
 • A complete test suite that always fails due to bugs way down on the priority list
 • A complete test suite that ignores tests but reports that it's ignoring tests
 • A complete test suite that runs all tests but is tolerant of hand-picked tests failing (xFAIL, etc...)
 • A complete test suite with all tests passing

The last one us unrealistic while under development. In fact, a "complete test suite" is also unrealistic, but a test suite should slowly become more and more thorough. Certainly, failing tests should not be removed.