[dmd-internals] dmd commit, revision 657

[Jason House]

On Sep 2, 2010, at 3:46 PM, Brad Roberts <braddr at puremagic.com> wrote:

> On Thu, 2 Sep 2010, Jason House wrote:
> 
>> Here's how I'd rank things by level of dislike:
>> ? A crippled test suite where failing tests are deleted
>> ? A crippled test suite with failing tests are commented out
>> ? A complete test suite that always fails due to bugs way down on the priority list
>> ? A complete test suite that ignores tests but reports that it's ignoring tests
>> ? A complete test suite that runs all tests but is tolerant of hand-picked tests failing (xFAIL, etc...)
>> ? A complete test suite with all tests passing
>> 
>> The last one us unrealistic while under development. In fact, a 
>> "complete test suite" is also unrealistic, but a test suite should 
>> slowly become more and more thorough. Certainly, failing tests should 
>> not be removed.
> 
> The last one is what we have, as long as you drop the word 'complete'.


At the moment, that makes sense for the public test suite. As I understand it, you're playing catch-up; you're actively converting tests from the private test suite into the public one.


>  DMD's test suite is monotomically increasing in completeness and releases 
> don't comment out tests except in some extreme exceptions.. such as 
> rolling back a change due to finding regression that wasn't covered by the 
> test suite.
> 
> ie:
>  fix a bug + add a test that showed the bug
>  discover a bug that fix caused that didn't have a test in the suite
>  roll back the fix and the new test
>  re-fix with both the new test and the newly discovered test

It should be simpler than that... When a bug is discovered, add a test. In your example above, two tests should be committed regardless of if the attempted patch is committed.

Let's take a hypothetical case of that Don attempted a patch and the new bug is best handled by Walter. The new tests represent a very clear communication of the problem space. Done right, that should make it easier for Walter to repeat Don's tests.

Similarly, many bugzilla entries gave a "reduced test case" posted to them when Don confirms the bug. Such tests should also make their way into the test suite.

I have no idea how Don currently concludes that patch for bugzilla 1234333 also fixes bugzilla 333322 and 3462663. A test suite with failing tests is one way to determine such things.

PS: Don, I'm not trying to pick on you. I'm using you as an example since you're such a prolific contributor. 

PSS: Brad, I'm not picking on you either. I'm just using recent emails as inspiration to bring up a long overdue topic. All your work into the public test suite is commendable. It's much easier for me to be an armchair philosopher.

> The major wish I have is that walter would start using the public test 
> suite rather than the private one.  I'm working on getting the public one 
> to work on windows -- making good progress, it runs most of the tests now.
> 
> Once walter actually uses that one, it'll be easier to monitor to make 
> sure fixes come with tests. :)


That's a great goal!
>