[dmd-internals] dmd test suite

[Don Clugston]

On 14 June 2010 10:32, Brad Roberts <braddr at puremagic.com> wrote:
> On 6/14/2010 1:05 AM, Don Clugston wrote:
>> On 13 June 2010 11:30, Brad Roberts <braddr at puremagic.com> wrote:
>>> I've just checked in the skeleton of a test suite for us to discuss and
>>> hopefully start adding to rather than continuing to expand the non-distributable
>>> one.
>>
>> Some comments:
>> 1) This seems to be following the dstress design rather than the
>> internal test suite design, in that you have one bug per file.
>> That's great if you're developing a new compiler, but for regression
>> testing, it's going to be really slow, since the testing time depends
>> far more strongly on the number of files rather than the number of
>> tests. Even with the internal test suite, running all the tests takes
>> an annoyingly long time.
>> 2) I think it'd be nice if the tests were run in the order
>> most-recently-added first, rather than oldest-first. The reason for
>> this is that the more recent tests are nearly always tougher than the
>> old tests.
>> 3) It's in the DMD repository, which means only Walter has write access
>> to it. Of course it is important that the test suite always compiles
>> with the latest compiler update. But still, I'd hope that we can
>> reduce Walter's workload somehow.
>> (It's a particular problem while initially building up the test suite).
>> _______________________________________________
>> dmd-internals mailing list
>> dmd-internals at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>
> 1) The few tests I've checked in are simple single tests, but nothing about the
> make file logic for either the compilable or runnable tests prevents it from
> taking the big file with lots of tests approach either.  The fail_compilation
> tests need to be single-unit tests or it's too easy for sub-sets to start
> compiling when they shouldn't and go unnoticed.

Cool. Incidentally, in most cases, you can use static
assert(!is(typeof(  xxx )))
to turn it into a should-pass test. That's what I've been doing with
the CTFE should-not-compile tests.
(It generally doesn't work for the syntactic pass, though).

  For the parts that have been
> checked in so far, I just wanted to make sure things worked and that doesn't
> need big chunks of tests.  I agree that the big chunks of tests style is a major
> win in total testing time.

Excellent.
Something I really like about your setup is the 'import' directory. It
really tidies things up.

> 2) Test ordering might be useful.  I really wanted to get away from having to
> maintain lists of tests to run... just drop new files in the right place and go.
>  Might be doable with something like $(shell ls -1tr runnable/*.d).. worth
> playing with.
>
> 3) Submit permissions are the easiest part.  For now, until Walter is comfy with
> others submitting changes, I'd like us to just include test changes in the diffs
> we attach to bug reports.  ie, fix + tests for it come as a single package.  I
> don't think that the tests need to be separate just for permission management.
> Just making it public is a huge step in the right direction.  As long as it get
> used.

Agreed. I think it can potentially save a huge amount of time.

> As for back filling from bugzilla.. after just a little while looking through it
> and the existing tests, there's just not enough cross referencing to make it
> easy to tell what's already in the test suite somewhere.  It'd be a whole lot
> easier to just add to it as new bugs are identified and fixed.

I have a personal repository with all of the test suites which Walter
has sent me.
So I can do a diff with everything from 25 Sept 2009. Everything added
since then is clean.
Might be a good place to start. (After adding in the tests from TDPL).