[dmd-internals] dmd test suite

[Brad Roberts]

On 6/14/2010 1:05 AM, Don Clugston wrote:
> On 13 June 2010 11:30, Brad Roberts <braddr at puremagic.com> wrote:
>> I've just checked in the skeleton of a test suite for us to discuss and
>> hopefully start adding to rather than continuing to expand the non-distributable
>> one.
> 
> Some comments:
> 1) This seems to be following the dstress design rather than the
> internal test suite design, in that you have one bug per file.
> That's great if you're developing a new compiler, but for regression
> testing, it's going to be really slow, since the testing time depends
> far more strongly on the number of files rather than the number of
> tests. Even with the internal test suite, running all the tests takes
> an annoyingly long time.
> 2) I think it'd be nice if the tests were run in the order
> most-recently-added first, rather than oldest-first. The reason for
> this is that the more recent tests are nearly always tougher than the
> old tests.
> 3) It's in the DMD repository, which means only Walter has write access
> to it. Of course it is important that the test suite always compiles
> with the latest compiler update. But still, I'd hope that we can
> reduce Walter's workload somehow.
> (It's a particular problem while initially building up the test suite).
> _______________________________________________
> dmd-internals mailing list
> dmd-internals at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals

1) The few tests I've checked in are simple single tests, but nothing about the
make file logic for either the compilable or runnable tests prevents it from
taking the big file with lots of tests approach either.  The fail_compilation
tests need to be single-unit tests or it's too easy for sub-sets to start
compiling when they shouldn't and go unnoticed.  For the parts that have been
checked in so far, I just wanted to make sure things worked and that doesn't
need big chunks of tests.  I agree that the big chunks of tests style is a major
win in total testing time.

2) Test ordering might be useful.  I really wanted to get away from having to
maintain lists of tests to run... just drop new files in the right place and go.
 Might be doable with something like $(shell ls -1tr runnable/*.d).. worth
playing with.

3) Submit permissions are the easiest part.  For now, until Walter is comfy with
others submitting changes, I'd like us to just include test changes in the diffs
we attach to bug reports.  ie, fix + tests for it come as a single package.  I
don't think that the tests need to be separate just for permission management.
Just making it public is a huge step in the right direction.  As long as it get
used.

As for back filling from bugzilla.. after just a little while looking through it
and the existing tests, there's just not enough cross referencing to make it
easy to tell what's already in the test suite somewhere.  It'd be a whole lot
easier to just add to it as new bugs are identified and fixed.

==============

Walter.. your 2 cents are really important on this thread.  You need to agree
that this is the direction you're willing to go in and support or it's a waste
of time to continue with.  Another dstress isn't useful.

Specifically, what I think is needed from you:

  1) some way to work out what parts of the existing test suite can be moved to
the public view.  The vast majority of it ought to be directly transportable
even if the ancestry isn't fully documented.  They're obviously
non-copyrightable snippits of code.. little 5-10 line blocks.  For example,
mixin1.d.

  2) agreement to start adding to it rather than the non-distributable tests

  3) agreement that you'll run it as part of your testing

#1 is key.. without it it'll take a long time for the suite to be useful for
anyone.  I'm hoping that the existing suite will be able to quickly shrink down
to rather few tests that aren't safe enough to make public.

I'm willing to do the file by file examination.  Moving only the parts that are
simple.  Anything not simple I'll leave for a second pass for either you to do
or at least to discuss and then me continue doing.

Thoughts?

Thanks,
Brad