[phobos] Time taken for running unit tests

[Sean Kelly]

On Sep 26, 2011, at 2:37 AM, Don Clugston wrote:
> 
> How about defining a version, eg:
> version=ExtendedPhobosTests;
> which contains the more exhaustive, black-box tests, which take almost
> all of the time. So that the standard tests would consist of
> (1) regression tests, which should have a corresponding bugzilla
> entry, unless they were discovered during development of the library
> (The feature of these tests, is that at some point, they failed);
> and
> (2) code coverage tests.
> 
> Both of these execute quickly, and since they are linear with the
> number of reported bugs + the number of lines of code, they should
> always remain manageable. The black-box tests, on the other hand, are
> potentially unlimited.

I'd definitely support something like this.  At a prior job, the product I was working on had extensive tests (a full regression run took over 24 hours) and every commit had to be preceded by a MAT (minimal acceptance test) run.  This caught most product-breaking bugs and completed in only a few minutes.

What if the modules each contained what are essentially a MAT validation suite and the exhaustive tests lived in other files?  This is already basically how Phobos testing works on Posix, and avoids cluttering up module files with thousands of lines of test code.