[phobos] Time taken for running unit tests

[Jonathan M Davis]

On Monday, September 26, 2011 11:37:30 Don Clugston wrote:
> I think we need to have a strategy for managing the amount of time
> required for running the unittests. Currently, on Windows, the time
> for all compiler tests + phobos tests is one hour. (The druntime tests
> are only 30 seconds).
> By contrast, running all compiler tests + phobos tests on D1, takes
> about four minutes.
> 
> I used to be able to use test-driven development extensively --
> running all tests after every change. But this is no longer viable on
> D2. It's quite terrible to have to wait for one hour before finding
> that you broke something.
> A very large fraction of the time is used in testing only a tiny part
> of the Phobos API. So I have some anxiety over what may happen in the
> long term -- if current trends continue, we could easily have ten
> hours of Phobos tests eventually.
> 
> How about defining a version, eg:
> version=ExtendedPhobosTests;
> which contains the more exhaustive, black-box tests, which take almost
> all of the time. So that the standard tests would consist of
> (1) regression tests, which should have a corresponding bugzilla
> entry, unless they were discovered during development of the library
> (The feature of these tests, is that at some point, they failed);
> and
> (2) code coverage tests.
> 
> Both of these execute quickly, and since they are linear with the
> number of reported bugs + the number of lines of code, they should
> always remain manageable. The black-box tests, on the other hand, are
> potentially unlimited.

In principle, I think that it's a solid idea, though I'm not sure that it's 
necessarily at all obvious where the line should be drawn between the normal 
and extended tests, and that could be a bit tricky. We wouldn't want the 
normal unit tests to become not extensive enough simply in an effort to reduce 
the amount of time that it takes to run tests, but on the other hand, we don't 
want to the tests to take forever, so it's a bit of a balancing act.

For whatever it's worth, on the whole, I believe that the main culprit for the 
length of time that it takes to run the unit tests is the amount of time that 
it takes to compile them. Actually running the tests is generally quite quick. 
And Windows does that much worse then, because it compiles all of the modules 
together for unit tests instead of separately like Posix does - though that 
has the advantage of helping to find module interdependencies with module 
constructors which don't get found otherwise, so I'm not sure that we 
necessarily want to change how Windows does the unit tests. Maybe one way to 
deal with that would be to adjust the makefiles so that the modules are 
normally tested separately on _all_ of the OSes, but there's make target on 
all OSes to build them all as one if we want to test for module constructor 
issues - though then that target risks not every being called.

The other question is what the autotester should be running. Should it run the 
normal unit tests or the extend unit tests? Should it run then the normal ones 
normally and the extended ones periodically? It'll be easier to miss platform-
specific bugs if the extended tests are only ever run on an individual's 
machine and never the autotester.

On the whole though, I think that this is a good idea.

- Jonathan M Davis