[phobos] Time taken for running unit tests

Don Clugston dclugston at googlemail.com
Mon Sep 26 02:37:30 PDT 2011


I think we need to have a strategy for managing the amount of time
required for running the unittests. Currently, on Windows, the time
for all compiler tests + phobos tests is one hour. (The druntime tests
are only 30 seconds).
By contrast, running all compiler tests + phobos tests on D1, takes
about four minutes.

I used to be able to use test-driven development extensively --
running all tests after every change. But this is no longer viable on
D2. It's quite terrible to have to wait for one hour before finding
that you broke something.
A very large fraction of the time is used in testing only a tiny part
of the Phobos API. So I have some anxiety over what may happen in the
long term -- if current trends continue, we could easily have ten
hours of Phobos tests eventually.

How about defining a version, eg:
version=ExtendedPhobosTests;
which contains the more exhaustive, black-box tests, which take almost
all of the time. So that the standard tests would consist of
(1) regression tests, which should have a corresponding bugzilla
entry, unless they were discovered during development of the library
(The feature of these tests, is that at some point, they failed);
and
(2) code coverage tests.

Both of these execute quickly, and since they are linear with the
number of reported bugs + the number of lines of code, they should
always remain manageable. The black-box tests, on the other hand, are
potentially unlimited.


More information about the phobos mailing list