Parallel execution of unittests

[Jason Spencer via Digitalmars-d]

On Thursday, 1 May 2014 at 17:57:05 UTC, Andrei Alexandrescu 
wrote:
> Well how complicated can we make it all? -- Andrei

As simple as possible, but no simpler :)

I've seen you favor this or that feature because it would make 
unit testing easier and more accessible, and eschew features that 
would cause folks to not bother.  In truth, we could leave it how 
it is.  But I surmise you started this thread to improve the 
feature and encourage more use of unit test.  So we're looking 
for the sweet spot.

I don't think it's important to support the sharing of state 
between unit tests.  But I do see value in being able to 
influence the order of test execution, largely for debugging 
reasons.  It's important for a module's tests to be able to 
depend on other modules--otherwise, unittest is not very 
enticing.  If it does and there's a failure, it's hugely helpful 
to know the failure is caused by the unit-under-test, and not the 
dependency(s).  The common way to do that is to run the tests in 
reverse order of dependency--i.e. levelize the design and test 
from the bottom up.  See "Large Scale C++ SW Design", Lakos, Chp. 
3-4.

I imagine there are other niche reasons for order, but for me, 
this is the driving reason.  So it seems a nice middle ground.

If order is important, it might be a workable approach to run 
unittest in the reverse module dependency order by default.  A 
careful programmer could arrange those classes/functions in 
modules to take advantage of that order if it were important.  
Seems like we'd have the dependency information--building and 
traversing a tree shouldn't be that tough....  To preserve it, 
you'd only be able to parallelize the UTs within a module at a 
time (unless there's a different flag or something.)

But it seems the key question is whether order can EVER be 
important for any reason.  I for one would be willing to give up 
parallelization to get levelized tests.  What are you seeing on 
your project?  How do you allow tests to have dependencies and 
avoid order issues?  Why is parallelization more important than  
that?