Parallel execution of unittests

[Jonathan M Davis via Digitalmars-d]

On Wed, 30 Apr 2014 22:32:33 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d at puremagic.com>
wrote:

> On 4/30/14, 10:01 PM, Jonathan M Davis via Digitalmars-d wrote:
> > I'm all for parallelizing all unittest blocks that are pure, as
> > doing so would be safe, but I think that we're making a big mistake
> > if we try and insist that all unittest blocks be able to be run in
> > parallel. Any that aren't pure are not guaranteed to be
> > parallelizable, and any which access system resources or other
> > global, mutable state stand a good chance of breaking.
> 
> There are a number of assumptions here: (a) most unittests that can
> be effectively parallelized can be actually inferred (or declared) as
> pure; (b) most unittests that cannot be inferred as pure are likely
> to break; (c) it's a big deal if unittests break. I question all of
> these assumptions. In particular I consider unittests that depend on
> one another an effective antipattern that needs to be eradicated.

Even if they don't depend on each other, they can depend on the system.
std.file's unit tests will break if we parallelize them, because it
operates on files and directories, and many of those tests operate on
the same temp directories. That can be fixed by changing the tests, but
it will break the tests. Other tests _can't_ be fixed if we force them
to run in parallel. For instance, some of std.datetime's unit tests set
the local time zone of the system in order to test that LocalTime works
correctly. That sets it for the whole program, so all threads will be
affected even if they're running other tests. Right now, this isn't a
problem, because those tests set the timezone at their start and reset
it at their end. But if they were made to run in parallel with any other
tests involving LocalTime, there's a good chance that those tests would
have random test failures. They simply can't be run in parallel due to
a system resource that we can't make thread-local. So, regardless of
how we want to mark up unittest blocks as parallelizable or not
parallelizable (be it explicit, implict, using pure, or using something
else), we do need a way to make it so that a unittest block is not run
in parallel with any other unittest block.

We can guarantee that pure functions can safely be run in parallel. We
_cannot_ guarantee that impure functions can safely be run in parallel.
I'm sure that many impure unittest functions could be safely run in
parallel, but it would require that the programmer verify that if we
don't want undefined behavior - just like programmers have to verify
that @system code is actually @safe. Simply running all unittest blocks
in parallel is akin to considering @system code @safe in a particular
piece of code simply because by convention that code should be @safe.

pure allows us to detect guaranteed, safe parallelizability. If we want
to define some other way to make it so a unittest block can be marked as
parallelizable regardless of purity, then fine. But
automatically parallelizing impure functions means that we're going to
have undefined behavior for those unittest functions, and I really
think that that is a bad idea - in addition to the fact that some
unittest blocks legitimately cannot be run in parallel due to the use
of system resources, so parallelizing them _will_ not only break them
but make them impossible to write in a way that's not broken without
adding mutexes to the unittest blocks to stop the test runner from
running them in parallel. And IMHO, if we end up having to do that
anywhere, we've done something very wrong with how unit tests work.

- Jonathan M Davis