Parallel execution of unittests

[Jonathan M Davis via Digitalmars-d]

On Wed, 30 Apr 2014 20:33:06 -0400
Steven Schveighoffer via Digitalmars-d <digitalmars-d at puremagic.com>
wrote:

> On Wed, 30 Apr 2014 13:50:10 -0400, Jonathan M Davis via
> Digitalmars-d <digitalmars-d at puremagic.com> wrote:
> 
> > On Wed, 30 Apr 2014 08:59:42 -0700
> > Andrei Alexandrescu via Digitalmars-d <digitalmars-d at puremagic.com>
> > wrote:
> >
> >> On 4/30/14, 8:54 AM, bearophile wrote:
> >> > Andrei Alexandrescu:
> >> >
> >> >> A coworker mentioned the idea that unittests could be run in
> >> >> parallel
> >> >
> >> > In D we have strong purity to make more safe to run code in
> >> > parallel:
> >> >
> >> > pure unittest {}
> >>
> >> This doesn't follow. All unittests should be executable
> >> concurrently. -- Andrei
> >>
> >
> > In general, I agree. In reality, there are times when having state
> > across unit tests makes sense - especially when there's expensive
> > setup required for the tests.
> 
> int a;
> unittest
> {
>     // set up a;
> }
> 
> unittest
> {
>     // use a;
> }
> 
> ==>
> 
> unittest
> {
>     int a;
>     {
>        // set up a;
>     }
>     {
>        // use a;
>     }
> }
> 
> It makes no sense to do it the first way, you are not gaining
> anything.

It can make sense to do it the first way when it's more like

LargeDocumentOrDatabase foo;

unittest
{
    // set up foo;
}

unittest
{
   // test something using foo
}

unittest
{
  // do other tests using foo which then take advantage of changes made
  // by the previous test rather than doing all of those changes to
  // foo in order to set up this test
}

In general, I agree that tests shouldn't be done that way, and I
don't think that I've ever done it personally, but I've seen it done,
and for stuff that requires a fair bit of initialization, it can save
time to have each test build on the state of the last. But even if we
all agree that that sort of testing is a horrible idea, the language
supports it right now, and automatically parallelizing unit tests will
break any code that does that.

> > Honestly, the idea of running unit tests in parallel makes me very
> > nervous. In general, across modules, I'd expect it to work, but
> > there will be occasional cases where it will break.
> 
> Then you didn't write your unit-tests correctly. True unit
> tests-anyway.
> 
> In fact, the very quality that makes unit tests so valuable (that
> they are independent of other code) is ruined by sharing state across
> tests. If you are going to share state, it really is one unit test.

All it takes is that tests in two separate modules which have separate
functionality access the file system or sockets or some other system
resource, and they could end up breaking due to the fact that the other
test is messing with the same resource. I'd expect that to be a
relatively rare case, but it _can_ happen, so simply parallelizing
tests across modules does risk test failures that would not have
occurred otherwise.

> > Across the unittest
> > blocks in a single module, I'd be _very_ worried about breakage.
> > There is nothing whatsoever in the language which guarantees that
> > running them in parallel will work or even makes sense. All that
> > protects us is the convention that unit tests are usually
> > independent of each other, and in my experience, it's common enough
> > that they're not independent that I think that blindly enabling
> > parallelization of unit tests across a single module is definitely
> > a bad idea.
> 
> I think that if we add the assumption, the resulting fallout would be
> easy to fix.
> 
> Note that we can't require unit tests to be pure -- non-pure
> functions need testing too :)

Sure, they need testing. Just don't test them in parallel, because
they're not guaranteed to work in parallel. That guarantee _does_ hold
for pure functions, because they don't access global, mutable state.
So, we can safely parallelize a unittest block that is pure, but we
_can't_ safely paralellize one that isn't - not in a guaranteed way.

> I can imagine that even if you could only parallelize 90% of unit
> tests, that would be an effective optimization for a large project.
> In such a case, the rare (and I mean rare to the point of I can't
> think of a single use-case) need to deny parallelization could be
> marked.

std.file's unit tests would break immediately. It wouldn't surprise me
if std.socket's unit tests broke. std.datetime's unit tests would
probably break on Posix systems, because some of them temporarily set
the local time zone - which sets it for the whole program, not just the
current thread (those tests aren't done on Windows, because Windows only
lets you set it for the whole OS, not just the program). Any tests which
aren't pure risk breakage due to changes in whatever global, mutable
state they're accessing.

I would strongly argue that automatically parallelizing any unittest
block which isn't pure is a bad idea, because it's not guaranteed to
work, and it _will_ result in bugs in at least some cases. If we make it
so that unittest blocks have their purity inferred (and allow you to
mark them as pure to enforce that they be pure if you want to require
it), then any unittest blocks which can safely be parallelized will be
known, and the test runner could then parallelize those unittest
functions and then _not_ parallelize the ones that it can't guarantee
are going to work in parallel.

So, then we get safe, unittest parallelization without having to insist
that folks write their unit tests in a particular way or that they do or
don't do particular things in a unit test. And maybe we can add add
some sort of UDA to tell the test runner that an impure test can be
safely parallelized, but automatically parallelizing impure unittest
functions would be akin to automatically treating @system functions as
@safe just because we thought that only @safe code should be used in
this particular context.

- Jonathan M Davis