Parallel execution of unittests

[Steven Schveighoffer via Digitalmars-d]

On Wed, 30 Apr 2014 13:50:10 -0400, Jonathan M Davis via Digitalmars-d  
<digitalmars-d at puremagic.com> wrote:

> On Wed, 30 Apr 2014 08:59:42 -0700
> Andrei Alexandrescu via Digitalmars-d <digitalmars-d at puremagic.com>
> wrote:
>
>> On 4/30/14, 8:54 AM, bearophile wrote:
>> > Andrei Alexandrescu:
>> >
>> >> A coworker mentioned the idea that unittests could be run in
>> >> parallel
>> >
>> > In D we have strong purity to make more safe to run code in
>> > parallel:
>> >
>> > pure unittest {}
>>
>> This doesn't follow. All unittests should be executable concurrently.
>> -- Andrei
>>
>
> In general, I agree. In reality, there are times when having state
> across unit tests makes sense - especially when there's expensive setup
> required for the tests.

int a;
unittest
{
    // set up a;
}

unittest
{
    // use a;
}

==>

unittest
{
    int a;
    {
       // set up a;
    }
    {
       // use a;
    }
}

It makes no sense to do it the first way, you are not gaining anything.

> Honestly, the idea of running unit tests in parallel makes me very
> nervous. In general, across modules, I'd expect it to work, but there
> will be occasional cases where it will break.

Then you didn't write your unit-tests correctly. True unit tests-anyway.

In fact, the very quality that makes unit tests so valuable (that they are  
independent of other code) is ruined by sharing state across tests. If you  
are going to share state, it really is one unit test.

> Across the unittest
> blocks in a single module, I'd be _very_ worried about breakage. There
> is nothing whatsoever in the language which guarantees that running
> them in parallel will work or even makes sense. All that protects us is
> the convention that unit tests are usually independent of each other,
> and in my experience, it's common enough that they're not independent
> that I think that blindly enabling parallelization of unit tests across
> a single module is definitely a bad idea.

I think that if we add the assumption, the resulting fallout would be easy  
to fix.

Note that we can't require unit tests to be pure -- non-pure functions  
need testing too :)

I can imagine that even if you could only parallelize 90% of unit tests,  
that would be an effective optimization for a large project. In such a  
case, the rare (and I mean rare to the point of I can't think of a single  
use-case) need to deny parallelization could be marked.

-Steve