Parallel execution of unittests

[Atila Neves via Digitalmars-d]

On Wednesday, 30 April 2014 at 17:50:34 UTC, Jonathan M Davis via 
Digitalmars-d wrote:
> On Wed, 30 Apr 2014 08:59:42 -0700
> Andrei Alexandrescu via Digitalmars-d 
> <digitalmars-d at puremagic.com>
> wrote:
>
>> On 4/30/14, 8:54 AM, bearophile wrote:
>> > Andrei Alexandrescu:
>> >
>> >> A coworker mentioned the idea that unittests could be run in
>> >> parallel
>> >
>> > In D we have strong purity to make more safe to run code in
>> > parallel:
>> >
>> > pure unittest {}
>> 
>> This doesn't follow. All unittests should be executable 
>> concurrently.
>> -- Andrei
>> 
>
> In general, I agree. In reality, there are times when having 
> state
> across unit tests makes sense - especially when there's 
> expensive setup
> required for the tests. While it's not something that I 
> generally
> like to do, I know that we have instances of that where I work. 
> Also, if
> the unit tests have to deal with shared resources, they may 
> very well be
> theoretically independent but would run afoul of each other if 
> run at
> the same time - a prime example of this would be std.file, 
> which has to
> operate on the file system. I fully expect that if std.file's 
> unit
> tests were run in parallel, they would break. Unit tests 
> involving
> sockets would be another type of test which would be at high 
> risk of
> breaking, depending on what sockets they need.
>
> Honestly, the idea of running unit tests in parallel makes me 
> very
> nervous. In general, across modules, I'd expect it to work, but 
> there
> will be occasional cases where it will break. Across the 
> unittest
> blocks in a single module, I'd be _very_ worried about 
> breakage. There
> is nothing whatsoever in the language which guarantees that 
> running
> them in parallel will work or even makes sense. All that 
> protects us is
> the convention that unit tests are usually independent of each 
> other,
> and in my experience, it's common enough that they're not 
> independent
> that I think that blindly enabling parallelization of unit 
> tests across
> a single module is definitely a bad idea.
>
> - Jonathan M Davis

You're right; blindly enabling parallelisation after the fact is 
likely to cause problems.

Unit tests though, by definition (and I'm aware there are more 
than one) have to be independent. Have to not touch the 
filesystem, or the network. Only CPU and RAM. In my case, and 
since I had the luxury of implementing a framework first and only 
writing tests after it was done, running them in parallel was an 
extra check that they are in fact independent.

Now, it does happen that you're testing code that isn't 
thread-safe itself, and yes, in that case you have to run them in 
a single thread. That's why I added the @SingleThreaded UDA to my 
library to enable that. As soon as I tried calling legacy C 
code...

We could always make running in threads opt-in.

Atila