Parallel execution of unittests

[Andrei Alexandrescu via Digitalmars-d]

On 4/30/14, 11:31 PM, Jonathan M Davis via Digitalmars-d wrote:
> On Wed, 30 Apr 2014 22:32:33 -0700
> Andrei Alexandrescu via Digitalmars-d <digitalmars-d at puremagic.com>
> wrote:
>> There are a number of assumptions here: (a) most unittests that can
>> be effectively parallelized can be actually inferred (or declared) as
>> pure; (b) most unittests that cannot be inferred as pure are likely
>> to break; (c) it's a big deal if unittests break. I question all of
>> these assumptions. In particular I consider unittests that depend on
>> one another an effective antipattern that needs to be eradicated.
>
> Even if they don't depend on each other, they can depend on the system.

Understood, no need to repeat, thanks.

> std.file's unit tests will break if we parallelize them, because it
> operates on files and directories, and many of those tests operate on
> the same temp directories.

Yah, I remember even times when make unittest -j broke unittests because 
I've used the file name "deleteme" in multiple places. We need to fix those.

> That can be fixed by changing the tests, but
> it will break the tests.

I'm not too worried about breaking tests. I have in mind that we'll 
display a banner at the beginning of unittesting explaining that tests 
are ran in parallel and to force serial execution they'd need to set 
this thing or that. In a way I don't see it as "breakage" in the 
traditional tests. Unittests are in a way supposed to break :o).

> Other tests _can't_ be fixed if we force them
> to run in parallel. For instance, some of std.datetime's unit tests set
> the local time zone of the system in order to test that LocalTime works
> correctly.

Sure. We could specify that tests are to be run serially within one 
specific module, or to use classic interlocking in the unittest code. I 
see it as a problem relatively easy to address.

> We can guarantee that pure functions can safely be run in parallel. We
> _cannot_ guarantee that impure functions can safely be run in parallel.
> I'm sure that many impure unittest functions could be safely run in
> parallel, but it would require that the programmer verify that if we
> don't want undefined behavior - just like programmers have to verify
> that @system code is actually @safe. Simply running all unittest blocks
> in parallel is akin to considering @system code @safe in a particular
> piece of code simply because by convention that code should be @safe.

I don't think undefined behavior is at stake here, and I find the simile 
invalid. Thread isolation is a done deal in D and we may as well take 
advantage of it. Worse that could happen is that a unittest sets a 
global and surprisingly the next one doesn't "see" it.

At any rate I think it's pointless to insist on limiting parallel 
running to pure - let me just say I understood the point (thanks) so 
there is no need to restate it, and that I think it doesn't take us a 
good place.


Andrei