Parallel execution of unittests

[Xavier Bigand via Digitalmars-d]

Le 30/04/2014 19:58, Atila Neves a écrit :
> On Wednesday, 30 April 2014 at 17:50:34 UTC, Jonathan M Davis via
> Digitalmars-d wrote:
>> On Wed, 30 Apr 2014 08:59:42 -0700
>> Andrei Alexandrescu via Digitalmars-d <digitalmars-d at puremagic.com>
>> wrote:
>>
>>> On 4/30/14, 8:54 AM, bearophile wrote:
>>> > Andrei Alexandrescu:
>>> >
>>> >> A coworker mentioned the idea that unittests could be run in
>>> >> parallel
>>> >
>>> > In D we have strong purity to make more safe to run code in
>>> > parallel:
>>> >
>>> > pure unittest {}
>>>
>>> This doesn't follow. All unittests should be executable concurrently.
>>> -- Andrei
>>>
>>
>> In general, I agree. In reality, there are times when having state
>> across unit tests makes sense - especially when there's expensive setup
>> required for the tests. While it's not something that I generally
>> like to do, I know that we have instances of that where I work. Also, if
>> the unit tests have to deal with shared resources, they may very well be
>> theoretically independent but would run afoul of each other if run at
>> the same time - a prime example of this would be std.file, which has to
>> operate on the file system. I fully expect that if std.file's unit
>> tests were run in parallel, they would break. Unit tests involving
>> sockets would be another type of test which would be at high risk of
>> breaking, depending on what sockets they need.
>>
>> Honestly, the idea of running unit tests in parallel makes me very
>> nervous. In general, across modules, I'd expect it to work, but there
>> will be occasional cases where it will break. Across the unittest
>> blocks in a single module, I'd be _very_ worried about breakage. There
>> is nothing whatsoever in the language which guarantees that running
>> them in parallel will work or even makes sense. All that protects us is
>> the convention that unit tests are usually independent of each other,
>> and in my experience, it's common enough that they're not independent
>> that I think that blindly enabling parallelization of unit tests across
>> a single module is definitely a bad idea.
>>
>> - Jonathan M Davis
>
> You're right; blindly enabling parallelisation after the fact is likely
> to cause problems.
>
> Unit tests though, by definition (and I'm aware there are more than one)
> have to be independent. Have to not touch the filesystem, or the
> network. Only CPU and RAM. In my case, and since I had the luxury of
> implementing a framework first and only writing tests after it was done,
> running them in parallel was an extra check that they are in fact
> independent.
>
Why a test don't have to touch filesystem? That really restrictive, you 
just can't have a good code coverage on a lot libraries with a such 
restriction. I had work on a Source Control Management software, and all 
tests have to deal with a DB which requires file system and network 
operations.
IMO it's pretty much like impossible to miss testing of functions 
relations, simple integration tests are often needed to ensure that the 
application is working correctly. If D integrate features to support 
automatized testing maybe it must not be to restrictive mainly if 
everybody will expect more features commonly used (like named tests, 
formated result output,...).
Some of those common features have to be added to phobos instead of the 
language.

> Now, it does happen that you're testing code that isn't thread-safe
> itself, and yes, in that case you have to run them in a single thread.
> That's why I added the @SingleThreaded UDA to my library to enable that.
> As soon as I tried calling legacy C code...
>
> We could always make running in threads opt-in.
>
> Atila