Parallel execution of unittests

[Atila Neves via Digitalmars-d]

On Thursday, 1 May 2014 at 18:38:15 UTC, w0rp wrote:
> On Thursday, 1 May 2014 at 17:04:53 UTC, Xavier Bigand wrote:
>> Le 01/05/2014 16:01, Atila Neves a écrit :
>>> On Thursday, 1 May 2014 at 11:44:12 UTC, w0rp wrote:
>>>> On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg 
>>>> wrote:
>>>>> On 2014-04-30 23:35, Andrei Alexandrescu wrote:
>>>>>
>>>>>> Agreed. I think we should look into parallelizing all 
>>>>>> unittests. --
>>>>>> Andrei
>>>>>
>>>>> I recommend running the tests in random order as well.
>>>>
>>>> This is a bad idea. Tests could fail only some of the time. 
>>>> Even if
>>>> bugs are missed, I would prefer it if tests did exactly the 
>>>> same thing
>>>> every time.
>>>
>>> They _should_ do exactly the same thing every time. Which is 
>>> why running
>>> in threads or at random is a great way to enforce that.
>>>
>>> Atila
>> +1
>
> Tests shouldn't be run in a random order all of the time, 
> perhaps once in a while, manually. Having continuous 
> integration randomly report build failures is crap. Either you 
> should always see a build failure, or you shouldn't see it. You 
> can only test things which are deterministic, at least as far 
> as what you observe. Running tests in a random order should be 
> something you do manually, only when you have some ability to 
> figure out why the tests just failed.

In my experience when a test fails randomly because of ordering, 
a while loop on the shell running until failure is enough to 
reproduce it in a few seconds. But as others have mentioned, 
being able to use a seed to reproduce it exactly is superior.

Atila