Running Phobos unit tests in threads: I have data

[Rikki Cattermole via Digitalmars-d]

On Saturday, 3 May 2014 at 12:08:56 UTC, Atila Neves wrote:
> I turned off all output to check. It was still slower with 
> multiple threads. That was the only "weird" thing I was doing I 
> could think of as the cause. Otherwise it's just a 
> foreach(test; tests.parallel) { test(); }.
>
> Atila
>
> On Saturday, 3 May 2014 at 11:54:55 UTC, Atila Neves wrote:
>> So I tried using unit-threaded to run Phobos unit tests again 
>> and had problems (which I'll look into later) with its 
>> compile-time reflection. Then I realised I was an idiot since 
>> I don't need to reflect on anything: all Phobos tests are in 
>> unittest blocks so all I need to do is include them in the 
>> build and unit-threaded will run them for me.
>>
>> I tried a basic sanity check by running them in one thread 
>> only with the -s option and got a segfault, and a failing test 
>> before that. None of this should happen, and I'll be taking a 
>> look at that as well.
>>
>> But I carried on by removing the troublesome modules from the 
>> build. These turned out to be:
>>
>> std.datetime (fails)
>> std.process (fails and causes the segfault)
>> std.stdio (fails)
>>
>> All the others pass in single threaded mode. After this I 
>> tried using threads and std.parallelism failed, so I took that 
>> away from the build as well.
>>
>> Another thing to mention is that although the tests are 
>> running in threads, since when I wrote the library the 
>> getUnitTests __traits wasn't available (and since then I 
>> wasn't interested in using it), each module's unit tests run 
>> as one test. So they only interleave with other modules, not 
>> with each other.
>>
>> Running in one thread took 39 +/- 1 seconds.
>> Running in 8 threads took... ~41 seconds.
>>
>> Oops. I noticed some tests take a lot longer so I tried 
>> removing those. They were:
>>
>> std.file
>> std.conv
>> std.regex
>> std.random
>> std.container
>> std.xml
>> std.utf
>> std.numeric
>> std.uuid
>> std.exception
>>
>> I also removed any modules that were likely to be problematic 
>> like std.concurrency and std.socket. With the reduced sample 
>> size the results were:
>>
>> 1 thread: ~1.9s
>> 8 threads: 4.1s +/- 0.2
>>
>> So the whole threading thing isn't looking so great. Or at 
>> least not how I implemented it. This got me thinking about my 
>> own projects. The tests run so fast I never really paid 
>> attention to how fast they were running. I compared running 
>> the unit tests in Cerealed in one or more threads and got the 
>> same result: running in one thread was faster.
>>
>> I have to look to be sure but maybe the bottleneck is output. 
>> As in actually printing the results to the screen. I had to 
>> jump through a few hoops to make sure the output wasn't 
>> interleaved, and in the end decided to have one thread be 
>> responsible for that, with the tests sending it output 
>> messages.
>>
>> For reference, I copied all of the std/*.d modules into a 
>> local std directory, compiled all of them with dmd -unittest 
>> -c, then used this as the build command:
>>
>> dmd -unittest -I~/coding/d/unit-threaded/source ut.d 
>> std/algorithm.o std/array.o std/ascii.o std/base64.o 
>> std/bigint.o std/bitmanip.o std/compiler.o std/complex.o 
>> std/container.o std/cstream.o std/csv.o std/demangle.o 
>> std/encoding.o std/format.o std/functional.o std/getopt.o 
>> std/json.o std/math.o std/mathspecial.o std/metastrings.o 
>> std/mmfile.o std/numeric.o std/outbuffer.o std/range.o  
>> std/signals.o  std/stdint.o std/stdiobase.o std/stream.o 
>> std/string.o std/syserror.o std/system.o std/traits.o 
>> std/typecons.o std/typelist.o std/typetuple.o std/uri.o 
>> std/variant.o std/zip.o std/zlib.o  libunit-threaded.a 
>> -ofphobos_ut
>>
>> I got libunit-threaded.a by running "dub build" in the root 
>> directory of unit-threaded.
>>
>> I might just implement a random order option now. Hmm.
>>
>> Atila

Out of curiosity are you on Windows?