Running Phobos unit tests in threads: I have data

[Atila Neves via Digitalmars-d]

I turned off all output to check. It was still slower with 
multiple threads. That was the only "weird" thing I was doing I 
could think of as the cause. Otherwise it's just a foreach(test; 
tests.parallel) { test(); }.

Atila

On Saturday, 3 May 2014 at 11:54:55 UTC, Atila Neves wrote:
> So I tried using unit-threaded to run Phobos unit tests again 
> and had problems (which I'll look into later) with its 
> compile-time reflection. Then I realised I was an idiot since I 
> don't need to reflect on anything: all Phobos tests are in 
> unittest blocks so all I need to do is include them in the 
> build and unit-threaded will run them for me.
>
> I tried a basic sanity check by running them in one thread only 
> with the -s option and got a segfault, and a failing test 
> before that. None of this should happen, and I'll be taking a 
> look at that as well.
>
> But I carried on by removing the troublesome modules from the 
> build. These turned out to be:
>
> std.datetime (fails)
> std.process (fails and causes the segfault)
> std.stdio (fails)
>
> All the others pass in single threaded mode. After this I tried 
> using threads and std.parallelism failed, so I took that away 
> from the build as well.
>
> Another thing to mention is that although the tests are running 
> in threads, since when I wrote the library the getUnitTests 
> __traits wasn't available (and since then I wasn't interested 
> in using it), each module's unit tests run as one test. So they 
> only interleave with other modules, not with each other.
>
> Running in one thread took 39 +/- 1 seconds.
> Running in 8 threads took... ~41 seconds.
>
> Oops. I noticed some tests take a lot longer so I tried 
> removing those. They were:
>
> std.file
> std.conv
> std.regex
> std.random
> std.container
> std.xml
> std.utf
> std.numeric
> std.uuid
> std.exception
>
> I also removed any modules that were likely to be problematic 
> like std.concurrency and std.socket. With the reduced sample 
> size the results were:
>
> 1 thread: ~1.9s
> 8 threads: 4.1s +/- 0.2
>
> So the whole threading thing isn't looking so great. Or at 
> least not how I implemented it. This got me thinking about my 
> own projects. The tests run so fast I never really paid 
> attention to how fast they were running. I compared running the 
> unit tests in Cerealed in one or more threads and got the same 
> result: running in one thread was faster.
>
> I have to look to be sure but maybe the bottleneck is output. 
> As in actually printing the results to the screen. I had to 
> jump through a few hoops to make sure the output wasn't 
> interleaved, and in the end decided to have one thread be 
> responsible for that, with the tests sending it output messages.
>
> For reference, I copied all of the std/*.d modules into a local 
> std directory, compiled all of them with dmd -unittest -c, then 
> used this as the build command:
>
> dmd -unittest -I~/coding/d/unit-threaded/source ut.d 
> std/algorithm.o std/array.o std/ascii.o std/base64.o 
> std/bigint.o std/bitmanip.o std/compiler.o std/complex.o 
> std/container.o std/cstream.o std/csv.o std/demangle.o 
> std/encoding.o std/format.o std/functional.o std/getopt.o 
> std/json.o std/math.o std/mathspecial.o std/metastrings.o 
> std/mmfile.o std/numeric.o std/outbuffer.o std/range.o  
> std/signals.o  std/stdint.o std/stdiobase.o std/stream.o 
> std/string.o std/syserror.o std/system.o std/traits.o 
> std/typecons.o std/typelist.o std/typetuple.o std/uri.o 
> std/variant.o std/zip.o std/zlib.o  libunit-threaded.a 
> -ofphobos_ut
>
> I got libunit-threaded.a by running "dub build" in the root 
> directory of unit-threaded.
>
> I might just implement a random order option now. Hmm.
>
> Atila