Running Phobos unit tests in threads: I have data

[Atila Neves via Digitalmars-d]

So I tried using unit-threaded to run Phobos unit tests again and 
had problems (which I'll look into later) with its compile-time 
reflection. Then I realised I was an idiot since I don't need to 
reflect on anything: all Phobos tests are in unittest blocks so 
all I need to do is include them in the build and unit-threaded 
will run them for me.

I tried a basic sanity check by running them in one thread only 
with the -s option and got a segfault, and a failing test before 
that. None of this should happen, and I'll be taking a look at 
that as well.

But I carried on by removing the troublesome modules from the 
build. These turned out to be:

std.datetime (fails)
std.process (fails and causes the segfault)
std.stdio (fails)

All the others pass in single threaded mode. After this I tried 
using threads and std.parallelism failed, so I took that away 
from the build as well.

Another thing to mention is that although the tests are running 
in threads, since when I wrote the library the getUnitTests 
__traits wasn't available (and since then I wasn't interested in 
using it), each module's unit tests run as one test. So they only 
interleave with other modules, not with each other.

Running in one thread took 39 +/- 1 seconds.
Running in 8 threads took... ~41 seconds.

Oops. I noticed some tests take a lot longer so I tried removing 
those. They were:

std.file
std.conv
std.regex
std.random
std.container
std.xml
std.utf
std.numeric
std.uuid
std.exception

I also removed any modules that were likely to be problematic 
like std.concurrency and std.socket. With the reduced sample size 
the results were:

1 thread: ~1.9s
8 threads: 4.1s +/- 0.2

So the whole threading thing isn't looking so great. Or at least 
not how I implemented it. This got me thinking about my own 
projects. The tests run so fast I never really paid attention to 
how fast they were running. I compared running the unit tests in 
Cerealed in one or more threads and got the same result: running 
in one thread was faster.

I have to look to be sure but maybe the bottleneck is output. As 
in actually printing the results to the screen. I had to jump 
through a few hoops to make sure the output wasn't interleaved, 
and in the end decided to have one thread be responsible for 
that, with the tests sending it output messages.

For reference, I copied all of the std/*.d modules into a local 
std directory, compiled all of them with dmd -unittest -c, then 
used this as the build command:

dmd -unittest -I~/coding/d/unit-threaded/source ut.d 
std/algorithm.o std/array.o std/ascii.o std/base64.o std/bigint.o 
std/bitmanip.o std/compiler.o std/complex.o std/container.o 
std/cstream.o std/csv.o std/demangle.o std/encoding.o 
std/format.o std/functional.o std/getopt.o std/json.o std/math.o 
std/mathspecial.o std/metastrings.o std/mmfile.o std/numeric.o 
std/outbuffer.o std/range.o  std/signals.o  std/stdint.o 
std/stdiobase.o std/stream.o std/string.o std/syserror.o 
std/system.o std/traits.o std/typecons.o std/typelist.o 
std/typetuple.o std/uri.o std/variant.o std/zip.o std/zlib.o  
libunit-threaded.a -ofphobos_ut

I got libunit-threaded.a by running "dub build" in the root 
directory of unit-threaded.

I might just implement a random order option now. Hmm.

Atila