D vs Go in real life, part 2. Also, Erlang.

[qznc]

On Wednesday, 4 December 2013 at 12:49:13 UTC, Atila Neves wrote:
> It all started here:
>
> http://forum.dlang.org/thread/lmkvmfpysiokpqgsynar@forum.dlang.org
>
> Not wanting to be outdone, my buddy Jeff (the Go guy) went and 
> wrote a new benchmark. My implementation didn't do as well on 
> that one so I went back to the code and optimised it. Also, our 
> other buddy Patrick from work decided to get in on this and use 
> it as an excuse to learn Erlang. So now 4 different 
> implementations in 4 different languages are duking it out.
>
> There are two benchmarks, both written in Go. The first is 
> loadtest and it measures throughput. I varied the number of 
> client connections to the server and measured how many messages 
> were sent per second.
>
> The second benchmark is pingtest and it measures latency. The 
> quantity I varied here was the number of wildcard subscribers 
> (it'd take a while to explain what they are to anyone who isn't 
> familiar with MQTT). It has a bunch of client connections talk 
> to each other, and instead of trying to send as many messages 
> as computerly possible they wait for the acknowledgement from 
> the partner connection. The results are as follows (I tried 
> making charts but LibreOffice wasn't helping and then I got 
> bored), with the numbers being thousands of messages per second:
>
> loadtest
> Connections:   100            500            750            1k
> D + vibe.d:    121.7 +/- 1.5  166.9 +/- 1.5  171.1 +/- 3.3  
> 167.9 +/- 1.3
> C (mosquitto): 106.1 +/- 0.8  122.4 +/- 0.4   95.2 +/- 1.3   
> 74.7 +/- 0.4
> Erlang:        104.1 +/- 2.2  124.2 +/- 5.9  117.6 +/- 4.6  
> 117.7 +/- 3.2
> Go:             90.9 +/- 11   100.1 +/- 0.1   99.3 +/- 0.2   
> 98.8 +/- 0.3
>
> pingtest
> wsubs:         20             200            400
> D + vibe.d:    50.9 +/- 0.3   38.3 +/- 0.2   20.1 +/- 0.1
> C (mosquitto): 65.4 +/- 4.4   45.2 +/- 0.2   20.0 +/- 0.0
> Erlang:        49.1 +/- 0.8   30.9 +/- 0.3   15.6 +/- 0.1
> Go:            45.2 +/- 0.2   27.5 +/- 0.1   16.0 +/- 0.1
>
> So, D was faster than the other contenders by far in 
> throughput, 2nd place losing to the C implementation on 
> latency. I'm still not sure why that is. Profiling in this case 
> is tricky. I'm pretty sure the profiler is still ticking away 
> when a fiber yields - the top function is the one that reads 
> from the network, which I can't do much about.
>
> GC was pretty much a non-issue, which I'm both surprised at and 
> happy about. I allocated as much memory as I wanted at first, 
> and by the time I tried to optmise memory usage by allocating 
> less all I managed to eke out in performance terms was a paltry 
> extra 8% or so.
>
> All in all though, so much for his "nothing is going to be 
> faster than Go because this is what it's good at". If he'd 
> measured it against mosquitto first and realised it was already 
> losing to C I probably wouldn't have written an MQTT broker in 
> D, so I guess that was a good thing. :)
>
> Atila
>
> P.S. vibe.d is awesome, although I wish it'd compile with ldc 
> or gdc

Nice benchmark! :)

So, a DMD-compiled D program vs GCC-compiled (?) C program 
fighting for latency? That might be a reason. Interesting we 
still win with throughput.

The variance/standard deviation of pingtest-C-20 with 4.4 stands 
out. What happens there?