Language performance benchmark be updated 2019/11/09

[Jacob Shtokolov]

On Friday, 15 November 2019 at 03:31:24 UTC, zoujiaqing wrote:
> https://github.com/kostya/benchmarks

Sorry, but have you tried it by yourself?

I'm running this benchmark outside of Docker, and the numbers I 
see are very interesting. First of all, the time measurement 
script is quite questionable. It's a Ruby script, and I found 
that the running time depends on the shell and environment. For 
example, when I run it under VSCode's console window, I get 20% 
worse results in time for all binaries than when I run it under 
the regular terminal emulator window.

Second, the numbers are, hmmm, how to say... bullshit? Pardon my 
French!

Here is what I get for the Brainfuck2 mandelbrot benchmark (a 
simple Brainfuck interpreter implemented in different languages):

C++ gcc version 7.4.0 (g++ -flto -O3 -o bin_cpp bf.cpp):
```
$ ../xtime.rb ./bin_cpp mandel.b
# ... here is the Mandelbrot set ASCII art printed ...
18.05s, 3.6Mb
```

D LDC2 1.18.0 (ldc2 -ofbin_d_ldc -O5 -release -boundscheck=off 
bf.d)
```
$ ../xtime.rb ./bin_d_ldc mandel.b
# ... here is the Mandelbrot set ASCII art printed ...
19.53s, 3.6Mb
```

Nim 1.0.2 (nim c -o:bin_nim_gcc -d:danger --cc:gcc --verbosity:0 
bf.nim)
```
$ ../xtime.rb ./bin_nim_gcc mandel.b
# ... here is the Mandelbrot set ASCII art printed ...
25.07s, 2.2Mb
```

Kotlin kotlinc-jvm 1.3.50 (JRE 1.8.0_201-b09) (kotlinc bf2.kt 
-include-runtime -d bf2-kt.jar)
```
$ ../xtime.rb java -jar bf2-kt.jar mandel.b
JIT warming up
time: 1.25s
run
# ... here is the Mandelbrot set ASCII art printed ...
26.81s, 36.6Mb
```

Golang go1.13.4 linux/amd64 (go build -o bin_go bf.go)
```
$ ../xtime.rb ./bin_go mandel.b
# ... here is the Mandelbrot set ASCII art printed ...
38.73s, 2.9Mb
```

============================================

So the results for Braunfuck2 mandel.b are:

```
C++ gcc: 18.05s, 3.6Mb
D LDC2:  19.53s, 3.6Mb
Nim:     25.07s, 2.2Mb
Kotlin:  26.81s, 36.6Mb
```

Please note that I've added `boundscheck=off` to LDC2 command 
line there.
Also, Kotlin is always printing `JIT warming up` and takes about 
1 to 2 seconds to warm up, so the results for Brainfuck2 
`bench.b` are VERY different. Kotlin is not the first one 
obviously.

I'm running everything on my laptop with Intel(R) Core(TM) 
i7-8550U CPU @ 1.80GHz
As I mentioned, no Docker containers, just using the included 
scripts to build the binaries.

Haven't tried other tests, but I feel like I'll get very 
interesting results for them also.

Would be great if someone else is able to confirm these results 
because these benchmarks look very manipulative.