std.variant benchmark

[Andrei Alexandrescu]

On 7/29/12 11:20 AM, Gor Gyolchanyan wrote:
> On Sun, Jul 29, 2012 at 6:43 PM, Dmitry Olshansky <dmitry.olsh at gmail.com
> <mailto:dmitry.olsh at gmail.com>> wrote:
>     This should be more relevant then:
>
>     //fib.d
>     import std.datetime, std.stdio, std.variant;
>
>     auto fib(Int)()
>     {
>              Int a = 1, b = 1;
>              for(size_t i=0; i<100; i++){
>                      Int c = a + b;
>                      a = b;
>                      b = c;
>              }
>              return a;
>     }
>
>     void main()
>     {
>              writeln(benchmark!(fib!int, fib!long, fib!Variant)(10_000));
>     }
>
>
>     dmd -O -inline -release fib.d
>
>     Output:
>
>     [TickDuration(197), TickDuration(276), TickDuration(93370107)]
>
>     I'm horrified. Who was working on std.variant enhancements? Please
>     chime in.
>
>     --
>     Dmitry Olshansky
>
>
> Thank you for demonstrating my point. :-)

I don't think he demonstrated your point (even leaving aside that the 
benchmark above is also flawed). I don't think there was a point, unless 
it was you wanted to vent and looked for a pretext - which is, by the 
way, entirely reasonable.

You mentioned you need "a very fast typeless storage with maximum 
performance and type safety." Well if it's typeless then it means you're 
using it for storage, not for computationally-intensive operations, as 
the code above does. So what you'd presumably want to test is the speed 
of data storage and retrieval. A loop that does a bunch of adds on 
Variant does not go in that direction.

Benchmarks are notoriously hard to get right. You need to think of 
reasonable baselines - if you want to use Variant for typeless storage, 
what is your vanilla implementation, the "obvious contender"? What are 
the primitives of which speed is important? Then you need to make sure 
you subtract noise from all your comparisons. Then you need to figure 
whether the issue is with the design or with the implementation of 
Variant. In the former case, maybe Variant isn't for you. In the latter, 
bug reports are always welcome.


Andrei