Review of Andrei's std.benchmark

[Andrei Alexandrescu]

On 9/20/12 2:37 PM, Jacob Carlborg wrote:
> On 2012-09-20 14:36, Andrei Alexandrescu wrote:
>
>> Let's use the minimum. It is understood it's not what you'll see in
>> production, but it is an excellent proxy for indicative and reproducible
>> performance numbers.
>
> Why not min, max and average?

For a very simple reason: unless the algorithm under benchmark is very 
long-running, max is completely useless, and it ruins average as well.

For virtually all benchmarks I've run, the distribution of timings is a 
half-Gaussian very concentrated around the minimum. Say you have a 
minimum of e.g. 73 us. Then there would be a lot of results close to 
that; the mode of the distribution would be very close, e.g. 75 us, and 
the more measurements you take, the closer the mode is to the minimum. 
Then you have a few timings up to e.g. 90 us. And finally you will 
inevitably have a few outliers at some milliseconds. Those are orders of 
magnitude larger than anything of interest and are caused by system 
interrupts that happened to fall in the middle of the measurement.

Taking those into consideration and computing the average with those 
outliers simply brings useless noise into the measurement process.


Andrei