Review of Andrei's std.benchmark

[Jacob Carlborg]

On 2012-09-21 06:23, Andrei Alexandrescu wrote:

> For a very simple reason: unless the algorithm under benchmark is very
> long-running, max is completely useless, and it ruins average as well.

I may have completely misunderstood this but aren't we talking about 
what do include in the output of the benchmark? In that case, if you 
don't like max and average just don't look at it.

> For virtually all benchmarks I've run, the distribution of timings is a
> half-Gaussian very concentrated around the minimum. Say you have a
> minimum of e.g. 73 us. Then there would be a lot of results close to
> that; the mode of the distribution would be very close, e.g. 75 us, and
> the more measurements you take, the closer the mode is to the minimum.
> Then you have a few timings up to e.g. 90 us. And finally you will
> inevitably have a few outliers at some milliseconds. Those are orders of
> magnitude larger than anything of interest and are caused by system
> interrupts that happened to fall in the middle of the measurement.
>
> Taking those into consideration and computing the average with those
> outliers simply brings useless noise into the measurement process.

After your replay to one of Manu's post, I think I misunderstood the 
std.benchmark module. I was thinking more of profiling. But are these 
quite similar tasks, couldn't std.benchmark work for both?

-- 
/Jacob Carlborg