std.benchmark ready for review. Manager sought after

[Andrei Alexandrescu]

On 4/8/12 11:59 AM, Denis Shelomovskij wrote:
> Very good but minimum isn't a best guess. Personally I (and there will
> be a lot of such maniacs I suppose) will think that this (minimum) time
> can be significantly smaller than average time.

I've analyzed this quite a bit at work and the average and median are 
not very informative. You need the mode, but in most benchmarks the mode 
is very close to the minimum, so using the minimum is even better.

In speed measurements, all noise is additive (there's no noise that may 
make a benchmark appear to run faster). There are also quite a few 
outliers. Recording the average will include a fair amount of noise.

Clearly there is noise during normal use as well, but incorporating it 
in benchmarks as a matter of course reduces the usefulness of benchmarks 
as a mean to improve performance of the benchmarked code.

> So a parameter (probably with a default value) should be added.
> Something like enum of flags telling what we want to know. At least
> these looks usable: minTime, <some mean time>, maxTime,
> standardDeviation, graph (yes, good old ASCII art).

Standard deviation is also not very useful because it includes all 
outliers (some of which are very far away from the mode).

> Yes, graph is needed.

I am not sure about that. We may provide the raw measurement data for 
programs that want to plot things, but plotting is beyond the charter of 
std.benchmark.


Andrei