Review of Andrei's std.benchmark

[Andrei Alexandrescu]

On 9/18/12 5:07 PM, "Øivind" wrote:
> I think the std.benchmark is definitely a useful library addition, but
> in my mind it currently a bit too limited.
>
> * All tests are run 1000 times. Depending on the length of the test to
> benchmark, this can be too much. In some cases it would be good to be
> able to trade the number of runs against accuracy.

It would be a good idea to make that a configurable parameter.

> * For all tests, the best run is selected, but would it not be
> reasonable in some cases to get the average value? Maybe excluding the
> runs that are more than a couple std. deviations away from the mean value..

After extensive tests with a variety of aggregate functions, I can say 
firmly that taking the minimum time is by far the best when it comes to 
assessing the speed of a function.

> * Is there a way of specifying a test name other than the function-name
> when using the 'mixin(scheduleForBenchmarking)' approach to register
> benchmarks?

Not currently. Probably a manual registration of an individual benchmark 
would make sense.

> * I would also like to be able (if possible) to register two mentioned
> things (number of runs and result strategy) with the mixin approach (or
> similar).

Makes sense.

> * It seems like the baseline for subtraction from subsequent test runs
> is taken from a call to the test function, passing 1 to it. Shouldn't 0
> be passed for this value?

I'll look into that.


Thanks,

Andrei