std.benchmark is in reviewable state

[Andrei Alexandrescu]

On 9/25/11 9:02 PM, Robert Jacques wrote:
> On Sun, 25 Sep 2011 21:08:45 -0400, Andrei Alexandrescu
> <SeeWebsiteForEmail at erdani.org> wrote:
>> I've had a good time with std.benchmark today and made it ready for
>> submission to Phobos. As a perk I've added the %h format specifier,
>> which formats a number in human-readable form using SI prefixes.
>>
>> For those interested in discussing this prior to the formal review
>> process, the code is at:
>>
>> https://github.com/andralex/phobos/blob/benchmark/std/benchmark.d
>> https://github.com/andralex/phobos/blob/benchmark/std/format.d
>>
>> and the dox at
>>
>> http://erdani.com/d/web/phobos-prerelease/std_benchmark.html
>> http://erdani.com/d/web/phobos-prerelease/std_format.html
>>
>> Comments and suggestions are welcome. If possible, it would be great if
>> this submission were given a bump in the priority queue. This is because
>> with robust benchmarking in Phobos we can ask new, performance-sensitive
>> submissions, to add benchmarking.
>>
>>
>> Thanks,
>>
>> Andrei
>
> Andrei, good job on the framework design, but you've left all of the
> design flaws of the current benchmark routine. To quote myself from last
> Tuesday:
>
>>> On Tue, 20 Sep 2011 14:01:05 -0400, Timon Gehr <timon.gehr at gmx.ch>
>>> wrote:
>> [snip]
>>
>>> Thank you for making this more meaningful! I assumed the standard
>>> library benchmark function would take care of those things. Should it?
>> Yes and no. Benchmark provides a good way to make a single measurement of
>> a function, as for really short functions you do have to loop many
>> timesto be able to get a reliable reading. However, actual
>> benchmarking requiresa) tuning the benchmark() call time to about
>> 10-20 ms and b) runningbenchmark() many times, taking the minimum. The
>> idea is that on any givenrun you could hit a context switch, etc. so
>> if you make multiple run, onewill get lucky and not be interrupted.
>> Worse, if a core switch happensbetween StopWatch start and end, the
>> number of ticks returned is random.Hence, the comment to manually
>> limit execution to a single core. So, itmight be nice if benchmark()
>> also took a second parameter, denoting thenumber of times to benchmark
>> the function and had some better documentationon how to do good
>> benchmarking.
>
> Wikipedia also lists some of the challenges to modern benchmarking:
> http://en.wikipedia.org/wiki/Time_Stamp_Counter
>
> So, std.benchmark should
> [ ] Set the affinity to a single thread at start (i.e. use
> SetProcessAffinityMask, etc)

I think that won't influence most measurements measurably.

> [ ] Repeat the measurement N times, taking the min.

std.benchmark repeats the measurement in a loop, discounts the times 
spent in the iteration proper, and divides total time by the number of 
iterations to figure the time per iteration. This has the advantage that 
works with even very fast functions without letting the counter itself 
affect the timing.

> [X] Adaptively increase the number of loop iterations to ensure a valid
> reading.

The current system tries 10, 100, 1000 iterations until it gets to a 
total run time of at least 0.5 seconds. That's quite a generous margin, 
I plan to tighten it.

> [ ] Adaptively decrease the number of loop iterations to ensure minimal
> context switching.

OK.


Andrei