avgtime - Small D util for your everyday benchmarking needs

[Don Clugston]

On 23/03/12 16:25, Andrei Alexandrescu wrote:
> On 3/23/12 12:51 AM, Manfred Nowak wrote:
>> Andrei Alexandrescu wrote:
>>
>>> You may want to also print the mode of the distribution,
>>> nontrivial but informative
>>
>> In case of this implementation and according to the given link: trivial
>> and noninformative, because
>>
>> | For samples, if it is known that they are drawn from a symmetric
>> | distribution, the sample mean can be used as an estimate of the
>> | population mode.
>>
>> and the program computes the variance as if the values of the sample
>> follow a normal distribution, which is symmetric.
>>
>> Therefore the mode of the sample is of interest only, when the variance
>> is calculated wrongly.
>
> Again, benchmarks I've seen are always asymmetric. Not sure why those
> shown here are symmetric. The mode should be very close to the minimum
> (and in fact I think taking the minimum is a pretty good approximation
> of the sought-after time).
>
> Andrei

Agreed, I think situations where you would get a normal distribution are 
rare in benchmarking code.
Small sections of code always have a best-case scenario, where there are 
no cache misses.
If there are task switches, the best case is zero task switches.

If you use the CPU performance counters, you can identify the *cause* of 
performance variations. When I've done this, I've always been able to 
get very stable numbers