std.benchmark ready for review. Manager sought after

[Andrei Alexandrescu]

On 4/8/12 3:03 PM, Manfred Nowak wrote:
> Andrei Alexandrescu wrote:
>
>> Clearly there is noise during normal use as well, but
>> incorporating it in benchmarks as a matter of course reduces the
>> usefulness of benchmarks
>
> On the contrary:
> 1) The "noise during normal use" has to be measured in order to detect
> the sensibility of the benchmarked program to that noise.

That sounds quite tenuous to me. How do you measure it, and what 
conclusions do you draw other than there's a more or less other stuff 
going on on the machine, and the machine itself has complex interactions?

Far as I can tell a time measurement result is:

T = A + Q + N

where:

A > 0 is actual benchmark time

Q > 0 quantization noise (uniform distribution)

N > 0 various other noises (interrupts, task switching, networking, CPU 
dynamically changing frequency, etc). Many people jump on Gaussian as an 
approximation, but my tests suggest it's hardly so because it has a lot 
of jerky outliers.

How do we estimate A given T?

> 2) The noise the benchmarked program produces has to be measured too,
> because the running benchmarked program probably increases the noise
> for all other running programs.

How to measure that? Also, that noise does not need to be measured as 
much as eliminated to the extent possible. This is because the benchmark 
app noise is a poor model of the application-induced noise.

> In addition: the noise produced by a machine under heavy load might
> bring the performance of the benchmarked program down to zero.

Of course. That's why the documentation emphasizes the necessity of 
baselines. A measurement without baselines is irrelevant.


Andrei