std.benchmark ready for review. Manager sought after

[Andrei Alexandrescu]

On 4/9/12 9:25 AM, Manfred Nowak wrote:
> Andrei Alexandrescu wrote:
>> all noise is additive (there's no noise that may make a benchmark
>> appear to run faster)
>
> This is in doubt, because you yourself wrote "the machine itself has
> complex interactions". This complex interactions might lower the time
> needed for an operation of the benchmarked program.
>
> Examples that come to mind:
> a) needed data is already in a (faster) cache because it belongs to a
> memory block, from which some data is needed by some program not
> belonging to the benchmarked set---and that block isnt replaced yet.

Which is great, unless the program wants to measure the cache memory 
itself, in which case it would use special assembler instructions or 
large memset()s. (We do such at Facebook.)

> b) needed data is stored in a hdd whose I/O scheduler uses the elevator
> algorithm and serves the request by pure chance instantly, because the
> position of the needed data is between two positions accessed by some
> programs not belonging to the benchmarked set.
>
> Especially a hdd, if used, will be responsible for a lot of noise you
> define as "quantization noise (uniform distribution)" even if the head
> stays at the same cylinder. Not recognizing this noise would only mean
> that the data is cached and interpreting the only true read from the
> hdd as a jerky outlier sems quite wrong.

If the goal is to measure the seek time of the HDD, the benchmark itself 
should make sure the HDD cache is cleared. (What I recall they do on 
Linux is unmounting and remounting the drive.) Otherwise, it adds a 
useless component to the timing.

>>> 1) The "noise during normal use" has to be measured in order to
>>> detect the sensibility of the benchmarked program to that noise.
>> How do you measure it, and what
>> conclusions do you draw other than there's a more or less other
>> stuff going on on the machine, and the machine itself has complex
>> interactions?
>>
>> Far as I can tell a time measurement result is:
>>
>> T = A + Q + N
>
> For example by running more than one instance of the benchmarked
> program in paralell and use the thereby gathered statistical routines
> to spread T into the additiv components A, Q and N.

I disagree with running two benchmarks in parallel because that exposes 
them to even more noise (scheduling, CPU count, current machine load 
etc). I don't understand the part of the sentence starting with "...use 
the thereby...", I'd be grateful if you elaborated.


Andrei