std.benchmark ready for review. Manager sought after

[Andrei Alexandrescu]

(Resuming a few past discussions about std.benchmark.)

On 4/9/12 1:26 PM, Manfred Nowak wrote:
> Andrei Alexandrescu wrote:
>> I disagree with running two benchmarks in parallel because that
>> exposes them to even more noise (scheduling, CPU count, current
>> machine load etc).
>
> I did not mean to run two or more benchmarks in parallel but only
> more instances of the benchmarked program _without_ the environment
> supplied by the benchmakr. Of course may there be more noise. But if
> so, that noise is dependent on the additional running instances and
> to nothing else.

I doubt that's the case. Once we discuss running several instances in 
parallel, I predict the vagaries of scheduling become much more prevalent.

> Now let T(n) be the time needed to run n instances in parallel on a
> single processor. According to your definition and your remark above:
>
> §   T(n) = n*A + Q + N + P(n)
>
> where P(n)>=0 and P(1)==0 is the additional noise with which the
> benchmarked program disturbs itself.
>
> Please observe that Q and N eliminate themselves on subtraction:
> §  T(2) - T(1) = A + P(2) - P(1)
> §  T(3) - T(2) = A + P(3) - P(2)
> ...
>
> P(n+1)-P(n) measures the sensitivity of the benchmarked program to
> its one noise. As you wrote yourself its value should be close to
> zero until the machine is close to falling flat.

It would be great if that applied, but I'm afraid things get 
significantly more complex. For example, the noise will have a 
per-processor component, and the processes will compete for cache among 
themselves.


Andrei