Performance

[Andrei Alexandrescu via Digitalmars-d]

On 5/31/14, 7:10 AM, Russel Winder via Digitalmars-d wrote:
> On Sat, 2014-05-31 at 07:02 -0700, Andrei Alexandrescu via Digitalmars-d
> wrote:
>> On 5/30/14, 11:36 PM, Russel Winder via Digitalmars-d wrote:
>>> As well as the average (mean), you must provide standard deviation and
>>> degrees of freedom so that a proper error analysis and t-tests are
>>> feasible. Or put it another way: even if you quote a mean with knowing
>>> how many in the sample and what the spread is you cannot judge the error
>>> and so cannot make deductions or inferences.
>>
>> No. Elapsed time in a benchmark does not follow a Student or Gaussian
>> distribution. Use the mode or (better) the minimum. -- Andrei
>
> We almost certainly need to unpack that more. I agree that behind my
> comment was an implicit assumption of a normal distribution of results.
> This is an easy assumption to make even if it is wrong. So is it
> provably wrong? What is the distribution? If we know that then there is
> knowledge of the parameters which then allow for statistical inference
> and deduction.

Well there's quantization noise which has uniform distribution. Then all 
other sources of noise are additive (no noise may make code run faster). 
So I speculate that the pdf is a half Gaussian mixed with a uniform 
distribution. Taking the mode (which is very close to the minimum in my 
measurements) would be the most accurate way to go. Taking the average 
would end up in some weird point on the half-Gaussian slope.

Andrei