Strange measurements when reproducing issue 5650

[H. S. Teoh]

On Wed, Apr 25, 2012 at 10:34:38AM +0200, SomeDude wrote:
[...]
> import std.stdio, std.container, std.range;
> 
> void main() {
>     enum int range = 100;
>     enum int n = 1_000_000;
> 
>     auto t = redBlackTree!int(0);
> 
>     for (int i = 0; i < n; i++) {
>         if (i > range)
>             t.removeFront();
>         t.insert(i);
>     }
> 
>     writeln(walkLength(t[]));
>     //writeln(t[]);
> }
> 
> runs in about 1793 ms.
> The strange thing is, if I comment out the writeln line, runtimes
> are in average *slower* by about 20 ms, with timings varying a
> little bit more than when the writeln is included.
[...]

First of all, differences as small as 20ms really should be considered
as background noise. The exact measurements depend on a lot of
system-specific and environment-specific factors, such as OS memory
usage, CPU cache behaviour, disk activity & speed, the exact points of
context switches, etc.. If you really want to check for substantial
performance differences, you need to magnify your test case so that
differences are measured >5 seconds.

Second, on my AMD hexacore 64-bit Linux system, the running time
consistently measures between 0.57 or 0.58 seconds for both cases. The
exact figure changes between runs, and as far as I can tell, there's no
discernible difference between the two.

Third, to make any timing differences stand out from the background
noise, I increased n to 20_000_000, and both versions of the program
consistently runs in about 11 seconds each time. There's no discernible
difference between the two.

What all this means is that a single call to writeln does not make
enough difference to be measurable compared to the rest of the program.
It doesn't mean that the version with writeln is "faster", just that the
difference is too small and you're probably just seeing background
noise. If you put the writeln inside a loop, on the other hand, you'll
see a big difference, because now its cost is magnified by the number of
times the loop runs. (Say if you put it inside a foreach(i;0..1000) at
the end of the program, you'll see the difference when you comment it
out.)

So I'd chalk it up to inherent measurement inaccuracies.


T

-- 
He who sacrifices functionality for ease of use, loses both and deserves neither. -- Slashdotter