Timing too good to be true

[John Colvin]

On Saturday, 21 September 2013 at 22:07:27 UTC, Paul Jurczak 
wrote:
> I'm running this test program with multiple compilers on 
> Xubuntu and I'm getting impossibly short timings with LDC 
> compared to DMD and GDC (timing details in comments below). I 
> would appreciate any ideas about what is going on here?
>
> /*
> Xubuntu 13.04 64-bit Core i5 3450S 2.8GHz:
> GDC 4.8.1:     gdc-4.8 -m64 -march=native -fno-bounds-check 
> -frename-registers -frelease -O3
> 669171001  826ns  e28_0
> 669171001  824ns  e28_1
>
> DMD64 2.063.2: dmd -O -noboundscheck -inline -release
> 669171001  1115ns  e28_0
> 669171001  1955ns  e28_1
>
> LDC 0.11.0: ldmd2 -m64 -O -noboundscheck -inline -release
> 669171001  25ns  e28_0
> 669171001  25ns  e28_1
> */
>
> module main;
>
> import std.stdio, std.algorithm, std.range, std.datetime, 
> std.typetuple;
>
>
> int e28_0(int N = 1002) {
> 	int diagNumber = 1;					
> 	int sum        = diagNumber;	
>
> 	for (int width = 2; width < N; width += 2)	
> 		for (int j = 0; j < 4; ++j) {			
> 			diagNumber += width;				
> 			sum        += diagNumber;			
> 		}
>
> 	return sum;
> }
>
>
> int e28_1(int N = 1002) {
> 	int diagNumber = 1;					
> 	int sum        = diagNumber;	
>
> 	foreach (width; iota(2, N, 2))
> 		foreach (_; 0..4) {			
> 			diagNumber += width;				
> 			sum        += diagNumber;			
> 		}
>
> 	return sum;
> }
>
>
> void main()
> {
>   enum      N = 100_000;
>   StopWatch sw;
>
>    foreach (iFun, fun; TypeTuple!(e28_0, e28_1)) {
>       int [N] results;
>       long[N] timings;
>
>       foreach (i; 0..N) {
>          sw.reset;
>          sw.start;
>          results[i] = fun();
>          sw.stop;
>          timings[i] = sw.peek.nsecs;
>       }
>
>       writeln(results.reduce!min, "  ", timings.reduce!min, "ns
>  e28_", iFun);
>   }
> }

Generally speaking, you should benchmark using outside input, 
unknown to the optimizer, that is representative of the use you 
are expecting. You'll be much less likely to fall foul of 
situations like this one.