std.benchmark ready for review. Manager sought after

[Andrei Alexandrescu]

On 4/10/12 5:40 AM, Jens Mueller wrote:
> How come that the times based relative report and the percentage based
> relative report are mixed in one result? And how do I choose which one
> I'd like to see in the output.

It's in the names. If the name of a benchmark starts with 
benchmark_relative_, then that benchmark is considered relative to the 
last non-relative benchmark. Using a naming convention allows complete 
automation in benchmarking a module.

I figure it's fine that all results appear together because the absence 
of data in the relative column clarifies which is which.

> When benchmarking you can measure different things at the same time. In
> this regard the current proposal is limited. It just measures wall clock
> time. I believe extending the StopWatch to measure e.g. user CPU time is
> a useful addition.

Generally I fear piling too much on StopWatch because every feature adds 
its own noise. But there's value in collecting the result of times(). 
What would be the Windows equivalent?

> In general, allowing user defined measurements would be great.
> E.g. to measure the time spend in user mode.
> () =>  {
>           tms t;
>           times(&t);
>           return t.tms_utime;
>        }
>
> Note, that this code does not need to be portable. You can also use
> version() else static assert.
>
> Things that come to mind that I'd like to measure.
> Time measurements:
>    * User CPU time
>    * System CPU time
>    * Time spent in memory allocations
> Count measurements:
>    * Memory usage
>    * L1/L2/L3 cache misses
>    * Number of executed instructions
>    * Number of memory allocations
>
> Of course wall clock time is the ultimate measure when benchmarking.
> But often you need to investigate further (doing more measurements).
>
> Do you think adding this is worthwhile?

Absolutely. I just fear about expanding the charter of the framework too 
much. Let's see:

* Memory usage is, I think, difficult in Windows.
* Estimating cache misses and executed instructions is significant research
* Number of memory allocations requires instrumenting druntime


Andrei