Review of Andrei's std.benchmark

Andrei Alexandrescu SeeWebsiteForEmail at erdani.org
Thu Sep 20 21:17:05 PDT 2012


On 9/20/12 10:05 AM, Manu wrote:
> On 20 September 2012 15:36, Andrei Alexandrescu
> <SeeWebsiteForEmail at erdani.org <mailto:SeeWebsiteForEmail at erdani.org>>
> wrote:
>     Let's use the minimum. It is understood it's not what you'll see in
>     production, but it is an excellent proxy for indicative and
>     reproducible performance numbers.
>
>
> If you do more than a single iteration, the minimum will virtually
> always be influenced by ideal cache pre-population, which is
> unrealistic.

To measure performance against cold cache, you could always clear the 
cache using one of the available methods, see 
http://stackoverflow.com/questions/1756825/cpu-cache-flush. Probably 
std.benchmark could include a routine that does that. But performance on 
cold would actually be most unrealistic and uninformative, as loading 
the memory into cache will dominate the work that the algorithm is 
doing, so essentially the benchmark would evaluate the memory bandwidth 
against the working set of the algorithm. That may be occasionally 
useful, but I'd argue that most often the interest in benchmarking is to 
measure repeated application of a function, not occasional use of it.

> Memory locality is often the biggest contributing
> performance hazard in many algorithms, and usually the most
> unpredictable. I want to know about that in my measurements.
> Reproducibility is not important to me as accuracy. And I'd rather be
> conservative(/pessimistic) with the error.
 >
> What guideline would you apply to estimate 'real-world' time spent when
> always working with hyper-optimistic measurements?

The purpose of std.benchmark is not to estimate real-world time. (That 
is the purpose of profiling.) Instead, benchmarking measures and 
provides a good proxy of that time for purposes of optimizing the 
algorithm. If work is done on improving the minimum time given by the 
benchmark framework, it is reasonable to expect that performance in-situ 
will also improve.


Andrei


More information about the Digitalmars-d mailing list