std.benchmark ready for review. Manager sought after

Denis Shelomovskij verylonglogin.reg at gmail.com
Mon Apr 9 00:06:36 PDT 2012


08.04.2012 21:31, Andrei Alexandrescu пишет:
> On 4/8/12 11:59 AM, Denis Shelomovskij wrote:
>> Very good but minimum isn't a best guess. Personally I (and there will
>> be a lot of such maniacs I suppose) will think that this (minimum) time
>> can be significantly smaller than average time.
>
> I've analyzed this quite a bit at work and the average and median are
> not very informative. You need the mode, but in most benchmarks the mode
> is very close to the minimum, so using the minimum is even better.
>
> In speed measurements, all noise is additive (there's no noise that may
> make a benchmark appear to run faster). There are also quite a few
> outliers. Recording the average will include a fair amount of noise.

Yes of course. I mean than an algorithm itself should follow some 
restrictions for such measurement. It should run with the same speed 
every time. Many algorithms follows this convention, but not all.

Why will recording the average produce so much noise? As I see, floating 
point arithmetic is now used without a strong reason so it looks like a 
time of this part isn't valuable. Or is it just a temporary solution?

Anyway it should be configurable using a CT parameter, so it will not 
abuse one who doesn't need it.

>
> Clearly there is noise during normal use as well, but incorporating it
> in benchmarks as a matter of course reduces the usefulness of benchmarks
> as a mean to improve performance of the benchmarked code.

A graph is needed exactly because of that. Without a graph it really 
gives very little.

>
>> So a parameter (probably with a default value) should be added.
>> Something like enum of flags telling what we want to know. At least
>> these looks usable: minTime, <some mean time>, maxTime,
>> standardDeviation, graph (yes, good old ASCII art).
>
> Standard deviation is also not very useful because it includes all
> outliers (some of which are very far away from the mode).

So a graph is needed.

>
>> Yes, graph is needed.
>
> I am not sure about that. We may provide the raw measurement data for
> programs that want to plot things, but plotting is beyond the charter of
> std.benchmark.
>

Sorry, meant a histogram, not a curve. A histogram can be shown in a 
console very well. And it is needed because its the easiest way to show 
benchmarked program behaviour (and noise behaviour). It also requires 
only about 80 integers to store information and shouldn't produce much 
noise.

IMHO, a histogram gives lots of information and will be a good addition.


-- 
Денис В. Шеломовский
Denis V. Shelomovskij


More information about the Digitalmars-d mailing list