Review of Andrei's std.benchmark

[Andrei Alexandrescu]

On 9/21/12 11:12 AM, Manu wrote:
> On 21 September 2012 07:45, Andrei Alexandrescu
> <SeeWebsiteForEmail at erdani.org <mailto:SeeWebsiteForEmail at erdani.org>>
> wrote:
>
>         As such, you're going to need a far more
>         convincing argument than "It worked well for me."
>
>
>     Sure. I have just detailed the choices made by std.benchmark in a
>     couple of posts.
>
>     At Facebook we measure using the minimum, and it's working for us.
>
>
> Facebook isn't exactly 'realtime' software. Obviously, faster is always
> better, but it's not in a situation where if you slip a sync point by
> 1ms in an off case, it's all over. You can lose 1ms here, and make it up
> at a later time, and the result is the same. But again, this feeds back
> to your distinction between benchmarking and profiling.

You'd be surprised at how much we care about e.g. 90 percentile time to 
interaction.

>         Otherwise, I think we'll need richer results. At the very least
>         there
>         should be an easy way to get at the raw results programmatically
>         so we can run whatever
>         stats/plots/visualizations/__output-formats we
>         want. I didn't see anything like that browsing through the docs, but
>         it's possible I may have missed it.
>
>
>     Currently std.benchmark does not expose raw results for the sake of
>     simplicity. It's easy to expose such, but I'd need a bit more
>     convincing about their utility.
>
>
> Custom visualisation, realtime charting/plotting, user supplied reduce
> function?

Hrm, that sounds like an entire new project.


Andrei