Choosing iteration length

[Aleksey Shipilev]

Hi Sebastian,

On 05/16/2014 11:46 AM, Millies, Sebastian wrote:
> I unsure about the trade-off between iteration length and number of iterations.
> I guess the trade-off will be different for different benchmarking modes.
> 
> For average time and throughput: should I have many short-running iterations
> or few long-running ones? (Amounting to approx. the same number of operations)

As the rule of thumb, you should have the iterations of the sufficient
length to:
 a) iterations run at steady state for the most part, e.g. running 50 us
per iteration is probably missing the scheduler rebalancing the threads;
my personal rule of thumb is to have the iterations no less than 100 ms.
 b) have many invocations of the workload so that the iteration result
is normally distributed (assuming unimodality, this follows from the
Central Limit Theorem).

On the other hand, there are reasons to have more iterations:
 a) More samples means better statistics; the difference in inference
power between sample sets of 5, 20, and 100 samples may be significant.
 b) Gets more chance to see how workload reaches the steady state; let's
you catch the non-steady-state behaviors early.

> For single-shot benchmarks I guess I should have just one iteration which may
> be short?

Single-shot benchmarks were originally destined to run a single
measurement iteration over multiple forks -- the scenarios to estimate
"cold" performance. But for many cases, you might want more measurement
iterations there especially if you are running only a single fork,
because more samples would be generated.

Thanks,
-Aleksey.