JMH automatic driver ?

[Laurent Bourgès]

Hi,

I wonder how I can add auto-tuning to JMH: adjust warmup & iterations
(time, counts) to ensure stddev < threshold, e.g 1 or 2%.

As array adaptive sorting algorithms imply the duration depends a lot on
the data distribution, having a fixed number of iterations or time is
counter-productive:
- easy data ~ 100ns, complex data ~ 10000ns
- if I set longer duration, lot of time is wasted for short tests
- I will look at GA sample 25, to run first short benchmarks and repeat
with longer duration if stddev is too high.

Ideally I could do the same for warmups, if avgTime is stabilized, then
early skip remaining warmup iterations

Finally I am interested by min time and its variance (cpu bound test) so I
will use many iterations (avg time) and estimate min + stddev.

Do you think I should try SamplingTime mode but it requires me to tune also
the batch size ?

Here is the jmh test:
https://github.com/bourgesl/nearly-optimal-mergesort-code/blob/master/sort-bench/src/main/java/edu/sorting/bench/ArraySortBenchmark.java

EA results having varying stddev at the end:
https://github.com/bourgesl/nearly-optimal-mergesort-code/blob/master/sort-bench/results/sort-100.out

I can not trust them for now, will run again with 10x longer duration.


Thank you for your advices & directions,
Laurent