JMH auto driver ?

[Laurent Bourgès]

Hi,

I wonder how I can add auto-tuning to JMH: adjust warmup & iterations
(time, counts) to ensure time stddev < threshold, e.g 1 or 2%.

As array adaptive sorting algorithms imply the sort() duration depends a
lot on the data distribution, having a fixed number of iterations or time
is counter-productive:
- easy data ~ 100ns, complex data ~ 10000ns
- if I set longer duration, lot of time is wasted for short tests
- I had a look at GA sample 25, to run first short benchmarks and repeat
with longer duration if stddev is too high. I will experiment writing my
auto-tuner ie reuse jmh Runner via java API to control its own parameters...

Ideally I could do the same for warmups, if avgTime is stabilized, then
early skip remaining warmup iterations

Finally I am interested by min time and its variance (cpu bound test) so I
will use many iterations (avg time) and estimate min + stddev.

Do you think I should try SamplingTime mode but it requires me to tune also
the batch size ?

Here is the jmh test:
https://github.com/bourgesl/nearly-optimal-mergesort-code/blob/master/sort-bench/src/main/java/edu/sorting/bench/ArraySortBenchmark.java

EA results having varying stddev at the end:
https://github.com/bourgesl/nearly-optimal-mergesort-code/blob/master/sort-bench/results/
<https://github.com/bourgesl/nearly-optimal-mergesort-code/blob/master/sort-bench/results/sort-100.out>

I can not trust them for now, some results have too large stddev ~ average.


Thank you for your advices & directions,
Laurent