JMH auto driver ?

[Laurent Bourgès]

Hi,


I made good progress on my JMH auto-tune benchmark driver:
https://github.com/bourgesl/nearly-optimal-mergesort-code/blob/master/sort-bench/src/main/java/edu/sorting/bench/ArraySortBenchmark2.java

It looks complicated but the approach is simple:
- generate all benchmarks with their parameters
- perform warmup iterative adjustments to minimize the warmup time but
ensure proper warmup with 1 jvm fork: rms(3 avg times) are close to the
minimum (plateau reached)
- perform measurements with more forks (5) on all data distributions (10
instances each) ie ~100 times
- filters values using sort, and only consider 90% percentiles: avg,
stddev, rms... to obtain the final benchmark estimation: rms (avg+stddev)
that is not perturbated by gc/jit pauses (hi cups) but properly estimates
test variance on given data
- produce outputs: results in stdout (tab separated values) and complete
log in stderr, written in log files.

I will now refactor & cleanup the code to generalize it to any jmh
benchmark in a new AutoTuneJMHDriver class, and run lots of tests...

First EA results:
https://github.com/bourgesl/nearly-optimal-mergesort-code/blob/master/sort-bench/results/sort-100-1204.log

Comments & code reviews are welcome !

Enjoy,
Laurent


Le ven. 30 nov. 2018 à 09:44, Laurent Bourgès <bourges.laurent at gmail.com> a
écrit :

> Hi,
>
> I wonder how I can add auto-tuning to JMH: adjust warmup & iterations
> (time, counts) to ensure time stddev < threshold, e.g 1 or 2%.
>
> As array adaptive sorting algorithms imply the sort() duration depends a
> lot on the data distribution, having a fixed number of iterations or time
> is counter-productive:
> - easy data ~ 100ns, complex data ~ 10000ns
> - if I set longer duration, lot of time is wasted for short tests
> - I had a look at GA sample 25, to run first short benchmarks and repeat
> with longer duration if stddev is too high. I will experiment writing my
> auto-tuner ie reuse jmh Runner via java API to control its own parameters...
>
> Ideally I could do the same for warmups, if avgTime is stabilized, then
> early skip remaining warmup iterations
>
> Finally I am interested by min time and its variance (cpu bound test) so I
> will use many iterations (avg time) and estimate min + stddev.
>
> Do you think I should try SamplingTime mode but it requires me to tune
> also the batch size ?
>
> Here is the jmh test:
>
> https://github.com/bourgesl/nearly-optimal-mergesort-code/blob/master/sort-bench/src/main/java/edu/sorting/bench/ArraySortBenchmark.java
>
> EA results having varying stddev at the end:
>
> https://github.com/bourgesl/nearly-optimal-mergesort-code/blob/master/sort-bench/results/
> <https://github.com/bourgesl/nearly-optimal-mergesort-code/blob/master/sort-bench/results/sort-100.out>
>
> I can not trust them for now, some results have too large stddev ~ average.
>
>
> Thank you for your advices & directions,
> Laurent
>
>