how reliable JMH's results are

[Aleksey Shipilev]

On 3/26/19 11:56 AM, Maurizio Cimadamore wrote:
> On 25/03/2019 23:54, Aleksey Shipilev wrote:
>> I spy with my little eye the bimodal result: either it is 170 Kops/ms, or 164 Kops/ms. The "dips" in
>> iteration performance is quite often a GC
> 
> I agree that smaller differences can often be down to GC - I've seen bimodal behavior as well. But
> note my other email, where we basically have one jump from 131417 to 216897 - that's seem a bit
> extreme in terms of bimodal (at least I have never encountered such a stark one); and follow up
> experiments have never been able to replicate the 131Kop/s rate, which to me suggests it's something
> else (most likely changes/improvements to the BSM code itself).

My point is that it is much more useful to profile and see why the difference is like that.

Silly problems (e.g. experimental setup problems) are found in minutes usually. See, the GC behavior
thing in my email: bimodal behavior; hypothesis it is GC; testing the hypothesis with GC profiler --
confirmed; testing the hypothesis that reducing heap size helps to alleviate -- confirmed; final
variance is lower -- done, moving on.

Do the similar thing for the jumps you observe in your setup. Benchmarking sessions are full of
these little phases, each of which brings you a step closer to reliable benchmark with
understandable results.

-Aleksey