Mean error bigger than mean?

[Aleksey Shipilev]

On 04/10/2014 10:00 PM, Dmitry Vyazelenko wrote:
> I came across interesting issue with one benchmark while testing it
> on JDK8. Basically I observe measurement error bigger than measured
> value. And I’m not sure if it is somehow JMH issue or JDK8 regression
> or something else. I hope you can give me a hint on what is going
> on.

Well, the mere fact the error is larger than the mean itself does not
strike me as the correctness issue. I think your benchmark is GC bound,
and so it might require:

 * Setting the explicit heap size to tolerate the allocations
 * Do longer warmup for GC heuristics to adopt properly
 * Using "-gc true" to force GC between iterations
 * Tracking "-prof gc" to see the GC impact

Let me know if you think there is a problem with JMH in that regard.

Thanks,
-Aleksey.