Mean error bigger than mean?

[Kirk Pepperdine]

On Apr 10, 2014, at 8:40 PM, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:

> On 04/10/2014 10:00 PM, Dmitry Vyazelenko wrote:
>> I came across interesting issue with one benchmark while testing it
>> on JDK8. Basically I observe measurement error bigger than measured
>> value. And I’m not sure if it is somehow JMH issue or JDK8 regression
>> or something else. I hope you can give me a hint on what is going
>> on.
> 
> Well, the mere fact the error is larger than the mean itself does not
> strike me as the correctness issue.

I fear this is a correctness issue. Large variations in results means the threads executing the algorithm/workload you’re measuring were being interrupted or the unit of work wasn’t a constant meaning that it most likely your result set is a mix of different result sets. Either way, with large error or variation in the measurements.. you can’t use the measurements until you sort out the source of the variance and see if you can smoothen things out.
 
> I think your benchmark is GC bound,

I would concur… as this can be a huge source of variance. Now the question is, should this be a part of your measurement.. or not...

Regards,
Kirk