Tracking Memory Consumption Metrics With JMH What&How

[Kirk Pepperdine]

> On Apr 6, 2017, at 12:23 PM, Jens Wilke <jw_list at headissue.com> wrote:
> 
> Hi Kirk,
> 
> thanks for the lively discussion.

It’s fun… 
> 
> On Donnerstag, 6. April 2017 14:31:15 ICT kirk at kodewerk.com wrote:
>> There is a lot going on here that remains hidden.
> 
> True, there is a lot going on, it is hidden but hopefully not undetected.
> 
> I have two working assumptions:
> 
> 1)
> 
> If any adoption (e.g. recompile, GC autotuning, OS+CPU cache warmup)  happens, 
> that would be noticeable indirectly in the measured performance.

Sometimes difficult to notice.

> 
> 2)
> 
> If the JVM has a consistent performance level for one minute, and the workload 
> is identical, it will possibly not change after more time has passed.

This is the assumption that I would expect but I’ve found not to be true in all cases. I’d investigate to find an appropriate run time.

> 
>> What I would suggest is that you get a handle on the JIT counts and run the
>> warmups until that value stabilizes for some period of time.
> 
> As you said, there is a lot going on. Which means, it's hard to miss critical 
> things. I think the statistical confidence is the better "general" truth to 
> verify that there is no warmup or any other cause of deviations.

I’d use the internal JIT counters.

I don’t want to talk about GC, it’s an orthogonal yet important topic to consider.

> 
> I consider minimizing the iteration runtime while meeting an accuracy goal as 
> best practice, in case you like to run benchmarks continuously and get fast 
> feedback.

That would be the dream.

Kind regards,
Kirk