Tracking Memory Consumption Metrics With JMH What&How

[Jens Wilke]

Hi Kirk,

thanks for the lively discussion.

On Donnerstag, 6. April 2017 14:31:15 ICT kirk at kodewerk.com wrote:
> There is a lot going on here that remains hidden.

True, there is a lot going on, it is hidden but hopefully not undetected.

I have two working assumptions:

1)

If any adoption (e.g. recompile, GC autotuning, OS+CPU cache warmup)  happens, 
that would be noticeable indirectly in the measured performance.

2)

If the JVM has a consistent performance level for one minute, and the workload 
is identical, it will possibly not change after more time has passed.

> What I would suggest is that you get a handle on the JIT counts and run the
> warmups until that value stabilizes for some period of time.

As you said, there is a lot going on. Which means, it's hard to miss critical 
things. I think the statistical confidence is the better "general" truth to 
verify that there is no warmup or any other cause of deviations.

NB: One critical counter is actually how many full GCs happen. Larger heap 
sizes require longer iteration times.

> I’d suggest an
> initial investigation that may last 30-60 minutes??? We can debate the
> duration but…. 20 seconds is simply unlikely to be long enough.

Yes, fair enough. That's actually challenging working assumption 2.
Verifying this cannot hurt.

In the JMH examples I can find iteration runtimes of 100 milliseconds and one 
second. Typically 5 warmup intervals are used, which means real measurement 
starts after 5 seconds. It always depends on the benchmark scenario and needs 
experimental verification. Other experiences may hold for other scenarios.

Side note:

My urge for short benchmark iteration times may look strange. However, there 
are a lot of different parameters to walk through (thread counts, hitrate, 
size, G1/CMS), which already leads to a total runtime of about two days.

I consider minimizing the iteration runtime while meeting an accuracy goal as 
best practice, in case you like to run benchmarks continuously and get fast 
feedback.

Best,

Jens

-- 
"Everything superfluous is wrong!"

   // Jens Wilke - headissue GmbH - Germany
 \//  https://headissue.com