Tracking Memory Consumption Metrics With JMH What&How

[Bernd Eckenfels]

Hello,

Since we are at that topic, I typically try to measure more Warmup iterations and make sure the times go down after the first two. If not I increase the runtime.

I mentioned before for this observation it would help to actually print the number of executions, so one can see it without doing the math from the totals. (If I see 20 executions or 0,1s each I know I need to make the warmup iteration more like 2 minutes each).

The variance is an indicator for that (but only if the number of iterations is not too small or too big)

Gruss
Bernd
--
http://bernd.eckenfels.net
________________________________
From: jmh-dev <jmh-dev-bounces at openjdk.java.net> on behalf of Jens Wilke <jw_list at headissue.com>
Sent: Thursday, April 6, 2017 12:23:34 PM
To: kirk at kodewerk.com
Cc: jmh-dev at openjdk.java.net
Subject: Re: Tracking Memory Consumption Metrics With JMH What&How

Hi Kirk,

thanks for the lively discussion.

On Donnerstag, 6. April 2017 14:31:15 ICT kirk at kodewerk.com wrote:
> There is a lot going on here that remains hidden.

True, there is a lot going on, it is hidden but hopefully not undetected.

I have two working assumptions:

1)

If any adoption (e.g. recompile, GC autotuning, OS+CPU cache warmup)  happens,
that would be noticeable indirectly in the measured performance.

2)

If the JVM has a consistent performance level for one minute, and the workload
is identical, it will possibly not change after more time has passed.

> What I would suggest is that you get a handle on the JIT counts and run the
> warmups until that value stabilizes for some period of time.

As you said, there is a lot going on. Which means, it's hard to miss critical
things. I think the statistical confidence is the better "general" truth to
verify that there is no warmup or any other cause of deviations.

NB: One critical counter is actually how many full GCs happen. Larger heap
sizes require longer iteration times.

> I’d suggest an
> initial investigation that may last 30-60 minutes??? We can debate the
> duration but…. 20 seconds is simply unlikely to be long enough.

Yes, fair enough. That's actually challenging working assumption 2.
Verifying this cannot hurt.

In the JMH examples I can find iteration runtimes of 100 milliseconds and one
second. Typically 5 warmup intervals are used, which means real measurement
starts after 5 seconds. It always depends on the benchmark scenario and needs
experimental verification. Other experiences may hold for other scenarios.

Side note:

My urge for short benchmark iteration times may look strange. However, there
are a lot of different parameters to walk through (thread counts, hitrate,
size, G1/CMS), which already leads to a total runtime of about two days.

I consider minimizing the iteration runtime while meeting an accuracy goal as
best practice, in case you like to run benchmarks continuously and get fast
feedback.

Best,

Jens

--
"Everything superfluous is wrong!"

   // Jens Wilke - headissue GmbH - Germany
 \//  https://headissue.com