Tracking Memory Consumption Metrics With JMH What&How

[kirk at kodewerk.com]

Hi Jens,


> On Apr 6, 2017, at 5:10 AM, Jens Wilke <jw_list at headissue.com> wrote:
> 
> On Mittwoch, 5. April 2017 18:56:24 ICT Kirk Pepperdine wrote:
>> Hi Jens,
>> 
>> I will humbly disagree with your dropping run times to 20 seconds and
>> restarting JVM’s often. HotSpot warmup can take several minutes even for
>> the simplest of benchmarks.
> 
> Kirk,
> 
> maybe there is a communications hickup. Of course you need warmup.
> 
> As described in the blog post:
> 
> Within the same VM instance I do 2 warmup iterations of 20 seconds each, and 
> then 3 measurement iterations. With 5 VM restarts (forks in JMH lingo) its 10 
> measurement iterations altogether.
> 
> Since I am doing 3 measurement iterations within the same VM, performance
> differences, maybe as a result of insufficient warmup, would be detected 
> reliably in the confidence interval.
I would suggest that under benchmarking conditions that you would be measuring the same warmup conditions which would of course lead to approximately the same measures. The two warm ups are actually stabilizing the OS also.
There is a lot going on here that remains hidden.

What I would suggest is that you get a handle on the JIT counts and run the warmups until that value stabilizes for some period of time. I’d suggest an initial investigation that may last 30-60 minutes??? We can debate the duration but…. 20 seconds is simply unlikely to be long enough.

Kind regards,
Kirk