JMH iterations slow

[Millies, Sebastian]

Hi Aleksey,

thanks for the quick answer - I was unaware that an iteration comprises as many calls to the benchmarked operation as would fit in a given time interval - as given by measurementTime(TimeValue), I believe

However, I still don't quite understand how you arrive at the "roughly one million" estimate.
JMH outputs ca. 22 us per operation (is that the same as @GMB invocation?) In one second (a million us)  that fits about 45000 operations.
In a 100 iterations, that would be ca 4.5 million operations, not "roughly a million".
Does that sound right?

Thanks also for the pointer to TextReportFormat.

My colleagues in general are skeptical about micro-benchmarking. I like jmh, but even if it creates good micro-benchmarks (which I don't know how to prove) there is a more fundamental worry:
The benchmarks are good because jmh is successful in avoiding many of the runtime optimizations that would normally be a problem in micro-benchmarking, but for that very reason, the timings may not relate well to what will be going on in productive code. Methods that have similar benchmark results may still perform very differently in a real environment.
Do you think that is a valid concern? I do definitely need to read up on this - would you have any pointers to material I should know?

Best regards,
Sebastian

-----Original Message-----
From: Aleksey Shipilev [mailto:aleksey.shipilev at oracle.com]
Sent: Thursday, May 15, 2014 1:26 PM
To: Millies, Sebastian; jmh-dev at openjdk.java.net
Subject: Re: JMH iterations slow

Hi Sebastian,

On 05/15/2014 02:51 PM, Millies, Sebastian wrote:
> Why are the iterations so slow? Looping inside the measurement method
> is actively discouraged, because of JIT optimizations. Are there other
> ways to speed things up?

I am a bit struggling to understand what do you mean by "slow"? By default, the measurement iterations are time-based, and the default run time for the iteration is 1 second. That helps to properly warm up the system, level out outliers, and get accurate timings.

> What if I ever want to run a million iterations?

I miss this point completely. With 22 us per @GMB invocation and 10x1 second iterations, you are already doing roughly a million of @GMB calls.

> PS: I’d like a verbosity mode that only gives me the configuration
> info, result and benchmark, but does not log each iteration, i. e.
> something between NORMAL and completely SILENT, again with a view to
> having many iterations.

This seems like a good idea. Do you want to hack it up yourself?
TextReportFormat is what you are looking at.

-Aleksey.

Software AG – Sitz/Registered office: Uhlandstraße 12, 64297 Darmstadt, Germany – Registergericht/Commercial register: Darmstadt HRB 1562 - Vorstand/Management Board: Karl-Heinz Streibich (Vorsitzender/Chairman), Dr. Wolfram Jost, Arnd Zinnhardt; - Aufsichtsratsvorsitzender/Chairman of the Supervisory Board: Dr. Andreas Bereczky - http://www.softwareag.com