JMH iterations slow

[Aleksey Shipilev]

On 05/15/2014 05:16 PM, Millies, Sebastian wrote:
> thanks for the quick answer - I was unaware that an iteration
> comprises as many calls to the benchmarked operation as would fit in
> a given time interval - as given by measurementTime(TimeValue), I
> believe

Yes, that's right.

> However, I still don't quite understand how you arrive at the
> "roughly one million" estimate. JMH outputs ca. 22 us per operation
> (is that the same as @GMB invocation?) 

Yes.

> In one second (a million us)
> that fits about 45000 operations. In a 100 iterations, that would be
> ca 4.5 million operations, not "roughly a million". Does that sound
> right?

Yeah, in 10x1 sec iterations, you have around half a million ops.

> Thanks also for the pointer to TextReportFormat.

If you think you are going to do a contribution, please do submit OCA
early: http://openjdk.java.net/contribute/

> My colleagues in general are skeptical about micro-benchmarking. I
> like jmh, but even if it creates good micro-benchmarks (which I don't
> know how to prove) there is a more fundamental worry: The benchmarks
> are good because jmh is successful in avoiding many of the runtime
> optimizations that would normally be a problem in micro-benchmarking,
> but for that very reason, the timings may not relate well to what
> will be going on in productive code. Methods that have similar
> benchmark results may still perform very differently in a real
> environment. Do you think that is a valid concern? 

That's a broad over-generalization. Benchmarking framework job is not to
intimidate the runtime optimizations everywhere, but only in the
specific places where it matters for benchmarking. Usually at the
borders of the executing code.

The best analogy off the top of my head: it is like studying a live
heart by taking it out of the body (moral issues notwithstanding), and
plugging it to the life support system which still flows blood through
it, maintains temperature, pressure, and oxygenation to make the modeled
environment realistic and put a heart into the same conditions it has
inside the body. Otherwise you are studying a dead heart pretending it
it alive, and making bold statements like "here, we studied this alive
heart, and it's not beating at all, ergo, all alive hearts are not beating",

Thanks,
-Aleksey.