FW: Basic JMH questions

[Kenneth Fogel]

(I may have used the wrong email address earlier today so sending this again)

Hi,

If there is a better mailing list to ask my questions about the JMH then please let me know. I will be presenting a session at JavaZone next month entitled "I Just Discovered the Java Microbenchmark Harness and You Should To". Some years ago, I used System.nanoTime to measure performance of the Java Collections for a course I was giving. The results were relatively correct, i.e. LinkedLists are slow,  but I knew they were far from accurate. They also changed radically from run to run. I did a similar presentation two years ago but for this version I wish to present what will effectively be a tutorial on the best use of the JMH. To that end I have just a few questions.

The first most obvious question is if there are JavaDocs. Experimentation has revealed to me much of how the JMH works. There are many good articles on the JMH, but none are complete which is what leads to the next two questions.

When is a new JVM spun up for testing? Does it happen for every fork, warmup, or iteration? In reviewing my results, I see that the first Warmup Iteration is usually twice what the subsequent warmup iterations are and that the Iteration values are usually all close together. I am using the default of 5 forks and iterations, both in 10s. I am using Java 21.

What does this option mean:
.jvmArgsAppend("-Djmh.stack.period=1") mean
I found it in this file at  https://github.com/xp-vit/java-logging-performance-tests in the BenchmarkUtils.java<https://github.com/xp-vit/java-logging-performance-tests%20in%20the%20BenchmarkUtils.java>. I do not perceive a difference if I use it or not.

Please feel free to recommend any topics or concepts that should be presented. The session is only an hour, but I fear I may already have enough for two or three hours. As running tests can be time consuming, I will show the results of tests I performed previously. A single test class with 23 methods took three and a half hours to run last night.

Here is the setup for each test running under Windows 11:
# JMH version: 1.37
# VM version: JDK 21.0.2, OpenJDK 64-Bit Server VM, 21.0.2+13-LTS
# VM invoker: C:\Program Files\Eclipse Adoptium\jdk-21.0.2.13-hotspot\bin\java.exe
# VM options: -server -Djmh.stack.period=1
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op

Thank you for your time.

Ken Fogel






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jmh-dev/attachments/20240816/16617cd2/attachment-0001.htm>