RFR: 8316814: NMT: A separate script or Java program is needed to analyze and make useful reports ouf of JMH benchmarks outputs.

[David Holmes]

On Fri, 6 Oct 2023 10:22:23 GMT, Afshin Zafari <azafari at openjdk.org> wrote:

> It reads the benchmark report file whose path is given as command line. Then extract percentage of overhead of NMT modes for different methods used in benchmarks.
> Locally tested with some sample report files.
> From an input benchmark report like this: 
> 
> # JMH version: 1.37
> # VM version: JDK 22-internal, OpenJDK 64-Bit Server VM,......in...
> # VM invoker:.....
> # VM options: -XX:NativeMemoryTracking=detail --add-exports java.base/jdk.internal.misc=ALL-UNNAMED -Djava.library.path=...
> # Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
> # Warmup: 2 iterations, 1 s each
> # Measurement: 5 iterations, 1 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Average time, time/op
> # Benchmark: org.openjdk.bench.vm.runtime.NMTBenchmark.NMTDetail.mixAallocateFreeMemory
> # Parameters: (N = 100000, THREADS = 4)
> 
> .
> .
> .
> 
> 
> Benchmark                                               (N)  (THREADS)  Mode  Cnt    Score    Error  Units
> NMTBenchmark.NMTDetail.mixAallocateFreeMemory        100000          0  avgt   10   87.010 ? 10.630  ms/op
> NMTBenchmark.NMTDetail.mixAallocateFreeMemory        100000          4  avgt   10    2.473 ?  0.728  ms/op
> NMTBenchmark.NMTDetail.mixAllocateReallocateMemory   100000          0  avgt   10  225.054 ?  4.381  ms/op
> NMTBenchmark.NMTDetail.mixAllocateReallocateMemory   100000          4  avgt   10    2.247 ?  0.599  ms/op
> NMTBenchmark.NMTDetail.onlyAllocateMemory            100000          0  avgt   10  147.346 ?  8.226  ms/op
> NMTBenchmark.NMTDetail.onlyAllocateMemory            100000          4  avgt   10    2.285 ?  0.611  ms/op
> NMTBenchmark.NMTOff.mixAallocateFreeMemory           100000          0  avgt   10   40.405 ?  8.115  ms/op
> NMTBenchmark.NMTOff.mixAallocateFreeMemory           100000          4  avgt   10    2.299 ?  0.658  ms/op
> NMTBenchmark.NMTOff.mixAllocateReallocateMemory      100000          0  avgt   10  158.700 ? 10.750  ms/op
> NMTBenchmark.NMTOff.mixAllocateReallocateMemory      100000          4  avgt   10    2.282 ?  0.723  ms/op
> NMTBenchmark.NMTOff.onlyAllocateMemory               100000          0  avgt   10  113.949 ? 21.257  ms/op
> NMTBenchmark.NMTOff.onlyAllocateMemory               100000          4  avgt   10    2.155 ?  0.682  ms/op
> NMTBenchmark.NMTSummary.mixAallocateFreeMemory       100000          0  avgt   10   50.436 ?  4.576  ms/op
> NMTBenchmark.NMTSummary.mixAallocateFreeMemory       100000          4  avgt   10    2.192 ?  0.7...

Not a review but as per my comment in JBS I'm unclear where this tool really belongs. Perhaps @shipilev  has an opinion?

test/micro/org/openjdk/bench/vm/runtime/NMTAnalyze.java line 149:

> 147:       System.out.println(ioe.getMessage());
> 148:     } finally {
> 149:     }

This serves no purpose.

-------------

PR Review: https://git.openjdk.org/jdk/pull/16070#pullrequestreview-1691628084
PR Review Comment: https://git.openjdk.org/jdk/pull/16070#discussion_r1368068089