RFR: 8316814: NMT: A separate script or Java program is needed to analyze and make useful reports ouf of JMH benchmarks outputs. [v2]

[Aleksey Shipilev]

On Mon, 23 Oct 2023 08:21:57 GMT, Afshin Zafari <azafari at openjdk.org> wrote:

>> It reads the benchmark report file whose path is given as command line. Then extract percentage of overhead of NMT modes for different methods used in benchmarks.
>> Locally tested with some sample report files.
>> From an input benchmark report like this: 
>> 
>> # JMH version: 1.37
>> # VM version: JDK 22-internal, OpenJDK 64-Bit Server VM,......in...
>> # VM invoker:.....
>> # VM options: -XX:NativeMemoryTracking=detail --add-exports java.base/jdk.internal.misc=ALL-UNNAMED -Djava.library.path=...
>> # Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
>> # Warmup: 2 iterations, 1 s each
>> # Measurement: 5 iterations, 1 s each
>> # Timeout: 10 min per iteration
>> # Threads: 1 thread, will synchronize iterations
>> # Benchmark mode: Average time, time/op
>> # Benchmark: org.openjdk.bench.vm.runtime.NMTBenchmark.NMTDetail.mixAallocateFreeMemory
>> # Parameters: (N = 100000, THREADS = 4)
>> 
>> .
>> .
>> .
>> 
>> 
>> Benchmark                                               (N)  (THREADS)  Mode  Cnt    Score    Error  Units
>> NMTBenchmark.NMTDetail.mixAallocateFreeMemory        100000          0  avgt   10   87.010 ? 10.630  ms/op
>> NMTBenchmark.NMTDetail.mixAallocateFreeMemory        100000          4  avgt   10    2.473 ?  0.728  ms/op
>> NMTBenchmark.NMTDetail.mixAllocateReallocateMemory   100000          0  avgt   10  225.054 ?  4.381  ms/op
>> NMTBenchmark.NMTDetail.mixAllocateReallocateMemory   100000          4  avgt   10    2.247 ?  0.599  ms/op
>> NMTBenchmark.NMTDetail.onlyAllocateMemory            100000          0  avgt   10  147.346 ?  8.226  ms/op
>> NMTBenchmark.NMTDetail.onlyAllocateMemory            100000          4  avgt   10    2.285 ?  0.611  ms/op
>> NMTBenchmark.NMTOff.mixAallocateFreeMemory           100000          0  avgt   10   40.405 ?  8.115  ms/op
>> NMTBenchmark.NMTOff.mixAallocateFreeMemory           100000          4  avgt   10    2.299 ?  0.658  ms/op
>> NMTBenchmark.NMTOff.mixAllocateReallocateMemory      100000          0  avgt   10  158.700 ? 10.750  ms/op
>> NMTBenchmark.NMTOff.mixAllocateReallocateMemory      100000          4  avgt   10    2.282 ?  0.723  ms/op
>> NMTBenchmark.NMTOff.onlyAllocateMemory               100000          0  avgt   10  113.949 ? 21.257  ms/op
>> NMTBenchmark.NMTOff.onlyAllocateMemory               100000          4  avgt   10    2.155 ?  0.682  ms/op
>> NMTBenchmark.NMTSummary.mixAallocateFreeMemory       100000          0  avgt   10   50.436 ?  4.576  ms/op
>> NMTBenchmark.NMTSummary...
>
> Afshin Zafari has updated the pull request incrementally with one additional commit since the last revision:
> 
>   empty 'finally' block is removed.

I agree having the JMH comparison tool is important, I just don't think that:
 a) JDK microbenchmark suite is the proper home for it; looks like we need to support this natively in JMH instead (maybe as separate tool as well);
 b) Parsing the human-readable result is a proper way to do this, as it is not guaranteed to be stable; JMH has machine-readable JSON output for external tools;

-------------

PR Review: https://git.openjdk.org/jdk/pull/16070#pullrequestreview-1692038856