Proposal: Java Microbenchmark Harness (JMH)

[Aleksey Shipilev]

Hi,

I'd like to propose the new tool for inclusion to Codetools Project
within the OpenJDK. Please find the proposal below:

----8<---------------------------------------------------------------

Tool Name:
Java Microbenchmark Harness (JMH)

Tool Purpose:
Java harness for building, running, and analysing the nano/micro/macro
benchmarks, written in Java and other languages targeting the JVM.

Proposed By:
Aleksey Shipilev, Oracle, Java SE Performance team

Rationale:
Measuring the performance is the fine art, and measuring the performance
on microbenchmarks is double so. There are multiple caveats one should
take a great care of while designing the experiment involving
microbenchmarks. The advanced experience with VM technology is required,
the exposure with particular nits on exact VM is also a plus.

JMH is designed to aid the development of reliable microbenchmarks. Our
experience shows there is the clear need for the user-friendly way to
build ad-hoc benchmarks for JDK development and performance research.
None of the tools in the wild are meeting all the needs required for
this kind of work.

Including JMH in the Code Tools project allows us to use HotSpot- and
OpenJDK-specific functionality available only on this platform to get
better support for benchmarking. (Examples are: sun.misc.*, compiler
whitebox testing API, etc.)

It is also important to have the harness to keep up with the latest
API/ABI changes, available ahead of GA for a new release of JDK. It had
already been proved useful when dealing with JSR166, JSR292, Lambda
performance assessments, and other bleeding edge platform improvements.

In addition, JMH samples will guide the developers through the general
benchmarking pitfalls, as well as OpenJDK-specific nits, which will
greatly improve the quality of the ad-hoc performance testing in OpenJDK.

----8<---------------------------------------------------------------

Thanks,
Aleksey.