Proposal: Java Microbenchmark Harness (JMH)

[Jonathan Gibbons]

This proposal was posted over two weeks ago.

If anyone has any comments, please post them by the
end of Friday, March 15.

-- Jon


On 02/21/2013 06:40 AM, Aleksey Shipilev wrote:
> Hi,
>
> I'd like to propose the new tool for inclusion to Codetools Project
> within the OpenJDK. Please find the proposal below:
>
> ----8<---------------------------------------------------------------
>
> Tool Name:
> Java Microbenchmark Harness (JMH)
>
> Tool Purpose:
> Java harness for building, running, and analysing the nano/micro/macro
> benchmarks, written in Java and other languages targeting the JVM.
>
> Proposed By:
> Aleksey Shipilev, Oracle, Java SE Performance team
>
> Rationale:
> Measuring the performance is the fine art, and measuring the performance
> on microbenchmarks is double so. There are multiple caveats one should
> take a great care of while designing the experiment involving
> microbenchmarks. The advanced experience with VM technology is required,
> the exposure with particular nits on exact VM is also a plus.
>
> JMH is designed to aid the development of reliable microbenchmarks. Our
> experience shows there is the clear need for the user-friendly way to
> build ad-hoc benchmarks for JDK development and performance research.
> None of the tools in the wild are meeting all the needs required for
> this kind of work.
>
> Including JMH in the Code Tools project allows us to use HotSpot- and
> OpenJDK-specific functionality available only on this platform to get
> better support for benchmarking. (Examples are: sun.misc.*, compiler
> whitebox testing API, etc.)
>
> It is also important to have the harness to keep up with the latest
> API/ABI changes, available ahead of GA for a new release of JDK. It had
> already been proved useful when dealing with JSR166, JSR292, Lambda
> performance assessments, and other bleeding edge platform improvements.
>
> In addition, JMH samples will guide the developers through the general
> benchmarking pitfalls, as well as OpenJDK-specific nits, which will
> greatly improve the quality of the ad-hoc performance testing in OpenJDK.
>
> ----8<---------------------------------------------------------------
>
> Thanks,
> Aleksey.