Reviving JEP 230: Microbenchmark Suite

Mikael Vidstedt mikael.vidstedt at oracle.com
Mon Sep 10 17:11:09 UTC 2018


I would very much like to see this move forward. With more of the development happening in project sandboxes the challenge mentioned of keeping benchmarks for new or changed APIs in sync with the sandboxes will only get worse. Co-locating the benchmarks does have some cons as Aleksey mentions, though other co-location exercise not entirely unlike the proposed one have in my experience always turned out positive. I do think we’ll find that the fact that the benchmarks are trivially *there* (no extra cloning etc. needed) will mean that more people actually run them, and more people will develop new ones.

Cheers,
Mikael

> On Sep 6, 2018, at 8:16 AM, Claes Redestad <claes.redestad at oracle.com> wrote:
> 
> Hi,
> 
> JEP 230: Microbenchmark Suite[1] was proposed during JDK 9 development, but was put on hold for various mundane reasons. I think time has come to revive this effort, and I've volunteered to take over ownership of this JEP.
> 
> Some time after JEP 230 was temporarily abandoned, the jmh-jdk-microbenchmarks project[2] was conceived to ensure some of the work done to prepare didn't go to waste. A set of previously closed source microbenchmarks was open sourced and contributed there, and artifacts from this project are actively being used to track performance. This situation is not ideal, however.
> 
> A side-effect of the more rapid release cadence is that the bulk of new feature development is being done in project repositories. This creates some very specific challenges when developing benchmarks out-of-tree, especially those that that aim to track new and emerging APIs.
> 
> For starters we're pushed toward setting up branches that mirror the project layout of the various OpenJDK projects (valhalla, amber, ...), then we need to set up automated builds from each such branch, then have some automated means to match these artifacts with appropriate builds of the project-under-test etc. And how do we even deal with the case when the changes we really want to test are in javac? Nothing is impossible, of course, but the workarounds and added work is turning out to be costly.
> 
> By co-locating microbenchmarks, matching the JDK-under-test with an appropriate microbenchmark bundle would be trivial and automatic in most cases, and while one always need to be wary of subtle changes creeping into benchmark bundles and the JDK between builds, this is something we already test for automatically as regressions are detected.
> 
> Also, we wouldn't really lose the ability to pick up an existing microbenchmark artifact as appropriate (a "golden" bundle; appropriate when comparing library and VM changes over a longer time). A standalone project can be considered a good enough fit for that case, so one alternative to moving all of jmh-jdk-microbenchmarks into the JDK would be keep maintaining the standalone project for benchmarks that are considered mature and stable. My preference is to make jmh-jdk-microbenchmarks redundant and then shut it down, though. I think most would typically build and keep a "golden" bundle of a specific version around for longer-term regression tests, so a separate standalone project of JDK micros doesn't make much sense.
> 
> Also of note is that one of the more controversial aspects of JEP 230 was the question of where to put them. We've since consolidated the OpenJDK source repositories into a single one, so much of what was discussed back then no longer applies. Mainly this simplifies the steps needed to integrate a microbenchmark suite into the JDK, and I've updated the JEP text accordingly.
> 
> Thanks!
> 
> /Claes
> 
> [1] http://openjdk.java.net/jeps/230
> [2] http://openjdk.java.net/projects/code-tools/jmh-jdk-microbenchmarks/



More information about the jdk-dev mailing list