RFR: 7904061: Enable JMH integration tests to be executed against other impls

Tue Jul 8 11:17:03 UTC 2025

**NOTE**: This PR should be marked as draft because not all tests have been migrated to this setup. I've migrated the ones in the root test package to get a feel on how things would work and get feedback on whether this approach is fine for the JMH community.

This is the **last** in a series of PRs extending JMH to benchmark Java code running as GraalVM native images more easily.

In this PR:
* We enable `jmh-core-it` test jar to be created so that JMH extensions can consume it.
* We create a test `RunnerFactory` interface with a method called `createRunner` that by default creates JMH's `Runner` instance. JMH extensions that want to run the same test with a different runner would apply changes like [these](https://github.com/galderz/fibula/commit/79f48a8eabd38af4a4a023dacd2a925d8052aeea). Here the JMH extension extends the default `RunnerFactory` interface, and extends individual test classes to make them implement the extended runner factory interface.

The result of both these changes enables JMH tests to be executed with the JMH extension:

[INFO] --- failsafe:3.5.2:integration-test (default) @ fibula-jmh-it ---
[INFO] Using auto detected provider org.apache.maven.surefire.junit4.JUnit4Provider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.mendrugo.fibula.jmh.it.SingleShotTest
# JMH version: fibula:999-SNAPSHOT
# VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1
# *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong.
# VM invoker: target/benchmarks
...
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.536 s -- in org.mendrugo.fibula.jmh.it.SingleShotTest
[INFO] Running org.mendrugo.fibula.jmh.it.IterationCountCmdTest
# JMH version: fibula:999-SNAPSHOT
# VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1
# *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong.
# VM invoker: target/benchmarks
...
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.712 s -- in org.mendrugo.fibula.jmh.it.IterationCountCmdTest
[INFO] Running org.mendrugo.fibula.jmh.it.IterationCountAnnTest
# JMH version: fibula:999-SNAPSHOT
# VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1
# *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong.
# VM invoker: target/benchmarks
...
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.345 s -- in org.mendrugo.fibula.jmh.it.IterationCountAnnTest
[INFO] Running org.mendrugo.fibula.jmh.it.WarmupIterationCountAnnTest
# JMH version: fibula:999-SNAPSHOT
# VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1
# *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong.
# VM invoker: target/benchmarks
...
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.751 s -- in org.mendrugo.fibula.jmh.it.WarmupIterationCountAnnTest
[INFO] Running org.mendrugo.fibula.jmh.it.WarmupIterationCountCmdTest
# JMH version: fibula:999-SNAPSHOT
# VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1
# *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong.
# VM invoker: target/benchmarks
...
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.756 s -- in org.mendrugo.fibula.jmh.it.WarmupIterationCountCmdTest
[INFO] Running org.mendrugo.fibula.jmh.it.StackTraceInThrowableTest
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 s -- in org.mendrugo.fibula.jmh.it.StackTraceInThrowableTest
[INFO] 
[INFO] Results:
[INFO] 
[WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 1

Note that above we can observe that one of the JMH tests is disabled. It could happen that several JMH tests are not applicable to native image, e.g. perfasm tests. In the usage changes above we can observe how JUnit's `@Ignore` annotation can be easily used to disable particular tests as needed. In the particular case of `PrintFlags`, the JMH extension could filter this flag and not pass on (there are already such cases) so with that modification the test could pass but I left it as is so that test exclusion can be demonstrated for this PR.

Here is the PR list for reference:
1. [Make OutputFormatAdapter public](https://github.com/openjdk/jmh/pull/158)
2. [Enable BenchmarkParams construction to be overriden](https://github.com/openjdk/jmh/pull/160)
3. [Enable alternative Runner instantiation and Main reuse](https://github.com/openjdk/jmh/pull/161)
4. [Enable profiler classes to be reused outside of JMH](https://github.com/openjdk/jmh/pull/162)
5. [Enable JMH integration tests to be executed against other impls](https://github.com/openjdk/jmh/pull/163)

-------------

Commit messages:
 - Enable Runner instance to test to be provided by subclasses
 - Enable test reuse by generating the core IT test jar

Changes: https://git.openjdk.org/jmh/pull/163/files
  Webrev: https://webrevs.openjdk.org/?repo=jmh&pr=163&range=00
  Issue: https://bugs.openjdk.org/browse/CODETOOLS-7904061
  Stats: 64 lines in 8 files changed: 45 ins; 6 del; 13 mod
  Patch: https://git.openjdk.org/jmh/pull/163.diff
  Fetch: git fetch https://git.openjdk.org/jmh.git pull/163/head:pull/163

PR: https://git.openjdk.org/jmh/pull/163