From sitnikov.vladimir at gmail.com Thu Jul 3 09:25:55 2025 From: sitnikov.vladimir at gmail.com (Vladimir Sitnikov) Date: Thu, 3 Jul 2025 12:25:55 +0300 Subject: Can OperationsPerInvocation vary for each invocation? Message-ID: Hi, I want to benchmark "insert a batch of records to the database" scenario, and I want to use batches with varying sizes. Ideally I would like to have something like the following: Random batchSize; // state variable @Benchmark Object insertBatch(Infra infra) { int size = rnd.nextInt(10000); infra.opsPerInvocation(size); // <-- I want this instead of having a fixed @OperationsPerInvocation var ps = ...prepareStatement(...); for(int i=0; i From galder at openjdk.org Tue Jul 8 09:23:27 2025 From: galder at openjdk.org (Galder =?UTF-8?B?WmFtYXJyZcOxbw==?=) Date: Tue, 8 Jul 2025 09:23:27 GMT Subject: RFR: 7904058: Enable BenchmarkParams construction to be overriden Message-ID: This is the **second** in a series of PRs that I will be sending over the next few days, which make extending JMH to benchmark Java code running as GraalVM native images more easily. The codename for this extension is Fibula. It can still work without this PRs but without them the extension has to duplicate code that exists in JMH, which is not desirable. This PR enables `BenchmarkParams` construction to be overriden, in order to easily support JMH use case for benchmarking native images. The JIRA contains a detailed explanation. Having this PR in simplifies construction of `BenchmarkParams` instance. The changes https://github.com/galderz/fibula/commit/8fd8cf3f4eb1c6f37de7f807bfc697ef981a3e05#diff-601b6d98eff18db8a804efb415d515c3d397d42b9c982713b72c37bea2ebbf18 demonstrate the simplification enabled by the changes here. Here is the PR list for reference: 1. [Make OutputFormatAdapter public](https://github.com/openjdk/jmh/pull/158) 2. [Enable BenchmarkParams construction to be overriden](https://github.com/openjdk/jmh/pull/160) ------------- Commit messages: - Enable BenchmarkParams construction to be overriden Changes: https://git.openjdk.org/jmh/pull/160/files Webrev: https://webrevs.openjdk.org/?repo=jmh&pr=160&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7904058 Stats: 2 lines in 2 files changed: 0 ins; 0 del; 2 mod Patch: https://git.openjdk.org/jmh/pull/160.diff Fetch: git fetch https://git.openjdk.org/jmh.git pull/160/head:pull/160 PR: https://git.openjdk.org/jmh/pull/160 From galder at openjdk.org Tue Jul 8 09:51:07 2025 From: galder at openjdk.org (Galder =?UTF-8?B?WmFtYXJyZcOxbw==?=) Date: Tue, 8 Jul 2025 09:51:07 GMT Subject: RFR: 7904059: Enable alternative Runner instantiation and Main reuse Message-ID: This is the **third** in a series of PRs that I will be sending over the next few days, which make extending JMH to benchmark Java code running as GraalVM native images more easily. The codename for this extension is Fibula. It can still work without this PRs but without them the extension has to duplicate code that exists in JMH, which is not desirable. This PR refactors the main class to enable reuse of most of its logic, while allowing the construction of the `Runner` instance to be overridden by a subclass. An example of this override can be seen [here](https://github.com/galderz/fibula/commit/11bb06a3e30cfae61fb6d9319ee9cac1bf16d2d3). Here is the PR list for reference: 1. [Make OutputFormatAdapter public](https://github.com/openjdk/jmh/pull/158) 2. [Enable BenchmarkParams construction to be overriden](https://github.com/openjdk/jmh/pull/160) 3. [Enable alternative Runner instantiation and Main reuse](https://github.com/openjdk/jmh/pull/161) ------------- Commit messages: - Enable alternative Runner instantiation and Main reuse Changes: https://git.openjdk.org/jmh/pull/161/files Webrev: https://webrevs.openjdk.org/?repo=jmh&pr=161&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7904059 Stats: 72 lines in 1 file changed: 21 ins; 10 del; 41 mod Patch: https://git.openjdk.org/jmh/pull/161.diff Fetch: git fetch https://git.openjdk.org/jmh.git pull/161/head:pull/161 PR: https://git.openjdk.org/jmh/pull/161 From galder at openjdk.org Tue Jul 8 10:30:29 2025 From: galder at openjdk.org (Galder =?UTF-8?B?WmFtYXJyZcOxbw==?=) Date: Tue, 8 Jul 2025 10:30:29 GMT Subject: RFR: 7904060: Enable profiler classes to be reused outside of JMH Message-ID: This is the **fourth** in a series of PRs that I will be sending over the next few days, which make extending JMH to benchmark Java code running as GraalVM native images more easily. The codename for this extension is Fibula. It can still work without this PRs but without them the extension has to duplicate code that exists in JMH, which is not desirable. This PR makes `ProfilerOptionFormatter` and `ProfilerUtils` classes public so that they can be reused by external profilers. An example of this usage can be seen [in this perf dwarf profiler used for native image JMH benchmarks](https://github.com/galderz/fibula/commit/43fffcf225ab163a97cf0703abe956144adf31e6). Here is the PR list for reference: 1. [Make OutputFormatAdapter public](https://github.com/openjdk/jmh/pull/158) 2. [Enable BenchmarkParams construction to be overriden](https://github.com/openjdk/jmh/pull/160) 3. [Enable alternative Runner instantiation and Main reuse](https://github.com/openjdk/jmh/pull/161) 4. [Enable profiler classes to be reused outside of JMH](https://github.com/openjdk/jmh/pull/162) ------------- Commit messages: - Enable profiler classes to be reused outside of JMH Changes: https://git.openjdk.org/jmh/pull/162/files Webrev: https://webrevs.openjdk.org/?repo=jmh&pr=162&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7904060 Stats: 2 lines in 2 files changed: 0 ins; 0 del; 2 mod Patch: https://git.openjdk.org/jmh/pull/162.diff Fetch: git fetch https://git.openjdk.org/jmh.git pull/162/head:pull/162 PR: https://git.openjdk.org/jmh/pull/162 From galder at openjdk.org Tue Jul 8 11:17:03 2025 From: galder at openjdk.org (Galder =?UTF-8?B?WmFtYXJyZcOxbw==?=) Date: Tue, 8 Jul 2025 11:17:03 GMT Subject: RFR: 7904061: Enable JMH integration tests to be executed against other impls Message-ID: **NOTE**: This PR should be marked as draft because not all tests have been migrated to this setup. I've migrated the ones in the root test package to get a feel on how things would work and get feedback on whether this approach is fine for the JMH community. This is the **last** in a series of PRs extending JMH to benchmark Java code running as GraalVM native images more easily. In this PR: * We enable `jmh-core-it` test jar to be created so that JMH extensions can consume it. * We create a test `RunnerFactory` interface with a method called `createRunner` that by default creates JMH's `Runner` instance. JMH extensions that want to run the same test with a different runner would apply changes like [these](https://github.com/galderz/fibula/commit/79f48a8eabd38af4a4a023dacd2a925d8052aeea). Here the JMH extension extends the default `RunnerFactory` interface, and extends individual test classes to make them implement the extended runner factory interface. The result of both these changes enables JMH tests to be executed with the JMH extension: [INFO] --- failsafe:3.5.2:integration-test (default) @ fibula-jmh-it --- [INFO] Using auto detected provider org.apache.maven.surefire.junit4.JUnit4Provider [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running org.mendrugo.fibula.jmh.it.SingleShotTest # JMH version: fibula:999-SNAPSHOT # VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1 # *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong. # VM invoker: target/benchmarks ... [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.536 s -- in org.mendrugo.fibula.jmh.it.SingleShotTest [INFO] Running org.mendrugo.fibula.jmh.it.IterationCountCmdTest # JMH version: fibula:999-SNAPSHOT # VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1 # *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong. # VM invoker: target/benchmarks ... [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.712 s -- in org.mendrugo.fibula.jmh.it.IterationCountCmdTest [INFO] Running org.mendrugo.fibula.jmh.it.IterationCountAnnTest # JMH version: fibula:999-SNAPSHOT # VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1 # *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong. # VM invoker: target/benchmarks ... [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.345 s -- in org.mendrugo.fibula.jmh.it.IterationCountAnnTest [INFO] Running org.mendrugo.fibula.jmh.it.WarmupIterationCountAnnTest # JMH version: fibula:999-SNAPSHOT # VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1 # *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong. # VM invoker: target/benchmarks ... [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.751 s -- in org.mendrugo.fibula.jmh.it.WarmupIterationCountAnnTest [INFO] Running org.mendrugo.fibula.jmh.it.WarmupIterationCountCmdTest # JMH version: fibula:999-SNAPSHOT # VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1 # *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong. # VM invoker: target/benchmarks ... [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.756 s -- in org.mendrugo.fibula.jmh.it.WarmupIterationCountCmdTest [INFO] Running org.mendrugo.fibula.jmh.it.StackTraceInThrowableTest [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 s -- in org.mendrugo.fibula.jmh.it.StackTraceInThrowableTest [INFO] [INFO] Results: [INFO] [WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 1 Note that above we can observe that one of the JMH tests is disabled. It could happen that several JMH tests are not applicable to native image, e.g. perfasm tests. In the usage changes above we can observe how JUnit's `@Ignore` annotation can be easily used to disable particular tests as needed. In the particular case of `PrintFlags`, the JMH extension could filter this flag and not pass on (there are already such cases) so with that modification the test could pass but I left it as is so that test exclusion can be demonstrated for this PR. Here is the PR list for reference: 1. [Make OutputFormatAdapter public](https://github.com/openjdk/jmh/pull/158) 2. [Enable BenchmarkParams construction to be overriden](https://github.com/openjdk/jmh/pull/160) 3. [Enable alternative Runner instantiation and Main reuse](https://github.com/openjdk/jmh/pull/161) 4. [Enable profiler classes to be reused outside of JMH](https://github.com/openjdk/jmh/pull/162) 5. [Enable JMH integration tests to be executed against other impls](https://github.com/openjdk/jmh/pull/163) ------------- Commit messages: - Enable Runner instance to test to be provided by subclasses - Enable test reuse by generating the core IT test jar Changes: https://git.openjdk.org/jmh/pull/163/files Webrev: https://webrevs.openjdk.org/?repo=jmh&pr=163&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7904061 Stats: 64 lines in 8 files changed: 45 ins; 6 del; 13 mod Patch: https://git.openjdk.org/jmh/pull/163.diff Fetch: git fetch https://git.openjdk.org/jmh.git pull/163/head:pull/163 PR: https://git.openjdk.org/jmh/pull/163