RFR: Allow setting of opsPerInvocation from setup functions

[Aleksey Shipilev]

On Fri, 17 Mar 2023 12:54:38 GMT, bbrehm <duke at openjdk.org> wrote:

> In many cases, opsPerInvocation is known at the @Setup function, but cannot be reasonably known at any other point. This example is somewhat lame in that regard; I actually needed that for some not-quite-micro benchmarks (the setup function loads real-ish data from disk, and the opsPerInvocation depends on the data that has been loaded).
> 
> Since every good example teaches multiple things, I added one that also warns about the branch history table stitching together surprisingly long patterns.
> 
> The super-duper correct way of doing this example (cf https://discourse.julialang.org/t/psa-microbenchmarks-remember-branch-history/17436) would be to have constant opsPerInvocation and one large array; then a smaller (@Param) array is copied into that multiple times. Thus we could exclude effects of memory hierarchy. That would look quite unnatural and harder to relate to "normal looking" code (and there are no effects of memory anyway, since first, the benchmark is too slow to saturate DRAM bandwidth, and second even the largest set fits into L2).

Interesting example!

However, the benchmark parameters (like all other params) are supposed to be immutable for the run. It is pure luck that all uses of `opsPerInvocation` happen after `@Setup`, and thus calculations are not subtly broken. This is not actually guaranteed. Trying to reconcile what to do when the benchmark code mutates its own configuration is a much larger task. (Things to think about: Do we allow to set `opsPerInvocation` multiple times during the run? Do we allow doing this in `@Setup(Level.Invocation)`?)

The usual answer for things that cannot be captured in declarative form via annotations is going to Java API and doing it in imperative form there. For your example, it would be something like (drafting from memory, might not readily compile):


    public static void main(String[] args) throws RunnerException {
        Options parent = new OptionsBuilder()
                .include(".*" + JMHSample_40_InfraParamNormalization.class.getSimpleName() + ".*")
                .addProfiler(LinuxPerfNormProfiler.class)
                .build();

        for (int size : new int[] { 100, 1000, 5000, 10000, 100000 }) {
            Options opts = new OptionsBuilder()
                  .parent(parent)
                  .param("size", String.valueOf(size))
                  .operationsPerInvocation(size)
                  .build();
             new Runner(opt).run();
        }
    }

-------------

PR Comment: https://git.openjdk.org/jmh/pull/97#issuecomment-1480941620