Ability to add "jvm arguments" based on "@Param"

[Aleksey Shipilev]

On 06/05/2015 11:08 AM, Vladimir Sitnikov wrote:
> I want the following:  java -jar benchmarks.jar.
> It runs several combinations of the parameters, and for each
> combination it sets relevant operationsPerInvocation.

> I do like how set of parameters to test can be specified at the command line.
> I do not want reimplementing my own
> org.openjdk.jmh.runner.Runner#explodeAllParams
> I do want to get all the results in the same report table.

Is it really *that* hard?

    @Param("1")
    private int p1;

    @Param("1")
    private int p2;

    @Param("1")
    private int p3;

    @Benchmark
    public void test() throws InterruptedException {
        TimeUnit.MILLISECONDS.sleep(1);
    }

    public static void main(String... args) throws Exception  {
        CommandLineOptions opts = new CommandLineOptions(args);

        Collection<RunResult> rrs = new ArrayList<RunResult>();
        for (String p1 : opts.getParameter("p1").get()) {
            for (String p2 : opts.getParameter("p2").get()) {
                for (String p3 : opts.getParameter("p3").get()) {
                    int opi = Integer.valueOf(p1) +
                              Integer.valueOf(p2) +
                              Integer.valueOf(p3);

                    Options o = new OptionsBuilder()
                            .parent(opts)
                            .param("p1", p1)
                            .param("p2", p2)
                            .param("p3", p3)
                            .operationsPerInvocation(opi)
                            .build();

                    rrs.add(new Runner(o).runSingle());
                }
            }
        }

        ResultFormatFactory.getInstance(
            ResultFormatType.TEXT, System.out).writeOut(rrs);
    }


> On top of that, it would be great if jmh could use certain parameter
> as a "treatment" boolean, so it pivots "before"/"after" measurements
> to "before"/"after" columns. That's a bit different story, but I think
> it is rather simple/common use case.

This is the way it was envisioned long ago:
 1. Annotations are limiting for you? Use API.
 2a. Have an API problem, and you can work better if API provides
something more? Suggest to extend the API to expose more knobs. (More
likely to fly)
 2b. Have an API use case with the code that looks generic enough?
Suggest to include it into JMH core. (Less likely to fly)

The changes in JMH core are painful and time-consuming, they require
working out many mundane details, close the possibilities for misuse and
abuse, etc. This is why we keep JMH core arguably nice and tidy, *and*
provide APIs for the extension.

What you want is to introduce the intrusive change within the JMH core
without a compelling reason (yet). This is how it should be done: you
need a coherent argument stating "this is how it can be done now", "this
is what is minimally desireable", "this is what we would like to see at
best". All with code samples please. Handwaving has a drawback that most
kinks are not visible until you try to put it into the code...

Thanks,
-Aleksey