A few questions / feature requests

[Julien Nicoulaud]

Hi,

First all thanks for the hard work on JMH, it's making fast progress and it
really has changed the way I work on my code :)

Here is a few questions/feature requests after using it for a while:

1) "Run until stable" mode
It would be great if JMH add a mode where it runs more iterations of the
benchmark until mean error/confidence interval is under X. I often find
myself spending a lot of time tuning the number and duration of
warmup+measurement iterations to find the right setup.

2) Manually flush CPU cache
Let's say I'm benchmarking an iterator(), my microbenchmark usually looks
something like this:

@GMB
@OperationsPerInvocation(MY_ITERATOR_SIZE)
public void benchmark() {
    while (it.hasNext())

         myBlackHole.consume(it.next());

}

So the microbenchmark benefits of cache effects, which does not happen on a
real application... It would be great if I could just do:

@GMB
@OperationsPerInvocation(MY_ITERATOR_SIZE)
public void benchmark() {
    while (it.hasNext()) {

                  myBlackHole.consume(it.next());
                  myBlackHole.flushL1L2L3();

    }

}

Or am I just doing it wrong ?

3) Annotation processor and incremental compilation
The annotation processor replaces META-INF/MicroBenchmarks at each
compilation. In Eclipse (and other IDEs, I guess), compilation is
incremental so the file only contains the last benchmark class I modified...

One cheap way to deal with this would be to just write it in append mode
and silently ignore duplicates or deleted classes at runtime, what do you
think ?

4) OperationsPerInvocation is too static
OperationsPerInvocation value must be a constant, but in some cases, it
would be nice if it could be set at runtime / per invocation, with somthing
similar as @Counters (eg: my previous iterator example)

5) GC control
IMHO the current set of options does not make it very clear when the GC is
invoked by the framework (at each trial/iteration/invocation ? before/after
setup ?) and what does it do exactly (simple System.gc() ? Several
System.gc() ? Some hidden Oracle API to force GC for real ?). I think it is
in the scope of JMH to provide a clear and safe way to control this to
users.

Regards,
Julien