A few questions / feature requests

[Aleksey Shipilev]

Hi Julien,

On 06/10/2014 09:06 PM, Julien Nicoulaud wrote:
> 1) "Run until stable" mode
> It would be great if JMH add a mode where it runs more iterations of the
> benchmark until mean error/confidence interval is under X. I often find
> myself spending a lot of time tuning the number and duration of
> warmup+measurement iterations to find the right setup.

I am a firm disbeliever these adaptive schemes converge reliably,
especially in the presence of several warmup plateaus. It would be bad
for people trusting JMH to do this reliably, and realize it trips over
in their interesting cases.

Therefore, I still maintain we should *push* people to see how their
warmup "profile" looks for their particular workload, and make their
call how many warmup/measurement iterations should happen.

> 2) Manually flush CPU cache
> @GMB
> @OperationsPerInvocation(MY_ITERATOR_SIZE)
> public void benchmark() {
>     while (it.hasNext()) {
>                   myBlackHole.consume(it.next());
>                   myBlackHole.flushL1L2L3();
> 
>     }
> }
> 
> Or am I just doing it wrong ?

I would say this lies out of the scope for core JMH. Arguably, we set
the precedent with Blackhole.consumeCPU, but in retrospect, we might
want to have jmh-utils subproject with all these useful tricks.

However, I fail to see how would one implement such a thing on modern
hardware. The best we know about x86 is "WBINV <addr>", which basically
requires us to walk the entire address space to invalidate LLC.
memset-ting the memory via e.g. Array.fill or System.arraycopy would
potentially use non-temporal stores, and cache is unaffected. Heck, even
the manual copying loop may get recognized as arraycopy, and made
non-temporal. Etc, etc, etc.

> 3) Annotation processor and incremental compilation
> The annotation processor replaces META-INF/MicroBenchmarks at each
> compilation. In Eclipse (and other IDEs, I guess), compilation is
> incremental so the file only contains the last benchmark class I modified...

Ouch. Do we really have to support incremental benchmark compilation?
Can you get around this by requiring the full compilation before running
the tests from IDE?

> One cheap way to deal with this would be to just write it in append mode
> and silently ignore duplicates or deleted classes at runtime, what do you
> think ?

While it seems doable on generator side, the filtering on runner side is
complicated. Hint: what happens when you _delete_ the benchmark in your
IDE session? The benchmark list entry is still there in the naive
"append" approach, but trying to run it crashes JMH. One could try to
plumb that leak and recover nicely, but really, this is an important
internal error in all other cases, and should be treated as such.

> 4) OperationsPerInvocation is too static
> OperationsPerInvocation value must be a constant, but in some cases, it
> would be nice if it could be set at runtime / per invocation, with somthing
> similar as @Counters (eg: my previous iterator example)

OptionBuilder.operationsPerInvocation(int) is available in API already
(first available in 0.9).

> 5) GC control
> IMHO the current set of options does not make it very clear when the GC is
> invoked by the framework (at each trial/iteration/invocation ? before/after
> setup ?) and what does it do exactly (simple System.gc() ? Several
> System.gc() ? Some hidden Oracle API to force GC for real ?). I think it is
> in the scope of JMH to provide a clear and safe way to control this to
> users.

There is no "GC control" in JVM, and therefore, we can't make any of
such promises, like we did with @CompileControl.

-Aleksey.