Auxiliary counters incremented during warmup/warmdown?

[Yoann Rodiere]

Hello,

I've been trying to use auxiliary counters (@AuxCounters) as part of a JMH
benchmark for the last few days, and I keep getting a throughput that is
approximately three times the value I would expect.

My benchmark is a bit unusual: the benchmark method starts many
asynchronous operations, then waits for the operations to finish. As a
result, it takes quite a long time to run, more than the time limit I set
on the benchmark. Thus, the method is executed exactly three times for each
iteration and each thread: once during warmup, once to measure the
execution time, and once during warmdown.

After a bit of digging, I noticed that, while execution time is not taken
into account during warmup/warmdown, auxiliary counters are still
incremented, and increments during warmup/warmdown are not ignored in the
final result.

This can be observed easily in the samples. This benchmark method [1]:

    @Benchmark
    public void splitBranch(OpCounters counters) {
        if (Math.random() < 0.1) {
            counters.case1++;
        } else {
            counters.case2++;
        }
    }

... generates a class where the method is executed during warmup,
measurement and warmdown with the *same* counter instance [2], even though
only the measurement time is used when computing throughput [3].

My question is: is this on purpose? Am I not supposed to increment counters
during warmup/warmdown, and if so how?

If it's a bug, I think generating two instances of the counter class for
each iteration, then using one for warmup/warmdown and the other for
measurement only, would fix it.

Thanks!

[1]
http://hg.openjdk.java.net/code-tools/jmh/file/a128fd4a5901/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_23_AuxCounters.java#l91
[2]
https://gist.github.com/yrodiere/5e6d64b84a83a826f3b00a61a2e1d88b#file-jmhsample_23_auxcounters_splitbranch_jmhtest-java-L145,L172
[3]
https://gist.github.com/yrodiere/5e6d64b84a83a826f3b00a61a2e1d88b#file-jmhsample_23_auxcounters_splitbranch_jmhtest-java-L187

Yoann Rodière
Hibernate NoORM Team
yoann at hibernate.org