Custom and not-constant "Operations per invocation" (Re: JMH 1.15)

[Aleksey Shipilev]

On 10/03/2016 07:12 PM, Lev Serebryakov wrote:
> On 30.09.2016 22:06, Aleksey Shipilev wrote:
> 
>>  *) @AuxCounters improvements. Note this is still a very much
>> experimental API. That said, it makes sense to improve it a little more:
>> now it handles more field/method types, has new counting mode "events"
>> that does not normalize to time, and does not require reset between
>> iterations. This API change is backwards-compatible. See the RFE:
>>    https://bugs.openjdk.java.net/browse/CODETOOLS-7901810
>> ...and new Javadoc:
>  Looks like, it is way to implement custom "operations per invocation"
> metric, am I right?

No, it helps with auxiliary counters that need to count only the
specific paths in the workload. These counters are not supposed to be as
accurate as primary metric you get from the workload, and may only serve
as the additional bits of data, together with the primary results.

>  Problem I have at hands looks like this: I benchmark implementation,
> which could do batching of low-level operations. To make benchmark more
> "realistic" I choose size of batch on each invocation at random (really,
> from pre-filled big array of random numbers, as random generators, even
> bad ones, are slow). So, each benchmark invocation could be counted as
> different number of operations, from 1 to, say, 16. and default JMH's
> "average time per invocation" is not very meaningful in this situation,
> as I need to know time per basic operation.

You have to use Java API for this, and override
.operationsPerInvocation(...) to match your .params(...) if you want to
normalize the operations like that.

> Looks like "@AuxCounters" is solution, am I right? I only need to count
> real number of basic operations, and I'm done!

No, this is not a reliable solution. As said in both the changelog and
the Javadoc, it is a very experimental API. The reliable solution
measures the performance with primary metrics, and adjusts it accordingly.

Thanks,
-Aleksey