AuxCounters & benchmark modes

[Nitsan Wakart]

For what it is worth, I'm a fan of AuxCounters.
Here are 2 use cases I found, perhaps they can find their way into the grand JMH vision:
1. Counting failed poll/offer on queue throughput benchmark.
2. Integrating hardware performance counters
I like the @Units annotation, it's would be nice.



On Monday, March 17, 2014 7:36 PM, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
 
On 03/14/2014 04:28 PM, Dmitry Chuyko wrote:

> Current AuxCounters's javadoc says: "NOTE: This functionality is not
> available for all {@link BenchmarkMode}-s". It works in throughput mode
> but not for singleshot (-bm ss) mode. Maybe it makes sense to allow all
> modes. For example consider a state that collects some system statistics
> after each invocation.

I think @AuxCounters are leaky abstraction, and we should not really try
to get more from them (to the extent we would probably eliminate them
before 1.0 release). They leak "how" to measure, not "what" to measure,
which is a bad API design. @AuxCounters interpretation and
implementation is trivial only for throughput/averagetime modes: these
are the related to the number of @GMB calls the run makes.

In other modes, what does @AuxCounter mean? In "sample time" and "single
shot", this can naively be the time spend executing some part of @GMB
call, which opens another can of worms: *how* you get that data? users
do timestamps? how do these measurement correlate with JMH sampling?

Making @AuxCounters to carry something not directly related to primary
metrics makes another horrible API choice: it is *very* unclear what
those secondary metrics are, what units they carry, what are their
aggregation policies, whether we can ignore some of the values or not,
etc. That being said, maybe there is a way to introduce secondary
metrics to the benchmark which still does not pollute the API...
@AuxCounters is not that way, unfortunately.

-Aleksey.