New benchmark modes

[Aleksey Shipilev]

On 11/16/2016 12:04 PM, Nitsan Wakart wrote:
> 1. Histogram: This is similar to SampleTime, but I would like to always
> measure.
>   I realize that pre/post timestamps may not be a good idea for small
> workloads, and that nanoTime doesn't scale.
>   This is so similar to Sample that one could argue for making this a
> configurable mode of SampleTime (e.g add controls for sample probability
> and max samples per ms), I'd be happy with that solution too.

SampleTime is supposed to capture all the samples if workload is light
(e.g. if we do the timestamps no more than once each 50 us). You could
probably try to to disable this behavior to capture all the samples. See
how "targetSamples" is calculated in the generated stub.

>   I would also like to have the timing data recorded in HdrHistogram.
> Not that there's anything wrong with the JMH histogram, they are very
> similar, but I'd like to serialize the data in the HdrHistogram format
> to support downstream processing with other tools.

I think we need to separate the runtime representation (SampleBuffer)
and on-the-wire format. Not sure we want to introduce HdrHistogram just
for output serialization, because that blocks us from tuning up
SampleBuffer that *is* on measurement hotpath. Probably dumping the raw
data from SampleBuffer in the format that is similar to HdrHistogram
format is better -- see how JSONResultFormat.PRINT_RAW_DATA works.

> 2. Fixed rate + Histogram (response+service): JMH currently supports
> only an all-out measurement mode. I'd like to have a fixed operation
> rate mode where a given operation is executed X times a second.
>   The measurement collected is both for response time(end time -
> scheduled start time) and service time(end time - actual start time).
> This to avoid coordinated omission in measurement.

Yup. Adding a new benchmark mode should be easy. The only troubling
thing here is that you need to pass in the rate parameter somehow...

> I realize benchmark modes and the code generation they require is
> fundamental to the workings of JMH, I'm not sure this part can be opened
> up to extension such that the above benchmark modes can be added as
> extensions rather than included in the JMH codebase. If you see a clear
> path to openning up JMH for extension that way it might be easier to
> allow users more freedom to experiment with custom benchmark modes as
> above while retaining the benefits of the rest of JMH.

No. We would not make pluggable benchmark modes. The core of JMH trust
and benefit is that the existing benchmark modes scaffolding is heavily
vetted, stable, bugfixed and in our ultimate control. In other words,
this is where our beef is: we provide the reliable scaffolding, users
provide the workload code.

Thanks,
-Aleksey