Multiple JVMs, different numbers of threads and BenchmarkMode

[Aleksey Shipilev]

On 08/05/2014 12:48 AM, Dmitry Vyazelenko wrote:
>>> Now because of dynamic nature of the work done by a benchmark method
>>> what would be the best BenchmarkMode to use and how the best to
>>> configuration @Warmup/@Measurement iterations?
>>
>> AverageTime or Throughput, obviously. I'm not sure how much of a problem
>> that is, actually. We should not care how many times @Benchmark was
>> called in throughput modes, we should only care it was called *enough*
>> times.
> Well, my use-case here is different one. I actually want to measure certain amount
> of work/invocations. For example I want to benchmark how long it takes to add 1000
> elements into my data structure. If I just rely on time-based modes like AverageTime
> it means that my @Benchmark will be called unknown/unpredictable number of times.

If your workload is about doing 1000 operations over the map, then your
@Benchmark should do 1000 operations with a subsequent clear().

> If inside this method I put a loop which adds 1000 elements into collection then this
> would not yield expected results (i.e. my collections will contain millions of entries and
> not 1000).

The problem seems to be not with batching per se, but with the crud left
after a particular batch. SingleShot would not help you much as well,
since you would need @Setup(Iteration) with map.clear().

> Therefore I think that I should use @Warmup/@Measurment annotations with batchSize
> defined and SingleShotTime mode, i.e. using similar technique shown in the 
> JMHSample_26_BatchSize.java example. And of course I’ll have to use Java API to code
> proper invocation of my benchmarks with batchSize set dynamically.

That... overcomplicates the issue, doesn't it? Having @Param with
batchSize, @Benchmark method which does batchSize put/get-s, and the
following clear() seems to do the same as well.

Thanks,
-Aleksey.