Multiple JVMs, different numbers of threads and BenchmarkMode

[vyazelenko at yahoo.com]

> On Aug 5, 2014, at 11:56, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
> 
> On 08/05/2014 12:48 AM, Dmitry Vyazelenko wrote:
>>>> Now because of dynamic nature of the work done by a benchmark method
>>>> what would be the best BenchmarkMode to use and how the best to
>>>> configuration @Warmup/@Measurement iterations?
>>> 
>>> AverageTime or Throughput, obviously. I'm not sure how much of a problem
>>> that is, actually. We should not care how many times @Benchmark was
>>> called in throughput modes, we should only care it was called *enough*
>>> times.
>> Well, my use-case here is different one. I actually want to measure certain amount
>> of work/invocations. For example I want to benchmark how long it takes to add 1000
>> elements into my data structure. If I just rely on time-based modes like AverageTime
>> it means that my @Benchmark will be called unknown/unpredictable number of times.
> 
> If your workload is about doing 1000 operations over the map, then your
> @Benchmark should do 1000 operations with a subsequent clear()
Actually the workload is 1K, 10K, 100K...100M. And yes the lower the value the more iterations I'll need. And naturally the map.clear() or creation of new map as part of @Setup(Iteration). ;)
> 
>> If inside this method I put a loop which adds 1000 elements into collection then this
>> would not yield expected results (i.e. my collections will contain millions of entries and
>> not 1000).
> 
> The problem seems to be not with batching per se, but with the crud left
> after a particular batch. SingleShot would not help you much as well,
> since you would need @Setup(Iteration) with map.clear().
But it helps ensuring exact number of invocations of the benchmark method .
> 
>> Therefore I think that I should use @Warmup/@Measurment annotations with batchSize
>> defined and SingleShotTime mode, i.e. using similar technique shown in the 
>> JMHSample_26_BatchSize.java example. And of course I’ll have to use Java API to code
>> proper invocation of my benchmarks with batchSize set dynamically.
> 
> That... overcomplicates the issue, doesn't it? Having @Param with
> batchSize, @Benchmark method which does batchSize put/get-s, and the
> following clear() seems to do the same as well.
Well, I thought that I'll remove my own @Param with batchSize and rely on @Benchmark behavior. That would also remove loops from the benchmark methods and the need to use Blackhole class (I'll just return value from the method).

On the other hand if I remove my @Param then in the result table I won't see batchSize anymore. However since I'll use Java API anyway that won't be an issue.
> 
> Thanks,
> -Aleksey.
Best regards,
Dmitry