Benchmarking with heavily-assymetric threads? (full message)

[Aleksey Shipilev]

Hi Lev,

Sorry for a very late reply!

On 11/19/2015 03:55 PM, Lev Serebryakov wrote:
>  I'm using @Group("name") feature of JMH to start different threads in
> one benchmark, I don't start threads by myself. But I'm blocking
> benchmarked methods, as I try to benchmark different implementations
> and strategies for blocking!
> 
>  Benchmark is rather simple (see at the end of message complete
> example, you could not run it as you don't have my buffer
> implementation, but it is straightforward *blocking* ring buffer for
> single producer and single consumer).
> 
>  Now I'm understanding, that there is one major problem: if one of
> group threads stops slightly before other one (perfectly possible),
> other benchmark method could block forever (till JMH timeout), because
> it will not have data to "consume" (or space in buffer to "produce").

Unfortunately, there is no way to properly solve this. JMH does not
"equalize" the number of benchmark method calls per subtests in most
measurement modes. Therefore, if you need the rendezvous tests that
should be perfectly matched, you are in the bind.

When faced with the problems like these, it is very instructive to ask
yourself: how would you do this without JMH? In other words, how exactly
would you want to measure things?

There are few approaches:

 a) Count reads and writes in both threads, and run until both counts
are matched. If readers and writers are symmetrical (heavily unlikely),
then you just run until time expires, and pray all pairs rendezvous
perfectly well. If they are not, then a hard cap on reader/writer counts
should be imposed, otherwise it would never converge. This can be
simulated with SingleShot mode with large batch sizes [1].

 b) Run without counting, and shutdown the parties in orderly fashion.
E.g. shutdown readers first, then shutdown writers -- that will avoid
the trouble you are describing. But, this trick cannot be applied
consistently. I.e. if the queue/ring buffer storage is bounded and
writers can block, you have to shutdown *writers* first. Oops.

 c) Run without counting, but make readers/writers to respond to
interrupts, and break away gracefully once interrupt is caught. This is
what JMHSample_30_Interrupts [2] sample tries to show. Those
"interrupted" messages is JMH trying to communicate to workload that
time is up, and workload should exit @Benchmark right now. (Technical
detail: interrupts are usually received in ramp-down catchup iterations,
and should have no effect on measurement).

Thanks,
-Aleksey

[1]
http://hg.openjdk.java.net/code-tools/jmh/file/bcec9a03787f/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_26_BatchSize.java
[2]
http://hg.openjdk.java.net/code-tools/jmh/file/bcec9a03787f/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_30_Interrupts.java