JMH hangs between benchmarks (unable to kill forked JVM?)

[Leszek Bartkowski]

Hello,

I am encountering weird issue while running bunch of my JMH
benchmarks. It looks like after executing few it gets stucked on
closing forked JVM. The issue does not occur if I launch any of those
benchmark separately.

I have a large set of macrobenchmarks that measure execution time of
various simulations using different execution engines (some of them
are heavily multi-threaded). Most of these jobs use jmh settings that
I specify in abstract base class as:

@ContextConfiguration(value = AbstractJobExecutorBenchmarker.CONTEXT_LOCATION)
@State(Scope.Benchmark)
@Fork(1)
@BenchmarkMode(Mode.SingleShotTime)
@Warmup(iterations = 1, timeUnit = TimeUnit.MILLISECONDS)
@Measurement(iterations = 5, timeUnit = TimeUnit.MILLISECONDS)
public abstract class AbstractJobExecutorBenchmarker extends
AbstractTestNGSpringContextTests

(I also add -gc -si false to command line)

When I fire any single one of those benchmarks or some small subset it
all works fine. However if I try to run all of them in one go (i.e. no
filtering options), jmh succesfully finishies few of them and then
suddenly frezees after one specific benchmark... alli terations for it
are done, final results are printed, but it doesn't look like forked
JVM was killed and new isn't starting, seems like whole process is
stucked. Example of last statements printed:

Iteration   3: 03:59:52.059 [pool-8-thread-1] INFO
MessageTrafficController: Starting MessageTrafficController
03:59:52.059 [pool-8-thread-3] INFO  MessageTrafficController:
Starting MessageTrafficController
03:59:52.060 [pool-8-thread-5] INFO  MessageTrafficController:
Starting MessageTrafficController
03:59:52.060 [pool-8-thread-6] INFO  MessageTrafficController:
Starting MessageTrafficController
03:59:52.061 [pool-8-thread-7] INFO  MessageTrafficController:
Starting MessageTrafficController
03:59:52.061 [pool-8-thread-8] INFO  MessageTrafficController:
Starting MessageTrafficController
03:59:52.061 [pool-8-thread-9] INFO  MessageTrafficController:
Starting MessageTrafficController
03:59:52.061 [pool-8-thread-10] INFO  MessageTrafficController:
Starting MessageTrafficController
03:59:52.062 [pool-8-thread-13] INFO  MessageTrafficController:
Starting MessageTrafficController
03:59:52.062 [pool-8-thread-14] INFO  MessageTrafficController:
Starting MessageTrafficController
03:59:52.063 [pool-8-thread-15] INFO  MessageTrafficController:
Starting MessageTrafficController
03:59:52.063 [pool-8-thread-18] INFO  MessageTrafficController:
Starting MessageTrafficController
03:59:52.063 [pool-8-thread-19] INFO  MessageTrafficController:
Starting MessageTrafficController
03:59:52.064 [pool-8-thread-20] INFO  MessageTrafficController:
Starting MessageTrafficController
03:59:52.064 [pool-8-thread-22] INFO  MessageTrafficController:
Starting MessageTrafficController
03:59:52.065 [pool-8-thread-25] INFO  MessageTrafficController:
Starting MessageTrafficController
04:00:57.600 [pool-8-thread-6] WARN  MessageTrafficController: Stopped
due to class java.lang.InterruptedException
04:00:57.601 [pool-8-thread-7] WARN  MessageTrafficController: Stopped
due to class java.lang.InterruptedException
04:00:57.604 [pool-8-thread-13] WARN  MessageTrafficController:
Stopped due to class java.lang.InterruptedException
04:00:57.604 [pool-8-thread-10] WARN  MessageTrafficController:
Stopped due to class java.lang.InterruptedException
04:00:57.603 [pool-8-thread-8] WARN  MessageTrafficController: Stopped
due to class java.lang.InterruptedException
04:00:57.604 [pool-8-thread-9] WARN  MessageTrafficController: Stopped
due to class java.lang.InterruptedException
04:00:57.606 [pool-8-thread-22] WARN  MessageTrafficController:
Stopped due to class java.lang.InterruptedException
04:00:57.608 [pool-8-thread-25] WARN  MessageTrafficController:
Stopped due to class java.lang.InterruptedException
04:00:57.606 [pool-8-thread-20] WARN  MessageTrafficController:
Stopped due to class java.lang.InterruptedException
04:00:57.605 [pool-8-thread-15] WARN  MessageTrafficController:
Stopped due to class java.lang.InterruptedException
04:00:57.606 [pool-8-thread-19] WARN  MessageTrafficController:
Stopped due to class java.lang.InterruptedException
04:00:57.606 [pool-8-thread-18] WARN  MessageTrafficController:
Stopped due to class java.lang.InterruptedException
04:00:57.604 [pool-8-thread-14] WARN  MessageTrafficController:
Stopped due to class java.lang.InterruptedException
65598.920 ms

Result : 76749.529 Âą(99.9%) 804946.698 ms
  Statistics: (min, avg, max) = (39272.712, 76749.529, 125376.954),
stdev = 44121.845
  Confidence interval (99.9%): [-728197.169, 881696.226]

If I kill forked JVM manually the whole process unblocks. However
after finishing next benchmark it gets stuck again... Any ideas why
does it happen? And why it does NOT happen when I fire off the same
benchmark separately?

Also, is there some timeout option in JMH, so it kills forked process
forcefully if normal shutdown takes too long?

I am using version 0.3 and problem occurs on both Windows and Linux
machine. However it looks like my build server one time managed to
execute all those benchmarks without problems. This seems like pure
luck though, because every other time I try it always gets stuck in
the same way.