Iteration/Warmup Timeout (was: Using jmh.shutdownTimeout)

[Aleksey Shipilev]

Hi Behrooz,

Thanks for the investigation, I will take a look next week.

Cheers,
-Aleksey

On 07/14/2015 10:37 PM, Behrooz Nobakht wrote:
> Hello again,
> 
> A bit of delay from me on this. Sorry.
> However, I've been trying to work around this challenge
> on and off trying and failing every and each one of ideas.
> 
> My last approach is as follows that actually helped to get
> into a better situation.
> 
> I started to use `-Djmh.executor=CUSTOM` and then 
> my custom ExecutorService and pass it in via `-Djmh.executor.class`.
> 
> As it can be seen from the implementation, it is basically just redundant
> implementation at the level of ThreadFactory and ExecutorService extension.
> There are two parts in this that help me:
> 
> - Custom logic in `CustomThread#interrupt` that first calls a static
> singleton
> `Runnable` as my "interruption handler" during the benchmarks.
> 
> - And, then my customer "interruption handler" which is basically a
> singleton
> Runnable instance such as:
> 
> static final Runnable INTERRUPT_HANDLER = new Runnable() {
>   private final AtomicBoolean interrupted = new AtomicBoolean(false);
> 
>   @Override
>   public void run() {
>     try {
>       if (interrupted.compareAndSet(false, true)) {
>         // DEAL WITH MY HANGING THREADS
>       }
>     } catch (Exception e) {
>       System.err.println(e.getMessage());
>     }
>   }
> }
> 
> Now, considering the above, could the following make sense to be added
> to JMH:
> 
> - Add a new system property as: `jmh.timeout.interrupt.handler`
> This can be also be a parameter for @Timeout
> 
> - The value for the above should be the FQN of a class implementing
> `Runnable`.
> 
> - The above means that ThreadFactory implementation at JMH should
> provide an extension of Thread that overrides #interrupt().
> 
> - When timeout is caught in BenchmarkHandler, and so #interrupt() is
> called through BenchmarkTask, the above property comes into the 
> pictures and "optimistically" handles this timeout by ensuring the 
> application threads are stopped.
> 
> The downside of this approach is that the "interrupt handler" Runnable
> should be able to access a static context to handler the hanging threads
> during the benchmark.
> 
> As another side effect of diving into this, I am not sure but I think this 
> can be a bug in Java's `ForkJoinWorkerThread`. Basically, unless instances
> of this type of thread are not interrupted, shutting Executors at JVM
> runtime
> does not kill those threads. So, I also had to explicitly interrupt all
> the threads
> of type ForkJoinWorkerThread so that my JVM can cleanly shut down.
> 
> If this makes sense, I'm willing to provide a patch on this for JMH.
> 
> Thanks,
> Behrooz
> 
> P.S. I really wish I could "watch" those JIRA tickets.
> 
> 
> 
> On Thu, May 28, 2015 at 10:28 AM, Aleksey Shipilev
> <aleksey.shipilev at oracle.com <mailto:aleksey.shipilev at oracle.com>> wrote:
> 
>     On 26.05.2015 18 <tel:26.05.2015%2018>:05, Behrooz Nobakht wrote:
>     > - Based on the benchmark configuration I expect that each iteration is
>     > allowed to be executed at most 1 second.
>     > Is this not correct?
> 
>     Current timeout mechanics only issues the Thread.interrupt to the
>     running threads. We can't do much better, since we can't magically stop
>     the non-cooperating thread stuck in @Benchmark. We can crash the forked
>     VM though, as alluded here:
>       https://bugs.openjdk.java.net/browse/CODETOOLS-7901376
> 
>     > - If not, what should I use to have control over the iteration/warmup
>     > explicit ending?
> 
>     You nominally have to make sure @Benchmark call finishes in reasonable
>     time. When you can't, e.g. it requires coordination for multiple
>     @Benchmark calls (mostly in asymmetric scenarios), you have to check for
>     interrupts in your @Benchmark code.
> 
>     > - When I add @Threads(N) to the benchmark configuration, although I expect
>     > exceptions happen due to shutdown of threads (and catch them at a higher
>     > level),
>     > yet the benchmark fails to make progress to next iterations/warmups.
>     > Any advice?
> 
>     This might be an instance of:
>      https://bugs.openjdk.java.net/browse/CODETOOLS-7901008
> 
>     It would be nice to see if you have another case that should be handled.
> 
>     Thanks,
>     -Aleksey
> 
> 
> 
> 
> -- 
> -- Behrooz Nobakht