JMH does not terminate on error

[Aleksey Shipilev]

On 09/08/2016 05:41 PM, Alex Averbuch wrote:
> I have a benchmark that throws an exception. In spirit, it does this:
> 
>     @Benchmark
>     @BenchmarkMode( {Mode.SampleTime} )
>     public void doError( )
>     {
>             throwsException();
>     }
> 
> Via OptionsBuilder, Runner has been configured to fail as soon as it
> encounters an error, like this:
> 
>   ChainedOptionsBuilder baseBuilder = new OptionsBuilder()
>                 .shouldFailOnError( true )
>                 .timeout( TimeValue.hours( 1 ) )
>                 ...
> 
> Note also that timeout is 1 hour <-- this is important to the behavior I am
> seeing, I think.

Note that shouldFailOnError concerns the exit code and/or continuing the
execution after some benchmark had failed. This does not override the
hardcoded behavior w.r.t. @Benchmark exceptions: if it throws, then it
is a benchmark error.


> When running the doError() benchmark it hangs.
> It hangs for a very long time. I suspect it would be for one hour, but have
> not had the patience to wait.
> Instead, I changed the timeout to 10 seconds, which resulted in the
> benchmark failing (as expected) in ~10 seconds.
> 
> Is this intended behavior?
> 
> It feels like a bug. At the least, it is non-intuitive.

AFAIU, there are several facets to the problem:

 a) If the benchmark is throwing the exception, we still unconditionally
sleep for the iteration time; we should instead ask all threads to stop,
and throw the exception early;

 b) If some benchmark thread does not cooperate with the regular
protocol -- e.g. checking the "isDone" flag, which requires exiting from
@Benchmark, we can try to interrupt it. This is where timeout mechanics
comes in: we would start issuing interrupts after the timeout exceeds.

 c) With (b), there is an interesting corner case: if some benchmark
thread had failed, and another one should have rendezvoused with it to
exit its own @Benchmark, then we would wait for timeout to expire to
kill the workload. But if one thread is failing, then benchmark score is
irrelevant, and we might as well start interrupting without waiting for
the timeout.

All these issues should be alleviated with this fix:
  https://bugs.openjdk.java.net/browse/CODETOOLS-7901789

Would you mind testing it?

Either way, if your @Benchmark cannot quit either by itself, or after
continuous bashing with Thread.interrupt(), then we will block. That is,
we will never finish the benchmark that has astray benchmark threads, as
it is a benchmark design error.

Thanks,
-Aleksey