"Nice" report for failed forks/benchmarks

[Aleksey Shipilev]

On 06/02/2015 09:49 PM, Vladimir Sitnikov wrote:
> I happen to test some finalizer performance, and it looks like jmh's
> report is not very nice there.

Yes, it is tracked by:
  https://bugs.openjdk.java.net/browse/CODETOOLS-7901010

> Does anybody have idea how that should be depicted?
> I guess it might need some un-biasing in case just a couple of runs
> fails. For instance, if 3 of 5 forks survive we would have to average
> apples with exceptions somehow.

I'd think if benchmark fails, then the result should be marked as
failure as well. No recovery is possible.

> # Warmup Iteration   4: 202029,343 ns/op
> # Warmup Iteration   5: 846066,057 ns/op
> # Warmup Iteration   6: <failure>
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> 
> ...
> 
> # Run complete. Total time: 00:00:18
> 
> Benchmark  Mode  Cnt  Score   Error  Units

Should probably be:

Benchmark          Mode  Cnt  Score     Error  Units
MyBenchmark.test   avgt       <failure>

Thanks,
-Aleksey