Exception in Benchmark isn't marked as failure in CSV file.

[Aleksey Shipilev]

Hi Samuel,

Thanks for the bug report!

On 08/21/2014 09:52 PM, Samuel Bishop wrote:
> I've been reading the provided JMH samples, and I see that in sample 2,
> that JMH will stop measuring if an exception is thrown.  To test this I
> wrote a small benchmark to intentionally throw an exception when given a
> particular parameter. I was surprised to see that the CSV file didn't mark
> that anything had gone wrong when given my "bad" parameter.

This affects not only the CSV file, but also any other facility that
feeds on JMH results, including the final table in the human-readable
output.

Notice the result is not produced for a faulty benchmark at all, which
is the easiest of the sane things for harness to do. Too bad users have
to refer to the benchmark log to figure out what went wrong, and why the
result is missing.

I have looked into printing some human-readable message in the cases
like this, and concluded the change is too disruptive for current
release candidate. It also adds to the bucket of fixes we have to have
for treating the failing benchmarks well. Since failing benchmarks are
not the things we want to measure anyway, we will revisit this behavior
after 1.0 ships.

Thanks,
-Aleksey.