Execution of fixture methods in the case of error during benchmark.

[Mehdi Ennaïme]

Hi,

I'd like to have some information about the expected behaviour of the
fixture methods in the case a prior benchmark iteration threw an exception.

I'm currently benchmarking several BinaryFunction implementations with
various input parameters, and in some cases the execution of a given
function under some parameters is expected to throw an
UnsupportedOperationException
.

I initially assumed that specifying the following in the run Options :

shouldFailOnError(false)

would lead to the execution of the benchmark for the crossjoin of
parameters, with simply the runs throwing leaving no output in the
final report.

The logs also seem to indicate that the parameters crossjoin is indeed benched.
While the parameters values seems to have been updated after a
"failed" run, the BinaryFunction I use are instantiated within a Setup
fixture method with the TRIAL level, which is not called anymore after
the first failure run, meaning the tested function is not
re-instantiated with the proper parameters, and thus all the trials
fail beyond that first failure.


Is this an expected behaviour (as in "failing" states are not supposed
to be used inside a parameterized benchmark) ?

Thanks,
Mehdi Ennaïme