Execution of fixture methods in the case of error during benchmark.

[Mehdi Ennaïme]

> ...so the subsequent iteration within the trial would not run. The next
trial should run (presumably
> in newly forked VM), and @Setup(TRIAL) should run the.

I think this was the issue, since I was debugging other parts of the code
in my IDE, the number of forks was set to 0 :

It seems that if the benchmark trials are being run in the same VM, a
failure in a benchmark somehow breaks fixture methods, with 1 or more
forks, it seems to work as intended.
I've included a MCVE-like file reproducing the issue in this mail.

Thanks,
Mehdi Ennaïme

On Fri, Jun 4, 2021 at 12:34 PM Aleksey Shipilev <shade at redhat.com> wrote:

> On 6/4/21 11:30 AM, Mehdi Ennaïme wrote:
> > I'd like to have some information about the expected behaviour of the
> > fixture methods in the case a prior benchmark iteration threw an
> exception.
>
> Unexpected exception fails the entire trial...
>
> > The logs also seem to indicate that the parameters crossjoin is indeed
> benched.
> > While the parameters values seems to have been updated after a
> > "failed" run, the BinaryFunction I use are instantiated within a Setup
> > fixture method with the TRIAL level, which is not called anymore after
> > the first failure run, meaning the tested function is not
> > re-instantiated with the proper parameters, and thus all the trials
> > fail beyond that first failure.
>
> ...so the subsequent iteration within the trial would not run. The next
> trial should run (presumably
> in newly forked VM), and @Setup(TRIAL) should run the.
>
> If you need more precise answer, please provide the MCVE capturing your
> case.
>
> --
> Thanks,
> -Aleksey
>
>