Setup time is counted against iteration time in Mode.SingleShotTime (JMH-1.0)

[Aleksey Shipilev]

Hi,

On 09/01/2014 06:48 PM, Dmitry Vyazelenko wrote:
> I've discovered that when Mode.SingleShotTime is used the time in @Setup
> is counted against iteration time. This results in benchmark being interrupted
> if @Setup takes long time. 

Yes, there is the implementation reason for that: @Setup is executed by
the generated code, and so the harness control thread is oblivious of
the time spent in @Setup vs. @Benchmark.

Anyway, that is arguably not-the-issue, here is why. The timeout-ing is
done to recover from the misbehaved benchmarks, and the @Setup taking 10
minutes to run is arguably misbehaving. We can't afford waiting
indefinitely for broken benchmarks if we are running the large benchmark
suites. Now, when we are facing the misbehaving workload, it does not
seem all that necessary to accurately track the workload time?

> What is also interesting is that iteration time for Mode.SingleShotTime is set 
> to 10 minutes. I would expect that in this mode JMH will wait forever for iteration
> to complete. 

See above.

> Or at least such behavior should be documented somewhere. :)

And what would be the best place to document this? I am puzzled about
that, and since the issue is about the corner case, it does not seem to
belong in the generic documentation.

Thanks,
-Aleksey.