Setup time is counted against iteration time in Mode.SingleShotTime (JMH-1.0)

[vyazelenko at yahoo.com]

Ok, that's interesting...

Sent from my iPhone

> On Sep 2, 2014, at 12:52, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
> 
> Hi,
> 
>> On 09/01/2014 06:48 PM, Dmitry Vyazelenko wrote:
>> I've discovered that when Mode.SingleShotTime is used the time in @Setup
>> is counted against iteration time. This results in benchmark being interrupted
>> if @Setup takes long time.
> 
> Yes, there is the implementation reason for that: @Setup is executed by
> the generated code, and so the harness control thread is oblivious of
> the time spent in @Setup vs. @Benchmark.
I have a problem with that. It is neither intuitive nor expected that @Setup/@TearDown times are considered part of actual measurement. Especially if Level.Trial is used.

Consider the case of kilo-benchmark which inserts millions of entries into the database as part of @Setup(Level.Trial) and it takes 20 min to execute.
Now assume that benchmark methods are executing SELECTs against database (with different indices used). And they run 2-10 seconds.
With such setup and usage of SingleShotTime with 10 iterations reported execution times and reality will be quite different since very first iteration takes 1200 seconds longer than any other iteration.
> 
> Anyway, that is arguably not-the-issue, here is why. The timeout-ing is
> done to recover from the misbehaved benchmarks, and the @Setup taking 10
> minutes to run is arguably misbehaving. We can't afford waiting
> indefinitely for broken benchmarks if we are running the large benchmark
> suites. Now, when we are facing the misbehaving workload, it does not
> seem all that necessary to accurately track the workload time?
Well if Setup is not considered part of the workload then there would no problem to start with.

I think now with 1.0 released such change can be considered. Also waiting indefinitely for @Setup/@TearDown and maybe even benchmark proper should be considered.
> 
>> What is also interesting is that iteration time for Mode.SingleShotTime is set 
>> to 10 minutes. I would expect that in this mode JMH will wait forever for iteration
>> to complete.
> 
> See above.
> 
>> Or at least such behavior should be documented somewhere. :)
> 
> And what would be the best place to document this? I am puzzled about
> that, and since the issue is about the corner case, it does not seem to
> belong in the generic documentation.
Why not document it in "time" property of the @Warmup/@Measurement annotations? Saying something like:
"If time is undefined then depending on the BenchmarkMode JMH will:
  - wait at least 5 seconds...
  - wait for 10 minutes in case SingleShotTime...
To override defaults please put time value explicitly..."
> 
> Thanks,
> -Aleksey.
> 
> 
Best regards,
Dmitry Vyazelenko