@Setup(Level.Invocation)

[Ivan Gerasimov]

Okay, I see now.
Thanks for the pointer!

On 09.07.2014 18:10, Sergey Kuksenko wrote:
> Had you read Level.Invocation docs?
> I'll juct copy it here:
>
> /**
>  * Invocation level: to be executed for each benchmark method execution.
>  *
>  * WARNING: HERE BE DRAGONS!
>  * MAKE SURE YOU UNDERSTAND THE REASONING AND THE IMPLICATIONS
>  * OF THE WARNINGS BELOW BEFORE EVEN CONSIDERING USING THIS LEVEL.
>  *
>  * It is almost never a good idea to use this in nano- and 
> micro-benchmarks.
>  *
>  * <p>WARNING #1: In order to subtract the helper time from the
>  * benchmark itself, we will have to have at least two timestamps
>  * per *each* benchmark invocation. If the benchmarked method is
>  * small, then we saturate the system with timestamp requests, which
>  * *both* make timestamp requests the critical part of the benchmark
>  * time, and inhibit workload scalability, introducing the artificial
>  * scalability bottleneck.</p>
>  *
>  * <p>Also, the hiccups in measurement can be hidden from these 
> individual
>  * timing measurement, which can introduce inconsistent results. The 
> largest
>  * caveat is measuring oversaturated system where the descheduling events
>  * will be missed, and the benchmark will perceive the fictionally large
>  * throughput.</p>
>  *
>  * <p>WARNING #2: In order to maintain the basic interference behavior
>  * of other Levels (e.g. the State(Scope.Benchmark) should only fire the
>  * helper method once per invocation, regardless of the thread count),
>  * we have to arbitrate the access to the state between worker thread,
>  * and do that on *critical path*, thus further offsetting the 
> measurement.</p>
>  *
>  * <p>WARNING #3: Current implementation in JMH allows the helper method
>  * execution to overlap with the benchmark method itself in order to 
> simplify
>  * arbitrage. (To be redefined in future).</p>
>  */
> Invocation,
>
> On 07/09/2014 06:05 PM, Ivan Gerasimov wrote:
>> Hello!
>>
>> I noticed a great influence of presence of @Setup(Level.Invocation) on
>> some benchmark results.
>>
>> This simple test gives the following numbers.
>>
>> @State(Scope.Thread)
>> public class MyBenchmark {
>>
>> //    @Setup(Level.Invocation)
>> //    public void setup() {}
>>
>>      @Benchmark
>>      public int testMethod() {
>>          return 42;
>>      }
>> }
>>
>> Benchmark                      Mode   Samples        Score Score
>> error    Units
>> o.s.MyBenchmark.testMethod    thrpt         5 521204947.564
>> 200284466.443    ops/s
>>
>>
>> When I uncomment the setup() function, the numbers drops to:
>>
>> Benchmark                      Mode   Samples        Score Score
>> error    Units
>> o.s.MyBenchmark.testMethod    thrpt         5   120471.248 32030.457 
>> ops/s
>>
>>
>> Am I correct that it's not practical to use @Setup(Level.Invocation)
>> with measuring anything that lasts less than a microsecond or something?
>>
>> Sincerely yours,
>> Ivan
>>
>