Setup and TearDown executed by different threads

[Dmitry Vyazelenko]

Hi,

I've found a problem in multi-threaded benchmarks that @Setup and
@TearDown are executed by different threads. This breaks the teardown
code, because it assumes that same thread that executed @Setup will 
execute @TearDown method.

Here is benchmark that illustrates the issue:
@Fork(1) 
@Warmup(iterations = 0) 
@Measurement(iterations = 1) 
@State(Scope.Benchmark) 
@BenchmarkMode(Mode.SingleShotTime) 
public class SetupAndTearDownBenchmarkScope { 
    private String trialThreadName; 
    private String iterationsThreadName; 
    private String invocationThreadName; 
    @Benchmark 
    public void baseline() { 
    } 
    private void printThread(String msg) { 
        System.out.println("\r\n[" + Thread.currentThread().getName() + "] => " + msg); 
    } 
    @Setup(Level.Trial) 
    public void setUpTrial() { 
        printThread("setUpTrial()"); 
    } 
    @TearDown(Level.Trial) 
    public void tearDownTrial() { 
        printThread("tearDownTrial()"); 
    } 
    @Setup(Level.Iteration) 
    public void setUpIteration() { 
        printThread("setUpIteration()"); 
    } 
    @TearDown(Level.Iteration) 
    public void tearDownIteration() { 
        printThread("tearDownIteration()"); 
    } 
    @Setup(Level.Invocation) 
    public void setUpInvocation() { 
        printThread("setUpInvocation()"); 
    } 
    @TearDown(Level.Invocation) 
    public void tearDownInvocation() { 
        printThread("tearDownInvocation()"); 
    } 
}

When I execute it using single thread (i.e. java -jar target\benchmarks.jar -t 1) I get:
  [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-1] => setUpTrial() 
  [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-1] => setUpIteration() 
  [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-1] => setUpInvocation() 
  [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-1] => tearDownInvocation() 
  [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-1] => tearDownIteration() 
  [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-1] => tearDownTrial()
Which basically confirms that same thread used for everything.

However with multiple threads (i.e. java -jar target\benchmarks.jar -t 4):
  [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-3] => setUpTrial() 
  [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-1] => setUpIteration() 
  [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-1] => setUpInvocation() 
  [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-1] => tearDownInvocation() 
  [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-3] => setUpInvocation() 
  [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-3] => tearDownInvocation() 
  [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-2] => setUpInvocation() 
  [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-2] => tearDownInvocation() 
  [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-2] => tearDownIteration() 
  [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-2] => tearDownTrial()

This clearly shows that setup/tearDown pairs are executed by different threads!

What I expect:
- For Scope.Benchmark single thread will execute all @Setup/@TearDown methods
(at least for Level.Trial and Level.Iteration, not sure about Level.Invocation though)
- For Scope.Thread each thread will execute it's own @Setup/@TearDown methods
- For Scope.Group probably use one of threads from the group? Or dedicated @Setup/@TrearDown
thread created in addition.

 
Best regards,
Dmitry