a situation when JHM 1.21 measurements are broken

[Valentin Kovalenko]

I encountered a simple situation where JMH 1.21 produces obviously
incorrect results. The test (is also temporarily available at
https://github.com/stIncMale/sandbox/blob/master/benchmarks/src/test/java/stincmale/sandbox/benchmarks/TmpTest.java
,
you can run it with `mvn clean test -f benchmarks/pom.xml -Dtest=TmpTest`):

public class TmpTest {
  public TmpTest() {
  }

  public final void runThroughputBenchmarks(final int numberOfThreads)
throws RunnerException {
    final OptionsBuilder opts = new OptionsBuilder();
    opts.jvmArgs("-Xfuture", "--illegal-access=deny", "-Xshare:off",
"-Xms1024m", "-Xmx1024m", "-server", "-disableassertions")
        .shouldDoGC(false)
        .syncIterations(true)
        .shouldFailOnError(true)
        .threads(1)
        .timeout(milliseconds(1000_000))
        .forks(2)
        .warmupTime(milliseconds(1000))
        .warmupIterations(6)
        .measurementTime(milliseconds(1000))
        .measurementIterations(2)
        .include(includeBenchmarks(TmpTest.class))
        .shouldDoGC(false)
        .mode(Mode.Throughput)
        .timeUnit(TimeUnit.MILLISECONDS)
        .threads(numberOfThreads);
    new Runner(opts.build()).run();
  }

  @Test
  public final void throughputThreads4() throws RunnerException {
    runThroughputBenchmarks(4);
  }

  @Test
  public final void throughputThreads32() throws RunnerException {
    runThroughputBenchmarks(32);
  }

  @Benchmark
  public final long reentrantRwLock(final BenchmarkState s) throws
InterruptedException {
    s.rwLock.readLock().lock();
    try {
      if (s.counter.getAndIncrement() < 1000) {
        return s.counter.get();
      }
    } finally {
      s.rwLock.readLock().unlock();
    }
    s.rwLock.writeLock().lock();
    try {
      Thread.sleep(10);
      s.counter.set(0);
    } finally {
      s.rwLock.writeLock().unlock();
    }
    return s.counter.get();
  }

  @Benchmark
  public final long stampedLock(final BenchmarkState state) throws
InterruptedException {
    long stamp = state.lock.readLock();
    try {
      if (state.counter.getAndIncrement() < 1000) {
        return state.counter.get();
      }
    } finally {
      state.lock.unlockRead(stamp);
    }
    stamp = state.lock.writeLock();
    try {
      Thread.sleep(10);
      state.counter.set(0);
    } finally {
      state.lock.unlockWrite(stamp);
    }
    return state.counter.get();
  }

  @State(Scope.Benchmark)
  public static class BenchmarkState {
    private StampedLock lock;
    private ReentrantReadWriteLock rwLock;
    private AtomicLong counter;

    public BenchmarkState() {
    }

    @Setup(Level.Trial)
    public final void setup() {
      lock = new StampedLock();
      rwLock = new ReentrantReadWriteLock();
      counter = new AtomicLong();
    }
  }
}

Obviously, there is no way the throughput for this benchmarks is greater
than 1000 ops / 10ms = 100 ops/ms. And JMH produces stable results lesser
than 50 ops/ms for 4 threads for both reentrantRwLock and stampedLock.
However, for 32 threads JMH produces sane results only for reentrantRwLock,
but for stampedLock the run looks like this:

# Fork: 1 of 2
# Warmup Iteration   1: 665.471 ops/ms
# Warmup Iteration   2: 782.521 ops/ms
# Warmup Iteration   3: 32.830 ops/ms
# Warmup Iteration   4: 43.356 ops/ms
# Warmup Iteration   5: 2573.165 ops/ms
# Warmup Iteration   6: 408.397 ops/ms
Iteration   1: 9072.582 ops/ms
Iteration   2: 2492.579 ops/ms

# Fork: 2 of 2
# Warmup Iteration   1: 10.445 ops/ms
# Warmup Iteration   2: 27.372 ops/ms
# Warmup Iteration   3: 2632.625 ops/ms
# Warmup Iteration   4: 13.445 ops/ms
# Warmup Iteration   5: 16.657 ops/ms
# Warmup Iteration   6: 28.452 ops/ms
Iteration   1: 30.703 ops/ms
Iteration   2: 27.807 ops/ms

These numbers make no sense because as I mentioned, in these benchmarks the
throughput can never be higher than 100 ops/ms. Is there a valid
explanation for this JMH behaviour, or is this actually a bug in JMH?

Regards,
Valentin
[image: LinkedIn] <https://www.linkedin.com/in/stIncMale>   [image: GitHub]
<https://github.com/stIncMale>   [image: YouTube]
<https://www.youtube.com/user/stIncMale>