Mean error bigger than mean?

[Dmitry Vyazelenko]

Hi,

I came across interesting issue with one benchmark while testing it on JDK8. Basically I observe measurement error bigger than measured value. And I’m not sure if it is somehow JMH issue or JDK8 regression or something else.

Here is offending benchmark:
package net.java;

import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.logic.BlackHole;

import java.util.HashMap;
import java.util.concurrent.TimeUnit;

@State(Scope.Benchmark)
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@Fork(3)
@Warmup(iterations = 5)
@Measurement(iterations = 5)
public class HashMapLongBenchmark {
 private static final int DATA_SIZE = 1024 * 1024 * 4;
 private static final long[] DATA;

 static {
     DATA = new long[DATA_SIZE];
     for (int i = 0; i < DATA_SIZE; i++) {
         DATA[i] = (long) i;
     }
 }

 private HashMap<Long, Boolean> hugeMap;

 @Setup(Level.Iteration)
 public void setUp() {
     HashMap<Long, Boolean> map = new HashMap<>(DATA_SIZE);
     for (long v : DATA) {
         map.put(v, Boolean.TRUE);
     }
     hugeMap = map;
 }

 @GenerateMicroBenchmark
 public void baseLine(BlackHole bh) {
     long[] data = DATA;
     for (long v : data) {
         bh.consume((Long) v);
     }
 }
}

This benchmark only tests how long it takes to iterate long[] and autobox each value to Long (original benchmark was also testing map access performance to measure boxing overhead and evaluate HashMap vs primitive maps).  As you can see before every iteration hugeMap variable is initialized. Here it is only to cause the problem, i.e. without this variable the issue is not reproducible.

The benchmark was created using jmh-0.5.5 but the results posted here were obtained using current HEAD from Mercurial, i.e. 1.0-SNAPSHOT. To run the benchmark the following command was used: 
java -XX:-TieredCompilation -showversion -jar target/microbenchmarks.jar .*HashMapLong.*

Here are the results:
 Benchmark                                                 Mode      Samples         Mean      Mean error    Units
1) n.j.HashMapLongBenchmark.baseLine     avgt        15            28889.400        5216.270    us/op
2) n.j.HashMapLongBenchmark.baseLine     avgt        15          681260.400    882189.619    us/op
3) n.j.HashMapLongBenchmark.baseLine     avgt        15          709108.800    904306.276    us/op

* where:
1)
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
2) 
java version "1.8.0"
Java(TM) SE Runtime Environment (build 1.8.0-b132)
Java HotSpot(TM) 64-Bit Server VM (build 25.0-b70, mixed mode)
3)
java version "1.8.0_20-ea"
Java(TM) SE Runtime Environment (build 1.8.0_20-ea-b05)
Java HotSpot(TM) 64-Bit Server VM (build 25.20-b05, mixed mode)

I hope you can give me a hint on what is going on.

Best regards,
Dmitry Vyazelenko