OOME in simple benchmark

[Bernd Eckenfels]

Iteration and Warmup is not „one shot“, it will run for as Long as configured (30s - resulting in millions of executions).

I dont think there would be any useful results if you measure such a fest activity only 30 times.


Maybe clear the list when it reaches a specific size?

Gruss
Bernd
-- 
http://bernd.eckenfels.net

Von: Сергей Цыпанов
Gesendet: Sonntag, 14. Januar 2018 20:24
An: jmh-dev at openjdk.java.net
Betreff: OOME in simple benchmark

I execute the following benchmark

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public class AddIntoArrayListBenchmark {

    @Benchmark
    public boolean add(Data data) {
        return data.list.add(data.integer);
    }

    @State(Scope.Thread)
    public static class Data {
        private Integer integer = 1;
        private ArrayList<Integer> list;

        @Setup
        public void setup() {
            list = new ArrayList<>();
        }
    }
}

with this runner:

public class BenchmarkRunner {
	public static void main(String[] args) throws RunnerException {
		Options opt = new OptionsBuilder()
				.warmupIterations(10)
				.measurementIterations(20)
				.forks(1)
				.shouldFailOnError(true)
				.build();

		new Runner(opt).run();
	}
}

and on the 2nd or 3rd iteration constantly get OOME.

java.lang.OutOfMemoryError: Java heap space
	at java.base/java.util.Arrays.copyOf(Arrays.java:3688)
	at java.base/java.util.ArrayList.grow(ArrayList.java:236)
	at java.base/java.util.ArrayList.grow(ArrayList.java:241)
	at java.base/java.util.ArrayList.add(ArrayList.java:466)
	at java.base/java.util.ArrayList.add(ArrayList.java:479)
	at com.luxoft.logeek.benchmark.AddIntoArrayListBenchmark.add(AddIntoArrayListBenchmark.java:16)
	at com.luxoft.logeek.benchmark.generated.AddIntoArrayListBenchmark_add_jmhTest.add_avgt_jmhStub(AddIntoArrayListBenchmark_add_jmhTest.java:191)
	at com.luxoft.logeek.benchmark.generated.AddIntoArrayListBenchmark_add_jmhTest.add_AverageTime(AddIntoArrayListBenchmark_add_jmhTest.java:154)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Here is run time output:

# JMH version: 1.19
# VM version: JDK 9, VM 9+181
# VM invoker: G:\jdk\jdk-9\bin\java.exe
# VM options: -Dvisualvm.id=3867492965950 -javaagent:G:\idea\IntelliJ IDEA 173.2463.16\lib\idea_rt.jar=56255:G:\idea\IntelliJ IDEA 173.2463.16\bin -Dfile.encoding=UTF-8
# Warmup: 10 iterations, 1 s each
# Measurement: 20 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: com.luxoft.logeek.benchmark.AddIntoArrayListBenchmark.add

As I understand benchmark method 'add' should be executed 30 times (10 for warmup and 20 for measurement), but for some reason it seems to be executed in an infinite loop causing error.

Using @Setup(Level.Iteration) solves the issue but I wonder what's behind it?
I've looked into the code of generated class AddIntoArrayListBenchmark_add_jmhTest and it seems the only difference is the usage of

AddIntoArrayListBenchmark_Data_jmhType l_data1_1 = _jmh_tryInit_f_data1_1(control);

When plain @Setup is used l_data1_1 is instantiated with calling setup() and then passed to benchmark method.

For @Setup(Level.Iteration) l_data1_1.setup() is called after control.preSetup() which seems to be logical.

As far as I have only one thread and one fork l_data1_1.setup() is executed only once for both Level.Trial and Level.Iteration in my case.
I do not see any other differences in the code of AddIntoArrayListBenchmark_add_jmhTest but the behaviour is different.

What am I doing wrong?