Usage of Blackhole in a loop distorts benchmark results

[Сергей Цыпанов]

Say I have this benchmark:

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Fork(jvmArgsAppend = {"-XX:+UseParallelGC", "-Xms2g", "-Xmx2g"})
public class IteratorFromStreamBenchmark {

    @Benchmark
    public void iteratorFromStream(Data data, Blackhole bh) {
        Iterator<Integer> iterator = data.items.stream()
                .iterator();

        while (iterator.hasNext())
            bh.consume(iterator.next());
    }

    @Benchmark
    public void forEach(Data data, Blackhole bh) {
        data.items.stream().forEach(bh::consume);
    }

    @State(Scope.Thread)
    public static class Data {
        private Collection<Integer> items;

        private int size = 1000;

        @Setup
        public void init() {
            items = IntStream.range(0, size).boxed().collect(toList());
        }
    }
}

which on Java 9 (JDK 9, VM 9+181) yields this output:

Benchmark               Mode  Cnt        Score         Error  Units
forEach                       avgt  100  6130,066 ± 308,597  ns/op
iteratorFromStream    avgt  100  4835,355 ±   57,886  ns/op

Here 'iteratorFromStream' appears to be faster than 'forEach'


Then I change the behaviour to accumulate the result of iteration over elements and return it:

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Fork(jvmArgsAppend = {"-XX:+UseParallelGC", "-Xms2g", "-Xmx2g"})
public class IteratorFromStreamBenchmark {

    @Benchmark
    public int iteratorFromStream(Data data) {
        int sum = 0;
        Iterator<Integer> iterator = data.list.stream()
                .iterator();

        while (iterator.hasNext())
            sum += iterator.next();

        return sum;
    }

    @Benchmark
    public int forEach(Data data) {
        int[] sum = {0};
        data.list.stream().forEach(integer -> sum[0] = sum[0] + integer);
        return sum[0];
    }

    @State(Scope.Thread)
    public static class Data {
        private List<Integer> list;

        private int size = 100;

        @Setup
        public void init() {
            list = IntStream.range(0, size).boxed().collect(toList());
        }
    }
}

Which yields:

Benchmark               Mode  Cnt       Score     Error  Units
forEach                       avgt  100   133,118 ± 1,580  ns/op
iteratorFromStream    avgt  100   228,061 ± 5,491  ns/op


The question here is not only huge difference in absolute values, but the fact 'forEach' now appears to be faster than 'iteratorFromStream'. Also error is lower in case of value returning benchmark.

Could anyone explain is it correct behaviour of Blackhole?

Best regards,
Sergei Tsypanov