@Benchmark method code not included in timings?

[Kenneth Fogel]

Hi everyone,

Thank you for allowing me to join this mailing list.

My name is Ken Fogel and I was a Computer Science instructor at Dawson College in Montreal for 31 years, having retired 3 years ago. I was chairperson for 25 of those years and led the change from COBOL to Java in 2000. I am a Java Champion and organize, with fellow JCs, the JChampions Conference. I also sit on the JCP Executive Committee. Like all retired academics I wrote a book entitle Transitioning to Java from Packt for devs who need to move to Java from any other language. Finally, I still get to present at conferences, and it is work I am doing with the JMH for JavaLand that has led me to this mailing list.

Some years ago, I wrote a program that demonstrated the relative differences between the various collection in Java. The program worked as required, made for a good lecture, and proved why no one should use a LinkedList (this remark is meant to be funny and true). A little over a year ago I discovered the jmh and created a version of my program using it. Unfortunately, I could not run the benchmarks as I did in my version as the results were presented in a Swing table but that is not the issue I am having. Here is the measurement code for adding an element at the end of an ArrayList from my pre-jmh code:

for (int y = 0; y < REPETITIONS; ++y) {
         arrayDeque = new ArrayDeque<>();
         for (int x = 0; x < SIZE; ++x) {
                arrayDeque.add(dataArray[x]);
         }
         startTime = System.nanoTime();
         arrayDeque.addFirst("Dawson College");
         endTime = System.nanoTime() - startTime;
         runningTotal += endTime;
  }

The dataArray is a simple array of 10000 strings. I use the same set of words for every test. This worked well enough. The actual time to insert was calculated after a new ArrayList was created and populated for every repetition. This was necessary because had I created a new copy with :

arrayDeque = new ArrayDeque<>(Arrays.asList(dataArray));

I end up with an ArrayDeques that is the same size as the Array I am initializing it with. The problem is that this ArrayDeque has no slack space so it must immediately grow, As I am interested in the performance when there is slack space, I create the ArrayDeque in a loop. This results in the ArrayDeque having slack space.

When I moved to the jmh I could not find a way to create and initialize the ArrayList as @Benchmark timed the entire method's code.  I thought I figured it out by using an inner static class annotated with @State. It did not work. I discovered that the ArrayList was being initialized for each iteration but, and this was a very big but, I discovered after reading numerous StackOverflows and then looking at the jmh source code that iterations that were not set to  "Mode.SingleShotTime" executed the @Benchmark method a random number of times per iteration. This would be fine except that the @State class/method was only invoked once per iteration and not every time the method was executed as I required.

If you have read this far you can probably guess my question. Is it possible to execute code that is part of a @Benchmark method but that is not included in the timing. Is there or could there be an annotation for this rather than use SingleShotTime and then have a large value for measurementIterations . Something like:

@Benchmark
public void do10InsertLastElementArrayList(Blackhole blackhole) {

       @IgnoreOn
         Deque <String> arrayDeque = new ArrayDeque<>();
         for (int x = 0; x < SIZE; ++x) {
                arrayDeque.add(dataArray[x]);
         }
        @IgnoreOff

      arrayDeque.add("Dawson College");
      blackhole.consume(arrayDeque); // Not certain I need this

    }

If the operation does not mutate anything then there is no issue. It is when I mutate something that I need to restore the state of variables to before the mutation every time and not have this included in the timing measurement.

Thank you for your patience in reading this and for any assistance you can provide.

Ken

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jmh-dev/attachments/20240724/c890c470/attachment-0001.htm>