@Benchmark method code not included in timings?

[Henri Tremblay]

Hi Ken,

Of course it is :-)
If you haven't done it yet, I highly recommend you go through all the JMH
samples <https://github.com/openjdk/jmh/tree/master/jmh-samples>.
This one in your case:
https://github.com/openjdk/jmh/blob/master/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_03_States.java
They are really useful to understand how it works.

That said, you can have a state with scope = Benchmark
(@State(Scope.Benchmark))
It will be executed only one time at the beginning.

You also have @Setup that you can put on a method.
However, it will be minimally called for every trial.

I see something like this:

@Fork(2)
@Warmup(iterations = 10, time = 1)
@Measurement(iterations = 4, time = 1)
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Thread)
public class KenBenchmark {

    private static final int A_NUMBER_BIG_ENOUGH_TO_PREVENT_RESIZE = 1_000_000;
    private static List<String> dataArray = Arrays.asList("a", "b", "...");
    private ArrayDeque<String> arrayDeque;

    @Setup(Level.Iteration)
    public void init() {
        arrayDeque = new ArrayDeque<>(dataArray.size() +
A_NUMBER_BIG_ENOUGH_TO_PREVENT_RESIZE);
        arrayDeque.addAll(dataArray);
    }

    @Benchmark
    public void addFirst() {
        arrayDeque.addFirst("Dawson College");
    }

}

Regards,
Henri


On Wed, 24 Jul 2024 at 23:01, Kenneth Fogel <kfogel at dawsoncollege.qc.ca>
wrote:

> Hi everyone,
>
>
>
> Thank you for allowing me to join this mailing list.
>
>
>
> My name is Ken Fogel and I was a Computer Science instructor at Dawson
> College in Montreal for 31 years, having retired 3 years ago. I was
> chairperson for 25 of those years and led the change from COBOL to Java in
> 2000. I am a Java Champion and organize, with fellow JCs, the JChampions
> Conference. I also sit on the JCP Executive Committee. Like all retired
> academics I wrote a book entitle Transitioning to Java from Packt for devs
> who need to move to Java from any other language. Finally, I still get to
> present at conferences, and it is work I am doing with the JMH for JavaLand
> that has led me to this mailing list.
>
>
>
> Some years ago, I wrote a program that demonstrated the relative
> differences between the various collection in Java. The program worked as
> required, made for a good lecture, and proved why no one should use a
> LinkedList (this remark is meant to be funny and true). A little over a
> year ago I discovered the jmh and created a version of my program using it.
> Unfortunately, I could not run the benchmarks as I did in my version as the
> results were presented in a Swing table but that is not the issue I am
> having. Here is the measurement code for adding an element at the end of an
> ArrayList from my pre-jmh code:
>
>
>
> for (int y = 0; y < REPETITIONS; ++y) {
>
>          arrayDeque = new ArrayDeque<>();
>
>          for (int x = 0; x < SIZE; ++x) {
>
>                 arrayDeque.add(dataArray[x]);
>
>          }
>
>          startTime = System.nanoTime();
>
>          arrayDeque.addFirst("Dawson College");
>
>          endTime = System.nanoTime() - startTime;
>
>          runningTotal += endTime;
>
>   }
>
>
>
> The dataArray is a simple array of 10000 strings. I use the same set of
> words for every test. This worked well enough. The actual time to insert
> was calculated after a new ArrayList was created and populated for every
> repetition. This was necessary because had I created a new copy with :
>
>
>
> arrayDeque = new ArrayDeque<>(Arrays.asList(dataArray));
>
>
>
> I end up with an ArrayDeques that is the same size as the Array I am
> initializing it with. The problem is that this ArrayDeque has no slack
> space so it must immediately grow, As I am interested in the performance
> when there is slack space, I create the ArrayDeque in a loop. This results
> in the ArrayDeque having slack space.
>
>
>
> When I moved to the jmh I could not find a way to create and initialize
> the ArrayList as @Benchmark timed the entire method’s code.  I thought I
> figured it out by using an inner static class annotated with @State. It did
> not work. I discovered that the ArrayList was being initialized for each
> iteration but, and this was a very big but, I discovered after reading
> numerous StackOverflows and then looking at the jmh source code that
> iterations that were not set to  “Mode.SingleShotTime” executed the
> @Benchmark method a random number of times per iteration. This would be
> fine except that the @State class/method was only invoked once per
> iteration and not every time the method was executed as I required.
>
>
>
> If you have read this far you can probably guess my question. Is it
> possible to execute code that is part of a @Benchmark method but that is
> not included in the timing. Is there or could there be an annotation for
> this rather than use SingleShotTime and then have a large value for
> measurementIterations . Something like:
>
>
>
> @Benchmark
>
> public void do10InsertLastElementArrayList(Blackhole blackhole) {
>
>
>
> *       @IgnoreOn*
>
> *         Deque <String> arrayDeque = new ArrayDeque<>();*
>
> *         for (int x = 0; x < SIZE; ++x) {*
>
> *                arrayDeque.add(dataArray[x]);*
>
> *         }*
>
> *        @IgnoreOff*
>
>
>
>       arrayDeque.add("Dawson College");
>
>       blackhole.consume(arrayDeque); // Not certain I need this
>
>
>
>     }
>
>
>
> If the operation does not mutate anything then there is no issue. It is
> when I mutate something that I need to restore the state of variables to
> before the mutation every time and not have this included in the timing
> measurement.
>
>
>
> Thank you for your patience in reading this and for any assistance you can
> provide.
>
>
>
> Ken
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jmh-dev/attachments/20240725/b1b8e876/attachment.htm>