ScopedValue performance declines as number of dynamic bindings increases

Tue Jun 24 18:48:46 UTC 2025

The problem with removing the loop is that you'll end up measuring the cost of setting up the dynamic bindings, and not really the cost of traversing them. I'm not sure how you can set up the dynamic bindings as setup and have the benchmarked code run within that dynamic context. It's unfortunate, but the blackhole should prevent (and seems to do so) most optimizations of the loop.

________________________________________
From: robert engels <rengels at ix.netcom.com>
Sent: Tuesday, June 24, 2025 18:59
To: Johan Sjolen
Cc: loom-dev at openjdk.org
Subject: Re: ScopedValue performance declines as number of dynamic bindings increases

If you are using JMH, a read of this section may be helpful:

<https://urldefense.com/v3/__https://jenkov.com/tutorials/java-performance/jmh.html*writing-good-benchmarks__;Iw!!ACWV5N9M2RV99hQ!Kf4V8MGGzPh3EaRPIZemTDbQl3VmKGTB7_FHY31MQsK2eZqGoNR_Xcv-a_r8ABZQpqHXt1JlbIgQqPP3Hfw$>
[java-performance-teaser-500-300.png]
JMH - Java Microbenchmark Harness<https://urldefense.com/v3/__https://jenkov.com/tutorials/java-performance/jmh.html*writing-good-benchmarks__;Iw!!ACWV5N9M2RV99hQ!Kf4V8MGGzPh3EaRPIZemTDbQl3VmKGTB7_FHY31MQsK2eZqGoNR_Xcv-a_r8ABZQpqHXt1JlbIgQqPP3Hfw$>
jenkov.com<https://urldefense.com/v3/__https://jenkov.com/tutorials/java-performance/jmh.html*writing-good-benchmarks__;Iw!!ACWV5N9M2RV99hQ!Kf4V8MGGzPh3EaRPIZemTDbQl3VmKGTB7_FHY31MQsK2eZqGoNR_Xcv-a_r8ABZQpqHXt1JlbIgQqPP3Hfw$>

On Jun 24, 2025, at 11:54 AM, Johan Sjolen <johan.sjolen at oracle.com> wrote:

One last benchmark for today: I added a benchmark that binds the MISSING ScopedValue at the end, so it's like the slow benchmark but we don't need to traverse anything to find the binding.
That turns out to be even faster:

# Benchmark: org.openjdk.bench.java.lang.ScopedValues.ScopedValuesAreSlow2

# Run progress: 15.79% complete, ETA 00:03:51
# Fork: 1 of 1
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by org.openjdk.jmh.util.Utils (file:/home/opc/views/master/build/linux-aarch64/images/test/micro/benchmarks.jar)
WARNING: Please consider reporting this to the maintainers of class org.openjdk.jmh.util.Utils
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release
# Warmup Iteration   1: 196769.758 ns/op
# Warmup Iteration   2: 59395.092 ns/op
# Warmup Iteration   3: 29831.474 ns/op
# Warmup Iteration   4: 30670.259 ns/op
Iteration   1: 29522.851 ns/op
Iteration   2: 29220.998 ns/op
Iteration   3: 30017.858 ns/op
Iteration   4: 30409.883 ns/op
Iteration   5: 30392.041 ns/op
Iteration   6: 30876.052 ns/op
Iteration   7: 31950.619 ns/op
Iteration   8: 30711.009 ns/op
Iteration   9: 30412.429 ns/op
Iteration  10: 30890.572 ns/op

Result "org.openjdk.bench.java.lang.ScopedValues.ScopedValuesAreSlow2":
 30440.431 ±(99.9%) 1155.565 ns/op [Average]
 (min, avg, max) = (29220.998, 30440.431, 31950.619), stdev = 764.335
 CI (99.9%): [29284.866, 31595.997] (assumes normal distribution)

I'm logging off for today,
Cheers and thanks Robert and Aleksey for looking at this.

/ Johan

________________________________________
From: loom-dev <loom-dev-retn at openjdk.org> on behalf of Johan Sjolen <johan.sjolen at oracle.com>
Sent: Tuesday, June 24, 2025 18:44
To: Aleksey Shipilev
Cc: loom-dev at openjdk.org
Subject: Re: ScopedValue performance declines as number of dynamic bindings increases

Hi Aleksey,

I re-wrote my benchmarks to use JMH and I used Blackhole::consume instead of the S.o.println. The entire point of the println was to signal to the compiler 'this has to happen'. I kept the 100k iterations, as I didn't want the setup of the 100 dynamic bindings to dominate (or perhaps they are? Not sure).

I ended up with the following results:

(This is the test of just doing MISSING.orElse)
Result "org.openjdk.bench.java.lang.ScopedValues.ScopedValuesAreFasT":
 815142.530 ±(99.9%) 17764.324 ns/op [Average]
 (min, avg, max) = (790345.594, 815142.530, 835566.013), stdev = 11750.000
 CI (99.9%): [797378.206, 832906.854] (assumes normal distribution)

(This is the test with MISSING.orElse within the 100 dynamic bindings)
Result "org.openjdk.bench.java.lang.ScopedValues.ScopedValuesAreSlow":
 18847997.203 ±(99.9%) 587930.099 ns/op [Average]
 (min, avg, max) = (18176298.357, 18847997.203, 19523500.462), stdev = 388879.358
 CI (99.9%): [18260067.104, 19435927.301] (assumes normal distribution)

That's 815142 ns/op vs 18847997ns/op, a pretty large diff still ('contested' case takes 23x longer).

Am I messing up my benchmarks once again, but in some different way?

Here's the branch and source: https://github.com/openjdk/jdk/compare/master...jdksjolen:jdk:scopedvaluejmhs<https://urldefense.com/v3/__https://github.com/openjdk/jdk/compare/master...jdksjolen:jdk:scopedvaluejmhs__;!!ACWV5N9M2RV99hQ!Kf4V8MGGzPh3EaRPIZemTDbQl3VmKGTB7_FHY31MQsK2eZqGoNR_Xcv-a_r8ABZQpqHXt1JlbIgQYAgzvMk$>

I just put the benchmark into the pre-existing ScopedValues.java, should just be a copy and paste and be done if you've got a JDK build set up.
The tests are run on linux-aarch64.

________________________________________
From: Aleksey Shipilev <shipilev at amazon.de>
Sent: Tuesday, June 24, 2025 18:08
To: Johan Sjolen; Robert Engels
Cc: loom-dev at openjdk.org
Subject: Re: ScopedValue performance declines as number of dynamic bindings increases

On 24.06.25 18:01, Johan Sjolen wrote:
Oooh, that could explain the issue completely. I didn't think about that at all.
The issue basically disappears when compiling first, let me see what happens with a JMH benchmark as well.

I predict JMH would show the same thing.

javac is not very fast when cold; this is also why we use it as go-to example in AOT cache studies.
I would also expect compiling deeply nested lambdas (like in ScopedTest.java) makes the javac code
fairly recursive, which again plays interestingly with JIT compilation. It is still
actionable/interesting to figure out what can be improved there. Looks fairly weird we spend >3
seconds compiling a seemingly "easy" source. Submit a tools/javac RFE, let compiler folks have fun
with it?

-Aleksey

Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597