ScopedValue performance declines as number of dynamic bindings increases

Tue Jun 24 16:59:52 UTC 2025

If you are using JMH, a read of this section may be helpful:

https://jenkov.com/tutorials/java-performance/jmh.html#writing-good-benchmarks
JMH - Java Microbenchmark Harness
jenkov.com

> On Jun 24, 2025, at 11:54 AM, Johan Sjolen <johan.sjolen at oracle.com> wrote:
> 
> 
> One last benchmark for today: I added a benchmark that binds the MISSING ScopedValue at the end, so it's like the slow benchmark but we don't need to traverse anything to find the binding.
> That turns out to be even faster:
> 
> # Benchmark: org.openjdk.bench.java.lang.ScopedValues.ScopedValuesAreSlow2
> 
> # Run progress: 15.79% complete, ETA 00:03:51
> # Fork: 1 of 1
> WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
> WARNING: sun.misc.Unsafe::objectFieldOffset has been called by org.openjdk.jmh.util.Utils (file:/home/opc/views/master/build/linux-aarch64/images/test/micro/benchmarks.jar)
> WARNING: Please consider reporting this to the maintainers of class org.openjdk.jmh.util.Utils
> WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release
> # Warmup Iteration   1: 196769.758 ns/op
> # Warmup Iteration   2: 59395.092 ns/op
> # Warmup Iteration   3: 29831.474 ns/op
> # Warmup Iteration   4: 30670.259 ns/op
> Iteration   1: 29522.851 ns/op
> Iteration   2: 29220.998 ns/op
> Iteration   3: 30017.858 ns/op
> Iteration   4: 30409.883 ns/op
> Iteration   5: 30392.041 ns/op
> Iteration   6: 30876.052 ns/op
> Iteration   7: 31950.619 ns/op
> Iteration   8: 30711.009 ns/op
> Iteration   9: 30412.429 ns/op
> Iteration  10: 30890.572 ns/op
> 
> 
> Result "org.openjdk.bench.java.lang.ScopedValues.ScopedValuesAreSlow2":
>  30440.431 ±(99.9%) 1155.565 ns/op [Average]
>  (min, avg, max) = (29220.998, 30440.431, 31950.619), stdev = 764.335
>  CI (99.9%): [29284.866, 31595.997] (assumes normal distribution)
> 
> I'm logging off for today,
> Cheers and thanks Robert and Aleksey for looking at this.
> 
> / Johan
> 
> ________________________________________
> From: loom-dev <loom-dev-retn at openjdk.org> on behalf of Johan Sjolen <johan.sjolen at oracle.com>
> Sent: Tuesday, June 24, 2025 18:44
> To: Aleksey Shipilev
> Cc: loom-dev at openjdk.org
> Subject: Re: ScopedValue performance declines as number of dynamic bindings increases
> 
> Hi Aleksey,
> 
> I re-wrote my benchmarks to use JMH and I used Blackhole::consume instead of the S.o.println. The entire point of the println was to signal to the compiler 'this has to happen'. I kept the 100k iterations, as I didn't want the setup of the 100 dynamic bindings to dominate (or perhaps they are? Not sure).
> 
> I ended up with the following results:
> 
> 
> (This is the test of just doing MISSING.orElse)
> Result "org.openjdk.bench.java.lang.ScopedValues.ScopedValuesAreFasT":
>  815142.530 ±(99.9%) 17764.324 ns/op [Average]
>  (min, avg, max) = (790345.594, 815142.530, 835566.013), stdev = 11750.000
>  CI (99.9%): [797378.206, 832906.854] (assumes normal distribution)
> 
> (This is the test with MISSING.orElse within the 100 dynamic bindings)
> Result "org.openjdk.bench.java.lang.ScopedValues.ScopedValuesAreSlow":
>  18847997.203 ±(99.9%) 587930.099 ns/op [Average]
>  (min, avg, max) = (18176298.357, 18847997.203, 19523500.462), stdev = 388879.358
>  CI (99.9%): [18260067.104, 19435927.301] (assumes normal distribution)
> 
> 
> That's 815142 ns/op vs 18847997ns/op, a pretty large diff still ('contested' case takes 23x longer).
> 
> Am I messing up my benchmarks once again, but in some different way?
> 
> Here's the branch and source: https://github.com/openjdk/jdk/compare/master...jdksjolen:jdk:scopedvaluejmhs
> 
> I just put the benchmark into the pre-existing ScopedValues.java, should just be a copy and paste and be done if you've got a JDK build set up.
> The tests are run on linux-aarch64.
> 
> ________________________________________
> From: Aleksey Shipilev <shipilev at amazon.de>
> Sent: Tuesday, June 24, 2025 18:08
> To: Johan Sjolen; Robert Engels
> Cc: loom-dev at openjdk.org
> Subject: Re: ScopedValue performance declines as number of dynamic bindings increases
> 
> On 24.06.25 18:01, Johan Sjolen wrote:
>> Oooh, that could explain the issue completely. I didn't think about that at all.
>> The issue basically disappears when compiling first, let me see what happens with a JMH benchmark as well.
> 
> I predict JMH would show the same thing.
> 
> javac is not very fast when cold; this is also why we use it as go-to example in AOT cache studies.
> I would also expect compiling deeply nested lambdas (like in ScopedTest.java) makes the javac code
> fairly recursive, which again plays interestingly with JIT compilation. It is still
> actionable/interesting to figure out what can be improved there. Looks fairly weird we spend >3
> seconds compiling a seemingly "easy" source. Submit a tools/javac RFE, let compiler folks have fun
> with it?
> 
> -Aleksey
> 
> 
> 
> Amazon Web Services Development Center Germany GmbH
> Tamara-Danz-Str. 13
> 10243 Berlin
> Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
> Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
> Sitz: Berlin
> Ust-ID: DE 365 538 597

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20250624/dd56834a/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: java-performance-teaser-500-300.png
Type: image/png
Size: 43208 bytes
Desc: not available
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20250624/dd56834a/java-performance-teaser-500-300-0001.png>