Potential OSR-related performance issue in jdk21

[Dean Long]

It wouldn't hurt to file a bug.  If the same method keeps getting 
compiled over and over then it would be good to find out why. Some flags 
that might help: -XX:+PrintCompilation -XX:+LogCompilation and also 
maybe -XX:+TraceDeoptimization.  It could be OSR if 
DispatchThread.run()+788 is a branch, or just a regular compile if it is 
a method call.

dl

On 11/14/25 12:28 AM, Wojciech KUDLA wrote:
>
> Hi,
>
> (Somehow this message didn’t make it through to hotspot-compiler-dev, 
> so trying here)
>
>
> We’ve observed a strange performance issue within our 
> latency-sensitive application when running on jdk21 (tested with 
> 21.0.3 and 21.0.8).
> The impacted threads are pinned to their respective isolated cores so 
> not expected to experience any disruptions with the exception of a 
> regular hrtick but that executes in user context and is always a 
> sub-microsecond thing.
> First, we noticed that these threads started showing voluntary context 
> switching. Since we avoid any non-vdso syscalls this was a strong 
> indicator of some unintended locking going on and so we decided to 
> capture user- and kernel-space stack traces for one such thread with a 
> bit of eBPF.
> It looks like we go into this death loop of OSRs (the only type of 
> compilation activity known to me to halt execution on the impacted 
> thread) and this happens tens of times per second.
> Here’s the stack traces:
>
> ustack:
>
> __lll_unlock_wake+26
>
> CompileBroker::compile_method(methodHandle const&, int, int, 
> methodHandle const&, int, CompileTask::CompileReason, JavaThread*)+85
>
> CompilationPolicy::compile(methodHandle const&, int, CompLevel, 
> JavaThread*)+456
>
> CompilationPolicy::event(methodHandle const&, methodHandle const&, 
> int, int, CompLevel, CompiledMethod*, JavaThread*)+553
>
> InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, 
> unsigned char*)+331
>
> InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned 
> char*)+27
>
> Interpreter+15968
>
>         void 
> com.hsbc.efx.actor.dispatcher.SingleThreadDispatcher$DispatchThread.run()+788
>
> kstack:
>
> syscall_trace_enter+686
>
> syscall_trace_enter+686
>
> do_syscall_64+326
>
> entry_SYSCALL_64_after_hwframe+102
>
> ustack:
>
>        __lll_unlock_wake+26
>
> CompileBroker::compile_method(methodHandle const&, int, int, 
> methodHandle const&, int, CompileTask::CompileReason, JavaThread*)+85
>
> CompilationPolicy::compile(methodHandle const&, int, CompLevel, 
> JavaThread*)+456
>
> CompilationPolicy::event(methodHandle const&, methodHandle const&, 
> int, int, CompLevel, CompiledMethod*, JavaThread*)+553
>
> InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, 
> unsigned char*)+331
>
> InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned 
> char*)+27
>
> Interpreter+15968
>
>         void 
> com.hsbc.efx.actor.dispatcher.SingleThreadDispatcher$DispatchThread.run()+788
>
> kstack:
>
> syscall_slow_exit_work+179
>
> syscall_slow_exit_work+179
>
> do_syscall_64+365
>
> entry_SYSCALL_64_after_hwframe+102
>
> ustack:
>
> __lll_unlock_wake+26
>
> CompilationPolicy::compile(methodHandle const&, int, CompLevel, 
> JavaThread*)+456
>
> CompilationPolicy::event(methodHandle const&, methodHandle const&, 
> int, int, CompLevel, CompiledMethod*, JavaThread*)+553
>
> InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, 
> unsigned char*)+331
>
> InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned 
> char*)+27
>
> Interpreter+15968
>
>         void 
> com.hsbc.efx.actor.dispatcher.SingleThreadDispatcher$DispatchThread.run()+788
>
> kstack:
>
> syscall_trace_enter+686
>
> syscall_trace_enter+686
>
> do_syscall_64+326
>
> entry_SYSCALL_64_after_hwframe+102
>
> ustack:
>
> __lll_unlock_wake+26
>
> CompilationPolicy::compile(methodHandle const&, int, CompLevel, 
> JavaThread*)+456
>
> CompilationPolicy::event(methodHandle const&, methodHandle const&, 
> int, int, CompLevel, CompiledMethod*, JavaThread*)+553
>
> InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, 
> unsigned char*)+331
>
> InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned 
> char*)+27
>
> Interpreter+15968
>
>         void 
> com.hsbc.efx.actor.dispatcher.SingleThreadDispatcher$DispatchThread.run()+788
>
> kstack:
>
> syscall_slow_exit_work+179
>
> syscall_slow_exit_work+179
>
> do_syscall_64+365
>
> entry_SYSCALL_64_after_hwframe+102
>
> ustack:
>
> __lll_lock_wait+29
>
> DirectivesStack::getMatchingDirective(methodHandle const&, 
> AbstractCompiler*)+50
>
> CompileBroker::compile_method(methodHandle const&, int, int, 
> methodHandle const&, int, CompileTask::CompileReason, JavaThread*)+85
>
> CompilationPolicy::compile(methodHandle const&, int, CompLevel, 
> JavaThread*)+456
>
> CompilationPolicy::event(methodHandle const&, methodHandle const&, 
> int, int, CompLevel, CompiledMethod*, JavaThread*)+553
>
> InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, 
> unsigned char*)+331
>
> InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned 
> char*)+27
>
> Interpreter+15968
>
>         void 
> com.hsbc.efx.actor.dispatcher.SingleThreadDispatcher$DispatchThread.run()+788
>
> kstack:
>
> syscall_trace_enter+686
>
> syscall_trace_enter+686
>
> do_syscall_64+326
>
> entry_SYSCALL_64_after_hwframe+102
>
> ustack:
>
> __lll_lock_wait+29
>
> DirectivesStack::getMatchingDirective(methodHandle const&, 
> AbstractCompiler*)+50
>
> CompileBroker::compile_method(methodHandle const&, int, int, 
> methodHandle const&, int, CompileTask::CompileReason, JavaThread*)+85
>
> CompilationPolicy::compile(methodHandle const&, int, CompLevel, 
> JavaThread*)+456
>
> CompilationPolicy::event(methodHandle const&, methodHandle const&, 
> int, int, CompLevel, CompiledMethod*, JavaThread*)+553
>
> InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, 
> unsigned char*)+331
>
> InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned 
> char*)+27
>
> Interpreter+15968
>
>         void 
> com.hsbc.efx.actor.dispatcher.SingleThreadDispatcher$DispatchThread.run()+788
>
> kstack:
>
> syscall_slow_exit_work+179
>
> syscall_slow_exit_work+179
>
> do_syscall_64+365
>
> entry_SYSCALL_64_after_hwframe+102
>
> This usually starts happening a few minutes after we start the 
> application, probably long enough to reach compilation thresholds. It 
> can last for few to tens of minutes after which it might fix itself 
> and all this activity disappears.
> This does not happen on any jdk17 version that we used. My gut feeling 
> is some sort of a live lock somewhere in the profiler? We have limited 
> means of reproducing it due to environment constraints but if you’d 
> like us to run with some extra flags or on a fast debug build, we 
> could arrange that.
> Also, I’m OpenJDK author with access to the bug tracker if you think 
> we should create and issue for this.
>
> Thanks
>
> *Wojciech KUDLA*
>
> eFX eRisk Infrastructure
> *HSBC Bank plc*
> 8 Canada Square, London E14 5HQ
> Telephone: +44 (0)203 359 3827
> Mobile: +44 7895 833 903
> E-mail: wojciech.kudla at hsbc.com <mailto:wojciech.kudla at hsbc.com>
>
>
> PUBLIC
>
> ------------------------------------------------------------------------
> -SAVE PAPER - THINK BEFORE YOU PRINT!
>
> This E-mail is confidential.
>
> It may also be legally privileged. If you are not the addressee you 
> may not copy,
> forward, disclose or use any part of it. If you have received this 
> message in error,
> please delete it and all copies from your system and notify the sender 
> immediately by
> return E-mail.
>
> Internet communications cannot be guaranteed to be timely secure, 
> error or virus-free.
> The sender does not accept liability for any errors or omissions.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-dev/attachments/20251114/adf0ef84/attachment-0001.htm>