Potential OSR-related performance issue in jdk21

[Wojciech KUDLA]

Hi,

(Somehow this message didn't make it through to hotspot-compiler-dev, so trying here)

We've observed a strange performance issue within our latency-sensitive application when running on jdk21 (tested with 21.0.3 and 21.0.8).
The impacted threads are pinned to their respective isolated cores so not expected to experience any disruptions with the exception of a regular hrtick but that executes in user context and is always a sub-microsecond thing.
First, we noticed that these threads started showing voluntary context switching. Since we avoid any non-vdso syscalls this was a strong indicator of some unintended locking going on and so we decided to capture user- and kernel-space stack traces for one such thread with a bit of eBPF.
It looks like we go into this death loop of OSRs (the only type of compilation activity known to me to halt execution on the impacted thread) and this happens tens of times per second.
Here's the stack traces:

ustack:
        __lll_unlock_wake+26
        CompileBroker::compile_method(methodHandle const&, int, int, methodHandle const&, int, CompileTask::CompileReason, JavaThread*)+85
        CompilationPolicy::compile(methodHandle const&, int, CompLevel, JavaThread*)+456
        CompilationPolicy::event(methodHandle const&, methodHandle const&, int, int, CompLevel, CompiledMethod*, JavaThread*)+553
        InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, unsigned char*)+331
        InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned char*)+27
        Interpreter+15968
        void com.hsbc.efx.actor.dispatcher.SingleThreadDispatcher$DispatchThread.run()+788
kstack:

        syscall_trace_enter+686
        syscall_trace_enter+686
        do_syscall_64+326
        entry_SYSCALL_64_after_hwframe+102



ustack:
       __lll_unlock_wake+26
        CompileBroker::compile_method(methodHandle const&, int, int, methodHandle const&, int, CompileTask::CompileReason, JavaThread*)+85
        CompilationPolicy::compile(methodHandle const&, int, CompLevel, JavaThread*)+456
        CompilationPolicy::event(methodHandle const&, methodHandle const&, int, int, CompLevel, CompiledMethod*, JavaThread*)+553
        InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, unsigned char*)+331
        InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned char*)+27
        Interpreter+15968
        void com.hsbc.efx.actor.dispatcher.SingleThreadDispatcher$DispatchThread.run()+788
kstack:

        syscall_slow_exit_work+179
        syscall_slow_exit_work+179
        do_syscall_64+365
        entry_SYSCALL_64_after_hwframe+102


ustack:
        __lll_unlock_wake+26
        CompilationPolicy::compile(methodHandle const&, int, CompLevel, JavaThread*)+456
        CompilationPolicy::event(methodHandle const&, methodHandle const&, int, int, CompLevel, CompiledMethod*, JavaThread*)+553
        InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, unsigned char*)+331
        InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned char*)+27
        Interpreter+15968
        void com.hsbc.efx.actor.dispatcher.SingleThreadDispatcher$DispatchThread.run()+788
kstack:
        syscall_trace_enter+686
        syscall_trace_enter+686
        do_syscall_64+326
        entry_SYSCALL_64_after_hwframe+102


ustack:
        __lll_unlock_wake+26
        CompilationPolicy::compile(methodHandle const&, int, CompLevel, JavaThread*)+456
        CompilationPolicy::event(methodHandle const&, methodHandle const&, int, int, CompLevel, CompiledMethod*, JavaThread*)+553
        InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, unsigned char*)+331
        InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned char*)+27
        Interpreter+15968
        void com.hsbc.efx.actor.dispatcher.SingleThreadDispatcher$DispatchThread.run()+788
kstack:
        syscall_slow_exit_work+179
        syscall_slow_exit_work+179
        do_syscall_64+365
        entry_SYSCALL_64_after_hwframe+102


ustack:
        __lll_lock_wait+29
        DirectivesStack::getMatchingDirective(methodHandle const&, AbstractCompiler*)+50
        CompileBroker::compile_method(methodHandle const&, int, int, methodHandle const&, int, CompileTask::CompileReason, JavaThread*)+85
        CompilationPolicy::compile(methodHandle const&, int, CompLevel, JavaThread*)+456
        CompilationPolicy::event(methodHandle const&, methodHandle const&, int, int, CompLevel, CompiledMethod*, JavaThread*)+553
        InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, unsigned char*)+331
        InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned char*)+27
        Interpreter+15968
        void com.hsbc.efx.actor.dispatcher.SingleThreadDispatcher$DispatchThread.run()+788
kstack:
        syscall_trace_enter+686
        syscall_trace_enter+686
        do_syscall_64+326
        entry_SYSCALL_64_after_hwframe+102


ustack:
        __lll_lock_wait+29
        DirectivesStack::getMatchingDirective(methodHandle const&, AbstractCompiler*)+50
        CompileBroker::compile_method(methodHandle const&, int, int, methodHandle const&, int, CompileTask::CompileReason, JavaThread*)+85
        CompilationPolicy::compile(methodHandle const&, int, CompLevel, JavaThread*)+456
        CompilationPolicy::event(methodHandle const&, methodHandle const&, int, int, CompLevel, CompiledMethod*, JavaThread*)+553
        InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, unsigned char*)+331
        InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned char*)+27
        Interpreter+15968
        void com.hsbc.efx.actor.dispatcher.SingleThreadDispatcher$DispatchThread.run()+788
kstack:
        syscall_slow_exit_work+179
        syscall_slow_exit_work+179
        do_syscall_64+365
        entry_SYSCALL_64_after_hwframe+102


This usually starts happening a few minutes after we start the application, probably long enough to reach compilation thresholds. It can last for few to tens of minutes after which it might fix itself and all this activity disappears.
This does not happen on any jdk17 version that we used. My gut feeling is some sort of a live lock somewhere in the profiler? We have limited means of reproducing it due to environment constraints but if you'd like us to run with some extra flags or on a fast debug build, we could arrange that.
Also, I'm OpenJDK author with access to the bug tracker if you think we should create and issue for this.

Thanks
Wojciech KUDLA
eFX eRisk Infrastructure
HSBC Bank plc
8 Canada Square, London E14 5HQ
Telephone:            +44 (0)203 359 3827
Mobile:                  +44 7895 833 903
E-mail:                     wojciech.kudla at hsbc.com<mailto:wojciech.kudla at hsbc.com>



PUBLIC

______________________________________________________________________
-SAVE PAPER - THINK BEFORE YOU PRINT!

This E-mail is confidential. 

It may also be legally privileged. If you are not the addressee you may not copy,
forward, disclose or use any part of it. If you have received this message in error,
please delete it and all copies from your system and notify the sender immediately by
return E-mail.

Internet communications cannot be guaranteed to be timely secure, error or virus-free.
The sender does not accept liability for any errors or omissions.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-dev/attachments/20251114/3e13702f/attachment-0001.htm>