<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Aptos;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-compose;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-GB" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi,<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">(Somehow this message didn’t make it through to hotspot-compiler-dev, so trying here)<o:p></o:p></p>
<p class="MsoNormal"><br>
We’ve observed a strange performance issue within our latency-sensitive application when running on jdk21 (tested with 21.0.3 and 21.0.8).<br>
The impacted threads are pinned to their respective isolated cores so not expected to experience any disruptions with the exception of a regular hrtick but that executes in user context and is always a sub-microsecond thing.<br>
First, we noticed that these threads started showing voluntary context switching. Since we avoid any non-vdso syscalls this was a strong indicator of some unintended locking going on and so we decided to capture user- and kernel-space stack traces for one such
thread with a bit of eBPF.<br>
It looks like we go into this death loop of OSRs (the only type of compilation activity known to me to halt execution on the impacted thread) and this happens tens of times per second.<br>
Here’s the stack traces:<br>
<br>
<span style="font-family:"Courier New"">ustack:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> __lll_unlock_wake+26<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> CompileBroker::compile_method(methodHandle const&, int, int, methodHandle const&, int, CompileTask::CompileReason, JavaThread*)+85<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> CompilationPolicy::compile(methodHandle const&, int, CompLevel, JavaThread*)+456<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> CompilationPolicy::event(methodHandle const&, methodHandle const&, int, int, CompLevel, CompiledMethod*, JavaThread*)+553<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, unsigned char*)+331<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned char*)+27<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> Interpreter+15968<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> void com.hsbc.efx.actor.dispatcher.SingleThreadDispatcher$DispatchThread.run()+788<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">kstack:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> syscall_trace_enter+686<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> syscall_trace_enter+686<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> do_syscall_64+326<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> entry_SYSCALL_64_after_hwframe+102<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">ustack:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> __lll_unlock_wake+26<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> CompileBroker::compile_method(methodHandle const&, int, int, methodHandle const&, int, CompileTask::CompileReason, JavaThread*)+85<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> CompilationPolicy::compile(methodHandle const&, int, CompLevel, JavaThread*)+456<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> CompilationPolicy::event(methodHandle const&, methodHandle const&, int, int, CompLevel, CompiledMethod*, JavaThread*)+553<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, unsigned char*)+331<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned char*)+27<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> Interpreter+15968<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> void com.hsbc.efx.actor.dispatcher.SingleThreadDispatcher$DispatchThread.run()+788<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">kstack:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> syscall_slow_exit_work+179<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> syscall_slow_exit_work+179<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> do_syscall_64+365<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> entry_SYSCALL_64_after_hwframe+102<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">ustack:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> __lll_unlock_wake+26<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> CompilationPolicy::compile(methodHandle const&, int, CompLevel, JavaThread*)+456<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> CompilationPolicy::event(methodHandle const&, methodHandle const&, int, int, CompLevel, CompiledMethod*, JavaThread*)+553<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, unsigned char*)+331<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned char*)+27<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> Interpreter+15968<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> void com.hsbc.efx.actor.dispatcher.SingleThreadDispatcher$DispatchThread.run()+788<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">kstack:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> syscall_trace_enter+686<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> syscall_trace_enter+686<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> do_syscall_64+326<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> entry_SYSCALL_64_after_hwframe+102<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">ustack:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> __lll_unlock_wake+26<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> CompilationPolicy::compile(methodHandle const&, int, CompLevel, JavaThread*)+456<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> CompilationPolicy::event(methodHandle const&, methodHandle const&, int, int, CompLevel, CompiledMethod*, JavaThread*)+553<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, unsigned char*)+331<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned char*)+27<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> Interpreter+15968<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> void com.hsbc.efx.actor.dispatcher.SingleThreadDispatcher$DispatchThread.run()+788<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">kstack:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> syscall_slow_exit_work+179<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> syscall_slow_exit_work+179<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> do_syscall_64+365<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> entry_SYSCALL_64_after_hwframe+102<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">ustack:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> __lll_lock_wait+29<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> DirectivesStack::getMatchingDirective(methodHandle const&, AbstractCompiler*)+50<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> CompileBroker::compile_method(methodHandle const&, int, int, methodHandle const&, int, CompileTask::CompileReason, JavaThread*)+85<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> CompilationPolicy::compile(methodHandle const&, int, CompLevel, JavaThread*)+456<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> CompilationPolicy::event(methodHandle const&, methodHandle const&, int, int, CompLevel, CompiledMethod*, JavaThread*)+553<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, unsigned char*)+331<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned char*)+27<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> Interpreter+15968<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> void com.hsbc.efx.actor.dispatcher.SingleThreadDispatcher$DispatchThread.run()+788<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">kstack:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> syscall_trace_enter+686<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> syscall_trace_enter+686<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> do_syscall_64+326<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> entry_SYSCALL_64_after_hwframe+102<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">ustack:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> __lll_lock_wait+29<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> DirectivesStack::getMatchingDirective(methodHandle const&, AbstractCompiler*)+50<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> CompileBroker::compile_method(methodHandle const&, int, int, methodHandle const&, int, CompileTask::CompileReason, JavaThread*)+85<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> CompilationPolicy::compile(methodHandle const&, int, CompLevel, JavaThread*)+456<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> CompilationPolicy::event(methodHandle const&, methodHandle const&, int, int, CompLevel, CompiledMethod*, JavaThread*)+553<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, unsigned char*)+331<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned char*)+27<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> Interpreter+15968<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> void com.hsbc.efx.actor.dispatcher.SingleThreadDispatcher$DispatchThread.run()+788<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">kstack:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> syscall_slow_exit_work+179<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> syscall_slow_exit_work+179<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> do_syscall_64+365<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> entry_SYSCALL_64_after_hwframe+102<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This usually starts happening a few minutes after we start the application, probably long enough to reach compilation thresholds. It can last for few to tens of minutes after which it might fix itself and all this activity disappears.<br>
This does not happen on any jdk17 version that we used. My gut feeling is some sort of a live lock somewhere in the profiler? We have limited means of reproducing it due to environment constraints but if you’d like us to run with some extra flags or on a fast
debug build, we could arrange that.<br>
Also, I’m OpenJDK author with access to the bug tracker if you think we should create and issue for this.<br>
<br>
Thanks<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:9.0pt;color:#BF0000;mso-fareast-language:EN-GB">Wojciech KUDLA</span></b><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="color:#3F3F3F;mso-fareast-language:EN-GB">eFX eRisk Infrastructure </span><span style="mso-fareast-language:EN-GB"><br>
<b><span style="color:#3F3F3F">HSBC Bank plc</span></b><br>
<span style="color:#3F3F3F">8 Canada Square, London E14 5HQ</span><br>
<span style="color:#BF0000">Telephone: </span><span style="color:#3F3F3F">+44 (0)203 359 3827</span><br>
<span style="color:#BF0000">Mobile: </span><span style="color:#3F3F3F">+44 7895 833 903</span><br>
<span style="color:#BF0000">E-mail: </span><span style="color:black">
<a href="mailto:wojciech.kudla@hsbc.com" target="_blank"><span style="color:blue">wojciech.kudla@hsbc.com</span></a></span><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br>
<p style="font-family:Calibri;font-size:10pt;color:#000000;margin:5pt;" align="Center">
PUBLIC<br>
</p>
<HR>-SAVE PAPER - THINK BEFORE YOU PRINT!<BR>
<BR>
This E-mail is confidential. <BR>
<BR>
It may also be legally privileged. If you are not the addressee you may not copy,<BR>
forward, disclose or use any part of it. If you have received this message in error,<BR>
please delete it and all copies from your system and notify the sender immediately by<BR>
return E-mail.<BR>
<BR>
Internet communications cannot be guaranteed to be timely secure, error or virus-free.<BR>
The sender does not accept liability for any errors or omissions.<BR>
</body>
</html>