<div dir="ltr">Hi Alan,<div><br></div><div>About deadlock on Java 21 while serving static contents (which is resolved on your build), I deep dived a bit. You are right. The culprit is most probably <i><b>not File I/O</b></i>. What <i><b>Spring-MVC</b></i> does is that it <i><b>caches</b></i> from which location (out of multiple available candidates) it eventually manages to resolve the static resource and then it proceeds to do <i><b>Classloader.getResourceAsStream()</b> </i>to get the file. The cache implementation is backed by <i><b>ConcurrentHashMap</b></i> and it calls <b><i>put(k,v)</i> </b>method on the map which involves <i><b>synchronized block.</b> </i>I just didn't understand how it can happen even with very few concurrent requests.</div><div><br></div><div>Thanks for instructing me to use <i><b>jcmd</b></i> and yes it's a <i><b>12 core</b></i> machine. I ran the test again and got 2 thread dumps. One from <i><b>jvisualvm</b></i> and one from <i><b>jcmd</b></i> so you can co-relate them. Please find them attached.</div><div>It's a deadlock on classloader. 11 out of 12 carrier threads are block on a <i>synchronised<b> block</b></i> at <i><b>java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:651)</b> </i>and the other one which is </div><div><i><b>virtual thread #120 (forkjoinpool-1-worker-14) </b></i>, is stuck in a <i>synchronized<b> </b>block</i> at <b style="font-style:italic">java.base/java.util.zip.ZipFile.getEntry(ZipFile.java:339).</b></div><div><b style="font-style:italic"><br></b></div><div>I was wondering how this one is happening on 21 and current EA of 23. </div><div><br></div><div>Looking forward to your analysis and please let me know if you need further tests.</div><div><br></div><div>Kind regards,</div><div>Masoud</div><div><b style="font-style:italic"><br></b></div><div><br></div></div><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 16, 2024 at 4:10 PM Alan Bateman <<a href="mailto:Alan.Bateman@oracle.com" target="_blank">Alan.Bateman@oracle.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u>
<div>
On 16/02/2024 14:18, masoud parvari wrote:<br>
<blockquote type="cite">
<div>Hi Alan, Thanks for your
very valuable changes on monitor implementation.
</div>
</blockquote>
<br>
(Credit goes to Patricio Chicano Mateo for most of the heavy
lifting).<br>
<br>
It's great that you were able to try out the EA builds. We need to
get as much testing as possible. Some questions/comments below.<br>
<br>
<blockquote type="cite">
<div>:<br>
<div><br>
</div>
<div>In my setup everything runs on VT and I must admit it
hasn’t been as smooth as I hoped. Pinning and hanging
(deadlock) was happening a lot. Specially hanging. I
eventually ended up fixing (almost) all of the issues by
switching to implementations that are loom friendly
(ReentrantLock in favour of Synchronized block). But hanging
still happens for example when serving static resources (File
I/O) with spring boot. I ended up writing custom code to fix
the problem.</div>
</div>
</blockquote>
Can you say a bit more about this? I can't think how file I/O would
lead to deadlock but you've listed several frameworks and libraries
so there is lot going on. Class initializers can be problematic and
may surface more once the issues with monitors pinning goes away.<br>
<br>
<br>
<blockquote type="cite">
<div>
<div><br>
</div>
<div>So as soon as your build came out, I gave it a try. It
helped immediately with serving static resources when I tested
with one user requesting multiple static resources and threads
were not hanging anymore while on 21 they would have hanged
even with one user.</div>
<div><br>
</div>
<div>So I started to perform local load test on the application
and that’s when things go wrong. For load test I even turned
off serving static resources. Hanging happens every single
time I hit around 300 concurrent users, while on my Java 21
setup I could go to 2000-2500 concurrent users before hitting
DB thread pool size. So there is definitely a regression
compared to Java 21. I also ran the test on Java 23-ea+8 and
it performed similar to 21. So new ForkjoinPool improvements
(assuming they exist in 23-ea+8) can’t be the problem.</div>
<div><br>
</div>
<div>When hanging happens , all Carrier threads are on WAITING
state with a stack trace like this:</div>
<div><br>
</div>
<div>
<div>Name: ForkJoinPool-1-worker-1</div>
<div>State: WAITING on java.lang.VirtualThread@17d1aaa2 owned
by: tomcat-virt-70</div>
<div>Total blocked: 4.978 Total waited: 2.387</div>
</div>
</div>
</blockquote>
<br>
There are 12 ForkJoinPool-1-worker-<n> threads in the thread
dump. I assume this is a 12 core system. All 12 include the text
"Carrying virtual thread #<m>" where #<m> is the thread
ID of the virtual thread is mounted on that carrier at the time.<br>
<br>
Would it be possible to run `jcmd <pid> Thread.dump_to_file
<file>` to capture a thread dump that includes the virtual
thread. This isn't the same thing as the HotSpot VM thread dump that
jvisualvm is printing. It's has less details but it will at least
let you see what the virtual threads are doing.<br>
<br>
<br>
<blockquote type="cite">
<div>
<div>
<div>:<br>
</div>
</div>
<div><br>
</div>
<div>I am also listening to <span style="color:rgb(0,0,0)">jdk.VirtualThreadPinnedEvent
and they never happened. I tested with both </span><span style="color:rgb(0,0,0)">-XX:LockingMode=1 and </span><font color="#000000">-XX:LockingMode=2 and hanging happened in
both cases and I didn<span>’</span>t notice a
significant difference there.</font></div>
<div><font color="#000000"><br>
</font></div>
</div>
</blockquote>
<span style="color:rgb(0,0,0)">Until recently,
a jdk.VirtualThreadPinnedEvent was only recorded when the thread
continues after pinning a carrier for more than 20ms. So maybe
something is pinned indefinitely so there is no event recorded.
This has since changed, but not in these EA builds, so this event
is always recorded.<br>
<br>
-Alan<br>
</span><br>
</div>
</blockquote></div></div>