<div dir="ltr">Thanks Alan for the detailed explanation. This reminds me of Java 21 deadlock scenarios when obtaining a lock both outside and inside a synchronized block by some number of threads greater than machine cores, could potentially cause deadlock.<div><br></div><div>I still have one question though. This deadlock scenario (a burst of virtual threads at startup with a mix of class loading (which comes with pinning) and resource loading from the same JAR files) is not happening on 21 and latest 23-ea. I suspect because Forkjoinpool creates more thread to compensate for pinning? Would you please shed some light on this?</div><div><br></div><div>And is there any way we can have that behaviour reintroduced into loom only for class loading scenarios as a temporary workaround until the fundamental work for classloading is done? Or is there anything at the developer's side that can be done to avoid this? The thing is that with 21 and 23-ea, at least we could opt for implementations that favour Reentrantlock over synchronized (even though a very painful and time consuming approach) and have a working setup, but with your build it is almost impossible to survive the load.</div><div><br></div><div>I am not undermining the loom team's work at all. It's a great milestone. And we definitely wanna help the team by testing it and I personally would love to have your work in 23 ea builds sooner than later. But the testing itself can't be accomplished because of deadlock. So it sounds like a chicken egg problem to me. I am looking forward to hearing your opinion on this.</div><div><br></div><div>Kind regards,</div><div>Masoud</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Feb 17, 2024 at 11:20 AM Alan Bateman <<a href="mailto:Alan.Bateman@oracle.com">Alan.Bateman@oracle.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u>
<div>
On 16/02/2024 19:58, masoud parvari wrote:<br>
<blockquote type="cite">
<div dir="ltr">Hi Alan,
<div><br>
</div>
<div>About deadlock on Java 21 while serving static contents
(which is resolved on your build), I deep dived a bit. You are
right. The culprit is most probably <i><b>not File I/O</b></i>.
What <i><b>Spring-MVC</b></i> does is that it <i><b>caches</b></i>
from which location (out of multiple available candidates) it
eventually manages to resolve the static resource and then it
proceeds to do <i><b>Classloader.getResourceAsStream()</b> </i>to
get the file. The cache implementation is backed by <i><b>ConcurrentHashMap</b></i>
and it calls <b><i>put(k,v)</i> </b>method on the map which
involves <i><b>synchronized block.</b> </i>I just didn't
understand how it can happen even with very few concurrent
requests.</div>
<div><br>
</div>
<div>Thanks for instructing me to use <i><b>jcmd</b></i> and
yes it's a <i><b>12 core</b></i> machine. I ran the test
again and got 2 thread dumps. One from <i><b>jvisualvm</b></i> and
one from <i><b>jcmd</b></i> so you can co-relate them. Please
find them attached.</div>
<div>It's a deadlock on classloader. 11 out of 12 carrier
threads are block on a <i>synchronised<b> block</b></i> at <i><b>java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:651)</b>
</i>and the other one which is </div>
<div><i><b>virtual thread #120 (forkjoinpool-1-worker-14) </b></i>,
is stuck in a <i>synchronized<b> </b>block</i> at <b style="font-style:italic">java.base/java.util.zip.ZipFile.getEntry(ZipFile.java:339).</b></div>
<br>
</div>
</blockquote>
<br>
Thanks for sharing the thread dumps. We can see 300 virtual threads.
12 are blocked trying to enter a monitor but are pinned due to
native frames on the stack. No other threads can run as a result.
You won't see these native frames in the stack traces but
essentially all 12 are in
nl.trifork.qti.model.processing.expression.general.BaseValue's
constructor and triggering a class load, which goes through the VM.
Of the 12, 11 are blocked at BuiltinClassLoader.loadClassOrNull as
you pointed out. The built-in class loaders are "parallel capable"
but they do contend when several threads are attempting to load the
same class at the same time. As you found, one of the 12, #120 has
got further but it blocks as a later point due to other threads
(#119 and #123) trying to locate resources in the same JAR file. I
think we can assume that one of these two has been unblocked,
meaning scheduled to continue, but can't continue as there are no
carriers available. If you run `jcmd <pid>
Thread.vthread_scheduler` a few times when this happens then you'll
see the counters stall.<br>
<br>
I agree this is unfortunate, and not easy to avoid. It's essentially
a burst of virtual threads at startup with a mix of class loading
(which comes with pinning) and resource loading from the same JAR
files. Right now, the focus is the pain point of object monitors but
class loading is something that does need attention too.<br>
<br>
-Alan.<br>
<br>
<br>
<br>
</div>
</blockquote></div>