<!DOCTYPE html><html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
On 16/02/2024 14:18, masoud parvari wrote:<br>
<blockquote type="cite" cite="mid:981467E7-BF7E-41F6-8966-55A3ABA858CA@gmail.com">
<div style="line-break:after-white-space">Hi Alan, Thanks for your
very valuable changes on monitor implementation.
</div>
</blockquote>
<br>
(Credit goes to Patricio Chicano Mateo for most of the heavy
lifting).<br>
<br>
It's great that you were able to try out the EA builds. We need to
get as much testing as possible. Some questions/comments below.<br>
<br>
<blockquote type="cite" cite="mid:981467E7-BF7E-41F6-8966-55A3ABA858CA@gmail.com">
<div style="line-break:after-white-space">:<br>
<div><br>
</div>
<div>In my setup everything runs on VT and I must admit it
hasn’t been as smooth as I hoped. Pinning and hanging
(deadlock) was happening a lot. Specially hanging. I
eventually ended up fixing (almost) all of the issues by
switching to implementations that are loom friendly
(ReentrantLock in favour of Synchronized block). But hanging
still happens for example when serving static resources (File
I/O) with spring boot. I ended up writing custom code to fix
the problem.</div>
</div>
</blockquote>
Can you say a bit more about this? I can't think how file I/O would
lead to deadlock but you've listed several frameworks and libraries
so there is lot going on. Class initializers can be problematic and
may surface more once the issues with monitors pinning goes away.<br>
<br>
<br>
<blockquote type="cite" cite="mid:981467E7-BF7E-41F6-8966-55A3ABA858CA@gmail.com">
<div style="line-break:after-white-space">
<div><br>
</div>
<div>So as soon as your build came out, I gave it a try. It
helped immediately with serving static resources when I tested
with one user requesting multiple static resources and threads
were not hanging anymore while on 21 they would have hanged
even with one user.</div>
<div><br>
</div>
<div>So I started to perform local load test on the application
and that’s when things go wrong. For load test I even turned
off serving static resources. Hanging happens every single
time I hit around 300 concurrent users, while on my Java 21
setup I could go to 2000-2500 concurrent users before hitting
DB thread pool size. So there is definitely a regression
compared to Java 21. I also ran the test on Java 23-ea+8 and
it performed similar to 21. So new ForkjoinPool improvements
(assuming they exist in 23-ea+8) can’t be the problem.</div>
<div><br>
</div>
<div>When hanging happens , all Carrier threads are on WAITING
state with a stack trace like this:</div>
<div><br>
</div>
<div>
<div>Name: ForkJoinPool-1-worker-1</div>
<div>State: WAITING on java.lang.VirtualThread@17d1aaa2 owned
by: tomcat-virt-70</div>
<div>Total blocked: 4.978 Total waited: 2.387</div>
</div>
</div>
</blockquote>
<br>
There are 12 ForkJoinPool-1-worker-<n> threads in the thread
dump. I assume this is a 12 core system. All 12 include the text
"Carrying virtual thread #<m>" where #<m> is the thread
ID of the virtual thread is mounted on that carrier at the time.<br>
<br>
Would it be possible to run `jcmd <pid> Thread.dump_to_file
<file>` to capture a thread dump that includes the virtual
thread. This isn't the same thing as the HotSpot VM thread dump that
jvisualvm is printing. It's has less details but it will at least
let you see what the virtual threads are doing.<br>
<br>
<br>
<blockquote type="cite" cite="mid:981467E7-BF7E-41F6-8966-55A3ABA858CA@gmail.com">
<div style="line-break:after-white-space">
<div>
<div>:<br>
</div>
</div>
<div><br>
</div>
<div style="orphans:2; widows:2">I am also listening to <span style="color:rgb(0,0,0); orphans:2; widows:2">jdk.VirtualThreadPinnedEvent
and they never happened. I tested with both </span><span style="color:rgb(0,0,0)">-XX:LockingMode=1 and </span><font color="#000000">-XX:LockingMode=2 and hanging happened in
both cases and I didn<span style="">’</span>t notice a
significant difference there.</font></div>
<div style="orphans:2; widows:2"><font color="#000000"><br>
</font></div>
</div>
</blockquote>
<span style="color:rgb(0,0,0); orphans:2; widows:2">Until recently,
a jdk.VirtualThreadPinnedEvent was only recorded when the thread
continues after pinning a carrier for more than 20ms. So maybe
something is pinned indefinitely so there is no event recorded.
This has since changed, but not in these EA builds, so this event
is always recorded.<br>
<br>
-Alan<br>
</span><br>
</body>
</html>