<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Hi Alan, Thanks for your very valuable changes on monitor implementation.<div><br></div><div>I have been recently working on the performance improvement of a Java enterprise application using Virtual threads on Java 21. The stack looks like this: Tomcat 9.0.85 (using VT) , spring boot 2.7.18 using VT, Hibernate (Using Ehcache Jcache implementation for second level caching), HikariPool, MySQL, RabbitMQ publisher.</div><div><br></div><div>In my setup everything runs on VT and I must admit it hasn’t been as smooth as I hoped. Pinning and hanging (deadlock) was happening a lot. Specially hanging. I eventually ended up fixing (almost) all of the issues by switching to implementations that are loom friendly (ReentrantLock in favour of Synchronized block). But hanging still happens for example when serving static resources (File I/O) with spring boot. I ended up writing custom code to fix the problem.</div><div><br></div><div>So as soon as your build came out, I gave it a try. It helped immediately with serving static resources when I tested with one user requesting multiple static resources and threads were not hanging anymore while on 21 they would have hanged even with one user.</div><div><br></div><div>So I started to perform local load test on the application and that’s when things go wrong. For load test I even turned off serving static resources. Hanging happens every single time I hit around 300 concurrent users, while on my Java 21 setup I could go to 2000-2500 concurrent users before hitting DB thread pool size. So there is definitely a regression compared to Java 21. I also ran the test on Java 23-ea+8 and it performed similar to 21. So new ForkjoinPool improvements (assuming they exist in 23-ea+8) can’t be the problem.</div><div><br></div><div>When hanging happens , all Carrier threads are on WAITING state with a stack trace like this:</div><div><br></div><div><div>Name: ForkJoinPool-1-worker-1</div><div>State: WAITING on java.lang.VirtualThread@17d1aaa2 owned by: tomcat-virt-70</div><div>Total blocked: 4.978 Total waited: 2.387</div><div><br></div><div>Stack trace: </div><div>java.base@23-loom/jdk.internal.vm.Continuation.run(Continuation.java:251)</div><div>java.base@23-loom/java.lang.VirtualThread.runContinuation(VirtualThread.java:270)</div><div>java.base@23-loom/java.lang.VirtualThread$$Lambda/0x00001e0001d695d8.run(Unknown Source)</div><div>java.base@23-loom/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1726)</div><div>java.base@23-loom/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1717)</div><div>java.base@23-loom/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(ForkJoinTask.java:1641)</div><div>java.base@23-loom/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)</div><div>java.base@23-loom/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:668)</div><div>java.base@23-loom/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1280)</div><div>java.base@23-loom/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1204)</div><div>java.base@23-loom/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)</div></div><div><br></div><div style="orphans: 2; widows: 2;">I am also listening to <span style="color: rgb(0, 0, 0); orphans: 2; widows: 2;">jdk.VirtualThreadPinnedEvent and they never happened. I tested with both </span><span style="color: rgb(0, 0, 0);">-XX:LockingMode=1 and </span><font color="#000000">-XX:LockingMode=2 and hanging happened in both cases and I didn<span style="caret-color: rgb(0, 0, 0);">’</span>t notice a significant difference there.</font></div><div style="orphans: 2; widows: 2;"><font color="#000000"><br></font></div><div style="orphans: 2; widows: 2;"><font color="#000000">I attached a jvisualvm <span style="caret-color: rgb(0, 0, 0);">threaddump as well to give you more insight. I hope it helps.</span></font></div><div style="orphans: 2; widows: 2;"><font color="#000000"><span style="caret-color: rgb(0, 0, 0);"><br></span></font></div><div style="orphans: 2; widows: 2;"><font color="#000000"><span style="caret-color: rgb(0, 0, 0);">Let me know if you need more feedback and testings.</span></font></div><div style="orphans: 2; widows: 2;"><font color="#000000"><span style="caret-color: rgb(0, 0, 0);"><br></span></font></div><div style="orphans: 2; widows: 2;"><font color="#000000"><span style="caret-color: rgb(0, 0, 0);">Kind regards,</span></font></div><div style="orphans: 2; widows: 2;"><font color="#000000"><span style="caret-color: rgb(0, 0, 0);">Masoud</span></font></div><div style="orphans: 2; widows: 2;"><font color="#000000"></font></div></body></html>