EA builds with changes to object monitor implementation to avoid pinning with virtual threads

[masoud parvari]

Hi Alan, Thanks for your very valuable changes on monitor implementation.

I have been recently working on the performance improvement of a Java enterprise application using Virtual threads on Java 21. The stack looks like this: Tomcat 9.0.85 (using VT) , spring boot 2.7.18 using VT, Hibernate (Using Ehcache Jcache implementation  for second level caching), HikariPool, MySQL, RabbitMQ publisher.

In my setup everything runs on VT and I must admit it hasn’t been as smooth as I hoped. Pinning and hanging (deadlock) was happening a lot. Specially hanging. I eventually ended up fixing (almost) all of the issues by switching to implementations that are loom friendly (ReentrantLock in favour of Synchronized block). But hanging still happens for example when serving static resources (File I/O) with spring boot. I ended up writing custom code to fix the problem.

So as soon as your build came out, I gave it a try. It helped immediately with serving static resources when I tested with one user requesting multiple static resources and threads were not hanging anymore while on 21 they would have hanged even with one user.

So I started to perform local load test on the application and that’s when things go wrong. For load test I even turned off serving static resources. Hanging happens every single time I hit around 300 concurrent users, while on my Java 21 setup I could go to 2000-2500 concurrent users before hitting DB thread pool size. So there is definitely a regression compared to Java 21. I also ran the test on Java 23-ea+8 and it performed similar to 21. So new ForkjoinPool improvements (assuming they exist in 23-ea+8) can’t be the problem.

When hanging happens , all Carrier threads are on WAITING state with a stack trace like this:

Name: ForkJoinPool-1-worker-1
State: WAITING on java.lang.VirtualThread at 17d1aaa2 owned by: tomcat-virt-70
Total blocked: 4.978  Total waited: 2.387

Stack trace: 
java.base at 23-loom/jdk.internal.vm.Continuation.run(Continuation.java:251)
java.base at 23-loom/java.lang.VirtualThread.runContinuation(VirtualThread.java:270)
java.base at 23-loom/java.lang.VirtualThread$$Lambda/0x00001e0001d695d8.run(Unknown Source)
java.base at 23-loom/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1726)
java.base at 23-loom/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1717)
java.base at 23-loom/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(ForkJoinTask.java:1641)
java.base at 23-loom/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
java.base at 23-loom/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:668)
java.base at 23-loom/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1280)
java.base at 23-loom/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1204)
java.base at 23-loom/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)

I am also listening to jdk.VirtualThreadPinnedEvent and they never happened. I tested with both -XX:LockingMode=1 and -XX:LockingMode=2 and hanging happened in both cases and I didn’t notice a significant difference there.

I attached a jvisualvm threaddump as well to give you more insight. I hope it helps.

Let me know if you need more feedback and testings.

Kind regards,
Masoud

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20240216/fe164a9e/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: threaddump-1708087673476.tdump
Type: application/octet-stream
Size: 89630 bytes
Desc: not available
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20240216/fe164a9e/threaddump-1708087673476-0001.tdump>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20240216/fe164a9e/attachment-0003.htm>