EA builds with changes to object monitor implementation to avoid pinning with virtual threads

[Alan Bateman]

On 16/02/2024 14:18, masoud parvari wrote:
> Hi Alan, Thanks for your very valuable changes on monitor implementation.

(Credit goes to Patricio Chicano Mateo for most of the heavy lifting).

It's great that you were able to try out the EA builds. We need to get 
as much testing as possible. Some questions/comments below.

> :
>
> In my setup everything runs on VT and I must admit it hasn’t been as 
> smooth as I hoped. Pinning and hanging (deadlock) was happening a lot. 
> Specially hanging. I eventually ended up fixing (almost) all of the 
> issues by switching to implementations that are loom friendly 
> (ReentrantLock in favour of Synchronized block). But hanging still 
> happens for example when serving static resources (File I/O) with 
> spring boot. I ended up writing custom code to fix the problem.
Can you say a bit more about this?  I can't think how file I/O would 
lead to deadlock but you've listed several frameworks and libraries so 
there is lot going on. Class initializers can be problematic and may 
surface more once the issues with monitors pinning goes away.


>
> So as soon as your build came out, I gave it a try. It helped 
> immediately with serving static resources when I tested with one user 
> requesting multiple static resources and threads were not hanging 
> anymore while on 21 they would have hanged even with one user.
>
> So I started to perform local load test on the application and that’s 
> when things go wrong. For load test I even turned off serving static 
> resources. Hanging happens every single time I hit around 300 
> concurrent users, while on my Java 21 setup I could go to 2000-2500 
> concurrent users before hitting DB thread pool size. So there is 
> definitely a regression compared to Java 21. I also ran the test on 
> Java 23-ea+8 and it performed similar to 21. So new ForkjoinPool 
> improvements (assuming they exist in 23-ea+8) can’t be the problem.
>
> When hanging happens , all Carrier threads are on WAITING state with a 
> stack trace like this:
>
> Name: ForkJoinPool-1-worker-1
> State: WAITING on java.lang.VirtualThread at 17d1aaa2 owned by: 
> tomcat-virt-70
> Total blocked: 4.978  Total waited: 2.387

There are 12 ForkJoinPool-1-worker-<n> threads in the thread dump. I 
assume this is a 12 core system. All 12 include the text "Carrying 
virtual thread #<m>" where #<m> is the thread ID of the virtual thread 
is mounted on that carrier at the time.

Would it be possible to run `jcmd <pid> Thread.dump_to_file <file>` to 
capture a thread dump that includes the virtual thread. This isn't the 
same thing as the HotSpot VM thread dump that jvisualvm is printing. 
It's has less details but it will at least let you see what the virtual 
threads are doing.


> :
>
> I am also listening to jdk.VirtualThreadPinnedEvent and they never 
> happened. I tested with both -XX:LockingMode=1 and -XX:LockingMode=2 
> and hanging happened in both cases and I didn’t notice a significant 
> difference there.
>
Until recently, a jdk.VirtualThreadPinnedEvent was only recorded when 
the thread continues after pinning a carrier for more than 20ms. So 
maybe something is pinned indefinitely so there is no event recorded. 
This has since changed, but not in these EA builds, so this event is 
always recorded.

-Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20240216/252f29b4/attachment.htm>