EA builds with changes to object monitor implementation to avoid pinning with virtual threads

[masoud parvari]

Hi Alan,

About deadlock on Java 21 while serving static contents (which is resolved
on your build), I deep dived a bit. You are right. The culprit is most
probably *not File I/O*. What *Spring-MVC* does is that it *caches* from
which location (out of multiple available candidates) it eventually manages
to resolve the static resource and then it proceeds to do
*Classloader.getResourceAsStream()
*to get the file. The cache implementation is backed by *ConcurrentHashMap*
and it calls *put(k,v) *method on the map which involves *synchronized
block. *I just didn't understand how it can happen even with very few
concurrent requests.

Thanks for instructing me to use *jcmd* and yes it's a *12 core* machine. I
ran the test again and got 2 thread dumps. One from *jvisualvm* and one
from *jcmd* so you can co-relate them. Please find them attached.
It's a deadlock on classloader. 11 out of 12 carrier threads are block
on a *synchronised
block* at *java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:651)
*and the other one which is
*virtual thread #120 (forkjoinpool-1-worker-14) *, is stuck in a *synchronized
block* at *java.base/java.util.zip.ZipFile.getEntry(ZipFile.java:339).*

I was wondering how this one is happening on 21 and current EA of 23.

Looking forward to your analysis and please let me know if you need further
tests.

Kind regards,
Masoud




On Fri, Feb 16, 2024 at 4:10 PM Alan Bateman <Alan.Bateman at oracle.com>
wrote:

> On 16/02/2024 14:18, masoud parvari wrote:
>
> Hi Alan, Thanks for your very valuable changes on monitor implementation.
>
>
> (Credit goes to Patricio Chicano Mateo for most of the heavy lifting).
>
> It's great that you were able to try out the EA builds. We need to get as
> much testing as possible. Some questions/comments below.
>
> :
>
> In my setup everything runs on VT and I must admit it hasn’t been as
> smooth as I hoped. Pinning and hanging (deadlock) was happening a lot.
> Specially hanging. I eventually ended up fixing (almost) all of the issues
> by switching to implementations that are loom friendly (ReentrantLock in
> favour of Synchronized block). But hanging still happens for example when
> serving static resources (File I/O) with spring boot. I ended up writing
> custom code to fix the problem.
>
> Can you say a bit more about this?  I can't think how file I/O would lead
> to deadlock but you've listed several frameworks and libraries so there is
> lot going on. Class initializers can be problematic and may surface more
> once the issues with monitors pinning goes away.
>
>
>
> So as soon as your build came out, I gave it a try. It helped immediately
> with serving static resources when I tested with one user requesting
> multiple static resources and threads were not hanging anymore while on 21
> they would have hanged even with one user.
>
> So I started to perform local load test on the application and that’s when
> things go wrong. For load test I even turned off serving static resources.
> Hanging happens every single time I hit around 300 concurrent users, while
> on my Java 21 setup I could go to 2000-2500 concurrent users before hitting
> DB thread pool size. So there is definitely a regression compared to Java
> 21. I also ran the test on Java 23-ea+8 and it performed similar to 21. So
> new ForkjoinPool improvements (assuming they exist in 23-ea+8) can’t be the
> problem.
>
> When hanging happens , all Carrier threads are on WAITING state with a
> stack trace like this:
>
> Name: ForkJoinPool-1-worker-1
> State: WAITING on java.lang.VirtualThread at 17d1aaa2 owned by:
> tomcat-virt-70
> Total blocked: 4.978  Total waited: 2.387
>
>
> There are 12 ForkJoinPool-1-worker-<n> threads in the thread dump. I
> assume this is a 12 core system. All 12 include the text "Carrying virtual
> thread #<m>" where #<m> is the thread ID of the virtual thread is mounted
> on that carrier at the time.
>
> Would it be possible to run `jcmd <pid> Thread.dump_to_file <file>` to
> capture a thread dump that includes the virtual thread. This isn't the same
> thing as the HotSpot VM thread dump that jvisualvm is printing. It's has
> less details but it will at least let you see what the virtual threads are
> doing.
>
>
> :
>
> I am also listening to jdk.VirtualThreadPinnedEvent and they never
> happened. I tested with both -XX:LockingMode=1 and -XX:LockingMode=2 and
> hanging happened in both cases and I didn’t notice a significant
> difference there.
>
> Until recently, a jdk.VirtualThreadPinnedEvent was only recorded when the
> thread continues after pinning a carrier for more than 20ms. So maybe
> something is pinned indefinitely so there is no event recorded. This has
> since changed, but not in these EA builds, so this event is always recorded.
>
> -Alan
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20240216/c9b6e80b/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jcmd.tdump
Type: application/octet-stream
Size: 292870 bytes
Desc: not available
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20240216/c9b6e80b/jcmd-0001.tdump>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jvisualvm.tdump
Type: application/octet-stream
Size: 65541 bytes
Desc: not available
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20240216/c9b6e80b/jvisualvm-0001.tdump>