Virtual Threads - deadlock with synchronized (Main.class) and System.out.println

[Ron Pressler]

The best fix is to get synchronized to not pin; we’re working hard on that.

The problem can occur whenever a shared j.u.c lock is acquired both inside and outside a synchronized (and println acquires a shared lock for the output stream). For now, this can be detected with the JFR pinning event, and the fix is to replace the synchronized that guards the lock acquisition with a j.u.c lock.

This can also happen with pinning due to a native methods on the stack, which, in practice, primarily means class loading (class loading uses a native VM method that calls back into Java). For the past couple of months we’ve been testing a specialized fix for that that uses compensation (just note that it isn’t the same as compensating due to a blocking native method — what we recently discussed here — only a pinning native method, i.e. a native method that upcalls into Java, and class loading in particular).

— Ron

> On 17 Jul 2023, at 23:37, Brian S O'Neill <bronee at gmail.com> wrote:
> 
> What's the fix? Is this just a problem with println or can any code be affected?
> 
> On 2023-07-17 02:47 PM, Ron Pressler wrote:
>> Hi.
>> The problem occurs when some shared j.u.c lock is acquired both when some virtual thread is pinned (i.e. inside a synchronized block/method) as well as unpinned. In this case, println acquires a lock, sometimes inside synchronized and sometimes outside it. What happens is that an unpinned acquire succeeds and the thread is then blocks for some reason, and then there are multiple pinned attempts to acquire the same lock. All those attempts will fail because the owner hasn’t released the lock, but the owner now can’t run and release it because the scheduler has no remaining worker threads.
>> — Ron
>>> On 17 Jul 2023, at 20:08, Denis <hixon10 at yandex.ru> wrote:
>>> 
>>> Hello everyone,
>>> 
>>> A day ago sidknot17 reddit user created a post on reddit https://www.reddit.com/r/java/comments/1512xuo/virtual_threads_interesting_deadlock/ I still cannot understand, whether it is bug, or not. Maybe someone can help me with it?
>>>  So, we have the following code (it is copied from reddit):
>>> for (int i = 0; i < NUMBER_OF_CORES + 1; i++) {
>>>    Thread.startVirtualThread(() -> {
>>>     System.out.println(Thread.currentThread() + ": Before synchronized block");
>>>     synchronized (Main.class) {
>>>    System.out.println(Thread.currentThread() + ": Inside synchronized block");
>>>     }
>>>    });
>>> }
>>>  I can reproduce deadlock in every 3-d run, using Macbook pro with M1 pro. For me, NUMBER_OF_CORES is 10.
>>>  Via debugger, I see the following state. We have 10 platform threads in defaultForkJoinPool (because I have 10 CPU). Also, this test creates 11 virtual threads.
>>>  Virtual threads have the following internal state field value:
>>>     • 9 virtual threads have (RUNNING  = 3; runnable-mounted), and their stack trace is finished on synchronized (Main.class).
>>>     • 1 virtual thread has (PINNED   = 6; mounted), and its stack trace is on line with the second System.out.println.
>>>     • 1 virtual thread has (RUNNABLE = 2; runnable-unmounted), and its stack trace in on line with the first System.out.println.
>>> Platform threads have the following state:
>>>     • 9 platform threads have BLOCKED, and they stack trace is finished on https://github.com/openjdk/jdk/blob/e73796879299c6170b63edb998439db4764ceae0/src/java.base/share/classes/jdk/internal/vm/Continuation.java#L251 - if (isStarted()) { enterSpecial(this, true
>>>     • 1 platform thread has WAITING state, and its stack trace is finished on https://github.com/openjdk/jdk/blob/e73796879299c6170b63edb998439db4764ceae0/src/java.base/share/classes/jdk/internal/vm/Continuation.java#L248 - if (!isStarted()) { enterSpecial(this, false
>>>     • I ran the program with -Djdk.tracePinnedThreads=full, and only one thread was pinned.
>>> 
>>> So, I don't understand, why do we have deadlock. As we see, many threads are not pinned, so I would expect, that we shouldn't have any deadlocks here.
>>>