RFR: 8281615: Deadlock caused by jdwp agent

[Chris Plummer]

On Mon, 14 Feb 2022 14:27:45 GMT, Zhengyu Gu <zgu at openjdk.org> wrote:

> There are scenarios that JDWP agent can deadlock on `classTrackLock` monitor. Following is the scenario in bug report.
> 
> **Java Thread** 
> -   loads a class and post `JVMTI_EVENT_CLASS_PREPARE` event
> - JDWP event callback handler calls `classTrack_processUnloads()` to handle the event.
> - `classTrack_processUnloads()` takes `classTrackLock` lock, then tries to allocate a new bag under the lock.
> - bag allocation code calls` jvmtiAllocate()`, which may be blocked by ongoing safepoint due to state transition.
> 
> If the safepoint is GC safepoint (prior to JDK16) or `VM_JvmtiPostObjectFree`  safepoint (JDK16 or later)
> 
> **VM Thread**
> - post `JVMTI_EVENT_OBJECT_FREE`
> - JDWP event callback handler calls `cbTrackingObjectFree()` to handle the event
> - `cbTrackingObjectFree()` tries to acquire `classTrackLock` lock, leads to deadlock
> 
> From my research, there are three events that may be posted at safepoints, `JVMTI_EVENT_GARBAGE_COLLECTION_START`, `JVMTI_EVENT_GARBAGE_COLLECTION_FINISH` and  `JVMTI_EVENT_OBJECT_FREE`, but only  `JVMTI_EVENT_OBJECT_FREE` is relevant to JDWP agent.
> 
> The solution I purpose here, is simply move allocation/deallocation code outside of `classTrackLock` lock.
> 
> 
> Test:
> - [x] tier1 
> - [x] vmTestbase_nsk_jdi
> - [x] vmTestbase_nsk_jdwp
> - [x] vmTestbase_nsk_jvmti

src/jdk.jdwp.agent/share/native/libjdwp/classTrack.c line 65:

> 63:  * handler may acquire the same monitor(s), e.g. classTrackLock in cbTrackingObjectFree(),
> 64:  * which can lead to deadlock.
> 65:  */

The debug agent does a lot alloc/dealloc calls while processing JVMTI events and holding locks. So the question is why this is not problematic other than this classTrackLock issue. I believe the answer is that cbTrackingObjectFree() is unique in that it is handled outside of the debug agent's normal event handling code, which is serialized. cbTrackingObjectFree() is called so the debug agent can maintain a list of loaded classes, and is not an event that gets passed on to the debugger like most JVMTI events are. So we have a case here where classTrackLock can be grabbed by both "typical" JVMTI event handling code via the classTrack_processUnloads() call, and then this special cbTrackingObjectFree() event handling.

I think having this comment here doesn't help the reader of the code below unless they somehow read the comment first and then recognized it's application in the code below. At the very least the code below should tersely state while the lock is being released, and then refer to this comment for details.

-------------

PR: https://git.openjdk.java.net/jdk/pull/7461