RFR: 8281615: Deadlock caused by jdwp agent [v5]

[Zhengyu Gu]

> There are scenarios that JDWP agent can deadlock on `classTrackLock` monitor. Following is the scenario in bug report.
> 
> **Java Thread** 
> -   loads a class and post `JVMTI_EVENT_CLASS_PREPARE` event
> - JDWP event callback handler calls `classTrack_processUnloads()` to handle the event.
> - `classTrack_processUnloads()` takes `classTrackLock` lock, then tries to allocate a new bag under the lock.
> - bag allocation code calls` jvmtiAllocate()`, which may be blocked by ongoing safepoint due to state transition.
> 
> If the safepoint is GC safepoint (prior to JDK16) or `VM_JvmtiPostObjectFree`  safepoint (JDK16 or later)
> 
> **VM Thread**
> - post `JVMTI_EVENT_OBJECT_FREE`
> - JDWP event callback handler calls `cbTrackingObjectFree()` to handle the event
> - `cbTrackingObjectFree()` tries to acquire `classTrackLock` lock, leads to deadlock
> 
> From my research, there are three events that may be posted at safepoints, `JVMTI_EVENT_GARBAGE_COLLECTION_START`, `JVMTI_EVENT_GARBAGE_COLLECTION_FINISH` and  `JVMTI_EVENT_OBJECT_FREE`, but only  `JVMTI_EVENT_OBJECT_FREE` is relevant to JDWP agent.
> 
> The solution I purpose here, is simply move allocation/deallocation code outside of `classTrackLock` lock.
> 
> 
> Test:
> - [x] tier1 
> - [x] vmTestbase_nsk_jdi
> - [x] vmTestbase_nsk_jdwp
> - [x] vmTestbase_nsk_jvmti

Zhengyu Gu has updated the pull request incrementally with one additional commit since the last revision:

  Missing space

-------------

Changes:
  - all: https://git.openjdk.java.net/jdk/pull/7461/files
  - new: https://git.openjdk.java.net/jdk/pull/7461/files/f93bf8d6..78198bb3

Webrevs:
 - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=7461&range=04
 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=7461&range=03-04

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.java.net/jdk/pull/7461.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/7461/head:pull/7461

PR: https://git.openjdk.java.net/jdk/pull/7461