[jdk19] RFR: 8289709: fatal error: stuck in JvmtiVTMSTransitionDisabler::disable_VTMS_transitions [v4]

[Serguei Spitsyn]

> This is a test bug. The test should filter out non-tested threads to avoid generating such kind of deadlocks. 
> 
> In short, the deadlock dependencies are:
> 
> - The `Common-Cleaner` thread is executing the JVM TI agent `MethodEntry` event callback which grabbed the `agent_lock` raw monitor and calls JVM TI `GetFrameCount`. The `GetFrameCount` is blocked in disabling VTMS transitions because the `ForkJoinPool-1-worker-2` is at a mount (VTMS) transition.
> - The `ForkJoinPool-1-worker-2` is at mount (VTMS) transition and blocked in `java.lang.ref.NativeReferenceQueue.poll()` when acquiring the `NativeReferenceQueue` lock which is held by the `Reference Handler` thread.
> - The `Reference Handler` thread grabbed the `NativeReferenceQueue` lock and is entering the `signal()` method. It triggered a JVM TI `MethodEntry` event. The JVM TI agent `MethodEntry` event callback is blocked on grabbing the `agent_lock` raw monitor which is held by the `Common-Cleaner` thread.
> 
> Also, the `timeout=360 `is explicitly set to avoid frequent timeouts in locals runs. 
> 
> Testing: submitted mach5 job with 100 runs on 3 debug platforms.

Serguei Spitsyn has updated the pull request incrementally with one additional commit since the last revision:

  remove trailing white space

-------------

Changes:
  - all: https://git.openjdk.org/jdk19/pull/129/files
  - new: https://git.openjdk.org/jdk19/pull/129/files/046d3525..9ae069c6

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk19&pr=129&range=03
 - incr: https://webrevs.openjdk.org/?repo=jdk19&pr=129&range=02-03

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jdk19/pull/129.diff
  Fetch: git fetch https://git.openjdk.org/jdk19 pull/129/head:pull/129

PR: https://git.openjdk.org/jdk19/pull/129