[jdk19] RFR: 8289709: fatal error: stuck in JvmtiVTMSTransitionDisabler::disable_VTMS_transitions [v4]

[Serguei Spitsyn]

On Sun, 10 Jul 2022 10:12:54 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

>> This is a test bug. The test should filter out non-tested threads to avoid generating such kind of deadlocks. 
>> 
>> In short, the deadlock dependencies are:
>> 
>> - The `Common-Cleaner` thread is executing the JVM TI agent `MethodEntry` event callback which grabbed the `agent_lock` raw monitor and calls JVM TI `GetFrameCount`. The `GetFrameCount` is blocked in disabling VTMS transitions because the `ForkJoinPool-1-worker-2` is at a mount (VTMS) transition.
>> - The `ForkJoinPool-1-worker-2` is at mount (VTMS) transition and blocked in `java.lang.ref.NativeReferenceQueue.poll()` when acquiring the `NativeReferenceQueue` lock which is held by the `Reference Handler` thread.
>> - The `Reference Handler` thread grabbed the `NativeReferenceQueue` lock and is entering the `signal()` method. It triggered a JVM TI `MethodEntry` event. The JVM TI agent `MethodEntry` event callback is blocked on grabbing the `agent_lock` raw monitor which is held by the `Common-Cleaner` thread.
>> 
>> Also, the `timeout=360 `is explicitly set to avoid frequent timeouts in locals runs. 
>> 
>> Testing: submitted mach5 job with 100 runs on 3 debug platforms.
>
> Serguei Spitsyn has updated the pull request incrementally with one additional commit since the last revision:
> 
>   remove trailing white space

Alan and Alex, thank you for reviews!

-------------

PR: https://git.openjdk.org/jdk19/pull/129