[jdk19] RFR: 8289709: fatal error: stuck in JvmtiVTMSTransitionDisabler::disable_VTMS_transitions [v2]

[Alan Bateman]

On Sat, 9 Jul 2022 07:34:48 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

>> This is a test bug. The test should filter out non-tested threads to avoid generating such kind of deadlocks. 
>> 
>> In short, the deadlock dependencies are:
>> 
>> - The `Common-Cleaner` thread is executing the JVM TI agent `MethodEntry` event callback which grabbed the `agent_lock` raw monitor and calls JVM TI `GetFrameCount`. The `GetFrameCount` is blocked in disabling VTMS transitions because the `ForkJoinPool-1-worker-2` is at a mount (VTMS) transition.
>> - The `ForkJoinPool-1-worker-2` is at mount (VTMS) transition and blocked in `java.lang.ref.NativeReferenceQueue.poll()` when acquiring the `NativeReferenceQueue` lock which is held by the `Reference Handler` thread.
>> - The `Reference Handler` thread grabbed the `NativeReferenceQueue` lock and is entering the `signal()` method. It triggered a JVM TI `MethodEntry` event. The JVM TI agent `MethodEntry` event callback is blocked on grabbing the `agent_lock` raw monitor which is held by the `Common-Cleaner` thread.
>> 
>> Also, the `timeout=360 `is explicitly set to avoid frequent timeouts in locals runs. 
>> 
>> Testing: submitted mach5 job with 100 runs on 3 debug platforms.
>
> Serguei Spitsyn has updated the pull request incrementally with one additional commit since the last revision:
> 
>   adjusted max number of threads instead of adding explicit timeout

test/hotspot/jtreg/serviceability/jvmti/events/FramePop/framepop02/libframepop02.cpp line 219:

> 217: 
> 218:   if (!isTestThread(jni, jvmti, thr)) {
> 219:     return; // not a tested thread

If I read the test correctly, NotifyFramePop will only be called with the test thread so therefore the FramePop callback should only be called by the test thread, is that correct? I'm asking because I'm wondering why FramePop also checks the thread is the test thread.

-------------

PR: https://git.openjdk.org/jdk19/pull/129