[jdk19] RFR: 8289709: fatal error: stuck in JvmtiVTMSTransitionDisabler::disable_VTMS_transitions [v4]

[Leonid Mesnik]

On Sun, 10 Jul 2022 10:12:54 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

>> This is a test bug. The test should filter out non-tested threads to avoid generating such kind of deadlocks. 
>> 
>> In short, the deadlock dependencies are:
>> 
>> - The `Common-Cleaner` thread is executing the JVM TI agent `MethodEntry` event callback which grabbed the `agent_lock` raw monitor and calls JVM TI `GetFrameCount`. The `GetFrameCount` is blocked in disabling VTMS transitions because the `ForkJoinPool-1-worker-2` is at a mount (VTMS) transition.
>> - The `ForkJoinPool-1-worker-2` is at mount (VTMS) transition and blocked in `java.lang.ref.NativeReferenceQueue.poll()` when acquiring the `NativeReferenceQueue` lock which is held by the `Reference Handler` thread.
>> - The `Reference Handler` thread grabbed the `NativeReferenceQueue` lock and is entering the `signal()` method. It triggered a JVM TI `MethodEntry` event. The JVM TI agent `MethodEntry` event callback is blocked on grabbing the `agent_lock` raw monitor which is held by the `Common-Cleaner` thread.
>> 
>> Also, the `timeout=360 `is explicitly set to avoid frequent timeouts in locals runs. 
>> 
>> Testing: submitted mach5 job with 100 runs on 3 debug platforms.
>
> Serguei Spitsyn has updated the pull request incrementally with one additional commit since the last revision:
> 
>   remove trailing white space

Looks good, assumint

test/hotspot/jtreg/serviceability/jvmti/events/FramePop/framepop02/libframepop02.cpp line 78:

> 76: 
> 77:   bool result = strncmp(inf.name, TEST_THREAD_NAME_BASE, strlen(TEST_THREAD_NAME_BASE)) == 0;
> 78:   jvmti->Deallocate((unsigned char *)inf.name);

Optional, we have 'deallocate' method in './jdk/test/lib/jvmti/jvmti_common.h' which check error status.

-------------

Marked as reviewed by lmesnik (Reviewer).

PR: https://git.openjdk.org/jdk19/pull/129