[jdk19] Integrated: 8289709: fatal error: stuck in JvmtiVTMSTransitionDisabler::disable_VTMS_transitions

[Serguei Spitsyn]

On Sat, 9 Jul 2022 06:43:12 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

> This is a test bug. The test should filter out non-tested threads to avoid generating such kind of deadlocks. 
> 
> In short, the deadlock dependencies are:
> 
> - The `Common-Cleaner` thread is executing the JVM TI agent `MethodEntry` event callback which grabbed the `agent_lock` raw monitor and calls JVM TI `GetFrameCount`. The `GetFrameCount` is blocked in disabling VTMS transitions because the `ForkJoinPool-1-worker-2` is at a mount (VTMS) transition.
> - The `ForkJoinPool-1-worker-2` is at mount (VTMS) transition and blocked in `java.lang.ref.NativeReferenceQueue.poll()` when acquiring the `NativeReferenceQueue` lock which is held by the `Reference Handler` thread.
> - The `Reference Handler` thread grabbed the `NativeReferenceQueue` lock and is entering the `signal()` method. It triggered a JVM TI `MethodEntry` event. The JVM TI agent `MethodEntry` event callback is blocked on grabbing the `agent_lock` raw monitor which is held by the `Common-Cleaner` thread.
> 
> Also, the `timeout=360 `is explicitly set to avoid frequent timeouts in locals runs. 
> 
> Testing: submitted mach5 job with 100 runs on 3 debug platforms.

This pull request has now been integrated.

Changeset: c3806b93
Author:    Serguei Spitsyn <sspitsyn at openjdk.org>
URL:       https://git.openjdk.org/jdk19/commit/c3806b93c48f826e940eecd0ba29995d7f0c796b
Stats:     13 lines in 2 files changed: 8 ins; 0 del; 5 mod

8289709: fatal error: stuck in JvmtiVTMSTransitionDisabler::disable_VTMS_transitions

Reviewed-by: alanb, amenkov, lmesnik

-------------

PR: https://git.openjdk.org/jdk19/pull/129