Integrated: 8311218: fatal error: stuck in JvmtiVTMSTransitionDisabler::VTMS_transition_disable

[Serguei Spitsyn]

On Thu, 7 Dec 2023 06:28:43 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

> This fix is for JDK 23 but the intention is to back port it to 22 in RDP-1 time frame.
> It is fixing a deadlock issue between `VirtualThread` class critical sections with the `interruptLock` (in methods: `unpark()`, `interrupt()`, `getAndClearInterrupt()`, `threadState()`, `toString()`), `JvmtiVTMSTransitionDisabler` and JVMTI `Suspend/Resume` mechanisms.
> The deadlocking scenario is well described by Patricio in a bug report comment.
> In simple words, a virtual thread should not be suspended during 'interruptLock' critical sections.
> 
> The fix is to record that a virtual thread is in a critical section (`JavaThread`'s `_in_critical_section` bit) by notifying the VM/JVMTI about begin/end of critical section.
> This bit is used in `HandshakeState::get_op_for_self()` to filter out any `HandshakeOperation` if a target `JavaThread` is in a critical section.
> 
> Some of new notifications with `notifyJvmtiSync()` method is on a performance critical path. It is why this method has been intrincified.
> 
> New test was developed by Patricio:
>  `test/hotspot/jtreg/serviceability/jvmti/vthread/SuspendWithInterruptLock`
> The test is very nice as it reliably in 100% reproduces the deadlock without the fix.
> The test is never failing with this fix.
> 
> Testing:
>  - tested with newly added test: `test/hotspot/jtreg/serviceability/jvmti/vthread/SuspendWithInterruptLock`
>  - tested with mach5 tiers 1-6

This pull request has now been integrated.

Changeset: 0f8e4e0a
Author:    Serguei Spitsyn <sspitsyn at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/0f8e4e0a81257c678e948c341a241dc0b810494f
Stats:     229 lines in 15 files changed: 196 ins; 0 del; 33 mod

8311218: fatal error: stuck in JvmtiVTMSTransitionDisabler::VTMS_transition_disable

Reviewed-by: lmesnik, alanb

-------------

PR: https://git.openjdk.org/jdk/pull/17011