RFR: 8346727: JvmtiVTMSTransitionDisabler deadlock [v3]

[Chris Plummer]

On Fri, 10 Jan 2025 02:27:16 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

>> This is a fix of one more deadlock issue related to `interruptLock` critical sections. When the `interruptLock` is hold by the target virtual thread it is unsafe to suspend or post JVMTI events. This update is to ignore the JVMTI events when the `interruptLock` is hold. It is additionally to the cases when the target JavaThread is in a `VTMS` transition. It is based on the existing mechanism disallowing JVMTI suspends while the `interruptLock` is hold. In order to support this the target JavaThread has the `_is_disable_suspend` bit.
>> 
>> Testing:
>> - Ran mach5 tiers 1-6
>> - There is no regression test for this issue as it is not hard to construct one. Originally, the issue was reported by JGroups which is using the `Async` profiler.
>
> Serguei Spitsyn has updated the pull request incrementally with one additional commit since the last revision:
> 
>   review: tweak some assert messages

src/hotspot/share/runtime/javaThread.hpp line 724:

> 722:   void set_VTMS_transition_mark(bool val)        { Atomic::store(&_VTMS_transition_mark, val); }
> 723: 
> 724:   // Temporarily skip posting JVMTI events for safety reasons when executions is in ia critical section:

Suggestion:

  // Temporarily skip posting JVMTI events for safety reasons when execution is in a critical section:

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/22997#discussion_r1910796441