RFR: 8373366: HandshakeState should disallow suspend ops for disabler threads

[Leonid Mesnik]

On Wed, 10 Dec 2025 10:18:11 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

> This update fixes the following problem. The function JavaThread::java_suspend has the following assert:
> 
>  bool JavaThread::java_suspend(bool register_vthread_SR) {
>    // Suspending a vthread transition disabler can cause deadlocks.
>   assert(!is_vthread_transition_disabler(), "no suspend allowed for vthread transition disablers");
>   . . .
> 
> This is an over-hard requirement/invariant for threads that execute JVMTI functions (e.g. in an agent event handler). Such threads need a chance to be eventually suspended after JVMTI function completes its work and its `mountUnmountDisabler` is destructed.
> 
> A more adequate approach is to tweak the function
> `HandshakeOperation* HandshakeState::get_op_for_self(bool allow_suspend, bool check_async_exception)`
> so it could skip a `HandshakeOperation` if `_handshakee->is_vthread_transition_disabler() == true`, so the same temporary suspension disabling mechanism would be used as for `_handshakee->is_disable_suspend() == true`.
> 
> Testing:
>  - In progress: mach5 tiers 1-6

The fix looks good. However, I think the it makes a sense to add regression test that tries to suspend thread while it processing event callback.

Can you please show where mounting/unmounting is disabled for jvmti events.  i was unable to fines this place.

-------------

PR Review: https://git.openjdk.org/jdk/pull/28740#pullrequestreview-3566121535