RFR: 8373366: HandshakeState should disallow suspend ops for disabler threads [v13]

[Serguei Spitsyn]

On Fri, 16 Jan 2026 08:00:01 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

>> An asynchronous handshake operation (`ThreadSelfSuspensionHandshakeClosure`) can be installed when the target thread is not in a `MountUnmountDisabler` scope. But the target thread can enter such scope by the time the operation is self-processed by the target thread.
>> 
>> This is fixed by a small tweak in the function
>> `HandshakeOperation* HandshakeState::get_op_for_self(bool allow_suspend, bool check_async_exception)`.
>> The tweak is to skip a `HandshakeOperation` if `_handshakee->is_vthread_transition_disabler() == true`, so the same temporary suspension disabling mechanism would be used as for `_handshakee->is_disable_suspend() == true`.
>> 
>> All other changes are to move the `is_vthread_transition_disabler()` out of DEBUG to product.
>> 
>> Testing:
>>  - In progress: mach5 tiers 1-6
>
> Serguei Spitsyn has updated the pull request incrementally with one additional commit since the last revision:
> 
>   8375362: deadlock with unmount of suspended virtual thread interrupting another virtual thread

I've added the following fix for [JDK-8375362](https://bugs.openjdk.org/browse/JDK-8375362) as we agreed with Patricio:


@@ -129,7 +129,8 @@ bool MountUnmountDisabler::is_start_transition_disabled(JavaThread* thread, oop
  int base_disable_count = notify_jvmti_events() ? 1 : 0;
  return java_lang_Thread::vthread_transition_disable_count(vthread) > 0
         || global_vthread_transition_disable_count() > base_disable_count
         JVMTI_ONLY(|| (JvmtiVTSuspender::is_vthread_suspended(java_lang_Thread::thread_id(vthread)) || thread->is_suspended()));
         JVMTI_ONLY(|| (!thread->is_vthread_transition_disabler() &&
                        (JvmtiVTSuspender::is_vthread_suspended(java_lang_Thread::thread_id(vthread)) || thread->is_suspended())));
}

void MountUnmountDisabler::start_transition(JavaThread* current, oop vthread, bool is_mount, bool is_thread_end) {

-------------

PR Comment: https://git.openjdk.org/jdk/pull/28740#issuecomment-3762080269