RFR: 8373366: HandshakeState should disallow suspend ops for disabler threads [v13]

[Serguei Spitsyn]

On Tue, 20 Jan 2026 20:57:29 GMT, Patricio Chilano Mateo <pchilanomate at openjdk.org> wrote:

>> Serguei Spitsyn has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   8375362: deadlock with unmount of suspended virtual thread interrupting another virtual thread
>
> src/hotspot/share/runtime/javaThread.cpp line 1186:
> 
>> 1184: bool JavaThread::java_suspend(bool register_vthread_SR) {
>> 1185:   // Suspending a vthread transition disabler can cause deadlocks.
>> 1186:   // The HandshakeState::has_operation does not allow such suspends.
> 
> So for a thread trying to self-suspend we don't use handshakes. We identify this case and call `do_owner_suspend()` directly, which would now hit the new assert added there. I think self-suspend within a `MountUnmountDisabler` scope is actually possible if some event is posted during the `interrupt` Java upcall. But maybe we should fix this case in a separate issue? (it's preexistent to this change)

Good check, thanks.
The `JvmtiJavaUpcallMark` hides the JVMTI events during the `interrupt` upcall:

JvmtiEnv::InterruptThread(jthread thread) {
  . . .
  if (java_lang_VirtualThread::is_instance(thread_obj)) {
    // For virtual threads we have to call into Java to interrupt:
    Handle obj(current_thread, thread_obj);
    JvmtiJavaUpcallMark jjum(current_thread); // hide JVMTI events for Java upcall          <== !!!
    JavaValue result(T_VOID);
    JavaCalls::call_virtual(&result,
                            obj,
                            vmClasses::Thread_klass(),
                            vmSymbols::interrupt_method_name(),
                            vmSymbols::void_method_signature(),
                            current_thread);

> src/hotspot/share/runtime/suspendResumeManager.cpp line 136:
> 
>> 134:   while (is_suspended()) {
>> 135:     log_trace(thread, suspend)("JavaThread:" INTPTR_FORMAT " suspended", p2i(_target));
>> 136:     assert(!_target->is_vthread_transition_disabler(), "attempt to suspend a vthread transition disabler");
> 
> We could check this once before the loop since only the current thread changes `_is_vthread_transition_disabler`.

Good suggestion, thanks! Fixed now.

> test/hotspot/jtreg/ProblemList.txt line 2:
> 
>> 1: #
>> 2: # Copyright (c) 2016, 2026, Oracle and/or its affiliates. All rights reserved.
> 
> Not needed anymore.

Good catch, thanks. Fixed now.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28740#discussion_r2711037099
PR Review Comment: https://git.openjdk.org/jdk/pull/28740#discussion_r2711038426
PR Review Comment: https://git.openjdk.org/jdk/pull/28740#discussion_r2711039636