RFR: 8373366: HandshakeState should disallow suspend ops for disabler threads [v11]

[David Holmes]

On Thu, 15 Jan 2026 01:34:28 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

>> An asynchronous handshake operation (`ThreadSelfSuspensionHandshakeClosure`) can be installed when the target thread is not in a `MountUnmountDisabler` scope. But the target thread can enter such scope by the time the operation is self-processed by the target thread.
>> 
>> This is fixed by a small tweak in the function
>> `HandshakeOperation* HandshakeState::get_op_for_self(bool allow_suspend, bool check_async_exception)`.
>> The tweak is to skip a `HandshakeOperation` if `_handshakee->is_vthread_transition_disabler() == true`, so the same temporary suspension disabling mechanism would be used as for `_handshakee->is_disable_suspend() == true`.
>> 
>> All other changes are to move the `is_vthread_transition_disabler()` out of DEBUG to product.
>> 
>> Testing:
>>  - In progress: mach5 tiers 1-6
>
> Serguei Spitsyn has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix trailing space in new test

test/hotspot/jtreg/serviceability/jvmti/vthread/ThreadStateTest2/ThreadStateTest2.java line 48:

> 46:  *  - VT-2: a virtual thread: in a loop calls JVMTI InterruptThread(VT-0) and GetThreadState(VT-1)
> 47:  *  - main: a platform thread: in a loop invokes native method testSuspendResume which suspends and resumes VT-2
> 48:  * The JVMTI functionis above install a MountUnmountDisabler for target virtual thread (VT-0 or VT-1).

Suggestion:

 * The JVMTI functions above install a MountUnmountDisabler for target virtual thread (VT-0 or VT-1).

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28740#discussion_r2693081790