RFR: 8373366: HandshakeState should disallow suspend ops for disabler threads [v15]

[Leonid Mesnik]

On Wed, 21 Jan 2026 20:11:53 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

>> An asynchronous handshake operation (`ThreadSelfSuspensionHandshakeClosure`) can be installed when the target thread is not in a `MountUnmountDisabler` scope. But the target thread can enter such scope by the time the operation is self-processed by the target thread.
>> 
>> This is fixed by a small tweak in the function
>> `HandshakeOperation* HandshakeState::get_op_for_self(bool allow_suspend, bool check_async_exception)`.
>> The tweak is to skip a `HandshakeOperation` if `_handshakee->is_vthread_transition_disabler() == true`, so the same temporary suspension disabling mechanism would be used as for `_handshakee->is_disable_suspend() == true`.
>> 
>> All other changes are to move the `is_vthread_transition_disabler()` out of DEBUG to product.
>> 
>> Testing:
>>  - In progress: mach5 tiers 1-6
>
> Serguei Spitsyn has updated the pull request incrementally with one additional commit since the last revision:
> 
>   review: restored the original assert with a different message

Looks good.
A couple of small nits,  feel free to ignore them.

src/hotspot/share/runtime/handshake.cpp line 525:

> 523: #if INCLUDE_JVMTI
> 524:   if (allow_suspend && (_handshakee->is_disable_suspend() || _handshakee->is_vthread_transition_disabler())) {
> 525:     // filter out suspend operations while JavaThread is in disable_suspend mode

does it makes sense to update comment to   
 // filter out suspend operations while JavaThread is in disable_suspend mode or in transition
or  something more generic
 // check if JavaThread can be suspended in this handshake

test/hotspot/jtreg/serviceability/jvmti/vthread/ThreadStateTest2/libThreadStateTest2.cpp line 63:

> 61: JNIEXPORT void JNICALL
> 62: Java_ThreadStateTest2_setMonitorContendedMode(JNIEnv* jni, jclass klass, jboolean enable) {
> 63:   jvmtiError err = jvmti->SetEventNotificationMode(enable ? JVMTI_ENABLE : JVMTI_DISABLE, JVMTI_EVENT_MONITOR_CONTENDED_ENTER, nullptr);

You can use
set_event_notification_mode(jvmtiEnv*, ... )
from jvmti_common.h

-------------

Marked as reviewed by lmesnik (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/28740#pullrequestreview-3689496071
PR Review Comment: https://git.openjdk.org/jdk/pull/28740#discussion_r2714417725
PR Review Comment: https://git.openjdk.org/jdk/pull/28740#discussion_r2714441116