RFR: 8373366: HandshakeState should disallow suspend ops for disabler threads [v2]

[Serguei Spitsyn]

On Fri, 12 Dec 2025 17:02:36 GMT, Patricio Chilano Mateo <pchilanomate at openjdk.org> wrote:

>> Not sure, I correctly understand your question. Are you asking about possible suspend points inside a `MountUnmountDisabler` scope? It is used for implementation of the JVMTI functions which have a deal with threads. They normally have suspend points (points where an asynchronous suspend operation can be picked and executed by the target thread itself with `HandshakeState::process_by_self()`), for instance, because they use handshakes.
>> As I understand, the `ThreadSelfSuspensionHandshakeClosure` is an `AsyncHandshakeClosure`. As the suspending operation is asynchronous, the suspending thread can finish its suspending work and get out of the exclusive operation scope. Then the target thread can enter its own `MountUnmountDisabler` scope and get suspended there. The tweak above is to disallow such a scenario.
>
> Right, I was just curious whether you ran into this issue or only found it by code inspection. But I'm guessing you found disablers processing `ThreadSelfSuspensionHandshakeClosure` handshakes because of the other issue in `enable_transition_for_all()`, and that accidentally led you to consider this other potential case where a thread is suspended before entering a `MountUnmountDisabler` scope but processes the async handshake inside it? Thanks for updating the PR description.

This is right, thanks!

>> You are asking right question, thanks!
>> I've found and fixed the issue caused this and also restored the assert.
>> I will also need to update the PR description to reflect this change.
>
> Thanks! Now it makes sense why we were hitting this assert. Since that was introduced in 8364343 I think we should separate that fix and backport it to 26.

Thanks. Okay, will do it.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28740#discussion_r2615590699
PR Review Comment: https://git.openjdk.org/jdk/pull/28740#discussion_r2615586743