RFR: 8352178: Add precondition in VMThread::execute to prevent deadlock [v2]

[Patricio Chilano Mateo]

On Thu, 20 Mar 2025 06:21:30 GMT, David Holmes <dholmes at openjdk.org> wrote:

>> Moved to `VMThread::execute` and edited the title.
>
> Aren't Handshakes a superset of VMoperations? `VMThread::execute` will only affect VM_Operations, not direct Handshakes with other threads. Is the problem limited to a STS thread trying to execute a safepoint VM operation?

There are two types of handshakes, the ones that target a single JavaThread and the ones that are executed on all JavaThreads. The latter one is a VM operation (VM_HandshakeAllThreads), i.e it’s executed by the VMThread. The original fix guarded against the deadlock scenario where the STS thread requested a VM_HandshakeAllThreads operation, but the same issue can happen by requesting any VM operation. As for direct handshakes, the original fix didn’t touch that code path, and I don’t think we need to because the handshaker can only be a JavaThread already, and AFAIK all these STS threads are not.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/24088#discussion_r2005825890