RFR: 8352178: Add precondition in VMThread::execute to prevent deadlock [v2]

[David Holmes]

On Thu, 20 Mar 2025 14:50:34 GMT, Patricio Chilano Mateo <pchilanomate at openjdk.org> wrote:

>> Aren't Handshakes a superset of VMoperations? `VMThread::execute` will only affect VM_Operations, not direct Handshakes with other threads. Is the problem limited to a STS thread trying to execute a safepoint VM operation?
>
> There are two types of handshakes, the ones that target a single JavaThread and the ones that are executed on all JavaThreads. The latter one is a VM operation (VM_HandshakeAllThreads), i.e it’s executed by the VMThread. The original fix guarded against the deadlock scenario where the STS thread requested a VM_HandshakeAllThreads operation, but the same issue can happen by requesting any VM operation. As for direct handshakes, the original fix didn’t touch that code path, and I don’t think we need to because the handshaker can only be a JavaThread already, and AFAIK all these STS threads are not.

Thanks Patricio. I had things inverted.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/24088#discussion_r2006904269