RFR: 8352116: Deadlock with GCLocker and JVMTI after JDK-8192647

[Erik Österlund]

On Thu, 3 Apr 2025 09:40:19 GMT, Albert Mingkun Yang <ayang at openjdk.org> wrote:

> Using a new lock (`JNICritical_lock`) in `GCLocker::block` to resolve a deadlock issue. The root cause of the deadlock is that holding `Heap_lock` while waiting in `GCLocker::block` is unsafe.
> 
> The new lock is held from the start of `GCLocker::block` to the end of `GCLocker::unblock`. This requires adjusting `Heap_lock`'s rank to allow acquiring `Heap_lock` while holding `JNICritical_lock`. The most important changes are in `gcVMOperations.cpp` and `mutexLocker.cpp`.
> 
> Test: tier1-8; verified failure can be observed 2/2000 and pass 8000 iterations.

Looks good. Would be nice to refactor the if (UseSerialGC || UseParallelGC) code to something that explains why it's there (those are the GCs that use the new improved GC locker). But that's pre existing so I don't mind if it's split to a separate RFE.

-------------

Marked as reviewed by eosterlund (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/24407#pullrequestreview-2739864515