RFR: 8352116: Deadlock with GCLocker and JVMTI after JDK-8192647

[Thomas Schatzl]

On Thu, 3 Apr 2025 09:40:19 GMT, Albert Mingkun Yang <ayang at openjdk.org> wrote:

> Using a new lock (`JNICritical_lock`) in `GCLocker::block` to resolve a deadlock issue. The root cause of the deadlock is that holding `Heap_lock` while waiting in `GCLocker::block` is unsafe.
> 
> The new lock is held from the start of `GCLocker::block` to the end of `GCLocker::unblock`. This requires adjusting `Heap_lock`'s rank to allow acquiring `Heap_lock` while holding `JNICritical_lock`. The most important changes are in `gcVMOperations.cpp` and `mutexLocker.cpp`.
> 
> Test: tier1-8; verified failure can be observed 2/2000 and pass 8000 iterations.

Marked as reviewed by tschatzl (Reviewer).

-------------

PR Review: https://git.openjdk.org/jdk/pull/24407#pullrequestreview-2745840662