RFR: 8293782: Shenandoah: some tests failed on lock rank check

[Aleksey Shipilev]

On Wed, 14 Sep 2022 07:01:52 GMT, Tongbao Zhang <duke at openjdk.org> wrote:

> After [JDK-8290025](https://bugs.openjdk.org/browse/JDK-8290025), some tests using ShenandoahGC failed on the lock rank check between AdapterHandlerLibrary_lock and ShenandoahRequestedGC_lock
> 
> Symptom
> 
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # Internal Error (/data1/ws/jdk/src/hotspot/share/runtime/mutex.cpp:454), pid=2018566, tid=2022220
> # assert(false) failed: Attempting to acquire lock ShenandoahRequestedGC_lock/safepoint-1 out of order with lock AdapterHandlerLibrary_lock/safepoint-1 -- possible deadlock
> #
> # JRE version: OpenJDK Runtime Environment (20.0) (slowdebug build 20-internal-adhoc.root.jdk)
> # Java VM: OpenJDK 64-Bit Server VM (slowdebug 20-internal-adhoc.root.jdk, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, shenandoah gc, linux-amd64)
> # Problematic frame:
> # V [libjvm.so+0x106fd6a] Mutex::check_rank(Thread*)+0x426

So we can now enter this code when holding `AdapterHandlerLibrary_lock`, which has a rank of `safepoint-1`. These locks should probably match the rank of `Heap_lock`, which is `safepoint-2` now. Please update `_alloc_failure_waiters_lock` rank as well.

-------------

Changes requested by shade (Reviewer).

PR: https://git.openjdk.org/jdk/pull/10264