RFR: 8325587: Shenandoah: ShenandoahLock should allow blocking in VM [v2]

[Aleksey Shipilev]

> `ShenandoahLock` is a spinlock that is supposed to guard heap state. That lock is normally only lightly contended, as threads normally only allocate large TLABs/GCLABs. So we just summarily delegated to `Thread::{SpinAcquire,SpinRelease}`, which spins a bit, and then starts going to sleep/yield dance on contention.
> 
> This does not work well when there are lots of threads near the OOM conditions. Then, most of these threads would fail to allocate the TLAB, go for out-of-TLAB alloc, start lock acquisition, and spend _a lot_ of time trying to acquire the lock. The handshake/safepoint would think those threads are running, even when they are actually yielding or sleeping. As seen in bug report, a handshake operation over many such threads could then take hundreds of seconds.
> 
> The solution is to notify VM that we are blocking before going for `sleep` or `yield`. This is similar to what other VM code does near such yields. This involves state transitions, so it is only cheap to do near the actual blocking. Protecting the whole lock with VM transition would be very slow.
> 
> I also de-uglified bits of adjacent code.
> 
> Additional testing:
>  - [x] Original Extremem reproducer does not have outliers anymore
>  - [x] Linux x86_64 server fastdebug, `hotspot_gc_shenandoah`
>  - [x] Linux x86_64 server fastdebug, `all` passes with `-XX:+UseShenandoahGC` (some old failures, but quite a few "timeouts" also disappear)

Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:

  Review comments

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/17813/files
  - new: https://git.openjdk.org/jdk/pull/17813/files/3711cf13..81af0e16

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=17813&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=17813&range=00-01

  Stats: 45 lines in 2 files changed: 8 ins; 17 del; 20 mod
  Patch: https://git.openjdk.org/jdk/pull/17813.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/17813/head:pull/17813

PR: https://git.openjdk.org/jdk/pull/17813