RFR: 8334482: Shenandoah: Deadlock when safepoint is pending during nmethods iteration

[Roman Kennke]

On Wed, 24 Jul 2024 09:10:35 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> There is a problem in interaction between STS taken in `ShenandoahConcurrentWeakRootsEvacUpdateTask::work` and nmethod list iteration. If we start the nmethod iteration in `ShCWREUT` constructor, proceed to STS and block, any waiter that waits for the end of iteration would block until safepoint is over. If that waiter waits without a safepoint check, like anyone waiting on `CodeCache_lock`, e.g. Sweeper (not really in mainline, but we seen a case in JDK 17), or a JIT compiler patching code, then we would have a deadlock.
> 
> A solution is to initialize iteration only immediately before the use of iterator. This will pass any pending STSes that we have seen could deadlock. As long as nothing in nmethod handling safepoints, the deadlock is resolved. I think `NoSafepointVerifier` would give us a warning if something safepoints while iteration is on.
> 
> Unfortunately, it is really hard to reproduce in mainline, since the absence of Sweeper makes this deadlock exceedingly rare.
> 
> Additional testing:
>  - [x] Reproducer on JDK 17u with this patch applied now does not deadlock
>  - [x] Linux AArch64 server fastdebug, `all` with `-XX:+UseShenandoahGC`
>  - [x] Linux x86_64 server fastdebug, `all` with `-XX:+UseShenandoahGC`

Looks good to me! Thanks!

-------------

Marked as reviewed by rkennke (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/20309#pullrequestreview-2201587027