RFR: 8351464: Shenandoah: Hang on ShenandoahController::handle_alloc_failure when run test TestAllocHumongousFragment#generational

[Xiaolong Peng]

On Tue, 11 Mar 2025 19:31:47 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Failed allocations may race to cancel the GC with the collector who is working to clear the cancelled GC. When the GC wins this race, it will fail to notify threads that are waiting for the failed GC cycle to complete. This change directly tracks the number of threads waiting due to an allocation failure, rather than indirectly tracking them through the cancelled gc state.
> 
> # Testing
> Ran TestAllocHumongousFragment#generational 6,500 times without failures.

The bug should also exist in classical Shenandoah w/o generation, I think ShenandoahControlThread also need to be updated to fix the bug, even it seems not happening in ShenandoahControlThread in the jtreg test.

src/hotspot/share/gc/shenandoah/shenandoahGenerationalControlThread.cpp line 281:

> 279: 
> 280:   {
> 281:     MonitorLocker ml(&_alloc_failure_waiters_lock);

Should the notification code be encapsulated in method `notify_alloc_failure_waiters()`?

-------------

PR Review: https://git.openjdk.org/jdk/pull/23997#pullrequestreview-2676193059
PR Review Comment: https://git.openjdk.org/jdk/pull/23997#discussion_r1990174134