RFR: 8351464: Shenandoah: Hang on ShenandoahController::handle_alloc_failure when run test TestAllocHumongousFragment#generational

[William Kemper]

On Tue, 11 Mar 2025 19:31:47 GMT, William Kemper <wkemper at openjdk.org> wrote:

> Failed allocations may race to cancel the GC with the collector who is working to clear the cancelled GC. When the GC wins this race, it will fail to notify threads that are waiting for the failed GC cycle to complete. This change directly tracks the number of threads waiting due to an allocation failure, rather than indirectly tracking them through the cancelled gc state.
> 
> # Testing
> Ran TestAllocHumongousFragment#generational 6,500 times without failures.

Not sure I want to change `ShenandoahControlThread.` It uses a different mechanism to track whether or not to notify. It only notifies when it services the alloc failure request (it doesn't depend on the shared `cancelled_gc` state the same way the generational mode does). In the scenario that leads to this live lock for the generational mode, the default mode would _not_ notify the waiters upon successful completion of the concurrent cycle. It would notify them after the subsequent degenerated cycle.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/23997#issuecomment-2715851851