RFR: 8355789: GenShen: assert(_degen_point == ShenandoahGC::_degenerated_unset) failed: Should not be set yet: Outside of Cycle

[Y. Srinivas Ramakrishna]

On Mon, 28 Apr 2025 22:57:09 GMT, William Kemper <wkemper at openjdk.org> wrote:

> When old generation marking is cancelled to run a young collection. we still set a `_degen_point ` for reasons that became vestigial after [JDK-8349094](https://bugs.openjdk.org/browse/JDK-8349094). When old marking is cancelled, the `_degen_point` should only be set if the marking was cancelled because of an allocation failure (and it should still only be set to "outside of cycle"). The following sequence could lead to this assertion failure:
> 1. Control thread is marking old
> 2. Young GC preempts it
> 3. Control thread sets the degen point because the old GC was "cancelled"
> 4. The concurrent young GC fails and attempts to set a degenerated point
> 5. This trips the assert because we already (incorrectly) set the degen point in `3`.

This looks fine, but it also leaves me with the feeling that we should sit down and review the interaction of the state machines for the young and old collections with the mutator allocations, and see if we can simplify the interaction protocol.

🚢

-------------

Marked as reviewed by ysr (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/24940#pullrequestreview-2805105686