RFR: 8352299: GenShen: Young cycles that interrupt old cycles cannot be cancelled [v3]

[William Kemper]

> The sequence of events that creates this state:
> 1. An old collection is trying to finish marking by flushing SATB buffers with a Handshake
> 2. The regulator thread cancels old marking to start a young collection
> 3. A mutator thread shortly follows and attempts to cancel the nascent young collection
> 4. Step `3` fails (because of this bug) and cancellation reason does _not_ become `allocation failure`
> 5. The mutator thread enters a tight loop in which it retries allocations without `waiting`
> 6. The mutator thread remains in the `thread_in_vm` state and prevents the VM thread from completing step `1`.

William Kemper has updated the pull request incrementally with one additional commit since the last revision:

  Widen type of shared enum value to unlock platform support of atomic xchg

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/24105/files
  - new: https://git.openjdk.org/jdk/pull/24105/files/adcb999b..ca45ff02

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=24105&range=02
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=24105&range=01-02

  Stats: 9 lines in 1 file changed: 1 ins; 0 del; 8 mod
  Patch: https://git.openjdk.org/jdk/pull/24105.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/24105/head:pull/24105

PR: https://git.openjdk.org/jdk/pull/24105