RFR: 8352299: GenShen: Young cycles that interrupt old cycles cannot be cancelled [v2]

[Kelvin Nilsen]

On Wed, 19 Mar 2025 18:33:50 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> The sequence of events that creates this state:
>> 1. An old collection is trying to finish marking by flushing SATB buffers with a Handshake
>> 2. The regulator thread cancels old marking to start a young collection
>> 3. A mutator thread shortly follows and attempts to cancel the nascent young collection
>> 4. Step `3` fails (because of this bug) and cancellation reason does _not_ become `allocation failure`
>> 5. The mutator thread enters a tight loop in which it retries allocations without `waiting`
>> 6. The mutator thread remains in the `thread_in_vm` state and prevents the VM thread from completing step `1`.
>
> William Kemper has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains three additional commits since the last revision:
> 
>  - Emulate single byte xchg with cmpxchg
>  - Merge remote-tracking branch 'jdk/master' into fix-uncancellable-young-gc
>  - Allow young cycles that interrupt old cycles to be cancelled

src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 2143:

> 2141: 
> 2142: bool ShenandoahHeap::try_cancel_gc(GCCause::Cause cause) {
> 2143:   const jbyte prev = _cancelled_gc.xchg(cause);

I guess maybe we want cause and prev to be integer type.   Then the template will expand into a type that is known to that Atomic::xchg operation.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/24105#discussion_r2010713586