RFR: 8352299: GenShen: Young cycles that interrupt old cycles cannot be cancelled [v4]

[Aleksey Shipilev]

On Tue, 25 Mar 2025 17:51:41 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> The sequence of events that creates this state:
>> 1. An old collection is trying to finish marking by flushing SATB buffers with a Handshake
>> 2. The regulator thread cancels old marking to start a young collection
>> 3. A mutator thread shortly follows and attempts to cancel the nascent young collection
>> 4. Step `3` fails (because of this bug) and cancellation reason does _not_ become `allocation failure`
>> 5. The mutator thread enters a tight loop in which it retries allocations without `waiting`
>> 6. The mutator thread remains in the `thread_in_vm` state and prevents the VM thread from completing step `1`.
>
> William Kemper has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Stop casting GCCause to jbyte

I think we want to remove `addr_of` and related methods for `ShenandoahSharedEnumFlag` to avoid accidents. `ShenandoahSharedValue` was defined specifically to stick to `jbyte` for the sake of generated code. If we are not expected to have accesses to generated code to this flag, we should remove the APIs that allow it.

Going forward, I think we should consider redefining `ShenandoahSharedValue` to `uint32_t` to begin with. This would require fiddling with barrier sets that might read them.

-------------

PR Review: https://git.openjdk.org/jdk/pull/24105#pullrequestreview-2714735454