RFR: 8367646: [GenShen] Control thread may overwrite gc cancellation cause set by mutator

[Kelvin Nilsen]

On Mon, 6 Oct 2025 22:00:01 GMT, William Kemper <wkemper at openjdk.org> wrote:

> I believe the following events could lead to this assertion failure:
> 1. Control thread reads the heap's gc cancellation cause as `shenandoah_concurrent_gc` 
> 2. Mutator thread has an allocation failure and sets the heap's gc cancellation cause to `shenandoah_alloc_failure`
> 3. Control thread uses stale value from `1` and decides to unconditionally clear the cancellation cause
> 4. Mutator thread assert that gc is still cancelled
> 
> The proposed fix here has the control thread use a CAS operation to only clear the gc if the existing value if `shenandoah_concurrent_gc`. This will stop the control thread from erroneously changing the value if a mutator has already set it to `shenandoah_alloc_failure`. A mutator thread may still have an allocation failure after the control thread has cleared the cancellation, but this is normal and expected.

There's a typo in description of the issue: s/if/is/

Have we been able to confirm through testing that this resolves the previously observed assertion failure?

-------------

Marked as reviewed by kdnilsen (Committer).

PR Review: https://git.openjdk.org/jdk/pull/27662#pullrequestreview-3307686078