RFR: 8370041: GenShen: Filter young pointers from thread local SATB buffers when only marking old

[Kelvin Nilsen]

On Fri, 24 Oct 2025 21:00:40 GMT, William Kemper <wkemper at openjdk.org> wrote:

> When GenShen is only marking the old generation, we do not need the SATB mechanism to preserve young pointers. We currently filter these out of the SATB buffers during the final-update-refs and init-mark safepoints. This increases latency and introduces no small amount of complexity. It should be possible to instead filter out these pointers when the SATB buffers are 'compacted' before being 'completed'.
> 
> # Background
> When GenShen is marking the old generation it leaves the SATB barrier enabled. When a young collection interrupts old marking, it creates a situation where a mutator thread could overwrite a field holding a pointer into a collection set region. The SATB barrier will dutifully place this object in the SATB queue. If this pointer makes it into a mark queue, the marking thread will crash. Prior to this change, GenShen filtered out such pointers _after_ the thread local SATB buffers were completed. After this change, such pointers are filtered out _before_ the buffers are completed. This is more inline with the natural way of things.

This is a great simplification.  Do we have any performance numbers, especially for reduction in p99.999 and p100 latencies with certain Extremem workloads, which I believe to be related to safepoint flushing of satb buffers?

src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.cpp line 1144:

> 1142:   //     becoming part of the old snapshot.
> 1143:   //  2. The region is allocated during evacuation of old. This is also not a concern because
> 1144:   //     we haven't yet finished marking old so no mixed evacuations will happen.

Might be worth mentioning that there may be some additional analysis and code required when we use the forwarding table to recycle cset regions during evacuation and/or updating.  If one of these regions becomes old before the SATB buffers have been flushed, then a young cset pointer that lingers in a SATB buffer will "all of a sudden" look like a valid old pointer and will not be purged from the SATB buffer.  When we subsequently scan the "object" referenced by this obsolete pointer, we are likely to find "garbage memory", possibly resulting in a crash.

I am thinking that an initial fix might be to do this flushing at init-update-refs instead of at final update refs, and to not recycle cset regions until evacuation is all done.  Is there a different handshake there that we might piggyback on?

-------------

PR Review: https://git.openjdk.org/jdk/pull/27983#pullrequestreview-3378867185
PR Review Comment: https://git.openjdk.org/jdk/pull/27983#discussion_r2461998590