RFR: 8370041: GenShen: Filter young pointers from thread local SATB buffers when only marking old

[William Kemper]

On Fri, 24 Oct 2025 21:16:10 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> When GenShen is only marking the old generation, we do not need the SATB mechanism to preserve young pointers. We currently filter these out of the SATB buffers during the final-update-refs and init-mark safepoints. This increases latency and introduces no small amount of complexity. It should be possible to instead filter out these pointers when the SATB buffers are 'compacted' before being 'completed'.
>> 
>> # Background
>> When GenShen is marking the old generation it leaves the SATB barrier enabled. When a young collection interrupts old marking, it creates a situation where a mutator thread could overwrite a field holding a pointer into a collection set region. The SATB barrier will dutifully place this object in the SATB queue. If this pointer makes it into a mark queue, the marking thread will crash. Prior to this change, GenShen filtered out such pointers _after_ the thread local SATB buffers were completed. After this change, such pointers are filtered out _before_ the buffers are completed. This is more inline with the natural way of things.
>
> src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.cpp line 1144:
> 
>> 1142:   //     becoming part of the old snapshot.
>> 1143:   //  2. The region is allocated during evacuation of old. This is also not a concern because
>> 1144:   //     we haven't yet finished marking old so no mixed evacuations will happen.
> 
> Might be worth mentioning that there may be some additional analysis and code required when we use the forwarding table to recycle cset regions during evacuation and/or updating.  If one of these regions becomes old before the SATB buffers have been flushed, then a young cset pointer that lingers in a SATB buffer will "all of a sudden" look like a valid old pointer and will not be purged from the SATB buffer.  When we subsequently scan the "object" referenced by this obsolete pointer, we are likely to find "garbage memory", possibly resulting in a crash.
> 
> I am thinking that an initial fix might be to do this flushing at init-update-refs instead of at final update refs, and to not recycle cset regions until evacuation is all done.  Is there a different handshake there that we might piggyback on?

Hmm, that's a good point about recycling regions concurrently. I don't think we can flush before `init-update-refs` because forwarded pointers still exist and the SATB barrier doesn't try to resolve them. That is to say, "bad" pointers can be created throughout the update-refs phase. 

Even in the (hypothetical) scenario with a forwarding table, a region would only become `old` through an in-place-promotion (in which case, it will never have been recycled) or we will be doing a mixed evacuation. If we are running a mixed evacuation, we will already have finished marking old.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/27983#discussion_r2462100316