RFR: 8377011: Shenandoah: assert_bounds should be only called when boundaries have changed [v3]

[William Kemper]

On Wed, 4 Feb 2026 00:58:58 GMT, Xiaolong Peng <xpeng at openjdk.org> wrote:

>> This fix the problem causing following 3 jtreg tests with tlab disabled to fail:
>> 
>> gc/shenandoah/TestRetainObjects.java#no-tlab 8361099 generic-all
>> gc/shenandoah/TestSieveObjects.java#no-tlab 8361099 generic-all
>> gc/shenandoah/TestSieveObjects.java#no-tlab-genshen 8361099 generic-all
>> 
>> 
>> 
>> The problem is assert_bounds is always called from function ShenandoahFreeSet::try_allocate_in, no matter the boundaries have changed or not after the allocation,  this is new behavior introduced by https://bugs.openjdk.org/browse/JDK-8365880, before which assert_bounds was only called after retiring a region.
>> 
>> The fix is to call assert_bounds only when boundaries have changed:
>> 1. When retire a region;
>> 2. When a the object is allocated in a new region - one region became non-empty.
>> 
>> There is also minor improvement to the if-else code in the PR.
>> 
>> Overall the change should be safe and only affect Shenandoah related tests, I have run Shenandoah test suite on MacOS.
>> 
>> Profiler data from the problematic jtreg test: [jtreg-gc_shenandoah_TestRetainObjects_no_tlab.html](https://github.com/user-attachments/files/25027324/jtreg-gc_shenandoah_TestRetainObjects_no_tlab.html) 
>> 
>> ### Tests
>> - [x] hotspot_gc_shenandoah
>> - [x] GHA
>
> Xiaolong Peng has updated the pull request incrementally with one additional commit since the last revision:
> 
>   code format

Marked as reviewed by wkemper (Reviewer).

src/hotspot/share/gc/shenandoah/shenandoahFreeSet.cpp line 1677:

> 1675:   }
> 1676:   DEBUG_ONLY(bool boundary_changed = false;)
> 1677:   if ((result != nullptr) && in_new_region) {

Why make this a standalone test instead of having it be subordinate to `alloc_capacity(r) < PLAB::min_size() * HeapWordSize` as it was before?

-------------

PR Review: https://git.openjdk.org/jdk/pull/29537#pullrequestreview-3765791072
PR Review Comment: https://git.openjdk.org/jdk/pull/29537#discussion_r2776535181