RFR: 8370198: Test gc/arguments/TestShrinkHeapInSteps.java crashed: assert(left >= right) failed: avoid underflow

[Albert Mingkun Yang]

On Wed, 19 Nov 2025 15:39:27 GMT, Albert Mingkun Yang <ayang at openjdk.org> wrote:

> Add an early-return for outside-heap address in `CollectedHeap::is_in` API.
> 
> While investigating this failure (JDK-8370198), I realized that some java-threads (compiler-threads) in `native` state can invoke `CollectedHeap` APIs. Since heap-resizing occurs inside safepoint but java-threads in `native` state just ignore safepoint, I have added some assert to catch such dangerous uses, where the return value might not be stable.
> 
> Test: tie1-5; can't reproduce the JDK-8370198 with or without this patch for >8000 runs.

There are three related but distinct parts in this problem/solution:

1. **Asserting that an address is outside the heap-reserved space when the caller is in native state.**

   For example, the assertion in `MacroAssembler::set_narrow_klass`.

2. **Adding an early return to `is_in` for addresses outside the heap-reserved space.**

   This is always correct and almost trivial.

3. **Adding an assertion inside `is_in` to catch potentially racy or unsafe uses of this API for addresses that fall inside the heap-reserved range.**

---

## Analysis

To resolve the specific crash observed once in JDK-8370198, removing the unnecessary assertion in `set_narrow_klass` is sufficient. However, because the reproduction rate is extremely low, I suspect there may be other call sites with similar usage patterns -- that is, calling `is_in` with addresses outside the heap-reserved range. For this reason, part (2) is beneficial.

Additionally, there may be other problematic uses of this API that became unsafe after the recent heap/young-generation resizing support. Part (3) aims to uncover such issues.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/28393#issuecomment-3563162859