RFR: 8366787: Test runtime/StackGuardPages/TestStackGuardPagesNative.java hangs on some platforms

[David Holmes]

On Fri, 5 Sep 2025 09:51:58 GMT, mazhen <duke at openjdk.org> wrote:

> #### Summary
> 
> This PR fixes a hang in the `TestStackGuardPagesNative.java` test that occurs on certain Linux distributions (e.g., CentOS 7). The fix replaces an unbounded `for(;;)` loop in the native test code (`exeinvoke.c`) with a bounded `while` loop, making the test's behavior deterministic and robust across all platforms.
> 
> #### Problem
> 
> The test would hang and eventually time out on some platforms. This was caused by an unbounded `for(;;)` loop in the `do_overflow` function, which was introduced as part of the "hardening" fix in `JDK-8295344`.
> 
> *   On platforms like **CentOS 7**, this unbounded loop would not encounter a terminating signal in a timely manner, causing the native process to hang indefinitely until killed by the test harness.
> *   In contrast, on platforms like **Ubuntu 24**, the test would coincidentally pass because a `SEGV_MAPERR` would happen to terminate the loop. This highlighted that the test's success was reliant on platform-specific side effects, masking the underlying issue.
> 
> #### Solution
> 
> The solution is to replace the unbounded `for(;;)` loop with a bounded `while` loop. The condition `while (_kp_rec_count == 0 || _rec_count < _kp_rec_count)` ensures that the loop terminates deterministically after a specific number of allocations, corresponding to the overflow depth detected in the test's first phase.
> 
> This change makes the test's logic robust and its behavior consistent across different environments.

Actual fix looks good, but the test has to remain in the PL.

Thanks

test/hotspot/jtreg/ProblemList.txt line 109:

> 107: runtime/os/TestTracePageSizes.java#Parallel 8267460 linux-aarch64
> 108: runtime/os/TestTracePageSizes.java#Serial 8267460 linux-aarch64
> 109: runtime/StackGuardPages/TestStackGuardPagesNative.java 8303612 linux-all

You can't remove this from the ProblemList unless 8303612 is actually fixed.

-------------

Changes requested by dholmes (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/27114#pullrequestreview-3195163355
PR Review Comment: https://git.openjdk.org/jdk/pull/27114#discussion_r2329286591