RFR: 8263476: Use reserved memory for stack guard pages [v3]

[Thomas Stuefe]

On Mon, 4 Aug 2025 14:04:15 GMT, Albert Mingkun Yang <ayang at openjdk.org> wrote:

>> Use reserved (instead of committed) memory for stack-guard-pages on linux like systems.
>> 
>> `os::must_commit_stack_guard_pages` uses `commit` in its name, but `commit` usually has specific meanings in OS memory context. The actual question the caller is asking is whether the caller needs to do some preparation work before marking the guard-pages as inaccessible. To avoid confusion, I changed it to "allocate". Other suggestions are welcome.
>> 
>> Test: tier1-3
>
> Albert Mingkun Yang has updated the pull request incrementally with one additional commit since the last revision:
> 
>   review

@albertnetymk @zhengyu123 I am not sure I get the point of this RFE, or the problem it tries to solve. The difference between /proc/smaps output and NMT committed output - isn't this just wrong accounting on behalf of NMT? The pages are marked in /proc/smaps with RSS=0 because they are untouched. If you were to touch them before the commit, they probably would show up as committed, but it would be better to just count them in NMT as uncommitted.

About the commit itself: 

- reserved memory (MAP_NORESERVE) is not guaranteed to crash on touch. It usually does, but there is nothing in the standard saying it has to
- so, we mprotect the pages. However, POSIX says its not valid to protect memory not allocated by mmap. Even if they were mmap'ed, there were platforms, not sure which, which did not allow mprotect on memory allocated with mmap(MAP_NORESERVE). Not an issue on Linux, but I think there were other platforms that had this problem.
- therefore we may need to commit before calling mprotect.

Fun fact: We also may run on *customer-provided* stacks. The thread may have been launched externally, e.g. by a third-party custom launcher, and attached later; the customer may have started the thread providing their own stacks, which could live wherever. I have only seen this once with a particularly enterprising customer, and they placed the stacks in bss.

-------------

PR Review: https://git.openjdk.org/jdk/pull/26571#pullrequestreview-3091512380