RFR: 8263476: Use reserved memory for stack guard pages [v3]

[Albert Mingkun Yang]

On Wed, 6 Aug 2025 09:40:22 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> I am not sure I get the point of this RFE, or the problem it tries to solve.

It's to reduce complexity of stack-guard-pages construction -- the end result changes files on Linux and BSD only. Note that this is the only place we can commit memory without first reserving the corresponding memory.

> The difference between /proc/smaps output and NMT committed output - isn't this just wrong accounting on behalf of NMT?

The end result, as a side effect, also removes such inconsistency. Both would show them as reserved-only.

> POSIX says its not valid to protect memory not allocated by mmap

Linux allows it:

"""
On Linux, it is always permissible to call mprotect() on any
address in a process's address space (except for the kernel
vsyscall area).
"""

> Apart from being hard to understand, it also opens a time window where the stack guard pages are unmapped;

Could you elaborate which part is hard to understand? I think the newly added diagram illustrates the stack-frame clearly.

> but in this time window someone else may have mapped pages in there.

Since Linux main-thread dynamically reserves memory for stack, it's always possible that address-ranges smaller than stack-top (including stack-guard-pages) are already reserved by others. I could "reuse" the reserved memory range for stack-guard-pages, but that can result in variance in NMT-accounting, depending on how OS stack reserving status.

>  If the primordial thread stack grows into this region, it will overwrite that misplaced memory. Errors would be very rare and intermittent.

According to `MAP_GROWSDOWN` (used for main-thread), it's a guaranteed segfault. (I also tried a small C program, artificially mmaping sth at stack-end, then allocating sth large on the stack. A segfault is generated as expected.)

"""
This growth can be repeated until the
mapping grows to within a page of the high end of the next
lower mapping, at which point touching the "guard" page
will result in a SIGSEGV signal.
"""

> Apart from all that, it also means that we add two unnecessary calls to mmap to likely every startup, since the condition at line 3475 is now likely always true.

This code is only for `is_primordial_thread == true`, but this is rarely used. Hence, David's concerns on testibility, "because we never use the primordial thread to run the VM".

Therefore, this actually removes one commit-call in the most common path, because non-main-thread performs zero preparation now.

> Pre-existing: to be honest, I have no idea why we even do the munmap in the first place. Why would it matter if the primordial thread stack is larger than what we think it should be?

It's covered in the doc right before `os::pd_create_stack_guard_pages` in Linux


// However, it's essential not to split the stack region by unmapping
// a region (leaving a hole) that's already part of the stack mapping,
// so if the stack mapping has already grown beyond the guard pages at
// the time we create them, we have to truncate the stack mapping.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/26571#issuecomment-3159809021