RFR: 8263476: Use reserved memory for stack guard pages [v3]

[Thomas Stuefe]

On Mon, 4 Aug 2025 13:48:05 GMT, Johan Sjölen <jsjolen at openjdk.org> wrote:

>> There is no pre-existing bug. The previous code ensures the memory range for stack-guard-pages are committed, note `os::commit_memory` is invoked in all paths. OTOH, the "reserve memory" call is only for main-thread in the new code. Also added a diagram, if that helps.
>
> Aha, now I understand. The diagram helped a lot!

I don't think we should make this change, since it introduces a subtle bug.

Apart from being hard to understand, it also opens a time window where the stack guard pages are unmapped; at line 3478, we re-establish MAP_NORESERVE mapping here, but in this time window someone else may have mapped pages in there. Now someone lives in our stack. The reservation will fail, we return false, write a "cannot establish guard pages" warning. If the primordial thread stack grows into this region, it will overwrite that misplaced memory. Errors would be very rare and intermittent.

Apart from all that, it also means that we add two unnecessary calls to mmap to likely every startup, since the condition at line 3475 is now likely always true. 

Pre-existing: to be honest, I have no idea why we even do the munmap in the first place. Why would it matter if the primordial thread stack is larger than what we think it should be?

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/26571#discussion_r2256483228