[jdk21u-dev] RFR: 8354922: ZGC: Use MAP_FIXED_NOREPLACE when reserving memory

[Sergey Chernyshev]

Hi all,

Please review the backport of JDK-8354922 to JDK 21.

There are versions of the Linux kernel that do not honor the address hint when mmapping memory without MAP_FIXED, that clobbers the older mappings overlapped with the requested range. A safer MAP_FIXED_NOREPLACE flag is used with ZGC since Java 25.

JDK 21 crashes with -XX:+UseZGC with RANDMMAP [1] kernel patch enabled, while JDK 25 works as expected.


$ ./25.0.1/bin/java -XX:+UseZGC -version
openjdk version "25.0.1" 2025-10-21 LTS
OpenJDK Runtime Environment (build 25.0.1+11-LTS)
OpenJDK 64-Bit Server VM (build 25.0.1+11-LTS, mixed mode, sharing)

$ ./21.0.9/bin/java -XX:+UseZGC -version
[0.070s][error][gc] Failed to reserve enough address space for Java heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.


ZGC relies on kernel that respects address hints in mmap system call, which is not the case with RANDMMAP enabled kernels. ZVirtualMemoryManager::pd_reserve() always returns false at line 50 because the mmap returned value never matches the requested address.

https://github.com/openjdk/jdk21u-dev/blob/634d3fa7f147fc6fb1a7765755a3feed82ffaf50/src/hotspot/os/posix/gc/z/zVirtualMemory_posix.cpp#L40-L51

Since Linux 4.17 there's MAP_FIXED_NOREPLACE flag in mmap, that satisfies the `addr` requests and reports failure when the requested range overlaps a pre-existing mapping.

The backport isn't clean. JDK 21 doesn't have JDK-8350441 that renamed zVirtualMemory_posix.cpp to zVirtualMemoryManager_posix.cpp, and JDK-8341692 that removed non-generational mode in ZGC. In absense of JDK-8341692 the same approach was taken on gc/x/ version of pd_reserve().

It is also proposed to backport JDK-8313319 that prevents unnecessary mmap-munmap cycle, that will follow in a separate PR.

[1] https://pax.grsecurity.net/docs/randmmap.txt

-------------

Commit messages:
 - 8354922: ZGC: Use MAP_FIXED_NOREPLACE when reserving memory

Changes: https://git.openjdk.org/jdk21u-dev/pull/2475/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk21u-dev&pr=2475&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8354922
  Stats: 20 lines in 4 files changed: 18 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/jdk21u-dev/pull/2475.diff
  Fetch: git fetch https://git.openjdk.org/jdk21u-dev.git pull/2475/head:pull/2475

PR: https://git.openjdk.org/jdk21u-dev/pull/2475