RFR: 8366434: THP not working properly with G1 after JDK-8345655

[Aleksey Shipilev]

On Tue, 2 Sep 2025 13:53:13 GMT, Stefan Johansson <sjohanss at openjdk.org> wrote:

> Please review this fix to enable transparent huge pages for G1,
> 
> **Summary**
> In [JDK-8345655](https://bugs.openjdk.org/browse/JDK-8345655) we refactored the memory reservation code to be more maintainable and easier to follow. When doing this one of the code paths changed to always pass in `os::vm_page_size()` where it previously had used a page size provided by the caller.
> 
> Even if the alignment for `ReservedSpaces` created this way show that they should be aligned to support large pages the page size member for the `ReservedSpace` does not convey that we want large pages for the space. In G1 when using `G1PageBaseVirtualSpace` we use the above mentioned page size as the alignment for the reservation. This leads to reservations (made using the API) not being THP eligible even if `-XX:+UseTransparentHugePages` is specified.
> 
> This is only an issue when the system is configured with the THP mode `madvise`. If the mode is `always`, we will get THP eligible reservations. So a fairly simple workaround for this issue (given you have access to configuring your system) is to configure the THP mode to always.
> 
> The fix is to simply change back to the old behavior and pass in the user provided page size to the `ReservedSpace`. We've also added a test that verifies that we try to back the heap with transparent huge pages when `-XX:+UseTransparentHugePages` is specified on the command-line.
> 
> **Testing**
> * Mach5 testing tier1-tier5
> * Manual testing of the new test both locally and on mach5. Making sure it has been executed on system with both `madvise` and `always` configured. Also made sure the test actually failed without the fix.

Marked as reviewed by shade (Reviewer).

-------------

PR Review: https://git.openjdk.org/jdk/pull/27051#pullrequestreview-3179634396