RFR: 8366434: THP not working properly with G1 after JDK-8345655

[Stefan Johansson]

On Wed, 3 Sep 2025 08:51:38 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> Please review this fix to enable transparent huge pages for G1,
>> 
>> **Summary**
>> In [JDK-8345655](https://bugs.openjdk.org/browse/JDK-8345655) we refactored the memory reservation code to be more maintainable and easier to follow. When doing this one of the code paths changed to always pass in `os::vm_page_size()` where it previously had used a page size provided by the caller.
>> 
>> Even if the alignment for `ReservedSpaces` created this way show that they should be aligned to support large pages the page size member for the `ReservedSpace` does not convey that we want large pages for the space. In G1 when using `G1PageBaseVirtualSpace` we use the above mentioned page size as the alignment for the reservation. This leads to reservations (made using the API) not being THP eligible even if `-XX:+UseTransparentHugePages` is specified.
>> 
>> This is only an issue when the system is configured with the THP mode `madvise`. If the mode is `always`, we will get THP eligible reservations. So a fairly simple workaround for this issue (given you have access to configuring your system) is to configure the THP mode to always.
>> 
>> The fix is to simply change back to the old behavior and pass in the user provided page size to the `ReservedSpace`. We've also added a test that verifies that we try to back the heap with transparent huge pages when `-XX:+UseTransparentHugePages` is specified on the command-line.
>> 
>> **Testing**
>> * Mach5 testing tier1-tier5
>> * Manual testing of the new test both locally and on mach5. Making sure it has been executed on system with both `madvise` and `always` configured. Also made sure the test actually failed without the fix.
>
> Marked as reviewed by shade (Reviewer).

Thanks for the reviews @shipilev and @stefank

-------------

PR Comment: https://git.openjdk.org/jdk/pull/27051#issuecomment-3252130640