RFR: 8366434: THP not working properly with G1 after JDK-8345655

[Stefan Johansson]

On Tue, 2 Sep 2025 15:10:48 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> Please review this fix to enable transparent huge pages for G1,
>> 
>> **Summary**
>> In [JDK-8345655](https://bugs.openjdk.org/browse/JDK-8345655) we refactored the memory reservation code to be more maintainable and easier to follow. When doing this one of the code paths changed to always pass in `os::vm_page_size()` where it previously had used a page size provided by the caller.
>> 
>> Even if the alignment for `ReservedSpaces` created this way show that they should be aligned to support large pages the page size member for the `ReservedSpace` does not convey that we want large pages for the space. In G1 when using `G1PageBaseVirtualSpace` we use the above mentioned page size as the alignment for the reservation. This leads to reservations (made using the API) not being THP eligible even if `-XX:+UseTransparentHugePages` is specified.
>> 
>> This is only an issue when the system is configured with the THP mode `madvise`. If the mode is `always`, we will get THP eligible reservations. So a fairly simple workaround for this issue (given you have access to configuring your system) is to configure the THP mode to always.
>> 
>> The fix is to simply change back to the old behavior and pass in the user provided page size to the `ReservedSpace`. We've also added a test that verifies that we try to back the heap with transparent huge pages when `-XX:+UseTransparentHugePages` is specified on the command-line.
>> 
>> **Testing**
>> * Mach5 testing tier1-tier5
>> * Manual testing of the new test both locally and on mach5. Making sure it has been executed on system with both `madvise` and `always` configured. Also made sure the test actually failed without the fix.
>
> test/hotspot/jtreg/gc/TestTransparentHugePagesHeap.java line 105:
> 
>> 103: 
>> 104:             final Pattern heapSection = Pattern.compile("^" + heapAddress + ".*");
>> 105:             final Pattern thpEligible = Pattern.compile("THPeligible:\\s+(\\d)\\s*");
> 
> I thought we had the `smaps` parser somewhere, and here it is: https://github.com/openjdk/jdk/blob/444a8fa14e8ab016b8aae018054c5dc1eb843fee/test/hotspot/jtreg/runtime/os/TestTracePageSizes.java#L149 -- maybe yank it from there to some testlib location, and use it?

I agree that this is what we want long-term, we talked about this internally as well. At one point we also just extended the `TestTracePageSizes.java` test to ensure THP for the heap to avoid some duplication. In the end we felt creating a new test was the best approach right now, to also allow for small and safe backport to 25u.

Created [JDK-8366716](https://bugs.openjdk.org/browse/JDK-8366716) to keep track of this. 

Are you good with this approach?

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/27051#discussion_r2316672407