Support for >4MB largepages in Intel/AMD

[David Holmes]

Hi Nitin,

On 6/10/2023 7:13 am, Nitin Singh wrote:
> Hello OpenJDK Community,
> 
> I've been immersing myself in understanding large pages within OpenJDK, 
> especially for the Windows OS. Here's an observation I'd like to share 
> and seek feedback on.
> 
> It appears that on Windows Server 2022 Datacenter (OS Version: 
> 10.0.20348), when employing AMD or Intel processors, there's an implicit 
> restriction that limits the use of large page sizes to 4MB. This 
> constraint seems to originate from line 3199 in |os_windows.cpp|
> 
> https://github.com/openjdk/jdk/blob/2e340e855b760e381793107f2a4d74095bd40199/src/hotspot/os/windows/os_windows.cpp#L3199. <https://github.com/openjdk/jdk/blob/2e340e855b760e381793107f2a4d74095bd40199/src/hotspot/os/windows/os_windows.cpp#L3199.>
> 
> To probe this further, I tweaked the code to see what would happen 
> without this restriction. The results were enlightening. I was able to 
> successfully use an 8MB large page size, and my subsequent benchmarks 
> post this change have been quite positive.
> 
> What heightens my curiosity is the knowledge that these processors 
> support up to 1G of large pages (HugeTLB pages). Given the widespread 
> support for these processors by various cloud providers, it does feel 
> like we're missing out on potential optimizations for HotSpot running on 
> Windows.
> 
> Before delving deeper, I'd love to get some historical context. Can 
> anyone share insights about why this 4MB limit was imposed in the first 
> place? Were there specific technical challenges or concerns that led to 
> this decision?
> 
> I'm currently working with the Microsoft JDK version 
> |microsoft-jdk-17.0.8-windows-x64|.
> 
> Thank you for shedding light on this matter. I deeply appreciate your 
> guidance as I tread these waters.

This is a very old limit that was put in place back in 2006 for JDK 6 
with backports to 5u11 and 1.4.2. Unfortunately the issue used for this 
is not publicly visible. The original issue with the large page 
selection algorithm was that it would select the largest available page 
size, and when Opteron supported 1GB large pages that caused a problem. 
So the decision was made to clamp at 4MB (which was much more suitable 
for the Windows hardware of the time) with a view initially that the 
user could override this using LargePageSizeInBytes. But it was then 
decided that the 4MB limit needed to be firm, the basic reasons given being:

1. The code cache uses the same page size as the Java heap, and it's
bad news to map the code cache with 1gb pages on Opteron.

2. On x86 (32-bit), rounding the Java heap size to a 1gb boundary can 
cause the vm to fail at startup due to inability to map the heap 
contiguously.Even if mmap succeeds, a user could be surprised by the 
increase in memory use.

3. On x64, we don't currently see heap sizes larger than ~14gb, so the 
use of 1gb pages isn't as urgent as it would be with the much larger 
heaps encountered on, say, large sparc SMPs.

I don't know if #1 is still true but see 
https://bugs.openjdk.org/browse/JDK-8304954.

#2 is no longer relevant, and #3 is probably also not accurate.

I suspect that because of this inherent limit a number of large-page 
related changes have not been applied to Windows, so there may be a bit 
of  "cross-porting" to do - seach JBS for "project = jdk AND component = 
hotspot AND text ~ LargePageSizeInBytes" to get some idea of the issues 
involved.

HTH.

David
-----

> Thanks,
> 
> Nitin Singh
>