RFR: 8303215: Make thread stacks not use huge pages [v3]

[David Holmes]

On Fri, 26 May 2023 20:18:02 GMT, Poonam Bajaj <poonam at openjdk.org> wrote:

>> When a system has Transparent Huge Pages (THP) enabled (/sys/kernel/mm/transparent_hugepage/enabled is set to 'always'), thread stacks can have significantly more resident set size (RSS) than they actually require. This occurs when the stack size is 2MB or larger, which makes the memory range of the stack more likely to be aligned on a Large Page Size boundary (2MB on most systems). This in turn makes the stack eligible to be backed by transparent huge pages resulting in more memory consumption than it would otherwise when standard small pages are used. This issue is more apparent on AArch64 platforms where the default stack size is 2MB. 
>> 
>> 
>> Example mapping from smaps illustrating this issue:
>> fffced200000-fffced204000 ---p 00000000 00:00 0 
>> Size:                16 kB     # guard pages
>> KernelPageSize:       4 kB
>> MMUPageSize:          4 kB
>> ...
>> fffced204000-fffced400000 rw-p 00000000 00:00 0 
>> Size:              2032 kB    # stack space
>> KernelPageSize:       4 kB
>> MMUPageSize:          4 kB
>> Rss:               2032 kB    # entire stack resident in memory
>> 
>> 
>> This fix addresses this issue with the following two main changes:
>> 
>> 1. Change the default stack size to 2040KB, which is 2 pages less than 2MB. This ensures that stacks don't get 2MB aligned. And why 2 pages less than 2MB, because for non-JavaThreads, glibc adds an additional guard page to the total stack size. To keep it simple and to keep the default stack size value for  all options - ThreadStackSize, CompilerThreadStackSize, and VMThreadStackSize same, we use the default value as 2040K.
>> 
>> Example mapping for a JavaThread:
>> 
>> ffff6e913000-ffff6e917000 ---p 00000000 00:00 0 
>> Size:                 16 kB
>> KernelPageSize:        4 kB
>> MMUPageSize:           4 kB
>> ...
>> ffff6e917000-ffff6eb11000 rw-p 00000000 00:00 0 
>> Size:               2024 kB
>> KernelPageSize:        4 kB
>> MMUPageSize:           4 kB
>> Rss:                  92 kB
>> 
>> Example Mapping for a non-JavaThread (WatcherThread):
>> 
>> ffff6eb11000-ffff6eb12000 ---p 00000000 00:00 0 
>> Size:                  4 kB
>> KernelPageSize:        4 kB
>> MMUPageSize:           4 kB
>> ...
>> ffff6eb12000-ffff6ed10000 rw-p 00000000 00:00 0 
>> Size:               2040 kB
>> KernelPageSize:        4 kB
>> MMUPageSize:           4 kB
>> Rss:                  12 kB
>> 
>> 
>> 2. If the requested stack size is greater than or equal to the default large page size (2MB on most systems) and can be large-page aligned, then add an ...
>
> Poonam Bajaj has updated the pull request incrementally with one additional commit since the last revision:
> 
>   addressed review comments

Changes requested by dholmes (Reviewer).

src/hotspot/os/linux/os_linux.cpp line 934:

> 932:   if (stack_size >= os::Linux::default_large_page_size() && is_aligned(stack_size, os::Linux::default_large_page_size())) {
> 933:     stack_size += os::vm_page_size();
> 934:   }

Do we need to watch for a zero default size here so that we don't expand all stacks by a page?

src/hotspot/os/linux/os_linux.cpp line 3754:

> 3752: 
> 3753: void os::large_page_init() {
> 3754:   // Scan default large page size

Suggestion:

// Always initialize the default large page size even if large pages are not being used.

src/hotspot/os/linux/os_linux.cpp line 3759:

> 3757: 
> 3758:   // 1) Handle the case where we do not want to use huge pages and hence
> 3759:   //    there is no need to scan the OS for related info

The comment needs updating as now you do the scan first.

src/hotspot/os_cpu/linux_aarch64/globals_linux_aarch64.hpp line 34:

> 32: define_pd_global(bool, DontYieldALot,            false);
> 33: 
> 34: // set default stack sizes < 2MB so as to prevent stacks from getting

/set/Set/

src/hotspot/os_cpu/linux_aarch64/globals_linux_aarch64.hpp line 36:

> 34: // set default stack sizes < 2MB so as to prevent stacks from getting
> 35: // large-page aligned and backed by THPs on systems where 2MB is the
> 36: // default huge page size. For non-JavaThreads, glibc adds an additional

s/adds/may add/

src/hotspot/os_cpu/linux_aarch64/globals_linux_aarch64.hpp line 38:

> 36: // default huge page size. For non-JavaThreads, glibc adds an additional
> 37: // guard page to the total stack size, so to keep the default sizes same
> 38: // for all the following flags, we set them to 2 pages less than 2MB.

I still think we need to at least mention we assume 4K page here.

-------------

PR Review: https://git.openjdk.org/jdk/pull/14105#pullrequestreview-1448579948
PR Review Comment: https://git.openjdk.org/jdk/pull/14105#discussion_r1208737751
PR Review Comment: https://git.openjdk.org/jdk/pull/14105#discussion_r1208738001
PR Review Comment: https://git.openjdk.org/jdk/pull/14105#discussion_r1208735912
PR Review Comment: https://git.openjdk.org/jdk/pull/14105#discussion_r1208745365
PR Review Comment: https://git.openjdk.org/jdk/pull/14105#discussion_r1208745459
PR Review Comment: https://git.openjdk.org/jdk/pull/14105#discussion_r1208746341