RFR: JDK-8293114: GC should trim the native heap [v10]

[Aleksey Shipilev]

On Thu, 2 Mar 2023 16:37:47 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> (*Updated 2023-07-05 to reflect the current state of the patch*)
>> 
>> This RFE adds the option to auto-trim the Glibc heap as part of the GC cycle. If the VM process suffered high temporary malloc spikes (regardless of whether from JVM- or user code), this could recover significant amounts of memory.
>> 
>> We discussed this a year ago [1], but the item got pushed to the bottom of my work pile, therefore, it took longer than I thought.
>> 
>> ### Motivation
>> 
>> The Glibc is reluctant to return memory to the OS, more so than other allocators. Temporary malloc spikes often carry over as permanent RSS increase. Note that C-heap retention is difficult to observe. Since it is freed memory, it won't appear in NMT; it is just a part of RSS.
>> 
>> This is, effectively, caching, and a performance tradeoff by the glibc. It makes a lot of sense with applications that cause high traffic on the C-heap (the typical native application). The JVM, however, clusters allocations and for a lot of use cases rolls its own memory management via mmap. And app's malloc load can fluctuate wildly, with temporary spikes and long idle periods.
>> 
>> To help, Glibc exports an API to trim the C-heap: `malloc_trim(3)`. With JDK 18 [2], SAP contributed a new jcmd command to *manually* trim the C-heap on Linux. This RFE adds a complementary way to trim automatically.
>> 
>> #### Is this even a problem?
>> 
>> Yes. 
>> 
>> The JVM clusters most native memory allocations and satisfies them with mmap. But there are enough C-heap allocations left to cause malloc spikes that are subject of memory retention. Note that one example are hotspot arenas themselves.
>> 
>> But many cases of high memory retention in Glibc I have seen in third-party JNI code. Libraries allocate large buffers via malloc as temporary buffers. In fact, since we introduced the jcmd "System.trim_native_heap", some of our customers started to call this command periodically in scripts to counter these issues.
>> 
>> ### How trimming works
>> 
>> Trimming is done via `malloc_trim(2)`. `malloc_trim` will iterate over all arenas and trim each one subsequently. While doing that, it will lock the arena, which may cause some (but not all) subsequent actions on the same arenas to block. glibc also trims automatically on free, but that is very limited (see https://github.com/openjdk/jdk/pull/10085#issuecomment-1619638641 for details).
>> 
>> `malloc_trim` offers almost no way to control its behavior; in particular, no way to limit its runtime. Its run...
>
> Thomas Stuefe has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 34 commits:
> 
>  - wip
>  - Merge branch 'master' into JDK-8293114-GC-trim-native
>  - wip
>  - merge master
>  - wip
>  - wip
>  - rename GCTrimNative TrimNative
>  - rename NativeTrimmer
>  - rename
>  - src/hotspot/share/gc/shared/gcTrimNativeHeap.cpp
>  - ... and 24 more: https://git.openjdk.org/jdk/compare/99f5687e...5d41312e

Okay, cool. The reason I am asking it that glibc "memory leaks" are not uncommon in production cases. There are quite a few libraries that churn native memory (looks at Netty), even with internal pooling. Having something that is backportable to 21u and 17u would be a plus.

So I see that `malloc_trim` actually has the parameter `pad`:


       The pad argument specifies the amount of free space to leave
       untrimmed at the top of the heap.  If this argument is 0, only
       the minimum amount of memory is maintained at the top of the heap
       (i.e., one page or less).  A nonzero argument can be used to
       maintain some trailing space at the top of the heap in order to
       allow future allocations to be made without having to extend the
       heap with [sbrk(2)](https://man7.org/linux/man-pages/man2/sbrk.2.html).


Does current glibc honor that argument at all? Can we use that to control the incrementality of the trim? Since `malloc_trim` return value also describes if trimming was successful, we could probably come up with the adaptive scheme that guesses which pad to use, so that we don't trim the entirety of the memory, but still trim something.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/10085#issuecomment-1614633876
PR Comment: https://git.openjdk.org/jdk/pull/10085#issuecomment-1621991402