RFR: JDK-8293114: GC should trim the native heap [v10]

[Volker Simonis]

On Thu, 2 Mar 2023 16:37:47 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> (*Updated 2023-07-05 to reflect the current state of the patch*)
>> 
>> This RFE adds the option to auto-trim the Glibc heap as part of the GC cycle. If the VM process suffered high temporary malloc spikes (regardless of whether from JVM- or user code), this could recover significant amounts of memory.
>> 
>> We discussed this a year ago [1], but the item got pushed to the bottom of my work pile, therefore, it took longer than I thought.
>> 
>> ### Motivation
>> 
>> The Glibc is reluctant to return memory to the OS, more so than other allocators. Temporary malloc spikes often carry over as permanent RSS increase. Note that C-heap retention is difficult to observe. Since it is freed memory, it won't appear in NMT; it is just a part of RSS.
>> 
>> This is, effectively, caching, and a performance tradeoff by the glibc. It makes a lot of sense with applications that cause high traffic on the C-heap (the typical native application). The JVM, however, clusters allocations and for a lot of use cases rolls its own memory management via mmap. And app's malloc load can fluctuate wildly, with temporary spikes and long idle periods.
>> 
>> To help, Glibc exports an API to trim the C-heap: `malloc_trim(3)`. With JDK 18 [2], SAP contributed a new jcmd command to *manually* trim the C-heap on Linux. This RFE adds a complementary way to trim automatically.
>> 
>> #### Is this even a problem?
>> 
>> Yes. 
>> 
>> The JVM clusters most native memory allocations and satisfies them with mmap. But there are enough C-heap allocations left to cause malloc spikes that are subject of memory retention. Note that one example are hotspot arenas themselves.
>> 
>> But many cases of high memory retention in Glibc I have seen in third-party JNI code. Libraries allocate large buffers via malloc as temporary buffers. In fact, since we introduced the jcmd "System.trim_native_heap", some of our customers started to call this command periodically in scripts to counter these issues.
>> 
>> ### How trimming works
>> 
>> Trimming is done via `malloc_trim(2)`. `malloc_trim` will iterate over all arenas and trim each one subsequently. While doing that, it will lock the arena, which may cause some (but not all) subsequent actions on the same arenas to block. glibc also trims automatically on free, but that is very limited (see https://github.com/openjdk/jdk/pull/10085#issuecomment-1619638641 for details).
>> 
>> `malloc_trim` offers almost no way to control its behavior; in particular, no way to limit its runtime. Its run...
>
> Thomas Stuefe has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 34 commits:
> 
>  - wip
>  - Merge branch 'master' into JDK-8293114-GC-trim-native
>  - wip
>  - merge master
>  - wip
>  - wip
>  - rename GCTrimNative TrimNative
>  - rename NativeTrimmer
>  - rename
>  - src/hotspot/share/gc/shared/gcTrimNativeHeap.cpp
>  - ... and 24 more: https://git.openjdk.org/jdk/compare/99f5687e...5d41312e

My main concern with this change is increased latency. You wrote "*..concurrent malloc/frees are usually not blocked while trimming if they are satisfied from the local arena..*". Not sure what "*usually*" means here and how many mallocs are satisfied from a local arena. But introducing pauses up to a second seems significant for some applications. 

The other question is that I still don't understand if glibc-malloc will ever call `malloc_trim()` automatically (and in that case introduce the latency anyway). The manpage says that `malloc_trim()` "*..is automatically called by  free(3)  in  certain  circumstances;  see  the  discussion  of  `M_TOP_PAD`  and `M_TRIM_THRESHOLD` in `mallopt(3)`..*" but you reported that you couldn't observe any cleanup effect when playing around with `M_TRIM_THRESHOLD`. In the end, calling `malloc_trim()` periodically might even help to decrease latency if this prevents seldom, but longer automatic invocations of `malloc_trim()` by glibc itself.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/10085#issuecomment-1617840919