RFR: JDK-8293114: GC should trim the native heap [v11]

[Thomas Stuefe]

> (*Updated 2023-07-05 to reflect the current state of the patch*)
> 
> This RFE adds the option to auto-trim the Glibc heap as part of the GC cycle. If the VM process suffered high temporary malloc spikes (regardless of whether from JVM- or user code), this could recover significant amounts of memory.
> 
> We discussed this a year ago [1], but the item got pushed to the bottom of my work pile, therefore, it took longer than I thought.
> 
> ### Motivation
> 
> The Glibc is reluctant to return memory to the OS, more so than other allocators. Temporary malloc spikes often carry over as permanent RSS increase. Note that C-heap retention is difficult to observe. Since it is freed memory, it won't appear in NMT; it is just a part of RSS.
> 
> This is, effectively, caching, and a performance tradeoff by the glibc. It makes a lot of sense with applications that cause high traffic on the C-heap (the typical native application). The JVM, however, clusters allocations and for a lot of use cases rolls its own memory management via mmap. And app's malloc load can fluctuate wildly, with temporary spikes and long idle periods.
> 
> To help, Glibc exports an API to trim the C-heap: `malloc_trim(3)`. With JDK 18 [2], SAP contributed a new jcmd command to *manually* trim the C-heap on Linux. This RFE adds a complementary way to trim automatically.
> 
> #### Is this even a problem?
> 
> Yes. 
> 
> The JVM clusters most native memory allocations and satisfies them with mmap. But there are enough C-heap allocations left to cause malloc spikes that are subject of memory retention. Note that one example are hotspot arenas themselves.
> 
> But many cases of high memory retention in Glibc I have seen in third-party JNI code. Libraries allocate large buffers via malloc as temporary buffers. In fact, since we introduced the jcmd "System.trim_native_heap", some of our customers started to call this command periodically in scripts to counter these issues.
> 
> ### How trimming works
> 
> Trimming is done via `malloc_trim(2)`. `malloc_trim` will iterate over all arenas and trim each one subsequently. While doing that, it will lock the arena, which may cause some (but not all) subsequent actions on the same arenas to block. glibc also trims automatically on free, but that is very limited (see https://github.com/openjdk/jdk/pull/10085#issuecomment-1619638641 for details).
> 
> `malloc_trim` offers almost no way to control its behavior; in particular, no way to limit its runtime. Its runtime will depend on the size of the reclaimed memory. ...

Thomas Stuefe has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 41 commits:

 - fix windows build
 - Merge branch 'master' into JDK-8293114-GC-trim-native
 - wip
 - Merge branch 'master' into JDK-8293114-GC-trim-native
 - wip
 - Remove adaptive stepdown coding
 - Merge master
 - wip
 - Merge branch 'master' into JDK-8293114-GC-trim-native
 - wip
 - ... and 31 more: https://git.openjdk.org/jdk/compare/22e17c29...162b880a

-------------

Changes: https://git.openjdk.org/jdk/pull/10085/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10085&range=10
  Stats: 653 lines in 21 files changed: 645 ins; 1 del; 7 mod
  Patch: https://git.openjdk.org/jdk/pull/10085.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/10085/head:pull/10085

PR: https://git.openjdk.org/jdk/pull/10085