Extend Native Memory Tracking over the JDK ? (was: Proposal: track zlib native memory usage with NMT)

[Stefan Johansson]

Hi Carter,

Your mail made me pick up an old item from my wishlist: to have native 
memory tracking information available in JFR recordings. When we, in GC, 
do improvements to decrease the native memory overhead of our 
algorithms, NMT is a very good tool to track the progress. We have 
scripts that sound very similar to what you describe and more than once 
I've been thinking about adding this information into JFR. But it has 
not been a priority and the greater value has been unclear.

Hearing that others might also benefit from such a change I took a 
discussion with the JFR team on how to best proceed with this. I have 
created a branch for this and will probably create a PR for it shortly, 
but I thought I would drop it here first:
https://github.com/kstefanj/jdk/tree/8157023-jfr-events-for-nmt

The change adds two new JFR events: one for the total usage and one for 
the usage of each memory type. These are sent only if Native Memory 
Tracking is turned on, and they are enabled in the default JFR profile 
with an interval of 1s. This might change during reviewing but it was a 
good starting point.

With this you will be able to use JFR streaming to access the events 
from within your running process. I hope this will help your use cases 
and please let us know if you have any comments or suggestions.

Thanks,
Stefan


On 2022-11-10 16:58, Carter Kozak wrote:
> /+serviceability-dev/
> 
> Firstly, thank you both for your time and work in this space. Apologies 
> if this should be a separate thread, but the new title “Extend Native 
> Memory Tracking over the JDK” aligns directly with some work I’ve been 
> investigating, and I hope my feedback will be helpful for prioritization 
> of zlib observability as well as the way users think about native memory 
> tracking in general.
> 
> Observability of native memory in the JVM is critically important, and 
> becomes even more valuable as the industry shifts to more and smaller 
> services deployed in right-sized container environments like kubernetes. 
> Each new JDK release (major and hotfix) offers dramatic improvements, 
> often based on some form of trade-off. To be clear, I cannot overstate 
> how impressed I am with quality and velocity of improvement! However, 
> these trade-offs impact the way that memory is used, and it’s a 
> difficult balance to ensure containers use the correct amount of memory 
> without being wasteful (over-provisioned) or oomkilled (under-provisioned).
> 
> In production, I have thousands of JVMs running with native memory 
> tracking summary enabled. Real-time monitoring of the output is painful 
> and inefficient. Currently the only supported option I’m aware of is 
> shelling out to create a new jcmd process and parsing the NMT summary 
> text output periodically. In older releases, it was possible to bypass 
> the jcmd process by self-attaching, but that was limited in jdk9 by 
> JDK-8178380 <https://bugs.openjdk.org/browse/JDK-8178380>, and still 
> required the caller to parse human-readable strings. In fact, attachment 
> issues in some JDK/environment combinations make automated attachment 
> /dangerous/ in a way that has crashed the JVM — that may be a story for 
> another day, but my point is that simple, efficient NMT data collection 
> would go a very long way. Many modern observability tools, especially 
> those used in container deployments, operate by reading data from within 
> the jvm process, and relaying it to a storage system (Prometheus may be 
> the most ubiquitous example).
> 
> For my use-case, I’d love to have a simple API I could invoke from java 
> code to access structured native-memory-tracking data, similar in a way 
> to MemoryPoolMXBean for heap pools (although JMX isn’t necessary for me, 
> it aligns with other observability APIs in the JDK). Additionally, I’d 
> like to provide JFR events to periodically record native memory tracking 
> metadata when enabled for better out-of-the-box experience with JMC. 
> I’ve begun investigating some options for JDK-8182634 
> <https://bugs.openjdk.org/browse/JDK-8182634>, but would appreciate 
> feedback before I propose any sort of code change.
> 
> Thank you all for beginning this discussion, I’m eager to see the ways 
> the JDK continues to improve upon observability features, and do what 
> small part I can to help!
> 
> Carter Kozak