jcmd VM.native_memory extremely large numbers when using ZGC

[Marçal Perapoch Amadó]

Hello!
First of all, congratulations on all the hard work with ZGC!

TLDR: Running a simple java main with generational ZGC, and NMT reports
221GB of reserved memory on a 32GB machine.

*Context*: at my current company, we're keen on switching from G1GC to ZGC
due to its ability to maintain very low pause times. Our problem in
particular, is that when we scale up our application, the new nodes get so
much traffic in that little time that even the node is technically ready to
accept new traffic, the amount of new allocations end up adding a lot of
pressure to g1 and that translates to multiple over the second pauses. So
we decided to give ZGC a try and although the numbers for those pauses were
looking amazing, our canary nodes were suddenly killed by OOM.
I've read about the ZGC multi-mapping technique and how that can trick the
Linux kernel. I found particularly useful this topic from this same mailing
list: https://mail.openjdk.org/pipermail/zgc-dev/2018-November/000511.html
and also read about using the -XX:+UseLargePages flag. Even saw a mailing
topic about kubernetes and containers having issues with ZCG here:
https://mail.openjdk.org/pipermail/zgc-dev/2023-August/001259.html.
However, despite this research, I have not been able to find a solution to
the issue. So I decided to reproduce the problem locally for further
investigation. Although my local environment is quite different from our
live setup, I encountered the same high reserved memory behavior.

I created a very simple java application (just a Main that loops forever
waiting for a number from the console and performs some allocations based
on that, but I don't think that matters that much).
I run my application with the following JVM args:
-XX:+UseZGC
-XX:+ZGenerational
-Xms12g
-Xmx12g
-XX:NativeMemoryTracking=summary
-Xlog:gc*:gc.log

And that produces the following report on my MacBook Pro M2, 32GB.

*Native Memory Tracking*:
(Omitting categories weighting less than 1GB)

Total: reserved=221GB, committed=12GB
       malloc: 0GB #38256
       mmap:   reserved=221GB, committed=12GB

-                 Java Heap (reserved=192GB, committed=12GB)
                            (mmap: reserved=192GB, committed=12GB, at peak)

-                     Class (reserved=1GB, committed=0GB)
                            (classes #2376)
                            (  instance classes #2142, array classes #234)
                            (mmap: reserved=1GB, committed=0GB, at peak)
                            (  Metadata:   )
                            (    reserved=0GB, committed=0GB)
                            (    used=0GB)
                            (    waste=0GB =0.79%)
                            (  Class space:)
                            (    reserved=1GB, committed=0GB)
                            (    used=0GB)
                            (    waste=0GB =7.49%)

-                        GC (reserved=16GB, committed=0GB)
                            (mmap: reserved=16GB, committed=0GB, at peak)

-                   Unknown (reserved=12GB, committed=0GB)
                            (mmap: reserved=12GB, committed=0GB, peak=0GB)

As you can see, it is reporting a total reserved of 221GB, which I find
very confusing. I understand it is related to the muli-mapping technique,
but my question is, how can I be sure how much memory my app is using if
even with jcmd I get reports like this one?

Also, launching the same application with G1, reports Total: reserved=14GB,
committed=12GB.

Sorry if that has already been reported/answered, I really tried to inform
myself before wasting your time, but I do have the impression that I am
missing something here.

Could you please provide any insights or suggestions on what might be
happening, or how we could mitigate this issue?
If not jcmd, which tool/command would you recommend to measure the memory
consumption? We’d greatly appreciate your advice on how to move forward.

Thank you very much for your time and help!


Marçal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/zgc-dev/attachments/20241028/652bcb15/attachment-0001.htm>