Resident/Shared memory size is showing 3 times the given heap size

[Per Liden]

Hi,

On 9/20/19 3:54 AM, Sundara Mohan M wrote:
> Hi,
>     We are running our server with ZGC and seeing the resident memory size
> is approximately 3 times the given heap size. I am a newbie still trying to
> understand basic concepts. Can someone help me understand this better
> 
> 1. Is this expected?

Yes

> 2. At high level i know ZGC is having multiple view of memory region. Can
> you explain how this can happen?

Exactly. RSS (Resident set size) basically just says how much memory the 
process has mapped at the moment. It doesn't take into account that some 
of these mappings might be backed by the same memory, or might be are 
shared with other process, etc. A more interesting number, to get a 
better view of how much memory a process is actually using is PSS 
(Proportional set size). There are various tools that display PSS (smem, 
procrank, ps_mem.py, etc), but the raw data is available in 
/proc/<pid>/smaps_rollup.

For example, a JVM running ZGC with a 4G heap looks like this:

$ cat /proc/4509/smaps_rollup
00400000-7ffd472e2000 ---p 00000000 00:00 0 
[rollup]
Rss:            12843604 kB
Pss:             4451825 kB
Shared_Clean:       3232 kB
Shared_Dirty:   12582912 kB
Private_Clean:     14380 kB
Private_Dirty:    243080 kB
Referenced:     12843604 kB
Anonymous:        243048 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB

Here the multi-mapping causes the RSS to be inflated by 3x, while PSS 
shows a more accurate number reflecting the fact that the three 
different heap mappings are backed by the same memory.

> 3. Also i see something like this in /proc/<pid>/maps
> 
> ...
> 13fff9a00000-13fff9c00000 rw-s 5f8e00000 00:12 1488924473
>   /mnt/tmpfs/java_heap.10087 (deleted)
> 13fff9c00000-13fff9e00000 rw-s 5fb000000 00:12 1488924473
>   /mnt/tmpfs/java_heap.10087 (deleted)
> 13fff9e00000-13fffa000000 rw-s 603200000 00:12 1488924473
>   /mnt/tmpfs/java_heap.10087 (deleted)
> 13fffa000000-13fffa200000 rw-s 612c00000 00:12 1488924473
>   /mnt/tmpfs/java_heap.10087 (deleted)
> 13fffa200000-13fffa400000 rw-s 61a800000 00:12 1488924473
>   /mnt/tmpfs/java_heap.10087 (deleted)
> 13fffa400000-13fffa600000 rw-s 326000000 00:12 1488924473
>   /mnt/tmpfs/java_heap.10087 (deleted)
> 13fffa600000-13fffa800000 rw-s 5d3600000 00:12 1488924473
>   /mnt/tmpfs/java_heap.10087 (deleted)
> ...
> Why is this still in deleted state?

The "(deleted)" state just means that the file has been unlinked. I.e. 
the file is still in use, but the directory entry 
/mnt/tmpfs/java_heap.10087 has been deleted (as it should be).

> 4. Trying to get heap dump with following command
>   jmap -heap <pid> (tried with same user as well root but it is not printing
> heap)
> Is there something changed regarding dumping heap with ZGC?

IIRC, jmap -heap was removed in JDK 9, use jcmd instead:

$ jcmd <pid> GC.heap_dump <filename>

cheers,
Per

> 
> TIA
> Sundar
>