Large number of VMAs for large ZGC heap

[Thomas Stüfe]

Hi ZGC experts,

I see a strangeness with one of our customers running JDK 17 with ZGC, THP
enabled (always), and a large heap of 4.6TB.

The number of VMAs exceeds 20 million. I try to understand whether that is
normal or pathological.

Looking at maps, I see millions of adjacent VMAs that point into the heap
to different offsets:

```
15fc5f600000-15fc5f800000 rw-s 24630400000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
15fc5f800000-15fc5fa00000 rw-s 2504e600000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
15fc5fa00000-15fc5fc00000 rw-s 25330000000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
15fc5fc00000-15fc5fe00000 rw-s 26324200000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
15fc5fe00000-15fc60000000 rw-s 26f03a00000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
```

The different offsets prevent these mappings from being folded.

The number of mappings surpasses what would be needed to map the heap.
Almost all are 2MB mappings:

Total number of mappings: 18634289
Number of 2MB mappings:        18529201
Per color: 6211420 / 6211429 / 6211439

The total address space covered by these 2MB mappings is 38TB. Taking into
account the triple-mapping, we still map about 12TB per color. That far
exceeds the necessary room for a 4.6TB heap.

Examining the mappings, I see that many offsets into the heap are mapped to
multiple points, even discounting the triple mapping. For example, offset
105fe800000 is mapped six times per color, for a total of 12 times:

13438de00000-13438e000000 rw-s 105fe800000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
15bf79400000-15bf79600000 rw-s 105fe800000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
165022800000-165022a00000 rw-s 105fe800000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
16fdad200000-16fdad400000 rw-s 105fe800000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
17b1b9600000-17b1b9800000 rw-s 105fe800000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
1d9860000000-1d9860200000 rw-s 105fe800000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)

23438de00000-23438e000000 rw-s 105fe800000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
25bf79400000-25bf79600000 rw-s 105fe800000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
265022800000-265022a00000 rw-s 105fe800000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
26fdad200000-26fdad400000 rw-s 105fe800000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
27b1b9600000-27b1b9800000 rw-s 105fe800000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
2d9860000000-2d9860200000 rw-s 105fe800000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)

43438de00000-43438e000000 rw-s 105fe800000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
45bf79400000-45bf79600000 rw-s 105fe800000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
465022800000-465022a00000 rw-s 105fe800000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
46fdad200000-46fdad400000 rw-s 105fe800000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
47b1b9600000-47b1b9800000 rw-s 105fe800000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)
4d9860000000-4d9860200000 rw-s 105fe800000 00:0f 373323680
/memfd:java_heap.hugetlb (deleted)

The ZGC Page table contains close to a million ZGC pages and looks okay for
a heap of that size:
Small: 739175
Medium: 10160
Large:   65495
               -------
                814830

My question: is such a high number of mappings for ZGC normal?

Thank you for your time,

Cheers, Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/zgc-dev/attachments/20230425/1b213b0e/attachment-0001.htm>