[ZGC] [aarch64] Unable to allocate heap for certain Linux kernel configurations

[Christoph Göttschkes]

Hi,

I didn't file a bug yet, because I first wanted to get some feedback on 
this topic. Maybe this was a design choice and is not a bug.

On one of our test devices (a rock64 [1] running Debian GNU/Linux) I am 
unable to use ZGC due to the heap allocation scheme used by the JVM and 
the Linux kernel configuration for the device.

$ uname -a
Linux rock64-aarch64-1 4.4.190-1233-rockchip-ayufan-gd3f1be0ed310 #1 SMP 
Wed Aug 28 08:59:34 UTC 2019 aarch64 GNU/Linux
$ getconf PAGE_SIZE
4096
$ zcat /proc/config.gz |grep PGTABLE_LEVELS
CONFIG_PGTABLE_LEVELS=3

The problem is, the Linux kernel is configured to only use 3 pagetable 
levels, not 4. According to the kernel documentation [2], using 3 
pagetable levels and a page size of 4k, the highest virtual address 
available to userspace programs is 0x007f_ffff_ffff. If 4 pagetable 
levels are used with 4k pages, the address range is bigger and goes up 
to 0xffff_ffff_ffff for userspace programs.

Because of the memory allocation scheme used by ZGC [2], the JVM is not 
able to allocate the heap if ZGC is used on an aarch64 device, if the 
Linux kernel is configured to only use 3 pagetable levels with a page 
size of 4k. According to the documentation, the heap layout is only 
determined by the size of the Java heap, not by the capabilities of the 
device. On the rock64, which has 4GB of RAM, the JVM only tries to 
allocate the heap on addresses > 0x0400_0000_0000, which is well beyond 
the maximum address for userspace programs. Because of this, the JVM 
terminates during startup, because it is unable to allocate the heap, 
even though there are still more than 3 GB of RAM available.

$ free -m
          total   used   free  shared  buff/cache  available
Mem:      3986     56   3714      18         215       3880
Swap:        0      0      0

I understand that using 3 pagetable levels is not a very common case. In 
fact, on the rock64, the Linux kernel is very well capable of using 4 
pagetable levels. We have an Arch Linux ARM based installation, and in 
this distribution, the Linux kernel is configured to use 4 pagetable 
levels. This is the first time I encounter this, so chances are very 
high that none of our other aarch64 test devices are using 3 pagetable 
levels.

I am bringing this topic up, because I wasn't aware of this limitation 
and couldn't find a discussion or documentation about it. Was the design 
decision for using such high addresses made while being aware of this 
possible limitation? Might it be feasible to make an auto detect 
mechanism, for such cases? Or maybe lower the address range used for 
small heaps (maybe a heap below 8GB could use a different address 
range)? I don't really know if there are any aarch64 devices which are 
only capable of using 3 pagetable levels around and this is a real world 
problem or not. But this limitation should at least be documented, and 
maybe a command line option should be introduced for these very specific 
Linux kernel configurations.

-- Christoph

[1] https://www.pine64.org/devices/single-board-computers/rock64/
[2] 
https://hg.openjdk.java.net/jdk/jdk/file/3123bebd1eff/src/hotspot/cpu/aarch64/gc/z/zGlobals_aarch64.cpp