[ZGC] [aarch64] Unable to allocate heap for certain Linux kernel configurations

[Stefan Karlsson]

Hi Christoph,

On 2020-08-28 10:40, Christoph Göttschkes wrote:
> Hi,
> 
> I didn't file a bug yet, because I first wanted to get some feedback on 
> this topic. Maybe this was a design choice and is not a bug.
> 
> On one of our test devices (a rock64 [1] running Debian GNU/Linux) I am 
> unable to use ZGC due to the heap allocation scheme used by the JVM and 
> the Linux kernel configuration for the device.
> 
> $ uname -a
> Linux rock64-aarch64-1 4.4.190-1233-rockchip-ayufan-gd3f1be0ed310 #1 SMP 
> Wed Aug 28 08:59:34 UTC 2019 aarch64 GNU/Linux
> $ getconf PAGE_SIZE
> 4096
> $ zcat /proc/config.gz |grep PGTABLE_LEVELS
> CONFIG_PGTABLE_LEVELS=3
> 
> The problem is, the Linux kernel is configured to only use 3 pagetable 
> levels, not 4. According to the kernel documentation [2], using 3 
> pagetable levels and a page size of 4k, the highest virtual address 
> available to userspace programs is 0x007f_ffff_ffff. If 4 pagetable 
> levels are used with 4k pages, the address range is bigger and goes up 
> to 0xffff_ffff_ffff for userspace programs.
> 
> Because of the memory allocation scheme used by ZGC [2], the JVM is not 
> able to allocate the heap if ZGC is used on an aarch64 device, if the 
> Linux kernel is configured to only use 3 pagetable levels with a page 
> size of 4k. According to the documentation, the heap layout is only 
> determined by the size of the Java heap, not by the capabilities of the 
> device. On the rock64, which has 4GB of RAM, the JVM only tries to 
> allocate the heap on addresses > 0x0400_0000_0000, which is well beyond 
> the maximum address for userspace programs. Because of this, the JVM 
> terminates during startup, because it is unable to allocate the heap, 
> even though there are still more than 3 GB of RAM available.
> 
> $ free -m
>           total   used   free  shared  buff/cache  available
> Mem:      3986     56   3714      18         215       3880
> Swap:        0      0      0
> 
> I understand that using 3 pagetable levels is not a very common case. In 
> fact, on the rock64, the Linux kernel is very well capable of using 4 
> pagetable levels. We have an Arch Linux ARM based installation, and in 
> this distribution, the Linux kernel is configured to use 4 pagetable 
> levels. This is the first time I encounter this, so chances are very 
> high that none of our other aarch64 test devices are using 3 pagetable 
> levels.
> 
> I am bringing this topic up, because I wasn't aware of this limitation 
> and couldn't find a discussion or documentation about it. Was the design 
> decision for using such high addresses made while being aware of this 
> possible limitation? Might it be feasible to make an auto detect 
> mechanism, for such cases? Or maybe lower the address range used for 
> small heaps (maybe a heap below 8GB could use a different address 
> range)? I don't really know if there are any aarch64 devices which are 
> only capable of using 3 pagetable levels around and this is a real world 
> problem or not. But this limitation should at least be documented, and 
> maybe a command line option should be introduced for these very specific 
> Linux kernel configurations.

I think we hit a very similar problem during some internal testing on 
one machine. I have a patch to workaround that problem:

https://cr.openjdk.java.net/~stefank/prototype/zaarch-va/webrev.01/

Unfortunately, this patch only solves the problem on a very specific 
setup, and I don't think it covers your use-case. Hopefully, someone 
with enough AArch64 machine config knowledge would be able to extend 
this patch to also cover all possible combinations.

I think creating a bug report would be a good start. Do you have an 
openjdk user name? If not I can create a bug report.

Thanks,
StefanK


> 
> -- Christoph
> 
> [1] https://www.pine64.org/devices/single-board-computers/rock64/
> [2] 
> https://hg.openjdk.java.net/jdk/jdk/file/3123bebd1eff/src/hotspot/cpu/aarch64/gc/z/zGlobals_aarch64.cpp 
> 
>