ZGC: Failed to truncate backing file (Permission denied) at startup after 14.0.2 to 15.0.1 upgrade

Fri Nov 20 15:21:31 UTC 2020

Hi,

I found a possible regression in customer production environment linked to  
JDK-8245203 "ZGC: Don't track size in ZPhysicalMemoryBacking" or JDK15  
"Fixed support for transparent huge pages".
After upgrade from jdk-14.0.2 to jdk-15.0.1, the JVM (using ZGC) is  
crashing at startup with the error message "Failed to truncate backing  
file (Permission denied)"

This error message was introduced in changelist  
http://hg.openjdk.java.net/jdk-updates/jdk15u/rev/556d5070c458
Cf source code :  
http://hg.openjdk.java.net/jdk-updates/jdk15u/file/556d5070c458/src/hotspot/os/linux/gc/z/zPhysicalMemoryBacking_linux.cpp

The issue was seen in Prod, not in Test, nor in labs, I suspect that  Heap  
Backing Filesystem  is not correctly set to tmpfs in Prod or a permission  
issue on a directory/file owned by root.

$ /opt/3rd_party/jdk_installed/jdk-15.0.1/bin/java -version
openjdk version "15.0.1" 2020-10-20
OpenJDK Runtime Environment (build 15.0.1+9-18)
OpenJDK 64-Bit Server VM (build 15.0.1+9-18, mixed mode, sharing)

$ /opt/3rd_party/jdk_installed/jdk-15.0.1/bin/java -XX:+UseZGC -version
[0.009s][error][gc] Failed to truncate backing file (Permission denied)
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

$ /opt/mycom/3rd_party/jdk_installed/jdk-14.0.2/bin/java  
-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -version
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-46)
OpenJDK 64-Bit Server VM (build 14.0.2+12-46, mixed mode)

Should I fill a bug at https://bugreport.java.com/bugreport/ ?
What additional information should I provide ?
Any workaround ?

Best Regards
-- 
David

[Additional details]

1. Not working in Prod with 15.0.1 :
openjdk version "15.0.1" 2020-10-20
OpenJDK Runtime Environment (build 15.0.1+9-18)
OpenJDK 64-Bit Server VM (build 15.0.1+9-18, mixed mode, sharing)

[2020-11-20T16:56:31.952+0530][debug][gc,heap] Minimum heap 96636764160  
Initial heap 96636764160 Maximum heap 289910292480
[2020-11-20T16:56:31.953+0530][info ][gc,init] Initializing The Z Garbage  
Collector
[2020-11-20T16:56:31.953+0530][info ][gc,init] Version: 15.0.1+9-18  
(release)
[2020-11-20T16:56:31.953+0530][info ][gc,init] NUMA Support: Disabled
[2020-11-20T16:56:31.953+0530][info ][gc,init] CPUs: 70 total, 45 available
[2020-11-20T16:56:31.953+0530][info ][gc,init] Memory: 307200M
[2020-11-20T16:56:31.953+0530][info ][gc,init] Large Page Support: Disabled
[2020-11-20T16:56:31.953+0530][info ][gc,init] Workers: 8 parallel, 8  
concurrent
[2020-11-20T16:56:31.956+0530][info ][gc,init] Address Space Type:  
Contiguous/Unrestricted/Complete
[2020-11-20T16:56:31.956+0530][info ][gc,init] Address Space Size:  
4423680M x 3 = 13271040M
[2020-11-20T16:56:31.956+0530][info ][gc,init] Heap Backing File:  
/memfd:java_heap
[2020-11-20T16:56:31.956+0530][error][gc ] Failed to truncate backing file  
(Permission denied)
[2020-11-20T16:56:31.992+0530][info ][gc,init] Runtime Workers: 8 parallel
[2020-11-20T16:56:31.994+0530][info ][gc ] Using The Z Garbage Collector

$ /opt/mycom/3rd_party/jdk_installed/jdk-15.0.1/bin/java -Xmx512m  
-XX:+PrintFlagsFinal -version | egrep  
"UseHugeTLBFS|UseLargePages|UseSHM|UseTransparentHugePages"
bool UseHugeTLBFS = false {product} {default}
bool UseLargePages = false {pd product} {default}
bool UseLargePagesInMetaspace = false {product} {default}
bool UseLargePagesIndividualAllocation = false {pd product} {default}
bool UseSHM = false {product} {default}
bool UseTransparentHugePages = false {product} {default}

2. Working fine in Prod with 14.0.2 :
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-46)
OpenJDK 64-Bit Server VM (build 14.0.2+12-46, mixed mode, sharing)

[2020-11-20T17:10:57.378+0530][debug][gc,heap] Minimum heap 96636764160  
Initial heap 96636764160 Maximum heap 289910292480
[2020-11-20T17:10:57.378+0530][info ][gc,init] Initializing The Z Garbage  
Collector
[2020-11-20T17:10:57.378+0530][info ][gc,init] Version: 14.0.2+12-46  
(release)
[2020-11-20T17:10:57.378+0530][info ][gc,init] NUMA Support: Disabled
[2020-11-20T17:10:57.378+0530][info ][gc,init] CPUs: 70 total, 45 available
[2020-11-20T17:10:57.378+0530][info ][gc,init] Memory: 307200M
[2020-11-20T17:10:57.378+0530][info ][gc,init] Large Page Support: Disabled
[2020-11-20T17:10:57.378+0530][info ][gc,init] Medium Page Size: 32M
[2020-11-20T17:10:57.378+0530][info ][gc,init] Workers: 8 parallel, 8  
concurrent
[2020-11-20T17:10:57.381+0530][info ][gc,init] Address Space Type:  
Contiguous/Unrestricted/Complete
[2020-11-20T17:10:57.381+0530][info ][gc,init] Address Space Size:  
4423680M x 3 = 13271040M
[2020-11-20T17:10:57.381+0530][info ][gc,init] Heap backed by file:  
/memfd:java_heap
[2020-11-20T17:10:57.381+0530][info ][gc,init] Min Capacity: 92160M
[2020-11-20T17:10:57.381+0530][info ][gc,init] Initial Capacity: 92160M
[2020-11-20T17:10:57.381+0530][info ][gc,init] Max Capacity: 276480M
[2020-11-20T17:10:57.381+0530][info ][gc,init] Max Reserve: 48M
[2020-11-20T17:10:57.381+0530][info ][gc,init] Pre-touch: Disabled
[2020-11-20T17:10:57.381+0530][info ][gc,init] Available space on backing  
filesystem: N/A
[2020-11-20T17:11:21.585+0530][info ][gc,init] Uncommit: Enabled, Delay:  
300s
[2020-11-20T17:11:21.618+0530][info ][gc,init] Runtime Workers: 8 parallel
[2020-11-20T17:11:21.620+0530][info ][gc ] Using The Z Garbage Collector

Working fine in Lab with 15.0.1 :
$ /opt/3rd_party/jdk_installed/jdk-15.0.1/bin/java -Xms90g -Xmx270g  
-XX:-UseNUMA -XX:+UseZGC -XX:ParallelGCThreads=8  
-Xlog:gc*,gc+heap=debug,gc+age=trace,safepoint:stdout:time,level,tags  
HelloWorld.java
[2020-11-20T14:16:08.635+0000][debug][gc,heap] Minimum heap 96636764160  
Initial heap 96636764160 Maximum heap 289910292480
[2020-11-20T14:16:08.635+0000][info ][gc,init] Initializing The Z Garbage  
Collector
[2020-11-20T14:16:08.635+0000][info ][gc,init] Version: 15.0.1+9-18  
(release)
[2020-11-20T14:16:08.635+0000][info ][gc,init] NUMA Support: Disabled
[2020-11-20T14:16:08.635+0000][info ][gc,init] CPUs: 64 total, 64 available
[2020-11-20T14:16:08.635+0000][info ][gc,init] Memory: 257775M
[2020-11-20T14:16:08.635+0000][info ][gc,init] Large Page Support: Disabled
[2020-11-20T14:16:08.635+0000][info ][gc,init] Workers: 8 parallel, 8  
concurrent
[2020-11-20T14:16:08.636+0000][info ][gc,init] Address Space Type:  
Contiguous/Unrestricted/Complete
[2020-11-20T14:16:08.636+0000][info ][gc,init] Address Space Size:  
4423680M x 3 = 13271040M
[2020-11-20T14:16:08.636+0000][info ][gc,init] Heap Backing File:  
/memfd:java_heap
[2020-11-20T14:16:08.637+0000][info ][gc,init] Heap Backing Filesystem:  
tmpfs (0x1021994)
[2020-11-20T14:16:08.637+0000][info ][gc,init] Min Capacity: 92160M
[2020-11-20T14:16:08.637+0000][info ][gc,init] Initial Capacity: 92160M
[2020-11-20T14:16:08.637+0000][info ][gc,init] Max Capacity: 276480M
[2020-11-20T14:16:08.637+0000][info ][gc,init] Max Reserve: 48M
[2020-11-20T14:16:08.637+0000][info ][gc,init] Medium Page Size: 32M
[2020-11-20T14:16:08.637+0000][info ][gc,init] Pre-touch: Disabled
[2020-11-20T14:16:08.637+0000][info ][gc,init] Available space on backing  
filesystem: N/A
[2020-11-20T14:16:08.637+0000][info ][gc,init] Uncommit: Enabled
[2020-11-20T14:16:08.637+0000][info ][gc,init] Uncommit Delay: 300s
[2020-11-20T14:16:24.652+0000][info ][gc,init] Runtime Workers: 8 parallel
[2020-11-20T14:16:24.653+0000][info ][gc ] Using The Z Garbage Collector