ZGC: Failed to truncate backing file (Permission denied) at startup after 14.0.2 to 15.0.1 upgrade
per.liden at oracle.com
Mon Nov 23 14:11:30 UTC 2020
On 11/20/20 4:21 PM, David Tavoularis wrote:
> I found a possible regression in customer production environment linked
> to JDK-8245203 "ZGC: Don't track size in ZPhysicalMemoryBacking" or
> JDK15 "Fixed support for transparent huge pages".
> After upgrade from jdk-14.0.2 to jdk-15.0.1, the JVM (using ZGC) is
> crashing at startup with the error message "Failed to truncate backing
> file (Permission denied)"
> This error message was introduced in changelist
> Cf source code :
> The issue was seen in Prod, not in Test, nor in labs, I suspect that
> Heap Backing Filesystem is not correctly set to tmpfs in Prod or a
> permission issue on a directory/file owned by root.
> $ /opt/3rd_party/jdk_installed/jdk-15.0.1/bin/java -version
> openjdk version "15.0.1" 2020-10-20
> OpenJDK Runtime Environment (build 15.0.1+9-18)
> OpenJDK 64-Bit Server VM (build 15.0.1+9-18, mixed mode, sharing)
> $ /opt/3rd_party/jdk_installed/jdk-15.0.1/bin/java -XX:+UseZGC -version
> [0.009s][error][gc] Failed to truncate backing file (Permission denied)
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> $ /opt/mycom/3rd_party/jdk_installed/jdk-14.0.2/bin/java
> -XX:+UnlockExperimentalVMOptions -XX:+UseZGC -version
> openjdk version "14.0.2" 2020-07-14
> OpenJDK Runtime Environment (build 14.0.2+12-46)
> OpenJDK 64-Bit Server VM (build 14.0.2+12-46, mixed mode)
> Should I fill a bug at https://bugreport.java.com/bugreport/ ?
> What additional information should I provide ?
> Any workaround ?
It's hard to tell exactly why this fails. One difference between JDK 14
and 15 is that the backing file will be truncated to the max heap size
up-front when the JVM starts (but it will not be backed by any memory
until we actually start to expand the heap), instead of gradually grown
as the heap expands. So, one theory is that you could have had this
problem even with 14, but the heap might not have grown large enough for
the problem to be exposed. To test this, run with JDK 14 and use
"-Xms270g -Xmx270g". This will force the backing file to be truncated
up-front, similar to what happens in JDK 15.
Also running with "-Xlog:gc*=debug" will print a few more things that
might be interesting.
The fact that ftruncate returns EACCESS suggests that there's some kind
of environment problem that is blocking the file from growing. In the
production system, is this process running in some constrained
More information about the zgc-dev