Native Memory Tracking Bug

[Zhengyu Gu]

Hi Jacob,

It looks like JDK-8204128 [1] strikes again. It would be very helpful if 
you can provide a reproducer.

Thanks,

-Zhengyu


[1] https://bugs.openjdk.java.net/browse/JDK-8204128


On 11/13/19 9:56 PM, Jacob Schlather wrote:
> We're currently in the process of upgrading our Java applications from Java
> 8 to Java 11. After deploying some of our production applications with Java
> 11, we began to see the resident memory size grow without bound until our
> orchestrator killed the applications for excessive memory usage. We've
> started to debug this issue, but noticed that the NMT output appears to be
> incorrect. In particular the Compiler section is displaying
> 
> Compiler (reserved=4528KB +1010KB, committed=4528KB +1010KB)
>                  (malloc=4896KB +1206KB #4132 +508)
>                  (arena=18014398509481617KB -196 #5)
> 
> Obviously the arena value here is quite wrong and there's no way the
> reserved memory can be less than the malloc memory. Further there's
> a 276305KB gap in the RSS size reported by our metrics and the amount of
> memory NMT reports as committed. Here's our JVM args and JDK version, I've
> additionally attached the full output of the NMT detailed diff.
> 
> Running java11 with JVM arguments: -Djava.net.preferIPv4Stack=true
> -Xms6144m -Xmx6g -Xss256k -XX:MetaspaceSize=128m
> -XX:MaxMetaspaceSize=256m -XX:CompressedClassSpaceSize=128m
> -XX:+UseG1GC -XX:MaxGCPauseMillis=350 -XX:+UnlockExperimentalVMOptions
> -XX:G1NewSizePercent=20 -XX:G1MaxNewSizePercent=45
> -XX:ParallelGCThreads=8 -XX:ConcGCThreads=6
> -XX:InitiatingHeapOccupancyPercent=35 -XX:+PerfDisableSharedMem
> -XX:-UseBiasedLocking -XX:G1HeapRegionSize=4m
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=../logs
> -Djava.io.tmpdir=REDACTED -XX:+ExitOnOutOfMemoryError
> -XX:-OmitStackTraceInFastThrow
> -javaagent:/usr/local/appoptics-6.5.1/appoptics-agent.jar=config=appoptics-agent.json
> -XX:-PreferContainerQuotaForCPUCount -XX:NativeMemoryTracking=detail
> -jar REDACTED
> openjdk version "11.0.5" 2019-10-15
> OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.5+10)
> OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.5+10, mixed mode)
>