Native Memory Tracking Bug

[Jacob Schlather]

We're currently in the process of upgrading our Java applications from Java
8 to Java 11. After deploying some of our production applications with Java
11, we began to see the resident memory size grow without bound until our
orchestrator killed the applications for excessive memory usage. We've
started to debug this issue, but noticed that the NMT output appears to be
incorrect. In particular the Compiler section is displaying

Compiler (reserved=4528KB +1010KB, committed=4528KB +1010KB)
                (malloc=4896KB +1206KB #4132 +508)
                (arena=18014398509481617KB -196 #5)

Obviously the arena value here is quite wrong and there's no way the
reserved memory can be less than the malloc memory. Further there's
a 276305KB gap in the RSS size reported by our metrics and the amount of
memory NMT reports as committed. Here's our JVM args and JDK version, I've
additionally attached the full output of the NMT detailed diff.

Running java11 with JVM arguments: -Djava.net.preferIPv4Stack=true
-Xms6144m -Xmx6g -Xss256k -XX:MetaspaceSize=128m
-XX:MaxMetaspaceSize=256m -XX:CompressedClassSpaceSize=128m
-XX:+UseG1GC -XX:MaxGCPauseMillis=350 -XX:+UnlockExperimentalVMOptions
-XX:G1NewSizePercent=20 -XX:G1MaxNewSizePercent=45
-XX:ParallelGCThreads=8 -XX:ConcGCThreads=6
-XX:InitiatingHeapOccupancyPercent=35 -XX:+PerfDisableSharedMem
-XX:-UseBiasedLocking -XX:G1HeapRegionSize=4m
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=../logs
-Djava.io.tmpdir=REDACTED -XX:+ExitOnOutOfMemoryError
-XX:-OmitStackTraceInFastThrow
-javaagent:/usr/local/appoptics-6.5.1/appoptics-agent.jar=config=appoptics-agent.json
-XX:-PreferContainerQuotaForCPUCount -XX:NativeMemoryTracking=detail
-jar REDACTED
openjdk version "11.0.5" 2019-10-15
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.5+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.5+10, mixed mode)