[RFC 8285277] - How should the JVM handle container memory limits

[Ioi Lam]

I would like to have a discussion on how (or whether) the JVM should 
handle container memory limits -- see JDK-8285277 [1]


The JVM may be terminated by the Linux OOM killer if it tries to use 
more memory than the memory limit specified for a container. JDK-8284900 
[2] tries to avoid OOM by checking InitialHeapSize against the memory 
limit. However, this is incomplete because the JVM can use memory in 
other ways:

- If -Xmx is larger than -Xms, the heap may expand
- malloc memory
- code cache
- thread stacks

We have several choices:

(a) Following the direction of JDK-8284900, check for the initial memory 
usage  of other types of memory as well. The problem with this is that 
checking the "initial" usage some type of memory usage (malloc or stack) 
is difficult or impossible.

(b) Avoid committing more memory when the total memory usage is close to 
the limit. E.g., avoid expanding the Java heap, or return NULL from 
os::malloc(). The problem is that some operations in the VM cannot 
handle malloc failure and will terminate the VM anyway. Also, trying to 
obey the memory limit with concurrent allocator/deallocator threads is 
probably difficult or impossible.

(c) Do not consider the memory limit and let the VM be killed by the OOM 
killer. This may be what the user wants -- the user knows that the app 
will normally run within the memory limit, so if the heap expands too 
much the app probably has a memory leak and should be killed, to be 
automatically restarted by an external mechanism (such as Kubernetes).


[1] https://bugs.openjdk.java.net/browse/JDK-8285277
[2] https://bugs.openjdk.java.net/browse/JDK-8284900