[RFC 8285277] - How should the JVM handle container memory limits

[Ioi Lam]

On 4/21/2022 2:49 AM, Severin Gehwolf wrote:
> On Wed, 2022-04-20 at 11:09 -0700, Ioi Lam wrote:
>> I would like to have a discussion on how (or whether) the JVM should
>> handle container memory limits -- see JDK-8285277 [1]
> It does handle container memory limits. os::physical_memory() returns
> the container limit instead of the host value if a container limit is
> in place. That was one big aspect of JDK-8146115 when it was done in
> JDK 10 timeframe. See:
>
> https://github.com/openjdk/jdk/blob/fa04d1f832ff201248f935939fa255988053a1d0/src/hotspot/os/linux/os_linux.cpp#L217..L232
>
> What am I missing?

Yes, the container support code does expose the memory limit through 
APIs like os::physical_memory(), and we use the same ergonomic logic as 
if we are running on a physical machine with a small amount of RAM.

However, I am wondering if we need to reconsider our logic if the 
"physical memory" is unusually small.

Currently, the ergonomic code will set the heap size to a "sensible" 
fraction of os::physical_memory(). This usually works fine on modern day 
machines with lots of RAM and swap. Even the below-$100 windows laptops 
have at least 4GB.

On my 64GB box, max heap is set to 16GB by default. This should be 
enough for most apps, and will practically never be killed by OOM unless 
the system is really overloaded.

================
$ java -XX:+PrintFlagsFinal --version | egrep 
'((MaxHeapSize)|(InitialHeapSize))'
    size_t InitialHeapSize                          = 
1056964608                                {product} {ergonomic}
    size_t MaxHeapSize                              = 
16869490688                               {product} {ergonomic}
================

But inside a container that has only a 128MB limit (which might be 
"typical" for micro-services??)

================
$ java -XX:+PrintFlagsFinal --version | egrep 
'((MaxHeapSize)|(InitialHeapSize))'
    size_t InitialHeapSize                          = 
8388608                                   {product} {ergonomic}
    size_t MaxHeapSize                              = 
67108864                                  {product} {ergonomic}
================

The default MaxHeapSize seems wasteful. We are paying for 128MB but 
really just use 64MB of heap. I am able to start with a 108MB heap size 
without getting killed.

======================
$ java -XshowSettings:system -Xms108m -XX:+AlwaysPreTouch --version
Operating System Metrics:
     Provider: cgroupv1
     ....
     Memory Limit: 128.00M
     Memory Soft Limit: Unlimited
     Memory & Swap Limit: 128.00M

java 19-internal 2022-09-20
Java(TM) SE Runtime Environment (build 19-internal-adhoc.iklam)
Java HotSpot(TM) 64-Bit Server VM (build 19-internal-adhoc.iklam, mixed 
mode, sharing)
==========================

The problem is

- If we want to efficiently use all the memory we paid for in a 
container, we need to tune the VM parameters manually

- But tuning is very difficult and error prone. With the 100mb heap 
setting, my app could easily be killed if it uses too much memory in 
malloc, native buffers, or even the JIT. Currently we don't fail early 
(no check of -Xms, -Xmx, etc, against available memory), and don't fail 
predictably.


So, I am wondering if there's anything we can do to make Java both safe 
and efficient when running under an small memory budget.

Thanks
- Ioi


>> The JVM may be terminated by the Linux OOM killer if it tries to use
>> more memory than the memory limit specified for a container. JDK-8284900
>> [2] tries to avoid OOM by checking InitialHeapSize against the memory
>> limit. However, this is incomplete because the JVM can use memory in
>> other ways:
>>
>> - If -Xmx is larger than -Xms, the heap may expand
>> - malloc memory
>> - code cache
>> - thread stacks
> JDK-8284900 is somewhat a special case which, IIUIC, tries to avoid
> user-errors detectable at JVM startup. For example setting -Xms to a
> value >= physical memory (including swap). In this case the JVM would
> be at risk to get killed by the OOM killer and doing this check on
> start-up would catch it before it comes to it.
>
> On the other hand, I don't see any handling of such a case on a
> *physical* machine. Consider a machine with 8GB of RAM and somebody
> spawning a JVM with -Xms8g on it. It'll happily oblige? In one of my
> VMs with 8GB I see this:
>
> $ ./jdk19-jdk/bin/java -Xms8g -XX:+PrintFlagsFinal -version 2>&1 | grep InitialHeapSize
>     size_t InitialHeapSize                          = 8589934592                                {product} {command line}
> $ free --giga
>                 total        used        free      shared  buff/cache   available
> Mem:               8           0           7           0           0           7
> Swap:              8           0           8
>
>> We have several choices:
>>
>> (a) Following the direction of JDK-8284900, check for the initial memory
>> usage  of other types of memory as well. The problem with this is that
>> checking the "initial" usage some type of memory usage (malloc or stack)
>> is difficult or impossible.
>>
>> (b) Avoid committing more memory when the total memory usage is close to
>> the limit. E.g., avoid expanding the Java heap, or return NULL from
>> os::malloc(). The problem is that some operations in the VM cannot
>> handle malloc failure and will terminate the VM anyway. Also, trying to
>> obey the memory limit with concurrent allocator/deallocator threads is
>> probably difficult or impossible.
>>
>> (c) Do not consider the memory limit and let the VM be killed by the OOM
>> killer. This may be what the user wants -- the user knows that the app
>> will normally run within the memory limit, so if the heap expands too
>> much the app probably has a memory leak and should be killed, to be
>> automatically restarted by an external mechanism (such as Kubernetes).
> Neither of the above?
>
> It would perhaps make sense to do some sanitiy checks on JVM start-up
> (somewhat akin to JDK-8284900). But this should probably be done with
> the main operating system API. Not using OSContainer API as proposed in
> JDK-8284900. For example, print a warning if MaxHeapSize,
> InitialHeapSize and MinHeapSize are larger than the reported physical
> memory (and/or cap it at a lower size if so).
>
> Isn't (b) being handled by setting os::pysical_memory() and
> os::available_memory() to the container limits (which we currently do)?
> It would behave similarly to a physical machine running out of memory,
> no?
>
> Thanks,
> Severin
>
>
>> [1] https://bugs.openjdk.java.net/browse/JDK-8285277
>> [2] https://bugs.openjdk.java.net/browse/JDK-8284900
>>