[RFC 8285277] - How should the JVM handle container memory limits

[David Holmes]

On 27/04/2022 4:12 am, Ioi Lam wrote:
> On 4/21/2022 2:49 AM, Severin Gehwolf wrote:
>> On Wed, 2022-04-20 at 11:09 -0700, Ioi Lam wrote:
>>> I would like to have a discussion on how (or whether) the JVM should
>>> handle container memory limits -- see JDK-8285277 [1]
>> It does handle container memory limits. os::physical_memory() returns
>> the container limit instead of the host value if a container limit is
>> in place. That was one big aspect of JDK-8146115 when it was done in
>> JDK 10 timeframe. See:
>>
>> https://github.com/openjdk/jdk/blob/fa04d1f832ff201248f935939fa255988053a1d0/src/hotspot/os/linux/os_linux.cpp#L217..L232 
>>
>>
>> What am I missing?
> 
> Yes, the container support code does expose the memory limit through 
> APIs like os::physical_memory(), and we use the same ergonomic logic as 
> if we are running on a physical machine with a small amount of RAM.
> 
> However, I am wondering if we need to reconsider our logic if the 
> "physical memory" is unusually small.
> 
> Currently, the ergonomic code will set the heap size to a "sensible" 
> fraction of os::physical_memory(). This usually works fine on modern day 
> machines with lots of RAM and swap. Even the below-$100 windows laptops 
> have at least 4GB.
> 
> On my 64GB box, max heap is set to 16GB by default. This should be 
> enough for most apps, and will practically never be killed by OOM unless 
> the system is really overloaded.
> 
> ================
> $ java -XX:+PrintFlagsFinal --version | egrep 
> '((MaxHeapSize)|(InitialHeapSize))'
>     size_t InitialHeapSize                          = 
> 1056964608                                {product} {ergonomic}
>     size_t MaxHeapSize                              = 
> 16869490688                               {product} {ergonomic}
> ================
> 
> But inside a container that has only a 128MB limit (which might be 
> "typical" for micro-services??)
> 
> ================
> $ java -XX:+PrintFlagsFinal --version | egrep 
> '((MaxHeapSize)|(InitialHeapSize))'
>     size_t InitialHeapSize                          = 
> 8388608                                   {product} {ergonomic}
>     size_t MaxHeapSize                              = 
> 67108864                                  {product} {ergonomic}
> ================
> 
> The default MaxHeapSize seems wasteful. We are paying for 128MB but 
> really just use 64MB of heap. I am able to start with a 108MB heap size 
> without getting killed.
> 
> ======================
> $ java -XshowSettings:system -Xms108m -XX:+AlwaysPreTouch --version
> Operating System Metrics:
>      Provider: cgroupv1
>      ....
>      Memory Limit: 128.00M
>      Memory Soft Limit: Unlimited
>      Memory & Swap Limit: 128.00M
> 
> java 19-internal 2022-09-20
> Java(TM) SE Runtime Environment (build 19-internal-adhoc.iklam)
> Java HotSpot(TM) 64-Bit Server VM (build 19-internal-adhoc.iklam, mixed 
> mode, sharing)
> ==========================
> 
> The problem is
> 
> - If we want to efficiently use all the memory we paid for in a 
> container, we need to tune the VM parameters manually
> 
> - But tuning is very difficult and error prone. With the 100mb heap 
> setting, my app could easily be killed if it uses too much memory in 
> malloc, native buffers, or even the JIT. Currently we don't fail early 
> (no check of -Xms, -Xmx, etc, against available memory), and don't fail 
> predictably.
> 
> So, I am wondering if there's anything we can do to make Java both safe 
> and efficient when running under an small memory budget.

I think the only thing you could do for the scenario outlined, when the 
VM has no idea what may be running in the container nor how much of the 
container memory it might have available to it, is to provide the "I'm 
running by myself so max out the memory use" flag. But even then 
determining how much slack you need to leave will be difficult given we 
only really have control over some aspects of memory usage.

Trying to pro-actively reduce memory use when you hit some soft limit is 
a good idea, but does such an API exist to let you know when you have 
hit that soft limit?

Cheers,
David

> Thanks
> - Ioi
> 
> 
>>> The JVM may be terminated by the Linux OOM killer if it tries to use
>>> more memory than the memory limit specified for a container. JDK-8284900
>>> [2] tries to avoid OOM by checking InitialHeapSize against the memory
>>> limit. However, this is incomplete because the JVM can use memory in
>>> other ways:
>>>
>>> - If -Xmx is larger than -Xms, the heap may expand
>>> - malloc memory
>>> - code cache
>>> - thread stacks
>> JDK-8284900 is somewhat a special case which, IIUIC, tries to avoid
>> user-errors detectable at JVM startup. For example setting -Xms to a
>> value >= physical memory (including swap). In this case the JVM would
>> be at risk to get killed by the OOM killer and doing this check on
>> start-up would catch it before it comes to it.
>>
>> On the other hand, I don't see any handling of such a case on a
>> *physical* machine. Consider a machine with 8GB of RAM and somebody
>> spawning a JVM with -Xms8g on it. It'll happily oblige? In one of my
>> VMs with 8GB I see this:
>>
>> $ ./jdk19-jdk/bin/java -Xms8g -XX:+PrintFlagsFinal -version 2>&1 | 
>> grep InitialHeapSize
>>     size_t InitialHeapSize                          = 
>> 8589934592                                {product} {command line}
>> $ free --giga
>>                 total        used        free      shared  
>> buff/cache   available
>> Mem:               8           0           7           0           
>> 0           7
>> Swap:              8           0           8
>>
>>> We have several choices:
>>>
>>> (a) Following the direction of JDK-8284900, check for the initial memory
>>> usage  of other types of memory as well. The problem with this is that
>>> checking the "initial" usage some type of memory usage (malloc or stack)
>>> is difficult or impossible.
>>>
>>> (b) Avoid committing more memory when the total memory usage is close to
>>> the limit. E.g., avoid expanding the Java heap, or return NULL from
>>> os::malloc(). The problem is that some operations in the VM cannot
>>> handle malloc failure and will terminate the VM anyway. Also, trying to
>>> obey the memory limit with concurrent allocator/deallocator threads is
>>> probably difficult or impossible.
>>>
>>> (c) Do not consider the memory limit and let the VM be killed by the OOM
>>> killer. This may be what the user wants -- the user knows that the app
>>> will normally run within the memory limit, so if the heap expands too
>>> much the app probably has a memory leak and should be killed, to be
>>> automatically restarted by an external mechanism (such as Kubernetes).
>> Neither of the above?
>>
>> It would perhaps make sense to do some sanitiy checks on JVM start-up
>> (somewhat akin to JDK-8284900). But this should probably be done with
>> the main operating system API. Not using OSContainer API as proposed in
>> JDK-8284900. For example, print a warning if MaxHeapSize,
>> InitialHeapSize and MinHeapSize are larger than the reported physical
>> memory (and/or cap it at a lower size if so).
>>
>> Isn't (b) being handled by setting os::pysical_memory() and
>> os::available_memory() to the container limits (which we currently do)?
>> It would behave similarly to a physical machine running out of memory,
>> no?
>>
>> Thanks,
>> Severin
>>
>>
>>> [1] https://bugs.openjdk.java.net/browse/JDK-8285277
>>> [2] https://bugs.openjdk.java.net/browse/JDK-8284900
>>>
>