[RFC 8285277] - How should the JVM handle container memory limits

Wed Apr 27 13:01:42 UTC 2022

On Tue, 2022-04-26 at 11:12 -0700, Ioi Lam wrote:
> On 4/21/2022 2:49 AM, Severin Gehwolf wrote:
> > On Wed, 2022-04-20 at 11:09 -0700, Ioi Lam wrote:
> > > I would like to have a discussion on how (or whether) the JVM should
> > > handle container memory limits -- see JDK-8285277 [1]
> > It does handle container memory limits. os::physical_memory() returns
> > the container limit instead of the host value if a container limit is
> > in place. That was one big aspect of JDK-8146115 when it was done in
> > JDK 10 timeframe. See:
> > 
> > https://github.com/openjdk/jdk/blob/fa04d1f832ff201248f935939fa255988053a1d0/src/hotspot/os/linux/os_linux.cpp#L217..L232
> > 
> > What am I missing?
> 
> Yes, the container support code does expose the memory limit through 
> APIs like os::physical_memory(), and we use the same ergonomic logic as 
> if we are running on a physical machine with a small amount of RAM.
> 
> However, I am wondering if we need to reconsider our logic if the 
> "physical memory" is unusually small.

Not just for small containers. Some tunings are needed for larger
containers too (considering that it's usually 1 process per container).

> Currently, the ergonomic code will set the heap size to a "sensible" 
> fraction of os::physical_memory(). This usually works fine on modern day 
> machines with lots of RAM and swap. Even the below-$100 windows laptops 
> have at least 4GB.

For some definition of "sensible". For a physical machine a value of
MaxRAMPercentage=25 makes sense. For a container where the typical use
case is some cloud orchestration framework spawning them and in 99% of
the cases the container runs a single process with some memory limit
imposed a value of MaxRAMPercentage=25 doesn't make as much sense. Say
a container limit is 500MB, then your heap limit (MaxHeapSize) without
any further tuning elsewhere will be 500/4 (126MB), which in that case
most of the time won't be what the user wants. Note that this means
that the JVM will throw OutOfMemory errors as soon as the MaxHeapSize
is being reached, even though there probably is unused container memory
still available (most likely 3*500/4 - N, where N the amount of native
JVM memory).

> On my 64GB box, max heap is set to 16GB by default. This should be 
> enough for most apps, and will practically never be killed by OOM unless 
> the system is really overloaded.
> 
> ================
> $ java -XX:+PrintFlagsFinal --version | egrep 
> '((MaxHeapSize)|(InitialHeapSize))'
>     size_t InitialHeapSize                          = 
> 1056964608                                {product} {ergonomic}
>     size_t MaxHeapSize                              = 
> 16869490688                               {product} {ergonomic}
> ================
> 
> But inside a container that has only a 128MB limit (which might be 
> "typical" for micro-services??)
> 
> ================
> $ java -XX:+PrintFlagsFinal --version | egrep 
> '((MaxHeapSize)|(InitialHeapSize))'
>     size_t InitialHeapSize                          = 
> 8388608                                   {product} {ergonomic}
>     size_t MaxHeapSize                              = 
> 67108864                                  {product} {ergonomic}
> ================
> 
> The default MaxHeapSize seems wasteful. We are paying for 128MB but 
> really just use 64MB of heap. I am able to start with a 108MB heap size 
> without getting killed.

Yes. While I got confused a little what you meant by "wasteful" it
sounds like you are saying it should set MaxHeapSize to a higher level.
Is that correct? If so that's basically what I was trying to say with
the (wrong) default of MaxRAMPercentage when run in a container. My
experimentations with this didn't go anywhere, though, as we cannot
reliably detect if we are on a plain Linux system or in a container.
See JDK-8261242.

> ======================
> $ java -XshowSettings:system -Xms108m -XX:+AlwaysPreTouch --version
> Operating System Metrics:
>      Provider: cgroupv1
>      ....
>      Memory Limit: 128.00M
>      Memory Soft Limit: Unlimited
>      Memory & Swap Limit: 128.00M
> 
> java 19-internal 2022-09-20
> Java(TM) SE Runtime Environment (build 19-internal-adhoc.iklam)
> Java HotSpot(TM) 64-Bit Server VM (build 19-internal-adhoc.iklam, mixed 
> mode, sharing)
> ==========================
> 
> The problem is
> 
> - If we want to efficiently use all the memory we paid for in a 
> container, we need to tune the VM parameters manually
> 
> - But tuning is very difficult and error prone. With the 100mb heap 
> setting, my app could easily be killed if it uses too much memory in 
> malloc, native buffers, or even the JIT. Currently we don't fail early 
> (no check of -Xms, -Xmx, etc, against available memory), and don't fail 
> predictably.

+1 to doing some more sanity checks on JVM startup when some of those
settings exceed physical memory.

> 
> So, I am wondering if there's anything we can do to make Java both safe 
> and efficient when running under an small memory budget.

I sure hope that we can improve this. One low hanging fruit would be to
adjust MaxRAMPercentage (in containers). The question is how to do this
without causing issues for JVMs running on physical machines on Linux.

Thanks,
Severin

> Thanks
> - Ioi
> 
> 
> > > The JVM may be terminated by the Linux OOM killer if it tries to use
> > > more memory than the memory limit specified for a container. JDK-8284900
> > > [2] tries to avoid OOM by checking InitialHeapSize against the memory
> > > limit. However, this is incomplete because the JVM can use memory in
> > > other ways:
> > > 
> > > - If -Xmx is larger than -Xms, the heap may expand
> > > - malloc memory
> > > - code cache
> > > - thread stacks
> > JDK-8284900 is somewhat a special case which, IIUIC, tries to avoid
> > user-errors detectable at JVM startup. For example setting -Xms to a
> > value >= physical memory (including swap). In this case the JVM would
> > be at risk to get killed by the OOM killer and doing this check on
> > start-up would catch it before it comes to it.
> > 
> > On the other hand, I don't see any handling of such a case on a
> > *physical* machine. Consider a machine with 8GB of RAM and somebody
> > spawning a JVM with -Xms8g on it. It'll happily oblige? In one of my
> > VMs with 8GB I see this:
> > 
> > $ ./jdk19-jdk/bin/java -Xms8g -XX:+PrintFlagsFinal -version 2>&1 | grep InitialHeapSize
> >     size_t InitialHeapSize                          = 8589934592                                {product} {command line}
> > $ free --giga
> >                 total        used        free      shared  buff/cache   available
> > Mem:               8           0           7           0           0           7
> > Swap:              8           0           8
> > 
> > > We have several choices:
> > > 
> > > (a) Following the direction of JDK-8284900, check for the initial memory
> > > usage  of other types of memory as well. The problem with this is that
> > > checking the "initial" usage some type of memory usage (malloc or stack)
> > > is difficult or impossible.
> > > 
> > > (b) Avoid committing more memory when the total memory usage is close to
> > > the limit. E.g., avoid expanding the Java heap, or return NULL from
> > > os::malloc(). The problem is that some operations in the VM cannot
> > > handle malloc failure and will terminate the VM anyway. Also, trying to
> > > obey the memory limit with concurrent allocator/deallocator threads is
> > > probably difficult or impossible.
> > > 
> > > (c) Do not consider the memory limit and let the VM be killed by the OOM
> > > killer. This may be what the user wants -- the user knows that the app
> > > will normally run within the memory limit, so if the heap expands too
> > > much the app probably has a memory leak and should be killed, to be
> > > automatically restarted by an external mechanism (such as Kubernetes).
> > Neither of the above?
> > 
> > It would perhaps make sense to do some sanitiy checks on JVM start-up
> > (somewhat akin to JDK-8284900). But this should probably be done with
> > the main operating system API. Not using OSContainer API as proposed in
> > JDK-8284900. For example, print a warning if MaxHeapSize,
> > InitialHeapSize and MinHeapSize are larger than the reported physical
> > memory (and/or cap it at a lower size if so).
> > 
> > Isn't (b) being handled by setting os::pysical_memory() and
> > os::available_memory() to the container limits (which we currently do)?
> > It would behave similarly to a physical machine running out of memory,
> > no?
> > 
> > Thanks,
> > Severin
> > 
> > 
> > > [1] https://bugs.openjdk.java.net/browse/JDK-8285277
> > > [2] https://bugs.openjdk.java.net/browse/JDK-8284900
> > > 
> 