Additional JEPS 8182070 requirement: avoiding container OOM

[Bob Vandette]

Hi Glyn,

The JEP you mention is going to be updated in a week or so to refocus it’s goals
on monitoring of the JVM running in containers.  This is due to the fact that I was
able to split out the work of dynamically configuring the VM based on container
settings and integrate this work into JDK 10.  This work was integrated under jira issue
https://bugs.openjdk.java.net/browse/JDK-8146115 <https://bugs.openjdk.java.net/browse/JDK-8146115>.

Although this change does not completely address your requirement, it should help.
This change causes the JVM to correctly configure the number of threads and memory
allocations based on the resources configured in the container.  This change also
introduces a new JVM option (-XX:ActiveProcessorCount=xx) which allows the user
to specify how many processors the JVM should use.  We will now honor cpu shares, quotas
in addition to cpusets for determining the number of threads that the JVM will use.

Prior to this change, I also added https://bugs.openjdk.java.net/browse/JDK-8186315 <https://bugs.openjdk.java.net/browse/JDK-8186315>.
This change allows the user to specify the percentage of container memory that
should be used by the Java Heap.  This allows the users to reserve memory outside
of the Java heap for other classes of memory allocations (C-Heap for example).

Both of these changes have been integrated into JDK 10 and are available in the
latest early access release of Linux x64 (http://jdk.java.net/10/ <http://jdk.java.net/10/>).

I will talk to the Hotspot runtime folks about your requirement but other than these 
options, have you considered configuring your containers to use some swap space
in addition to memory configurations so that temporary spikes in memory consumption 
won’t cause an OOM ?  There are also ways of disabling the OOM killer in containers.
If we get a failed memory allocation, we will try to deliver a Java OutOfMemory
exception which would at least allow you to figure out what went wrong.  I haven’t looked 
into this yet but it is something worth considering.

Bob.



> On Dec 13, 2017, at 4:13 AM, Glyn Normington <gnormington at pivotal.io> wrote:
> 
> I wonder if someone involved in JEPS 8182070 (Container aware Java) would
> care to comment on the additional requirement described below?
> 
> On Mon, Nov 13, 2017 at 9:30 AM, Glyn Normington <gnormington at pivotal.io>
> wrote:
> 
>> I would like to mention an additional requirement for JEPS 8182070 (
>> http://openjdk.java.net/jeps/8182070): avoid the JVM hitting container
>> OOM by strictly bounding the amount of (physical) memory that the JVM
>> consumes. This may be implicit in the document, but I think it should be
>> made an explicit goal.
>> 
>> If a java application hits container OOM, no detailed diagnostics, such as
>> those associated with an OutOfMemoryError or a JVMTI resource exhaustion
>> event, are presented to the user, so the user finds it very difficult to
>> know how to fix the problem.
>> 
>> The Cloud Foundry OSS project has done quite a bit of work on this problem
>> and provides a couple of utilities which help when running a JVM in a
>> container:
>> 
>> * Java memory calculator ([1], [2]) to determine JVM memory settings,
>> * jvmkill JVMTI agent ([3]) to report detailed diagnostics on a resource
>> exhaustion event.
>> 
>> Regards,
>> Glyn
>> 
>> [1] https://github.com/cloudfoundry/java-buildpack-memory-calculator
>> 
>> [2] Design doc: https://docs.google.com/document/d/
>> 1vlXBiwRIjwiVcbvUGYMrxx2Aw1RVAtxq3iuZ3UK2vXA/edit#heading=h.uy41ishpv9zc
>> 
>> [3] https://github.com/cloudfoundry/jvmkill
>> 
> 
> 
> 
> -- 
> Regards,
> Glyn