Additional JEPS 8182070 requirement: avoiding container OOM

Wed Dec 13 17:00:58 UTC 2017

On Wed, Dec 13, 2017 at 4:51 PM, Bob Vandette <bob.vandette at oracle.com>
wrote:

>
> On Dec 13, 2017, at 11:01 AM, Glyn Normington <gnormington at pivotal.io>
> wrote:
>
> Hi Bob
>
> Thanks very much for that information. Some comments inline. This one
> could be relevant to the refocussed JEP: the JVM should monitor its
> approach to the container memory limit and trigger "out of memory"
> processing at some threshold close to the limit.
>
>
> Memory allocations in hotspot are difficult to track since there are many
> different allocators
> and we don’t control them all.  Thread stacks get dynamically allocated
> upon use, the C++ runtime allocates
> memory for native code and third party JNI library could be calling
> malloc.  These are all in addition
> to the allocators we do control - Java Heap and internal Memory Arenas for
> runtime metadata.
>
> One possible solution is to at least attempt threshold checking during
> allocations we
> do control and to do OOM processing at that time.  This wouldn’t catch
> every case but could
> increase the robustness of the Java runtime.  With the change I introduced
> in JDK-8146155,
> we now at least have a way of finding out how much free memory is
> available in the container.
>

Another solution you might like to mull over is for the JVM to poll the
memory cgroup value memory.usage_in_bytes (assuming the memory cgroup is
mounted read-only inside the container) and trigger "out of memory" on some
threshold.

>
> Bob.
>
>
> Regards,
> Glyn
>
> On Wed, Dec 13, 2017 at 2:58 PM, Bob Vandette <bob.vandette at oracle.com>
> wrote:
>
>> Hi Glyn,
>>
>> The JEP you mention is going to be updated in a week or so to refocus
>> it’s goals
>> on monitoring of the JVM running in containers.  This is due to the fact
>> that I was
>> able to split out the work of dynamically configuring the VM based on
>> container
>> settings and integrate this work into JDK 10.  This work was integrated
>> under jira issue
>> https://bugs.openjdk.java.net/browse/JDK-8146115.
>>
>> Although this change does not completely address your requirement, it
>> should help.
>> This change causes the JVM to correctly configure the number of threads
>> and memory
>> allocations based on the resources configured in the container.  This
>> change also
>> introduces a new JVM option (-XX:ActiveProcessorCount=xx) which allows
>> the user
>> to specify how many processors the JVM should use.  We will now honor cpu
>> shares, quotas
>> in addition to cpusets for determining the number of threads that the JVM
>> will use.
>>
>
> Yes, that does sound like a useful set of changes.
>
>
>>
>> Prior to this change, I also added https://bugs.openjdk.jav
>> a.net/browse/JDK-8186315.
>> This change allows the user to specify the percentage of container memory
>> that
>> should be used by the Java Heap.  This allows the users to reserve memory
>> outside
>> of the Java heap for other classes of memory allocations (C-Heap for
>> example).
>>
>> Both of these changes have been integrated into JDK 10 and are available
>> in the
>> latest early access release of Linux x64 (http://jdk.java.net/10/).
>>
>
> How interesting! The previous version of our Java memory calculator
> allowed the heap size to be specified as a proportion of container memory,
> but we could only guess at a default proportion and users found it quite
> hard to know what value to use when they needed to increase or decrease
> heap size.
>
> The current version adopts a different approach: we try to calculate JVM
> memory consumption (apart from heap) from the JVM options and then subtract
> that value from the container memory to get the maximum heap size. This
> approach suffers from the difficulty of predicting the memory consumption
> of the JVM with any accuracy (see my recent thread entitled 'Excessive "GC"
> memory area size'). The up-side of the new approach is that the user can
> override the behaviour using standard JVM options rather than proportions.
>
>
>>
>> I will talk to the Hotspot runtime folks about your requirement
>>
>
> Thanks very much.
>
>
>> but other than these
>> options, have you considered configuring your containers to use some swap
>> space
>> in addition to memory configurations so that temporary spikes in memory
>> consumption
>> won’t cause an OOM ?
>>
>
> We talked about this but we need to bound the amount of swap space because
> our environment is a PaaS and we can't allow users to consume more than
> their fair share of system resources. If we did enable a bounded swap
> space, I think we'd hit the same issue when the RAM+swap limit was exceeded.
>
>
>> There are also ways of disabling the OOM killer in containers.
>>
>
> We actually tried that. We started the container with the OOM killer
> disabled and when the container hit OOM, we gathered diagnostics and
> re-enabled the OOM killer to terminate the container. The main issue with
> that is the container-level diagnostics (number of committed pages of RAM
> etc.) don't tell the Java programmer what JVM options need changing to
> avoid the problem in future.
>
> A more sophisticated approach along those lines would be to disable the
> OOM killer and when the container *nears* its limit somehow trigger the JVM
> to do "out of memory" processing. Then our JVMTI agent would be driven and
> produce plenty of diagnostics suitable for a Java programmer. This is a bit
> speculative though as it requires fairly deep changes to the container
> configuration as well as solving the problem of how to trigger the JVM to
> do its "out of memory" processing. Maybe a more practical solution there is
> for the JVM itself to monitor its approach to the container memory limit
> and trigger "out of memory" processing at some threshold close to the
> limit. This could be relevant to the JEP.
>
>
>> If we get a failed memory allocation, we will try to deliver a Java
>> OutOfMemory
>> exception which would at least allow you to figure out what went wrong.
>> I haven’t looked
>> into this yet but it is something worth considering.
>>
>
> Yeah, OutOfMemory exceptions provide some useful information, although we
> found it was insufficiently detailed which is why we wrote a JVMTI agent to
> dump out detailed diagnostics.
>
>
>>
>> Bob.
>>
>>
>>
>> On Dec 13, 2017, at 4:13 AM, Glyn Normington <gnormington at pivotal.io>
>> wrote:
>>
>> I wonder if someone involved in JEPS 8182070 (Container aware Java) would
>> care to comment on the additional requirement described below?
>>
>> On Mon, Nov 13, 2017 at 9:30 AM, Glyn Normington <gnormington at pivotal.io>
>> wrote:
>>
>> I would like to mention an additional requirement for JEPS 8182070 (
>> http://openjdk.java.net/jeps/8182070): avoid the JVM hitting container
>> OOM by strictly bounding the amount of (physical) memory that the JVM
>> consumes. This may be implicit in the document, but I think it should be
>> made an explicit goal.
>>
>> If a java application hits container OOM, no detailed diagnostics, such as
>> those associated with an OutOfMemoryError or a JVMTI resource exhaustion
>> event, are presented to the user, so the user finds it very difficult to
>> know how to fix the problem.
>>
>> The Cloud Foundry OSS project has done quite a bit of work on this problem
>> and provides a couple of utilities which help when running a JVM in a
>> container:
>>
>> * Java memory calculator ([1], [2]) to determine JVM memory settings,
>> * jvmkill JVMTI agent ([3]) to report detailed diagnostics on a resource
>> exhaustion event.
>>
>> Regards,
>> Glyn
>>
>> [1] https://github.com/cloudfoundry/java-buildpack-memory-calculator
>>
>> [2] Design doc: https://docs.google.com/document/d/
>> 1vlXBiwRIjwiVcbvUGYMrxx2Aw1RVAtxq3iuZ3UK2vXA/edit#heading=h.uy41ishpv9zc
>>
>> [3] https://github.com/cloudfoundry/jvmkill
>>
>>
>>
>>
>> --
>> Regards,
>> Glyn
>>
>>
>>
>
>
> --
> Regards,
> Glyn
>
>
>

-- 
Regards,
Glyn