Heap Size Ergonomics in docker containers

Wed Nov 1 10:30:50 UTC 2017

Hi Thomas, thank you for the feedback. Please see inline.

On 31 October 2017 at 16:22, Thomas Schatzl <thomas.schatzl at oracle.com>
wrote:

> Hi Ruslan,
>
> On Mon, 2017-10-30 at 19:06 +0100, Ruslan Synytsky wrote:
> > Hi guys, I'm interested in improvements in this direction as well.
> > Let me share the statistical experience from Jelastic customers who
> > run Java stacks in containers for a long time.
> >
> > We set the minimum Xmx to 32m by default as we believe majority wants
> > to start from the minimum, specifically in the "micro" world
> > (services, containers). G1 GC provides good enough elasticity of
> > memory allocation. You can grow up quickly, and important to scale
> > down later when the load goes away.
> >
> > By default, Xmx is 80% of available RAM with container limits. It
> > works great for the majority of our users. We track OOM Kill events
> > and send alerts to the users. So the numbers are based on the real
> > end users applications.
> >
> > We keep 80% from max by default always, even if users allocate a
> > bigger amount of RAM, for two reasons: 1 - it's easier to remember
> > the defaults and 2 - unused resources are not reserved with
> > containers, so if Java does not use it - no problem, they are free,
> > and at the same time additional memory available for native non-heap
> > data or for other processes, just like an insurance.
>
> Thanks for your insights into real-world usage. So maybe the suggested
> 50% as default for small heaps is indeed a bit too conservative.
>
Understanding of the situations when people rely on the default heuristic
will be really helpful to make a final decision... From my personal
experience, if you do not specify Xmx you always get troubles at the end.

> > Big memory and big data solutions - everything like this requires a
> > fine-tuning, the customers who run such stacks definitely do not rely
> > on the default heuristic. You can always find instructions and
> > recommended RAM usage configs for these stacks. Some solutions even
> > use direct memory access, a small heap size is preferable for them.
> > In general, it means you have to do a customization if you want to
> > get a stable solution.
> >
> > To make it simple, we came up with a supervisor script that
> > applies custom memory configs. As result, customers can define the
> > needed parameters with env variables in a container if they do not
> > want to rely on the heuristic.
> >
> > Btw, is there any guaranteed way to pass Xmx and Xms to java process
> > w/o any external scripts? I mean, it will be cool if java can check
> > env variable like XMX and if it exists apply.
> >
>
> There is JAVA_TOOL_OPTIONS, which are always prepended to the java VM
> command line. See the description [0] for more information.
>
Indeed this option is close to what I was looking for! But seems
JAVA_TOOL_OPTIONS does not work for Xmx. Just a quick test with Java
HotSpot(TM) 64-Bit Server VM (build 9+181, mixed mode)

*Default*
$ java -XX:+PrintFlagsFinal -version | grep MaxHeapSize
   size_t MaxHeapSize                              = 643825664
                   {product} {command line}

*Define Xmx = 2GB via JAVA_TOOL_OPTIONS *
$ export JAVA_TOOL_OPTIONS="-Xmx2g"
$ java -XX:+PrintFlagsFinal -version | grep MaxHeapSize
Picked up JAVA_TOOL_OPTIONS: -Xmx2g
   size_t MaxHeapSize                              = 643825664
                   {product} {command line}

It says that Xmx has been picked up, but in reality it has not been
applied.

*Define Xmx = 3GB via _JAVA_OPTIONS *
$ export _JAVA_OPTIONS="-Xmx3g"
$ java -XX:+PrintFlagsFinal -version | grep MaxHeapSize
Picked up JAVA_TOOL_OPTIONS: -Xmx2g
Picked up _JAVA_OPTIONS: -Xmx3g
   size_t MaxHeapSize                              = 3221225472
                  {product} {command line}

Now it works as expected. However, another issue is that _JAVA_OPTIONS is
undocumented and hotspot-specific?...

Also important to mention that _JAVA_OPTIONS trumps command-line arguments,
and command-line arguments trump JAVA_TOOL_OPTIONS. That gives a quite good
flexibility. Because sometimes you may want to override command line args
and sometimes you need just to provide better deafults, but if users
specify command line args use them. In other words, I believe, if we can a)
improve JAVA_TOOL_OPTIONS (or introduce a new name like
JAVA_OPTIONS_DEFAULT) for enabling people to specify Xmx via this env var
and b) make _JAVA_OPTIONS as a standard/documented option, then this will
solve the existig dificulties and provide enough flexibility.

Thanks

>
> I think this is what you were looking for.
>
> Thanks,
>   Thomas
>
> [0] https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot
> /envvars002.html
> <https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars002.html>
>

-- 
Ruslan
CEO @ Jelastic <https://jelastic.com/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20171101/29842c5b/attachment.htm>