Container-aware heap sizing for OpenJDK

Sat Sep 17 10:08:21 UTC 2022

Hi Jonathan, thank you for sharing your ideas and results. My team and
myself are also interested in the suggested improvements, and happy to
collaborate on this.

First of all, I do agree that modern cloud environments can be smarter in
terms of memory management and adaptive right-sizing, because there are
many applications and services with variable load. Based on our experience,
the inability to provide an optimal runtime for different kinds of
workloads leads to a poor user experience as the end users have to do
manual fine tuning, which is a difficult task, or they simply allocate (pay
for) more resources than they really need.

As Volker mentioned, your suggestion appears to be somewhat similar to "JEP
Draft: Dynamic Maximum Memory Limit" https://openjdk.org/jeps/8204088. Here
is a more detailed description of the work and ideas behind the JEP
https://rodrigo-bruno.github.io/papers/rbruno-ismm18.pdf. In other words,
"Current maximum heap expansion size" is the same as CurrentMaxHeapSize (or
HardMaxHeapSize - alt name discussed previously here in the group) from the
JEP. Also "Current target heap size" is the same as SoftMaxHeapSize (for G1
it's not yet implemented https://bugs.openjdk.org/browse/JDK-8236073).
Regarding the "Separate thread runs along with the JVM" - this can be done
in the way you described, or since the container limits are configured by
an external orchestrator, the orchestrator can send commands to the JVM
(before or after setting the container limits depending on the case).

Looking forward to getting more details on the implementation by the Google
team.

Thanks
--
*Ruslan Synytsky *| CTO | Virtuozzo | rs at virtuozzo.com

> ---------- Forwarded message ----------
> From: Volker Simonis <volker.simonis at gmail.com>
> Date: Fri, 16 Sep 2022 09:39:04 +0200
> Subject: Re: Container-aware heap sizing for OpenJDK
> Hi Jonathan,
>
> thanks for starting this discussion. It's a relevant topic and we're
> definitely interested as well.
>
> I have a few questions:
> 1. Is there a public prototype available?
> 2. For which GCs have you implemented it?
> 3. On which platforms does it currently work?
> 4. Do you use information from procfs on Linux (or what else) to get
> the memory state of the system?
> 5. How often do you query the system and JVM state and update the
> settings? Is this (or can it be) correlated to the JVMs allocation
> rate?
> 6. Can you please explain your approach in some more detail (e.g. how
> does heap shrinking work)? I suppose you have "Current target heap
> size" <= "Current maximum heap expansion size" <= Xmx. I can
> understand how your monitoring thread updates "Current maximum heap
> expansion size" based on the systems memory usage. But it's harder to
> understand how you set "Current target heap size" based GC CPU
> overhead and when and how do you trigger heap resizing (both
> increasing and shrinking) based on these values? And why do you need
> two new variables? Wouldn't "Current target heap size" be enough"
> (also see next question for more context)?
> 7. What you propose is very similar to the CurrentMaxHeapSize proposed
> by "JEP draft: Dynamic Max Memory Limit" [1] except that the latter
> proposal misses the part about how to automatically set and update
> this value.
> 8. We already have "JEP 346: Promptly Return Unused Committed Memory
> from G1" [2] since JDK 12. Can you explain why this isn't enough? I.e.
> if the GC already returns as much heap memory as possible back to the
> system, what else can you do to further improve the situation?
>
> [1] https://openjdk.org/jeps/8204088
> [2] https://openjdk.org/jeps/346
>
> On Tue, Sep 13, 2022 at 12:17 PM Jonathan Joo <jonathanjoo at google.com>
wrote:

> Hello hotspot-dev and hotspot-gc-dev,
>
> My name is Jonathan, and I'm working on the Java Platform Team at Google.
> Here, we are working on a project to address Java container memory issues,
> as we noticed that a significant number of Java servers hit container OOM
> issues due to people incorrectly tuning their heap size with respect to the
> container size. Because our containers have other RAM consumers which
> fluctuate over time, it is often difficult to determine a priori what is an
> appropriate Xmx to set for a particular server.
>
> We set about trying to solve this by dynamically adjusting the Java
> heap/gc behavior based on the container usage information that we pass into
> the JVM. We have seen promising results so far, reducing container OOMs by
> a significant amount, and oftentimes also reducing average heap usage (with
> the tradeoff of more CPU time spent doing GC).
>
> Below (under the dotted line) is a more detailed explanation of our
> initial approach. Does this sound like something that may be useful for the
> general OpenJDK community? If so, would some of you be open to further
> discussion? I would also like to better understand what container
> environments look like outside of Google, to see how we could modify our
> approach for the more general case.
>
> Thank you!
>
>
> Jonathan
> ------------------------------------------------------------------------
> Introduction:
>
> Adaptable Heap Sizing (AHS) is a project internal to Google that is meant
> to simplify configuration and improve the stability of applications in
> container environments. The key is that in a containerized environment, we
> have access to container usage and limit information. This can be used as a
> signal to modify Java heap behavior, helping prevent container OOMs.
> Problem:
>
>    -
>
>    Containers at Google must be properly sized to not only the JVM heap,
>    but other memory consumers as well. These consumers include non-heap Java
>    (e.g. native code allocations), and simultaneously running non-Java
>    processes.
>    -
>
>    Common antipattern we see here at Google:
>    -
>
>       We have an application running into container OOMs.
>       -
>
>       An engineer raises both container memory limit and Xmx by the same
>       amount, since there appears to be insufficient memory.
>       -
>
>       The application has reduced container OOMs, but is still prone to
>       them, since G1 continues to use most of Xmx.
>       -
>
>    This results in many jobs being configured with much more RAM than
>    they need, but still running into container OOM issues.
>
> Hypothesis:
>
>    -
>
>    For preventing container OOM: Why can't heap expansions be bounded by
>    the remaining free space in the container?
>    -
>
>    For preventing the `unnecessarily high Xmx` antipattern: Why can't
>    target heap size be set based on GC CPU overhead?
>    -
>
>    From our work on Adaptable Heap Sizing, it appears they can!
>
> Design:
>
>    -
>
>    We add two manageable flags in the JVM
>    -
>
>       Current maximum heap expansion size
>       -
>
>       Current target heap size
>       -
>
>    A separate thread runs alongside the JVM, querying:
>    -
>
>       Container memory usage/limits
>       -
>
>       GC CPU overhead metrics from the JVM.
>       -
>
>    This thread then uses this information to calculate new values for the
>    two new JVM flags, and continually updates them at runtime.
>    -
>
>    The `Current maximum heap expansion size` informs the JVM what is the
>    maximum amount we can expand the heap by, while staying within container
>    limits. This is a hard limit, and trying to expand more than this amount
>    results in behavior equivalent to hitting the Xmx limit.
>    -
>
>    The `Current target heap size` is a soft target value, which is used to
>    resize the heap (when possible) so as to bring GC CPU overhead toward its
>    target value.
>
>
> Results:
>
>    -
>
>    At Google, we have found that this design works incredibly well in our
>    initial rollout, even for large and complex workloads.
>    -
>
>    After deploying this to dozens of applications:
>    -
>
>       Significant memory savings for previously misconfigured jobs (many
>       of which reduced their heap usage by 50% or more)
>       -
>
>       Significantly reduced occurrences of container OOM (100% reduction
>       in vast majority of cases)
>       -
>
>       No correctness issues
>       -
>
>       No latency regressions*
>       -
>
>       We plan to deploy AHS across a much wider subset of applications by
>       EOY '22.
>
>
> *Caveats:
>
>    - Enabling this feature might require tuning of the newly introduced
>    default GC CPU overhead target to avoid regressions.
>    -
>
>    Time spent doing GC for an application may increase significantly
>    (though generally we've seen in practice that even if this is the case,
>    end-to-end latency does not increase a noticeable amount)
>    -
>
>    Enabling AHS results in frequent heap resizings, but we have not seen
>    evidence of any negative effects as a result of these more frequent heap
>    resizings.
>    -
>
>    AHS is not necessarily a replacement for proper JVM tuning, but should
>    generally work better than an untuned or improperly tuned configuration.
>    -
>
>    AHS is not intended for every possible workload, and there could be
>    pathological cases where AHS results in worse behavior.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20220917/6c273be2/attachment.htm>