Heap Size Ergonomics in docker containers

[Thomas Schatzl]

Hi Bob,

  

On Wed, 2017-10-25 at 08:44 -0400, Bob Vandette wrote:
> > On Oct 25, 2017, at 5:36 AM, Thomas Schatzl <thomas.schatzl at oracle.
> > com> wrote:
> > 
> > On Tue, 2017-10-24 at 14:43 -0400, Bob Vandette wrote:
> > > I’m trying to test the container support that I’m adding to 
> > > JDK 18.3 and I’m running into an issue with the Heap size
> > > calculation.
> > > 
> > > If I create a docker container that has 100MB of total memory on
> > > a
> > > 64-bit Linux box, I’m ending up with a 126MB Max Heap which
> > > results
> > > in the process getting killed by the OOM killer.
> > > 
> > > This is due to the fact that MaxHeapSize is 126MB and phys_mem
> > > (100MB) is
> > > not less than 50% of MinRAMPercentage of MaxHeapSize.
> > > 
> > > I would think in a small memory system, you don’t ever want the
> > > Heap 
> > > size to be more than somewhere around 70% of total RAM.  
> > > 
> > > A quick fix for my problem might be this, but this still
> > > leaves some pathological cases where phys_mem is greater than but
> > > close to MaxHeapSize.
> > > 
> > > if phys_mem < MaxHeapSize || phys_mem < MaxHeapSize *
> > > MinRAMPercentage
> > >    use phys_mem * MinRAMPercentage
> > > 
> > > I could improve on this with:
> > > 
> > > if MaxHeapSize > 70% of phys_mem || phys_mem < MaxHeapSize *
> > > MinRAMPercentage
> > >    use phys_mem * MinRAMPercentage
> > > 
> > > [...]
> > > Let me know what you think.  
> > > 
> > 
> >  that seems to be almost the same as making the default value of
> > MaxHeapSize 70% smaller.
> 
> Not really.  That just creates the same problem if the containers
> memory is set to some number near the lower value.
> 
> > Also that 70% only seems to serve the same
> > purpose as MaxRAMPercentage, i.e. imo MaxRAMPercentage should have
> > covered that.
> 
> I think it makes sense to have a MaxRAMPercentage with a
> MinRAMPercentage. You don’t want to use the same % for large GB’s of
> RAM as you’d use with 128M so I don’t agree that we should cover all
> cases with MaxRAMPercentage.

The general consensus within in the gc team is that the less
complicated any such heuristic is, the less surprising it is, the
easier to explain, with less maintenance and so more favorable from our
point of view. :)

The reasons are simple: this acknowledges the fact that you can't
capture the needs of billions of users in any kind of manageable
heuristic; there is also a limit on how complicated you want to make
this particular heuristic, because the more complicated to make it,
most likely you will get more corner cases for which you might want to
expose knobs to the user; and at that point the ultimate fix to this
issue is to tell to "set -Xmx" anyway.

A user is already going through the trouble of setting a max memory
size limit for your container...

Also, we have never seen anyone setting Min/MaxRAMFraction ever except
in error.

I do not think for this issue we should go down this road, making the
heuristics even more complicated.

> > 
> > I kind of question this line in the code:
> > 
> >       // Not-small physical memory, so require a heap at least
> >       // as large as MaxHeapSize
> >       reasonable_max = MAX2(reasonable_max, (julong)MaxHeapSize);
> > 
> > So we just found out a "reasonable max" as a part of the
> > calculation, and then we ignore it…
> 
> I don’t think it’s ignored.  reasonable_max will be set to 25% of
> phys_mem which could be larger than MaxHeapSize.
> 

I filed JDK-8190283 [0], as this is a bug however you look at it.

The suggested fix for this is to use physmem*MinRAMFraction while it is
below the default MaxHeapSize value, then switch over to
physmem*MaxRAMFraction. Since physmem*MaxRAMFraction is at that point
smaller than MaxHeapSize, keep using MaxHeapSize as long as
physmem*MaxRAMFraction is smaller.
As such, default MaxHeapSize would act as kind of switchover threshold
between the two heuristics (that switchover point is pretty random as
before) - that should ensure that the selected heap size is always
smaller than the amount of physical memory, and fix the problem.

This would make the heap 50% (MinRAMFraction) of physical memory for
machines with < 252M, 126M for machines with physical memory < 504M,
and 25% (MaxRAMFraction) of physical memory otherwise.

A prototype patch is available at http://cr.openjdk.java.net/~tschatzl/
8190283/webrev/ (did not test that, I simply tried to implement the
formula from a spreadsheet I used, but I think it shows the idea) -
it's also very small.

What do you think?

------

Now there still may be the question whether the use case you presented,
i.e. "with a "small heap" with 100MB use 70% of the available memory"
is worth changing some additional default options for.

We think that we should not go that direction: the VM does not give
outright bad values for heap size any more (50MB with default
MinRAMFraction), providing safe and fairly good settings. There has
been no indication on whether a 100MB container is that pervasive
either.

Particularly a MinRAMFraction of 70% seems a bit too tight as default
for the current default collector, i.e. G1. As we have seen it can, in
extreme cases as a ballpark figure use another 20% of Java heap size
just for its remembered sets (theoretical max is even higher), and
other subsystems also need memory (compiler, runtime).

Note that I think that maybe G1 isn't the correct choice for such small
heaps at the moment - serial or parallel would probably run as fine if
not better with less overhead(*) - but that's also something we would
not like to change haphazardly; and it is possible and might be better
to improve G1 here in the future instead of changing defaults. (And if
the user is already selecting the GC, he might as well set max heap
size).

Thanks,
  Thomas

[0] https://bugs.openjdk.java.net/browse/JDK-8190283
(*) haven't tried lately.