JVM heaps can scale up but not down

[Andrew Ash]

Hi Jesper,

Many thanks for the extensive answer!

It sounds like with the newer features the scale-down scenario will be
possible.  Here's how I think it could work from the new features:

1. Mesos starts a task with X memory of Y memory on a machine
2. JVM starts with -Xmx=Y and a MinHeapFree ratio of (X-Y)/Y to keep the
max actual utilization at X
3. Application running in the JVM does work
4. Mesos decides to take back memory and sends a message to the app that it
should shrink to Z memory (Z < X)
5. App reduces its memory usage by the appropriate amount
6. App signals to the JVM to use a lower MaxHeapFreeRatio and triggers Full
GC
7. JVM performs Full GC and shrinks heap
8. App checks if memory usage is below Z; goto step 5 if not
9. App tells Mesos it has shrunk its usage
10. Mesos gives the memory to some other task

The Xmx/MinHeapFree calculation in step 2 is to support increasing the
memory usage up to the maximum of the machine (is Xmx manageable now?).

There are some difficulties in step 5 of how the app knows how much memory
to remove.  In my motivating case (Apache Spark) the app keeps a pool of
cached data with size estimates, and could potentially drop the appropriate
amount of data from its cache.  It would require recalculation but that can
be streamed and never held in a JVM's heap at once.

The combination of step 6 and step 7 are what the recent JDK feature work
will make possible.


Is this how you envision the scale-down scenario working in practice?


Many thanks,
Andrew


On Wed, Aug 20, 2014 at 11:47 PM, Jesper Wilhelmsson <
jesper.wilhelmsson at oracle.com> wrote:

> Hi Andrew,
>
> The flag -XX:MaxHeapFreeRatio was recently made manageable so that you can
> change its value during runtime. The motivating use case behind that change
> was exactly the scenario you describe, that you want to force a running JVM
> to shrink the heap.
>
> Lowering MaxHeapFreeRatio may decrease performance since it will likely
> increase garbage collection frequency. The way to use it in a production
> server would probably be to lower the ratio, force a System.gc to shrink
> the heap and then increase the ratio again.
>
> Unfortunately there can never be any guaranties that the heap can shrink
> unless you force a full GC that compacts the entire heap. The heap has to
> be allocated as one consecutive memory area and if you're unlucky there can
> be some long lived object allocated near the top of the heap which makes it
> impossible to shrink the memory area.
>
> Recently there has been plenty of work in the G1 collector to deal with
> this situation. In 8u20 we started sorting the memory regions to increase
> the chances of allocating long lived objects at the bottom of the heap. In
> 8u40 we will introduce the possibility to decommit memory within the heap.
> This will make it possible to shrink the heap by cutting out a hole
> somewhere within the heap rather than being forced to only shrink from the
> top. This change will most likely be pushed to 8u40 within the next few
> days, it's already present in the JDK 9 repository.
>
> These features are all default behavior in G1, so if you haven't already,
> switch to G1 (-XX:+UseG1GC) and try out the latest bits to see if it solves
> your problem.
>
> Best regards,
> /Jesper
>
>
> Andrew Ash skrev 21/8/14 00:42:
>
>  // please redirect if this is the wrong list
>>
>> Hi JDK devs,
>>
>> I'm occasionally in a situation where I have a JVM running and want to
>> tell
>> it to "scale down" to use less heap than its current active set.  This
>> would be in the space between -Xms and -Xmx bounds.  Scaling "up" from
>> -Xms
>> towards -Xmx obviously happens automatically, and I've heard conventional
>> wisdom that you can't decrease heap size on a running JVM, but I've never
>> heard of any work being done to make that possible.
>>
>> The use case is when I have a long-running JVM in an Apache Mesos [1]
>> context, and Mesos wants to "take back" some memory resources from a
>> running JVM task.
>>
>> Is there any work being done to make scaling down possible?
>>
>> Cheers,
>> Andrew
>>
>>
>> [1] https://mesos.apache.org/
>>
>>