RFC: Adaptively resize heap at any GC/SoftMaxHeapSize for G1

[Thomas Schatzl]

Hi Ruslan,

On 18.06.20 14:16, Ruslan Synytsky wrote:
> Hi Thomas and Liang, thank you for moving this improvement forward. A quick

:) Thanks.

> question regarding the naming: did we agree on how this parameter should be
> called? What happens when heap usage goes higher than SoftMaxHeapSize -
> OOMError or JVM gets a little bit more memory? If JVM throws OOMError I
> believe the right naming should be HardMaxHeapSize. Sorry in advance if I
> missed this point in the previous conversations.

SoftMaxHeapSize is not what you describe here - SoftMaxHeapSize is only 
an internal goal for heap sizing without guarantees. Hence the name 
*Soft*MaxHeapSize.

See https://bugs.openjdk.java.net/browse/JDK-8222145.

There has been no progress on anything like Current/HardMaxHeapSize.

> 
> Also, some news regarding analysis automation of memory usage efficiency
> I'm working on in the background. We built a relatively small script that
> collects memory usage metrics from many containers running inside the same
> large host machine. After executing it in one of our dev environments with
> about 150 containers we got interesting results - the used heap is very
> close to the committed heap while Xmx is much higher compared to committed
> value. Please note, almost all containers use JEP 346 improvement or
> javaagent which triggers GC at idle state in the older JDK versions.
> 
> [image: Screenshot 2020-06-18 at 13.20.19.jpg] >
> Zoomed
> 
> [image: Screenshot 2020-06-18 at 14.40.18.jpg]

While the screenshots have been scrubbed by the mailing list it's very 
nice to hear that you have had success with these approaches.

> There is a challenge to get metrics from a running java process. I as
> understand the most accurate metrics can be collected via MBean, for
> example
> 
> JMXConnector c =
> JMXConnectorFactory.newJMXConnector(createConnectionURL(host, port),
> null);
> c.connect();
> MBeanServerConnection mbsc = c.getMBeanServerConnection();
> MemoryMXBean mem = ManagementFactory.getPlatformMXBean(mbsc,
> MemoryMXBean.class);
> MemoryUsage heap = mem.*getHeapMemoryUsage*();
> 
> However, enabling JMX ManagementAgent via jcmd and connecting to JVM with a
> JMX client is a quite complex operation for getting such a simple metric
> about heap memory usage. Also, some java processes may already
> start ManagementAgent on a custom port with auth protection, so we can't
> collect statistics from such java processes without contacting the
> application owner (you can see the gaps on the chart). Do you know any
> other way for collecting accurate heap usage statistics from a running java
> process? We plan to run this analysis tool on productions with a large
> number of containers, so we can get a more realistic picture.
> 

Jcmd with the GC.heap_info command provides some information, probably 
not enough (I filed JDK-8248136) though. More information can be 
retrieved with the "VM.info" command, the detailed per-region printout 
which might be too much information.

There is also JFR with its event streaming API that could be an option, 
however it is JDK14 only (https://openjdk.java.net/jeps/349).

Finally, there is jstat to gather some information.

Thanks,
   Thomas