G1GC fine tuning under heavy load

[David Weeda]

Hello Ahmed,

What we have done is following the documentation at
https://www.oracle.com/technetwork/articles/java/g1gc-1984535.html
We noted the following comment. To us it meant that ideally your HEAP size should be a power of 2 also. We settled on 2 x 8GB.

-XX:G1HeapRegionSize=n
Sets the size of a G1 region. The value will be a power of two and can range from 1MB to 32MB. The goal is to have around 2048 regions based on the minimum Java heap size.
From our java trace
J  [0.013s][info   ][gc,heap] Heap region size: 4M
J  [0.128s][info   ][gc     ] Using G1

Regards,
David Weeda
SAP Technical Architect

From: hotspot-gc-use <hotspot-gc-use-bounces at openjdk.java.net> On Behalf Of Wajih Ahmed
Sent: Wednesday, 12 September 2018 11:26 PM
To: hotspot-gc-use at openjdk.java.net
Subject: G1GC fine tuning under heavy load

Hello,

I have an application running on two nodes in a kubernetes cluster. It is handling about 70 million requests per day.  I have noticed a gradual decline in the throughput so much so that in about 7 days the througput falls about 50%.  Although large percent of this decline is in the first hour and then a gradual decline.
This graph<https://drive.google.com/open?id=19pG4j2ezNj-jm69Br7HqGKtR_c7L_-r6> to shows this pattern.  Some of the decline i can attribute to the application and use case itself. As database starts growing rapidly the system come under memory and cpu pressure and the database itself is also a java application.  So perhaps ignoring the decline of the first hour is prudent but i am still interested in seeing if i can tune the jvm of the app so that the throughput is more linear after the first hour.

I am also providing a gceasy.io<http://gceasy.io> report<https://drive.google.com/open?id=1s0akdn6ztj2-oeOHwEjFMqbDRpYOweJJ> that will
give the required information about GC activity.  You will see i have done some rudementary tuning already.

What i am curious about is if the young gen size needs to be reduced by tunring G1NewSizePercent to reduce the duration of the pauses in particular the object copy stage.

Secondly what GCEasy is calling "consecutive full gc" don't appear to be full GC's.  But it might be CMS (initial-mark) activity which accouts for most of the GC activity and has some long pause times.  Will increasing InitiatingHeapOccupancyPercent be recommended to reduce this activity and give the application more time?

Any other advise will be helpful as i start to learn and unfold the mystries of GC tuning :-)

Just in case you don't want to open the pdf report these are my JVM args

-XX:G1MixedGCCountTarget=12 -XX:InitialHeapSize=7516192768 -XX:MaxGCPauseMillis=200 -XX:MaxHeapSize=7516192768 - XX:MetaspaceSize=268435456 -XX:+PrintAdaptiveSizePolicy -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintPromotionFailure -XX:+PrintTenuringDistribution -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:- UseNUMA


Regards

**********************************************************************
This message is intended for the addressee named and may contain
privileged information or confidential information or both. If you
are not the intended recipient please delete it and notify the sender.
**********************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20180913/2f5d9d81/attachment.html>