G1 GC Tuning parameters

[Thomas Schatzl]

Hi Narashimha,

On Sat, 2018-08-04 at 07:58 +0000, Narasimha C Achi wrote:
> Hi Team,
>  
> Please find attached logs. java web services are deployed in these
> servers.
> Attachment 1: shows to-space exhausted and allocation failure
> Attachment2: Log GC Pauses.
>  
> Please find below G1 GC setting ini all our 4 managed servers.
>  
> -XX:GCLogFileSize=10485760 -XX:InitialHeapSize=6442450944
> -XX:MaxGCPauseMillis=50 -XX:MaxHeapSize=6442450944
> -XX:NumberOfGCLogFiles=10 -XX:+PrintGC -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps -XX:+ReduceSignalUsage
> -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC
> -XX:+UseGCLogFileRotation -XX:concGCThread=4 –XX:+ExplicitGC
>  
> Our observation is that heap utilization is more and free Heap
> Percentage comes to 8-9% and suddenly goes to 34%. Does this create
> any issues like outofmemory error and server crash?
> How to fine tune these parameters and what other parameters to change
> for proper GC process with less stop world times.
>  
> Each server serves almost 50000 requests every day. Few times we
> observed to-space exhausted and Allocation Failure in GC logs.
>  
> Please provide us details about any other parameters to tune to avoid
> GC Issues.

paidj2ee_server1*.log:

  from the logs I can observe that even after full gc (full heap
compaction) the heap does not significantly decrease.
Actually, the heap usage after full gc grows.

E.g. after the fourth full gc, the heap is ~5.6GB full out of 6GB (and
young gen size is 300M, so you have 200M left).

There are mixed gcs, but they typically do not free much (a few megs),
and the mixed gc cycle (the amount of times G1 tries to clean out old
gen) is typically just one.

This strongly suggests this setup has either a memory leak, the heap is
too small or both.

serv1459.txt

  all full gcs are triggered by the application by a system.gc() call.
Either fix them in your application, or one could mitigate these with
either -XX:+DisableExplicitGC or  -XX:+ExplicitGCInvokesConcurrent.

Other than that there is also a frightening trend of the application
not leveling out its application memory usage. I.e. the VM is on the
way to memory exhaustion in another ~200 hours (the setup "looses"
~10MB/hour) like the other already did.

Are these applications stable on a different collector? I would be
surprised.

Hth,
  Thomas