Very long full G1 GC pauses

Thu Nov 21 07:35:29 UTC 2024

Hi,

A quick look at the GCeasy pages show that the old-generation start to 
increase right before the FullGCs. This is likely caused by an increase 
in allocation rate (likely caused higher load on the application or a 
change in behavior) which lead to more frequent GCs and objects getting 
promoted to the old generation quicker.

Without the complete logs, its hard to say exactly what tuning could or 
should be done. We already seem to run the concurrent marking quite 
frequently, but lowering the InitiatingHeapOccupancyPercent could 
perhaps help avoid the Full GCs. You also seem to have some GCs caused 
by humongous allocations around the Full GCs, depending on how big those 
objects are increasing the G1 regions size could possibly help. Looking 
at your heap size the default region size should be 4m, so increasing it 
to 8m could help: -XX:G1HeapRegionSize=8m.

If you don't have the complete logs easy available, you can collect them 
into a separate file using -Xloggc:<filename>.

Hope this helps,
Stefan

On 2024-11-21 02:20, Mohammed wrote:
> Dear GC Community,
>
> I hope this message finds you well... I'm currently experiencing 
> performance issues related to very long garbage collection (GC) pauses 
> in our application, which uses the G1 on Java 8.
>
> https://gceasy.io/my-gc-report.jsp?p=YXJjaGl2ZWQvMjAyNC8xMS8xOC9qYXZhX2djIC0gMTh0aCBOb3YubG9nLS0xMi01Mi0xMA==&channel=WEB 
> <https://gceasy.io/my-gc-report.jsp?p=YXJjaGl2ZWQvMjAyNC8xMS8xOC9qYXZhX2djIC0gMTh0aCBOb3YubG9nLS0xMi01Mi0xMA==&channel=WEB>
>
> We increased the heap size to 14 GB but it didn't help:
>
> https://gceasy.io/my-gc-report.jsp?p=YXJjaGl2ZWQvMjAyNC8xMS8yMC9qYXZhX2djLmxvZy0tMTctNDEtMjk=&channel=WEB 
> <https://gceasy.io/my-gc-report.jsp?p=YXJjaGl2ZWQvMjAyNC8xMS8yMC9qYXZhX2djLmxvZy0tMTctNDEtMjk=&channel=WEB>
>
> I highly appreciate any insights or recommendations to fine tune it, 
> please.
>
> Thank you in advance for your time and support.
>
>
> OpenJDK 64-Bit Server VM (25.422-b05) for linux-amd64 JRE 
> (1.8.0_422-b05), built on Jul 10 2024 20:58:03 by "mockbuild" with gcc 
> 4.8.5 20150623 (Red Hat 4.8.5-44)
> Memory: 4k page, physical 22439452k(19045348k free), swap 
> 4194300k(4194300k free)
> CommandLine flags: -XX:InitialHeapSize=15032385536 
> -XX:InitialTenuringThreshold=4 -XX:InitiatingHeapOccupancyPercent=30 
> -XX:+ManagementServer -XX:MaxHeapSize=15032385536 
> -XX:MaxTenuringThreshold=4 -XX:+ParallelRefProcEnabled -XX:+PrintGC 
> -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps 
> -XX:+PrintHeapAtGC -XX:+PrintReferenceGC 
> -XX:+PrintTenuringDistribution -XX:+UseCompressedClassPointers 
> -XX:+UseCompressedOops -XX:+UseG1GC
>
>
> 2024-11-20T10:01:23.823+0100: 1.757: [GC pause (Metadata GC Threshold) 
> (young) (initial-mark)
> 2024-11-20T10:01:23.839+0100: 1.773: [SoftReference, 0 refs, 0.0010719 
> secs]
> 2024-11-20T10:01:23.840+0100: 1.774: [WeakReference, 492 refs, 
> 0.0005143 secs]
> 2024-11-20T10:01:23.841+0100: 1.775: [FinalReference, 1914 refs, 
> 0.0025155 secs]
> 2024-11-20T10:01:23.843+0100: 1.777: [PhantomReference, 0 refs, 
> 0.0006570 secs]
> 2024-11-20T10:01:23.844+0100: 1.778: [JNI Weak Reference, 0.0000576 
> secs], 0.0223584 secs]
> 2024-11-20T10:01:23.846+0100: 1.779: [GC 
> concurrent-root-region-scan-start]
> 2024-11-20T10:01:23.850+0100: 1.783: [GC 
> concurrent-root-region-scan-end, 0.0039879 secs]
> 2024-11-20T10:01:23.850+0100: 1.783: [GC concurrent-mark-start]
> 2024-11-20T10:01:23.855+0100: 1.789: [GC concurrent-mark-end, 
> 0.0053844 secs]
> 2024-11-20T10:01:29.883+0100: 7.817: [GC pause (G1 Evacuation Pause) 
> (young)
> 2024-11-20T10:01:31.675+0100: 9.609: [GC pause (G1 Evacuation Pause) 
> (young)
> 2024-11-20T10:01:33.353+0100: 11.286: [GC pause (G1 Evacuation Pause) 
> (young)
> 2024-11-20T10:01:35.989+0100: 13.923: [GC pause (G1 Evacuation Pause) 
> (young)
> 2024-11-20T10:01:37.481+0100: 15.414: [GC pause (G1 Evacuation Pause) 
> (young)
> 2024-11-20T10:01:40.874+0100: 18.807: [GC pause (G1 Evacuation Pause) 
> (young)
> 2024-11-20T10:01:43.229+0100: 21.163: [GC pause (G1 Evacuation Pause) 
> (young)
> 2024-11-20T10:01:46.703+0100: 24.637: [GC pause (Metadata GC 
> Threshold) (young) (initial-mark)
> 2024-11-20T10:01:55.044+0100: 32.977: [GC pause (G1 Evacuation Pause) 
> (young)
>
> 2024-11-20T15:00:04.646+0100: 17922.579: [GC pause (G1 Evacuation 
> Pause) (young) (initial-mark)
> 2024-11-20T15:00:04.689+0100: 17922.622: [GC 
> concurrent-root-region-scan-start]
> 2024-11-20T15:00:04.725+0100: 17922.659: [GC 
> concurrent-root-region-scan-end, 0.0365063 secs]
> 2024-11-20T15:00:04.725+0100: 17922.659: [GC concurrent-mark-start]
> 2024-11-20T15:00:06.107+0100: 17924.040: [GC pause (G1 Evacuation 
> Pause) (young)
> 2024-11-20T15:00:12.337+0100: 17930.270: [GC cleanup 
> 10530M->10490M(14336M), 0.0788429 secs]
> 2024-11-20T15:00:12.416+0100: 17930.349: [GC concurrent-cleanup-start]
> 2024-11-20T15:00:15.700+0100: 17933.633: [GC pause (G1 Evacuation 
> Pause) (mixed)
> 2024-11-20T15:00:19.262+0100: 17937.196: [GC pause (G1 Evacuation 
> Pause) (mixed)
> 2024-11-20T15:00:21.759+0100: 17939.692: [GC pause (G1 Evacuation 
> Pause) (young) (initial-mark)
> 2024-11-20T15:00:29.523+0100: 17947.457: [GC concurrent-mark-end, 
> 7.6961570 secs]
> 2024-11-20T15:00:29.688+0100: 17947.622: [GC cleanup 
> 10369M->10329M(14336M), 0.0734577 secs]
> 2024-11-20T15:00:47.950+0100: 17965.884: [GC pause (G1 Evacuation 
> Pause) (young) (initial-mark)
> 2024-11-20T15:00:55.434+0100: 17973.368: [GC concurrent-mark-end, 
> 7.3457392 secs]
>
> 2024-11-20T18:02:39.789+0100: [Full GC (Allocation Failure) 
> 13373M->10294M(14336M), 15.5035776 secs]
> 2024-11-20T18:02:39.789+0100: [SoftReference, 13012 refs, 0.0020701 secs]
> 2024-11-20T18:02:39.789+0100: [WeakReference, 140239 refs, 0.0156066 secs]
> 2024-11-20T18:02:39.789+0100: [FinalReference, 3610 refs, 0.0003202 secs]
> 2024-11-20T18:02:39.789+0100: [PhantomReference, 194 refs, 0.0000238 secs]
> 2024-11-20T18:02:39.789+0100: [JNI Weak Reference, 0.0003217 secs]
>
> 2024-11-20T18:02:53.371+0100: [Full GC (Allocation Failure) 
> 13472M->10131M(14336M), 16.4023915 secs]
> 2024-11-20T18:02:53.371+0100: [SoftReference, 13771 refs, 0.0022936 secs]
> 2024-11-20T18:02:53.371+0100: [WeakReference, 138881 refs, 0.0158819 secs]
> 2024-11-20T18:02:53.371+0100: [FinalReference, 3644 refs, 0.0003447 secs]
> 2024-11-20T18:02:53.371+0100: [PhantomReference, 194 refs, 0.0000328 secs]
> 2024-11-20T18:02:53.371+0100: [JNI Weak Reference, 0.0003272 secs]
>
> 2024-11-20T18:03:12.492+0100: [Full GC (Allocation Failure) 
> 12600M->10282M(14336M), 14.9648411 secs]
> 2024-11-20T18:03:12.492+0100: [SoftReference, 13027 refs, 0.0022496 secs]
> 2024-11-20T18:03:12.492+0100: [WeakReference, 135613 refs, 0.0150724 secs]
> 2024-11-20T18:03:12.492+0100: [FinalReference, 3643 refs, 0.0003302 secs]
> 2024-11-20T18:03:12.492+0100: [PhantomReference, 194 refs, 0.0000216 secs]
> 2024-11-20T18:03:12.492+0100: [JNI Weak Reference, 0.0003229 secs]
>
> 2024-11-20T18:03:31.107+0100: [Full GC (Allocation Failure) 
> 13365M->10281M(14336M), 15.4983270 secs]
> 2024-11-20T18:03:31.107+0100: [SoftReference, 13110 refs, 0.0021606 secs]
> 2024-11-20T18:03:31.107+0100: [WeakReference, 135613 refs, 0.0149449 secs]
> 2024-11-20T18:03:31.107+0100: [FinalReference, 3641 refs, 0.0003340 secs]
> 2024-11-20T18:03:31.107+0100: [PhantomReference, 194 refs, 0.0000244 secs]
> 2024-11-20T18:03:31.107+0100: [JNI Weak Reference, 0.0003422 secs]
>
> 2024-11-20T18:03:50.243+0100: [Full GC (Allocation Failure) 
> 13360M->10280M(14336M), 15.1688211 secs]
> 2024-11-20T18:03:50.243+0100: [SoftReference, 13110 refs, 0.0020750 secs]
> 2024-11-20T18:03:50.243+0100: [WeakReference, 135612 refs, 0.0149417 secs]
> 2024-11-20T18:03:50.243+0100: [FinalReference, 3641 refs, 0.0003294 secs]
> 2024-11-20T18:03:50.243+0100: [PhantomReference, 194 refs, 0.0000235 secs]
> 2024-11-20T18:03:50.243+0100: [JNI Weak Reference, 0.0003195 secs]
>
>
> Best regards
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.org
> https://mail.openjdk.org/mailman/listinfo/hotspot-gc-use
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-use/attachments/20241121/9221c1d7/attachment.htm>