Large heap size, slow concurrent marking causing frequent full GC

[James Sun]

Dear

We observed frequent full GCs due to long concurrent marking phase (about 30 seconds to a minute). The GC log with heap histogram during full GC is attached.
The Java version we use is 8_144 with G1 GC.  The machines are with 56 cores and a heap size around 180 – 210GB.

Example concurrent mark duration:
2017-11-16T09:32:04.565-0800: 167543.159: [GC concurrent-mark-end, 45.7020802 secs]
2017-11-16T09:33:16.314-0800: 167614.908: [GC concurrent-mark-end, 51.0809053 secs]
2017-11-16T09:34:28.343-0800: 167686.938: [GC concurrent-mark-end, 48.7335047 secs]


Wonder if anyone could help in terms of:

  1.  How in general we can make concurrent marking faster. We bumped up the ConcGCThread to 20 but it didn’t help that much.
  2.  We also turned on -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark but nothing related to marking shows up.
  3.  General advice in tuning GC in other aspects

Thanks in advance

James


Here is the JVM config we have

-Xss2048k
-XX:MaxMetaspaceSize=4G
-XX:+PreserveFramePointer
-XX:-UseBiasedLocking
-XX:+PrintGCApplicationConcurrentTime
-XX:+PrintGCApplicationStoppedTime
-XX:+UnlockExperimentalVMOptions
-XX:+UseG1GC
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-XX:+UseGCOverheadLimit
-XX:+ExitOnOutOfMemoryError
-agentpath:/packages/presto.presto/bin/libjvmkill.so
-agentpath:/packages/presto.presto/bin/libperfagent.so
-XX:+PrintReferenceGC
-XX:+PrintGCCause
-XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
-XX:+PrintClassHistogramAfterFullGC
-XX:+PrintClassHistogramBeforeFullGC
-XX:PrintFLSStatistics=2
-XX:+PrintAdaptiveSizePolicy
-XX:+PrintSafepointStatistics
-XX:PrintSafepointStatisticsCount=1
-XX:+PrintJNIGCStalls
-XX:+UnlockDiagnosticVMOptions
-XX:+AlwaysPreTouch
-XX:+G1SummarizeRSetStats
-XX:G1SummarizeRSetStatsPeriod=100
-Dorg.eclipse.jetty.io.SelectorManager.submitKeyUpdates=true
-XX:-OmitStackTraceInFastThrow
-XX:ReservedCodeCacheSize=1G
-Djdk.nio.maxCachedBufferSize=30000000
-XX:G1MaxNewSizePercent=20
-XX:G1HeapRegionSize=32M
-Xms180G
-Xmx180G
-XX:MarkStackSize=64M
-XX:G1HeapWastePercent=2
-XX:ConcGCThreads=20
-XX:MaxGCPauseMillis=500
-XX:GCLockerRetryAllocationCount=5
-XX:MarkStackSizeMax=256M
-XX:G1OldCSetRegionThresholdPercent=20
-XX:InitiatingHeapOccupancyPercent=40

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20171116/2e739a4f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log.zip
Type: application/zip
Size: 48297 bytes
Desc: log.zip
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20171116/2e739a4f/log-0001.zip>