Tuning ShenandoahGC with 420 GB heaps

[Aleksey Shipilev]

On 3/4/19 5:48 PM, Jeroen Borgers wrote:
> Please find 3 gc log files at:
> www.jpinpoint.com/resources/gc_Shenandoah_Thu.log.0.current.zip
> <http://www.jpinpoint.com/resources/gc_Shenandoah_Thu.log.0.current.zip>
> gc_Shenandoah_Thu.log.1.zip
> gc_shenandoah_Weekend_25.zip

Briefly looking at one of the logs, Weekend_25.zip.

*) First things first, command line options:

CommandLine flags: -XX:-AlwaysPreTouch -XX:+ClassUnloadingWithConcurrentMark -XX:ConcGCThreads=16
-XX:GCLogFileSize=10485760 -XX:InitialHeapSize=450971566080 -XX:InitialTenuringThreshold=2
-XX:+LogVMOutput -XX:+ManagementServer -XX:MaxHeapSize=450971566080 -XX:MaxTenuringThreshold=2
-XX:NumberOfGCLogFiles=10 -XX:ParallelGCThreads=44 -XX:+PrintGC -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps -XX:ShenandoahAllocSpikeFactor=5 -XX:+ShenandoahAllocationTrace
-XX:ShenandoahPacingMaxDelay=50 -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions
-XX:+UseGCLogFileRotation -XX:+UseNUMA -XX:+UseNUMAInterleaving -XX:+UseShenandoahGC
-XX:+UseTransparentHugePages

 - Consider enabling -XX:+AlwaysPreTouch, especially as you are running with NUMA turned on and THP
turned on. If memory allocator and/or defragger kicks in at unfortunate times, it might stall the
collector enough to trip off the the concurrent mode;

 - These options are useless for Shenandoah: "-XX:InitialTenuringThreshold=2 -XX:MaxTenuringThreshold=2"

 - "-XX:InitialHeapSize=450971566080 -XX:MaxHeapSize=450971566080", but machine has "physical
528157524k(460445396k free)". There is some native memory spent on top of heap size, are you sure
the machine never swaps?


*) 8u191 Shenandoah is a bit old, which might not have all the performance touchups. Sorry about
that. Can you try our 8u nightlies? https://builds.shipilev.net/openjdk-shenandoah-jdk8/

*) From very far out, it seems that the normal GC cycle takes around 16 seconds in that config. And
the live data set is around 260G (the heap size after Full GC) of 420G max. Which means, any
allocation spike at 10+ GB/sec would tank the whole thing. I'd say if you cannot crank up the heap,
then you'd have to choose whether you want much larger allocation pacing delay (i.e. seconds) or
accept more Degen/Full GCs.

*) Looking at progression of some counters:

 $ grep "Actual Free" gc_shenandoah_Weekend_25.logfull | less

...it seems it slowly goes down, until Full GC recovers from it. We have seen those as heuristics
issues before, and it should be fixed in recent 8u's, but probably not in 8u191.


-Aleksey