Gen ZGC EAB in K8S

[Peter Booth]

The default geode heartbeat timeout interval is 5 seconds, which is an eternity.

Some points/questions:

I’d recommend using either Solarflare’s sysjitter tool or Gil Tene’s jhiccup to quantify your OS jitter
Can you capture the output of vmstat 1 60 ?
What kind of EC2 instances are you using? Are they RHEL?
How much physical RAM does each instance have?
Do you have THP enabled?
What is the value of vm.min_free_kbytes ?


How often do you see missed heartbeats?
What length of time do you see the Adjusting Workers message?







> On Mar 13, 2023, at 4:42 AM, Evaristo José Camarero <evaristojosec at yahoo.es> wrote:
> 
> Hi there,
> 
> We are trying latest ZGC EAB for testing an Apache Geode Cluster (distributed Key Value store with similar use cases that Apache Cassandra)
> 
> We are using K8s, and we have PODs with 32 cores request (limit with 60 cores) per data node with 150GB heap per node.
> 
>           - -Xmx152000m
>           - -Xms152000m
>           - -XX:+UseZGC
>           - -XX:SoftMaxHeapSize=136000m
>           - -XX:ZAllocationSpikeTolerance=4.0   // We have some spiky workloads periodically 
>           - -XX:+UseNUMA
> 
> 
> ZGC is working great in regard GC pauses with no allocation stalls at all during almost all the time. We observe higher CPU utilization that G1 (Around 20% for a heavy workload using flag -XX:ZAllocationSpikeTolerance=4.0 that maybe is making ZGC more hungry than needed. We will play further with this)
> 
> 
> BUT from time to time we see that Geode Clusters are missing heartbeats between and Geode logic shutdowns the JVM of the node with missing heartbeats. We believe that the main problem could be CPU starvation, because some seconds before this is happening we observe ZGC to use more workers for making the job done
> 
> [2023-03-12T18:29:38.980+0000] Adjusting Workers for Young Generation: 1 -> 2
> [2023-03-12T18:29:39.781+0000] Adjusting Workers for Young Generation: 1 -> 3
> [2023-03-12T18:29:40.181+0000] Adjusting Workers for Young Generation: 1 -> 4
> [2023-03-12T18:29:40.382+0000] Adjusting Workers for Young Generation: 1 -> 5
> [2023-03-12T18:29:40.582+0000] Adjusting Workers for Young Generation: 1 -> 6
> [2023-03-12T18:29:40.782+0000] Adjusting Workers for Young Generation: 1 -> 7
> [2023-03-12T18:29:40.882+0000] Adjusting Workers for Young Generation: 1 -> 8
> [2023-03-12T18:29:40.982+0000] Adjusting Workers for Young Generation: 1 -> 10
> [2023-03-12T18:29:41.083+0000] Adjusting Workers for Young Generation: 1 -> 13
> [2023-03-12T18:29:41.183+0000] Adjusting Workers for Young Generation: 1 -> 16
> 
> 
> As commented we are using K8S with PODs that have 32 cores for request and 60 cores for limit (and it also true that our K8s workers are close to the limit in CPU utilization). ZGC is assuming on booting that machine has 60 cores (as logged). What is the best way to configure the ZGC to provide a hint to be tuned for a host with 32 cores (basically the 60 cores limit is just to avoid K8s produced throttling)? Is it using ParallelGCThreads flag? Any other thoughts?
> 
> Regards,
> 
> Evaristo
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/zgc-dev/attachments/20230313/ccefb930/attachment-0001.htm>