Running into Allocation Stalls during class unloading

[charlie hunt]

Hi Mary,

Thanks for reaching out.

Since you are observing allocation stalls, there a couple options to 
consider.

1.) If you have CPU cycles available, you can increase the number of 
concurrent GC threads. You can see the default number ZGC is currently 
using by doing:  java -XX:+UseZGC -XX:+PrintFlagsFinal -version | grep 
-i concgcthreads.  Increasing the number of concurrent GC threads should 
allow ZGC to do its concurrent work before the Java heap space becomes 
exhausted resulting in allocation stalls. But, additional concurrent GC 
threads will use more CPU.

2.) Another option is to size the Java heap larger, if you have the 
available RAM on the system. By increasing the size of the Java heap, 
you also increase the time the concurrent GC threads can do their work 
to free space before exhausting Java heap space (which results in 
allocation stalls).

3.) Another option is profile the application and look for opportunities 
to reduce unnecessary object allocations. This will reduce the speed at 
which the Java heap fills that available free space and thus allows 
ZGC's concurrent GC threads to keep up and avoid allocation stalls.

Fwiw, I tend to like the first two options better since I would rather 
see folks write their Java application(s) in their most natural form and 
let the JVM figure out how to best run the Java app.

Also, as a general comment, having 37% head room for ZGC to operate is 
not a lot of space. Whether that is enough space largely depends on the 
application, i.e. its allocation rate, object lifetimes, amount live 
data in the Java heap, etc., and whether concurrent GC threads can keep 
up with the pace of allocations with the amount of Java heap space 
that's available.

hths,

charlie

On 2/2/21 1:30 PM, Mary Sunitha Joseph wrote:
> Hi team,
>
> Our Production application runs on a 320G heap and uses ZGC with large
> pages enabled. We have not done any tuning and are using ZGC with defaults.
> Since upgrading to JDK 15.0.1 we've started to notice that once a day the
> app experiences allocation stalls (during peak hours) and this happens when
> there is a huge drop in the number of classes loaded. We have a bi-monthly
> release cycle and can see that the allocation stalls start small a business
> day after a  release and slowly increase as the week progresses.
>
> At the moment the app seems to be doing fine but it could escalate anytime
> by the looks of it. There is an increase in the app's response time as well
> at the same time and a small spike in heap which seem like side effects.
> Any pointers in terms of tuning would be much appreciated.
>
> The app currently always makes use of at least 200G of heap space which
> leaves a 37% head space for ZGC.
>
>
> Regards
> Mary