Running into Allocation Stalls during class unloading

Wed Feb 3 14:40:53 UTC 2021

Hi Mary,

No, allocation stalls where there is concurrent class unloading is not 
an expected occurrence.

What may be a possibility here, and I will try to explain what I am 
thinking.

With the introduction of concurrent class unloading, the elapsed time it 
takes ZGC to complete a concurrent collection cycle may be slightly 
longer than when class unloading was a GC pause. If your application 
happened to be very close to a point where ZGC was just ahead of "losing 
the race" and exhausting Java heap space, then the additional concurrent 
class unloading work may be just enough for ZGC to lose that race. One 
thing to keep in mind here is that concurrent class unloading removes a 
GC pause, the pause that did class unloading.

hths,

charlie

On 2/3/21 8:10 AM, Mary Sunitha Joseph wrote:
> Hi Charlie,
>
> Thank you for going over our case and for the recommendations. Increasing
> the heap size or the number of ZGC threads is definitely something we can
> try out.
>
> I'm also trying to understand if the allocation stalls during the time of
> class unloading is an expected occurrence. It's almost as if at that point
> ZGC's entire focus is on class unloading and not clearing out the heap
> which leads to that spike in heap and subsequent allocation stalls. Perhaps
> class unloading and freeing heap are not concurrent themselves and ZGC is
> able to do one or the other?
>
> Regards,
> Mary
>
> On Wed, Feb 3, 2021 at 6:55 AM <zgc-dev-request at openjdk.java.net> wrote:
>
>> Send zgc-dev mailing list submissions to
>>          zgc-dev at openjdk.java.net
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>          https://mail.openjdk.java.net/mailman/listinfo/zgc-dev
>> or, via email, send a message with subject or body 'help' to
>>          zgc-dev-request at openjdk.java.net
>>
>> You can reach the person managing the list at
>>          zgc-dev-owner at openjdk.java.net
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of zgc-dev digest..."
>>
>>
>> Today's Topics:
>>
>>     1. Running into Allocation Stalls during class unloading
>>        (Mary Sunitha Joseph)
>>     2. Re: Running into Allocation Stalls during class unloading
>>        (charlie hunt)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Tue, 2 Feb 2021 14:30:40 -0500
>> From: Mary Sunitha Joseph <mary.joseph at fiixsoftware.com>
>> To: zgc-dev at openjdk.java.net
>> Subject: Running into Allocation Stalls during class unloading
>> Message-ID:
>>          <CAL2XQ3sa_txDTQQPLK2SEVkpmDnotCDKDqMKbGLrXQykt=
>> MXcA at mail.gmail.com>
>> Content-Type: text/plain; charset="UTF-8"
>>
>> Hi team,
>>
>> Our Production application runs on a 320G heap and uses ZGC with large
>> pages enabled. We have not done any tuning and are using ZGC with defaults.
>> Since upgrading to JDK 15.0.1 we've started to notice that once a day the
>> app experiences allocation stalls (during peak hours) and this happens when
>> there is a huge drop in the number of classes loaded. We have a bi-monthly
>> release cycle and can see that the allocation stalls start small a business
>> day after a  release and slowly increase as the week progresses.
>>
>> At the moment the app seems to be doing fine but it could escalate anytime
>> by the looks of it. There is an increase in the app's response time as well
>> at the same time and a small spike in heap which seem like side effects.
>> Any pointers in terms of tuning would be much appreciated.
>>
>> The app currently always makes use of at least 200G of heap space which
>> leaves a 37% head space for ZGC.
>>
>>
>> Regards
>> Mary
>> --
>>
>> Mary Sunitha Joseph (She/her)
>>
>> Lead Developer
>>
>> Fiix Software
>>
>> p: 1 (855) 884-5619
>>
>> e: mary.joseph at fiixsoftware.com
>>
>> w: www.fiixsoftware.com
>>
>> <
>> https://www.fiixsoftware.com/foresight/#insights?utm_source=signature&utm_medium=email
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Tue, 2 Feb 2021 15:02:37 -0600
>> From: charlie hunt <charlie.hunt at oracle.com>
>> To: zgc-dev at openjdk.java.net
>> Subject: Re: Running into Allocation Stalls during class unloading
>> Message-ID: <c9040834-5b30-6b6e-6804-9fd55233873f at oracle.com>
>> Content-Type: text/plain; charset=utf-8; format=flowed
>>
>> Hi Mary,
>>
>> Thanks for reaching out.
>>
>> Since you are observing allocation stalls, there a couple options to
>> consider.
>>
>> 1.) If you have CPU cycles available, you can increase the number of
>> concurrent GC threads. You can see the default number ZGC is currently
>> using by doing:? java -XX:+UseZGC -XX:+PrintFlagsFinal -version | grep
>> -i concgcthreads.? Increasing the number of concurrent GC threads should
>> allow ZGC to do its concurrent work before the Java heap space becomes
>> exhausted resulting in allocation stalls. But, additional concurrent GC
>> threads will use more CPU.
>>
>> 2.) Another option is to size the Java heap larger, if you have the
>> available RAM on the system. By increasing the size of the Java heap,
>> you also increase the time the concurrent GC threads can do their work
>> to free space before exhausting Java heap space (which results in
>> allocation stalls).
>>
>> 3.) Another option is profile the application and look for opportunities
>> to reduce unnecessary object allocations. This will reduce the speed at
>> which the Java heap fills that available free space and thus allows
>> ZGC's concurrent GC threads to keep up and avoid allocation stalls.
>>
>> Fwiw, I tend to like the first two options better since I would rather
>> see folks write their Java application(s) in their most natural form and
>> let the JVM figure out how to best run the Java app.
>>
>> Also, as a general comment, having 37% head room for ZGC to operate is
>> not a lot of space. Whether that is enough space largely depends on the
>> application, i.e. its allocation rate, object lifetimes, amount live
>> data in the Java heap, etc., and whether concurrent GC threads can keep
>> up with the pace of allocations with the amount of Java heap space
>> that's available.
>>
>> hths,
>>
>> charlie
>>
>> On 2/2/21 1:30 PM, Mary Sunitha Joseph wrote:
>>> Hi team,
>>>
>>> Our Production application runs on a 320G heap and uses ZGC with large
>>> pages enabled. We have not done any tuning and are using ZGC with
>> defaults.
>>> Since upgrading to JDK 15.0.1 we've started to notice that once a day the
>>> app experiences allocation stalls (during peak hours) and this happens
>> when
>>> there is a huge drop in the number of classes loaded. We have a
>> bi-monthly
>>> release cycle and can see that the allocation stalls start small a
>> business
>>> day after a  release and slowly increase as the week progresses.
>>>
>>> At the moment the app seems to be doing fine but it could escalate
>> anytime
>>> by the looks of it. There is an increase in the app's response time as
>> well
>>> at the same time and a small spike in heap which seem like side effects.
>>> Any pointers in terms of tuning would be much appreciated.
>>>
>>> The app currently always makes use of at least 200G of heap space which
>>> leaves a 37% head space for ZGC.
>>>
>>>
>>> Regards
>>> Mary
>>
>> End of zgc-dev Digest, Vol 36, Issue 1
>> **************************************
>>
>