Running into Allocation Stalls during class unloading

Per Liden per.liden at oracle.com
Wed Feb 3 15:38:20 UTC 2021


Hi Mary,

Would it be possible for you to share GC logs (preferably generated 
using -Xlog:gc*)? That would help us understand if what you experience 
here is a consequence of what Charlie describes.

If that's the case, i.e. a prolonged concurrent GC cycle because of lots 
of classes to unload, it could be that the GC is simply kicking in a bit 
too late. If so, you might want to have a look at the 
-XX:+SoftMaxHeapSize option. This option tells ZGC's heuristics that it 
shouldn't aim at using the whole heap (-Xmx), but instead some lower 
number (-XX:SoftMaxHeapSize). This will increase ZGC's tolerance to 
variance in GC cycle length and make it more resilient to allocation spikes.

cheers,
Per

On 2/3/21 3:40 PM, charlie hunt wrote:
> Hi Mary,
> 
> No, allocation stalls where there is concurrent class unloading is not 
> an expected occurrence.
> 
> What may be a possibility here, and I will try to explain what I am 
> thinking.
> 
> With the introduction of concurrent class unloading, the elapsed time it 
> takes ZGC to complete a concurrent collection cycle may be slightly 
> longer than when class unloading was a GC pause. If your application 
> happened to be very close to a point where ZGC was just ahead of "losing 
> the race" and exhausting Java heap space, then the additional concurrent 
> class unloading work may be just enough for ZGC to lose that race. One 
> thing to keep in mind here is that concurrent class unloading removes a 
> GC pause, the pause that did class unloading.
> 
> hths,
> 
> charlie
> 
> On 2/3/21 8:10 AM, Mary Sunitha Joseph wrote:
>> Hi Charlie,
>>
>> Thank you for going over our case and for the recommendations. Increasing
>> the heap size or the number of ZGC threads is definitely something we can
>> try out.
>>
>> I'm also trying to understand if the allocation stalls during the time of
>> class unloading is an expected occurrence. It's almost as if at that 
>> point
>> ZGC's entire focus is on class unloading and not clearing out the heap
>> which leads to that spike in heap and subsequent allocation stalls. 
>> Perhaps
>> class unloading and freeing heap are not concurrent themselves and ZGC is
>> able to do one or the other?
>>
>> Regards,
>> Mary
>>
>> On Wed, Feb 3, 2021 at 6:55 AM <zgc-dev-request at openjdk.java.net> wrote:
>>
>>> Send zgc-dev mailing list submissions to
>>>          zgc-dev at openjdk.java.net
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>          https://mail.openjdk.java.net/mailman/listinfo/zgc-dev
>>> or, via email, send a message with subject or body 'help' to
>>>          zgc-dev-request at openjdk.java.net
>>>
>>> You can reach the person managing the list at
>>>          zgc-dev-owner at openjdk.java.net
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of zgc-dev digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>>     1. Running into Allocation Stalls during class unloading
>>>        (Mary Sunitha Joseph)
>>>     2. Re: Running into Allocation Stalls during class unloading
>>>        (charlie hunt)
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Tue, 2 Feb 2021 14:30:40 -0500
>>> From: Mary Sunitha Joseph <mary.joseph at fiixsoftware.com>
>>> To: zgc-dev at openjdk.java.net
>>> Subject: Running into Allocation Stalls during class unloading
>>> Message-ID:
>>>          <CAL2XQ3sa_txDTQQPLK2SEVkpmDnotCDKDqMKbGLrXQykt=
>>> MXcA at mail.gmail.com>
>>> Content-Type: text/plain; charset="UTF-8"
>>>
>>> Hi team,
>>>
>>> Our Production application runs on a 320G heap and uses ZGC with large
>>> pages enabled. We have not done any tuning and are using ZGC with 
>>> defaults.
>>> Since upgrading to JDK 15.0.1 we've started to notice that once a day 
>>> the
>>> app experiences allocation stalls (during peak hours) and this 
>>> happens when
>>> there is a huge drop in the number of classes loaded. We have a 
>>> bi-monthly
>>> release cycle and can see that the allocation stalls start small a 
>>> business
>>> day after a  release and slowly increase as the week progresses.
>>>
>>> At the moment the app seems to be doing fine but it could escalate 
>>> anytime
>>> by the looks of it. There is an increase in the app's response time 
>>> as well
>>> at the same time and a small spike in heap which seem like side effects.
>>> Any pointers in terms of tuning would be much appreciated.
>>>
>>> The app currently always makes use of at least 200G of heap space which
>>> leaves a 37% head space for ZGC.
>>>
>>>
>>> Regards
>>> Mary
>>> -- 
>>>
>>> Mary Sunitha Joseph (She/her)
>>>
>>> Lead Developer
>>>
>>> Fiix Software
>>>
>>> p: 1 (855) 884-5619
>>>
>>> e: mary.joseph at fiixsoftware.com
>>>
>>> w: www.fiixsoftware.com
>>>
>>> <
>>> https://www.fiixsoftware.com/foresight/#insights?utm_source=signature&utm_medium=email 
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 2
>>> Date: Tue, 2 Feb 2021 15:02:37 -0600
>>> From: charlie hunt <charlie.hunt at oracle.com>
>>> To: zgc-dev at openjdk.java.net
>>> Subject: Re: Running into Allocation Stalls during class unloading
>>> Message-ID: <c9040834-5b30-6b6e-6804-9fd55233873f at oracle.com>
>>> Content-Type: text/plain; charset=utf-8; format=flowed
>>>
>>> Hi Mary,
>>>
>>> Thanks for reaching out.
>>>
>>> Since you are observing allocation stalls, there a couple options to
>>> consider.
>>>
>>> 1.) If you have CPU cycles available, you can increase the number of
>>> concurrent GC threads. You can see the default number ZGC is currently
>>> using by doing:? java -XX:+UseZGC -XX:+PrintFlagsFinal -version | grep
>>> -i concgcthreads.? Increasing the number of concurrent GC threads should
>>> allow ZGC to do its concurrent work before the Java heap space becomes
>>> exhausted resulting in allocation stalls. But, additional concurrent GC
>>> threads will use more CPU.
>>>
>>> 2.) Another option is to size the Java heap larger, if you have the
>>> available RAM on the system. By increasing the size of the Java heap,
>>> you also increase the time the concurrent GC threads can do their work
>>> to free space before exhausting Java heap space (which results in
>>> allocation stalls).
>>>
>>> 3.) Another option is profile the application and look for opportunities
>>> to reduce unnecessary object allocations. This will reduce the speed at
>>> which the Java heap fills that available free space and thus allows
>>> ZGC's concurrent GC threads to keep up and avoid allocation stalls.
>>>
>>> Fwiw, I tend to like the first two options better since I would rather
>>> see folks write their Java application(s) in their most natural form and
>>> let the JVM figure out how to best run the Java app.
>>>
>>> Also, as a general comment, having 37% head room for ZGC to operate is
>>> not a lot of space. Whether that is enough space largely depends on the
>>> application, i.e. its allocation rate, object lifetimes, amount live
>>> data in the Java heap, etc., and whether concurrent GC threads can keep
>>> up with the pace of allocations with the amount of Java heap space
>>> that's available.
>>>
>>> hths,
>>>
>>> charlie
>>>
>>> On 2/2/21 1:30 PM, Mary Sunitha Joseph wrote:
>>>> Hi team,
>>>>
>>>> Our Production application runs on a 320G heap and uses ZGC with large
>>>> pages enabled. We have not done any tuning and are using ZGC with
>>> defaults.
>>>> Since upgrading to JDK 15.0.1 we've started to notice that once a 
>>>> day the
>>>> app experiences allocation stalls (during peak hours) and this happens
>>> when
>>>> there is a huge drop in the number of classes loaded. We have a
>>> bi-monthly
>>>> release cycle and can see that the allocation stalls start small a
>>> business
>>>> day after a  release and slowly increase as the week progresses.
>>>>
>>>> At the moment the app seems to be doing fine but it could escalate
>>> anytime
>>>> by the looks of it. There is an increase in the app's response time as
>>> well
>>>> at the same time and a small spike in heap which seem like side 
>>>> effects.
>>>> Any pointers in terms of tuning would be much appreciated.
>>>>
>>>> The app currently always makes use of at least 200G of heap space which
>>>> leaves a 37% head space for ZGC.
>>>>
>>>>
>>>> Regards
>>>> Mary
>>>
>>> End of zgc-dev Digest, Vol 36, Issue 1
>>> **************************************
>>>
>>


More information about the zgc-dev mailing list