From lxw263044 at alibaba-inc.com  Thu Jan 13 02:58:25 2022
From: lxw263044 at alibaba-inc.com (=?UTF-8?B?6YCv5pmT5aiBKOWzsOmjjik=?=)
Date: Thu, 13 Jan 2022 10:58:25 +0800
Subject: =?UTF-8?B?WkdDIEFBcmNoNjQgaGFuZyBvbiBTUEVDamJiMjAxNQ==?=
Message-ID: <4ed59695-af34-43f4-8798-7655574a19a4.lxw263044@alibaba-inc.com>

Hi, I have tested zgc on jdk-15-GA and found occasionally SPECjbb2015 hangs when it is about to shut down. This is quite hard to reproduce, one out of 20 iterations probably. Since it only reproduces during shut down, I prefer to use "preset" mode to run SPECjbb to save time.
jdk-15-GA/bin/java -XX:+UseZGC -Xmx10g -Dspecjbb.controller.type=PRESET -Dspecjbb.controller.preset.ir=20000 -Dspecjbb.controller.preset.duration=60000 -jar specjbb2015.jar -m COMPOSITE
And SPECjbb log is as follow:
<Tue Jan 04 03:06:13 EST 2022> org.spec.jbb: Failed to stop executor service java.util.concurrent.ForkJoinPool at 9d2cbbc[Terminating, parallelism = 160, size = 2, active = 0, running = 2, steals = 40255, tasks = 0, submissions = 0], trying again
<Tue Jan 04 03:06:14 EST 2022> org.spec.jbb: Failed to stop executor service java.util.concurrent.ForkJoinPool at 9d2cbbc[Terminating, parallelism = 160, size = 2, active = 0, running = 2, steals = 40255, tasks = 0, submissions = 0], trying again

This seems quite similar to a previous issue: https://bugs.openjdk.java.net/browse/JDK-8214285, but on AArch64 other than Skylake. In addition, JDK-8214285 has been solved in jdk12, but we reproduced this issue on jdk-13.0.2+8 and jdk-14.0.2+12

Regards,Xiaowei Lu

From per.liden at oracle.com  Thu Jan 13 08:37:43 2022
From: per.liden at oracle.com (Per Liden)
Date: Thu, 13 Jan 2022 09:37:43 +0100
Subject: ZGC AArch64 hang on SPECjbb2015
In-Reply-To: <4ed59695-af34-43f4-8798-7655574a19a4.lxw263044@alibaba-inc.com>
References: <4ed59695-af34-43f4-8798-7655574a19a4.lxw263044@alibaba-inc.com>
Message-ID: <608d169c-2692-95c5-93b6-61208cd03a0f@oracle.com>

Hi,

Before digging deeper, please verify if this is a problem on the latest 
JDK 17. Also note that SPECjbb2015 1.03 contains a fix for a hang we 
discovered and reported some time ago. So please verify that you're 
using that version.

/Per


On 1/13/22 03:58, ???(??) wrote:
> Hi, I have tested zgc on jdk-15-GA and found occasionally SPECjbb2015 hangs when it is about to shut down. This is quite hard to reproduce, one out of 20 iterations probably. Since it only reproduces during shut down, I prefer to use "preset" mode to run SPECjbb to save time.
> jdk-15-GA/bin/java -XX:+UseZGC -Xmx10g -Dspecjbb.controller.type=PRESET -Dspecjbb.controller.preset.ir=20000 -Dspecjbb.controller.preset.duration=60000 -jar specjbb2015.jar -m COMPOSITE
> And SPECjbb log is as follow:
> <Tue Jan 04 03:06:13 EST 2022> org.spec.jbb: Failed to stop executor service java.util.concurrent.ForkJoinPool at 9d2cbbc[Terminating, parallelism = 160, size = 2, active = 0, running = 2, steals = 40255, tasks = 0, submissions = 0], trying again
> <Tue Jan 04 03:06:14 EST 2022> org.spec.jbb: Failed to stop executor service java.util.concurrent.ForkJoinPool at 9d2cbbc[Terminating, parallelism = 160, size = 2, active = 0, running = 2, steals = 40255, tasks = 0, submissions = 0], trying again
> 
> This seems quite similar to a previous issue: https://bugs.openjdk.java.net/browse/JDK-8214285, but on AArch64 other than Skylake. In addition, JDK-8214285 has been solved in jdk12, but we reproduced this issue on jdk-13.0.2+8 and jdk-14.0.2+12
> 
> Regards,Xiaowei Lu
> 

From lxw263044 at alibaba-inc.com  Fri Jan 14 06:23:45 2022
From: lxw263044 at alibaba-inc.com (=?UTF-8?B?6YCv5pmT5aiBKOWzsOmjjik=?=)
Date: Fri, 14 Jan 2022 14:23:45 +0800
Subject: =?UTF-8?B?5Zue5aSN77yaWkdDIEFBcmNoNjQgaGFuZyBvbiBTUEVDamJiMjAxNQ==?=
In-Reply-To: <608d169c-2692-95c5-93b6-61208cd03a0f@oracle.com>
References: <4ed59695-af34-43f4-8798-7655574a19a4.lxw263044@alibaba-inc.com>,
 <608d169c-2692-95c5-93b6-61208cd03a0f@oracle.com>
Message-ID: <eb18f0e2-c206-47bc-8bbf-8568d3508697.lxw263044@alibaba-inc.com>

Hi,

Thanks for your suggestion. Yes, we found out this problem resulted from SPECjbb and hasn't reproduced on the latest SPECjbb version.

Regards,
Xiaowei Lu



------------------------------------------------------------------
????Per Liden <per.liden at oracle.com>
?????2022?1?13?(???) 16:38
???????(??) <lxw263044 at alibaba-inc.com>; zgc-dev <zgc-dev at openjdk.java.net>
????Re: ZGC AArch64 hang on SPECjbb2015

Hi,

Before digging deeper, please verify if this is a problem on the latest 
JDK 17. Also note that SPECjbb2015 1.03 contains a fix for a hang we 
discovered and reported some time ago. So please verify that you're 
using that version.

/Per


On 1/13/22 03:58, ???(??) wrote:
> Hi, I have tested zgc on jdk-15-GA and found occasionally SPECjbb2015 hangs when it is about to shut down. This is quite hard to reproduce, one out of 20 iterations probably. Since it only reproduces during shut down, I prefer to use "preset" mode to run SPECjbb to save time.
> jdk-15-GA/bin/java -XX:+UseZGC -Xmx10g -Dspecjbb.controller.type=PRESET -Dspecjbb.controller.preset.ir=20000 -Dspecjbb.controller.preset.duration=60000 -jar specjbb2015.jar -m COMPOSITE
> And SPECjbb log is as follow:
> <Tue Jan 04 03:06:13 EST 2022> org.spec.jbb: Failed to stop executor service java.util.concurrent.ForkJoinPool at 9d2cbbc[Terminating, parallelism = 160, size = 2, active = 0, running = 2, steals = 40255, tasks = 0, submissions = 0], trying again
> <Tue Jan 04 03:06:14 EST 2022> org.spec.jbb: Failed to stop executor service java.util.concurrent.ForkJoinPool at 9d2cbbc[Terminating, parallelism = 160, size = 2, active = 0, running = 2, steals = 40255, tasks = 0, submissions = 0], trying again
> 
> This seems quite similar to a previous issue: https://bugs.openjdk.java.net/browse/JDK-8214285, but on AArch64 other than Skylake. In addition, JDK-8214285 has been solved in jdk12, but we reproduced this issue on jdk-13.0.2+8 and jdk-14.0.2+12
> 
> Regards,Xiaowei Lu
> 

From stefan.karlsson at oracle.com  Thu Jan 27 08:49:54 2022
From: stefan.karlsson at oracle.com (Stefan Karlsson)
Date: Thu, 27 Jan 2022 09:49:54 +0100
Subject: [External] : Re: ZGC unstable behaviour on java 17
In-Reply-To: <CAGJiV__3cTSpBZvS_nYuHsUhEt35ykb3QJsFk4nSsmxkinhyjw@mail.gmail.com>
References: <CAGJiV__0ktNpyGgE=_mQZDA_jtA1ZDLouXg4rXYetJ03c8=GgQ@mail.gmail.com>
 <7A1AE5B9-04D8-4BD9-8582-5E158141C5A9@oracle.com>
 <CAGJiV__nvuCAjFTQxe3PMbmA1vfuiyhZ+jEDjFsnTUGog+Kf0w@mail.gmail.com>
 <BN0PR10MB51766D1B585959F0459A9024F4929@BN0PR10MB5176.namprd10.prod.outlook.com>
 <CAA-vtUwJjv8u7Wv-swZ+6AbZpSsHbvq-gSXvpXeipN1V82qm=A@mail.gmail.com>
 <CAGJiV_9ViTUrMmeAHtmvCQQ4_CekVjSM1NN_yt+SwTErRyk8Ug@mail.gmail.com>
 <048B6073-AEF0-488D-A674-56FB64B6779F@oracle.com>
 <CAGJiV_9of7=+azBFOMEM0EU_PDfvtJ36_auzbNFUKfdMF3vqyA@mail.gmail.com>
 <BN0PR10MB5176E0B8223B213FAF369EC2F4939@BN0PR10MB5176.namprd10.prod.outlook.com>
 <57b16ac3-2083-6544-9822-aacdac934314@oracle.com>
 <CAGJiV_8ToEGc-uQXoaaxr6a_PEr3iTC9j60VuMnwsuaWn9WYMQ@mail.gmail.com>
 <b28a300a-f53b-5044-77b7-1619bc6db79b@oracle.com>
 <44682fad-a1de-5ba7-e281-f4d832368977@oracle.com>
 <CAGJiV__3cTSpBZvS_nYuHsUhEt35ykb3QJsFk4nSsmxkinhyjw@mail.gmail.com>
Message-ID: <2634d807-f927-62c9-9495-23aa178914c9@oracle.com>

Hi Anatoly,

JDK 17.0.2 has now been released, and it contains the fix I mentioned 
below. I hope this resolves the issues you were seeing. If
you get around to retest with the latest binaries, feel free to report 
back and tell us if the problem went away or not.

Thanks,
StefanK

On 2021-11-19 11:45, Anatoly Deyneka wrote:
> Thank you,
>
> I'm surprised how fast it has been fixed.
> Looking forward to checking it out.
>
> Anatoly
>
>
> On Thu, 18 Nov 2021 at 13:30, Stefan Karlsson 
> <stefan.karlsson at oracle.com> wrote:
>
>     Hi Anatoly,
>
>     We've identified a 10-year-old bug in the inline cache cleaning code.
>     See this pull request:
>     https://github.com/openjdk/jdk/pull/6450
>     <https://urldefense.com/v3/__https://github.com/openjdk/jdk/pull/6450__;!!ACWV5N9M2RV99hQ!Y-s2AS9txeUlTkMY36_KkkfTFUkb7yuFvhvAQYJwyIBlNPpzooN8tCWE6462IR6I9OYm$>
>
>     It's likely that this is the cause of the problems you are seeing in
>     your workload. Note that this bug is particularly bad for concurrent
>     class unloading GCs (ZGC, Shenandoah), while the problem seems less
>     sever for stop-the-world class unloading GCs (G1, Parallel, Serial).
>
>     The plan right now is to get this into JDK 18, and then backport
>     it to
>     JDK 17.
>
>     Thanks again for reporting this!
>
>     StefanK
>
>     On 2021-11-12 16:46, Stefan Karlsson wrote:
>     > Hi Anatoly,
>     >
>     > On 2021-11-11 11:19, Anatoly Deyneka wrote:
>     >> It's great you can reproduce it.
>     >> Is there anything else we can help you with?
>     >
>     > I think we have everything we need for now. The hints about the
>     > problem with refill_ic_stubs were great, and that's what we are
>     > currently investigating on our side.
>     >
>     > Thanks,
>     > StefanK
>     >
>     >>
>     >> ??? On that note, it isn?t clear to me if the 2 orders of magnitude
>     >> ??? higher metaspace allocation rate was observed before or
>     after the
>     >> ??? JSON issues were addressed, as you described.
>     >>
>     >> Before the fix of ObjectMapper.copy() we had a 250MB/s allocation
>     >> rate and OutOfMemory(metaspace).
>     >> BTW with?shenandoah we have about 6MB/s with rare spikes up to
>     100MB/s
>     >>
>     >> Regards,
>     >> Anatoly Deyneka
>     >>
>     >>
>     >>
>     >> On Wed, 10 Nov 2021 at 20:07, Stefan Karlsson
>     >> <stefan.karlsson at oracle.com> wrote:
>     >>
>     >> ??? Hi,
>     >>
>     >> ??? I've managed to reproduce the long Concurrent Classes
>     Unlink phase
>     >> ??? by running an edited Blackbird microbenchmark.
>     >>
>     >> ??? First add an ObjectMapper#copy call to beanArrayDeser:
>     >>
>     https://github.com/FasterXML/jackson-modules-base/blob/master/blackbird/benchmarks/src/main/java/com/fasterxml/jackson/module/blackbird/BaseBenchmark.java
>     <https://urldefense.com/v3/__https://github.com/FasterXML/jackson-modules-base/blob/master/blackbird/benchmarks/src/main/java/com/fasterxml/jackson/module/blackbird/BaseBenchmark.java__;!!ACWV5N9M2RV99hQ!Y-s2AS9txeUlTkMY36_KkkfTFUkb7yuFvhvAQYJwyIBlNPpzooN8tCWE6462IRbCx0SB$>
>     >>
>     <https://urldefense.com/v3/__https://github.com/FasterXML/jackson-modules-base/blob/master/blackbird/benchmarks/src/main/java/com/fasterxml/jackson/module/blackbird/BaseBenchmark.java__;!!ACWV5N9M2RV99hQ!aVQ1g1KY8-TN5Yjuvx8qsBOlt--iQzL4LrUyJBf4QKIkyntGlvpoq0XLuC1BmIMuUEN8$>
>     >>
>     >> ??? ---
>     >> ??? diff --git
>     >>
>     a/blackbird/benchmarks/src/main/java/com/fasterxml/jackson/module/blackbird/BaseBenchmark.java
>     >>
>     b/blackbird/benchmarks/src/main/java/com/fasterxml/jackson/module/blackbird/BaseBenchmark.java
>     >> ??? index 3675286..af05d54 100644
>     >> ??? ---
>     >>
>     a/blackbird/benchmarks/src/main/java/com/fasterxml/jackson/module/blackbird/BaseBenchmark.java
>     >> ??? +++
>     >>
>     b/blackbird/benchmarks/src/main/java/com/fasterxml/jackson/module/blackbird/BaseBenchmark.java
>     >> ??? @@ -80,7 +80,7 @@ public abstract class BaseBenchmark {
>     >>
>     >> ??? ???? @Benchmark
>     >> ??? ???? public SomeBean[] beanArrayDeser() throws Exception {
>     >> ??? -??????? return mapper.readValue(beansBytes, SomeBean[].class);
>     >> ??? +??????? return mapper.copy().readValue(beansBytes,
>     >> SomeBean[].class);
>     >> ??? ???? }
>     >>
>     >> ??? ???? @Benchmark
>     >> ??? ---
>     >>
>     >> ??? Then change the ICBuffer size to 100, and run the following:
>     >> ??? java -Xlog:gc* -Xmx20g -XX:+UseZGC
>     -XX:MetaspaceReclaimPolicy=none
>     >> ??? -XX:-ZProactive -jar target/benchmarks.jar
>     >>
>     >> ??? First run:
>     >> ??? [180,168s][info][gc,stats ]??? Subphase: Concurrent Classes
>     >> ??? Unlink?????????????????? 495,386 / 495,386 7002,041 /
>     >> ??? 24368,855? 7002,041 / 24368,855? 7002,041 / 24368,855 ms
>     >>
>     >> ??? Second run:
>     >> ??? [140,170s][info][gc,stats??? ]??? Subphase: Concurrent Classes
>     >> ??? Unlink???????????????? 67606,542 / 134428,068 67606,542 /
>     >> ??? 134428,068 67606,542 / 134428,068 67606,542 / 134428,068 ms
>     >>
>     >> ??? StefanK
>     >>
>     >> ??? On 2021-11-10 17:18, Erik Osterlund wrote:
>     >>> ??? Hi Anatoly,
>     >>>
>     >>> ??? I think that we can now see why a large cluster of classes
>     get
>     >>> unloaded, and as you say possibly control that, as a workaround,
>     >>> with the SoftRefLRUPolicyMSPerMB flag. However, the bigger
>     question
>     >>> is what it is that makes the unloading take so long.
>     >>>
>     >>> ??? One theory is that it is related to metaspace uncommitting
>     being
>     >>> more aggressive. It seemed though like you had tried the more
>     >>> balanced mode and it still was an issue, if I understood you
>     correctly.
>     >>>
>     >>> ??? The second theory would be that it is something in the code
>     >>> cache unloading that takes a long time. And it seems like when
>     you
>     >>> introspected it, the flame graph suggests that it is the code
>     cache
>     >>> unloading that takes long. So this seems like a more likely
>     >>> explanation. And I can see from the GC logs that you have a
>     *lot* of
>     >>> nmethods being loaded. And many thousands of them get unloaded,
>     >>> which leads to a huge inline cache cleaning task, unlinking all
>     >>> these thousands of nmethods. This inline cache cleaning fills
>     up a
>     >>> buffer (ICBuffer) that is currently 10 KB large (rather small and
>     >>> has not changed since ancient times, perhaps it is time to change
>     >>> it). When this buffer fills up, you need to trigger an IC buffer
>     >>> refilling safepoint, as you have observed in the logs. So the
>     fact
>     >>> that loads of these safepoints incrementally trigger, seems to
>     >>> suggest that for this large number of nmethods being unloaded,
>     and
>     >>> the buffer size is too small. When it is way too small, the code
>     >>> cache walk during the unlinking phase becomes quadratic with the
>     >>> number of nmethods. And you have *loads* of nmethods. So that
>     could
>     >>> explain why it takes so long with the code cache unloading.
>     >>>
>     >>> ??? In the solution space of scalability of class unloading to
>     these
>     >>> large levels, the long-term fix is probably to remove inline
>     caches
>     >>> completely, which I have worked on a bit, but it is not going to
>     >>> happen over night. A simpler fix that might actually happen over
>     >>> night, is to increase the size of the IC buffer, like this:
>     >>>
>     >>> ??? diff --git a/src/hotspot/share/code/icBuffer.cpp
>     >>> b/src/hotspot/share/code/icBuffer.cpp
>     >>> ??? index 1da48f0bd13..382fa8650c5 100644
>     >>> ??? --- a/src/hotspot/share/code/icBuffer.cpp
>     >>> ??? +++ b/src/hotspot/share/code/icBuffer.cpp
>     >>> ??? @@ -140,7 +140,7 @@ void ICStub::print() {
>     >>>
>     >>> ??? void InlineCacheBuffer::initialize() {
>     >>> ??????? if (_buffer != NULL) return; // already initialized
>     >>> ??? -? _buffer = new StubQueue(new ICStubInterface, 10*K,
>     >>> InlineCacheBuffer_lock, "InlineCacheBuffer");
>     >>> ??? +? _buffer = new StubQueue(new ICStubInterface, 1*M,
>     >>> InlineCacheBuffer_lock, "InlineCacheBuffer");
>     >>> ??????? assert (_buffer != NULL, "cannot allocate
>     InlineCacheBuffer");
>     >>> ??? }
>     >>>
>     >>> ??? That would probably bring down the time spent doing nmethod
>     >>> unlinking by a *lot*. If you know how to build the JDK, that
>     could
>     >>> be an easy patch to try out, to significantly improve the
>     >>> scalability of code cache unlinking.
>     >>>
>     >>> ??? However, one relevant question is also if this is really a
>     >>> scalability problem in the GC unloading path, or if the real
>     culprit
>     >>> is elsewhere. The fact that you see two orders of magnitude
>     higher
>     >>> allocation rates in metaspace (!!!) when upgrading from 15 to 17,
>     >>> might explain why the unloading code suddenly gets scalability
>     >>> stressed so hard in the first place. And perhaps whatever caused
>     >>> that 2 orders of magnitude increase, is the real culprit here.
>     >>> Having said that, we might want to improve on the class unloading
>     >>> scalabiliy anyway, but I do wonder why this is being stressed at
>     >>> all, and why there are so many dying classes and nmethods. On
>     that
>     >>> note, it isn?t clear to me if the 2 orders of magnitude higher
>     >>> metaspace allocation rate was observed before or after the JSON
>     >>> issues were addressed, as you described.
>     >>>
>     >>> ??? /Erik
>     >>>
>     >>> ??? From: Anatoly Deyneka<adeyneka at gmail.com>
>     >>> <mailto:adeyneka at gmail.com>
>     >>> ??? Sent: Wednesday, 10 November 2021 14:04
>     >>> ??? To: Erik Osterlund<erik.osterlund at oracle.com>
>     >>> <mailto:erik.osterlund at oracle.com>
>     >>> ??? Cc: Thomas St?fe<thomas.stuefe at gmail.com>
>     >>> <mailto:thomas.stuefe at gmail.com>;zgc-dev at openjdk.java.net
>     >>> ??? Subject: Re: [External] : Re: ZGC unstable behaviour on
>     java 17
>     >>>
>     >>> ??? Thank you Erik,
>     >>>
>     >>> ??? Do you have a understanding of how it can be fixed?
>     >>> ??? SoftRefLRUPolicyMSPerMB - is a workaround and not a fix.
>     >>> ??? We have working shenandoah with default settings. It unloads
>     >>> classes without performance regression.
>     >>> ??? BTW ZGC also can return to normal state if we temporary
>     reduce
>     >>> allocation rate by redirecting new users to another node(old
>     users
>     >>> still stay on the same node).
>     >>> ??? We did that many times when we saw such behaviour. ZGC
>     returned
>     >>> to normal state quite fast.
>     >>>
>     >>> ??? I guess ZGC does the slow unloading class job in the same
>     >>> threads that affects overall performance of GC.
>     >>> ??? It explains why zgc cycles time is dramatically increased.
>     >>>
>     >>> ??? According to Risks and Assumptions#Uncommit speed
>     >>> (https://openjdk.java.net/jeps/387):
>     >>> ??? "Metaspace reclamation can happen during a garbage-collection
>     >>> pause, so this could be a problem.
>     >>> ??? We haven?t observed this problem so far, but if uncommit
>     times
>     >>> become an issue then we could offload the uncommitting work to a
>     >>> separate thread so that it could be done independently of GC
>     pauses."
>     >>>
>     >>> ??? We observed and fixed class allocation problem in some
>     scenarios
>     >>> of using jackson+blackbird.
>     >>> ??? So we can try to reproduce this issue.
>     >>>
>     >>> ??? Regards,
>     >>> ??? Anatoly Deyneka
>     >>>
>     >>> ??? On Wed, 10 Nov 2021 at 00:23, Erik Osterlund
>     >>> <erik.osterlund at oracle.com<mailto:erik.osterlund at oracle.com>
>     >>> <mailto:erik.osterlund at oracle.com>> wrote:
>     >>> ??? Hi Anatoly,
>     >>>
>     >>> ??? Thank you for the additional details. Very interesting
>     indeed.
>     >>> Everything seems to run well until you get an allocation stall
>     due
>     >>> to class unloading taking a long time. The very same GC where the
>     >>> first stall happens, nuked quite a lot of soft references, while
>     >>> previous GCs did not. It really looks like a clear pattern wrt
>     soft
>     >>> reference count and the class unloading headache.
>     >>>
>     >>> ??? This seems to support the original intuition that the lambda
>     >>> form cache might have caused excessive class unloading after
>     lots of
>     >>> soft references get cleared.
>     >>>
>     >>> ??? The GC itself didn?t explicitly clear all soft references,
>     but
>     >>> they might have timed out at the same time. By setting
>     >>> SoftRefLRUPolicyMSPerMB to a *very* high value, the soft ref
>     >>> clearing won?t get triggered. That might make things better
>     for you.
>     >>>
>     >>> ??? Having said that, it seems like this cache really
>     shouldn?t be
>     >>> using soft references if the result is this horrible when they
>     get
>     >>> cleared. Unless of course that this is a red herring.
>     >>>
>     >>> ??? Thanks,
>     >>> ??? /Erik
>     >>>
>     >>>
>     >>> ??? On 9 Nov 2021, at 17:39, Anatoly Deyneka
>     >>> <adeyneka at gmail.com<mailto:adeyneka at gmail.com>
>     >>> <mailto:adeyneka at gmail.com>> wrote:
>     >>> ??? ?
>     >>> ??? Please find the log here:
>     >>>
>     https://github.com/adeyneka/ZGC-java-16-17-unstable-behaviour/blob/main/gc-1-2021-10-28_09-14-36-blue.log.0
>     <https://urldefense.com/v3/__https://github.com/adeyneka/ZGC-java-16-17-unstable-behaviour/blob/main/gc-1-2021-10-28_09-14-36-blue.log.0__;!!ACWV5N9M2RV99hQ!Y-s2AS9txeUlTkMY36_KkkfTFUkb7yuFvhvAQYJwyIBlNPpzooN8tCWE6462IULTQgU1$>
>
>     >>>
>     <https://urldefense.com/v3/__https://github.com/adeyneka/ZGC-java-16-17-unstable-behaviour/blob/main/gc-1-2021-10-28_09-14-36-blue.log.0__;!!ACWV5N9M2RV99hQ!aVQ1g1KY8-TN5Yjuvx8qsBOlt--iQzL4LrUyJBf4QKIkyntGlvpoq0XLuC1BmBsZKftr$><https://urldefense.com/v3/__https:/github.com/adeyneka/ZGC-java-16-17-unstable-behaviour/blob/main/gc-1-2021-10-28_09-14-36-blue.log.0__;!!ACWV5N9M2RV99hQ!a6mYXhBdfLivRQIJkyn10Z2tlh6iDn_TfHQpHBldzNBIJexWK-nR7FeTgxiyA0IE_N4$>
>
>     >>>
>     <https://urldefense.com/v3/__https:/github.com/adeyneka/ZGC-java-16-17-unstable-behaviour/blob/main/gc-1-2021-10-28_09-14-36-blue.log.0__;!!ACWV5N9M2RV99hQ!a6mYXhBdfLivRQIJkyn10Z2tlh6iDn_TfHQpHBldzNBIJexWK-nR7FeTgxiyA0IE_N4$>
>
>     >>>
>     >>>
>     >>> ??? - Maybe try without reflection inflation? E.g.
>     >>> `-Dsun.reflect.inflationThreshold=2147483647`. Which would
>     inhibit
>     >>> the generation of reflection class loaders and rule out that they
>     >>> are the problem.
>     >>> ??? we would like to go with inflation. We use a lot of
>     reflection.
>     >>> ??? - Does it happen with other GCs?
>     >>> ??? we tried shenandoah today and it worked fine(just 1 stall for
>     >>> the whole day)
>     >>> ??? - Elastic Metaspace did away with huge pages in Metaspace for
>     >>> several reasons. Did the old scenario use
>     >>> -XX:+UseLargePagesInMetaspace ?
>     >>> ??? no, we didn't use this flag
>     >>> ??? - Can you switch metaspace reclamation off using
>     >>> `-XX:MetaspaceReclaimPolicy=none`? I'd be curious if this
>     helps - if
>     >>> yes, there may be something wrong with the reclamation algorithm.
>     >>> ??? we can try this
>     >>> ??? - If the VM becomes unresponsive, could you also give us the
>     >>> output of `jcmd VM.metaspace` ? Also, if possible, `jcmd
>     >>> VM.metaspace show-loaders show-classes` though with 1M classes
>     >>> output may be large.
>     >>> ??? good point. we will analize it.
>     >>>
>     >>> ??? Regards,
>     >>> ??? Anatoly Deyneka
>     >>>
>     >>> ??? On Tue, 9 Nov 2021 at 18:26, Thomas St?fe
>     >>> <thomas.stuefe at gmail.com<mailto:thomas.stuefe at gmail.com>
>     >>> <mailto:thomas.stuefe at gmail.com>> wrote:
>     >>> ??? Additionally to Eriks proposal, some more remarks from the
>     side
>     >>> (I'll just lurk in the background otherwise) :
>     >>>
>     >>> ??? - Maybe try without reflection inflation? E.g.
>     >>> `-Dsun.reflect.inflationThreshold=2147483647`. Which would
>     inhibit
>     >>> the generation of reflection class loaders and rule out that they
>     >>> are the problem.
>     >>>
>     >>> ??? - Does it happen with other GCs?
>     >>>
>     >>> ??? - Elastic Metaspace did away with huge pages in Metaspace for
>     >>> several reasons. Did the old scenario use
>     >>> -XX:+UseLargePagesInMetaspace ?
>     >>>
>     >>> ??? - Can you switch metaspace reclamation off using
>     >>> `-XX:MetaspaceReclaimPolicy=none`? I'd be curious if this
>     helps - if
>     >>> yes, there may be something wrong with the reclamation algorithm.
>     >>>
>     >>> ??? - If the VM becomes unresponsive, could you also give us the
>     >>> output of `jcmd VM.metaspace` ? Also, if possible, `jcmd
>     >>> VM.metaspace show-loaders show-classes` though with 1M classes
>     >>> output may be large.
>     >>>
>     >>> ??? Thanks, Thomas
>     >>>
>     >>> ??? On Tue, Nov 9, 2021 at 3:45 PM Erik Osterlund
>     >>> <erik.osterlund at oracle.com<mailto:erik.osterlund at oracle.com>
>     >>> <mailto:erik.osterlund at oracle.com>> wrote:
>     >>> ??? Hi Anatoly,
>     >>>
>     >>> ??? Thank you for the additional details. This is very helpful. I
>     >>> talked to Per a bit about this, and we feel like understanding
>     what
>     >>> increased the metaspace allocation rate from 1.5 KB/s to 40
>     MB/s, is
>     >>> rather likely to lead us to the problem. That?s more than 3
>     orders
>     >>> of magnitude higher allocation rate of metaspace memory, which
>     might
>     >>> explain why class unloading is taking much longer than usual.
>     >>>
>     >>> ??? The culprit that is being repeatedly allocated, is rather
>     likely
>     >>> to also be repeatedly unloaded, or you would hit an out of memory
>     >>> issue. And the log snippet you showed, shows that there is a
>     bunch
>     >>> of LambdaForms being unloaded. Interestingly enough, the
>     >>> LambdaFormEditor has a SoftReference based cache, so that you
>     don?t
>     >>> have to create a new LambdaForm, when there is one in the cache.
>     >>> This soft reference based policy might not work very well if all
>     >>> SoftReferences are cleared by the GC, because then it would seem
>     >>> like a lot more LambdaForms would have to be created from
>     scratch,
>     >>> as opposed to being reused from the cache.
>     >>>
>     >>> ??? This might explain why it takes a while before this starts
>     >>> happening. The interesting thing is figuring out what could
>     have led
>     >>> to all the soft references being cleared. This could be
>     because of
>     >>> metaspace exhaustion, or due to allocation stalls. In the
>     charts we
>     >>> can see that the allocation rate was at the highest point, right
>     >>> before the incident. So without seeing the log, it seems like you
>     >>> could have gotten an allocation stall, which triggered a full
>     >>> collection clearing all soft references, which suddenly leads to
>     >>> class unloading taking much longer. If class unloading takes much
>     >>> longer, then you are quite likely to reach the same situation
>     >>> repeatedly, in a bad cycle.
>     >>>
>     >>> ??? Is there any way you could share an -Xlog:gc* log with us? If
>     >>> not, finding out if there was an allocation stall or a
>     ?Metadata GC
>     >>> Clear Soft References? GC just before everything went bad, might
>     >>> help with figuring out what went wrong. Something that did happen
>     >>> between JDK 15 and JDK 17 is dynamic thread selection for the
>     GC. I
>     >>> wonder if that has somehow led to a stall, which you wouldn?t
>     see in
>     >>> JDK 15 with the more static thread selection. If so, then maybe
>     >>> setting a SoftMaxHeapSize lower than the MaxHeapSize might help
>     >>> avoiding stall in the first place, due to fluctuations in the
>     >>> workload, or alternatively turning off
>     UseDynamicNumberOfGCThreads
>     >>> and setting ConcGCThreads high enough for allocation stalls
>     not happen.
>     >>>
>     >>> ??? Anyway, figuring out what kind of GC triggered right
>     before the
>     >>> incident would help a lot I think, to figure out if forced
>     soft ref
>     >>> clearing is indeed what instigated this, or if this is all just a
>     >>> red herring.
>     >>>
>     >>> ??? Thanks,
>     >>> ??? /Erik
>     >>>
>     >>> ??? From: Anatoly Deyneka
>     >>> <adeyneka at gmail.com<mailto:adeyneka at gmail.com>
>     >>> <mailto:adeyneka at gmail.com>>
>     >>> ??? Sent: Tuesday, 9 November 2021 14:13
>     >>> ??? To: Erik Osterlund
>     >>> <erik.osterlund at oracle.com<mailto:erik.osterlund at oracle.com>
>     >>> <mailto:erik.osterlund at oracle.com>>
>     >>> Cc:zgc-dev at openjdk.java.net
>     <mailto:Cc%3Azgc-dev at openjdk.java.net><mailto:zgc-dev at openjdk.java.net>
>
>     >>> <mailto:zgc-dev at openjdk.java.net>
>     >>> ??? Subject: [External] : Re: ZGC unstable behaviour on java 17
>     >>>
>     >>> ??? Hi Erik,
>     >>>
>     >>> ??? Probably we haven't reached this condition on java 15.
>     >>> ??? We see a big difference in metaspace allocation starting from
>     >>> java 16.
>     >>> ??? The behaviour is not changed for any option of
>     >>> -XX:MetaspaceReclaimPolicy
>     >>>
>     >>> ??? Here is all our knowledge about the problem:
>     >>>
>     https://github.com/adeyneka/ZGC-java-16-17-unstable-behaviour/blob/main/README.md
>     <https://urldefense.com/v3/__https://github.com/adeyneka/ZGC-java-16-17-unstable-behaviour/blob/main/README.md__;!!ACWV5N9M2RV99hQ!Y-s2AS9txeUlTkMY36_KkkfTFUkb7yuFvhvAQYJwyIBlNPpzooN8tCWE6462Icu2dE2x$>
>
>     >>>
>     <https://urldefense.com/v3/__https://github.com/adeyneka/ZGC-java-16-17-unstable-behaviour/blob/main/README.md__;!!ACWV5N9M2RV99hQ!aVQ1g1KY8-TN5Yjuvx8qsBOlt--iQzL4LrUyJBf4QKIkyntGlvpoq0XLuC1BmOvvay_E$><https://urldefense.com/v3/__https:/github.com/adeyneka/ZGC-java-16-17-unstable-behaviour/blob/main/README.md__;!!ACWV5N9M2RV99hQ!a6mYXhBdfLivRQIJkyn10Z2tlh6iDn_TfHQpHBldzNBIJexWK-nR7FeTgxiy18HZddw$>
>
>     >>>
>     <https://urldefense.com/v3/__https:/github.com/adeyneka/ZGC-java-16-17-unstable-behaviour/blob/main/README.md__;!!ACWV5N9M2RV99hQ!a6mYXhBdfLivRQIJkyn10Z2tlh6iDn_TfHQpHBldzNBIJexWK-nR7FeTgxiy18HZddw$><https://urldefense.com/v3/__https:/github.com/adeyneka/ZGC-java-16-17-unstable-behaviour/blob/main/README.md__;!!ACWV5N9M2RV99hQ!cjW1jni1cEfXVuMQMjBMHiYL-HeLf0WosxB4QltNGKaNiAf1NynZSboJeEbNiPRrUpo$>
>
>     >>>
>     <https://urldefense.com/v3/__https:/github.com/adeyneka/ZGC-java-16-17-unstable-behaviour/blob/main/README.md__;!!ACWV5N9M2RV99hQ!cjW1jni1cEfXVuMQMjBMHiYL-HeLf0WosxB4QltNGKaNiAf1NynZSboJeEbNiPRrUpo$>
>
>     >>>
>     >>>
>     >>> ??? Unfortunately we can reproduce this issue only on our
>     production
>     >>> platform.
>     >>> ??? If we can provide more details or logs please let me know.
>     >>>
>     >>> ??? Regards,
>     >>> ??? Anatoly Deyneka
>     >>>
>     >>> ??? On Mon, 8 Nov 2021 at 16:46, Erik Osterlund
>     >>> <erik.osterlund at oracle.com<mailto:erik.osterlund at oracle.com>
>     >>>
>     <mailto:erik.osterlund at oracle.com><mailto:erik.osterlund at oracle.com
>     >>>
>     <mailto:erik.osterlund at oracle.com><mailto:erik.osterlund at oracle.com>
>     >>> <mailto:erik.osterlund at oracle.com>>> wrote:
>     >>> ??? Hi Anatoly,
>     >>>
>     >>> ??? Thanks for reporting this. This looks rather odd indeed.
>     >>> Unfortunately, the attached images have been stripped from your
>     >>> original email.
>     >>>
>     >>> ??? It?s particularly interesting that you see this behaviour
>     when
>     >>> migrating from JDK 15 to 17. There have been a few bug fixes, but
>     >>> nothing I can easily imagine causing this. I have been surprised
>     >>> before though. One notable thing in the class unloading path that
>     >>> has changed though, is the elastic metaspace. I see that you
>     use the
>     >>> -XX:MetaspaceReclaimPolicy=aggressive option, which did not
>     exist in
>     >>> JDK15. This makes me wonder if this reproduces also without that
>     >>> option, i.e. the way that you run the workload is more similar.
>     >>>
>     >>> ??? Anyway, is there any way for me to get access to try to
>     >>> reproduce your issue?
>     >>>
>     >>> ??? Thanks,
>     >>> ??? /Erik
>     >>>
>     >>>> ??? On 4 Nov 2021, at 17:00, Anatoly Deyneka
>     >>>> <adeyneka at gmail.com<mailto:adeyneka at gmail.com>
>     >>>> <mailto:adeyneka at gmail.com><mailto:adeyneka at gmail.com
>     >>>> <mailto:adeyneka at gmail.com><mailto:adeyneka at gmail.com>
>     >>>> <mailto:adeyneka at gmail.com>>> wrote:
>     >>>>
>     >>>> ??? ?We are running a GC intensive application with
>     allocation rate
>     >>>> up to
>     >>>> ??? 1.5GB/s(4k requests/sec) on 32v cores instance (-Xms20G
>     -Xmx20G).
>     >>>> ??? openjdk 17 2021-09-14
>     >>>> ??? OpenJDK Runtime Environment (build 17+35-2724)
>     >>>> ??? OpenJDK 64-Bit Server VM (build 17+35-2724, mixed mode,
>     sharing)
>     >>>>
>     >>>> ??? During migration from java 15 to java 17 we see unstable
>     >>>> behavior when zgc
>     >>>> ??? goes to infinite loop and doesn't return back to the
>     normal state.
>     >>>> ??? We see the following behavior:
>     >>>> ??? - it runs perfectly(~30% better then on java 15) - 5-7GB
>     used
>     >>>> memory with
>     >>>> ??? 16% CPU utilization
>     >>>> ??? - something happens and CPU utilization jumps to 40%, used
>     >>>> memory jumps to
>     >>>> ??? 19GB, zgc cycles time is increased in 10times
>     >>>> ??? - sometimes app works about 30min before the crash sometimes
>     >>>> it's crashed
>     >>>> ??? instantly
>     >>>> ??? - if it's still alive memory is not released and the app
>     works
>     >>>> with used
>     >>>> ??? memory 18-20GB
>     >>>> ??? - we see allocation stalls
>     >>>> ??? - we see the following GC stats
>     >>>> ??? [17606.140s][info][gc,phases?? ] GC(719) Concurrent Process
>     >>>> Non-Strong
>     >>>> ??? References 25781.928ms
>     >>>> ??? [17610.181s][info][gc,stats??? ] Subphase: Concurrent
>     Classes
>     >>>> Unlink
>     >>>> ??? 14280.772 / 25769.511? 1126.563 / 25769.511?? 217.882 /
>     >>>> 68385.750?? 217.882
>     >>>> ??? / 68385.750?? ms
>     >>>> ??? -? we see JVM starts massively unload classes
>     >>>> (jvm_classes_unloaded_total
>     >>>> ??? counter keep increasing) .
>     >>>> ??? - we?ve managed to profile JVM in such ?unstable? state
>     and see
>     >>>> ZTask
>     >>>> ??? consuming a lot of cpu doing
>     >>>> CompiledMethod::unload_nmethod_caches(bool)
>     >>>> ??? job. See attached image.
>     >>>>
>     >>>> ??? We played with options:
>     >>>> ??? -XX:ZCollectionInterval=5
>     >>>> ??? -XX:SoftMaxHeapSize=12G
>     >>>> ??? -XX:ZAllocationSpikeTolerance=4
>     >>>> ??? -XX:MetaspaceReclaimPolicy=aggressive
>     >>>>
>     >>>> ??? But it doesn't help. it just postpones this event.
>     >>>> ??? Only if we redirect traffic to another node it goes to
>     normal.
>     >>>>
>     >>>> ??? Regards,
>     >>>> ??? Anatoly Deyneka
>     >>>>
>     >>>> ??? [image: image.png]
>     >>>> ??? [image: image.png]
>     >>>> ??? [image: image.png]
>     >>
>     >
>

From m.sundar85 at gmail.com  Thu Jan 27 19:00:17 2022
From: m.sundar85 at gmail.com (Sundara Mohan M)
Date: Thu, 27 Jan 2022 11:00:17 -0800
Subject: ZGC Concurrent Mark Work distribution
Message-ID: <CACGCMVpmYWS=tGPa8mOc3v5EB5mpYJr5F8yNmnRmT-au=SbDBA@mail.gmail.com>

Hi,
   We are using ZGC in production environment with following settings
*[2022-01-27T09:12:55.684+0000][0.046s][info][gc,init] Version: 17.0.1+12
(release)*
..
*[2022-01-27T09:12:55.684+0000][0.046s][info][gc,init] GC Workers: 18
(dynamic)*
...
[2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Min Capacity: 64000M
[2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Initial Capacity:
64000M
*[2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Max Capacity: 64000M*
[2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Medium Page Size: 32M
[2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Pre-touch: Disabled
[2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Available space on
backing filesystem: N/A
[2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Uncommit: Implicitly
Disabled (-Xms equals -Xmx)
*[2022-01-27T09:13:05.275+0000][9.636s][info][gc,init] Runtime Workers: 44*
[2022-01-27T09:13:05.278+0000][9.640s][info][gc     ] Using The Z Garbage
Collector

We run into GC cycle every 8-10 seconds. In some GC cycle Concurrent Mark
phase is taking 2*s* and in some cycle it is taking 200*ms.*

After enabling debug log for gc,phase saw in some cases more ZWorkers are
doing work in other case it is given to only 1 or 2 ZWorker.

Example log where work is done by multiple thread and took only 200ms
[2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#12) 150.092ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#0) 1.059ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#0) 150.279ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#14) 1.055ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#14) 149.355ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#8) 1.060ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#8) 150.312ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#6) 1.060ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#6) 150.323ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#17) 1.062ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#11) 1.058ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#13) 1.060ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#2) 1.059ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#2) 150.423ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#13) 150.437ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#11) 150.374ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#3) 1.061ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#3) 150.537ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#7) 1.061ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#7) 150.550ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#17) 150.364ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#15) 1.056ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#16) 1.059ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#1) 1.060ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#1) 150.613ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#4) 1.064ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#16) 150.584ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#10) 1.059ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#10) 150.706ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#4) 150.629ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#15) 150.601ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#9) 1.028ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#9) 150.938ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#5) 1.057ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#5) 151.063ms
*[2022-01-27T18:51:41.325+0000][34725.687s][info ][gc,phases   ] GC(2927)
Concurrent Mark 204.119ms*

Another case where it was run by only 1/2 worker and took 1/2s respectively
...
*[2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
Concurrent Mark (ZWorker#12) 934.304ms*
[2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
Concurrent Mark Try Terminate (ZWorker#14) 1.062ms
*[2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
Concurrent Mark (ZWorker#14) 934.339ms*
*[2022-01-27T18:51:52.342+0000][34736.704s][info ][gc,phases   ] GC(2928)
Concurrent Mark 1021.048ms*


[2022-01-27T18:52:00.822+0000][34745.184s][info ][safepoint   ] Safepoint
"ZMarkStart", Time since last: 8436197260 ns, Reaching safepoint: 1019671
ns, At safepoint: 515965 ns, Total: 1
535636 ns
[2022-01-27T18:52:00.835+0000][34745.197s][debug][gc,phases   ] GC(2929)
Concurrent Roots OopStorageSet (ZWorker#5) 0.991ms
[2022-01-27T18:52:00.851+0000][34745.213s][debug][gc,phases   ] GC(2929)
Concurrent Roots ClassLoaderDataGraph (ZWorker#5) 15.777ms
[2022-01-27T18:52:00.864+0000][34745.226s][debug][gc,phases   ] GC(2929)
Concurrent Roots JavaThreads (ZWorker#5) 12.769ms
[2022-01-27T18:52:00.945+0000][34745.307s][debug][gc,phases   ] GC(2929)
Concurrent Roots CodeCache (ZWorker#5) 81.380ms
[2022-01-27T18:52:02.766+0000][34747.127s][debug][gc,phases   ] GC(2929)
Concurrent Mark Try Flush (ZWorker#16) 1.613ms
[2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
Concurrent Mark Try Flush (ZWorker#16) 0.717ms
[2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
Concurrent Mark Try Flush (ZWorker#16) 0.292ms
[2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
Concurrent Mark Try Terminate (ZWorker#16) 0.306ms
[2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
Concurrent Mark Try Terminate (ZWorker#16) 0.000ms


*[2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
Concurrent Mark (ZWorker#16)
1825.719ms[2022-01-27T18:52:02.771+0000][34747.133s][info ][gc,phases   ]
GC(2929) Concurrent Mark 1949.325ms*

Even though we have 18 threads, does anyone have any idea why the work is
not distributed among multiple workers?

Thanks
Sundar

From m.sundar85 at gmail.com  Thu Jan 27 19:20:37 2022
From: m.sundar85 at gmail.com (Sundara Mohan M)
Date: Thu, 27 Jan 2022 11:20:37 -0800
Subject: ZGC Concurrent Mark Work distribution
In-Reply-To: <CACGCMVpmYWS=tGPa8mOc3v5EB5mpYJr5F8yNmnRmT-au=SbDBA@mail.gmail.com>
References: <CACGCMVpmYWS=tGPa8mOc3v5EB5mpYJr5F8yNmnRmT-au=SbDBA@mail.gmail.com>
Message-ID: <CACGCMVppLjbxCHU7zBbMm5vUec_hAiqtKPJ9Qic-Bbo93J_gfA@mail.gmail.com>

Adding complete log of that phase in case if it helps
-01-27T18:51:41.121+0000][34725.483s][info ][gc,phases   ] GC(2927) Pause
Mark Start 0.046ms
[2022-01-27T18:51:41.121+0000][34725.483s][info ][safepoint   ] Safepoint
"ZMarkStart", Time since last: 8392610571 ns, Reaching safepoint: 606586
ns, At safepoint: 438685 ns, Total: 1045271 ns
[2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
Concurrent Roots OopStorageSet (ZWorker#11) 0.000ms
[2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
Concurrent Roots OopStorageSet (ZWorker#8) 0.102ms
[2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
Concurrent Roots OopStorageSet (ZWorker#0) 0.110ms
[2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
Concurrent Roots OopStorageSet (ZWorker#2) 0.062ms
[2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
Concurrent Roots OopStorageSet (ZWorker#9) 0.183ms
[2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
Concurrent Roots OopStorageSet (ZWorker#13) 0.209ms
[2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
Concurrent Roots OopStorageSet (ZWorker#6) 0.001ms
[2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
Concurrent Roots OopStorageSet (ZWorker#1) 0.098ms
[2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
Concurrent Roots OopStorageSet (ZWorker#12) 0.109ms
[2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
Concurrent Roots OopStorageSet (ZWorker#5) 0.112ms
[2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
Concurrent Roots OopStorageSet (ZWorker#17) 0.000ms
[2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
Concurrent Roots OopStorageSet (ZWorker#15) 0.103ms
[2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
Concurrent Roots OopStorageSet (ZWorker#16) 0.147ms
[2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
Concurrent Roots OopStorageSet (ZWorker#7) 0.067ms
[2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
Concurrent Roots OopStorageSet (ZWorker#4) 0.143ms
[2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
Concurrent Roots OopStorageSet (ZWorker#10) 0.048ms
[2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
Concurrent Roots OopStorageSet (ZWorker#3) 0.057ms
[2022-01-27T18:51:41.135+0000][34725.496s][debug][gc,phases   ] GC(2927)
Concurrent Roots OopStorageSet (ZWorker#14) 0.249ms
[2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
Concurrent Roots ClassLoaderDataGraph (ZWorker#5) 4.032ms
[2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
Concurrent Roots ClassLoaderDataGraph (ZWorker#16) 3.921ms
[2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
Concurrent Roots ClassLoaderDataGraph (ZWorker#7) 3.888ms
[2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
Concurrent Roots ClassLoaderDataGraph (ZWorker#3) 3.685ms
[2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
Concurrent Roots ClassLoaderDataGraph (ZWorker#4) 3.857ms
[2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
Concurrent Roots ClassLoaderDataGraph (ZWorker#10) 3.725ms
[2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
Concurrent Roots ClassLoaderDataGraph (ZWorker#12) 4.064ms
[2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
Concurrent Roots ClassLoaderDataGraph (ZWorker#6) 4.137ms
[2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
Concurrent Roots ClassLoaderDataGraph (ZWorker#17) 3.998ms
[2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
Concurrent Roots ClassLoaderDataGraph (ZWorker#0) 4.301ms
[2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
Concurrent Roots ClassLoaderDataGraph (ZWorker#8) 4.335ms
[2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
Concurrent Roots ClassLoaderDataGraph (ZWorker#14) 3.658ms
[2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
Concurrent Roots ClassLoaderDataGraph (ZWorker#15) 3.966ms
[2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
Concurrent Roots ClassLoaderDataGraph (ZWorker#13) 4.267ms
[2022-01-27T18:51:41.139+0000][34725.501s][debug][gc,phases   ] GC(2927)
Concurrent Roots ClassLoaderDataGraph (ZWorker#1) 4.165ms
[2022-01-27T18:51:41.139+0000][34725.501s][debug][gc,phases   ] GC(2927)
Concurrent Roots ClassLoaderDataGraph (ZWorker#2) 4.619ms
[2022-01-27T18:51:41.140+0000][34725.502s][debug][gc,phases   ] GC(2927)
Concurrent Roots ClassLoaderDataGraph (ZWorker#9) 5.835ms
[2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
Concurrent Roots JavaThreads (ZWorker#7) 4.247ms
[2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
Concurrent Roots JavaThreads (ZWorker#3) 4.243ms
[2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
Concurrent Roots JavaThreads (ZWorker#6) 4.168ms
[2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
Concurrent Roots JavaThreads (ZWorker#17) 4.091ms
[2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
Concurrent Roots JavaThreads (ZWorker#9) 2.713ms
[2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
Concurrent Roots JavaThreads (ZWorker#10) 4.187ms
[2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
Concurrent Roots JavaThreads (ZWorker#0) 4.071ms
[2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
Concurrent Roots JavaThreads (ZWorker#2) 3.897ms
[2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
Concurrent Roots JavaThreads (ZWorker#16) 4.277ms
[2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
Concurrent Roots JavaThreads (ZWorker#5) 4.319ms
[2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
Concurrent Roots JavaThreads (ZWorker#4) 4.231ms
[2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
Concurrent Roots JavaThreads (ZWorker#15) 3.997ms
[2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
Concurrent Roots JavaThreads (ZWorker#13) 3.983ms
[2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
Concurrent Roots JavaThreads (ZWorker#14) 4.049ms
[2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
Concurrent Roots JavaThreads (ZWorker#1) 3.968ms
[2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
Concurrent Roots JavaThreads (ZWorker#8) 4.092ms
[2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
Concurrent Roots JavaThreads (ZWorker#12) 4.155ms
[2022-01-27T18:51:41.148+0000][34725.510s][debug][gc,phases   ] GC(2927)
Concurrent Roots ClassLoaderDataGraph (ZWorker#11) 13.665ms
[2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
Concurrent Roots CodeCache (ZWorker#12) 28.078ms
[2022-01-27T18:51:41.174+0000][34725.536s][debug][gc,phases   ] GC(2927)
Concurrent Roots CodeCache (ZWorker#0) 30.133ms
[2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
Concurrent Roots CodeCache (ZWorker#3) 30.557ms
[2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
Concurrent Roots CodeCache (ZWorker#13) 29.600ms
[2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
Concurrent Roots CodeCache (ZWorker#5) 29.829ms
[2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
Concurrent Roots CodeCache (ZWorker#9) 30.148ms
[2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
Concurrent Roots CodeCache (ZWorker#10) 30.083ms
[2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
Concurrent Roots CodeCache (ZWorker#15) 29.707ms
[2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
Concurrent Roots CodeCache (ZWorker#7) 30.647ms
[2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
Concurrent Roots CodeCache (ZWorker#4) 29.814ms
[2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
Concurrent Roots CodeCache (ZWorker#8) 29.350ms
[2022-01-27T18:51:41.174+0000][34725.536s][debug][gc,phases   ] GC(2927)
Concurrent Roots CodeCache (ZWorker#14) 29.569ms
[2022-01-27T18:51:41.174+0000][34725.536s][debug][gc,phases   ] GC(2927)
Concurrent Roots CodeCache (ZWorker#1) 29.496ms
[2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
Concurrent Roots CodeCache (ZWorker#17) 30.227ms
[2022-01-27T18:51:41.174+0000][34725.536s][debug][gc,phases   ] GC(2927)
Concurrent Roots CodeCache (ZWorker#6) 30.641ms
[2022-01-27T18:51:41.174+0000][34725.536s][debug][gc,phases   ] GC(2927)
Concurrent Roots CodeCache (ZWorker#16) 29.984ms
[2022-01-27T18:51:41.174+0000][34725.536s][debug][gc,phases   ] GC(2927)
Concurrent Roots CodeCache (ZWorker#2) 30.087ms
[2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
Concurrent Roots CodeCache (ZWorker#11) 25.501ms
[2022-01-27T18:51:41.319+0000][34725.681s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#9) 1.061ms
[2022-01-27T18:51:41.319+0000][34725.681s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#6) 1.062ms
[2022-01-27T18:51:41.319+0000][34725.681s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#15) 1.060ms
[2022-01-27T18:51:41.319+0000][34725.681s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#16) 1.061ms
[2022-01-27T18:51:41.319+0000][34725.681s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#3) 1.065ms
[2022-01-27T18:51:41.319+0000][34725.681s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#5) 1.061ms
[2022-01-27T18:51:41.320+0000][34725.681s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Flush (ZWorker#12) 1.921ms
[2022-01-27T18:51:41.322+0000][34725.684s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#9) 1.061ms
[2022-01-27T18:51:41.322+0000][34725.684s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#8) 1.062ms
[2022-01-27T18:51:41.322+0000][34725.684s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#1) 1.063ms
[2022-01-27T18:51:41.322+0000][34725.684s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#13) 1.059ms
[2022-01-27T18:51:41.322+0000][34725.684s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#14) 1.066ms
[2022-01-27T18:51:41.322+0000][34725.684s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#7) 1.065ms
[2022-01-27T18:51:41.322+0000][34725.684s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#16) 1.061ms
[2022-01-27T18:51:41.322+0000][34725.684s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#15) 1.063ms
[2022-01-27T18:51:41.323+0000][34725.685s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#5) 1.061ms
[2022-01-27T18:51:41.323+0000][34725.685s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#9) 1.057ms
[2022-01-27T18:51:41.323+0000][34725.685s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Flush (ZWorker#12) 0.743ms
[2022-01-27T18:51:41.323+0000][34725.685s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#0) 1.061ms
[2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#14) 1.057ms
[2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#8) 1.060ms
[2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#6) 1.059ms
[2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#17) 1.061ms
[2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#11) 1.063ms
[2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#13) 1.063ms
[2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#2) 1.058ms
[2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#3) 1.061ms
[2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#7) 1.060ms
[2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Flush (ZWorker#12) 0.356ms
[2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#4) 1.061ms
[2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#15) 1.059ms
[2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#10) 1.060ms
[2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#16) 1.060ms
[2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#1) 1.060ms
[2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Flush (ZWorker#12) 0.311ms
[2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#9) 1.060ms
[2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#5) 1.058ms
[2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Flush (ZWorker#12) 0.285ms
[2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#12) 0.297ms
[2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#12) 0.000ms
[2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#12) 150.092ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#0) 1.059ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#0) 150.279ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#14) 1.055ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#14) 149.355ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#8) 1.060ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#8) 150.312ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#6) 1.060ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#6) 150.323ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#17) 1.062ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#11) 1.058ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#13) 1.060ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#2) 1.059ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#2) 150.423ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#13) 150.437ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#11) 150.374ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#3) 1.061ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#3) 150.537ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#7) 1.061ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#7) 150.550ms
[2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#17) 150.364ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#15) 1.056ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#16) 1.059ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#1) 1.060ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#1) 150.613ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#4) 1.064ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#16) 150.584ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#10) 1.059ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#10) 150.706ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#4) 150.629ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#15) 150.601ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#9) 1.028ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#9) 150.938ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark Try Terminate (ZWorker#5) 1.057ms
[2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
Concurrent Mark (ZWorker#5) 151.063ms
*[2022-01-27T18:51:41.325+0000][34725.687s][info ][gc,phases   ] GC(2927)
Concurrent Mark 204.119ms*
[2022-01-27T18:51:41.326+0000][34725.688s][info ][gc,phases   ] GC(2927)
Pause Mark End 0.070ms
[2022-01-27T18:51:41.326+0000][34725.688s][info ][safepoint   ] Safepoint
"ZMarkEnd", Time since last: 204207403 ns, Reaching safepoint: 210401 ns,
At safepoint: 237941 ns, Total: 448342 ns
[2022-01-27T18:51:41.326+0000][34725.688s][info ][gc,phases   ] GC(2927)
Concurrent Mark Free 0.001ms
[2022-01-27T18:51:41.327+0000][34725.688s][debug][gc,phases   ] GC(2927)
Concurrent References Process (ZDriver) 0.738ms
[2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
Concurrent Weak Roots OopStorageSet (ZWorker#13) 3.344ms
[2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
Concurrent Weak Roots OopStorageSet (ZWorker#4) 3.354ms
[2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
Concurrent Weak Roots OopStorageSet (ZWorker#8) 3.366ms
[2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
Concurrent Weak Roots OopStorageSet (ZWorker#0) 3.328ms
[2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
Concurrent Weak Roots OopStorageSet (ZWorker#11) 3.365ms
[2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
Concurrent Weak Roots OopStorageSet (ZWorker#14) 3.336ms
[2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
Concurrent Weak Roots OopStorageSet (ZWorker#5) 3.362ms
[2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
Concurrent Weak Roots OopStorageSet (ZWorker#1) 3.354ms
[2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
Concurrent Weak Roots OopStorageSet (ZWorker#9) 3.351ms
[2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
Concurrent Weak Roots OopStorageSet (ZWorker#3) 3.328ms
[2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
Concurrent Weak Roots OopStorageSet (ZWorker#17) 3.333ms
[2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
Concurrent Weak Roots OopStorageSet (ZWorker#10) 3.334ms
[2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
Concurrent Weak Roots OopStorageSet (ZWorker#2) 3.345ms
[2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
Concurrent Weak Roots OopStorageSet (ZWorker#7) 3.338ms
[2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
Concurrent Weak Roots OopStorageSet (ZWorker#6) 3.329ms
[2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
Concurrent Weak Roots OopStorageSet (ZWorker#15) 3.350ms
[2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
Concurrent Weak Roots OopStorageSet (ZWorker#16) 3.366ms
[2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
Concurrent Weak Roots OopStorageSet (ZWorker#12) 3.370ms
[2022-01-27T18:51:41.331+0000][34725.693s][debug][gc,phases   ] GC(2927)
Concurrent References Enqueue (ZDriver) 0.011ms
[2022-01-27T18:51:41.331+0000][34725.693s][info ][gc,phases   ] GC(2927)
Concurrent Process Non-Strong References 5.192ms
[2022-01-27T18:51:41.333+0000][34725.695s][info ][gc,phases   ] GC(2927)
Concurrent Reset Relocation Set 2.186ms
[2022-01-27T18:51:41.349+0000][34725.711s][info ][gc,phases   ] GC(2927)
Concurrent Select Relocation Set 15.721ms
[2022-01-27T18:51:41.350+0000][34725.711s][info ][gc,phases   ] GC(2927)
Pause Relocate Start 0.016ms
[2022-01-27T18:51:41.350+0000][34725.712s][info ][safepoint   ] Safepoint
"ZRelocateStart", Time since last: 23235346 ns, Reaching safepoint: 315064
ns, At safepoint: 290299 ns, Total: 605363 ns
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,phases   ] GC(2927)
Concurrent Relocate 19.017ms
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,marking  ] GC(2927)
Mark: 16 stripe(s), 4 proactive flush(es), 1 terminate flush(es), 0
completion(s), 0 continuation(s)
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,marking  ] GC(2927)
Mark Stack Usage: 32M
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,metaspace] GC(2927)
Metaspace: 169M used, 178M committed, 1184M reserved
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,ref      ] GC(2927)
Soft: 40709 encountered, 0 discovered, 0 enqueued
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,ref      ] GC(2927)
Weak: 33476 encountered, 7667 discovered, 4604 enqueued
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,ref      ] GC(2927)
Final: 415 encountered, 0 discovered, 0 enqueued
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,ref      ] GC(2927)
Phantom: 5155 encountered, 3828 discovered, 205 enqueued
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,reloc    ] GC(2927)
Small Pages: 23233 / 46466M, Empty: 17842M, Relocated: 61M, In-Place: 0
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,reloc    ] GC(2927)
Medium Pages: 161 / 5152M, Empty: 5056M, Relocated: 9M, In-Place: 0
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,reloc    ] GC(2927)
Large Pages: 0 / 0M, Empty: 0M, Relocated: 0M, In-Place: 0
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,reloc    ] GC(2927)
Forwarding Usage: 13M
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
Min Capacity: 64000M(100%)
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
Max Capacity: 64000M(100%)
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
Soft Max Capacity: 64000M(100%)
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
             Mark Start          Mark End        Relocate Start
 Relocate End           High               Low
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
 Capacity:    64000M (100%)      64000M (100%)      64000M (100%)
 64000M (100%)      64000M (100%)      64000M (100%)
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
   Free:    12382M (19%)       11392M (18%)       34172M (53%)       61184M
(96%)       61184M (96%)       11326M (18%)
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
   Used:    51618M (81%)       52608M (82%)       29828M (47%)        2816M
(4%)        52674M (82%)        2816M (4%)
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
   Live:         -              1303M (2%)         1303M (2%)         1303M
(2%)             -                  -
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
Allocated:         -               990M (2%)         1108M (2%)
1195M (2%)             -                  -
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
Garbage:         -             50314M (79%)       27416M (43%)         316M
(0%)             -                  -
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
Reclaimed:         -                  -             22898M (36%)
49997M (78%)            -                  -
[2022-01-27T18:51:41.369+0000][34725.731s][info ][gc          ] GC(2927)
Garbage Collection (Proactive) 51618M(81%)->2816M(4%)
[2022-01-27T18:51:51.319+0000][34735.681s][info ][gc,start    ] GC(2928)
Garbage Collection (Allocation Rate)
[2022-01-27T18:51:51.321+0000][34735.682s][info ][gc,phases   ] GC(2928)
Pause Mark Start 0.042ms
[2022-01-27T18:51:51.321+0000][34735.683s][info ][safepoint   ] Safepoint
"ZMarkStart", Time since last: 9969777477 ns, Reaching safepoint: 887200
ns, At safepoint: 523517 ns, Total: 1410717 ns
[2022-01-27T18:51:51.334+0000][34735.696s][debug][gc,phases   ] GC(2928)
Concurrent Roots OopStorageSet (ZWorker#17) 0.542ms
[2022-01-27T18:51:51.334+0000][34735.696s][debug][gc,phases   ] GC(2928)
Concurrent Roots OopStorageSet (ZWorker#10) 0.575ms
[2022-01-27T18:51:51.343+0000][34735.705s][debug][gc,phases   ] GC(2928)
Concurrent Roots ClassLoaderDataGraph (ZWorker#17) 9.161ms
[2022-01-27T18:51:51.344+0000][34735.706s][debug][gc,phases   ] GC(2928)
Concurrent Roots ClassLoaderDataGraph (ZWorker#10) 9.817ms
[2022-01-27T18:51:51.351+0000][34735.713s][debug][gc,phases   ] GC(2928)
Concurrent Roots JavaThreads (ZWorker#10) 7.569ms
[2022-01-27T18:51:51.351+0000][34735.713s][debug][gc,phases   ] GC(2928)
Concurrent Roots JavaThreads (ZWorker#17) 8.243ms
[2022-01-27T18:51:51.407+0000][34735.769s][debug][gc,phases   ] GC(2928)
Concurrent Roots CodeCache (ZWorker#17) 55.952ms
[2022-01-27T18:51:51.407+0000][34735.769s][debug][gc,phases   ] GC(2928)
Concurrent Roots CodeCache (ZWorker#10) 55.988ms
[2022-01-27T18:51:52.336+0000][34736.698s][debug][gc,phases   ] GC(2928)
Concurrent Mark Try Terminate (ZWorker#14) 1.064ms
[2022-01-27T18:51:52.337+0000][34736.699s][debug][gc,phases   ] GC(2928)
Concurrent Mark Try Flush (ZWorker#12) 2.343ms
[2022-01-27T18:51:52.342+0000][34736.703s][debug][gc,phases   ] GC(2928)
Concurrent Mark Try Flush (ZWorker#12) 0.685ms
[2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
Concurrent Mark Try Flush (ZWorker#12) 0.294ms
[2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
Concurrent Mark Try Terminate (ZWorker#12) 0.320ms
[2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
Concurrent Mark Try Terminate (ZWorker#12) 0.000ms
[2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
Concurrent Mark (ZWorker#12) 934.304ms
[2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
Concurrent Mark Try Terminate (ZWorker#14) 1.062ms
[2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
Concurrent Mark (ZWorker#14) 934.339ms

*[2022-01-27T18:51:52.342+0000][34736.704s][info ][gc,phases   ] GC(2928)
Concurrent Mark 1021.048ms*[2022-01-27T18:51:52.342+0000][34736.704s][info
][gc,phases   ] GC(2928) Pause Mark End 0.095ms
[2022-01-27T18:51:52.343+0000][34736.704s][info ][safepoint   ] Safepoint
"ZMarkEnd", Time since last: 1021138351 ns, Reaching safepoint: 256951 ns,
At safepoint: 333825 ns, Total: 590776 ns
[2022-01-27T18:51:52.343+0000][34736.704s][info ][gc,phases   ] GC(2928)
Concurrent Mark Free 0.001ms
[2022-01-27T18:51:52.345+0000][34736.707s][debug][gc,phases   ] GC(2928)
Concurrent References Process (ZDriver) 2.019ms
[2022-01-27T18:51:52.364+0000][34736.726s][debug][gc,phases   ] GC(2928)
Concurrent Weak Roots OopStorageSet (ZWorker#13) 19.096ms
[2022-01-27T18:51:52.364+0000][34736.726s][debug][gc,phases   ] GC(2928)
Concurrent Weak Roots OopStorageSet (ZWorker#7) 19.068ms
[2022-01-27T18:51:52.365+0000][34736.727s][debug][gc,phases   ] GC(2928)
Concurrent References Enqueue (ZDriver) 0.010ms
[2022-01-27T18:51:52.365+0000][34736.727s][info ][gc,phases   ] GC(2928)
Concurrent Process Non-Strong References 22.133ms
[2022-01-27T18:51:52.368+0000][34736.730s][info ][gc,phases   ] GC(2928)
Concurrent Reset Relocation Set 2.875ms
[2022-01-27T18:51:52.384+0000][34736.745s][info ][gc,phases   ] GC(2928)
Concurrent Select Relocation Set 15.834ms
[2022-01-27T18:51:52.384+0000][34736.746s][info ][gc,phases   ] GC(2928)
Pause Relocate Start 0.015ms
[2022-01-27T18:51:52.384+0000][34736.746s][info ][safepoint   ] Safepoint
"ZRelocateStart", Time since last: 41003214 ns, Reaching safepoint: 303280
ns, At safepoint: 155476 ns, Total: 458756 ns
[2022-01-27T18:51:52.445+0000][34736.806s][info ][gc,phases   ] GC(2928)
Concurrent Relocate 60.419ms
[2022-01-27T18:51:52.445+0000][34736.806s][info ][gc,marking  ] GC(2928)
Mark: 2 stripe(s), 2 proactive flush(es), 1 terminate flush(es), 0
completion(s), 0 continuation(s)
[2022-01-27T18:51:52.445+0000][34736.806s][info ][gc,marking  ] GC(2928)
Mark Stack Usage: 32M
[2022-01-27T18:51:52.445+0000][34736.806s][info ][gc,metaspace] GC(2928)
Metaspace: 169M used, 178M committed, 1184M reserved
[2022-01-27T18:51:52.445+0000][34736.806s][info ][gc,ref      ] GC(2928)
Soft: 40719 encountered, 0 discovered, 0 enqueued
[2022-01-27T18:51:52.445+0000][34736.806s][info ][gc,ref      ] GC(2928)
Weak: 33189 encountered, 9016 discovered, 4318 enqueued
[2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,ref      ] GC(2928)
Final: 795 encountered, 387 discovered, 380 enqueued
[2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,ref      ] GC(2928)
Phantom: 5175 encountered, 3937 discovered, 219 enqueued
[2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,reloc    ] GC(2928)
Small Pages: 20919 / 41838M, Empty: 15862M, Relocated: 87M, In-Place: 0
[2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,reloc    ] GC(2928)
Medium Pages: 130 / 4160M, Empty: 4064M, Relocated: 11M, In-Place: 0
[2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,reloc    ] GC(2928)
Large Pages: 20 / 128M, Empty: 128M, Relocated: 0M, In-Place: 0
[2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,reloc    ] GC(2928)
Forwarding Usage: 19M
[2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
Min Capacity: 64000M(100%)
[2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
Max Capacity: 64000M(100%)
[2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
Soft Max Capacity: 64000M(100%)
[2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
             Mark Start          Mark End        Relocate Start
 Relocate End           High               Low
[2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
 Capacity:    64000M (100%)      64000M (100%)      64000M (100%)
 64000M (100%)      64000M (100%)      64000M (100%)
[2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
   Free:    17874M (28%)       13228M (21%)       33154M (52%)       57394M
(90%)       57434M (90%)       13142M (21%)
[2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
   Used:    46126M (72%)       50772M (79%)       30846M (48%)        6606M
(10%)       50858M (79%)        6566M (10%)
[2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
   Live:         -              1301M (2%)         1301M (2%)         1301M
(2%)             -                  -
[2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
Allocated:         -              4646M (7%)         4774M (7%)
5004M (8%)             -                  -
[2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
Garbage:         -             44824M (70%)       24770M (39%)         299M
(0%)             -                  -
[2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
Reclaimed:         -                  -             20054M (31%)
44524M (70%)            -                  -
[2022-01-27T18:51:52.445+0000][34736.807s][info ][gc          ] GC(2928)
Garbage Collection (Allocation Rate) 46126M(72%)->6606M(10%)
[2022-01-27T18:52:00.820+0000][34745.182s][info ][gc,start    ] GC(2929)
Garbage Collection (Allocation Rate)
[2022-01-27T18:52:00.821+0000][34745.183s][info ][gc,phases   ] GC(2929)
Pause Mark Start 0.057ms
[2022-01-27T18:52:00.822+0000][34745.184s][info ][safepoint   ] Safepoint
"ZMarkStart", Time since last: 8436197260 ns, Reaching safepoint: 1019671
ns, At safepoint: 515965 ns, Total: 1535636 ns
[2022-01-27T18:52:00.835+0000][34745.197s][debug][gc,phases   ] GC(2929)
Concurrent Roots OopStorageSet (ZWorker#5) 0.991ms
[2022-01-27T18:52:00.851+0000][34745.213s][debug][gc,phases   ] GC(2929)
Concurrent Roots ClassLoaderDataGraph (ZWorker#5) 15.777ms
[2022-01-27T18:52:00.864+0000][34745.226s][debug][gc,phases   ] GC(2929)
Concurrent Roots JavaThreads (ZWorker#5) 12.769ms
[2022-01-27T18:52:00.945+0000][34745.307s][debug][gc,phases   ] GC(2929)
Concurrent Roots CodeCache (ZWorker#5) 81.380ms
[2022-01-27T18:52:02.766+0000][34747.127s][debug][gc,phases   ] GC(2929)
Concurrent Mark Try Flush (ZWorker#16) 1.613ms
[2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
Concurrent Mark Try Flush (ZWorker#16) 0.717ms
[2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
Concurrent Mark Try Flush (ZWorker#16) 0.292ms
[2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
Concurrent Mark Try Terminate (ZWorker#16) 0.306ms
[2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
Concurrent Mark Try Terminate (ZWorker#16) 0.000ms
[2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
Concurrent Mark (ZWorker#16) 1825.719ms

*[2022-01-27T18:52:02.771+0000][34747.133s][info ][gc,phases   ] GC(2929)
Concurrent Mark 1949.325ms*[2022-01-27T18:52:02.774+0000][34747.136s][info
][gc,phases   ] GC(2929) Pause Mark End 0.212ms
[2022-01-27T18:52:02.775+0000][34747.136s][info ][safepoint   ] Safepoint
"ZMarkEnd", Time since last: 1949455739 ns, Reaching safepoint: 2536816 ns,
At safepoint: 606241 ns, Total: 3143057 ns
[2022-01-27T18:52:02.775+0000][34747.136s][info ][gc,phases   ] GC(2929)
Concurrent Mark Free 0.001ms
[2022-01-27T18:52:02.778+0000][34747.139s][debug][gc,phases   ] GC(2929)
Concurrent References Process (ZDriver) 3.161ms
[2022-01-27T18:52:02.813+0000][34747.175s][debug][gc,phases   ] GC(2929)
Concurrent Weak Roots OopStorageSet (ZWorker#4) 35.717ms
[2022-01-27T18:52:02.814+0000][34747.176s][debug][gc,phases   ] GC(2929)
Concurrent References Enqueue (ZDriver) 0.007ms
[2022-01-27T18:52:02.814+0000][34747.176s][info ][gc,phases   ] GC(2929)
Concurrent Process Non-Strong References 39.943ms
[2022-01-27T18:52:02.818+0000][34747.180s][info ][gc,phases   ] GC(2929)
Concurrent Reset Relocation Set 3.587ms
[2022-01-27T18:52:02.833+0000][34747.195s][info ][gc,phases   ] GC(2929)
Concurrent Select Relocation Set 15.208ms
[2022-01-27T18:52:02.834+0000][34747.195s][info ][gc,phases   ] GC(2929)
Pause Relocate Start 0.017ms
[2022-01-27T18:52:02.834+0000][34747.196s][info ][safepoint   ] Safepoint
"ZRelocateStart", Time since last: 58881245 ns, Reaching safepoint: 286775
ns, At safepoint: 232929 ns, Total: 519704 ns
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,phases   ] GC(2929)
Concurrent Relocate 107.558ms
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,marking  ] GC(2929)
Mark: 1 stripe(s), 2 proactive flush(es), 1 terminate flush(es), 0
completion(s), 0 continuation(s)
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,marking  ] GC(2929)
Mark Stack Usage: 32M
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,metaspace] GC(2929)
Metaspace: 169M used, 178M committed, 1184M reserved
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,ref      ] GC(2929)
Soft: 40717 encountered, 0 discovered, 0 enqueued
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,ref      ] GC(2929)
Weak: 32517 encountered, 8597 discovered, 3652 enqueued
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,ref      ] GC(2929)
Final: 415 encountered, 6 discovered, 0 enqueued
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,ref      ] GC(2929)
Phantom: 5031 encountered, 3633 discovered, 77 enqueued
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,reloc    ] GC(2929)
Small Pages: 18075 / 36150M, Empty: 12846M, Relocated: 76M, In-Place: 0
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,reloc    ] GC(2929)
Medium Pages: 117 / 3744M, Empty: 3584M, Relocated: 10M, In-Place: 0
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,reloc    ] GC(2929)
Large Pages: 16 / 104M, Empty: 104M, Relocated: 0M, In-Place: 0
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,reloc    ] GC(2929)
Forwarding Usage: 15M
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,heap     ] GC(2929)
Min Capacity: 64000M(100%)
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,heap     ] GC(2929)
Max Capacity: 64000M(100%)
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,heap     ] GC(2929)
Soft Max Capacity: 64000M(100%)
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,heap     ] GC(2929)
             Mark Start          Mark End        Relocate Start
 Relocate End           High               Low
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,heap     ] GC(2929)
 Capacity:    64000M (100%)      64000M (100%)      64000M (100%)
 64000M (100%)      64000M (100%)      64000M (100%)
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,heap     ] GC(2929)
   Free:    24002M (38%)       15544M (24%)       31836M (50%)       53224M
(83%)       53414M (83%)       15410M (24%)
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,heap     ] GC(2929)
   Used:    39998M (62%)       48456M (76%)       32164M (50%)       10776M
(17%)       48590M (76%)       10586M (17%)
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,heap     ] GC(2929)
   Live:         -              1283M (2%)         1283M (2%)         1283M
(2%)             -                  -
[2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,heap     ] GC(2929)
Allocated:         -              8458M (13%)        8700M (14%)
 9205M (14%)            -                  -
[2022-01-27T18:52:02.942+0000][34747.304s][info ][gc,heap     ] GC(2929)
Garbage:         -             38714M (60%)       22180M (35%)         286M
(0%)             -                  -
[2022-01-27T18:52:02.942+0000][34747.304s][info ][gc,heap     ] GC(2929)
Reclaimed:         -                  -             16534M (26%)
38427M (60%)            -                  -
[2022-01-27T18:52:02.942+0000][34747.304s][info ][gc          ] GC(2929)
Garbage Collection (Allocation Rate) 39998M(62%)->10776M(17%)


Thanks
Sundar


On Thu, Jan 27, 2022 at 11:00 AM Sundara Mohan M <m.sundar85 at gmail.com>
wrote:

> Hi,
>    We are using ZGC in production environment with following settings
> *[2022-01-27T09:12:55.684+0000][0.046s][info][gc,init] Version: 17.0.1+12
> (release)*
> ..
> *[2022-01-27T09:12:55.684+0000][0.046s][info][gc,init] GC Workers: 18
> (dynamic)*
> ...
> [2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Min Capacity: 64000M
> [2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Initial Capacity:
> 64000M
> *[2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Max Capacity:
> 64000M*
> [2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Medium Page Size: 32M
> [2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Pre-touch: Disabled
> [2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Available space on
> backing filesystem: N/A
> [2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Uncommit: Implicitly
> Disabled (-Xms equals -Xmx)
> *[2022-01-27T09:13:05.275+0000][9.636s][info][gc,init] Runtime Workers: 44*
> [2022-01-27T09:13:05.278+0000][9.640s][info][gc     ] Using The Z Garbage
> Collector
>
> We run into GC cycle every 8-10 seconds. In some GC cycle Concurrent Mark
> phase is taking 2*s* and in some cycle it is taking 200*ms.*
>
> After enabling debug log for gc,phase saw in some cases more ZWorkers are
> doing work in other case it is given to only 1 or 2 ZWorker.
>
> Example log where work is done by multiple thread and took only 200ms
> [2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#12) 150.092ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#0) 1.059ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#0) 150.279ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#14) 1.055ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#14) 149.355ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#8) 1.060ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#8) 150.312ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#6) 1.060ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#6) 150.323ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#17) 1.062ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#11) 1.058ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#13) 1.060ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#2) 1.059ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#2) 150.423ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#13) 150.437ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#11) 150.374ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#3) 1.061ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#3) 150.537ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#7) 1.061ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#7) 150.550ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#17) 150.364ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#15) 1.056ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#16) 1.059ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#1) 1.060ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#1) 150.613ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#4) 1.064ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#16) 150.584ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#10) 1.059ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#10) 150.706ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#4) 150.629ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#15) 150.601ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#9) 1.028ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#9) 150.938ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#5) 1.057ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#5) 151.063ms
> *[2022-01-27T18:51:41.325+0000][34725.687s][info ][gc,phases   ] GC(2927)
> Concurrent Mark 204.119ms*
>
> Another case where it was run by only 1/2 worker and took 1/2s respectively
> ...
> *[2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
> Concurrent Mark (ZWorker#12) 934.304ms*
> [2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
> Concurrent Mark Try Terminate (ZWorker#14) 1.062ms
> *[2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
> Concurrent Mark (ZWorker#14) 934.339ms*
> *[2022-01-27T18:51:52.342+0000][34736.704s][info ][gc,phases   ] GC(2928)
> Concurrent Mark 1021.048ms*
>
>
> [2022-01-27T18:52:00.822+0000][34745.184s][info ][safepoint   ] Safepoint
> "ZMarkStart", Time since last: 8436197260 ns, Reaching safepoint: 1019671
> ns, At safepoint: 515965 ns, Total: 1
> 535636 ns
> [2022-01-27T18:52:00.835+0000][34745.197s][debug][gc,phases   ] GC(2929)
> Concurrent Roots OopStorageSet (ZWorker#5) 0.991ms
> [2022-01-27T18:52:00.851+0000][34745.213s][debug][gc,phases   ] GC(2929)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#5) 15.777ms
> [2022-01-27T18:52:00.864+0000][34745.226s][debug][gc,phases   ] GC(2929)
> Concurrent Roots JavaThreads (ZWorker#5) 12.769ms
> [2022-01-27T18:52:00.945+0000][34745.307s][debug][gc,phases   ] GC(2929)
> Concurrent Roots CodeCache (ZWorker#5) 81.380ms
> [2022-01-27T18:52:02.766+0000][34747.127s][debug][gc,phases   ] GC(2929)
> Concurrent Mark Try Flush (ZWorker#16) 1.613ms
> [2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
> Concurrent Mark Try Flush (ZWorker#16) 0.717ms
> [2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
> Concurrent Mark Try Flush (ZWorker#16) 0.292ms
> [2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
> Concurrent Mark Try Terminate (ZWorker#16) 0.306ms
> [2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
> Concurrent Mark Try Terminate (ZWorker#16) 0.000ms
>
>
> *[2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
> Concurrent Mark (ZWorker#16)
> 1825.719ms[2022-01-27T18:52:02.771+0000][34747.133s][info ][gc,phases   ]
> GC(2929) Concurrent Mark 1949.325ms*
>
> Even though we have 18 threads, does anyone have any idea why the work is
> not distributed among multiple workers?
>
> Thanks
> Sundar
>
>

From per.liden at oracle.com  Mon Jan 31 08:35:41 2022
From: per.liden at oracle.com (Per Liden)
Date: Mon, 31 Jan 2022 09:35:41 +0100
Subject: ZGC Concurrent Mark Work distribution
In-Reply-To: <CACGCMVppLjbxCHU7zBbMm5vUec_hAiqtKPJ9Qic-Bbo93J_gfA@mail.gmail.com>
References: <CACGCMVpmYWS=tGPa8mOc3v5EB5mpYJr5F8yNmnRmT-au=SbDBA@mail.gmail.com>
 <CACGCMVppLjbxCHU7zBbMm5vUec_hAiqtKPJ9Qic-Bbo93J_gfA@mail.gmail.com>
Message-ID: <99897504-f8da-9127-50a5-fe98041ddf84@oracle.com>

Hi,

It's hard to tell without seeing the complete GC log, but I suspect that 
the heuristics just picks different number of threads here. The 
heuristics will pick the number of threads currently needed, based on 
available memory, allocation rate, etc. Each GC cycle will print "Using 
XX workers" in the log. I wrote about this feature in my latest blog 
post under "Dynamic Number of GC Threads" 
(https://malloc.se/blog/zgc-jdk17).

cheers,
Per

On 1/27/22 20:20, Sundara Mohan M wrote:
> Adding complete log of that phase in case if it helps
> -01-27T18:51:41.121+0000][34725.483s][info ][gc,phases   ] GC(2927) Pause
> Mark Start 0.046ms
> [2022-01-27T18:51:41.121+0000][34725.483s][info ][safepoint   ] Safepoint
> "ZMarkStart", Time since last: 8392610571 ns, Reaching safepoint: 606586
> ns, At safepoint: 438685 ns, Total: 1045271 ns
> [2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
> Concurrent Roots OopStorageSet (ZWorker#11) 0.000ms
> [2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
> Concurrent Roots OopStorageSet (ZWorker#8) 0.102ms
> [2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
> Concurrent Roots OopStorageSet (ZWorker#0) 0.110ms
> [2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
> Concurrent Roots OopStorageSet (ZWorker#2) 0.062ms
> [2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
> Concurrent Roots OopStorageSet (ZWorker#9) 0.183ms
> [2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
> Concurrent Roots OopStorageSet (ZWorker#13) 0.209ms
> [2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
> Concurrent Roots OopStorageSet (ZWorker#6) 0.001ms
> [2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
> Concurrent Roots OopStorageSet (ZWorker#1) 0.098ms
> [2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
> Concurrent Roots OopStorageSet (ZWorker#12) 0.109ms
> [2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
> Concurrent Roots OopStorageSet (ZWorker#5) 0.112ms
> [2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
> Concurrent Roots OopStorageSet (ZWorker#17) 0.000ms
> [2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
> Concurrent Roots OopStorageSet (ZWorker#15) 0.103ms
> [2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
> Concurrent Roots OopStorageSet (ZWorker#16) 0.147ms
> [2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
> Concurrent Roots OopStorageSet (ZWorker#7) 0.067ms
> [2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
> Concurrent Roots OopStorageSet (ZWorker#4) 0.143ms
> [2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
> Concurrent Roots OopStorageSet (ZWorker#10) 0.048ms
> [2022-01-27T18:51:41.134+0000][34725.496s][debug][gc,phases   ] GC(2927)
> Concurrent Roots OopStorageSet (ZWorker#3) 0.057ms
> [2022-01-27T18:51:41.135+0000][34725.496s][debug][gc,phases   ] GC(2927)
> Concurrent Roots OopStorageSet (ZWorker#14) 0.249ms
> [2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#5) 4.032ms
> [2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#16) 3.921ms
> [2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#7) 3.888ms
> [2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#3) 3.685ms
> [2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#4) 3.857ms
> [2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#10) 3.725ms
> [2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#12) 4.064ms
> [2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#6) 4.137ms
> [2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#17) 3.998ms
> [2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#0) 4.301ms
> [2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#8) 4.335ms
> [2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#14) 3.658ms
> [2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#15) 3.966ms
> [2022-01-27T18:51:41.139+0000][34725.500s][debug][gc,phases   ] GC(2927)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#13) 4.267ms
> [2022-01-27T18:51:41.139+0000][34725.501s][debug][gc,phases   ] GC(2927)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#1) 4.165ms
> [2022-01-27T18:51:41.139+0000][34725.501s][debug][gc,phases   ] GC(2927)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#2) 4.619ms
> [2022-01-27T18:51:41.140+0000][34725.502s][debug][gc,phases   ] GC(2927)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#9) 5.835ms
> [2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
> Concurrent Roots JavaThreads (ZWorker#7) 4.247ms
> [2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
> Concurrent Roots JavaThreads (ZWorker#3) 4.243ms
> [2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
> Concurrent Roots JavaThreads (ZWorker#6) 4.168ms
> [2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
> Concurrent Roots JavaThreads (ZWorker#17) 4.091ms
> [2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
> Concurrent Roots JavaThreads (ZWorker#9) 2.713ms
> [2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
> Concurrent Roots JavaThreads (ZWorker#10) 4.187ms
> [2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
> Concurrent Roots JavaThreads (ZWorker#0) 4.071ms
> [2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
> Concurrent Roots JavaThreads (ZWorker#2) 3.897ms
> [2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
> Concurrent Roots JavaThreads (ZWorker#16) 4.277ms
> [2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
> Concurrent Roots JavaThreads (ZWorker#5) 4.319ms
> [2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
> Concurrent Roots JavaThreads (ZWorker#4) 4.231ms
> [2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
> Concurrent Roots JavaThreads (ZWorker#15) 3.997ms
> [2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
> Concurrent Roots JavaThreads (ZWorker#13) 3.983ms
> [2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
> Concurrent Roots JavaThreads (ZWorker#14) 4.049ms
> [2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
> Concurrent Roots JavaThreads (ZWorker#1) 3.968ms
> [2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
> Concurrent Roots JavaThreads (ZWorker#8) 4.092ms
> [2022-01-27T18:51:41.143+0000][34725.505s][debug][gc,phases   ] GC(2927)
> Concurrent Roots JavaThreads (ZWorker#12) 4.155ms
> [2022-01-27T18:51:41.148+0000][34725.510s][debug][gc,phases   ] GC(2927)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#11) 13.665ms
> [2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
> Concurrent Roots CodeCache (ZWorker#12) 28.078ms
> [2022-01-27T18:51:41.174+0000][34725.536s][debug][gc,phases   ] GC(2927)
> Concurrent Roots CodeCache (ZWorker#0) 30.133ms
> [2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
> Concurrent Roots CodeCache (ZWorker#3) 30.557ms
> [2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
> Concurrent Roots CodeCache (ZWorker#13) 29.600ms
> [2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
> Concurrent Roots CodeCache (ZWorker#5) 29.829ms
> [2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
> Concurrent Roots CodeCache (ZWorker#9) 30.148ms
> [2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
> Concurrent Roots CodeCache (ZWorker#10) 30.083ms
> [2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
> Concurrent Roots CodeCache (ZWorker#15) 29.707ms
> [2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
> Concurrent Roots CodeCache (ZWorker#7) 30.647ms
> [2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
> Concurrent Roots CodeCache (ZWorker#4) 29.814ms
> [2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
> Concurrent Roots CodeCache (ZWorker#8) 29.350ms
> [2022-01-27T18:51:41.174+0000][34725.536s][debug][gc,phases   ] GC(2927)
> Concurrent Roots CodeCache (ZWorker#14) 29.569ms
> [2022-01-27T18:51:41.174+0000][34725.536s][debug][gc,phases   ] GC(2927)
> Concurrent Roots CodeCache (ZWorker#1) 29.496ms
> [2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
> Concurrent Roots CodeCache (ZWorker#17) 30.227ms
> [2022-01-27T18:51:41.174+0000][34725.536s][debug][gc,phases   ] GC(2927)
> Concurrent Roots CodeCache (ZWorker#6) 30.641ms
> [2022-01-27T18:51:41.174+0000][34725.536s][debug][gc,phases   ] GC(2927)
> Concurrent Roots CodeCache (ZWorker#16) 29.984ms
> [2022-01-27T18:51:41.174+0000][34725.536s][debug][gc,phases   ] GC(2927)
> Concurrent Roots CodeCache (ZWorker#2) 30.087ms
> [2022-01-27T18:51:41.174+0000][34725.535s][debug][gc,phases   ] GC(2927)
> Concurrent Roots CodeCache (ZWorker#11) 25.501ms
> [2022-01-27T18:51:41.319+0000][34725.681s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#9) 1.061ms
> [2022-01-27T18:51:41.319+0000][34725.681s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#6) 1.062ms
> [2022-01-27T18:51:41.319+0000][34725.681s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#15) 1.060ms
> [2022-01-27T18:51:41.319+0000][34725.681s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#16) 1.061ms
> [2022-01-27T18:51:41.319+0000][34725.681s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#3) 1.065ms
> [2022-01-27T18:51:41.319+0000][34725.681s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#5) 1.061ms
> [2022-01-27T18:51:41.320+0000][34725.681s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Flush (ZWorker#12) 1.921ms
> [2022-01-27T18:51:41.322+0000][34725.684s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#9) 1.061ms
> [2022-01-27T18:51:41.322+0000][34725.684s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#8) 1.062ms
> [2022-01-27T18:51:41.322+0000][34725.684s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#1) 1.063ms
> [2022-01-27T18:51:41.322+0000][34725.684s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#13) 1.059ms
> [2022-01-27T18:51:41.322+0000][34725.684s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#14) 1.066ms
> [2022-01-27T18:51:41.322+0000][34725.684s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#7) 1.065ms
> [2022-01-27T18:51:41.322+0000][34725.684s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#16) 1.061ms
> [2022-01-27T18:51:41.322+0000][34725.684s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#15) 1.063ms
> [2022-01-27T18:51:41.323+0000][34725.685s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#5) 1.061ms
> [2022-01-27T18:51:41.323+0000][34725.685s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#9) 1.057ms
> [2022-01-27T18:51:41.323+0000][34725.685s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Flush (ZWorker#12) 0.743ms
> [2022-01-27T18:51:41.323+0000][34725.685s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#0) 1.061ms
> [2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#14) 1.057ms
> [2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#8) 1.060ms
> [2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#6) 1.059ms
> [2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#17) 1.061ms
> [2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#11) 1.063ms
> [2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#13) 1.063ms
> [2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#2) 1.058ms
> [2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#3) 1.061ms
> [2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#7) 1.060ms
> [2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Flush (ZWorker#12) 0.356ms
> [2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#4) 1.061ms
> [2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#15) 1.059ms
> [2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#10) 1.060ms
> [2022-01-27T18:51:41.324+0000][34725.685s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#16) 1.060ms
> [2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#1) 1.060ms
> [2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Flush (ZWorker#12) 0.311ms
> [2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#9) 1.060ms
> [2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#5) 1.058ms
> [2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Flush (ZWorker#12) 0.285ms
> [2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#12) 0.297ms
> [2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#12) 0.000ms
> [2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#12) 150.092ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#0) 1.059ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#0) 150.279ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#14) 1.055ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#14) 149.355ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#8) 1.060ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#8) 150.312ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#6) 1.060ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#6) 150.323ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#17) 1.062ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#11) 1.058ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#13) 1.060ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#2) 1.059ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#2) 150.423ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#13) 150.437ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#11) 150.374ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#3) 1.061ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#3) 150.537ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#7) 1.061ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#7) 150.550ms
> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#17) 150.364ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#15) 1.056ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#16) 1.059ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#1) 1.060ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#1) 150.613ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#4) 1.064ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#16) 150.584ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#10) 1.059ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#10) 150.706ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#4) 150.629ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#15) 150.601ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#9) 1.028ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#9) 150.938ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark Try Terminate (ZWorker#5) 1.057ms
> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
> Concurrent Mark (ZWorker#5) 151.063ms
> *[2022-01-27T18:51:41.325+0000][34725.687s][info ][gc,phases   ] GC(2927)
> Concurrent Mark 204.119ms*
> [2022-01-27T18:51:41.326+0000][34725.688s][info ][gc,phases   ] GC(2927)
> Pause Mark End 0.070ms
> [2022-01-27T18:51:41.326+0000][34725.688s][info ][safepoint   ] Safepoint
> "ZMarkEnd", Time since last: 204207403 ns, Reaching safepoint: 210401 ns,
> At safepoint: 237941 ns, Total: 448342 ns
> [2022-01-27T18:51:41.326+0000][34725.688s][info ][gc,phases   ] GC(2927)
> Concurrent Mark Free 0.001ms
> [2022-01-27T18:51:41.327+0000][34725.688s][debug][gc,phases   ] GC(2927)
> Concurrent References Process (ZDriver) 0.738ms
> [2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
> Concurrent Weak Roots OopStorageSet (ZWorker#13) 3.344ms
> [2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
> Concurrent Weak Roots OopStorageSet (ZWorker#4) 3.354ms
> [2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
> Concurrent Weak Roots OopStorageSet (ZWorker#8) 3.366ms
> [2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
> Concurrent Weak Roots OopStorageSet (ZWorker#0) 3.328ms
> [2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
> Concurrent Weak Roots OopStorageSet (ZWorker#11) 3.365ms
> [2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
> Concurrent Weak Roots OopStorageSet (ZWorker#14) 3.336ms
> [2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
> Concurrent Weak Roots OopStorageSet (ZWorker#5) 3.362ms
> [2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
> Concurrent Weak Roots OopStorageSet (ZWorker#1) 3.354ms
> [2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
> Concurrent Weak Roots OopStorageSet (ZWorker#9) 3.351ms
> [2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
> Concurrent Weak Roots OopStorageSet (ZWorker#3) 3.328ms
> [2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
> Concurrent Weak Roots OopStorageSet (ZWorker#17) 3.333ms
> [2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
> Concurrent Weak Roots OopStorageSet (ZWorker#10) 3.334ms
> [2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
> Concurrent Weak Roots OopStorageSet (ZWorker#2) 3.345ms
> [2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
> Concurrent Weak Roots OopStorageSet (ZWorker#7) 3.338ms
> [2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
> Concurrent Weak Roots OopStorageSet (ZWorker#6) 3.329ms
> [2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
> Concurrent Weak Roots OopStorageSet (ZWorker#15) 3.350ms
> [2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
> Concurrent Weak Roots OopStorageSet (ZWorker#16) 3.366ms
> [2022-01-27T18:51:41.330+0000][34725.692s][debug][gc,phases   ] GC(2927)
> Concurrent Weak Roots OopStorageSet (ZWorker#12) 3.370ms
> [2022-01-27T18:51:41.331+0000][34725.693s][debug][gc,phases   ] GC(2927)
> Concurrent References Enqueue (ZDriver) 0.011ms
> [2022-01-27T18:51:41.331+0000][34725.693s][info ][gc,phases   ] GC(2927)
> Concurrent Process Non-Strong References 5.192ms
> [2022-01-27T18:51:41.333+0000][34725.695s][info ][gc,phases   ] GC(2927)
> Concurrent Reset Relocation Set 2.186ms
> [2022-01-27T18:51:41.349+0000][34725.711s][info ][gc,phases   ] GC(2927)
> Concurrent Select Relocation Set 15.721ms
> [2022-01-27T18:51:41.350+0000][34725.711s][info ][gc,phases   ] GC(2927)
> Pause Relocate Start 0.016ms
> [2022-01-27T18:51:41.350+0000][34725.712s][info ][safepoint   ] Safepoint
> "ZRelocateStart", Time since last: 23235346 ns, Reaching safepoint: 315064
> ns, At safepoint: 290299 ns, Total: 605363 ns
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,phases   ] GC(2927)
> Concurrent Relocate 19.017ms
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,marking  ] GC(2927)
> Mark: 16 stripe(s), 4 proactive flush(es), 1 terminate flush(es), 0
> completion(s), 0 continuation(s)
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,marking  ] GC(2927)
> Mark Stack Usage: 32M
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,metaspace] GC(2927)
> Metaspace: 169M used, 178M committed, 1184M reserved
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,ref      ] GC(2927)
> Soft: 40709 encountered, 0 discovered, 0 enqueued
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,ref      ] GC(2927)
> Weak: 33476 encountered, 7667 discovered, 4604 enqueued
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,ref      ] GC(2927)
> Final: 415 encountered, 0 discovered, 0 enqueued
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,ref      ] GC(2927)
> Phantom: 5155 encountered, 3828 discovered, 205 enqueued
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,reloc    ] GC(2927)
> Small Pages: 23233 / 46466M, Empty: 17842M, Relocated: 61M, In-Place: 0
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,reloc    ] GC(2927)
> Medium Pages: 161 / 5152M, Empty: 5056M, Relocated: 9M, In-Place: 0
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,reloc    ] GC(2927)
> Large Pages: 0 / 0M, Empty: 0M, Relocated: 0M, In-Place: 0
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,reloc    ] GC(2927)
> Forwarding Usage: 13M
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
> Min Capacity: 64000M(100%)
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
> Max Capacity: 64000M(100%)
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
> Soft Max Capacity: 64000M(100%)
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
>               Mark Start          Mark End        Relocate Start
>   Relocate End           High               Low
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
>   Capacity:    64000M (100%)      64000M (100%)      64000M (100%)
>   64000M (100%)      64000M (100%)      64000M (100%)
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
>     Free:    12382M (19%)       11392M (18%)       34172M (53%)       61184M
> (96%)       61184M (96%)       11326M (18%)
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
>     Used:    51618M (81%)       52608M (82%)       29828M (47%)        2816M
> (4%)        52674M (82%)        2816M (4%)
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
>     Live:         -              1303M (2%)         1303M (2%)         1303M
> (2%)             -                  -
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
> Allocated:         -               990M (2%)         1108M (2%)
> 1195M (2%)             -                  -
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
> Garbage:         -             50314M (79%)       27416M (43%)         316M
> (0%)             -                  -
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc,heap     ] GC(2927)
> Reclaimed:         -                  -             22898M (36%)
> 49997M (78%)            -                  -
> [2022-01-27T18:51:41.369+0000][34725.731s][info ][gc          ] GC(2927)
> Garbage Collection (Proactive) 51618M(81%)->2816M(4%)
> [2022-01-27T18:51:51.319+0000][34735.681s][info ][gc,start    ] GC(2928)
> Garbage Collection (Allocation Rate)
> [2022-01-27T18:51:51.321+0000][34735.682s][info ][gc,phases   ] GC(2928)
> Pause Mark Start 0.042ms
> [2022-01-27T18:51:51.321+0000][34735.683s][info ][safepoint   ] Safepoint
> "ZMarkStart", Time since last: 9969777477 ns, Reaching safepoint: 887200
> ns, At safepoint: 523517 ns, Total: 1410717 ns
> [2022-01-27T18:51:51.334+0000][34735.696s][debug][gc,phases   ] GC(2928)
> Concurrent Roots OopStorageSet (ZWorker#17) 0.542ms
> [2022-01-27T18:51:51.334+0000][34735.696s][debug][gc,phases   ] GC(2928)
> Concurrent Roots OopStorageSet (ZWorker#10) 0.575ms
> [2022-01-27T18:51:51.343+0000][34735.705s][debug][gc,phases   ] GC(2928)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#17) 9.161ms
> [2022-01-27T18:51:51.344+0000][34735.706s][debug][gc,phases   ] GC(2928)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#10) 9.817ms
> [2022-01-27T18:51:51.351+0000][34735.713s][debug][gc,phases   ] GC(2928)
> Concurrent Roots JavaThreads (ZWorker#10) 7.569ms
> [2022-01-27T18:51:51.351+0000][34735.713s][debug][gc,phases   ] GC(2928)
> Concurrent Roots JavaThreads (ZWorker#17) 8.243ms
> [2022-01-27T18:51:51.407+0000][34735.769s][debug][gc,phases   ] GC(2928)
> Concurrent Roots CodeCache (ZWorker#17) 55.952ms
> [2022-01-27T18:51:51.407+0000][34735.769s][debug][gc,phases   ] GC(2928)
> Concurrent Roots CodeCache (ZWorker#10) 55.988ms
> [2022-01-27T18:51:52.336+0000][34736.698s][debug][gc,phases   ] GC(2928)
> Concurrent Mark Try Terminate (ZWorker#14) 1.064ms
> [2022-01-27T18:51:52.337+0000][34736.699s][debug][gc,phases   ] GC(2928)
> Concurrent Mark Try Flush (ZWorker#12) 2.343ms
> [2022-01-27T18:51:52.342+0000][34736.703s][debug][gc,phases   ] GC(2928)
> Concurrent Mark Try Flush (ZWorker#12) 0.685ms
> [2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
> Concurrent Mark Try Flush (ZWorker#12) 0.294ms
> [2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
> Concurrent Mark Try Terminate (ZWorker#12) 0.320ms
> [2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
> Concurrent Mark Try Terminate (ZWorker#12) 0.000ms
> [2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
> Concurrent Mark (ZWorker#12) 934.304ms
> [2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
> Concurrent Mark Try Terminate (ZWorker#14) 1.062ms
> [2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
> Concurrent Mark (ZWorker#14) 934.339ms
> 
> *[2022-01-27T18:51:52.342+0000][34736.704s][info ][gc,phases   ] GC(2928)
> Concurrent Mark 1021.048ms*[2022-01-27T18:51:52.342+0000][34736.704s][info
> ][gc,phases   ] GC(2928) Pause Mark End 0.095ms
> [2022-01-27T18:51:52.343+0000][34736.704s][info ][safepoint   ] Safepoint
> "ZMarkEnd", Time since last: 1021138351 ns, Reaching safepoint: 256951 ns,
> At safepoint: 333825 ns, Total: 590776 ns
> [2022-01-27T18:51:52.343+0000][34736.704s][info ][gc,phases   ] GC(2928)
> Concurrent Mark Free 0.001ms
> [2022-01-27T18:51:52.345+0000][34736.707s][debug][gc,phases   ] GC(2928)
> Concurrent References Process (ZDriver) 2.019ms
> [2022-01-27T18:51:52.364+0000][34736.726s][debug][gc,phases   ] GC(2928)
> Concurrent Weak Roots OopStorageSet (ZWorker#13) 19.096ms
> [2022-01-27T18:51:52.364+0000][34736.726s][debug][gc,phases   ] GC(2928)
> Concurrent Weak Roots OopStorageSet (ZWorker#7) 19.068ms
> [2022-01-27T18:51:52.365+0000][34736.727s][debug][gc,phases   ] GC(2928)
> Concurrent References Enqueue (ZDriver) 0.010ms
> [2022-01-27T18:51:52.365+0000][34736.727s][info ][gc,phases   ] GC(2928)
> Concurrent Process Non-Strong References 22.133ms
> [2022-01-27T18:51:52.368+0000][34736.730s][info ][gc,phases   ] GC(2928)
> Concurrent Reset Relocation Set 2.875ms
> [2022-01-27T18:51:52.384+0000][34736.745s][info ][gc,phases   ] GC(2928)
> Concurrent Select Relocation Set 15.834ms
> [2022-01-27T18:51:52.384+0000][34736.746s][info ][gc,phases   ] GC(2928)
> Pause Relocate Start 0.015ms
> [2022-01-27T18:51:52.384+0000][34736.746s][info ][safepoint   ] Safepoint
> "ZRelocateStart", Time since last: 41003214 ns, Reaching safepoint: 303280
> ns, At safepoint: 155476 ns, Total: 458756 ns
> [2022-01-27T18:51:52.445+0000][34736.806s][info ][gc,phases   ] GC(2928)
> Concurrent Relocate 60.419ms
> [2022-01-27T18:51:52.445+0000][34736.806s][info ][gc,marking  ] GC(2928)
> Mark: 2 stripe(s), 2 proactive flush(es), 1 terminate flush(es), 0
> completion(s), 0 continuation(s)
> [2022-01-27T18:51:52.445+0000][34736.806s][info ][gc,marking  ] GC(2928)
> Mark Stack Usage: 32M
> [2022-01-27T18:51:52.445+0000][34736.806s][info ][gc,metaspace] GC(2928)
> Metaspace: 169M used, 178M committed, 1184M reserved
> [2022-01-27T18:51:52.445+0000][34736.806s][info ][gc,ref      ] GC(2928)
> Soft: 40719 encountered, 0 discovered, 0 enqueued
> [2022-01-27T18:51:52.445+0000][34736.806s][info ][gc,ref      ] GC(2928)
> Weak: 33189 encountered, 9016 discovered, 4318 enqueued
> [2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,ref      ] GC(2928)
> Final: 795 encountered, 387 discovered, 380 enqueued
> [2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,ref      ] GC(2928)
> Phantom: 5175 encountered, 3937 discovered, 219 enqueued
> [2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,reloc    ] GC(2928)
> Small Pages: 20919 / 41838M, Empty: 15862M, Relocated: 87M, In-Place: 0
> [2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,reloc    ] GC(2928)
> Medium Pages: 130 / 4160M, Empty: 4064M, Relocated: 11M, In-Place: 0
> [2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,reloc    ] GC(2928)
> Large Pages: 20 / 128M, Empty: 128M, Relocated: 0M, In-Place: 0
> [2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,reloc    ] GC(2928)
> Forwarding Usage: 19M
> [2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
> Min Capacity: 64000M(100%)
> [2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
> Max Capacity: 64000M(100%)
> [2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
> Soft Max Capacity: 64000M(100%)
> [2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
>               Mark Start          Mark End        Relocate Start
>   Relocate End           High               Low
> [2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
>   Capacity:    64000M (100%)      64000M (100%)      64000M (100%)
>   64000M (100%)      64000M (100%)      64000M (100%)
> [2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
>     Free:    17874M (28%)       13228M (21%)       33154M (52%)       57394M
> (90%)       57434M (90%)       13142M (21%)
> [2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
>     Used:    46126M (72%)       50772M (79%)       30846M (48%)        6606M
> (10%)       50858M (79%)        6566M (10%)
> [2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
>     Live:         -              1301M (2%)         1301M (2%)         1301M
> (2%)             -                  -
> [2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
> Allocated:         -              4646M (7%)         4774M (7%)
> 5004M (8%)             -                  -
> [2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
> Garbage:         -             44824M (70%)       24770M (39%)         299M
> (0%)             -                  -
> [2022-01-27T18:51:52.445+0000][34736.807s][info ][gc,heap     ] GC(2928)
> Reclaimed:         -                  -             20054M (31%)
> 44524M (70%)            -                  -
> [2022-01-27T18:51:52.445+0000][34736.807s][info ][gc          ] GC(2928)
> Garbage Collection (Allocation Rate) 46126M(72%)->6606M(10%)
> [2022-01-27T18:52:00.820+0000][34745.182s][info ][gc,start    ] GC(2929)
> Garbage Collection (Allocation Rate)
> [2022-01-27T18:52:00.821+0000][34745.183s][info ][gc,phases   ] GC(2929)
> Pause Mark Start 0.057ms
> [2022-01-27T18:52:00.822+0000][34745.184s][info ][safepoint   ] Safepoint
> "ZMarkStart", Time since last: 8436197260 ns, Reaching safepoint: 1019671
> ns, At safepoint: 515965 ns, Total: 1535636 ns
> [2022-01-27T18:52:00.835+0000][34745.197s][debug][gc,phases   ] GC(2929)
> Concurrent Roots OopStorageSet (ZWorker#5) 0.991ms
> [2022-01-27T18:52:00.851+0000][34745.213s][debug][gc,phases   ] GC(2929)
> Concurrent Roots ClassLoaderDataGraph (ZWorker#5) 15.777ms
> [2022-01-27T18:52:00.864+0000][34745.226s][debug][gc,phases   ] GC(2929)
> Concurrent Roots JavaThreads (ZWorker#5) 12.769ms
> [2022-01-27T18:52:00.945+0000][34745.307s][debug][gc,phases   ] GC(2929)
> Concurrent Roots CodeCache (ZWorker#5) 81.380ms
> [2022-01-27T18:52:02.766+0000][34747.127s][debug][gc,phases   ] GC(2929)
> Concurrent Mark Try Flush (ZWorker#16) 1.613ms
> [2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
> Concurrent Mark Try Flush (ZWorker#16) 0.717ms
> [2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
> Concurrent Mark Try Flush (ZWorker#16) 0.292ms
> [2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
> Concurrent Mark Try Terminate (ZWorker#16) 0.306ms
> [2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
> Concurrent Mark Try Terminate (ZWorker#16) 0.000ms
> [2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
> Concurrent Mark (ZWorker#16) 1825.719ms
> 
> *[2022-01-27T18:52:02.771+0000][34747.133s][info ][gc,phases   ] GC(2929)
> Concurrent Mark 1949.325ms*[2022-01-27T18:52:02.774+0000][34747.136s][info
> ][gc,phases   ] GC(2929) Pause Mark End 0.212ms
> [2022-01-27T18:52:02.775+0000][34747.136s][info ][safepoint   ] Safepoint
> "ZMarkEnd", Time since last: 1949455739 ns, Reaching safepoint: 2536816 ns,
> At safepoint: 606241 ns, Total: 3143057 ns
> [2022-01-27T18:52:02.775+0000][34747.136s][info ][gc,phases   ] GC(2929)
> Concurrent Mark Free 0.001ms
> [2022-01-27T18:52:02.778+0000][34747.139s][debug][gc,phases   ] GC(2929)
> Concurrent References Process (ZDriver) 3.161ms
> [2022-01-27T18:52:02.813+0000][34747.175s][debug][gc,phases   ] GC(2929)
> Concurrent Weak Roots OopStorageSet (ZWorker#4) 35.717ms
> [2022-01-27T18:52:02.814+0000][34747.176s][debug][gc,phases   ] GC(2929)
> Concurrent References Enqueue (ZDriver) 0.007ms
> [2022-01-27T18:52:02.814+0000][34747.176s][info ][gc,phases   ] GC(2929)
> Concurrent Process Non-Strong References 39.943ms
> [2022-01-27T18:52:02.818+0000][34747.180s][info ][gc,phases   ] GC(2929)
> Concurrent Reset Relocation Set 3.587ms
> [2022-01-27T18:52:02.833+0000][34747.195s][info ][gc,phases   ] GC(2929)
> Concurrent Select Relocation Set 15.208ms
> [2022-01-27T18:52:02.834+0000][34747.195s][info ][gc,phases   ] GC(2929)
> Pause Relocate Start 0.017ms
> [2022-01-27T18:52:02.834+0000][34747.196s][info ][safepoint   ] Safepoint
> "ZRelocateStart", Time since last: 58881245 ns, Reaching safepoint: 286775
> ns, At safepoint: 232929 ns, Total: 519704 ns
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,phases   ] GC(2929)
> Concurrent Relocate 107.558ms
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,marking  ] GC(2929)
> Mark: 1 stripe(s), 2 proactive flush(es), 1 terminate flush(es), 0
> completion(s), 0 continuation(s)
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,marking  ] GC(2929)
> Mark Stack Usage: 32M
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,metaspace] GC(2929)
> Metaspace: 169M used, 178M committed, 1184M reserved
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,ref      ] GC(2929)
> Soft: 40717 encountered, 0 discovered, 0 enqueued
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,ref      ] GC(2929)
> Weak: 32517 encountered, 8597 discovered, 3652 enqueued
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,ref      ] GC(2929)
> Final: 415 encountered, 6 discovered, 0 enqueued
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,ref      ] GC(2929)
> Phantom: 5031 encountered, 3633 discovered, 77 enqueued
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,reloc    ] GC(2929)
> Small Pages: 18075 / 36150M, Empty: 12846M, Relocated: 76M, In-Place: 0
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,reloc    ] GC(2929)
> Medium Pages: 117 / 3744M, Empty: 3584M, Relocated: 10M, In-Place: 0
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,reloc    ] GC(2929)
> Large Pages: 16 / 104M, Empty: 104M, Relocated: 0M, In-Place: 0
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,reloc    ] GC(2929)
> Forwarding Usage: 15M
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,heap     ] GC(2929)
> Min Capacity: 64000M(100%)
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,heap     ] GC(2929)
> Max Capacity: 64000M(100%)
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,heap     ] GC(2929)
> Soft Max Capacity: 64000M(100%)
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,heap     ] GC(2929)
>               Mark Start          Mark End        Relocate Start
>   Relocate End           High               Low
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,heap     ] GC(2929)
>   Capacity:    64000M (100%)      64000M (100%)      64000M (100%)
>   64000M (100%)      64000M (100%)      64000M (100%)
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,heap     ] GC(2929)
>     Free:    24002M (38%)       15544M (24%)       31836M (50%)       53224M
> (83%)       53414M (83%)       15410M (24%)
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,heap     ] GC(2929)
>     Used:    39998M (62%)       48456M (76%)       32164M (50%)       10776M
> (17%)       48590M (76%)       10586M (17%)
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,heap     ] GC(2929)
>     Live:         -              1283M (2%)         1283M (2%)         1283M
> (2%)             -                  -
> [2022-01-27T18:52:02.942+0000][34747.303s][info ][gc,heap     ] GC(2929)
> Allocated:         -              8458M (13%)        8700M (14%)
>   9205M (14%)            -                  -
> [2022-01-27T18:52:02.942+0000][34747.304s][info ][gc,heap     ] GC(2929)
> Garbage:         -             38714M (60%)       22180M (35%)         286M
> (0%)             -                  -
> [2022-01-27T18:52:02.942+0000][34747.304s][info ][gc,heap     ] GC(2929)
> Reclaimed:         -                  -             16534M (26%)
> 38427M (60%)            -                  -
> [2022-01-27T18:52:02.942+0000][34747.304s][info ][gc          ] GC(2929)
> Garbage Collection (Allocation Rate) 39998M(62%)->10776M(17%)
> 
> 
> Thanks
> Sundar
> 
> 
> On Thu, Jan 27, 2022 at 11:00 AM Sundara Mohan M <m.sundar85 at gmail.com>
> wrote:
> 
>> Hi,
>>     We are using ZGC in production environment with following settings
>> *[2022-01-27T09:12:55.684+0000][0.046s][info][gc,init] Version: 17.0.1+12
>> (release)*
>> ..
>> *[2022-01-27T09:12:55.684+0000][0.046s][info][gc,init] GC Workers: 18
>> (dynamic)*
>> ...
>> [2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Min Capacity: 64000M
>> [2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Initial Capacity:
>> 64000M
>> *[2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Max Capacity:
>> 64000M*
>> [2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Medium Page Size: 32M
>> [2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Pre-touch: Disabled
>> [2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Available space on
>> backing filesystem: N/A
>> [2022-01-27T09:12:55.686+0000][0.048s][info][gc,init] Uncommit: Implicitly
>> Disabled (-Xms equals -Xmx)
>> *[2022-01-27T09:13:05.275+0000][9.636s][info][gc,init] Runtime Workers: 44*
>> [2022-01-27T09:13:05.278+0000][9.640s][info][gc     ] Using The Z Garbage
>> Collector
>>
>> We run into GC cycle every 8-10 seconds. In some GC cycle Concurrent Mark
>> phase is taking 2*s* and in some cycle it is taking 200*ms.*
>>
>> After enabling debug log for gc,phase saw in some cases more ZWorkers are
>> doing work in other case it is given to only 1 or 2 ZWorker.
>>
>> Example log where work is done by multiple thread and took only 200ms
>> [2022-01-27T18:51:41.324+0000][34725.686s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark (ZWorker#12) 150.092ms
>> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark Try Terminate (ZWorker#0) 1.059ms
>> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark (ZWorker#0) 150.279ms
>> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark Try Terminate (ZWorker#14) 1.055ms
>> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark (ZWorker#14) 149.355ms
>> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark Try Terminate (ZWorker#8) 1.060ms
>> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark (ZWorker#8) 150.312ms
>> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark Try Terminate (ZWorker#6) 1.060ms
>> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark (ZWorker#6) 150.323ms
>> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark Try Terminate (ZWorker#17) 1.062ms
>> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark Try Terminate (ZWorker#11) 1.058ms
>> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark Try Terminate (ZWorker#13) 1.060ms
>> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark Try Terminate (ZWorker#2) 1.059ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark (ZWorker#2) 150.423ms
>> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark (ZWorker#13) 150.437ms
>> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark (ZWorker#11) 150.374ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark Try Terminate (ZWorker#3) 1.061ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark (ZWorker#3) 150.537ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark Try Terminate (ZWorker#7) 1.061ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark (ZWorker#7) 150.550ms
>> [2022-01-27T18:51:41.325+0000][34725.686s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark (ZWorker#17) 150.364ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark Try Terminate (ZWorker#15) 1.056ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark Try Terminate (ZWorker#16) 1.059ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark Try Terminate (ZWorker#1) 1.060ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark (ZWorker#1) 150.613ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark Try Terminate (ZWorker#4) 1.064ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark (ZWorker#16) 150.584ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark Try Terminate (ZWorker#10) 1.059ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark (ZWorker#10) 150.706ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark (ZWorker#4) 150.629ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark (ZWorker#15) 150.601ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark Try Terminate (ZWorker#9) 1.028ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark (ZWorker#9) 150.938ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark Try Terminate (ZWorker#5) 1.057ms
>> [2022-01-27T18:51:41.325+0000][34725.687s][debug][gc,phases   ] GC(2927)
>> Concurrent Mark (ZWorker#5) 151.063ms
>> *[2022-01-27T18:51:41.325+0000][34725.687s][info ][gc,phases   ] GC(2927)
>> Concurrent Mark 204.119ms*
>>
>> Another case where it was run by only 1/2 worker and took 1/2s respectively
>> ...
>> *[2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
>> Concurrent Mark (ZWorker#12) 934.304ms*
>> [2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
>> Concurrent Mark Try Terminate (ZWorker#14) 1.062ms
>> *[2022-01-27T18:51:52.342+0000][34736.704s][debug][gc,phases   ] GC(2928)
>> Concurrent Mark (ZWorker#14) 934.339ms*
>> *[2022-01-27T18:51:52.342+0000][34736.704s][info ][gc,phases   ] GC(2928)
>> Concurrent Mark 1021.048ms*
>>
>>
>> [2022-01-27T18:52:00.822+0000][34745.184s][info ][safepoint   ] Safepoint
>> "ZMarkStart", Time since last: 8436197260 ns, Reaching safepoint: 1019671
>> ns, At safepoint: 515965 ns, Total: 1
>> 535636 ns
>> [2022-01-27T18:52:00.835+0000][34745.197s][debug][gc,phases   ] GC(2929)
>> Concurrent Roots OopStorageSet (ZWorker#5) 0.991ms
>> [2022-01-27T18:52:00.851+0000][34745.213s][debug][gc,phases   ] GC(2929)
>> Concurrent Roots ClassLoaderDataGraph (ZWorker#5) 15.777ms
>> [2022-01-27T18:52:00.864+0000][34745.226s][debug][gc,phases   ] GC(2929)
>> Concurrent Roots JavaThreads (ZWorker#5) 12.769ms
>> [2022-01-27T18:52:00.945+0000][34745.307s][debug][gc,phases   ] GC(2929)
>> Concurrent Roots CodeCache (ZWorker#5) 81.380ms
>> [2022-01-27T18:52:02.766+0000][34747.127s][debug][gc,phases   ] GC(2929)
>> Concurrent Mark Try Flush (ZWorker#16) 1.613ms
>> [2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
>> Concurrent Mark Try Flush (ZWorker#16) 0.717ms
>> [2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
>> Concurrent Mark Try Flush (ZWorker#16) 0.292ms
>> [2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
>> Concurrent Mark Try Terminate (ZWorker#16) 0.306ms
>> [2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
>> Concurrent Mark Try Terminate (ZWorker#16) 0.000ms
>>
>>
>> *[2022-01-27T18:52:02.771+0000][34747.133s][debug][gc,phases   ] GC(2929)
>> Concurrent Mark (ZWorker#16)
>> 1825.719ms[2022-01-27T18:52:02.771+0000][34747.133s][info ][gc,phases   ]
>> GC(2929) Concurrent Mark 1949.325ms*
>>
>> Even though we have 18 threads, does anyone have any idea why the work is
>> not distributed among multiple workers?
>>
>> Thanks
>> Sundar
>>
>>