From ycraig at cysystems.com  Mon Jan  5 12:17:30 2009
From: ycraig at cysystems.com (craig yeldell)
Date: Mon, 5 Jan 2009 15:17:30 -0500
Subject: 1.6 jdk CMSPermGenSweepingEnbaled 
Message-ID: <245065D7-8408-47AD-AA9E-9A9C98A810EE@cysystems.com>

I have run into some PermGen OOM issues, and thought I remember  
reading that the following args are enabled in the 1.6 jdk.  Does  
anyone know where I can find the info on the default settings?

CMSPermGenSweepingEnabled
CMSClassUnloadingEnabled


1.6.0_10 64bit

Regards,
Craig


From Y.S.Ramakrishna at Sun.COM  Mon Jan  5 12:29:35 2009
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Mon, 05 Jan 2009 12:29:35 -0800
Subject: 1.6 jdk CMSPermGenSweepingEnbaled
In-Reply-To: <245065D7-8408-47AD-AA9E-9A9C98A810EE@cysystems.com>
References: <245065D7-8408-47AD-AA9E-9A9C98A810EE@cysystems.com>
Message-ID: <fa11b47a2e88.4961fd2f@sun.com>

Hi Craig -- No, they are not enabled by default (even in 6.0),
so you'll need to explicitly enable +CMSClassUnloadingEnabled
to get the perm gen to be collected by CMS (as of
6.0 CMSPermGenSweepingEnabled became a no-op with a
suitable warning issued).

If you know the name of a specific JVM option,
i think there's some way you can have jconsole give you
the value from a running JVM. Others on this list might
be able to provide details on how to get jconsole to
reveal these secrets. (One would hope jinfo might have
a flag/query/option for such as well.)

I'm cross-posting into the serviceability alias
where the latter part of this discussion likely belongs
and where you'll get more concrete information than
my musings above.

-- ramki

----- Original Message -----
From: craig yeldell <ycraig at cysystems.com>
Date: Monday, January 5, 2009 12:18 pm
Subject: 1.6 jdk CMSPermGenSweepingEnbaled
To: hotspot-gc-use at openjdk.java.net


> I have run into some PermGen OOM issues, and thought I remember  
> reading that the following args are enabled in the 1.6 jdk.  Does  
> anyone know where I can find the info on the default settings?
> 
> CMSPermGenSweepingEnabled
> CMSClassUnloadingEnabled
> 
> 
> 1.6.0_10 64bit
> 
> Regards,
> Craig
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From Alan.Bateman at Sun.COM  Mon Jan  5 13:07:34 2009
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Mon, 05 Jan 2009 21:07:34 +0000
Subject: 1.6 jdk CMSPermGenSweepingEnbaled
In-Reply-To: <fa11b47a2e88.4961fd2f@sun.com>
References: <245065D7-8408-47AD-AA9E-9A9C98A810EE@cysystems.com>
	<fa11b47a2e88.4961fd2f@sun.com>
Message-ID: <49627696.7080406@sun.com>

Y Srinivas Ramakrishna wrote:
> :
>
> If you know the name of a specific JVM option,
> i think there's some way you can have jconsole give you
> the value from a running JVM. Others on this list might
> be able to provide details on how to get jconsole to
> reveal these secrets. (One would hope jinfo might have
> a flag/query/option for such as well.)
>
> I'm cross-posting into the serviceability alias
> where the latter part of this discussion likely belongs
> and where you'll get more concrete information than
> my musings above.
>   
You are right. In the MBeans tab you'll see 
com.sun.management.HotSpotDiagnostic. It defines the getVMOption 
operation. Type CMSClassUnloadingEnabled into the text field and press 
the getVMOption button and it will popup a dialog window with its 
current and default value.

-Alan.


From higuava at gmail.com  Thu Jan 15 09:27:09 2009
From: higuava at gmail.com (Hi Guava)
Date: Thu, 15 Jan 2009 12:27:09 -0500
Subject: Extremely long parallel rescan
Message-ID: <bae1306e0901150927m6465f5ffs878e28c5126a9dd2@mail.gmail.com>

We have been having problem with CMS collector for a long time. The rescan
phase of CMS takes very very long time (20 minutes to several hours). We
tried searching on Internet, asking in Sun's developer forum, etc. No one
seems know what's going on.

The typical hardware has 4 to 8 cpu/cores with 20 GB to 64GB of memory. It
happens on both Windows and Linux version of 64-bit JVMs.
The GC parameters that we usually use are: -XX:+UseConcMarkSweepGC
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps

The following is an example of the problem.
Hardware: 2 dual-core CPUs (4 core in total) with 28 GB memory.
JVM: Java HotSpot(TM) 64-Bit Server VM Version 1.5.0_17-b04 from Sun
Microsystems Inc. on Windows XP Version 5.2 on amd64
Settings: -Xms20000m -Xmx20000m -XX:NewSize=500m
-XX:+CMSPermGenSweepingEnabled -XX:+DisableExplicitGC
-XX:CMSMarkStackSize=2M -XX:CMSMarkStackSizeMax=8M -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=30 -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
Note that we set many other GC parameters in this particular example.
Usually we only use -XX:+UseConcMarkSweepGC.

910.659: [GC [1 CMS-initial-mark: 6465020K(19968000K)] 6679872K(20479552K),
0.3206026 secs]
910.980: [CMS-concurrent-mark-start]
919.131: [GC 919.131: [ParNew: 511104K->0K(511552K), 0.6801310 secs]
6976124K->6657978K(20479552K), 0.6802201 secs]
933.413: [CMS-concurrent-mark: 21.739/22.433 secs]
933.413: [CMS-concurrent-preclean-start]
934.813: [CMS-concurrent-preclean: 1.348/1.400 secs]
934.813: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 935.929:
[CMS-concurrent-abortable-preclean: 0.662/1.116 secs]
935.933: [GC[YG occupancy: 443765 K (511552 K)]935.933: [Rescan (parallel)
, 3533.9840087 secs]4469.917: [weak refs processing, 0.0620489 secs] [1
CMS-remark: 6657978K(19968000K)] 7101744K(20479552K), 3534.0462074 secs]
4469.980: [CMS-concurrent-sweep-start]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090115/104d53a0/attachment.html 

From Y.S.Ramakrishna at Sun.COM  Thu Jan 15 09:44:26 2009
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Thu, 15 Jan 2009 09:44:26 -0800
Subject: Extremely long parallel rescan
In-Reply-To: <bae1306e0901150927m6465f5ffs878e28c5126a9dd2@mail.gmail.com>
References: <bae1306e0901150927m6465f5ffs878e28c5126a9dd2@mail.gmail.com>
Message-ID: <fccca5176304.496f057a@sun.com>


Can you run with -XX:PrintCMSStatistics=2 and send me the output?

I suspect a performance badness related to code for
manipulating overflow lists that we recently discovered
and which is being fixed under:

6786503 Overflow list performance can be improved
6787254 Work queue capacity can be increased substantially on some platforms

We can get you a fix verification binary
through your support channels once it is established
that work queue overflow is the issue here (and it
sounds very much like it is).

-- ramki

----- Original Message -----
From: Hi Guava <higuava at gmail.com>
Date: Thursday, January 15, 2009 9:27 am
Subject: Extremely long parallel rescan
To: hotspot-gc-use at openjdk.java.net


> We have been having problem with CMS collector for a long time. The rescan
> phase of CMS takes very very long time (20 minutes to several hours). 
> We
> tried searching on Internet, asking in Sun's developer forum, etc. No 
> one
> seems know what's going on.
> 
> The typical hardware has 4 to 8 cpu/cores with 20 GB to 64GB of 
> memory. It
> happens on both Windows and Linux version of 64-bit JVMs.
> The GC parameters that we usually use are: -XX:+UseConcMarkSweepGC
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
> 
> The following is an example of the problem.
> Hardware: 2 dual-core CPUs (4 core in total) with 28 GB memory.
> JVM: Java HotSpot(TM) 64-Bit Server VM Version 1.5.0_17-b04 from Sun
> Microsystems Inc. on Windows XP Version 5.2 on amd64
> Settings: -Xms20000m -Xmx20000m -XX:NewSize=500m
> -XX:+CMSPermGenSweepingEnabled -XX:+DisableExplicitGC
> -XX:CMSMarkStackSize=2M -XX:CMSMarkStackSizeMax=8M -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=30 -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps
> Note that we set many other GC parameters in this particular example.
> Usually we only use -XX:+UseConcMarkSweepGC.
> 
> 910.659: [GC [1 CMS-initial-mark: 6465020K(19968000K)] 6679872K(20479552K),
> 0.3206026 secs]
> 910.980: [CMS-concurrent-mark-start]
> 919.131: [GC 919.131: [ParNew: 511104K->0K(511552K), 0.6801310 secs]
> 6976124K->6657978K(20479552K), 0.6802201 secs]
> 933.413: [CMS-concurrent-mark: 21.739/22.433 secs]
> 933.413: [CMS-concurrent-preclean-start]
> 934.813: [CMS-concurrent-preclean: 1.348/1.400 secs]
> 934.813: [CMS-concurrent-abortable-preclean-start]
>  CMS: abort preclean due to time 935.929:
> [CMS-concurrent-abortable-preclean: 0.662/1.116 secs]
> 935.933: [GC[YG occupancy: 443765 K (511552 K)]935.933: [Rescan (parallel)
> , 3533.9840087 secs]4469.917: [weak refs processing, 0.0620489 secs] [1
> CMS-remark: 6657978K(19968000K)] 7101744K(20479552K), 3534.0462074 secs]
> 4469.980: [CMS-concurrent-sweep-start]
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From higuava at gmail.com  Fri Jan 16 08:49:41 2009
From: higuava at gmail.com (Hi Guava)
Date: Fri, 16 Jan 2009 11:49:41 -0500
Subject: Extremely long parallel rescan
In-Reply-To: <fccca5176304.496f057a@sun.com>
References: <bae1306e0901150927m6465f5ffs878e28c5126a9dd2@mail.gmail.com>
	<fccca5176304.496f057a@sun.com>
Message-ID: <bae1306e0901160849k1f706f76p16c6f2ff1b0d921@mail.gmail.com>

Ramki,

Thanks for your quick response.
I ran again with the parameter you suggested. The rescan is very long this
time. It has been in that state for 20 hours now and is still going. At the
beginning, 7 out of 8 cpus are running at about 100%. The machine has 2
quad-core cpus. After several hours, 4 out of 8 cpus are busy. It seems that
the rescan stuck at the "stealing" stage. I haven't killed it yet since I
don't know if it will print useful information after rescan is done. Please
scroll to the end of the log to see the rescan that never finishes.
Thanks a lot for the help!

-Xms20000m -Xmx20000m
-XX:NewSize=500m -XX:PrintCMSStatistics=2 -XX:+CMSPermGenSweepingEnabled
-XX:+DisableExplicitGC -XX:CMSMarkStackSize=2M -XX:CMSMarkStackSizeMax=8M
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=30
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps

Java HotSpot(TM) 64-Bit Server VM Version 1.5.0_17-b04 from Sun Microsystems
Inc. on Windows XP Version 5.2 on amd64

131.068: [GC 131.068: [ParNew: 482175K->0K(511552K), 0.1216606 secs]
482175K->140325K(20479552K), 0.1217908 secs]
133.271: [GC 133.271: [ParNew: 501962K->0K(511552K), 0.6634415 secs]
642288K->421458K(20479552K), 0.6635278 secs]
135.663: [GC 135.663: [ParNew: 457307K->0K(511552K), 0.6851882 secs]
878766K->710570K(20479552K), 0.6852731 secs]
139.225: [GC 139.225: [ParNew: 511104K->0K(511552K), 1.2118508 secs]
1221674K->964210K(20479552K), 1.2119400 secs]
141.498: [GC 141.498: [ParNew: 511104K->0K(511552K), 1.0503047 secs]
1475314K->1394330K(20479552K), 1.0504021 secs]
147.200: [GC 147.200: [ParNew: 511104K->0K(511552K), 1.1423852 secs]
1905434K->1668390K(20479552K), 1.1424733 secs]
151.790: [GC 151.790: [ParNew: 511104K->0K(511552K), 1.0925588 secs]
2179494K->1932668K(20479552K), 1.0926648 secs]
156.593: [GC 156.593: [ParNew: 511104K->0K(511552K), 1.0724513 secs]
2443772K->2197164K(20479552K), 1.0725535 secs]
161.616: [GC 161.616: [ParNew: 511104K->0K(511552K), 1.0866788 secs]
2708268K->2451306K(20479552K), 1.0867676 secs]
168.402: [GC 168.403: [ParNew: 511104K->0K(511552K), 0.7680119 secs]
2962410K->2650469K(20479552K), 0.7681083 secs]
224.690: [GC 224.690: [ParNew: 511104K->0K(511552K), 0.7515573 secs]
3161573K->2923798K(20479552K), 0.7516401 secs]
227.457: [GC 227.457: [ParNew: 511104K->0K(511552K), 0.8830924 secs]
3434902K->3174766K(20479552K), 0.8831806 secs]
233.210: [GC 233.210: [ParNew: 511104K->0K(511552K), 1.2646489 secs]
3685870K->3509255K(20479552K), 1.2647575 secs]
274.928: [GC 274.928: [ParNew: 511104K->0K(511552K), 0.8556637 secs]
4020359K->3691515K(20479552K), 0.8557548 secs]
279.750: [GC 279.750: [ParNew: 511104K->0K(511552K), 1.1509608 secs]
4202619K->3980773K(20479552K), 1.1510539 secs]
284.343: [GC 284.343: [ParNew: 511104K->0K(511552K), 0.3914217 secs]
4491877K->4080587K(20479552K), 0.3915111 secs]
288.167: [GC 288.167: [ParNew: 511104K->0K(511552K), 0.3882978 secs]
4591691K->4183268K(20479552K), 0.3883930 secs]
292.015: [GC 292.015: [ParNew: 511104K->0K(511552K), 0.3809847 secs]
4694372K->4290902K(20479552K), 0.3810756 secs]
295.859: [GC 295.859: [ParNew: 511104K->0K(511552K), 0.3902037 secs]
4802006K->4389069K(20479552K), 0.3902926 secs]
299.707: [GC 299.707: [ParNew: 511104K->0K(511552K), 0.3903764 secs]
4900173K->4487123K(20479552K), 0.3904713 secs]
303.708: [GC 303.708: [ParNew: 511104K->0K(511552K), 0.3407708 secs]
4998227K->4597766K(20479552K), 0.3408627 secs]
307.651: [GC 307.651: [ParNew: 511104K->0K(511552K), 0.2816556 secs]
5108870K->4673884K(20479552K), 0.2817383 secs]
311.571: [GC 311.571: [ParNew: 511104K->0K(511552K), 0.2831275 secs]
5184988K->4749829K(20479552K), 0.2832155 secs]
315.455: [GC 315.455: [ParNew: 511104K->0K(511552K), 0.2840821 secs]
5260933K->4826064K(20479552K), 0.2841665 secs]
319.338: [GC 319.338: [ParNew: 511104K->0K(511552K), 0.2929911 secs]
5337168K->4902045K(20479552K), 0.2930778 secs]
323.299: [GC 323.299: [ParNew: 511104K->0K(511552K), 0.2300986 secs]
5413149K->4966222K(20479552K), 0.2301841 secs]
327.220: [GC 327.220: [ParNew: 511104K->0K(511552K), 0.2266575 secs]
5477326K->5028784K(20479552K), 0.2267576 secs]
331.140: [GC 331.140: [ParNew: 511104K->0K(511552K), 0.2264157 secs]
5539888K->5091458K(20479552K), 0.2265016 secs]
335.062: [GC 335.062: [ParNew: 511104K->0K(511552K), 0.2225824 secs]
5602562K->5152768K(20479552K), 0.2226688 secs]
339.011: [GC 339.011: [ParNew: 511104K->0K(511552K), 0.2051935 secs]
5663872K->5210406K(20479552K), 0.2052908 secs]
342.942: [GC 342.942: [ParNew: 511104K->0K(511552K), 0.2066628 secs]
5721510K->5267895K(20479552K), 0.2067891 secs]
346.879: [GC 346.879: [ParNew: 511104K->0K(511552K), 0.2069097 secs]
5778999K->5325712K(20479552K), 0.2069984 secs]
350.821: [GC 350.821: [ParNew: 511104K->0K(511552K), 0.1989961 secs]
5836816K->5381923K(20479552K), 0.1990833 secs]
354.712: [GC 354.712: [ParNew: 511104K->0K(511552K), 0.1996306 secs]
5893027K->5437101K(20479552K), 0.1997210 secs]
358.669: [GC 358.669: [ParNew: 511104K->0K(511552K), 0.2017006 secs]
5948205K->5493076K(20479552K), 0.2017893 secs]
363.621: [GC 363.621: [ParNew: 511104K->0K(511552K), 0.2974636 secs]
6004180K->5574559K(20479552K), 0.2975465 secs]
368.774: [GC 368.774: [ParNew: 511104K->0K(511552K), 0.6974476 secs]
6085663K->5731873K(20479552K), 0.6975423 secs]
374.311: [GC 374.311: [ParNew: 511104K->0K(511552K), 0.7011833 secs]
6242977K->5891664K(20479552K), 0.7012883 secs]
380.995: [GC 380.995: [ParNew: 511104K->0K(511552K), 0.6870053 secs]
6402768K->6036024K(20479552K), 0.6870927 secs]
381.683: [GC [1 CMS-initial-mark: 6036024K(19968000K)] 6046258K(20479552K),
0.0010084 secs]
381.684: [CMS-concurrent-mark-start]
387.028: [GC 387.028: [ParNew: 511104K->0K(511552K), 0.6279217 secs]
6547128K->6189219K(20479552K), 0.6280151 secs]
393.651: [GC 393.651: [ParNew: 511104K->0K(511552K), 0.5878509 secs]
6700323K->6334466K(20479552K), 0.5879383 secs]
396.401: [CMS-concurrent-mark: 13.495/14.718 secs]
 (CMS-concurrent-mark yielded 5 times)
396.401: [CMS-concurrent-preclean-start]
 (cardTable: 523923 cards, re-scanned 1922942 cards, 2 iterations)
398.835: [CMS-concurrent-preclean: 2.278/2.433 secs]
 (CMS-concurrent-preclean yielded 6 times)
398.835: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 112875 cards, re-scanned 112875 cards, 1 iterations)
 (cardTable: 48863 cards, re-scanned 48863 cards, 1 iterations)
 (cardTable: 34189 cards, re-scanned 34189 cards, 1 iterations)
 (cardTable: 30811 cards, re-scanned 30811 cards, 1 iterations)
 (cardTable: 29949 cards, re-scanned 29949 cards, 1 iterations)
399.824: [GC 399.825: [ParNew: 511104K->0K(511552K), 0.6672418 secs]
6845570K->6489950K(20479552K), 0.6673266 secs]
 (cardTable: 16542 cards, re-scanned 16542 cards, 1 iterations)
 CMS: abort preclean due to time  [6 iterations, 0 waits, 273229 cards)]
400.541: [CMS-concurrent-abortable-preclean: 0.944/1.706 secs]
 (CMS-concurrent-abortable-preclean yielded 4 times)
400.545: [GC[YG occupancy: 10233 K (511552 K)]400.545: [Rescan (parallel)
Finished root rescan work in 7th thread: 0.000 sec
Finished root rescan work in 2th thread: 0.000 sec
Finished root rescan work in 5th thread: 0.000 sec
Finished root rescan work in 6th thread: 0.000 sec
Finished root rescan work in 3th thread: 0.000 sec
Finished root rescan work in 1th thread: 0.001 sec
Finished root rescan work in 4th thread: 0.001 sec
Finished root rescan work in 0th thread: 0.010 sec
Finished dirty card rescan work in 0th thread: 0.151 sec
Finished dirty card rescan work in 7th thread: 0.161 sec
Finished dirty card rescan work in 1th thread: 0.161 sec
Finished dirty card rescan work in 4th thread: 0.160 sec
Finished dirty card rescan work in 6th thread: 0.161 sec
Finished dirty card rescan work in 2th thread: 0.161 sec
Finished dirty card rescan work in 5th thread: 0.163 sec
Finished dirty card rescan work in 3th thread: 0.163 sec
Finished work stealing in 3th thread: 0.000 sec
Finished work stealing in 2th thread: 0.002 sec
Finished work stealing in 6th thread: 0.003 sec
Finished work stealing in 4th thread: 0.003 sec
Finished work stealing in 5th thread: 0.002 sec
Finished work stealing in 0th thread: 0.004 sec
Finished work stealing in 7th thread: 0.004 sec
Finished work stealing in 1th thread: 0.004 sec
, 0.1661181 secs]400.711: [weak refs processing, 0.0039589 secs] [1
CMS-remark: 6489950K(19968000K)] 6500184K(20479552K), 0.1701777 secs]
400.717: [CMS-concurrent-sweep-start]
405.492: [CMS-concurrent-sweep: 4.775/4.775 secs]
 (CMS-concurrent-sweep yielded 0 times)
405.492: [CMS-concurrent-reset-start]
405.545: [CMS-concurrent-reset: 0.053/0.053 secs]
 (CMS-concurrent-reset yielded 0 times)
406.207: [GC 406.207: [ParNew: 511104K->0K(511552K), 0.3996612 secs]
4935390K->4576286K(20479552K), 0.3997547 secs]
412.606: [GC 412.606: [ParNew: 511104K->0K(511552K), 0.4807867 secs]
5087390K->4730624K(20479552K), 0.4808747 secs]
418.685: [GC 418.685: [ParNew: 511104K->0K(511552K), 0.4314910 secs]
5241728K->4887033K(20479552K), 0.4315852 secs]
425.110: [GC 425.110: [ParNew: 511104K->0K(511552K), 0.4495350 secs]
5398137K->5033284K(20479552K), 0.4496249 secs]
431.196: [GC 431.196: [ParNew: 511104K->0K(511552K), 0.4870862 secs]
5544388K->5190382K(20479552K), 0.4871778 secs]
492.182: [GC 492.182: [ParNew: 504623K->0K(511552K), 0.3876369 secs]
5695006K->5349543K(20479552K), 0.3877360 secs]
494.770: [GC 494.771: [ParNew: 511104K->0K(511552K), 0.6916765 secs]
5860647K->5608632K(20479552K), 0.6917607 secs]
498.779: [GC 498.779: [ParNew: 511104K->0K(511552K), 0.9550482 secs]
6119736K->5940233K(20479552K), 0.9551500 secs]
511.704: [GC 511.704: [ParNew: 511104K->0K(511552K), 0.6890528 secs]
6451337K->6142170K(20479552K), 0.6891443 secs]
512.396: [GC [1 CMS-initial-mark: 6142170K(19968000K)] 6142375K(20479552K),
0.0009855 secs]
512.397: [CMS-concurrent-mark-start]
519.473: [GC 519.473: [ParNew: 511104K->0K(511552K), 0.5543093 secs]
6653274K->6332433K(20479552K), 0.5543996 secs]
525.948: [GC 525.948: [ParNew: 511104K->0K(511552K), 0.2174852 secs]
6843537K->6402868K(20479552K), 0.2175736 secs]
530.190: [CMS-concurrent-mark: 17.010/17.794 secs]
 (CMS-concurrent-mark yielded 13 times)
530.190: [CMS-concurrent-preclean-start]
531.866: [GC 531.866: [ParNew: 511104K->0K(511552K), 0.2369285 secs]
6913972K->6474400K(20479552K), 0.2370139 secs]
 (cardTable: 80762 cards, re-scanned 1261062 cards, 2 iterations)
532.162: [CMS-concurrent-preclean: 1.663/1.972 secs]
 (CMS-concurrent-preclean yielded 1 times)
532.162: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 125332 cards, re-scanned 735220 cards, 2 iterations)
 CMS: abort preclean due to time  [1 iterations, 0 waits, 735220 cards)]
533.358: [CMS-concurrent-abortable-preclean: 1.080/1.195 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
533.363: [GC[YG occupancy: 110410 K (511552 K)]533.363: [Rescan (parallel)
Finished root rescan work in 4th thread: 0.020 sec
Finished root rescan work in 5th thread: 0.020 sec
Finished root rescan work in 6th thread: 0.021 sec
Finished root rescan work in 1th thread: 0.021 sec
Finished root rescan work in 7th thread: 0.037 sec
Finished root rescan work in 2th thread: 0.039 sec
Finished root rescan work in 3th thread: 0.040 sec
Finished root rescan work in 0th thread: 0.040 sec
Finished dirty card rescan work in 0th thread: 0.001 sec
Finished dirty card rescan work in 3th thread: 0.001 sec
Finished dirty card rescan work in 7th thread: 0.004 sec
Finished dirty card rescan work in 2th thread: 0.002 sec
Finished dirty card rescan work in 6th thread: 0.020 sec
Finished dirty card rescan work in 4th thread: 0.020 sec
Finished dirty card rescan work in 1th thread: 0.020 sec
Finished dirty card rescan work in 5th thread: 0.020 sec
Finished work stealing in 0th thread: 0.000 sec
Finished work stealing in 3th thread: 0.000 sec
Finished work stealing in 5th thread: 0.000 sec
Finished work stealing in 7th thread: 0.000 sec
Finished work stealing in 6th thread: 0.000 sec
Finished work stealing in 2th thread: 0.000 sec
Finished work stealing in 1th thread: 0.000 sec
Finished work stealing in 4th thread: 0.000 sec
, 0.0422009 secs]533.405: [weak refs processing, 0.0041444 secs] [1
CMS-remark: 6474400K(19968000K)] 6584810K(20479552K), 0.0464604 secs]
533.411: [CMS-concurrent-sweep-start]
537.936: [GC 537.936: [ParNew: 511104K->0K(511552K), 0.2274676 secs]
6355567K->5914776K(20479552K), 0.2275568 secs]
539.902: [CMS-concurrent-sweep: 6.262/6.491 secs]
 (CMS-concurrent-sweep yielded 1 times)
539.903: [CMS-concurrent-reset-start]
539.954: [CMS-concurrent-reset: 0.051/0.051 secs]
 (CMS-concurrent-reset yielded 0 times)
544.373: [GC 544.373: [ParNew: 511104K->0K(511552K), 0.3436686 secs]
5928593K->5568531K(20479552K), 0.3437637 secs]
550.775: [GC 550.775: [ParNew: 511104K->0K(511552K), 0.2799366 secs]
6079635K->5679860K(20479552K), 0.2800241 secs]
555.517: [GC 555.517: [ParNew: 511104K->0K(511552K), 0.2173836 secs]
6190964K->5773027K(20479552K), 0.2174715 secs]
559.787: [GC 559.787: [ParNew: 511104K->0K(511552K), 0.1941384 secs]
6284131K->5859172K(20479552K), 0.1942224 secs]
562.693: [GC 562.693: [ParNew: 511104K->0K(511552K), 0.1068970 secs]
6370276K->5919835K(20479552K), 0.1069881 secs]
564.241: [GC 564.241: [ParNew: 511104K->0K(511552K), 0.0310694 secs]
6430939K->5935570K(20479552K), 0.0311656 secs]
565.717: [GC 565.717: [ParNew: 511104K->0K(511552K), 0.0312986 secs]
6446674K->5951305K(20479552K), 0.0313942 secs]
567.401: [GC 567.401: [ParNew: 511104K->0K(511552K), 0.0440493 secs]
6462409K->5974447K(20479552K), 0.0441344 secs]
570.574: [GC 570.574: [ParNew: 511104K->0K(511552K), 0.1122891 secs]
6485551K->6044567K(20479552K), 0.1123965 secs]
570.687: [GC [1 CMS-initial-mark: 6044567K(19968000K)] 6054843K(20479552K),
0.0007692 secs]
570.688: [CMS-concurrent-mark-start]
574.829: [GC 574.829: [ParNew: 511104K->0K(511552K), 0.1551952 secs]
6555671K->6143277K(20479552K), 0.1552766 secs]
578.927: [GC 578.927: [ParNew: 511104K->0K(511552K), 0.2672402 secs]
6654381K->6264990K(20479552K), 0.2673294 secs]
582.971: [GC 582.971: [ParNew: 511104K->0K(511552K), 0.2799158 secs]
6776094K->6387625K(20479552K), 0.2800000 secs]
587.239: [GC 587.239: [ParNew: 511104K->0K(511552K), 0.3188795 secs]
6898729K->6518431K(20479552K), 0.3189651 secs]
590.449: [CMS-concurrent-mark: 18.722/19.762 secs]
 (CMS-concurrent-mark yielded 14 times)
590.449: [CMS-concurrent-preclean-start]
 (cardTable: 363417 cards, re-scanned 2058169 cards, 2 iterations)
593.528: [CMS-concurrent-preclean: 2.949/3.079 secs]
 (CMS-concurrent-preclean yielded 0 times)
593.528: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 58884 cards, re-scanned 58884 cards, 1 iterations)
 (cardTable: 26087 cards, re-scanned 26087 cards, 1 iterations)
593.916: [GC 593.916: [ParNew: 511104K->0K(511552K), 0.2728526 secs]
7029535K->6655040K(20479552K), 0.2729535 secs]
 (cardTable: 15469 cards, re-scanned 15469 cards, 1 iterations)
 (cardTable: 140613 cards, re-scanned 821161 cards, 2 iterations)
 CMS: abort preclean due to time  [4 iterations, 0 waits, 921601 cards)]
595.404: [CMS-concurrent-abortable-preclean: 1.495/1.876 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
595.409: [GC[YG occupancy: 102823 K (511552 K)]595.409: [Rescan (parallel)
Finished root rescan work in 5th thread: 0.027 sec
Finished root rescan work in 1th thread: 0.027 sec
Finished root rescan work in 0th thread: 0.027 sec
Finished root rescan work in 4th thread: 0.027 sec
Finished root rescan work in 6th thread: 0.027 sec
Finished root rescan work in 7th thread: 0.028 sec
Finished dirty card rescan work in 0th thread: 0.019 sec
Finished dirty card rescan work in 4th thread: 0.019 sec
Finished dirty card rescan work in 6th thread: 0.019 sec
Finished dirty card rescan work in 1th thread: 0.019 sec
Finished dirty card rescan work in 7th thread: 0.019 sec
Finished dirty card rescan work in 5th thread: 0.019 sec
Finished root rescan work in 2th thread: 0.047 sec
Finished dirty card rescan work in 2th thread: 0.000 sec
Finished root rescan work in 3th thread: 0.051 sec
Finished dirty card rescan work in 3th thread: 0.000 sec
Finished work stealing in 3th thread: 0.000 sec
Finished work stealing in 0th thread: 0.005 sec
Finished work stealing in 7th thread: 0.005 sec
Finished work stealing in 6th thread: 0.005 sec
Finished work stealing in 4th thread: 0.005 sec
Finished work stealing in 2th thread: 0.005 sec
Finished work stealing in 5th thread: 0.006 sec
Finished work stealing in 1th thread: 0.006 sec
, 0.0539030 secs]595.463: [weak refs processing, 0.0041174 secs] [1
CMS-remark: 6655040K(19968000K)] 6757863K(20479552K), 0.0581083 secs]
595.469: [CMS-concurrent-sweep-start]
600.669: [GC 600.669: [ParNew: 511104K->0K(511552K), 0.2996252 secs]
6687227K->6323867K(20479552K), 0.2997161 secs]
602.214: [CMS-concurrent-sweep: 6.441/6.745 secs]
 (CMS-concurrent-sweep yielded 6 times)
602.215: [CMS-concurrent-reset-start]
602.266: [CMS-concurrent-reset: 0.051/0.051 secs]
 (CMS-concurrent-reset yielded 0 times)
604.265: [GC [1 CMS-initial-mark: 6111790K(19968000K)] 6369708K(20479552K),
0.5012984 secs]
604.767: [CMS-concurrent-mark-start]
606.696: [GC 606.696: [ParNew: 511104K->0K(511552K), 0.2422757 secs]
6622894K->6208475K(20479552K), 0.2423755 secs]
610.966: [GC 610.966: [ParNew: 511104K->0K(511552K), 0.1636319 secs]
6719579K->6282545K(20479552K), 0.1637219 secs]
613.548: [GC 613.548: [ParNew: 511104K->0K(511552K), 0.0980719 secs]
6793649K->6335058K(20479552K), 0.0981610 secs]
615.105: [GC 615.105: [ParNew: 511104K->0K(511552K), 0.0325480 secs]
6846162K->6350791K(20479552K), 0.0326367 secs]
616.670: [GC 616.670: [ParNew: 511104K->0K(511552K), 0.0341120 secs]
6861895K->6366523K(20479552K), 0.0341927 secs]
618.544: [GC 618.544: [ParNew: 511104K->0K(511552K), 0.0556126 secs]
6877627K->6392526K(20479552K), 0.0557086 secs]
622.029: [GC 622.029: [ParNew: 511104K->0K(511552K), 0.1234057 secs]
6903630K->6455571K(20479552K), 0.1234923 secs]
625.203: [CMS-concurrent-mark: 19.670/20.436 secs]
 (CMS-concurrent-mark yielded 31 times)
625.203: [CMS-concurrent-preclean-start]
626.771: [GC 626.771: [ParNew: 511104K->0K(511552K), 0.1720826 secs]
6966675K->6552594K(20479552K), 0.1721880 secs]
 (cardTable: 257359 cards, re-scanned 2255154 cards, 3 iterations)
629.246: [CMS-concurrent-preclean: 3.665/4.043 secs]
 (CMS-concurrent-preclean yielded 4 times)
629.246: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 68046 cards, re-scanned 68046 cards, 1 iterations)
 [1 iterations, 0 waits, 68046 cards)] 629.474:
[CMS-concurrent-abortable-preclean: 0.206/0.227 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
629.479: [GC[YG occupancy: 328846 K (511552 K)]629.479: [Rescan (parallel)
Finished root rescan work in 0th thread: 0.089 sec
Finished root rescan work in 5th thread: 0.089 sec
Finished root rescan work in 3th thread: 0.089 sec
Finished root rescan work in 7th thread: 0.089 sec
Finished root rescan work in 6th thread: 0.089 sec
Finished root rescan work in 1th thread: 0.090 sec
Finished root rescan work in 4th thread: 0.090 sec
Finished root rescan work in 2th thread: 0.090 sec
Finished dirty card rescan work in 3th thread: 0.012 sec
Finished dirty card rescan work in 2th thread: 0.012 sec
Finished dirty card rescan work in 6th thread: 0.012 sec
Finished dirty card rescan work in 5th thread: 0.013 sec
Finished dirty card rescan work in 4th thread: 0.012 sec
Finished dirty card rescan work in 0th thread: 0.013 sec
Finished dirty card rescan work in 1th thread: 0.012 sec
Finished dirty card rescan work in 7th thread: 0.012 sec
Finished work stealing in 3th thread: 0.000 sec
Finished work stealing in 0th thread: 0.000 sec
Finished work stealing in 1th thread: 0.000 sec
Finished work stealing in 5th thread: 0.000 sec
Finished work stealing in 4th thread: 0.000 sec
Finished work stealing in 7th thread: 0.000 sec
Finished work stealing in 6th thread: 0.000 sec
Finished work stealing in 2th thread: 0.000 sec
, 0.1037862 secs]629.583: [weak refs processing, 0.0040878 secs] [1
CMS-remark: 6552594K(19968000K)] 6881441K(20479552K), 0.1079643 secs]
629.589: [CMS-concurrent-sweep-start]
631.086: [GC 631.086: [ParNew: 511104K->0K(511552K), 0.2139077 secs]
6959656K->6548746K(20479552K), 0.2139938 secs]
636.189: [GC 636.189: [ParNew: 511104K->0K(511552K), 0.2391917 secs]
6831320K->6435331K(20479552K), 0.2392789 secs]
636.747: [CMS-concurrent-sweep: 6.703/7.158 secs]
 (CMS-concurrent-sweep yielded 3 times)
636.747: [CMS-concurrent-reset-start]
636.801: [CMS-concurrent-reset: 0.054/0.054 secs]
 (CMS-concurrent-reset yielded 0 times)
638.803: [GC [1 CMS-initial-mark: 6358688K(19968000K)] 6543780K(20479552K),
0.3865509 secs]
639.189: [CMS-concurrent-mark-start]
643.414: [GC 643.414: [ParNew: 511104K->0K(511552K), 0.2569440 secs]
6869792K->6486466K(20479552K), 0.2570434 secs]
650.635: [GC 650.635: [ParNew: 511104K->0K(511552K), 0.5590758 secs]
6997570K->6666452K(20479552K), 0.5591728 secs]
659.189: [GC 659.189: [ParNew: 511104K->0K(511552K), 0.5971190 secs]
7177556K->6830118K(20479552K), 0.5972131 secs]
662.572: [CMS-concurrent-mark: 21.945/23.382 secs]
 (CMS-concurrent-mark yielded 195 times)
662.572: [CMS-concurrent-preclean-start]
 (cardTable: 247095 cards, re-scanned 1964810 cards, 2 iterations)
665.578: [CMS-concurrent-preclean: 2.877/3.006 secs]
 (CMS-concurrent-preclean yielded 203 times)
665.578: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 20 cards, re-scanned 20 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 CMS: abort preclean due to time  [10 iterations, 9 waits, 20 cards)]
666.693: [CMS-concurrent-abortable-preclean: 0.663/1.115 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
666.696: [GC[YG occupancy: 451043 K (511552 K)]666.696: [Rescan (parallel)
Finished root rescan work in 7th thread: 0.000 sec
Finished root rescan work in 2th thread: 0.000 sec
Finished root rescan work in 4th thread: 0.000 sec
Finished root rescan work in 3th thread: 0.000 sec
Finished root rescan work in 1th thread: 0.000 sec
Finished root rescan work in 6th thread: 0.000 sec
Finished root rescan work in 5th thread: 0.001 sec
Finished dirty card rescan work in 1th thread: 0.005 sec
Finished dirty card rescan work in 6th thread: 0.005 sec
Finished dirty card rescan work in 5th thread: 0.004 sec
Finished dirty card rescan work in 2th thread: 0.005 sec
Finished dirty card rescan work in 4th thread: 0.005 sec
Finished dirty card rescan work in 7th thread: 0.005 sec
Finished dirty card rescan work in 3th thread: 0.005 sec
Finished root rescan work in 0th thread: 0.672 sec
Finished dirty card rescan work in 0th thread: 0.000 sec
Finished work stealing in 0th thread: 0.000 sec
Finished work stealing in 5th thread: 0.667 sec
Finished work stealing in 2th thread: 0.667 sec
Finished work stealing in 3th thread: 0.667 sec
Finished work stealing in 6th thread: 0.667 sec
Finished work stealing in 1th thread: 0.667 sec
Finished work stealing in 4th thread: 0.667 sec
Finished work stealing in 7th thread: 0.667 sec
, 0.6746487 secs]667.371: [weak refs processing, 0.0039127 secs] [1
CMS-remark: 6830118K(19968000K)] 7281162K(20479552K), 0.6786481 secs]
667.377: [CMS-concurrent-sweep-start]
668.533: [GC 668.533: [ParNew: 511104K->0K(511552K), 0.6987918 secs]
7171531K->6843652K(20479552K), 0.6988830 secs]
674.699: [CMS-concurrent-sweep: 6.622/7.322 secs]
 (CMS-concurrent-sweep yielded 29 times)
674.699: [CMS-concurrent-reset-start]
674.766: [CMS-concurrent-reset: 0.068/0.068 secs]
 (CMS-concurrent-reset yielded 0 times)
675.235: [GC [1 CMS-initial-mark: 6514340K(19968000K)] 6850985K(20479552K),
0.5171104 secs]
675.752: [CMS-concurrent-mark-start]
678.255: [GC 678.255: [ParNew: 511104K->0K(511552K), 0.5613177 secs]
7025444K->6680399K(20479552K), 0.5614320 secs]
686.862: [GC 686.862: [ParNew: 511104K->0K(511552K), 0.6123381 secs]
7191503K->6894577K(20479552K), 0.6124581 secs]
692.569: [GC 692.569: [ParNew: 511104K->0K(511552K), 1.0117371 secs]
7405681K->7168634K(20479552K), 1.0118288 secs]
700.570: [GC 700.570: [ParNew: 511104K->0K(511552K), 0.7273545 secs]
7679738K->7362685K(20479552K), 0.7274479 secs]
702.364: [CMS-concurrent-mark: 23.678/26.611 secs]
 (CMS-concurrent-mark yielded 135 times)
702.364: [CMS-concurrent-preclean-start]
705.164: [GC 705.164: [ParNew: 511104K->0K(511552K), 0.9172538 secs]
7873789K->7667034K(20479552K), 0.9173413 secs]
 (cardTable: 50463 cards, re-scanned 3549643 cards, 3 iterations)
707.648: [CMS-concurrent-preclean: 4.240/5.284 secs]
 (CMS-concurrent-preclean yielded 21 times)
707.648: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 42625 cards, re-scanned 42625 cards, 1 iterations)
 (cardTable: 37161 cards, re-scanned 37161 cards, 1 iterations)
 (cardTable: 370 cards, re-scanned 370 cards, 1 iterations)
 (cardTable: 26349 cards, re-scanned 26349 cards, 1 iterations)
 (cardTable: 37774 cards, re-scanned 37774 cards, 1 iterations)
 (cardTable: 37642 cards, re-scanned 37642 cards, 1 iterations)
 (cardTable: 15363 cards, re-scanned 15363 cards, 1 iterations)
 (cardTable: 8 cards, re-scanned 8 cards, 1 iterations)
 (cardTable: 44285 cards, re-scanned 44285 cards, 1 iterations)
 CMS: abort preclean due to time  [9 iterations, 1 waits, 241577 cards)]
708.651: [CMS-concurrent-abortable-preclean: 0.917/1.002 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
708.654: [GC[YG occupancy: 202539 K (511552 K)]708.654: [Rescan (parallel)
Finished root rescan work in 3th thread: 0.061 sec
Finished root rescan work in 0th thread: 0.061 sec
Finished root rescan work in 2th thread: 0.061 sec
Finished root rescan work in 1th thread: 0.061 sec
Finished root rescan work in 7th thread: 0.062 sec
Finished root rescan work in 4th thread: 0.062 sec
Finished root rescan work in 5th thread: 0.071 sec
Finished dirty card rescan work in 5th thread: 0.004 sec
Finished dirty card rescan work in 3th thread: 0.014 sec
Finished dirty card rescan work in 4th thread: 0.012 sec
Finished dirty card rescan work in 7th thread: 0.013 sec
Finished dirty card rescan work in 1th thread: 0.013 sec
Finished dirty card rescan work in 2th thread: 0.013 sec
Finished dirty card rescan work in 0th thread: 0.014 sec
Finished root rescan work in 6th thread: 0.083 sec
Finished dirty card rescan work in 6th thread: 0.000 sec
Finished work stealing in 6th thread: 0.000 sec
Finished work stealing in 4th thread: 0.009 sec
Finished work stealing in 5th thread: 0.009 sec
Finished work stealing in 3th thread: 0.009 sec
Finished work stealing in 0th thread: 0.009 sec
Finished work stealing in 1th thread: 0.009 sec
Finished work stealing in 7th thread: 0.009 sec
Finished work stealing in 2th thread: 0.009 sec
, 0.0857799 secs]708.740: [weak refs processing, 0.0034750 secs] [1
CMS-remark: 7667034K(19968000K)] 7869573K(20479552K), 0.0893439 secs]
708.745: [CMS-concurrent-sweep-start]
712.505: [GC 712.505: [ParNew: 511104K->0K(511552K), 0.4207328 secs]
8124807K->7747587K(20479552K), 0.4208246 secs]
716.134: [CMS-concurrent-sweep: 6.968/7.390 secs]
 (CMS-concurrent-sweep yielded 50 times)
716.134: [CMS-concurrent-reset-start]
716.186: [CMS-concurrent-reset: 0.052/0.052 secs]
 (CMS-concurrent-reset yielded 0 times)
718.188: [GC [1 CMS-initial-mark: 7578019K(19968000K)] 7643223K(20479552K),
0.1262421 secs]
718.314: [CMS-concurrent-mark-start]
724.085: [GC 724.085: [ParNew: 511104K->0K(511552K), 0.5070473 secs]
8089123K->7741504K(20479552K), 0.5071396 secs]
729.497: [GC 729.497: [ParNew: 511104K->0K(511552K), 0.3191565 secs]
8252608K->7861660K(20479552K), 0.3192474 secs]
734.721: [GC 734.721: [ParNew: 511104K->0K(511552K), 0.3763020 secs]
8372764K->7981059K(20479552K), 0.3763969 secs]
744.769: [CMS-concurrent-mark: 25.243/26.455 secs]
 (CMS-concurrent-mark yielded 5 times)
744.769: [CMS-concurrent-preclean-start]
 (cardTable: 273967 cards, re-scanned 2078622 cards, 2 iterations)
748.050: [CMS-concurrent-preclean: 3.120/3.280 secs]
 (CMS-concurrent-preclean yielded 0 times)
748.050: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 21 cards, re-scanned 21 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 1 cards, re-scanned 1 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 3 cards, re-scanned 3 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 CMS: abort preclean due to time  [9 iterations, 8 waits, 25 cards)]
749.079: [CMS-concurrent-abortable-preclean: 0.627/1.029 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
749.084: [GC[YG occupancy: 428465 K (511552 K)]749.084: [Rescan (parallel)
Finished root rescan work in 5th thread: 0.000 sec
Finished root rescan work in 4th thread: 0.000 sec
Finished root rescan work in 2th thread: 0.000 sec
Finished root rescan work in 6th thread: 0.000 sec
Finished root rescan work in 3th thread: 0.000 sec
Finished root rescan work in 7th thread: 0.001 sec
Finished root rescan work in 1th thread: 0.001 sec
Finished dirty card rescan work in 5th thread: 0.006 sec
Finished dirty card rescan work in 1th thread: 0.005 sec
Finished dirty card rescan work in 4th thread: 0.006 sec
Finished dirty card rescan work in 3th thread: 0.006 sec
Finished dirty card rescan work in 7th thread: 0.005 sec
Finished dirty card rescan work in 2th thread: 0.006 sec
Finished dirty card rescan work in 6th thread: 0.006 sec
Finished root rescan work in 0th thread: 0.647 sec
Finished dirty card rescan work in 0th thread: 0.000 sec
Finished work stealing in 0th thread: 0.000 sec
Finished work stealing in 1th thread: 0.641 sec
Finished work stealing in 3th thread: 0.641 sec
Finished work stealing in 7th thread: 0.641 sec
Finished work stealing in 5th thread: 0.641 sec
Finished work stealing in 4th thread: 0.641 sec
Finished work stealing in 6th thread: 0.641 sec
Finished work stealing in 2th thread: 0.641 sec
, 0.6487285 secs]749.733: [weak refs processing, 0.0038747 secs] [1
CMS-remark: 7981059K(19968000K)] 8409525K(20479552K), 0.6526974 secs]
749.739: [CMS-concurrent-sweep-start]
750.162: [GC 750.162: [ParNew: 511104K->0K(511552K), 0.4654205 secs]
8450886K->8062505K(20479552K), 0.4655102 secs]
758.152: [CMS-concurrent-sweep: 7.943/8.413 secs]
 (CMS-concurrent-sweep yielded 4 times)
758.152: [CMS-concurrent-reset-start]
758.202: [CMS-concurrent-reset: 0.050/0.050 secs]
 (CMS-concurrent-reset yielded 0 times)
759.631: [GC [1 CMS-initial-mark: 7637037K(19968000K)] 7778462K(20479552K),
0.1727130 secs]
759.804: [CMS-concurrent-mark-start]
762.833: [GC 762.833: [ParNew: 511104K->0K(511552K), 0.2362660 secs]
8148141K->7717274K(20479552K), 0.2363653 secs]
777.934: [GC 777.934: [ParNew: 511104K->0K(511552K), 0.1780274 secs]
8228378K->7768683K(20479552K), 0.1781192 secs]
782.664: [GC 782.664: [ParNew: 511104K->0K(511552K), 0.1654980 secs]
8279787K->7822935K(20479552K), 0.1655849 secs]
785.996: [CMS-concurrent-mark: 25.594/26.193 secs]
 (CMS-concurrent-mark yielded 47 times)
785.996: [CMS-concurrent-preclean-start]
787.183: [GC 787.183: [ParNew: 511104K->0K(511552K), 0.2082788 secs]
8334039K->7883477K(20479552K), 0.2083668 secs]
 (cardTable: 153623 cards, re-scanned 1673787 cards, 3 iterations)
789.522: [CMS-concurrent-preclean: 3.103/3.526 secs]
 (CMS-concurrent-preclean yielded 0 times)
789.522: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 24425 cards, re-scanned 24425 cards, 1 iterations)
 (cardTable: 11658 cards, re-scanned 11658 cards, 1 iterations)
 [2 iterations, 0 waits, 36083 cards)] 789.756:
[CMS-concurrent-abortable-preclean: 0.220/0.234 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
789.763: [GC[YG occupancy: 259227 K (511552 K)]789.763: [Rescan (parallel)
Finished root rescan work in 2th thread: 0.055 sec
Finished root rescan work in 1th thread: 0.056 sec
Finished root rescan work in 0th thread: 0.056 sec
Finished root rescan work in 3th thread: 0.069 sec
Finished root rescan work in 7th thread: 0.072 sec
Finished root rescan work in 4th thread: 0.072 sec
Finished root rescan work in 5th thread: 0.073 sec
Finished root rescan work in 6th thread: 0.073 sec
Finished dirty card rescan work in 5th thread: 0.000 sec
Finished dirty card rescan work in 3th thread: 0.004 sec
Finished dirty card rescan work in 4th thread: 0.000 sec
Finished dirty card rescan work in 1th thread: 0.017 sec
Finished dirty card rescan work in 7th thread: 0.000 sec
Finished dirty card rescan work in 0th thread: 0.017 sec
Finished dirty card rescan work in 2th thread: 0.018 sec
Finished dirty card rescan work in 6th thread: 0.000 sec
Finished work stealing in 7th thread: 0.000 sec
Finished work stealing in 6th thread: 0.000 sec
Finished work stealing in 0th thread: 0.000 sec
Finished work stealing in 3th thread: 0.000 sec
Finished work stealing in 4th thread: 0.000 sec
Finished work stealing in 1th thread: 0.000 sec
Finished work stealing in 2th thread: 0.000 sec
Finished work stealing in 5th thread: 0.002 sec
, 0.0748844 secs]789.838: [weak refs processing, 0.0039506 secs] [1
CMS-remark: 7883477K(19968000K)] 8142705K(20479552K), 0.0789296 secs]
789.844: [CMS-concurrent-sweep-start]
793.524: [GC 793.524: [ParNew: 511104K->0K(511552K), 0.2793121 secs]
8359118K->7946170K(20479552K), 0.2794056 secs]
798.260: [CMS-concurrent-sweep: 8.133/8.416 secs]
 (CMS-concurrent-sweep yielded 4 times)
798.260: [CMS-concurrent-reset-start]
798.312: [CMS-concurrent-reset: 0.052/0.052 secs]
 (CMS-concurrent-reset yielded 0 times)
800.315: [GC [1 CMS-initial-mark: 7782344K(19968000K)] 8161000K(20479552K),
0.6957516 secs]
801.011: [CMS-concurrent-mark-start]
804.122: [GC 804.122: [ParNew: 511104K->0K(511552K), 0.2372039 secs]
8293448K->7862317K(20479552K), 0.2372839 secs]
808.756: [GC 808.756: [ParNew: 511104K->0K(511552K), 0.1191233 secs]
8373421K->7900123K(20479552K), 0.1192169 secs]
813.166: [GC 813.166: [ParNew: 511104K->0K(511552K), 0.1048696 secs]
8411227K->7938704K(20479552K), 0.1049700 secs]
817.774: [GC 817.774: [ParNew: 511104K->0K(511552K), 0.1095556 secs]
8449808K->7973780K(20479552K), 0.1096427 secs]
822.225: [GC 822.225: [ParNew: 510645K->0K(511552K), 0.1956629 secs]
8484425K->8038630K(20479552K), 0.1957720 secs]
829.083: [CMS-concurrent-mark: 27.289/28.072 secs]
 (CMS-concurrent-mark yielded 60 times)
829.083: [CMS-concurrent-preclean-start]
830.242: [GC 830.242: [ParNew: 511104K->0K(511552K), 0.5830426 secs]
8549734K->8213662K(20479552K), 0.5831452 secs]
 (cardTable: 111730 cards, re-scanned 2016509 cards, 3 iterations)
833.312: [CMS-concurrent-preclean: 3.437/4.229 secs]
 (CMS-concurrent-preclean yielded 6 times)
833.312: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 1 cards, re-scanned 1 cards, 1 iterations)
 (cardTable: 1 cards, re-scanned 1 cards, 1 iterations)
 (cardTable: 1 cards, re-scanned 1 cards, 1 iterations)
 (cardTable: 1 cards, re-scanned 1 cards, 1 iterations)
 (cardTable: 1 cards, re-scanned 1 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 1 cards, re-scanned 1 cards, 1 iterations)
 (cardTable: 1 cards, re-scanned 1 cards, 1 iterations)
 (cardTable: 5 cards, re-scanned 5 cards, 1 iterations)
 CMS: abort preclean due to time  [9 iterations, 8 waits, 12 cards)]
834.334: [CMS-concurrent-abortable-preclean: 0.620/1.022 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
834.341: [GC[YG occupancy: 145687 K (511552 K)]834.341: [Rescan (parallel)
Finished root rescan work in 0th thread: 0.039 sec
Finished root rescan work in 2th thread: 0.040 sec
Finished root rescan work in 4th thread: 0.041 sec
Finished root rescan work in 1th thread: 0.041 sec
Finished root rescan work in 7th thread: 0.043 sec
Finished root rescan work in 5th thread: 0.044 sec
Finished root rescan work in 6th thread: 0.045 sec
Finished root rescan work in 3th thread: 0.045 sec
Finished dirty card rescan work in 0th thread: 0.008 sec
Finished dirty card rescan work in 7th thread: 0.004 sec
Finished dirty card rescan work in 3th thread: 0.002 sec
Finished dirty card rescan work in 1th thread: 0.006 sec
Finished dirty card rescan work in 5th thread: 0.003 sec
Finished dirty card rescan work in 6th thread: 0.002 sec
Finished dirty card rescan work in 2th thread: 0.007 sec
Finished dirty card rescan work in 4th thread: 0.006 sec
Finished work stealing in 0th thread: 0.000 sec
Finished work stealing in 4th thread: 0.000 sec
Finished work stealing in 3th thread: 0.000 sec
Finished work stealing in 7th thread: 0.000 sec
Finished work stealing in 1th thread: 0.000 sec
Finished work stealing in 6th thread: 0.000 sec
Finished work stealing in 5th thread: 0.000 sec
Finished work stealing in 2th thread: 0.000 sec
, 0.0491305 secs]834.390: [weak refs processing, 0.0049754 secs] [1
CMS-remark: 8213662K(19968000K)] 8359349K(20479552K), 0.0542113 secs]
834.397: [CMS-concurrent-sweep-start]
840.693: [GC 840.694: [ParNew: 511104K->0K(511552K), 0.2906592 secs]
8654473K->8224927K(20479552K), 0.2907565 secs]
842.762: [CMS-concurrent-sweep: 8.071/8.365 secs]
 (CMS-concurrent-sweep yielded 31 times)
842.762: [CMS-concurrent-reset-start]
842.814: [CMS-concurrent-reset: 0.051/0.051 secs]
 (CMS-concurrent-reset yielded 0 times)
844.817: [GC [1 CMS-initial-mark: 8140494K(19968000K)] 8371327K(20479552K),
0.3368048 secs]
845.154: [CMS-concurrent-mark-start]
850.137: [GC 850.137: [ParNew: 511104K->0K(511552K), 0.5255524 secs]
8651598K->8294544K(20479552K), 0.5256546 secs]
859.754: [GC 859.754: [ParNew: 511101K->0K(511552K), 0.2970488 secs]
8805646K->8387248K(20479552K), 0.2971573 secs]
869.681: [GC 869.681: [ParNew: 511104K->0K(511552K), 0.4707484 secs]
8898352K->8522811K(20479552K), 0.4708463 secs]
875.059: [CMS-concurrent-mark: 28.602/29.905 secs]
 (CMS-concurrent-mark yielded 151 times)
875.059: [CMS-concurrent-preclean-start]
 (cardTable: 159831 cards, re-scanned 1255841 cards, 2 iterations)
877.126: [CMS-concurrent-preclean: 1.983/2.067 secs]
 (CMS-concurrent-preclean yielded 1 times)
877.126: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 21 cards, re-scanned 21 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 1 cards, re-scanned 1 cards, 1 iterations)
 CMS: abort preclean due to time  [10 iterations, 9 waits, 22 cards)]
878.239: [CMS-concurrent-abortable-preclean: 0.661/1.113 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
878.243: [GC[YG occupancy: 481934 K (511552 K)]878.243: [Rescan (parallel)
Finished root rescan work in 6th thread: 0.000 sec
Finished root rescan work in 7th thread: 0.000 sec
Finished root rescan work in 3th thread: 0.000 sec
Finished root rescan work in 1th thread: 0.000 sec
Finished root rescan work in 2th thread: 0.000 sec
Finished root rescan work in 5th thread: 0.000 sec
Finished root rescan work in 4th thread: 0.001 sec
Finished dirty card rescan work in 6th thread: 0.007 sec
Finished dirty card rescan work in 4th thread: 0.006 sec
Finished dirty card rescan work in 1th thread: 0.007 sec
Finished dirty card rescan work in 2th thread: 0.007 sec
Finished dirty card rescan work in 3th thread: 0.007 sec
Finished dirty card rescan work in 5th thread: 0.000 sec
Finished dirty card rescan work in 7th thread: 0.007 sec
Finished root rescan work in 0th thread: 0.799 sec
Finished dirty card rescan work in 0th thread: 0.000 sec
Finished work stealing in 0th thread: 0.000 sec
Finished work stealing in 6th thread: 0.792 sec
Finished work stealing in 7th thread: 0.793 sec
Finished work stealing in 1th thread: 0.793 sec
Finished work stealing in 2th thread: 0.793 sec
Finished work stealing in 3th thread: 0.793 sec
Finished work stealing in 5th thread: 0.793 sec
Finished work stealing in 4th thread: 0.793 sec
, 0.8015677 secs]879.044: [weak refs processing, 0.0035149 secs] [1
CMS-remark: 8522811K(19968000K)] 9004746K(20479552K), 0.8051625 secs]
879.049: [CMS-concurrent-sweep-start]
879.528: [GC 879.528: [ParNew: 511104K->0K(511552K), 0.3601451 secs]
9029683K->8612491K(20479552K), 0.3602350 secs]
887.767: [CMS-concurrent-sweep: 8.353/8.718 secs]
 (CMS-concurrent-sweep yielded 5 times)
887.767: [CMS-concurrent-reset-start]
887.819: [CMS-concurrent-reset: 0.051/0.051 secs]
 (CMS-concurrent-reset yielded 0 times)
889.715: [GC 889.715: [ParNew: 511104K->0K(511552K), 0.3684124 secs]
8989644K->8623042K(20479552K), 0.3685156 secs]
890.085: [GC [1 CMS-initial-mark: 8623042K(19968000K)] 8628032K(20479552K),
0.0007955 secs]
890.086: [CMS-concurrent-mark-start]
900.484: [GC 900.485: [ParNew: 511104K->0K(511552K), 0.5899399 secs]
9134146K->8787247K(20479552K), 0.5900505 secs]
909.118: [GC 909.119: [ParNew: 511104K->0K(511552K), 0.8387118 secs]
9298351K->9053545K(20479552K), 0.8388006 secs]
918.540: [GC 918.540: [ParNew: 511104K->0K(511552K), 0.7218656 secs]
9564649K->9285203K(20479552K), 0.7219738 secs]
923.175: [CMS-concurrent-mark: 30.922/33.089 secs]
 (CMS-concurrent-mark yielded 248 times)
923.175: [CMS-concurrent-preclean-start]
924.799: [GC 924.800: [ParNew: 511104K->0K(511552K), 0.2914306 secs]
9796307K->9387045K(20479552K), 0.2915123 secs]
 (cardTable: 316898 cards, re-scanned 3567167 cards, 3 iterations)
928.732: [CMS-concurrent-preclean: 4.999/5.557 secs]
 (CMS-concurrent-preclean yielded 1 times)
928.732: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 58312 cards, re-scanned 58312 cards, 1 iterations)
 [1 iterations, 0 waits, 58312 cards)] 928.934:
[CMS-concurrent-abortable-preclean: 0.185/0.202 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
928.941: [GC[YG occupancy: 462263 K (511552 K)]928.941: [Rescan (parallel)
Finished root rescan work in 1th thread: 0.114 sec
Finished root rescan work in 0th thread: 0.114 sec
Finished root rescan work in 4th thread: 0.114 sec
Finished root rescan work in 7th thread: 0.114 sec
Finished root rescan work in 5th thread: 0.117 sec
Finished root rescan work in 6th thread: 0.128 sec
Finished root rescan work in 3th thread: 0.129 sec
Finished dirty card rescan work in 5th thread: 0.015 sec
Finished dirty card rescan work in 4th thread: 0.018 sec
Finished dirty card rescan work in 0th thread: 0.018 sec
Finished dirty card rescan work in 3th thread: 0.003 sec
Finished dirty card rescan work in 1th thread: 0.018 sec
Finished dirty card rescan work in 7th thread: 0.017 sec
Finished dirty card rescan work in 6th thread: 0.004 sec
Finished root rescan work in 2th thread: 0.156 sec
Finished dirty card rescan work in 2th thread: 0.000 sec
Finished work stealing in 2th thread: 0.000 sec
Finished work stealing in 7th thread: 0.025 sec
Finished work stealing in 0th thread: 0.025 sec
Finished work stealing in 1th thread: 0.025 sec
Finished work stealing in 3th thread: 0.025 sec
Finished work stealing in 6th thread: 0.025 sec
Finished work stealing in 5th thread: 0.025 sec
Finished work stealing in 4th thread: 0.025 sec
, 0.1592149 secs]929.101: [weak refs processing, 0.0040734 secs] [1
CMS-remark: 9387045K(19968000K)] 9849308K(20479552K), 0.1633733 secs]
929.106: [CMS-concurrent-sweep-start]
929.517: [GC 929.517: [ParNew: 511104K->0K(511552K), 0.2243274 secs]
9894392K->9467222K(20479552K), 0.2244136 secs]
932.011: [GC 932.011: [ParNew: 511104K->0K(511552K), 0.0801678 secs]
9974315K->9502811K(20479552K), 0.0802626 secs]
933.721: [GC 933.721: [ParNew: 511104K->0K(511552K), 0.0387052 secs]
10010818K->9516322K(20479552K), 0.0387929 secs]
936.011: [GC 936.011: [ParNew: 511104K->0K(511552K), 0.0799769 secs]
10024229K->9549526K(20479552K), 0.0800657 secs]
938.396: [CMS-concurrent-sweep: 8.856/9.290 secs]
 (CMS-concurrent-sweep yielded 5 times)
938.397: [CMS-concurrent-reset-start]
938.451: [CMS-concurrent-reset: 0.054/0.054 secs]
 (CMS-concurrent-reset yielded 0 times)
939.723: [GC 939.723: [ParNew: 511104K->0K(511552K), 0.1427382 secs]
9991830K->9559431K(20479552K), 0.1428303 secs]
939.868: [GC [1 CMS-initial-mark: 9559431K(19968000K)] 9569447K(20479552K),
0.0008202 secs]
939.869: [CMS-concurrent-mark-start]
947.478: [GC 947.479: [ParNew: 511104K->0K(511552K), 0.2587277 secs]
10070535K->9676816K(20479552K), 0.2588171 secs]
954.350: [GC 954.350: [ParNew: 511104K->0K(511552K), 0.4023464 secs]
10187920K->9844118K(20479552K), 0.4024424 secs]
961.480: [GC 961.480: [ParNew: 511104K->0K(511552K), 0.3926957 secs]
10355222K->9990001K(20479552K), 0.3928016 secs]
966.744: [GC 966.744: [ParNew: 511104K->0K(511552K), 0.2423191 secs]
10501105K->10074081K(20479552K), 0.2424164 secs]
971.032: [GC 971.032: [ParNew: 511104K->0K(511552K), 0.1767435 secs]
10585185K->10134759K(20479552K), 0.1768684 secs]
972.777: [CMS-concurrent-mark: 31.427/32.908 secs]
 (CMS-concurrent-mark yielded 12 times)
972.777: [CMS-concurrent-preclean-start]
975.219: [GC 975.219: [ParNew: 511104K->0K(511552K), 0.1726277 secs]
10645863K->10195002K(20479552K), 0.1727317 secs]
 (cardTable: 226347 cards, re-scanned 3143987 cards, 3 iterations)
977.736: [CMS-concurrent-preclean: 4.540/4.960 secs]
 (CMS-concurrent-preclean yielded 1 times)
977.736: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 45975 cards, re-scanned 45975 cards, 1 iterations)
 (cardTable: 21682 cards, re-scanned 21682 cards, 1 iterations)
 (cardTable: 17914 cards, re-scanned 17914 cards, 1 iterations)
 [3 iterations, 0 waits, 85571 cards)] 978.155:
[CMS-concurrent-abortable-preclean: 0.388/0.419 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
978.159: [GC[YG occupancy: 257299 K (511552 K)]978.159: [Rescan (parallel)
Finished root rescan work in 4th thread: 0.070 sec
Finished root rescan work in 7th thread: 0.070 sec
Finished root rescan work in 2th thread: 0.070 sec
Finished root rescan work in 5th thread: 0.070 sec
Finished root rescan work in 3th thread: 0.070 sec
Finished root rescan work in 6th thread: 0.071 sec
Finished root rescan work in 1th thread: 0.072 sec
Finished dirty card rescan work in 4th thread: 0.012 sec
Finished dirty card rescan work in 1th thread: 0.010 sec
Finished dirty card rescan work in 5th thread: 0.012 sec
Finished dirty card rescan work in 2th thread: 0.012 sec
Finished dirty card rescan work in 7th thread: 0.012 sec
Finished dirty card rescan work in 3th thread: 0.011 sec
Finished dirty card rescan work in 6th thread: 0.011 sec
Finished root rescan work in 0th thread: 0.087 sec
Finished dirty card rescan work in 0th thread: 0.000 sec
Finished work stealing in 0th thread: 0.000 sec
Finished work stealing in 3th thread: 0.006 sec
Finished work stealing in 5th thread: 0.006 sec
Finished work stealing in 1th thread: 0.006 sec
Finished work stealing in 6th thread: 0.006 sec
Finished work stealing in 7th thread: 0.006 sec
Finished work stealing in 2th thread: 0.006 sec
Finished work stealing in 4th thread: 0.006 sec
, 0.0897017 secs]978.249: [weak refs processing, 0.0040880 secs] [1
CMS-remark: 10195002K(19968000K)] 10452301K(20479552K), 0.0938730 secs]
978.255: [CMS-concurrent-sweep-start]
981.149: [GC 981.149: [ParNew: 511104K->0K(511552K), 0.2851940 secs]
10636932K->10216282K(20479552K), 0.2852903 secs]
986.345: [GC 986.346: [ParNew: 511104K->0K(511552K), 0.2381327 secs]
10574224K->10143919K(20479552K), 0.2382335 secs]
988.091: [GC 988.091: [ParNew: 511104K->0K(511552K), 0.0381336 secs]
10299700K->9803219K(20479552K), 0.0382412 secs]
988.234: [CMS-concurrent-sweep: 9.417/9.980 secs]
 (CMS-concurrent-sweep yielded 3 times)
988.235: [CMS-concurrent-reset-start]
988.291: [CMS-concurrent-reset: 0.056/0.056 secs]
 (CMS-concurrent-reset yielded 0 times)
989.592: [GC 989.592: [ParNew: 511104K->0K(511552K), 0.0373781 secs]
10173778K->9677298K(20479552K), 0.0374650 secs]
989.630: [GC [1 CMS-initial-mark: 9677298K(19968000K)] 9677298K(20479552K),
0.0008160 secs]
989.631: [CMS-concurrent-mark-start]
991.147: [GC 991.147: [ParNew: 511104K->0K(511552K), 0.0386906 secs]
10188402K->9691927K(20479552K), 0.0387842 secs]
992.640: [GC 992.640: [ParNew: 511104K->0K(511552K), 0.0393215 secs]
10203031K->9706549K(20479552K), 0.0394326 secs]
994.140: [GC 994.140: [ParNew: 511104K->0K(511552K), 0.0389781 secs]
10217653K->9721179K(20479552K), 0.0390642 secs]
995.860: [GC 995.860: [ParNew: 511104K->0K(511552K), 0.0538745 secs]
10232283K->9741271K(20479552K), 0.0539799 secs]
999.562: [GC 999.562: [ParNew: 511104K->0K(511552K), 0.1324355 secs]
10252375K->9801946K(20479552K), 0.1325237 secs]
1003.217: [GC 1003.217: [ParNew: 511104K->0K(511552K), 0.1348373 secs]
10313050K->9862578K(20479552K), 0.1349193 secs]
1008.202: [GC 1008.202: [ParNew: 511104K->0K(511552K), 0.2197866 secs]
10373682K->9985552K(20479552K), 0.2198775 secs]
1012.756: [GC 1012.756: [ParNew: 511104K->0K(511552K), 0.1972383 secs]
10496656K->10074078K(20479552K), 0.1973386 secs]
1017.206: [GC 1017.206: [ParNew: 511104K->0K(511552K), 0.2606849 secs]
10585182K->10159316K(20479552K), 0.2607717 secs]
1021.713: [GC 1021.713: [ParNew: 511104K->0K(511552K), 0.2138427 secs]
10670420K->10244591K(20479552K), 0.2139337 secs]
1024.314: [CMS-concurrent-mark: 33.350/34.683 secs]
 (CMS-concurrent-mark yielded 10 times)
1024.314: [CMS-concurrent-preclean-start]
 (cardTable: 444225 cards, re-scanned 2562099 cards, 2 iterations)
1028.506: [CMS-concurrent-preclean: 4.018/4.192 secs]
 (CMS-concurrent-preclean yielded 0 times)
1028.506: [CMS-concurrent-abortable-preclean-start]
1028.657: [GC 1028.657: [ParNew: 511104K->0K(511552K), 0.3699737 secs]
10755695K->10394576K(20479552K), 0.3700648 secs]
 (cardTable: 55503 cards, re-scanned 55503 cards, 1 iterations)
 (cardTable: 173253 cards, re-scanned 973694 cards, 2 iterations)
 CMS: abort preclean due to time  [2 iterations, 0 waits, 1029197 cards)]
1030.548: [CMS-concurrent-abortable-preclean: 1.563/2.043 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
1030.552: [GC[YG occupancy: 122769 K (511552 K)]1030.552: [Rescan (parallel)
Finished root rescan work in 5th thread: 0.028 sec
Finished root rescan work in 3th thread: 0.028 sec
Finished root rescan work in 7th thread: 0.028 sec
Finished root rescan work in 2th thread: 0.028 sec
Finished root rescan work in 4th thread: 0.040 sec
Finished root rescan work in 0th thread: 0.052 sec
Finished root rescan work in 1th thread: 0.054 sec
Finished root rescan work in 6th thread: 0.055 sec
Finished dirty card rescan work in 6th thread: 0.004 sec
Finished dirty card rescan work in 5th thread: 0.031 sec
Finished dirty card rescan work in 3th thread: 0.031 sec
Finished dirty card rescan work in 1th thread: 0.005 sec
Finished dirty card rescan work in 0th thread: 0.007 sec
Finished dirty card rescan work in 7th thread: 0.031 sec
Finished dirty card rescan work in 2th thread: 0.031 sec
Finished dirty card rescan work in 4th thread: 0.019 sec
Finished work stealing in 6th thread: 0.000 sec
Finished work stealing in 0th thread: 0.000 sec
Finished work stealing in 2th thread: 0.000 sec
Finished work stealing in 3th thread: 0.000 sec
Finished work stealing in 5th thread: 0.000 sec
Finished work stealing in 1th thread: 0.000 sec
Finished work stealing in 7th thread: 0.000 sec
Finished work stealing in 4th thread: 0.000 sec
, 0.0604143 secs]1030.613: [weak refs processing, 0.0041506 secs] [1
CMS-remark: 10394576K(19968000K)] 10517345K(20479552K), 0.0646520 secs]
1030.618: [CMS-concurrent-sweep-start]
1035.554: [GC 1035.554: [ParNew: 511104K->0K(511552K), 0.3376513 secs]
10863541K->10518937K(20479552K), 0.3377508 secs]
1040.881: [CMS-concurrent-sweep: 9.925/10.263 secs]
 (CMS-concurrent-sweep yielded 1 times)
1040.881: [CMS-concurrent-reset-start]
1040.935: [CMS-concurrent-reset: 0.053/0.053 secs]
 (CMS-concurrent-reset yielded 0 times)
1042.466: [GC 1042.466: [ParNew: 511104K->0K(511552K), 0.3357408 secs]
10513843K->10199567K(20479552K), 0.3358269 secs]
1042.804: [GC [1 CMS-initial-mark: 10199567K(19968000K)]
10199568K(20479552K), 0.0008126 secs]
1042.805: [CMS-concurrent-mark-start]
1048.274: [GC 1048.274: [ParNew: 511104K->0K(511552K), 0.2176103 secs]
10710671K->10281912K(20479552K), 0.2177058 secs]
1051.989: [GC 1051.989: [ParNew: 511104K->0K(511552K), 0.1336105 secs]
10793016K->10336497K(20479552K), 0.1336999 secs]
1053.669: [GC 1053.669: [ParNew: 511104K->0K(511552K), 0.0403536 secs]
10847601K->10351670K(20479552K), 0.0404649 secs]
1055.312: [GC 1055.312: [ParNew: 511104K->0K(511552K), 0.0397924 secs]
10862774K->10366844K(20479552K), 0.0398939 secs]
1058.662: [GC 1058.662: [ParNew: 511104K->0K(511552K), 0.1181157 secs]
10877948K->10420381K(20479552K), 0.1182020 secs]
1063.737: [GC 1063.737: [ParNew: 511104K->0K(511552K), 0.1548948 secs]
10931485K->10491673K(20479552K), 0.1549824 secs]
1069.058: [GC 1069.058: [ParNew: 511104K->0K(511552K), 0.2051767 secs]
11002777K->10583138K(20479552K), 0.2052763 secs]
1076.357: [GC 1076.357: [ParNew: 511104K->0K(511552K), 0.3305497 secs]
11094242K->10718363K(20479552K), 0.3306379 secs]
1078.393: [CMS-concurrent-mark: 34.345/35.588 secs]
 (CMS-concurrent-mark yielded 8 times)
1078.393: [CMS-concurrent-preclean-start]
 (cardTable: 639333 cards, re-scanned 2790874 cards, 2 iterations)
1083.267: [CMS-concurrent-preclean: 4.583/4.874 secs]
 (CMS-concurrent-preclean yielded 4 times)
1083.267: [CMS-concurrent-abortable-preclean-start]
1083.453: [GC 1083.453: [ParNew: 511104K->0K(511552K), 0.4489793 secs]
11229467K->10887944K(20479552K), 0.4490654 secs]
 (cardTable: 82676 cards, re-scanned 82676 cards, 1 iterations)
 (cardTable: 214098 cards, re-scanned 1204846 cards, 2 iterations)
 CMS: abort preclean due to time  [2 iterations, 0 waits, 1287522 cards)]
1085.733: [CMS-concurrent-abortable-preclean: 1.866/2.466 secs]
 (CMS-concurrent-abortable-preclean yielded 1 times)
1085.736: [GC[YG occupancy: 132996 K (511552 K)]1085.736: [Rescan (parallel)
Finished root rescan work in 6th thread: 0.028 sec
Finished root rescan work in 4th thread: 0.028 sec
Finished root rescan work in 2th thread: 0.028 sec
Finished root rescan work in 3th thread: 0.042 sec
Finished root rescan work in 7th thread: 0.053 sec
Finished root rescan work in 5th thread: 0.053 sec
Finished root rescan work in 1th thread: 0.054 sec
Finished root rescan work in 0th thread: 0.054 sec
Finished dirty card rescan work in 1th thread: 0.009 sec
Finished dirty card rescan work in 5th thread: 0.009 sec
Finished dirty card rescan work in 0th thread: 0.009 sec
Finished dirty card rescan work in 2th thread: 0.034 sec
Finished dirty card rescan work in 3th thread: 0.020 sec
Finished dirty card rescan work in 4th thread: 0.035 sec
Finished dirty card rescan work in 6th thread: 0.035 sec
Finished dirty card rescan work in 7th thread: 0.009 sec
Finished work stealing in 3th thread: 0.000 sec
Finished work stealing in 7th thread: 0.000 sec
Finished work stealing in 0th thread: 0.000 sec
Finished work stealing in 5th thread: 0.000 sec
Finished work stealing in 2th thread: 0.000 sec
Finished work stealing in 4th thread: 0.000 sec
Finished work stealing in 6th thread: 0.000 sec
Finished work stealing in 1th thread: 0.000 sec
, 0.0643763 secs]1085.801: [weak refs processing, 0.0043164 secs] [1
CMS-remark: 10887944K(19968000K)] 11020941K(20479552K), 0.0687798 secs]
1085.807: [CMS-concurrent-sweep-start]
1090.339: [GC 1090.339: [ParNew: 511104K->0K(511552K), 0.3083312 secs]
11346876K->10953533K(20479552K), 0.3084172 secs]
1094.892: [GC 1094.892: [ParNew: 511104K->0K(511552K), 0.1464757 secs]
11363112K->10907037K(20479552K), 0.1465629 secs]
1096.507: [CMS-concurrent-sweep: 10.245/10.700 secs]
 (CMS-concurrent-sweep yielded 2 times)
1096.507: [CMS-concurrent-reset-start]
1096.558: [CMS-concurrent-reset: 0.052/0.052 secs]
 (CMS-concurrent-reset yielded 0 times)
1098.559: [GC [1 CMS-initial-mark: 10238099K(19968000K)]
10688167K(20479552K), 0.7348449 secs]
1099.293: [CMS-concurrent-mark-start]
1099.723: [GC 1099.724: [ParNew: 511104K->0K(511552K), 0.1454485 secs]
10749203K->10291688K(20479552K), 0.1455364 secs]
1104.961: [GC 1104.961: [ParNew: 511104K->0K(511552K), 0.2109422 secs]
10802792K->10368965K(20479552K), 0.2110266 secs]
1110.900: [GC 1110.900: [ParNew: 511104K->0K(511552K), 0.2196373 secs]
10880069K->10457322K(20479552K), 0.2197353 secs]
1113.066: [GC 1113.066: [ParNew: 511104K->0K(511552K), 0.0647019 secs]
10968426K->10481572K(20479552K), 0.0647892 secs]
1114.588: [GC 1114.588: [ParNew: 511104K->0K(511552K), 0.0382566 secs]
10992676K->10496198K(20479552K), 0.0383687 secs]
1116.143: [GC 1116.143: [ParNew: 511104K->0K(511552K), 0.0387661 secs]
11007302K->10510825K(20479552K), 0.0388557 secs]
1117.648: [GC 1117.648: [ParNew: 511104K->0K(511552K), 0.0397888 secs]
11021929K->10525449K(20479552K), 0.0398804 secs]
1119.225: [GC 1119.225: [ParNew: 511104K->0K(511552K), 0.0402370 secs]
11036553K->10540078K(20479552K), 0.0403254 secs]
1120.710: [GC 1120.710: [ParNew: 511104K->0K(511552K), 0.0396171 secs]
11051182K->10554701K(20479552K), 0.0397017 secs]
1124.047: [GC 1124.047: [ParNew: 511104K->0K(511552K), 0.1194040 secs]
11065805K->10606965K(20479552K), 0.1194925 secs]
1127.932: [GC 1127.932: [ParNew: 511104K->0K(511552K), 0.1323603 secs]
11118069K->10665975K(20479552K), 0.1324455 secs]
1132.141: [GC 1132.141: [ParNew: 511104K->0K(511552K), 0.1857145 secs]
11177079K->10768287K(20479552K), 0.1858023 secs]
1135.758: [CMS-concurrent-mark: 35.183/36.464 secs]
 (CMS-concurrent-mark yielded 12 times)
1135.758: [CMS-concurrent-preclean-start]
1137.346: [GC 1137.346: [ParNew: 511104K->0K(511552K), 0.1694393 secs]
11279391K->10846877K(20479552K), 0.1695246 secs]
 (cardTable: 355359 cards, re-scanned 2912466 cards, 3 iterations)
1140.924: [CMS-concurrent-preclean: 4.730/5.166 secs]
 (CMS-concurrent-preclean yielded 0 times)
1140.924: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 85087 cards, re-scanned 85087 cards, 1 iterations)
 [1 iterations, 0 waits, 85087 cards)] 1141.195:
[CMS-concurrent-abortable-preclean: 0.244/0.271 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
1141.199: [GC[YG occupancy: 460291 K (511552 K)]1141.199: [Rescan (parallel)
Finished root rescan work in 1th thread: 0.115 sec
Finished root rescan work in 6th thread: 0.115 sec
Finished root rescan work in 7th thread: 0.116 sec
Finished root rescan work in 3th thread: 0.119 sec
Finished root rescan work in 5th thread: 0.135 sec
Finished root rescan work in 4th thread: 0.135 sec
Finished root rescan work in 0th thread: 0.136 sec
Finished root rescan work in 2th thread: 0.137 sec
Finished dirty card rescan work in 5th thread: 0.007 sec
Finished dirty card rescan work in 2th thread: 0.006 sec
Finished dirty card rescan work in 4th thread: 0.007 sec
Finished dirty card rescan work in 3th thread: 0.023 sec
Finished dirty card rescan work in 0th thread: 0.006 sec
Finished dirty card rescan work in 1th thread: 0.027 sec
Finished dirty card rescan work in 6th thread: 0.027 sec
Finished dirty card rescan work in 7th thread: 0.027 sec
Finished work stealing in 5th thread: 0.000 sec
Finished work stealing in 6th thread: 0.000 sec
Finished work stealing in 1th thread: 0.000 sec
Finished work stealing in 3th thread: 0.000 sec
Finished work stealing in 2th thread: 0.000 sec
Finished work stealing in 7th thread: 0.000 sec
Finished work stealing in 0th thread: 0.000 sec
Finished work stealing in 4th thread: 0.000 sec
, 0.1443962 secs]1141.343: [weak refs processing, 0.0040867 secs] [1
CMS-remark: 10846877K(19968000K)] 11307168K(20479552K), 0.1485643 secs]
1141.349: [CMS-concurrent-sweep-start]
1141.790: [GC 1141.790: [ParNew: 511104K->0K(511552K), 0.1640300 secs]
11350286K->10912612K(20479552K), 0.1641182 secs]
1146.214: [GC 1146.214: [ParNew: 511104K->0K(511552K), 0.1881758 secs]
11374028K->10943490K(20479552K), 0.1882665 secs]
1152.290: [CMS-concurrent-sweep: 10.588/10.941 secs]
 (CMS-concurrent-sweep yielded 2 times)
1152.290: [CMS-concurrent-reset-start]
1152.344: [CMS-concurrent-reset: 0.054/0.054 secs]
 (CMS-concurrent-reset yielded 0 times)
1153.586: [GC 1153.586: [ParNew: 511104K->0K(511552K), 0.2787759 secs]
10861822K->10504126K(20479552K), 0.2788630 secs]
1153.865: [GC [1 CMS-initial-mark: 10504126K(19968000K)]
10504126K(20479552K), 0.0008140 secs]
1153.866: [CMS-concurrent-mark-start]
1160.401: [GC 1160.401: [ParNew: 511104K->0K(511552K), 0.3305136 secs]
11015230K->10677714K(20479552K), 0.3306071 secs]
1167.520: [GC 1167.520: [ParNew: 511104K->0K(511552K), 0.4205922 secs]
11188818K->10893058K(20479552K), 0.4206949 secs]
1174.871: [GC 1174.871: [ParNew: 511104K->0K(511552K), 0.3851338 secs]
11404162K->11049802K(20479552K), 0.3852324 secs]
1180.394: [GC 1180.394: [ParNew: 511104K->0K(511552K), 0.2326485 secs]
11560906K->11139682K(20479552K), 0.2327363 secs]
1184.720: [GC 1184.720: [ParNew: 511104K->0K(511552K), 0.1502694 secs]
11650786K->11193958K(20479552K), 0.1503868 secs]
1188.954: [GC 1188.954: [ParNew: 511104K->0K(511552K), 0.1607400 secs]
11705062K->11252329K(20479552K), 0.1608336 secs]
1192.032: [CMS-concurrent-mark: 36.484/38.166 secs]
 (CMS-concurrent-mark yielded 10 times)
1192.032: [CMS-concurrent-preclean-start]
1194.722: [GC 1194.722: [ParNew: 511104K->0K(511552K), 0.2618601 secs]
11763433K->11337152K(20479552K), 0.2620287 secs]
 (cardTable: 229158 cards, re-scanned 3738696 cards, 3 iterations)
1198.118: [CMS-concurrent-preclean: 5.553/6.085 secs]
 (CMS-concurrent-preclean yielded 0 times)
1198.118: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 24687 cards, re-scanned 24687 cards, 1 iterations)
 [1 iterations, 0 waits, 24687 cards)] 1198.247:
[CMS-concurrent-abortable-preclean: 0.122/0.129 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
1198.250: [GC[YG occupancy: 300820 K (511552 K)]1198.250: [Rescan (parallel)
Finished root rescan work in 1th thread: 0.076 sec
Finished root rescan work in 3th thread: 0.076 sec
Finished root rescan work in 0th thread: 0.088 sec
Finished root rescan work in 5th thread: 0.089 sec
Finished root rescan work in 2th thread: 0.089 sec
Finished root rescan work in 6th thread: 0.089 sec
Finished root rescan work in 4th thread: 0.089 sec
Finished root rescan work in 7th thread: 0.066 sec
Finished dirty card rescan work in 4th thread: 0.006 sec
Finished dirty card rescan work in 3th thread: 0.019 sec
Finished dirty card rescan work in 1th thread: 0.019 sec
Finished dirty card rescan work in 0th thread: 0.007 sec
Finished dirty card rescan work in 2th thread: 0.006 sec
Finished dirty card rescan work in 5th thread: 0.006 sec
Finished dirty card rescan work in 7th thread: 0.006 sec
Finished dirty card rescan work in 6th thread: 0.006 sec
Finished work stealing in 7th thread: 0.000 sec
Finished work stealing in 6th thread: 0.000 sec
Finished work stealing in 5th thread: 0.000 sec
Finished work stealing in 2th thread: 0.000 sec
Finished work stealing in 0th thread: 0.002 sec
Finished work stealing in 4th thread: 0.002 sec
Finished work stealing in 3th thread: 0.002 sec
Finished work stealing in 1th thread: 0.002 sec
, 0.0966873 secs]1198.347: [weak refs processing, 0.0042343 secs] [1
CMS-remark: 11337152K(19968000K)] 11637973K(20479552K), 0.1010335 secs]
1198.353: [CMS-concurrent-sweep-start]
1199.548: [GC 1199.548: [ParNew: 511104K->0K(511552K), 0.2133766 secs]
11825902K->11385875K(20479552K), 0.2134628 secs]
1201.212: [GC 1201.212: [ParNew: 511104K->0K(511552K), 0.0389662 secs]
11872072K->11375573K(20479552K), 0.0390651 secs]
1202.766: [GC 1202.766: [ParNew: 511104K->0K(511552K), 0.0383736 secs]
11869646K->11373164K(20479552K), 0.0384725 secs]
1204.258: [GC 1204.258: [ParNew: 511104K->0K(511552K), 0.0392435 secs]
11855272K->11358795K(20479552K), 0.0393489 secs]
1205.790: [GC 1205.790: [ParNew: 511104K->0K(511552K), 0.0409362 secs]
11856780K->11360305K(20479552K), 0.0410378 secs]
1207.261: [GC 1207.261: [ParNew: 511104K->0K(511552K), 0.0363010 secs]
11816129K->11319650K(20479552K), 0.0363947 secs]
1209.190: [GC 1209.190: [ParNew: 511104K->0K(511552K), 0.0608038 secs]
11310471K->10823161K(20479552K), 0.0608947 secs]
1209.417: [CMS-concurrent-sweep: 10.594/11.065 secs]
 (CMS-concurrent-sweep yielded 7 times)
1209.417: [CMS-concurrent-reset-start]
1209.469: [CMS-concurrent-reset: 0.051/0.051 secs]
 (CMS-concurrent-reset yielded 0 times)
1211.471: [GC [1 CMS-initial-mark: 10741434K(19968000K)]
11058502K(20479552K), 0.5295450 secs]
1212.000: [CMS-concurrent-mark-start]
1213.374: [GC 1213.375: [ParNew: 511104K->0K(511552K), 0.1264412 secs]
11252538K->10800504K(20479552K), 0.1265427 secs]
1217.131: [GC 1217.131: [ParNew: 511104K->0K(511552K), 0.1285049 secs]
11311608K->10859907K(20479552K), 0.1286073 secs]
1222.695: [GC 1222.695: [ParNew: 511104K->0K(511552K), 0.1977017 secs]
11371011K->10974506K(20479552K), 0.1977985 secs]
1227.208: [GC 1227.208: [ParNew: 511104K->0K(511552K), 0.1568749 secs]
11485610K->11046953K(20479552K), 0.1569797 secs]
1231.474: [GC 1231.474: [ParNew: 511104K->0K(511552K), 0.1612471 secs]
11558057K->11118711K(20479552K), 0.1613706 secs]
1237.270: [GC 1237.270: [ParNew: 511104K->0K(511552K), 0.2288151 secs]
11629815K->11231320K(20479552K), 0.2289008 secs]
1244.273: [GC 1244.273: [ParNew: 511104K->0K(511552K), 0.3091513 secs]
11742424K->11394813K(20479552K), 0.3092449 secs]
1250.646: [CMS-concurrent-mark: 37.334/38.646 secs]
 (CMS-concurrent-mark yielded 7 times)
1250.646: [CMS-concurrent-preclean-start]
1250.829: [GC 1250.829: [ParNew: 511104K->0K(511552K), 0.3680981 secs]
11905917K->11575129K(20479552K), 0.3681840 secs]
1256.071: [GC 1256.071: [ParNew: 511104K->0K(511552K), 0.8464974 secs]
12086233K->11819982K(20479552K), 0.8465890 secs]
 (cardTable: 167191 cards, re-scanned 3997841 cards, 3 iterations)
1258.547: [CMS-concurrent-preclean: 6.464/7.901 secs]
 (CMS-concurrent-preclean yielded 8 times)
1258.547: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 21786 cards, re-scanned 21786 cards, 1 iterations)
 (cardTable: 32607 cards, re-scanned 32607 cards, 1 iterations)
 (cardTable: 34088 cards, re-scanned 34088 cards, 1 iterations)
 (cardTable: 34139 cards, re-scanned 34139 cards, 1 iterations)
 (cardTable: 34206 cards, re-scanned 34206 cards, 1 iterations)
 (cardTable: 84823 cards, re-scanned 84823 cards, 1 iterations)
 (cardTable: 38864 cards, re-scanned 38864 cards, 1 iterations)
 CMS: abort preclean due to time  [7 iterations, 0 waits, 280513 cards)]
1259.596: [CMS-concurrent-abortable-preclean: 0.990/1.049 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
1259.599: [GC[YG occupancy: 208790 K (511552 K)]1259.599: [Rescan (parallel)
Finished root rescan work in 1th thread: 0.084 sec
Finished root rescan work in 3th thread: 0.085 sec
Finished root rescan work in 5th thread: 0.088 sec
Finished root rescan work in 7th thread: 0.088 sec
Finished root rescan work in 2th thread: 0.089 sec
Finished root rescan work in 6th thread: 0.089 sec
Finished root rescan work in 4th thread: 0.090 sec
Finished root rescan work in 0th thread: 0.090 sec
Finished dirty card rescan work in 2th thread: 0.011 sec
Finished dirty card rescan work in 5th thread: 0.011 sec
Finished dirty card rescan work in 4th thread: 0.010 sec
Finished dirty card rescan work in 6th thread: 0.010 sec
Finished dirty card rescan work in 1th thread: 0.015 sec
Finished dirty card rescan work in 0th thread: 0.009 sec
Finished dirty card rescan work in 3th thread: 0.014 sec
Finished dirty card rescan work in 7th thread: 0.011 sec
Finished work stealing in 2th thread: 0.000 sec
Finished work stealing in 7th thread: 0.000 sec
Finished work stealing in 4th thread: 0.000 sec
Finished work stealing in 0th thread: 0.000 sec
Finished work stealing in 5th thread: 0.000 sec
Finished work stealing in 6th thread: 0.000 sec
Finished work stealing in 1th thread: 0.000 sec
Finished work stealing in 3th thread: 0.000 sec
, 0.1014253 secs]1259.701: [weak refs processing, 0.0040730 secs] [1
CMS-remark: 11819982K(19968000K)] 12028772K(20479552K), 0.1055813 secs]
1259.707: [CMS-concurrent-sweep-start]
1265.795: [GC 1265.795: [ParNew: 511104K->0K(511552K), 0.6058385 secs]
12239405K->11928765K(20479552K), 0.6059309 secs]
1271.477: [CMS-concurrent-sweep: 11.165/11.771 secs]
 (CMS-concurrent-sweep yielded 1 times)
1271.477: [CMS-concurrent-reset-start]
1271.534: [CMS-concurrent-reset: 0.057/0.057 secs]
 (CMS-concurrent-reset yielded 0 times)
1273.536: [GC [1 CMS-initial-mark: 11295825K(19968000K)]
11555492K(20479552K), 0.3846378 secs]
1273.920: [CMS-concurrent-mark-start]
1279.373: [GC 1279.373: [ParNew: 511104K->0K(511552K), 0.1274697 secs]
11806929K->11360021K(20479552K), 0.1275599 secs]
1285.466: [GC 1285.466: [ParNew: 508382K->0K(511552K), 1.0100917 secs]
11868404K->11621483K(20479552K), 1.0101838 secs]
1297.511: [GC 1297.511: [ParNew: 511104K->0K(511552K), 0.3895762 secs]
12132587K->11754331K(20479552K), 0.3896735 secs]
1312.426: [GC 1312.426: [ParNew: 511104K->0K(511552K), 0.0244052 secs]
12265435K->11754590K(20479552K), 0.0244968 secs]
1315.683: [CMS-concurrent-mark: 40.208/41.763 secs]
 (CMS-concurrent-mark yielded 5 times)
1315.683: [CMS-concurrent-preclean-start]
1316.386: [GC 1316.386: [ParNew: 511104K->0K(511552K), 0.8812461 secs]
12265694K->12114100K(20479552K), 0.8813349 secs]
 (cardTable: 231985 cards, re-scanned 2814616 cards, 2 iterations)
1321.230: [CMS-concurrent-preclean: 4.550/5.546 secs]
 (CMS-concurrent-preclean yielded 1 times)
1321.230: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 62048 cards, re-scanned 62048 cards, 1 iterations)
 [1 iterations, 0 waits, 62048 cards)] 1321.382:
[CMS-concurrent-abortable-preclean: 0.146/0.152 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
1321.386: [GC[YG occupancy: 294014 K (511552 K)]1321.386: [Rescan (parallel)
Finished root rescan work in 1th thread: 0.086 sec
Finished root rescan work in 3th thread: 0.087 sec
Finished root rescan work in 4th thread: 0.092 sec
Finished root rescan work in 7th thread: 0.096 sec
Finished root rescan work in 6th thread: 0.096 sec
Finished root rescan work in 2th thread: 0.102 sec
Finished dirty card rescan work in 1th thread: 0.023 sec
Finished dirty card rescan work in 4th thread: 0.018 sec
Finished dirty card rescan work in 6th thread: 0.013 sec
Finished dirty card rescan work in 7th thread: 0.014 sec
Finished dirty card rescan work in 3th thread: 0.023 sec
Finished dirty card rescan work in 2th thread: 0.008 sec
Finished root rescan work in 0th thread: 0.139 sec
Finished dirty card rescan work in 0th thread: 0.000 sec
Finished root rescan work in 5th thread: 0.148 sec
Finished dirty card rescan work in 5th thread: 0.000 sec
Finished work stealing in 5th thread: 0.000 sec
Finished work stealing in 3th thread: 0.039 sec
Finished work stealing in 0th thread: 0.009 sec
Finished work stealing in 1th thread: 0.039 sec
Finished work stealing in 6th thread: 0.039 sec
Finished work stealing in 4th thread: 0.039 sec
Finished work stealing in 7th thread: 0.039 sec
Finished work stealing in 2th thread: 0.039 sec
, 0.1502378 secs]1321.536: [weak refs processing, 0.0040858 secs] [1
CMS-remark: 12114100K(19968000K)] 12408114K(20479552K), 0.1544129 secs]
1321.542: [CMS-concurrent-sweep-start]
1326.498: [GC 1326.498: [ParNew: 511104K->0K(511552K), 0.6562131 secs]
12581551K->12263311K(20479552K), 0.6563043 secs]
1334.363: [CMS-concurrent-sweep: 12.163/12.821 secs]
 (CMS-concurrent-sweep yielded 2 times)
1334.363: [CMS-concurrent-reset-start]
1334.414: [CMS-concurrent-reset: 0.051/0.051 secs]
 (CMS-concurrent-reset yielded 0 times)
1336.416: [GC [1 CMS-initial-mark: 11355530K(19968000K)]
11702101K(20479552K), 0.5035177 secs]
1336.920: [CMS-concurrent-mark-start]
1341.208: [GC 1341.208: [ParNew: 511104K->0K(511552K), 0.0227975 secs]
11866634K->11355539K(20479552K), 0.0228953 secs]
1347.378: [GC 1347.378: [ParNew: 511104K->0K(511552K), 0.7659223 secs]
11866643K->11597058K(20479552K), 0.7660278 secs]
1357.649: [GC 1357.649: [ParNew: 511104K->0K(511552K), 0.4226757 secs]
12108162K->11762237K(20479552K), 0.4227831 secs]
1371.765: [GC 1371.765: [ParNew: 511104K->0K(511552K), 0.0244932 secs]
12273341K->11762244K(20479552K), 0.0245982 secs]
1377.338: [GC 1377.338: [ParNew: 511104K->0K(511552K), 0.6383523 secs]
12273348K->11959973K(20479552K), 0.6384535 secs]
1378.677: [CMS-concurrent-mark: 39.881/41.757 secs]
 (CMS-concurrent-mark yielded 120 times)
1378.677: [CMS-concurrent-preclean-start]
 (cardTable: 359647 cards, re-scanned 2169751 cards, 2 iterations)
1382.369: [CMS-concurrent-preclean: 3.594/3.692 secs]
 (CMS-concurrent-preclean yielded 0 times)
1382.369: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 100999 cards, re-scanned 100999 cards, 1 iterations)
 [1 iterations, 0 waits, 100999 cards)] 1382.610:
[CMS-concurrent-abortable-preclean: 0.220/0.241 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
1382.613: [GC[YG occupancy: 273024 K (511552 K)]1382.613: [Rescan (parallel)
Finished root rescan work in 3th thread: 0.130 sec
Finished root rescan work in 6th thread: 0.130 sec
Finished root rescan work in 1th thread: 0.130 sec
Finished root rescan work in 7th thread: 0.135 sec
Finished root rescan work in 2th thread: 0.136 sec
Finished root rescan work in 0th thread: 0.137 sec
Finished root rescan work in 5th thread: 0.141 sec
Finished root rescan work in 4th thread: 0.141 sec
Finished dirty card rescan work in 0th thread: 0.011 sec
Finished dirty card rescan work in 7th thread: 0.013 sec
Finished dirty card rescan work in 4th thread: 0.008 sec
Finished dirty card rescan work in 2th thread: 0.012 sec
Finished dirty card rescan work in 6th thread: 0.019 sec
Finished dirty card rescan work in 3th thread: 0.019 sec
Finished dirty card rescan work in 5th thread: 0.008 sec
Finished dirty card rescan work in 1th thread: 0.018 sec
Finished work stealing in 0th thread: 0.000 sec
Finished work stealing in 2th thread: 0.000 sec
Finished work stealing in 5th thread: 0.000 sec
Finished work stealing in 1th thread: 0.000 sec
Finished work stealing in 3th thread: 0.000 sec
Finished work stealing in 4th thread: 0.000 sec
Finished work stealing in 7th thread: 0.000 sec
Finished work stealing in 6th thread: 0.000 sec
, 0.1502418 secs]1382.763: [weak refs processing, 0.0041741 secs] [1
CMS-remark: 11959973K(19968000K)] 12232998K(20479552K), 0.1545105 secs]
1382.769: [CMS-concurrent-sweep-start]
1389.171: [GC 1389.171: [ParNew: 511104K->0K(511552K), 0.3098344 secs]
12460273K->12067950K(20479552K), 0.3099492 secs]
1395.537: [CMS-concurrent-sweep: 12.457/12.768 secs]
 (CMS-concurrent-sweep yielded 1 times)
1395.537: [CMS-concurrent-reset-start]
1395.588: [CMS-concurrent-reset: 0.051/0.051 secs]
 (CMS-concurrent-reset yielded 0 times)
1397.590: [GC [1 CMS-initial-mark: 11607445K(19968000K)]
11912509K(20479552K), 0.4413456 secs]
1398.031: [CMS-concurrent-mark-start]
1403.287: [GC 1403.287: [ParNew: 511104K->0K(511552K), 0.0235646 secs]
12118549K->11607708K(20479552K), 0.0236438 secs]
1408.445: [GC 1408.445: [ParNew: 511104K->0K(511552K), 0.6727795 secs]
12118812K->11822964K(20479552K), 0.6728882 secs]
1420.576: [GC 1420.576: [ParNew: 511104K->0K(511552K), 0.3059516 secs]
12334068K->11922555K(20479552K), 0.3060466 secs]
1433.576: [GC 1433.576: [ParNew: 511104K->0K(511552K), 0.0777266 secs]
12433659K->11942618K(20479552K), 0.0778274 secs]
1440.219: [CMS-concurrent-mark: 41.107/42.188 secs]
 (CMS-concurrent-mark yielded 9 times)
1440.219: [CMS-concurrent-preclean-start]
1442.756: [GC 1442.757: [ParNew: 511104K->0K(511552K), 0.5738822 secs]
12453722K->12102666K(20479552K), 0.5739725 secs]
 (cardTable: 3486 cards, re-scanned 1177512 cards, 2 iterations)
1443.392: [CMS-concurrent-preclean: 2.554/3.173 secs]
 (CMS-concurrent-preclean yielded 0 times)
1443.392: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 516 cards, re-scanned 568892 cards, 2 iterations)
 (cardTable: 4 cards, re-scanned 4 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 4 cards, re-scanned 4 cards, 1 iterations)
 CMS: abort preclean due to time  [4 iterations, 2 waits, 568900 cards)]
1444.425: [CMS-concurrent-abortable-preclean: 0.913/1.033 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
1444.429: [GC[YG occupancy: 39924 K (511552 K)]1444.429: [Rescan (parallel)
Finished root rescan work in 5th thread: 0.006 sec
Finished root rescan work in 1th thread: 0.007 sec
Finished root rescan work in 7th thread: 0.007 sec
Finished root rescan work in 6th thread: 0.008 sec
Finished root rescan work in 3th thread: 0.009 sec
Finished root rescan work in 4th thread: 0.011 sec
Finished root rescan work in 0th thread: 0.012 sec
Finished root rescan work in 2th thread: 0.012 sec
Finished dirty card rescan work in 6th thread: 0.009 sec
Finished dirty card rescan work in 4th thread: 0.005 sec
Finished dirty card rescan work in 3th thread: 0.007 sec
Finished dirty card rescan work in 1th thread: 0.010 sec
Finished dirty card rescan work in 5th thread: 0.010 sec
Finished dirty card rescan work in 7th thread: 0.010 sec
Finished dirty card rescan work in 0th thread: 0.005 sec
Finished dirty card rescan work in 2th thread: 0.005 sec
Finished work stealing in 6th thread: 0.000 sec
Finished work stealing in 0th thread: 0.000 sec
Finished work stealing in 5th thread: 0.000 sec
Finished work stealing in 7th thread: 0.000 sec
Finished work stealing in 4th thread: 0.000 sec
Finished work stealing in 2th thread: 0.000 sec
Finished work stealing in 3th thread: 0.000 sec
Finished work stealing in 1th thread: 0.000 sec
, 0.0184101 secs]1444.448: [weak refs processing, 0.0040122 secs] [1
CMS-remark: 12102666K(19968000K)] 12142591K(20479552K), 0.0225227 secs]
1444.453: [CMS-concurrent-sweep-start]
1450.848: [GC 1450.848: [ParNew: 511104K->0K(511552K), 0.3937120 secs]
12606510K->12235880K(20479552K), 0.3938206 secs]
1456.841: [CMS-concurrent-sweep: 11.993/12.387 secs]
 (CMS-concurrent-sweep yielded 14 times)
1456.841: [CMS-concurrent-reset-start]
1456.892: [CMS-concurrent-reset: 0.052/0.052 secs]
 (CMS-concurrent-reset yielded 0 times)
1458.894: [GC [1 CMS-initial-mark: 11955610K(19968000K)]
12359634K(20479552K), 0.8733672 secs]
1459.767: [CMS-concurrent-mark-start]
1462.704: [GC 1462.704: [ParNew: 511104K->0K(511552K), 0.3029162 secs]
12466714K->12075623K(20479552K), 0.3030067 secs]
1493.793: [CMS-concurrent-mark: 33.722/34.026 secs]
 (CMS-concurrent-mark yielded 77 times)
1493.793: [CMS-concurrent-preclean-start]
 (cardTable: 939 cards, re-scanned 476804 cards, 2 iterations)
1496.790: [CMS-concurrent-preclean: 2.966/2.997 secs]
 (CMS-concurrent-preclean yielded 0 times)
1496.790: [CMS-concurrent-abortable-preclean-start]
 (cardTable: 1 cards, re-scanned 1 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 (cardTable: 0 cards, re-scanned 0 cards, 1 iterations)
 CMS: abort preclean due to time  [10 iterations, 9 waits, 1 cards)]
1497.905: [CMS-concurrent-abortable-preclean: 0.662/1.115 secs]
 (CMS-concurrent-abortable-preclean yielded 0 times)
1497.908: [GC[YG occupancy: 112414 K (511552 K)]1497.908: [Rescan (parallel)
Finished root rescan work in 7th thread: 0.000 sec
Finished root rescan work in 5th thread: 0.000 sec
Finished root rescan work in 3th thread: 0.000 sec
Finished root rescan work in 4th thread: 0.000 sec
Finished root rescan work in 2th thread: 0.000 sec
Finished root rescan work in 1th thread: 0.000 sec
Finished root rescan work in 6th thread: 0.001 sec
Finished dirty card rescan work in 1th thread: 0.009 sec
Finished dirty card rescan work in 5th thread: 0.009 sec
Finished dirty card rescan work in 4th thread: 0.009 sec
Finished dirty card rescan work in 7th thread: 0.009 sec
Finished dirty card rescan work in 6th thread: 0.008 sec
Finished dirty card rescan work in 3th thread: 0.009 sec
Finished dirty card rescan work in 2th thread: 0.009 sec
Finished root rescan work in 0th thread: 0.146 sec
Finished dirty card rescan work in 0th thread: 0.000 sec


On Thu, Jan 15, 2009 at 12:44 PM, Y Srinivas Ramakrishna <
Y.S.Ramakrishna at sun.com> wrote:

>
> Can you run with -XX:PrintCMSStatistics=2 and send me the output?
>
> I suspect a performance badness related to code for
> manipulating overflow lists that we recently discovered
> and which is being fixed under:
>
> 6786503 Overflow list performance can be improved
> 6787254 Work queue capacity can be increased substantially on some
> platforms
>
> We can get you a fix verification binary
> through your support channels once it is established
> that work queue overflow is the issue here (and it
> sounds very much like it is).
>
> -- ramki
>
> ----- Original Message -----
> From: Hi Guava <higuava at gmail.com>
> Date: Thursday, January 15, 2009 9:27 am
> Subject: Extremely long parallel rescan
> To: hotspot-gc-use at openjdk.java.net
>
>
> > We have been having problem with CMS collector for a long time. The
> rescan
> > phase of CMS takes very very long time (20 minutes to several hours).
> > We
> > tried searching on Internet, asking in Sun's developer forum, etc. No
> > one
> > seems know what's going on.
> >
> > The typical hardware has 4 to 8 cpu/cores with 20 GB to 64GB of
> > memory. It
> > happens on both Windows and Linux version of 64-bit JVMs.
> > The GC parameters that we usually use are: -XX:+UseConcMarkSweepGC
> > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
> >
> > The following is an example of the problem.
> > Hardware: 2 dual-core CPUs (4 core in total) with 28 GB memory.
> > JVM: Java HotSpot(TM) 64-Bit Server VM Version 1.5.0_17-b04 from Sun
> > Microsystems Inc. on Windows XP Version 5.2 on amd64
> > Settings: -Xms20000m -Xmx20000m -XX:NewSize=500m
> > -XX:+CMSPermGenSweepingEnabled -XX:+DisableExplicitGC
> > -XX:CMSMarkStackSize=2M -XX:CMSMarkStackSizeMax=8M
> -XX:+UseConcMarkSweepGC
> > -XX:CMSInitiatingOccupancyFraction=30 -XX:+PrintGCDetails
> > -XX:+PrintGCTimeStamps
> > Note that we set many other GC parameters in this particular example.
> > Usually we only use -XX:+UseConcMarkSweepGC.
> >
> > 910.659: [GC [1 CMS-initial-mark: 6465020K(19968000K)]
> 6679872K(20479552K),
> > 0.3206026 secs]
> > 910.980: [CMS-concurrent-mark-start]
> > 919.131: [GC 919.131: [ParNew: 511104K->0K(511552K), 0.6801310 secs]
> > 6976124K->6657978K(20479552K), 0.6802201 secs]
> > 933.413: [CMS-concurrent-mark: 21.739/22.433 secs]
> > 933.413: [CMS-concurrent-preclean-start]
> > 934.813: [CMS-concurrent-preclean: 1.348/1.400 secs]
> > 934.813: [CMS-concurrent-abortable-preclean-start]
> >  CMS: abort preclean due to time 935.929:
> > [CMS-concurrent-abortable-preclean: 0.662/1.116 secs]
> > 935.933: [GC[YG occupancy: 443765 K (511552 K)]935.933: [Rescan
> (parallel)
> > , 3533.9840087 secs]4469.917: [weak refs processing, 0.0620489 secs] [1
> > CMS-remark: 6657978K(19968000K)] 7101744K(20479552K), 3534.0462074 secs]
> > 4469.980: [CMS-concurrent-sweep-start]
> > _______________________________________________
> > hotspot-gc-use mailing list
> > hotspot-gc-use at openjdk.java.net
> > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090116/2b680a69/attachment.html 

From xguan at cse.unl.edu  Thu Jan 29 09:25:54 2009
From: xguan at cse.unl.edu (Tony Guan)
Date: Thu, 29 Jan 2009 11:25:54 -0600
Subject: parallel adaptive gc
Message-ID: <2fcb552b0901290925o2d4b6909p9809e910eb3c0186@mail.gmail.com>

Hi,
I just reviewed some of the code of the class PSAdaptiveSizePolicy.
There is one thing I don't understand, why the mutator cost is
caculated 2 times: one is measured from the end of the minor
collection till the beginning of the next minor collection:
(_latest_minor_mutator_interval_seconds=_minor_timer.seconds), and the
other is measured from the end of a major collection till the
beginning of the next major collection.(
_latest_major_mutator_interval_seconds = _major_timer.seconds())

And the cost of minor collection is calculated as (this minor gc
time)/(this minor gc time+last MINOR mutator time)
the cost of major collection is :  (this major gc time)/(this major gc
time+last MAJOR mutator time)

But in fact, the two mutator time can be overlapping, since we don't
stop minor timer even when we are doing the major timer, and vice
versa.
So we end up with such possible counting:
time1                              time2
time3
minor gc stops               minor gc starts            minor gc stops
minor_mutator_starts     minor_mutator_ends    minor_mutator_starts

time4                              time5                            time6
major gc starts                major gc stops             minor gc starts
major mutator ends         major mutator starts     minor mutator ends

time 7
minor gc ends
minor mutator starts.

If we are at time 7, and want to caculate minor gc cost, then
last_minor_mutator is (time6-time3), and minor gc time is
(time7-time6)
but here (time6-time3)=
(time4-time3)+(time5-time4)+(time6-time5)=mutator time + full gc
time+mutator time.

based on such cost calculation, I really doubt whether we could be
precise enough to do any prediction?

Or that's maybe just my incomplete understanding of it, can somebody
help me out?

Thanks!


Tony


From Jon.Masamitsu at Sun.COM  Thu Jan 29 10:44:31 2009
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Thu, 29 Jan 2009 10:44:31 -0800
Subject: parallel adaptive gc
In-Reply-To: <2fcb552b0901290925o2d4b6909p9809e910eb3c0186@mail.gmail.com>
References: <2fcb552b0901290925o2d4b6909p9809e910eb3c0186@mail.gmail.com>
Message-ID: <4981F90F.2020401@sun.com>

Tony,

The definition of GC cost used in the adaptive size policy
work is the fraction

      GC-pause-time  /  time-interval

where time-interval is the time from the start of
of one GC to the start of the next GC.  The time-interval
is calculated as

    time-interval = GC-pause-time + time-between-GC

where time-between-GC is time from the end of a GC
to the start of the next GC.  

Each GC cost is calculated separately (minor GC cost
and major GC cost).  The intent is that the
time-between-GC includes all the times (mutator
times and times for other types of GC's).

Does that make sense?

I can't say why the GC cost is modeled as this fraction.
Before the current adaptive size policy there was
an implementation in 1.4.2 that did the sizing of the
generations and the GC cost model may have come
from that previous implementation.

Jon

Tony Guan wrote On 01/29/09 09:25,:

>Hi,
>I just reviewed some of the code of the class PSAdaptiveSizePolicy.
>There is one thing I don't understand, why the mutator cost is
>caculated 2 times: one is measured from the end of the minor
>collection till the beginning of the next minor collection:
>(_latest_minor_mutator_interval_seconds=_minor_timer.seconds), and the
>other is measured from the end of a major collection till the
>beginning of the next major collection.(
>_latest_major_mutator_interval_seconds = _major_timer.seconds())
>
>And the cost of minor collection is calculated as (this minor gc
>time)/(this minor gc time+last MINOR mutator time)
>the cost of major collection is :  (this major gc time)/(this major gc
>time+last MAJOR mutator time)
>
>But in fact, the two mutator time can be overlapping, since we don't
>stop minor timer even when we are doing the major timer, and vice
>versa.
>So we end up with such possible counting:
>time1                              time2
>time3
>minor gc stops               minor gc starts            minor gc stops
>minor_mutator_starts     minor_mutator_ends    minor_mutator_starts
>
>time4                              time5                            time6
>major gc starts                major gc stops             minor gc starts
>major mutator ends         major mutator starts     minor mutator ends
>
>time 7
>minor gc ends
>minor mutator starts.
>
>If we are at time 7, and want to caculate minor gc cost, then
>last_minor_mutator is (time6-time3), and minor gc time is
>(time7-time6)
>but here (time6-time3)=
>(time4-time3)+(time5-time4)+(time6-time5)=mutator time + full gc
>time+mutator time.
>
>based on such cost calculation, I really doubt whether we could be
>precise enough to do any prediction?
>
>Or that's maybe just my incomplete understanding of it, can somebody
>help me out?
>
>Thanks!
>
>
>Tony
>_______________________________________________
>hotspot-gc-use mailing list
>hotspot-gc-use at openjdk.java.net
>http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>  
>



From xguan at cse.unl.edu  Thu Jan 29 11:59:51 2009
From: xguan at cse.unl.edu (Tony Guan)
Date: Thu, 29 Jan 2009 13:59:51 -0600
Subject: parallel adaptive gc
In-Reply-To: <4981F90F.2020401@sun.com>
References: <2fcb552b0901290925o2d4b6909p9809e910eb3c0186@mail.gmail.com>
	<4981F90F.2020401@sun.com>
Message-ID: <2fcb552b0901291159u6880b626q7320588e49f97660@mail.gmail.com>

Hi Jon,

Thank you for the information.
I am OK with the for the cost part, but just don't understand the
time-interval part.
the time-interval should be (last mutator time)+(gc pause)
but in the implementation, it turns to be (last this type gc ends to
this time this type gc starts) which may include 1 or more mutator
times and 1 or more other type gc pause.

is this valid?


On Thu, Jan 29, 2009 at 12:44 PM, Jon Masamitsu <Jon.Masamitsu at sun.com> wrote:
> Tony,
>
> The definition of GC cost used in the adaptive size policy
> work is the fraction
>
>      GC-pause-time  /  time-interval
>
> where time-interval is the time from the start of
> of one GC to the start of the next GC.  The time-interval
> is calculated as
>
>    time-interval = GC-pause-time + time-between-GC
>
> where time-between-GC is time from the end of a GC
> to the start of the next GC.
>
> Each GC cost is calculated separately (minor GC cost
> and major GC cost).  The intent is that the
> time-between-GC includes all the times (mutator
> times and times for other types of GC's).
>
> Does that make sense?
>
> I can't say why the GC cost is modeled as this fraction.
> Before the current adaptive size policy there was
> an implementation in 1.4.2 that did the sizing of the
> generations and the GC cost model may have come
> from that previous implementation.
>
> Jon
>
> Tony Guan wrote On 01/29/09 09:25,:
>
>>Hi,
>>I just reviewed some of the code of the class PSAdaptiveSizePolicy.
>>There is one thing I don't understand, why the mutator cost is
>>caculated 2 times: one is measured from the end of the minor
>>collection till the beginning of the next minor collection:
>>(_latest_minor_mutator_interval_seconds=_minor_timer.seconds), and the
>>other is measured from the end of a major collection till the
>>beginning of the next major collection.(
>>_latest_major_mutator_interval_seconds = _major_timer.seconds())
>>
>>And the cost of minor collection is calculated as (this minor gc
>>time)/(this minor gc time+last MINOR mutator time)
>>the cost of major collection is :  (this major gc time)/(this major gc
>>time+last MAJOR mutator time)
>>
>>But in fact, the two mutator time can be overlapping, since we don't
>>stop minor timer even when we are doing the major timer, and vice
>>versa.
>>So we end up with such possible counting:
>>time1                              time2
>>time3
>>minor gc stops               minor gc starts            minor gc stops
>>minor_mutator_starts     minor_mutator_ends    minor_mutator_starts
>>
>>time4                              time5                            time6
>>major gc starts                major gc stops             minor gc starts
>>major mutator ends         major mutator starts     minor mutator ends
>>
>>time 7
>>minor gc ends
>>minor mutator starts.
>>
>>If we are at time 7, and want to caculate minor gc cost, then
>>last_minor_mutator is (time6-time3), and minor gc time is
>>(time7-time6)
>>but here (time6-time3)=
>>(time4-time3)+(time5-time4)+(time6-time5)=mutator time + full gc
>>time+mutator time.
>>
>>based on such cost calculation, I really doubt whether we could be
>>precise enough to do any prediction?
>>
>>Or that's maybe just my incomplete understanding of it, can somebody
>>help me out?
>>
>>Thanks!
>>
>>
>>Tony
>>_______________________________________________
>>hotspot-gc-use mailing list
>>hotspot-gc-use at openjdk.java.net
>>http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>>
>
>


From xguan at cse.unl.edu  Thu Jan 29 13:23:14 2009
From: xguan at cse.unl.edu (Tony Guan)
Date: Thu, 29 Jan 2009 15:23:14 -0600
Subject: parallel adaptive gc
In-Reply-To: <2fcb552b0901291159u6880b626q7320588e49f97660@mail.gmail.com>
References: <2fcb552b0901290925o2d4b6909p9809e910eb3c0186@mail.gmail.com>
	<4981F90F.2020401@sun.com>
	<2fcb552b0901291159u6880b626q7320588e49f97660@mail.gmail.com>
Message-ID: <2fcb552b0901291323s46a8f924ydd6af3483f764e5d@mail.gmail.com>

Hi Jon,

I am coming clearer about this issue, by implementing this way, we are
only interested in the overall time proportion of each type of
collection, such that minor gc occupies 10% of the time, and major gc
occupies 5% of the time, and thus the mutator utilizes (1-10%-5%)=95%
of the whole execution.

This makes more sense to me now.

Thanks a lot!

Tony

On Thu, Jan 29, 2009 at 1:59 PM, Tony Guan <xguan at cse.unl.edu> wrote:
> Hi Jon,
>
> Thank you for the information.
> I am OK with the for the cost part, but just don't understand the
> time-interval part.
> the time-interval should be (last mutator time)+(gc pause)
> but in the implementation, it turns to be (last this type gc ends to
> this time this type gc starts) which may include 1 or more mutator
> times and 1 or more other type gc pause.
>
> is this valid?
>
>
> On Thu, Jan 29, 2009 at 12:44 PM, Jon Masamitsu <Jon.Masamitsu at sun.com> wrote:
>> Tony,
>>
>> The definition of GC cost used in the adaptive size policy
>> work is the fraction
>>
>>      GC-pause-time  /  time-interval
>>
>> where time-interval is the time from the start of
>> of one GC to the start of the next GC.  The time-interval
>> is calculated as
>>
>>    time-interval = GC-pause-time + time-between-GC
>>
>> where time-between-GC is time from the end of a GC
>> to the start of the next GC.
>>
>> Each GC cost is calculated separately (minor GC cost
>> and major GC cost).  The intent is that the
>> time-between-GC includes all the times (mutator
>> times and times for other types of GC's).
>>
>> Does that make sense?
>>
>> I can't say why the GC cost is modeled as this fraction.
>> Before the current adaptive size policy there was
>> an implementation in 1.4.2 that did the sizing of the
>> generations and the GC cost model may have come
>> from that previous implementation.
>>
>> Jon
>>
>> Tony Guan wrote On 01/29/09 09:25,:
>>
>>>Hi,
>>>I just reviewed some of the code of the class PSAdaptiveSizePolicy.
>>>There is one thing I don't understand, why the mutator cost is
>>>caculated 2 times: one is measured from the end of the minor
>>>collection till the beginning of the next minor collection:
>>>(_latest_minor_mutator_interval_seconds=_minor_timer.seconds), and the
>>>other is measured from the end of a major collection till the
>>>beginning of the next major collection.(
>>>_latest_major_mutator_interval_seconds = _major_timer.seconds())
>>>
>>>And the cost of minor collection is calculated as (this minor gc
>>>time)/(this minor gc time+last MINOR mutator time)
>>>the cost of major collection is :  (this major gc time)/(this major gc
>>>time+last MAJOR mutator time)
>>>
>>>But in fact, the two mutator time can be overlapping, since we don't
>>>stop minor timer even when we are doing the major timer, and vice
>>>versa.
>>>So we end up with such possible counting:
>>>time1                              time2
>>>time3
>>>minor gc stops               minor gc starts            minor gc stops
>>>minor_mutator_starts     minor_mutator_ends    minor_mutator_starts
>>>
>>>time4                              time5                            time6
>>>major gc starts                major gc stops             minor gc starts
>>>major mutator ends         major mutator starts     minor mutator ends
>>>
>>>time 7
>>>minor gc ends
>>>minor mutator starts.
>>>
>>>If we are at time 7, and want to caculate minor gc cost, then
>>>last_minor_mutator is (time6-time3), and minor gc time is
>>>(time7-time6)
>>>but here (time6-time3)=
>>>(time4-time3)+(time5-time4)+(time6-time5)=mutator time + full gc
>>>time+mutator time.
>>>
>>>based on such cost calculation, I really doubt whether we could be
>>>precise enough to do any prediction?
>>>
>>>Or that's maybe just my incomplete understanding of it, can somebody
>>>help me out?
>>>
>>>Thanks!
>>>
>>>
>>>Tony
>>>_______________________________________________
>>>hotspot-gc-use mailing list
>>>hotspot-gc-use at openjdk.java.net
>>>http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>
>>>
>>
>>
>