From java7 at AngelikaLanger.com  Thu Aug 19 02:52:12 2010
From: java7 at AngelikaLanger.com (Angelika Langer)
Date: Thu, 19 Aug 2010 11:52:12 +0200
Subject: negative pause times
Message-ID: <4C6CFECC.7050002@AngelikaLanger.com>

Hi!

Does anybody know off hand why the G1 log file shows negative pause times?


222.121: [GC pause (young) 561M->528M(1005M), 0.0586070 secs]
222.915: [GC pause (young) 576M->540M(1005M), 0.0447820 secs]
223.763: [GC pause (young) 602M->562M(1005M), -0.7354870 secs]  <=======
225.572: [GC pause (young) 772M->632M(1005M), 0.0719350 secs]


Thanks
Angelika Langer



From y.s.ramakrishna at oracle.com  Mon Aug 30 11:28:40 2010
From: y.s.ramakrishna at oracle.com (Y. Srinivas Ramakrishna)
Date: Mon, 30 Aug 2010 11:28:40 -0700
Subject: negative pause times
In-Reply-To: <4C6CFECC.7050002@AngelikaLanger.com>
References: <4C6CFECC.7050002@AngelikaLanger.com>
Message-ID: <4C7BF858.5000105@oracle.com>

Angelika Langer wrote:
> Hi!
> 
> Does anybody know off hand why the G1 log file shows negative pause times?
> 
> 
> 222.121: [GC pause (young) 561M->528M(1005M), 0.0586070 secs]
> 222.915: [GC pause (young) 576M->540M(1005M), 0.0447820 secs]
> 223.763: [GC pause (young) 602M->562M(1005M), -0.7354870 secs]  <=======
> 225.572: [GC pause (young) 772M->632M(1005M), 0.0719350 secs]
> 

Is this on Linux? Are you running NTP or otherwise causing TOD to
be adjusted? (you might want to turn off NTP if the negative times
bother you.) Are you running on VMware? (try a non-virtualized
run to see if the problem reproduces.) The negative pause times
we have seen in the past have usually been related to one of the
other of the above.

Please file a formal bug using your support credentials should
you believe this is an issue that needs delving deeper, especially
if you have a usable test case.

thanks.
-- ramki


> 
> Thanks
> Angelika Langer
> 
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From noam_kat at hotmail.com  Tue Aug 31 08:22:20 2010
From: noam_kat at hotmail.com (noam katzowicz)
Date: Tue, 31 Aug 2010 15:22:20 +0000
Subject: Long GC pause time
Message-ID: <SNT108-W11C7575D28055DFA3A6544E38A0@phx.gbl>


Hello.
 
We are running JBoss AS over RH5, using java 6u21, with the following GC params:
 
-Xms6750m -Xmx6750m 
-XX:+UseConcMarkSweepGC 
-XX:SurvivorRatio=128 
-XX:MaxTenuringThreshold=0 
-XX:+UseCompressedOops 
-XX:CMSInitiatingOccupancyFraction=60 
-Dsun.rmi.dgc.client.gcInterval=3600000 
-Dsun.rmi.dgc.server.gcInterval=3600000 
-XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+PrintTenuringDistribution
 
We are experiencing relatively long GC pause time of over 1 second.
Our application allocates hundreds of MBs per second. We?ve tried different generation sizes but the above was the best we came up with. Your help is appreciated.
 

Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0596370 secs] 4422502K->4173928K(6909888K), 0.0606730 secs] [Times: user=0.44 sys=0.03, real=0.06 secs] 
89.486: [CMS-concurrent-preclean: 1.308/1.649 secs] [Times: user=10.32 sys=2.30, real=1.65 secs] 
89.487: [CMS-concurrent-abortable-preclean-start]
89.804: [GC 89.804: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0603670 secs] 4446568K->4198190K(6909888K), 0.0614310 secs] [Times: user=0.48 sys=0.03, real=0.07 secs] 
89.866: [CMS-concurrent-abortable-preclean: 0.293/0.379 secs] [Times: user=2.42 sys=0.54, real=0.38 secs] 
89.867: [GC[YG occupancy: 420 K (274752 K)]89.867: [Rescan (parallel) , 0.0242770 secs]89.892: [weak refs processing, 1.3860230 secs] [1 CMS-remark: 4198190K(6635136K)] 4198611K(6909888K), 1.4164670 secs] [Times: user=1.68 sys=0.01, real=1.41 secs] 
91.284: [CMS-concurrent-sweep-start]
91.590: [GC 91.591: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0675550 secs] 4371287K->4128966K(6909888K), 0.0683470 secs] [Times: user=0.56 sys=0.01, real=0.07 secs] 
91.860: [GC 91.860: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0604320 secs] 4308368K->4053829K(6909888K), 0.0612020 secs] [Times: user=0.41 sys=0.01, real=0.06 secs] 
92.786: [GC 92.787: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.1028730 secs] 3964957K->3753351K(6909888K), 0.1038490 secs] [Times: user=0.80 sys=0.02, real=0.10 secs] 
93.106: [GC 93.107: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0487000 secs] 3949085K->3699496K(6909888K), 0.0494470 secs] [Times: user=0.32 sys=0.01, real=0.05 secs] 
93.371: [GC 93.371: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0430620 secs] 3905862K->3652769K(6909888K), 0.0438800 secs] [Times: user=0.33 sys=0.01, real=0.05 secs] 
94.637: [GC 94.638: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0564930 secs] 3416325K->3164258K(6909888K), 0.0575000 secs] [Times: user=0.44 sys=0.02, real=0.06 secs] 
94.863: [CMS-concurrent-sweep: 3.107/3.578 secs] [Times: user=16.85 sys=3.23, real=3.58 secs] 
94.863: [CMS-concurrent-reset-start]
94.878: [CMS-concurrent-reset: 0.015/0.015 secs] [Times: user=0.03 sys=0.01, real=0.02 secs] 
95.981: [GC 95.981: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0514980 secs] 3405940K->3152334K(6909888K), 0.0524610 secs] [Times: user=0.44 sys=0.01, real=0.05 secs] 
100.179: [GC 100.180: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0676510 secs] 3424974K->3178150K(6909888K), 0.0686240 secs] [Times: user=0.58 sys=0.00, real=0.07 secs] 
100.909: [GC 100.909: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0707060 secs] 3450790K->3202638K(6909888K), 0.0717030 secs] [Times: user=0.55 sys=0.01, real=0.07 secs] 
101.613: [GC 101.613: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272636K->0K(274752K), 0.0667270 secs] 3475274K->3224061K(6909888K), 0.0678240 secs] [Times: user=0.50 sys=0.01, real=0.07 secs] 
102.072: [GC 102.073: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272635K->0K(274752K), 0.0590350 secs] 3496696K->3241246K(6909888K), 0.0601190 secs] [Times: user=0.44 sys=0.00, real=0.06 secs] 
102.756: [GC 102.756: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272613K->0K(274752K), 0.0630060 secs] 3513860K->3262602K(6909888K), 0.0640320 secs] [Times: user=0.49 sys=0.00, real=0.07 secs] 
103.452: [GC 103.453: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0533390 secs] 3535242K->3280184K(6909888K), 0.0543760 secs] [Times: user=0.41 sys=0.01, real=0.06 secs] 
104.462: [GC 104.462: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0729890 secs] 3552824K->3299003K(6909888K), 0.0740260 secs] [Times: user=0.48 sys=0.01, real=0.07 secs] 
106.274: [GC 106.274: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0609880 secs] 3571643K->3314370K(6909888K), 0.0619510 secs] [Times: user=0.41 sys=0.00, real=0.06 secs] 
108.154: [GC 108.155: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.1251050 secs] 3587010K->3343996K(6909888K), 0.1261630 secs] [Times: user=0.71 sys=0.02, real=0.13 secs] 
108.882: [GC 108.883: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0952060 secs] 3616636K->3369490K(6909888K), 0.0962930 secs] [Times: user=0.64 sys=0.01, real=0.10 secs] 
109.482: [GC 109.482: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272936K->0K(274752K), 0.0791270 secs] 3642426K->3392040K(6909888K), 0.0800830 secs] [Times: user=0.53 sys=0.01, real=0.08 secs] 
109.948: [GC 109.949: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 273038K->0K(274752K), 0.0835410 secs] 3665079K->3417813K(6909888K), 0.0845680 secs] [Times: user=0.57 sys=0.00, real=0.08 secs] 
110.035: [GC 110.035: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 64K->0K(274752K), 0.0088490 secs] 3417877K->3417828K(6909888K), 0.0093860 secs] [Times: user=0.09 sys=0.00, real=0.01 secs] 
110.484: [GC 110.485: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0880810 secs] 3690468K->3443727K(6909888K), 0.0890560 secs] [Times: user=0.59 sys=0.00, real=0.09 secs] 
110.910: [GC 110.910: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0860790 secs] 3716367K->3475451K(6909888K), 0.0870070 secs] [Times: user=0.58 sys=0.02, real=0.08 secs] 
111.276: [GC 111.277: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272639K->0K(274752K), 0.0772230 secs] 3748091K->3503686K(6909888K), 0.0782780 secs] [Times: user=0.47 sys=0.02, real=0.08 secs] 
111.658: [GC 111.659: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0754700 secs] 3776326K->3532682K(6909888K), 0.0763440 secs] [Times: user=0.54 sys=0.01, real=0.07 secs] 
112.068: [GC 112.068: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0705680 secs] 3805322K->3559619K(6909888K), 0.0714610 secs] [Times: user=0.47 sys=0.02, real=0.07 secs] 
112.497: [GC 112.498: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0779750 secs] 3832259K->3589461K(6909888K), 0.0788830 secs] [Times: user=0.50 sys=0.01, real=0.08 secs] 
112.911: [GC 112.911: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0728900 secs] 3862101K->3616744K(6909888K), 0.0738560 secs] [Times: user=0.52 sys=0.00, real=0.07 secs] 
113.362: [GC 113.363: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0609030 secs] 3889384K->3638477K(6909888K), 0.0619050 secs] [Times: user=0.44 sys=0.01, real=0.06 secs] 
113.829: [GC 113.829: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272614K->0K(274752K), 0.0641940 secs] 3911091K->3659305K(6909888K), 0.0653490 secs] [Times: user=0.48 sys=0.01, real=0.06 secs] 
114.311: [GC 114.311: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0618170 secs] 3931945K->3681619K(6909888K), 0.0626970 secs] [Times: user=0.50 sys=0.01, real=0.06 secs] 
114.841: [GC 114.842: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0670890 secs] 3954259K->3711036K(6909888K), 0.0680540 secs] [Times: user=0.52 sys=0.02, real=0.07 secs] 
115.356: [GC 115.357: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0623700 secs] 3983676K->3737945K(6909888K), 0.0633630 secs] [Times: user=0.49 sys=0.00, real=0.07 secs] 
115.829: [GC 115.830: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0665900 secs] 4010585K->3766876K(6909888K), 0.0676240 secs] [Times: user=0.51 sys=0.01, real=0.06 secs] 
116.271: [GC 116.271: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272630K->0K(274752K), 0.0594280 secs] 4039506K->3796861K(6909888K), 0.0603620 secs] [Times: user=0.46 sys=0.02, real=0.06 secs] 
116.716: [GC 116.717: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0724150 secs] 4069501K->3825191K(6909888K), 0.0733430 secs] [Times: user=0.52 sys=0.01, real=0.08 secs] 
117.145: [GC 117.146: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0664990 secs] 4097831K->3853057K(6909888K), 0.0675760 secs] [Times: user=0.48 sys=0.00, real=0.07 secs] 
117.543: [GC 117.545: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0650350 secs] 4125697K->3880562K(6909888K), 0.0673450 secs] [Times: user=0.49 sys=0.01, real=0.07 secs] 
117.956: [GC 117.956: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0664410 secs] 4153202K->3907935K(6909888K), 0.0673980 secs] [Times: user=0.49 sys=0.02, real=0.07 secs] 
118.385: [GC 118.386: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272631K->0K(274752K), 0.0655070 secs] 4180566K->3934925K(6909888K), 0.0664120 secs] [Times: user=0.50 sys=0.01, real=0.07 secs] 
118.826: [GC 118.827: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272608K->0K(274752K), 0.0605390 secs] 4207533K->3964211K(6909888K), 0.0616050 secs] [Times: user=0.50 sys=0.02, real=0.06 secs] 
119.233: [GC 119.234: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0656870 secs] 4236851K->3991065K(6909888K), 0.0666900 secs] [Times: user=0.50 sys=0.02, real=0.07 secs] 
119.301: [GC [1 CMS-initial-mark: 3991065K(6635136K)] 3991071K(6909888K), 0.0091650 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
119.311: [CMS-concurrent-mark-start]
119.699: [GC 119.699: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0647090 secs] 4263705K->4019961K(6909888K), 0.0656290 secs] [Times: user=0.53 sys=0.01, real=0.06 secs] 
120.081: [GC 120.082: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0641110 secs] 4292601K->4048334K(6909888K), 0.0649070 secs] [Times: user=0.50 sys=0.01, real=0.06 secs] 
120.538: [GC 120.539: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0677200 secs] 4320974K->4075491K(6909888K), 0.0687990 secs] [Times: user=0.49 sys=0.03, real=0.07 secs] 
120.621: [CMS-concurrent-mark: 1.076/1.310 secs] [Times: user=10.66 sys=1.73, real=1.31 secs] 
120.621: [CMS-concurrent-preclean-start]
120.942: [GC 120.943: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0736770 secs] 4348131K->4104969K(6909888K), 0.0747430 secs] [Times: user=0.51 sys=0.03, real=0.07 secs] 
121.337: [GC 121.338: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0569980 secs] 4377609K->4126623K(6909888K), 0.0579070 secs] [Times: user=0.48 sys=0.03, real=0.06 secs] 
121.766: [GC 121.767: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272615K->0K(274752K), 0.0590140 secs] 4399238K->4148770K(6909888K), 0.0599510 secs] [Times: user=0.44 sys=0.03, real=0.06 secs] 
122.174: [GC 122.175: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0531420 secs] 4421410K->4167895K(6909888K), 0.0540310 secs] [Times: user=0.45 sys=0.03, real=0.05 secs] 
122.257: [CMS-concurrent-preclean: 1.306/1.636 secs] [Times: user=10.25 sys=2.21, real=1.63 secs] 
122.257: [CMS-concurrent-abortable-preclean-start]
122.631: [GC 122.631: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0528220 secs] 4440535K->4187654K(6909888K), 0.0537160 secs] [Times: user=0.46 sys=0.02, real=0.05 secs] 
122.696: [CMS-concurrent-abortable-preclean: 0.355/0.439 secs] [Times: user=2.50 sys=0.52, real=0.44 secs] 
122.698: [GC[YG occupancy: 7874 K (274752 K)]122.698: [Rescan (parallel) , 0.0253430 secs]122.723: [weak refs processing, 1.1038100 secs] [1 CMS-remark: 4187654K(6635136K)] 4195529K(6909888K), 1.1357970 secs] [Times: user=1.42 sys=0.01, real=1.14 secs] 
123.834: [CMS-concurrent-sweep-start]
124.060: [GC 124.061: [ParNew
Desired survivor size 1081344 bytes, new threshold 0 (max 0)
: 272640K->0K(274752K), 0.0690300 secs] 4399395K->4167257K(6909888K), 0.0698010 secs] [Times: user=0.52 sys=0.02, real=0.07 secs] 
  		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100831/ffc5c073/attachment-0001.html 

From jon.masamitsu at oracle.com  Tue Aug 31 11:28:52 2010
From: jon.masamitsu at oracle.com (Jon Masamitsu)
Date: Tue, 31 Aug 2010 11:28:52 -0700
Subject: Long GC pause time
In-Reply-To: <SNT108-W11C7575D28055DFA3A6544E38A0@phx.gbl>
References: <SNT108-W11C7575D28055DFA3A6544E38A0@phx.gbl>
Message-ID: <4C7D49E4.6060908@oracle.com>

Try -XX+ParallelRefProcEnabled

Your spending much of the pause time doing Reference processing

89.867: [GC[YG occupancy: 420 K (274752 K)]89.867: [Rescan (parallel) , 
0.0242770 secs]89.892: [weak refs processing, 1.3860230 secs] [1 
CMS-remark: 4198190K(6635136K)] 4198611K(6909888K), 1.4164670 secs] 
[Times: user=1.68 sys=0.01, real=1.41 secs]

(see the "[weak refs processing, 1.3860230 secs]") so doing it in 
parallel might
help.  ParallelRefProcEnabled isn't always a win so it's not "on" by 
default.

You might also try -XX:+CMSScavengeBeforeRemark.  That's a just a guess.
Sometimes doing a minor collection before the remark helps.

On 08/31/10 08:22, noam katzowicz wrote:
> Hello.
>  
> We are running JBoss AS over RH5, using java 6u21, with the following 
> GC params:
>  
> -Xms6750m -Xmx6750m
> -XX:+UseConcMarkSweepGC
> -XX:SurvivorRatio=128
> -XX:MaxTenuringThreshold=0
> -XX:+UseCompressedOops
> -XX:CMSInitiatingOccupancyFraction=60
> -Dsun.rmi.dgc.client.gcInterval=3600000
> -Dsun.rmi.dgc.server.gcInterval=3600000
> -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+PrintTenuringDistribution
>  
> We are experiencing relatively long GC pause time of over 1 second.
> Our application allocates hundreds of MBs per second. We?ve tried 
> different generation sizes but the above was the best we came up with. 
> Your help is appreciated.
>  
>
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0596370 secs] 4422502K->4173928K(6909888K), 
> 0.0606730 secs] [Times: user=0.44 sys=0.03, real=0.06 secs]
> 89.486: [CMS-concurrent-preclean: 1.308/1.649 secs] [Times: user=10.32 
> sys=2.30, real=1.65 secs]
> 89.487: [CMS-concurrent-abortable-preclean-start]
> 89.804: [GC 89.804: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0603670 secs] 4446568K->4198190K(6909888K), 
> 0.0614310 secs] [Times: user=0.48 sys=0.03, real=0.07 secs]
> 89.866: [CMS-concurrent-abortable-preclean: 0.293/0.379 secs] [Times: 
> user=2.42 sys=0.54, real=0.38 secs]
> 89.867: [GC[YG occupancy: 420 K (274752 K)]89.867: [Rescan (parallel) 
> , 0.0242770 secs]89.892: [weak refs processing, 1.3860230 secs] [1 
> CMS-remark: 4198190K(6635136K)] 4198611K(6909888K), 1.4164670 secs] 
> [Times: user=1.68 sys=0.01, real=1.41 secs]
> 91.284: [CMS-concurrent-sweep-start]
> 91.590: [GC 91.591: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0675550 secs] 4371287K->4128966K(6909888K), 
> 0.0683470 secs] [Times: user=0.56 sys=0.01, real=0.07 secs]
> 91.860: [GC 91.860: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0604320 secs] 4308368K->4053829K(6909888K), 
> 0.0612020 secs] [Times: user=0.41 sys=0.01, real=0.06 secs]
> 92.786: [GC 92.787: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.1028730 secs] 3964957K->3753351K(6909888K), 
> 0.1038490 secs] [Times: user=0.80 sys=0.02, real=0.10 secs]
> 93.106: [GC 93.107: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0487000 secs] 3949085K->3699496K(6909888K), 
> 0.0494470 secs] [Times: user=0.32 sys=0.01, real=0.05 secs]
> 93.371: [GC 93.371: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0430620 secs] 3905862K->3652769K(6909888K), 
> 0.0438800 secs] [Times: user=0.33 sys=0.01, real=0.05 secs]
> 94.637: [GC 94.638: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0564930 secs] 3416325K->3164258K(6909888K), 
> 0.0575000 secs] [Times: user=0.44 sys=0.02, real=0.06 secs]
> 94.863: [CMS-concurrent-sweep: 3.107/3.578 secs] [Times: user=16.85 
> sys=3.23, real=3.58 secs]
> 94.863: [CMS-concurrent-reset-start]
> 94.878: [CMS-concurrent-reset: 0.015/0.015 secs] [Times: user=0.03 
> sys=0.01, real=0.02 secs]
> 95.981: [GC 95.981: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0514980 secs] 3405940K->3152334K(6909888K), 
> 0.0524610 secs] [Times: user=0.44 sys=0.01, real=0.05 secs]
> 100.179: [GC 100.180: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0676510 secs] 3424974K->3178150K(6909888K), 
> 0.0686240 secs] [Times: user=0.58 sys=0.00, real=0.07 secs]
> 100.909: [GC 100.909: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0707060 secs] 3450790K->3202638K(6909888K), 
> 0.0717030 secs] [Times: user=0.55 sys=0.01, real=0.07 secs]
> 101.613: [GC 101.613: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272636K->0K(274752K), 0.0667270 secs] 3475274K->3224061K(6909888K), 
> 0.0678240 secs] [Times: user=0.50 sys=0.01, real=0.07 secs]
> 102.072: [GC 102.073: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272635K->0K(274752K), 0.0590350 secs] 3496696K->3241246K(6909888K), 
> 0.0601190 secs] [Times: user=0.44 sys=0.00, real=0.06 secs]
> 102.756: [GC 102.756: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272613K->0K(274752K), 0.0630060 secs] 3513860K->3262602K(6909888K), 
> 0.0640320 secs] [Times: user=0.49 sys=0.00, real=0.07 secs]
> 103.452: [GC 103.453: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0533390 secs] 3535242K->3280184K(6909888K), 
> 0.0543760 secs] [Times: user=0.41 sys=0.01, real=0.06 secs]
> 104.462: [GC 104.462: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0729890 secs] 3552824K->3299003K(6909888K), 
> 0.0740260 secs] [Times: user=0.48 sys=0.01, real=0.07 secs]
> 106.274: [GC 106.274: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0609880 secs] 3571643K->3314370K(6909888K), 
> 0.0619510 secs] [Times: user=0.41 sys=0.00, real=0.06 secs]
> 108.154: [GC 108.155: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.1251050 secs] 3587010K->3343996K(6909888K), 
> 0.1261630 secs] [Times: user=0.71 sys=0.02, real=0.13 secs]
> 108.882: [GC 108.883: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0952060 secs] 3616636K->3369490K(6909888K), 
> 0.0962930 secs] [Times: user=0.64 sys=0.01, real=0.10 secs]
> 109.482: [GC 109.482: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272936K->0K(274752K), 0.0791270 secs] 3642426K->3392040K(6909888K), 
> 0.0800830 secs] [Times: user=0.53 sys=0.01, real=0.08 secs]
> 109.948: [GC 109.949: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 273038K->0K(274752K), 0.0835410 secs] 3665079K->3417813K(6909888K), 
> 0.0845680 secs] [Times: user=0.57 sys=0.00, real=0.08 secs]
> 110.035: [GC 110.035: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 64K->0K(274752K), 0.0088490 secs] 3417877K->3417828K(6909888K), 
> 0.0093860 secs] [Times: user=0.09 sys=0.00, real=0.01 secs]
> 110.484: [GC 110.485: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0880810 secs] 3690468K->3443727K(6909888K), 
> 0.0890560 secs] [Times: user=0.59 sys=0.00, real=0.09 secs]
> 110.910: [GC 110.910: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0860790 secs] 3716367K->3475451K(6909888K), 
> 0.0870070 secs] [Times: user=0.58 sys=0.02, real=0.08 secs]
> 111.276: [GC 111.277: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272639K->0K(274752K), 0.0772230 secs] 3748091K->3503686K(6909888K), 
> 0.0782780 secs] [Times: user=0.47 sys=0.02, real=0.08 secs]
> 111.658: [GC 111.659: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0754700 secs] 3776326K->3532682K(6909888K), 
> 0.0763440 secs] [Times: user=0.54 sys=0.01, real=0.07 secs]
> 112.068: [GC 112.068: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0705680 secs] 3805322K->3559619K(6909888K), 
> 0.0714610 secs] [Times: user=0.47 sys=0.02, real=0.07 secs]
> 112.497: [GC 112.498: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0779750 secs] 3832259K->3589461K(6909888K), 
> 0.0788830 secs] [Times: user=0.50 sys=0.01, real=0.08 secs]
> 112.911: [GC 112.911: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0728900 secs] 3862101K->3616744K(6909888K), 
> 0.0738560 secs] [Times: user=0.52 sys=0.00, real=0.07 secs]
> 113.362: [GC 113.363: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0609030 secs] 3889384K->3638477K(6909888K), 
> 0.0619050 secs] [Times: user=0.44 sys=0.01, real=0.06 secs]
> 113.829: [GC 113.829: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272614K->0K(274752K), 0.0641940 secs] 3911091K->3659305K(6909888K), 
> 0.0653490 secs] [Times: user=0.48 sys=0.01, real=0.06 secs]
> 114.311: [GC 114.311: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0618170 secs] 3931945K->3681619K(6909888K), 
> 0.0626970 secs] [Times: user=0.50 sys=0.01, real=0.06 secs]
> 114.841: [GC 114.842: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0670890 secs] 3954259K->3711036K(6909888K), 
> 0.0680540 secs] [Times: user=0.52 sys=0.02, real=0.07 secs]
> 115.356: [GC 115.357: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0623700 secs] 3983676K->3737945K(6909888K), 
> 0.0633630 secs] [Times: user=0.49 sys=0.00, real=0.07 secs]
> 115.829: [GC 115.830: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0665900 secs] 4010585K->3766876K(6909888K), 
> 0.0676240 secs] [Times: user=0.51 sys=0.01, real=0.06 secs]
> 116.271: [GC 116.271: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272630K->0K(274752K), 0.0594280 secs] 4039506K->3796861K(6909888K), 
> 0.0603620 secs] [Times: user=0.46 sys=0.02, real=0.06 secs]
> 116.716: [GC 116.717: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0724150 secs] 4069501K->3825191K(6909888K), 
> 0.0733430 secs] [Times: user=0.52 sys=0.01, real=0.08 secs]
> 117.145: [GC 117.146: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0664990 secs] 4097831K->3853057K(6909888K), 
> 0.0675760 secs] [Times: user=0.48 sys=0.00, real=0.07 secs]
> 117.543: [GC 117.545: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0650350 secs] 4125697K->3880562K(6909888K), 
> 0.0673450 secs] [Times: user=0.49 sys=0.01, real=0.07 secs]
> 117.956: [GC 117.956: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0664410 secs] 4153202K->3907935K(6909888K), 
> 0.0673980 secs] [Times: user=0.49 sys=0.02, real=0.07 secs]
> 118.385: [GC 118.386: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272631K->0K(274752K), 0.0655070 secs] 4180566K->3934925K(6909888K), 
> 0.0664120 secs] [Times: user=0.50 sys=0.01, real=0.07 secs]
> 118.826: [GC 118.827: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272608K->0K(274752K), 0.0605390 secs] 4207533K->3964211K(6909888K), 
> 0.0616050 secs] [Times: user=0.50 sys=0.02, real=0.06 secs]
> 119.233: [GC 119.234: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0656870 secs] 4236851K->3991065K(6909888K), 
> 0.0666900 secs] [Times: user=0.50 sys=0.02, real=0.07 secs]
> 119.301: [GC [1 CMS-initial-mark: 3991065K(6635136K)] 
> 3991071K(6909888K), 0.0091650 secs] [Times: user=0.01 sys=0.00, 
> real=0.01 secs]
> 119.311: [CMS-concurrent-mark-start]
> 119.699: [GC 119.699: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0647090 secs] 4263705K->4019961K(6909888K), 
> 0.0656290 secs] [Times: user=0.53 sys=0.01, real=0.06 secs]
> 120.081: [GC 120.082: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0641110 secs] 4292601K->4048334K(6909888K), 
> 0.0649070 secs] [Times: user=0.50 sys=0.01, real=0.06 secs]
> 120.538: [GC 120.539: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0677200 secs] 4320974K->4075491K(6909888K), 
> 0.0687990 secs] [Times: user=0.49 sys=0.03, real=0.07 secs]
> 120.621: [CMS-concurrent-mark: 1.076/1.310 secs] [Times: user=10.66 
> sys=1.73, real=1.31 secs]
> 120.621: [CMS-concurrent-preclean-start]
> 120.942: [GC 120.943: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0736770 secs] 4348131K->4104969K(6909888K), 
> 0.0747430 secs] [Times: user=0.51 sys=0.03, real=0.07 secs]
> 121.337: [GC 121.338: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0569980 secs] 4377609K->4126623K(6909888K), 
> 0.0579070 secs] [Times: user=0.48 sys=0.03, real=0.06 secs]
> 121.766: [GC 121.767: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272615K->0K(274752K), 0.0590140 secs] 4399238K->4148770K(6909888K), 
> 0.0599510 secs] [Times: user=0.44 sys=0.03, real=0.06 secs]
> 122.174: [GC 122.175: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0531420 secs] 4421410K->4167895K(6909888K), 
> 0.0540310 secs] [Times: user=0.45 sys=0.03, real=0.05 secs]
> 122.257: [CMS-concurrent-preclean: 1.306/1.636 secs] [Times: 
> user=10.25 sys=2.21, real=1.63 secs]
> 122.257: [CMS-concurrent-abortable-preclean-start]
> 122.631: [GC 122.631: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0528220 secs] 4440535K->4187654K(6909888K), 
> 0.0537160 secs] [Times: user=0.46 sys=0.02, real=0.05 secs]
> 122.696: [CMS-concurrent-abortable-preclean: 0.355/0.439 secs] [Times: 
> user=2.50 sys=0.52, real=0.44 secs]
> 122.698: [GC[YG occupancy: 7874 K (274752 K)]122.698: [Rescan 
> (parallel) , 0.0253430 secs]122.723: [weak refs processing, 1.1038100 
> secs] [1 CMS-remark: 4187654K(6635136K)] 4195529K(6909888K), 1.1357970 
> secs] [Times: user=1.42 sys=0.01, real=1.14 secs]
> 123.834: [CMS-concurrent-sweep-start]
> 124.060: [GC 124.061: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0690300 secs] 4399395K->4167257K(6909888K), 
> 0.0698010 secs] [Times: user=0.52 sys=0.02, real=0.07 secs]
>  
> ------------------------------------------------------------------------
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100831/d6e7cd34/attachment-0001.html 

From adamh at basis.com  Tue Aug 31 12:36:53 2010
From: adamh at basis.com (Adam Hawthorne)
Date: Tue, 31 Aug 2010 15:36:53 -0400
Subject: iCMS question
Message-ID: <AANLkTi=D23ym9Nj7mv68oucB1wKfcUrh=b64A8N4Oqyu@mail.gmail.com>

Hi all,

I'm looking at an i-cms log, and I don't understand why I'm seeing what I'm
seeing.  Any help is appreciated.  At about 38718.035 in the log below,
icms_dc begins to report 100, and just after that I see several back-to-back
concurrent mode failures.  During that time, the ParNew collections are
reporting that they're doing no work.  This happens on and off for about 20
minutes in the log.  Obviously the collections are taking seconds, but I
don't see a Full GC every time.  What is it doing when it's not doing a full
GC?  Is the continual CMS collection preventing the young generation from
being collected (or perhaps collecting it simultaneously)?  When
icms_dc=100, does that imply the application is stopped?

There was some uninformed command line twiddling going on here, so the GC
parameters were:
-Xmx1152m
-Xms256m
-XX:NewRatio=4
-XX:MaxPermSize=128m
-XX:+UseParNewGC
-XX:-CMSParallelRemarkEnabled
-XX:+UseConcMarkSweepGC
-XX:+CMSPermGenSweepingEnabled
-XX:+UseCMSCompactAtFullCollection
-XX:CMSFullGCsBeforeCompaction=1
-XX:+CMSClassUnloadingEnabled
-XX:+CMSIncrementalMode
-XX:+CMSIncrementalPacing
-XX:CMSInitiatingOccupancyFraction=50
-XX:ParallelGCThreads=1
-XX:MaxTenuringThreshold=0
-XX:SurvivorRatio=1024
-XX:SoftRefLRUPolicyMSPerMB=1000
-XX:+DisableExplicitGC
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintGCDateStamps

I had them remove -XX:ParallelGCThreads=1, -XX:MaxTenuringThreshold=0,
and -XX:SurvivorRatio=1024 , thinking that since the duty cycle was
approaching 100%, and because the ParNew pause times were so low, we could
probably afford to spend more time doing young gc's instead of promoting.
 I'm thinking I should probably tell them to increase their -Xmx and
decrease their NewRatio as well.  Is there anything else?  I haven't done
much tuning of i-cms in general, but the standard i-cms safety options don't
seem like they would help here.

Here's the log excerpt:

2010-08-26T15:44:42.836+0200: 38680.510: [GC 38680.510: [ParNew:
235520K->0K(235712K), 0.0187551 secs] 1158126K->923156K(1179456K) icms_dc=41
, 0.0189305 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:44:43.179+0200: 38680.861: [GC [1 CMS-initial-mark:
923156K(943744K)] 996964K(1179456K), 0.0526219 secs] [Times: user=0.06
sys=0.00, real=0.06 secs]
2010-08-26T15:44:43.241+0200: 38680.914: [CMS-concurrent-mark-start]
2010-08-26T15:44:44.021+0200: 38681.695: [GC 38681.695: [ParNew:
235520K->0K(235712K), 0.0196054 secs] 1158676K->923705K(1179456K) icms_dc=41
, 0.0197785 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:44:45.160+0200: 38682.831: [GC 38682.831: [ParNew:
235520K->0K(235712K), 0.0199663 secs] 1159225K->924310K(1179456K) icms_dc=41
, 0.0201448 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2010-08-26T15:44:46.127+0200: 38683.797: [GC 38683.797: [ParNew:
235520K->0K(235712K), 0.0549963 secs] 1159830K->930710K(1179456K) icms_dc=56
, 0.0551800 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
2010-08-26T15:44:47.562+0200: 38685.246: [CMS-concurrent-mark: 1.903/4.332
secs] [Times: user=6.77 sys=0.25, real=4.32 secs]
2010-08-26T15:44:47.562+0200: 38685.246: [CMS-concurrent-preclean-start]
2010-08-26T15:44:47.828+0200: 38685.506: [GC 38685.507: [ParNew:
235520K->0K(235712K), 0.0180149 secs] 1166230K->931150K(1179456K) icms_dc=56
, 0.0182051 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2010-08-26T15:44:48.108+0200: 38685.790: [CMS-concurrent-preclean:
0.023/0.543 secs] [Times: user=0.55 sys=0.05, real=0.55 secs]
2010-08-26T15:44:48.108+0200: 38685.790:
[CMS-concurrent-abortable-preclean-start]
2010-08-26T15:44:48.483+0200: 38686.158: [CMS-concurrent-abortable-preclean:
0.040/0.368 secs] [Times: user=0.41 sys=0.02, real=0.38 secs]
2010-08-26T15:44:48.483+0200: 38686.159: [GC[YG occupancy: 136856 K (235712
K)]38686.159: [Rescan (non-parallel) 38686.159: [grey object rescan,
0.0041031 secs]38686.163: [root rescan, 0.1153851 secs], 0.1195876
secs]38686.279: [weak refs processing, 0.0012088 secs]38686.280: [class
unloading, 0.0143079 secs]38686.294: [scrub symbol & string tables,
0.0159790 secs] [1 CMS-remark: 931150K(943744K)] 1068006K(1179456K),
0.1563910 secs] [Times: user=0.16 sys=0.00, real=0.16 secs]
2010-08-26T15:44:48.639+0200: 38686.316: [CMS-concurrent-sweep-start]
2010-08-26T15:44:49.138+0200: 38686.808: [GC 38686.808: [ParNew:
235520K->0K(235712K), 0.0183463 secs] 1166540K->931466K(1179456K) icms_dc=56
, 0.0185256 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:44:49.637+0200: 38687.307: [CMS-concurrent-sweep: 0.470/0.992
secs] [Times: user=1.47 sys=0.02, real=1.00 secs]
2010-08-26T15:44:49.637+0200: 38687.307: [CMS-concurrent-reset-start]
2010-08-26T15:44:49.637+0200: 38687.313: [CMS-concurrent-reset: 0.005/0.005
secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-08-26T15:44:50.277+0200: 38687.948: [GC 38687.948: [ParNew:
235520K->0K(235712K), 0.0177987 secs] 1158207K->923110K(1179456K) icms_dc=42
, 0.0179840 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:44:50.604+0200: 38688.280: [GC [1 CMS-initial-mark:
923110K(943744K)] 992484K(1179456K), 0.0509414 secs] [Times: user=0.05
sys=0.00, real=0.05 secs]
2010-08-26T15:44:50.651+0200: 38688.331: [CMS-concurrent-mark-start]
2010-08-26T15:44:51.478+0200: 38689.157: [GC 38689.157: [ParNew:
235520K->0K(235712K), 0.0178124 secs] 1158630K->923567K(1179456K) icms_dc=42
, 0.0179821 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:44:52.632+0200: 38690.306: [GC 38690.306: [ParNew:
235520K->0K(235712K), 0.0180970 secs] 1159087K->924012K(1179456K) icms_dc=42
, 0.0182661 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:44:53.740+0200: 38691.422: [GC 38691.422: [ParNew:
235520K->0K(235712K), 0.0267182 secs] 1159532K->925696K(1179456K) icms_dc=42
, 0.0269020 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2010-08-26T15:44:54.520+0200: 38692.201: [CMS-concurrent-mark: 1.894/3.870
secs] [Times: user=5.74 sys=0.13, real=3.87 secs]
2010-08-26T15:44:54.520+0200: 38692.201: [CMS-concurrent-preclean-start]
2010-08-26T15:44:54.551+0200: 38692.222: [CMS-concurrent-preclean:
0.015/0.020 secs] [Times: user=0.05 sys=0.02, real=0.03 secs]
2010-08-26T15:44:54.551+0200: 38692.222:
[CMS-concurrent-abortable-preclean-start]
2010-08-26T15:44:54.894+0200: 38692.573: [GC 38692.573: [ParNew:
235520K->0K(235712K), 0.0186037 secs] 1161216K->926149K(1179456K) icms_dc=42
, 0.0187843 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:44:55.581+0200: 38693.250: [CMS-concurrent-abortable-preclean:
0.050/1.028 secs] [Times: user=1.05 sys=0.01, real=1.03 secs]
2010-08-26T15:44:55.581+0200: 38693.251: [GC[YG occupancy: 140686 K (235712
K)]38693.251: [Rescan (non-parallel) 38693.251: [grey object rescan,
0.0043953 secs]38693.255: [root rescan, 0.1209735 secs], 0.1254680
secs]38693.376: [weak refs processing, 0.0013433 secs]38693.378: [class
unloading, 0.0142815 secs]38693.392: [scrub symbol & string tables,
0.0160091 secs] [1 CMS-remark: 926149K(943744K)] 1066835K(1179456K),
0.1624119 secs] [Times: user=0.16 sys=0.00, real=0.16 secs]
2010-08-26T15:44:55.737+0200: 38693.414: [CMS-concurrent-sweep-start]
2010-08-26T15:44:56.205+0200: 38693.879: [GC 38693.880: [ParNew:
235520K->0K(235712K), 0.0188069 secs] 1161648K->926599K(1179456K) icms_dc=42
, 0.0189967 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:44:56.891+0200: 38694.563: [CMS-concurrent-sweep: 0.474/1.149
secs] [Times: user=1.64 sys=0.02, real=1.15 secs]
2010-08-26T15:44:56.891+0200: 38694.563: [CMS-concurrent-reset-start]
2010-08-26T15:44:56.891+0200: 38694.569: [CMS-concurrent-reset: 0.006/0.006
secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-08-26T15:44:57.343+0200: 38695.018: [GC 38695.018: [ParNew:
235520K->0K(235712K), 0.0187471 secs] 1156567K->921554K(1179456K) icms_dc=32
, 0.0189332 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:44:57.718+0200: 38695.402: [GC [1 CMS-initial-mark:
921554K(943744K)] 1001837K(1179456K), 0.0572313 secs] [Times: user=0.06
sys=0.00, real=0.06 secs]
2010-08-26T15:44:57.780+0200: 38695.460: [CMS-concurrent-mark-start]
2010-08-26T15:44:58.545+0200: 38696.224: [GC 38696.225: [ParNew:
235520K->0K(235712K), 0.0183404 secs] 1157074K->922009K(1179456K) icms_dc=32
, 0.0185097 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2010-08-26T15:45:00.167+0200: 38697.850: [GC 38697.850: [ParNew:
235520K->0K(235712K), 0.0196135 secs] 1157529K->922547K(1179456K) icms_dc=32
, 0.0197977 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2010-08-26T15:45:01.321+0200: 38698.992: [GC 38698.992: [ParNew:
235520K->0K(235712K), 0.0187905 secs] 1158067K->923023K(1179456K) icms_dc=32
, 0.0189818 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2010-08-26T15:45:02.476+0200: 38700.146: [GC 38700.146: [ParNew:
235520K->0K(235712K), 0.0180207 secs] 1158543K->923471K(1179456K) icms_dc=32
, 0.0181892 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2010-08-26T15:45:03.615+0200: 38701.284: [GC 38701.284: [ParNew:
235520K->0K(235712K), 0.0186408 secs] 1158991K->923976K(1179456K) icms_dc=32
, 0.0188287 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:45:04.051+0200: 38701.735: [CMS-concurrent-mark: 1.899/6.275
secs] [Times: user=8.16 sys=0.06, real=6.27 secs]
2010-08-26T15:45:04.051+0200: 38701.735: [CMS-concurrent-preclean-start]
2010-08-26T15:45:04.083+0200: 38701.756: [CMS-concurrent-preclean:
0.016/0.021 secs] [Times: user=0.05 sys=0.02, real=0.03 secs]
2010-08-26T15:45:04.083+0200: 38701.756:
[CMS-concurrent-abortable-preclean-start]
2010-08-26T15:45:04.753+0200: 38702.434: [GC 38702.435: [ParNew:
235520K->0K(235712K), 0.0184004 secs] 1159496K->924427K(1179456K) icms_dc=32
, 0.0185851 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:45:05.409+0200: 38703.087: [CMS-concurrent-abortable-preclean:
0.063/1.331 secs] [Times: user=1.39 sys=0.03, real=1.33 secs]
2010-08-26T15:45:05.409+0200: 38703.088: [GC[YG occupancy: 141725 K (235712
K)]38703.088: [Rescan (non-parallel) 38703.088: [grey object rescan,
0.0041240 secs]38703.093: [root rescan, 0.1200361 secs], 0.1242579
secs]38703.213: [weak refs processing, 0.0010203 secs]38703.214: [class
unloading, 0.0142923 secs]38703.228: [scrub symbol & string tables,
0.0170438 secs] [1 CMS-remark: 924427K(943744K)] 1066152K(1179456K),
0.1618864 secs] [Times: user=0.17 sys=0.00, real=0.17 secs]
2010-08-26T15:45:05.580+0200: 38703.250: [CMS-concurrent-sweep-start]
2010-08-26T15:45:06.017+0200: 38703.687: [GC 38703.687: [ParNew:
235520K->0K(235712K), 0.0425827 secs] 1159944K->929053K(1179456K) icms_dc=32
, 0.0427672 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2010-08-26T15:45:07.202+0200: 38704.873: [GC 38704.873: [ParNew:
235520K->0K(235712K), 0.0179344 secs] 1161562K->926484K(1179456K) icms_dc=32
, 0.0181045 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2010-08-26T15:45:07.608+0200: 38705.282: [CMS-concurrent-sweep: 0.472/2.032
secs] [Times: user=2.53 sys=0.08, real=2.03 secs]
2010-08-26T15:45:07.608+0200: 38705.282: [CMS-concurrent-reset-start]
2010-08-26T15:45:07.608+0200: 38705.288: [CMS-concurrent-reset: 0.005/0.005
secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-08-26T15:45:08.341+0200: 38706.012: [GC 38706.012: [ParNew:
235520K->0K(235712K), 0.0180667 secs] 1162004K->926949K(1179456K) icms_dc=24
, 0.0182417 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:45:08.747+0200: 38706.425: [GC [1 CMS-initial-mark:
926949K(943744K)] 1019208K(1179456K), 0.0677621 secs] [Times: user=0.06
sys=0.00, real=0.06 secs]
2010-08-26T15:45:08.809+0200: 38706.493: [CMS-concurrent-mark-start]
2010-08-26T15:45:09.480+0200: 38707.158: [GC 38707.159: [ParNew:
235520K->0K(235712K), 0.0477314 secs] 1162469K->932300K(1179456K) icms_dc=39
, 0.0479217 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
2010-08-26T15:45:10.463+0200: 38708.133: [GC 38708.133: [ParNew:
235520K->0K(235712K), 0.0371965 secs] 1167820K->936491K(1179456K) icms_dc=54
, 0.0373840 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2010-08-26T15:45:12.085+0200: 38709.770: [GC 38709.770: [ParNew:
235520K->0K(235712K), 0.0185757 secs] 1172011K->936953K(1179456K) icms_dc=69
, 0.0187606 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2010-08-26T15:45:13.130+0200: 38710.806: [GC 38710.806: [ParNew (promotion
failed): 235520K->235520K(235712K), 1.0952192 secs]38711.901:
[CMS2010-08-26T15:45:14.222+0200: 38711.903: [CMS-concurrent-mark:
1.934/5.410 secs] [Times: user=8.20 sys=0.25, real=5.41 secs]
 (concurrent mode failure): 943043K->933853K(943744K), 4.0906398 secs]
1172473K->933853K(1179456K), [CMS Perm : 97361K->97361K(131072K)] icms_dc=86
, 5.1861184 secs] [Times: user=5.18 sys=0.00, real=5.18 secs]
2010-08-26T15:45:18.372+0200: 38716.052: [GC [1 CMS-initial-mark:
933853K(943744K)] 950390K(1179456K), 0.0155216 secs] [Times: user=0.01
sys=0.00, real=0.02 secs]
2010-08-26T15:45:18.387+0200: 38716.068: [CMS-concurrent-mark-start]
2010-08-26T15:45:19.355+0200: 38717.035: [GC 38717.035: [ParNew:
235520K->0K(235712K), 0.0623713 secs] 1169373K->942749K(1179456K)
icms_dc=100 , 0.0625660 secs] [Times: user=0.06 sys=0.00, real=0.06 secs]
2010-08-26T15:45:20.509+0200: 38718.182: [CMS-concurrent-mark: 1.964/2.114
secs] [Times: user=4.57 sys=0.11, real=2.12 secs]
2010-08-26T15:45:20.509+0200: 38718.182: [CMS-concurrent-preclean-start]
2010-08-26T15:45:20.525+0200: 38718.202: [CMS-concurrent-preclean:
0.015/0.020 secs] [Times: user=0.02 sys=0.02, real=0.02 secs]
2010-08-26T15:45:20.525+0200: 38718.202:
[CMS-concurrent-abortable-preclean-start]
2010-08-26T15:45:20.540+0200: 38718.222: [GC 38718.222: [ParNew:
235520K->235520K(235712K), 0.0000592 secs]38718.222:
[CMS2010-08-26T15:45:20.540+0200: 38718.223:
[CMS-concurrent-abortable-preclean: 0.004/0.021 secs] [Times: user=0.02
sys=0.00, real=0.01 secs]
 (concurrent mode failure): 942749K->936038K(943744K), 4.0650806 secs]
1178269K->936038K(1179456K), [CMS Perm : 97371K->97371K(131072K)]
icms_dc=100 , 4.0653451 secs] [Times: user=4.07 sys=0.00, real=4.07 secs]
2010-08-26T15:45:24.612+0200: 38722.289: [GC [1 CMS-initial-mark:
936038K(943744K)] 936042K(1179456K), 0.0023357 secs] [Times: user=0.00
sys=0.00, real=0.00 secs]
2010-08-26T15:45:24.612+0200: 38722.291: [CMS-concurrent-mark-start]
2010-08-26T15:45:26.141+0200: 38723.817: [GC 38723.817: [ParNew:
235520K->235520K(235712K), 0.0000588 secs]38723.817:
[CMS2010-08-26T15:45:27.030+0200: 38724.709: [CMS-concurrent-mark:
1.935/2.418 secs] [Times: user=3.65 sys=0.19, real=2.42 secs]
 (concurrent mode failure): 936038K->942227K(943744K), 4.9718511 secs]
1171558K->942227K(1179456K), [CMS Perm : 97393K->97393K(131072K)]
icms_dc=100 , 4.9721115 secs] [Times: user=4.98 sys=0.00, real=4.98 secs]
2010-08-26T15:45:31.117+0200: 38728.790: [GC [1 CMS-initial-mark:
942227K(943744K)] 942270K(1179456K), 0.0025839 secs] [Times: user=0.00
sys=0.00, real=0.00 secs]
2010-08-26T15:45:31.117+0200: 38728.793: [CMS-concurrent-mark-start]
2010-08-26T15:45:32.084+0200: 38729.758: [GC 38729.759: [ParNew:
235520K->235520K(235712K), 0.0000623 secs]38729.759:
[CMS2010-08-26T15:45:33.051+0200: 38730.736: [CMS-concurrent-mark:
1.934/1.943 secs] [Times: user=3.28 sys=0.19, real=1.93 secs]
 (concurrent mode failure): 942227K->943743K(943744K), 4.9442127 secs]
1177747K->948024K(1179456K), [CMS Perm : 97396K->97396K(131072K)]
icms_dc=100 , 4.9446265 secs] [Times: user=4.91 sys=0.02, real=4.94 secs]
2010-08-26T15:45:37.029+0200: 38734.704: [GC [1 CMS-initial-mark:
943743K(943744K)] 948026K(1179456K), 0.0067733 secs] [Times: user=0.00
sys=0.00, real=0.00 secs]
2010-08-26T15:45:37.029+0200: 38734.711: [CMS-concurrent-mark-start]
2010-08-26T15:45:38.745+0200: 38736.419: [Full GC 38736.419:
[CMS2010-08-26T15:45:39.463+0200: 38737.138: [CMS-concurrent-mark:
1.952/2.427 secs] [Times: user=3.95 sys=0.16, real=2.43 secs]
 (concurrent mode failure): 943743K->943743K(943744K), 4.7856092 secs]
1179455K->949021K(1179456K), [CMS Perm : 97406K->97406K(131072K)]
icms_dc=100 , 4.7857927 secs] [Times: user=4.79 sys=0.00, real=4.79 secs]
2010-08-26T15:45:43.534+0200: 38741.206: [GC [1 CMS-initial-mark:
943743K(943744K)] 949287K(1179456K), 0.0080127 secs] [Times: user=0.00
sys=0.00, real=0.00 secs]
2010-08-26T15:45:43.534+0200: 38741.214: [CMS-concurrent-mark-start]
2010-08-26T15:45:44.455+0200: 38742.136: [Full GC 38742.136:
[CMS2010-08-26T15:45:45.484+0200: 38743.168: [CMS-concurrent-mark:
1.952/1.954 secs] [Times: user=3.26 sys=0.05, real=1.95 secs]
 (concurrent mode failure): 943743K->943743K(943744K), 4.9692415 secs]
1179440K->949759K(1179456K), [CMS Perm : 97406K->97406K(131072K)]
icms_dc=100 , 4.9694250 secs] [Times: user=4.96 sys=0.02, real=4.98 secs]
2010-08-26T15:45:49.883+0200: 38747.569: [GC [1 CMS-initial-mark:
943743K(943744K)] 953372K(1179456K), 0.0086197 secs] [Times: user=0.02
sys=0.00, real=0.02 secs]
2010-08-26T15:45:49.899+0200: 38747.578: [CMS-concurrent-mark-start]
2010-08-26T15:45:50.960+0200: 38748.634: [Full GC 38748.634:
[CMS2010-08-26T15:45:51.833+0200: 38749.515: [CMS-concurrent-mark:
1.927/1.938 secs] [Times: user=3.23 sys=0.11, real=1.93 secs]
 (concurrent mode failure): 943743K->937800K(943744K), 4.9818453 secs]
1179455K->937800K(1179456K), [CMS Perm : 97407K->97407K(131072K)]
icms_dc=100 , 4.9820296 secs] [Times: user=4.95 sys=0.02, real=4.98 secs]
2010-08-26T15:45:55.936+0200: 38753.617: [GC [1 CMS-initial-mark:
937800K(943744K)] 938528K(1179456K), 0.0024408 secs] [Times: user=0.00
sys=0.00, real=0.00 secs]
2010-08-26T15:45:55.936+0200: 38753.620: [CMS-concurrent-mark-start]
2010-08-26T15:45:56.950+0200: 38754.630: [GC 38754.631: [ParNew:
235520K->235520K(235712K), 0.0000597 secs]38754.631:
[CMS2010-08-26T15:45:57.995+0200: 38755.679: [CMS-concurrent-mark:
2.057/2.059 secs] [Times: user=3.29 sys=0.23, real=2.06 secs]
 (concurrent mode failure): 937800K->936173K(943744K), 5.1274815 secs]
1173320K->936173K(1179456K), [CMS Perm : 97407K->97407K(131072K)]
icms_dc=100 , 5.1277524 secs] [Times: user=5.13 sys=0.00, real=5.13 secs]
2010-08-26T15:46:02.535+0200: 38760.215: [GC [1 CMS-initial-mark:
936173K(943744K)] 936184K(1179456K), 0.0024186 secs] [Times: user=0.00
sys=0.00, real=0.00 secs]
2010-08-26T15:46:02.535+0200: 38760.218: [CMS-concurrent-mark-start]
2010-08-26T15:46:03.611+0200: 38761.290: [GC 38761.290: [ParNew (promotion
failed): 235520K->235520K(235712K), 0.1302956 secs]38761.420:
[CMS2010-08-26T15:46:04.609+0200: 38762.289: [CMS-concurrent-mark:
1.938/2.071 secs] [Times: user=3.54 sys=0.11, real=2.07 secs]
 (concurrent mode failure): 943229K->939271K(943744K), 4.9629545 secs]
1171693K->939271K(1179456K), [CMS Perm : 97407K->97407K(131072K)]
icms_dc=100 , 5.0934938 secs] [Times: user=5.09 sys=0.00, real=5.10 secs]

Thanks again for any help,

Adam

--
Adam Hawthorne
Software Engineer
BASIS International Ltd.
www.basis.com
+1.505.345.5232 Phone
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100831/16a5b52f/attachment-0001.html 

From jon.masamitsu at oracle.com  Tue Aug 31 13:54:35 2010
From: jon.masamitsu at oracle.com (Jon Masamitsu)
Date: Tue, 31 Aug 2010 13:54:35 -0700
Subject: iCMS question
In-Reply-To: <AANLkTi=D23ym9Nj7mv68oucB1wKfcUrh=b64A8N4Oqyu@mail.gmail.com>
References: <AANLkTi=D23ym9Nj7mv68oucB1wKfcUrh=b64A8N4Oqyu@mail.gmail.com>
Message-ID: <4C7D6C0B.9070103@oracle.com>

I think the short answer is that your heap is full.

This is a minor collection that turns into a full collection (so you
don't see the telltale "Full GC" message).

2010-08-26T15:45:13.130+0200: 38710.806: [GC 38710.806: [ParNew 
(promotion failed): 235520K->235520K(235712K), 1.0952192 secs]38711.901: 
[CMS2010-08-26T15:45:14.222+0200: 38711.903: [CMS-concurrent-mark: 
1.934/5.410 secs] [Times: user=8.20 sys=0.25, real=5.41 secs] 
 (concurrent mode failure): 943043K->933853K(943744K), 4.0906398 secs] 
1172473K->933853K(1179456K), [CMS Perm : 97361K->97361K(131072K)] 
icms_dc=86 , 5.1861184 secs] [Times: user=5.18 sys=0.00, real=5.18 secs]

After the full collection the heap is still mostly full.

943043K->933853K(943744K)

I think your advice to increase the size of the should will help.


On 08/31/10 12:36, Adam Hawthorne wrote:
> Hi all,
>
> I'm looking at an i-cms log, and I don't understand why I'm seeing 
> what I'm seeing.  Any help is appreciated.  At about  38718.035 in the 
> log below, icms_dc begins to report 100, and just after that I see 
> several back-to-back concurrent mode failures.  During that time, the 
> ParNew collections are reporting that they're doing no work.  This 
> happens on and off for about 20 minutes in the log.  Obviously the 
> collections are taking seconds, but I don't see a Full GC every time. 
>  What is it doing when it's not doing a full GC?  Is the continual CMS 
> collection preventing the young generation from being collected (or 
> perhaps collecting it simultaneously)?  When icms_dc=100, does that 
> imply the application is stopped?
>
> There was some uninformed command line twiddling going on here, so the 
> GC parameters were:
> -Xmx1152m
> -Xms256m
> -XX:NewRatio=4
> -XX:MaxPermSize=128m
> -XX:+UseParNewGC
> -XX:-CMSParallelRemarkEnabled
> -XX:+UseConcMarkSweepGC
> -XX:+CMSPermGenSweepingEnabled
> -XX:+UseCMSCompactAtFullCollection
> -XX:CMSFullGCsBeforeCompaction=1
> -XX:+CMSClassUnloadingEnabled
> -XX:+CMSIncrementalMode
> -XX:+CMSIncrementalPacing
> -XX:CMSInitiatingOccupancyFraction=50
> -XX:ParallelGCThreads=1
> -XX:MaxTenuringThreshold=0
> -XX:SurvivorRatio=1024
> -XX:SoftRefLRUPolicyMSPerMB=1000
> -XX:+DisableExplicitGC
> -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps
> -XX:+PrintGCDateStamps
>
> I had them remove -XX:ParallelGCThreads=1, -XX:MaxTenuringThreshold=0, 
> and -XX:SurvivorRatio=1024 , thinking that since the duty cycle was 
> approaching 100%, and because the ParNew pause times were so low, we 
> could probably afford to spend more time doing young gc's instead of 
> promoting.  I'm thinking I should probably tell them to increase their 
> -Xmx and decrease their NewRatio as well.  Is there anything else?  I 
> haven't done much tuning of i-cms in general, but the standard i-cms 
> safety options don't seem like they would help here.
>
> Here's the log excerpt:
>
> 2010-08-26T15:44:42.836+0200: 38680.510: [GC 38680.510: [ParNew: 
> 235520K->0K(235712K), 0.0187551 secs] 1158126K->923156K(1179456K) 
> icms_dc=41 , 0.0189305 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:43.179+0200: 38680.861: [GC [1 CMS-initial-mark: 
> 923156K(943744K)] 996964K(1179456K), 0.0526219 secs] [Times: user=0.06 
> sys=0.00, real=0.06 secs] 
> 2010-08-26T15:44:43.241+0200: 38680.914: [CMS-concurrent-mark-start]
> 2010-08-26T15:44:44.021+0200: 38681.695: [GC 38681.695: [ParNew: 
> 235520K->0K(235712K), 0.0196054 secs] 1158676K->923705K(1179456K) 
> icms_dc=41 , 0.0197785 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:45.160+0200: 38682.831: [GC 38682.831: [ParNew: 
> 235520K->0K(235712K), 0.0199663 secs] 1159225K->924310K(1179456K) 
> icms_dc=41 , 0.0201448 secs] [Times: user=0.01 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:46.127+0200: 38683.797: [GC 38683.797: [ParNew: 
> 235520K->0K(235712K), 0.0549963 secs] 1159830K->930710K(1179456K) 
> icms_dc=56 , 0.0551800 secs] [Times: user=0.05 sys=0.00, real=0.05 secs] 
> 2010-08-26T15:44:47.562+0200: 38685.246: [CMS-concurrent-mark: 
> 1.903/4.332 secs] [Times: user=6.77 sys=0.25, real=4.32 secs] 
> 2010-08-26T15:44:47.562+0200: 38685.246: [CMS-concurrent-preclean-start]
> 2010-08-26T15:44:47.828+0200: 38685.506: [GC 38685.507: [ParNew: 
> 235520K->0K(235712K), 0.0180149 secs] 1166230K->931150K(1179456K) 
> icms_dc=56 , 0.0182051 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
> 2010-08-26T15:44:48.108+0200: 38685.790: [CMS-concurrent-preclean: 
> 0.023/0.543 secs] [Times: user=0.55 sys=0.05, real=0.55 secs] 
> 2010-08-26T15:44:48.108+0200: 38685.790: 
> [CMS-concurrent-abortable-preclean-start]
> 2010-08-26T15:44:48.483+0200: 38686.158: 
> [CMS-concurrent-abortable-preclean: 0.040/0.368 secs] [Times: 
> user=0.41 sys=0.02, real=0.38 secs] 
> 2010-08-26T15:44:48.483+0200: 38686.159: [GC[YG occupancy: 136856 K 
> (235712 K)]38686.159: [Rescan (non-parallel) 38686.159: [grey object 
> rescan, 0.0041031 secs]38686.163: [root rescan, 0.1153851 secs], 
> 0.1195876 secs]38686.279: [weak refs processing, 0.0012088 
> secs]38686.280: [class unloading, 0.0143079 secs]38686.294: [scrub 
> symbol & string tables, 0.0159790 secs] [1 CMS-remark: 
> 931150K(943744K)] 1068006K(1179456K), 0.1563910 secs] [Times: 
> user=0.16 sys=0.00, real=0.16 secs] 
> 2010-08-26T15:44:48.639+0200: 38686.316: [CMS-concurrent-sweep-start]
> 2010-08-26T15:44:49.138+0200: 38686.808: [GC 38686.808: [ParNew: 
> 235520K->0K(235712K), 0.0183463 secs] 1166540K->931466K(1179456K) 
> icms_dc=56 , 0.0185256 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:49.637+0200: 38687.307: [CMS-concurrent-sweep: 
> 0.470/0.992 secs] [Times: user=1.47 sys=0.02, real=1.00 secs] 
> 2010-08-26T15:44:49.637+0200: 38687.307: [CMS-concurrent-reset-start]
> 2010-08-26T15:44:49.637+0200: 38687.313: [CMS-concurrent-reset: 
> 0.005/0.005 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
> 2010-08-26T15:44:50.277+0200: 38687.948: [GC 38687.948: [ParNew: 
> 235520K->0K(235712K), 0.0177987 secs] 1158207K->923110K(1179456K) 
> icms_dc=42 , 0.0179840 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:50.604+0200: 38688.280: [GC [1 CMS-initial-mark: 
> 923110K(943744K)] 992484K(1179456K), 0.0509414 secs] [Times: user=0.05 
> sys=0.00, real=0.05 secs] 
> 2010-08-26T15:44:50.651+0200: 38688.331: [CMS-concurrent-mark-start]
> 2010-08-26T15:44:51.478+0200: 38689.157: [GC 38689.157: [ParNew: 
> 235520K->0K(235712K), 0.0178124 secs] 1158630K->923567K(1179456K) 
> icms_dc=42 , 0.0179821 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:52.632+0200: 38690.306: [GC 38690.306: [ParNew: 
> 235520K->0K(235712K), 0.0180970 secs] 1159087K->924012K(1179456K) 
> icms_dc=42 , 0.0182661 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:53.740+0200: 38691.422: [GC 38691.422: [ParNew: 
> 235520K->0K(235712K), 0.0267182 secs] 1159532K->925696K(1179456K) 
> icms_dc=42 , 0.0269020 secs] [Times: user=0.03 sys=0.00, real=0.03 secs] 
> 2010-08-26T15:44:54.520+0200: 38692.201: [CMS-concurrent-mark: 
> 1.894/3.870 secs] [Times: user=5.74 sys=0.13, real=3.87 secs] 
> 2010-08-26T15:44:54.520+0200: 38692.201: [CMS-concurrent-preclean-start]
> 2010-08-26T15:44:54.551+0200: 38692.222: [CMS-concurrent-preclean: 
> 0.015/0.020 secs] [Times: user=0.05 sys=0.02, real=0.03 secs] 
> 2010-08-26T15:44:54.551+0200: 38692.222: 
> [CMS-concurrent-abortable-preclean-start]
> 2010-08-26T15:44:54.894+0200: 38692.573: [GC 38692.573: [ParNew: 
> 235520K->0K(235712K), 0.0186037 secs] 1161216K->926149K(1179456K) 
> icms_dc=42 , 0.0187843 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:55.581+0200: 38693.250: 
> [CMS-concurrent-abortable-preclean: 0.050/1.028 secs] [Times: 
> user=1.05 sys=0.01, real=1.03 secs] 
> 2010-08-26T15:44:55.581+0200: 38693.251: [GC[YG occupancy: 140686 K 
> (235712 K)]38693.251: [Rescan (non-parallel) 38693.251: [grey object 
> rescan, 0.0043953 secs]38693.255: [root rescan, 0.1209735 secs], 
> 0.1254680 secs]38693.376: [weak refs processing, 0.0013433 
> secs]38693.378: [class unloading, 0.0142815 secs]38693.392: [scrub 
> symbol & string tables, 0.0160091 secs] [1 CMS-remark: 
> 926149K(943744K)] 1066835K(1179456K), 0.1624119 secs] [Times: 
> user=0.16 sys=0.00, real=0.16 secs] 
> 2010-08-26T15:44:55.737+0200: 38693.414: [CMS-concurrent-sweep-start]
> 2010-08-26T15:44:56.205+0200: 38693.879: [GC 38693.880: [ParNew: 
> 235520K->0K(235712K), 0.0188069 secs] 1161648K->926599K(1179456K) 
> icms_dc=42 , 0.0189967 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:56.891+0200: 38694.563: [CMS-concurrent-sweep: 
> 0.474/1.149 secs] [Times: user=1.64 sys=0.02, real=1.15 secs] 
> 2010-08-26T15:44:56.891+0200: 38694.563: [CMS-concurrent-reset-start]
> 2010-08-26T15:44:56.891+0200: 38694.569: [CMS-concurrent-reset: 
> 0.006/0.006 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
> 2010-08-26T15:44:57.343+0200: 38695.018: [GC 38695.018: [ParNew: 
> 235520K->0K(235712K), 0.0187471 secs] 1156567K->921554K(1179456K) 
> icms_dc=32 , 0.0189332 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:57.718+0200: 38695.402: [GC [1 CMS-initial-mark: 
> 921554K(943744K)] 1001837K(1179456K), 0.0572313 secs] [Times: 
> user=0.06 sys=0.00, real=0.06 secs] 
> 2010-08-26T15:44:57.780+0200: 38695.460: [CMS-concurrent-mark-start]
> 2010-08-26T15:44:58.545+0200: 38696.224: [GC 38696.225: [ParNew: 
> 235520K->0K(235712K), 0.0183404 secs] 1157074K->922009K(1179456K) 
> icms_dc=32 , 0.0185097 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
> 2010-08-26T15:45:00.167+0200: 38697.850: [GC 38697.850: [ParNew: 
> 235520K->0K(235712K), 0.0196135 secs] 1157529K->922547K(1179456K) 
> icms_dc=32 , 0.0197977 secs] [Times: user=0.03 sys=0.00, real=0.03 secs] 
> 2010-08-26T15:45:01.321+0200: 38698.992: [GC 38698.992: [ParNew: 
> 235520K->0K(235712K), 0.0187905 secs] 1158067K->923023K(1179456K) 
> icms_dc=32 , 0.0189818 secs] [Times: user=0.01 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:45:02.476+0200: 38700.146: [GC 38700.146: [ParNew: 
> 235520K->0K(235712K), 0.0180207 secs] 1158543K->923471K(1179456K) 
> icms_dc=32 , 0.0181892 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
> 2010-08-26T15:45:03.615+0200: 38701.284: [GC 38701.284: [ParNew: 
> 235520K->0K(235712K), 0.0186408 secs] 1158991K->923976K(1179456K) 
> icms_dc=32 , 0.0188287 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:45:04.051+0200: 38701.735: [CMS-concurrent-mark: 
> 1.899/6.275 secs] [Times: user=8.16 sys=0.06, real=6.27 secs] 
> 2010-08-26T15:45:04.051+0200: 38701.735: [CMS-concurrent-preclean-start]
> 2010-08-26T15:45:04.083+0200: 38701.756: [CMS-concurrent-preclean: 
> 0.016/0.021 secs] [Times: user=0.05 sys=0.02, real=0.03 secs] 
> 2010-08-26T15:45:04.083+0200: 38701.756: 
> [CMS-concurrent-abortable-preclean-start]
> 2010-08-26T15:45:04.753+0200: 38702.434: [GC 38702.435: [ParNew: 
> 235520K->0K(235712K), 0.0184004 secs] 1159496K->924427K(1179456K) 
> icms_dc=32 , 0.0185851 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:45:05.409+0200: 38703.087: 
> [CMS-concurrent-abortable-preclean: 0.063/1.331 secs] [Times: 
> user=1.39 sys=0.03, real=1.33 secs] 
> 2010-08-26T15:45:05.409+0200: 38703.088: [GC[YG occupancy: 141725 K 
> (235712 K)]38703.088: [Rescan (non-parallel) 38703.088: [grey object 
> rescan, 0.0041240 secs]38703.093: [root rescan, 0.1200361 secs], 
> 0.1242579 secs]38703.213: [weak refs processing, 0.0010203 
> secs]38703.214: [class unloading, 0.0142923 secs]38703.228: [scrub 
> symbol & string tables, 0.0170438 secs] [1 CMS-remark: 
> 924427K(943744K)] 1066152K(1179456K), 0.1618864 secs] [Times: 
> user=0.17 sys=0.00, real=0.17 secs] 
> 2010-08-26T15:45:05.580+0200: 38703.250: [CMS-concurrent-sweep-start]
> 2010-08-26T15:45:06.017+0200: 38703.687: [GC 38703.687: [ParNew: 
> 235520K->0K(235712K), 0.0425827 secs] 1159944K->929053K(1179456K) 
> icms_dc=32 , 0.0427672 secs] [Times: user=0.03 sys=0.00, real=0.03 secs] 
> 2010-08-26T15:45:07.202+0200: 38704.873: [GC 38704.873: [ParNew: 
> 235520K->0K(235712K), 0.0179344 secs] 1161562K->926484K(1179456K) 
> icms_dc=32 , 0.0181045 secs] [Times: user=0.01 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:45:07.608+0200: 38705.282: [CMS-concurrent-sweep: 
> 0.472/2.032 secs] [Times: user=2.53 sys=0.08, real=2.03 secs] 
> 2010-08-26T15:45:07.608+0200: 38705.282: [CMS-concurrent-reset-start]
> 2010-08-26T15:45:07.608+0200: 38705.288: [CMS-concurrent-reset: 
> 0.005/0.005 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
> 2010-08-26T15:45:08.341+0200: 38706.012: [GC 38706.012: [ParNew: 
> 235520K->0K(235712K), 0.0180667 secs] 1162004K->926949K(1179456K) 
> icms_dc=24 , 0.0182417 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:45:08.747+0200: 38706.425: [GC [1 CMS-initial-mark: 
> 926949K(943744K)] 1019208K(1179456K), 0.0677621 secs] [Times: 
> user=0.06 sys=0.00, real=0.06 secs] 
> 2010-08-26T15:45:08.809+0200: 38706.493: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:09.480+0200: 38707.158: [GC 38707.159: [ParNew: 
> 235520K->0K(235712K), 0.0477314 secs] 1162469K->932300K(1179456K) 
> icms_dc=39 , 0.0479217 secs] [Times: user=0.05 sys=0.00, real=0.05 secs] 
> 2010-08-26T15:45:10.463+0200: 38708.133: [GC 38708.133: [ParNew: 
> 235520K->0K(235712K), 0.0371965 secs] 1167820K->936491K(1179456K) 
> icms_dc=54 , 0.0373840 secs] [Times: user=0.03 sys=0.00, real=0.03 secs] 
> 2010-08-26T15:45:12.085+0200: 38709.770: [GC 38709.770: [ParNew: 
> 235520K->0K(235712K), 0.0185757 secs] 1172011K->936953K(1179456K) 
> icms_dc=69 , 0.0187606 secs] [Times: user=0.03 sys=0.00, real=0.03 secs] 
> 2010-08-26T15:45:13.130+0200: 38710.806: [GC 38710.806: [ParNew 
> (promotion failed): 235520K->235520K(235712K), 1.0952192 
> secs]38711.901: [CMS2010-08-26T15:45:14.222+0200: 38711.903: 
> [CMS-concurrent-mark: 1.934/5.410 secs] [Times: user=8.20 sys=0.25, 
> real=5.41 secs] 
>  (concurrent mode failure): 943043K->933853K(943744K), 4.0906398 secs] 
> 1172473K->933853K(1179456K), [CMS Perm : 97361K->97361K(131072K)] 
> icms_dc=86 , 5.1861184 secs] [Times: user=5.18 sys=0.00, real=5.18 secs] 
> 2010-08-26T15:45:18.372+0200: 38716.052: [GC [1 CMS-initial-mark: 
> 933853K(943744K)] 950390K(1179456K), 0.0155216 secs] [Times: user=0.01 
> sys=0.00, real=0.02 secs] 
> 2010-08-26T15:45:18.387+0200: 38716.068: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:19.355+0200: 38717.035: [GC 38717.035: [ParNew: 
> 235520K->0K(235712K), 0.0623713 secs] 1169373K->942749K(1179456K) 
> icms_dc=100 , 0.0625660 secs] [Times: user=0.06 sys=0.00, real=0.06 secs] 
> 2010-08-26T15:45:20.509+0200: 38718.182: [CMS-concurrent-mark: 
> 1.964/2.114 secs] [Times: user=4.57 sys=0.11, real=2.12 secs] 
> 2010-08-26T15:45:20.509+0200: 38718.182: [CMS-concurrent-preclean-start]
> 2010-08-26T15:45:20.525+0200: 38718.202: [CMS-concurrent-preclean: 
> 0.015/0.020 secs] [Times: user=0.02 sys=0.02, real=0.02 secs] 
> 2010-08-26T15:45:20.525+0200: 38718.202: 
> [CMS-concurrent-abortable-preclean-start]
> 2010-08-26T15:45:20.540+0200: 38718.222: [GC 38718.222: [ParNew: 
> 235520K->235520K(235712K), 0.0000592 secs]38718.222: 
> [CMS2010-08-26T15:45:20.540+0200: 38718.223: 
> [CMS-concurrent-abortable-preclean: 0.004/0.021 secs] [Times: 
> user=0.02 sys=0.00, real=0.01 secs] 
>  (concurrent mode failure): 942749K->936038K(943744K), 4.0650806 secs] 
> 1178269K->936038K(1179456K), [CMS Perm : 97371K->97371K(131072K)] 
> icms_dc=100 , 4.0653451 secs] [Times: user=4.07 sys=0.00, real=4.07 secs] 
> 2010-08-26T15:45:24.612+0200: 38722.289: [GC [1 CMS-initial-mark: 
> 936038K(943744K)] 936042K(1179456K), 0.0023357 secs] [Times: user=0.00 
> sys=0.00, real=0.00 secs] 
> 2010-08-26T15:45:24.612+0200: 38722.291: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:26.141+0200: 38723.817: [GC 38723.817: [ParNew: 
> 235520K->235520K(235712K), 0.0000588 secs]38723.817: 
> [CMS2010-08-26T15:45:27.030+0200: 38724.709: [CMS-concurrent-mark: 
> 1.935/2.418 secs] [Times: user=3.65 sys=0.19, real=2.42 secs] 
>  (concurrent mode failure): 936038K->942227K(943744K), 4.9718511 secs] 
> 1171558K->942227K(1179456K), [CMS Perm : 97393K->97393K(131072K)] 
> icms_dc=100 , 4.9721115 secs] [Times: user=4.98 sys=0.00, real=4.98 secs] 
> 2010-08-26T15:45:31.117+0200: 38728.790: [GC [1 CMS-initial-mark: 
> 942227K(943744K)] 942270K(1179456K), 0.0025839 secs] [Times: user=0.00 
> sys=0.00, real=0.00 secs] 
> 2010-08-26T15:45:31.117+0200: 38728.793: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:32.084+0200: 38729.758: [GC 38729.759: [ParNew: 
> 235520K->235520K(235712K), 0.0000623 secs]38729.759: 
> [CMS2010-08-26T15:45:33.051+0200: 38730.736: [CMS-concurrent-mark: 
> 1.934/1.943 secs] [Times: user=3.28 sys=0.19, real=1.93 secs] 
>  (concurrent mode failure): 942227K->943743K(943744K), 4.9442127 secs] 
> 1177747K->948024K(1179456K), [CMS Perm : 97396K->97396K(131072K)] 
> icms_dc=100 , 4.9446265 secs] [Times: user=4.91 sys=0.02, real=4.94 secs] 
> 2010-08-26T15:45:37.029+0200: 38734.704: [GC [1 CMS-initial-mark: 
> 943743K(943744K)] 948026K(1179456K), 0.0067733 secs] [Times: user=0.00 
> sys=0.00, real=0.00 secs] 
> 2010-08-26T15:45:37.029+0200: 38734.711: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:38.745+0200: 38736.419: [Full GC 38736.419: 
> [CMS2010-08-26T15:45:39.463+0200: 38737.138: [CMS-concurrent-mark: 
> 1.952/2.427 secs] [Times: user=3.95 sys=0.16, real=2.43 secs] 
>  (concurrent mode failure): 943743K->943743K(943744K), 4.7856092 secs] 
> 1179455K->949021K(1179456K), [CMS Perm : 97406K->97406K(131072K)] 
> icms_dc=100 , 4.7857927 secs] [Times: user=4.79 sys=0.00, real=4.79 secs] 
> 2010-08-26T15:45:43.534+0200: 38741.206: [GC [1 CMS-initial-mark: 
> 943743K(943744K)] 949287K(1179456K), 0.0080127 secs] [Times: user=0.00 
> sys=0.00, real=0.00 secs] 
> 2010-08-26T15:45:43.534+0200: 38741.214: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:44.455+0200: 38742.136: [Full GC 38742.136: 
> [CMS2010-08-26T15:45:45.484+0200: 38743.168: [CMS-concurrent-mark: 
> 1.952/1.954 secs] [Times: user=3.26 sys=0.05, real=1.95 secs] 
>  (concurrent mode failure): 943743K->943743K(943744K), 4.9692415 secs] 
> 1179440K->949759K(1179456K), [CMS Perm : 97406K->97406K(131072K)] 
> icms_dc=100 , 4.9694250 secs] [Times: user=4.96 sys=0.02, real=4.98 secs] 
> 2010-08-26T15:45:49.883+0200: 38747.569: [GC [1 CMS-initial-mark: 
> 943743K(943744K)] 953372K(1179456K), 0.0086197 secs] [Times: user=0.02 
> sys=0.00, real=0.02 secs] 
> 2010-08-26T15:45:49.899+0200: 38747.578: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:50.960+0200: 38748.634: [Full GC 38748.634: 
> [CMS2010-08-26T15:45:51.833+0200: 38749.515: [CMS-concurrent-mark: 
> 1.927/1.938 secs] [Times: user=3.23 sys=0.11, real=1.93 secs] 
>  (concurrent mode failure): 943743K->937800K(943744K), 4.9818453 secs] 
> 1179455K->937800K(1179456K), [CMS Perm : 97407K->97407K(131072K)] 
> icms_dc=100 , 4.9820296 secs] [Times: user=4.95 sys=0.02, real=4.98 secs] 
> 2010-08-26T15:45:55.936+0200: 38753.617: [GC [1 CMS-initial-mark: 
> 937800K(943744K)] 938528K(1179456K), 0.0024408 secs] [Times: user=0.00 
> sys=0.00, real=0.00 secs] 
> 2010-08-26T15:45:55.936+0200: 38753.620: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:56.950+0200: 38754.630: [GC 38754.631: [ParNew: 
> 235520K->235520K(235712K), 0.0000597 secs]38754.631: 
> [CMS2010-08-26T15:45:57.995+0200: 38755.679: [CMS-concurrent-mark: 
> 2.057/2.059 secs] [Times: user=3.29 sys=0.23, real=2.06 secs] 
>  (concurrent mode failure): 937800K->936173K(943744K), 5.1274815 secs] 
> 1173320K->936173K(1179456K), [CMS Perm : 97407K->97407K(131072K)] 
> icms_dc=100 , 5.1277524 secs] [Times: user=5.13 sys=0.00, real=5.13 secs] 
> 2010-08-26T15:46:02.535+0200: 38760.215: [GC [1 CMS-initial-mark: 
> 936173K(943744K)] 936184K(1179456K), 0.0024186 secs] [Times: user=0.00 
> sys=0.00, real=0.00 secs] 
> 2010-08-26T15:46:02.535+0200: 38760.218: [CMS-concurrent-mark-start]
> 2010-08-26T15:46:03.611+0200: 38761.290: [GC 38761.290: [ParNew 
> (promotion failed): 235520K->235520K(235712K), 0.1302956 
> secs]38761.420: [CMS2010-08-26T15:46:04.609+0200: 38762.289: 
> [CMS-concurrent-mark: 1.938/2.071 secs] [Times: user=3.54 sys=0.11, 
> real=2.07 secs] 
>  (concurrent mode failure): 943229K->939271K(943744K), 4.9629545 secs] 
> 1171693K->939271K(1179456K), [CMS Perm : 97407K->97407K(131072K)] 
> icms_dc=100 , 5.0934938 secs] [Times: user=5.09 sys=0.00, real=5.10 secs] 
>
> Thanks again for any help,
>
> Adam
>
> --
> Adam Hawthorne
> Software Engineer
> BASIS International Ltd.
> www.basis.com <http://www.basis.com>
> +1.505.345.5232 Phone
> ------------------------------------------------------------------------
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100831/04c8eeeb/attachment-0001.html 

From adamh at basis.com  Tue Aug 31 14:05:12 2010
From: adamh at basis.com (Adam Hawthorne)
Date: Tue, 31 Aug 2010 17:05:12 -0400
Subject: iCMS question
In-Reply-To: <4C7D6C0B.9070103@oracle.com>
References: <AANLkTi=D23ym9Nj7mv68oucB1wKfcUrh=b64A8N4Oqyu@mail.gmail.com>
	<4C7D6C0B.9070103@oracle.com>
Message-ID: <AANLkTinGk8S7hZ6OKq-B3Y+dqypWYJA+D+1quhbQaVXi@mail.gmail.com>

Thanks, somehow I missed the obvious answer.

Adam

--
Adam Hawthorne
Software Engineer
BASIS International Ltd.
www.basis.com
+1.505.345.5232 Phone


On Tue, Aug 31, 2010 at 16:54, Jon Masamitsu <jon.masamitsu at oracle.com>wrote:

>  I think the short answer is that your heap is full.
>
> This is a minor collection that turns into a full collection (so you
> don't see the telltale "Full GC" message).
>
>
> 2010-08-26T15:45:13.130+0200: 38710.806: [GC 38710.806: [ParNew (promotion
> failed): 235520K->235520K(235712K), 1.0952192 secs]38711.901:
> [CMS2010-08-26T15:45:14.222+0200: 38711.903: [CMS-concurrent-mark:
> 1.934/5.410 secs] [Times: user=8.20 sys=0.25, real=5.41 secs]
>  (concurrent mode failure): 943043K->933853K(943744K), 4.0906398 secs]
> 1172473K->933853K(1179456K), [CMS Perm : 97361K->97361K(131072K)] icms_dc=86
> , 5.1861184 secs] [Times: user=5.18 sys=0.00, real=5.18 secs]
>
> After the full collection the heap is still mostly full.
>
> 943043K->933853K(943744K)
>
> I think your advice to increase the size of the should will help.
>
>
> On 08/31/10 12:36, Adam Hawthorne wrote:
>
> Hi all,
>
>  I'm looking at an i-cms log, and I don't understand why I'm seeing what
> I'm seeing.  Any help is appreciated.  At about  38718.035 in the log below,
> icms_dc begins to report 100, and just after that I see several back-to-back
> concurrent mode failures.  During that time, the ParNew collections are
> reporting that they're doing no work.  This happens on and off for about 20
> minutes in the log.  Obviously the collections are taking seconds, but I
> don't see a Full GC every time.  What is it doing when it's not doing a full
> GC?  Is the continual CMS collection preventing the young generation from
> being collected (or perhaps collecting it simultaneously)?  When
> icms_dc=100, does that imply the application is stopped?
>
>  There was some uninformed command line twiddling going on here, so the GC
> parameters were:
> -Xmx1152m
> -Xms256m
> -XX:NewRatio=4
> -XX:MaxPermSize=128m
> -XX:+UseParNewGC
> -XX:-CMSParallelRemarkEnabled
> -XX:+UseConcMarkSweepGC
> -XX:+CMSPermGenSweepingEnabled
> -XX:+UseCMSCompactAtFullCollection
> -XX:CMSFullGCsBeforeCompaction=1
> -XX:+CMSClassUnloadingEnabled
> -XX:+CMSIncrementalMode
> -XX:+CMSIncrementalPacing
> -XX:CMSInitiatingOccupancyFraction=50
> -XX:ParallelGCThreads=1
> -XX:MaxTenuringThreshold=0
> -XX:SurvivorRatio=1024
> -XX:SoftRefLRUPolicyMSPerMB=1000
> -XX:+DisableExplicitGC
> -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps
> -XX:+PrintGCDateStamps
>
>  I had them remove -XX:ParallelGCThreads=1, -XX:MaxTenuringThreshold=0,
> and -XX:SurvivorRatio=1024 , thinking that since the duty cycle was
> approaching 100%, and because the ParNew pause times were so low, we could
> probably afford to spend more time doing young gc's instead of promoting.
>  I'm thinking I should probably tell them to increase their -Xmx and
> decrease their NewRatio as well.  Is there anything else?  I haven't done
> much tuning of i-cms in general, but the standard i-cms safety options don't
> seem like they would help here.
>
>  Here's the log excerpt:
>
>  2010-08-26T15:44:42.836+0200: 38680.510: [GC 38680.510: [ParNew:
> 235520K->0K(235712K), 0.0187551 secs] 1158126K->923156K(1179456K) icms_dc=41
> , 0.0189305 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:43.179+0200: 38680.861: [GC [1 CMS-initial-mark:
> 923156K(943744K)] 996964K(1179456K), 0.0526219 secs] [Times: user=0.06
> sys=0.00, real=0.06 secs]
> 2010-08-26T15:44:43.241+0200: 38680.914: [CMS-concurrent-mark-start]
> 2010-08-26T15:44:44.021+0200: 38681.695: [GC 38681.695: [ParNew:
> 235520K->0K(235712K), 0.0196054 secs] 1158676K->923705K(1179456K) icms_dc=41
> , 0.0197785 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:45.160+0200: 38682.831: [GC 38682.831: [ParNew:
> 235520K->0K(235712K), 0.0199663 secs] 1159225K->924310K(1179456K) icms_dc=41
> , 0.0201448 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:46.127+0200: 38683.797: [GC 38683.797: [ParNew:
> 235520K->0K(235712K), 0.0549963 secs] 1159830K->930710K(1179456K) icms_dc=56
> , 0.0551800 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
> 2010-08-26T15:44:47.562+0200: 38685.246: [CMS-concurrent-mark: 1.903/4.332
> secs] [Times: user=6.77 sys=0.25, real=4.32 secs]
> 2010-08-26T15:44:47.562+0200: 38685.246: [CMS-concurrent-preclean-start]
> 2010-08-26T15:44:47.828+0200: 38685.506: [GC 38685.507: [ParNew:
> 235520K->0K(235712K), 0.0180149 secs] 1166230K->931150K(1179456K) icms_dc=56
> , 0.0182051 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
> 2010-08-26T15:44:48.108+0200: 38685.790: [CMS-concurrent-preclean:
> 0.023/0.543 secs] [Times: user=0.55 sys=0.05, real=0.55 secs]
> 2010-08-26T15:44:48.108+0200: 38685.790:
> [CMS-concurrent-abortable-preclean-start]
> 2010-08-26T15:44:48.483+0200: 38686.158:
> [CMS-concurrent-abortable-preclean: 0.040/0.368 secs] [Times: user=0.41
> sys=0.02, real=0.38 secs]
> 2010-08-26T15:44:48.483+0200: 38686.159: [GC[YG occupancy: 136856 K (235712
> K)]38686.159: [Rescan (non-parallel) 38686.159: [grey object rescan,
> 0.0041031 secs]38686.163: [root rescan, 0.1153851 secs], 0.1195876
> secs]38686.279: [weak refs processing, 0.0012088 secs]38686.280: [class
> unloading, 0.0143079 secs]38686.294: [scrub symbol & string tables,
> 0.0159790 secs] [1 CMS-remark: 931150K(943744K)] 1068006K(1179456K),
> 0.1563910 secs] [Times: user=0.16 sys=0.00, real=0.16 secs]
> 2010-08-26T15:44:48.639+0200: 38686.316: [CMS-concurrent-sweep-start]
> 2010-08-26T15:44:49.138+0200: 38686.808: [GC 38686.808: [ParNew:
> 235520K->0K(235712K), 0.0183463 secs] 1166540K->931466K(1179456K) icms_dc=56
> , 0.0185256 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:49.637+0200: 38687.307: [CMS-concurrent-sweep: 0.470/0.992
> secs] [Times: user=1.47 sys=0.02, real=1.00 secs]
> 2010-08-26T15:44:49.637+0200: 38687.307: [CMS-concurrent-reset-start]
> 2010-08-26T15:44:49.637+0200: 38687.313: [CMS-concurrent-reset: 0.005/0.005
> secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-08-26T15:44:50.277+0200: 38687.948: [GC 38687.948: [ParNew:
> 235520K->0K(235712K), 0.0177987 secs] 1158207K->923110K(1179456K) icms_dc=42
> , 0.0179840 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:50.604+0200: 38688.280: [GC [1 CMS-initial-mark:
> 923110K(943744K)] 992484K(1179456K), 0.0509414 secs] [Times: user=0.05
> sys=0.00, real=0.05 secs]
> 2010-08-26T15:44:50.651+0200: 38688.331: [CMS-concurrent-mark-start]
> 2010-08-26T15:44:51.478+0200: 38689.157: [GC 38689.157: [ParNew:
> 235520K->0K(235712K), 0.0178124 secs] 1158630K->923567K(1179456K) icms_dc=42
> , 0.0179821 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:52.632+0200: 38690.306: [GC 38690.306: [ParNew:
> 235520K->0K(235712K), 0.0180970 secs] 1159087K->924012K(1179456K) icms_dc=42
> , 0.0182661 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:53.740+0200: 38691.422: [GC 38691.422: [ParNew:
> 235520K->0K(235712K), 0.0267182 secs] 1159532K->925696K(1179456K) icms_dc=42
> , 0.0269020 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
> 2010-08-26T15:44:54.520+0200: 38692.201: [CMS-concurrent-mark: 1.894/3.870
> secs] [Times: user=5.74 sys=0.13, real=3.87 secs]
> 2010-08-26T15:44:54.520+0200: 38692.201: [CMS-concurrent-preclean-start]
> 2010-08-26T15:44:54.551+0200: 38692.222: [CMS-concurrent-preclean:
> 0.015/0.020 secs] [Times: user=0.05 sys=0.02, real=0.03 secs]
> 2010-08-26T15:44:54.551+0200: 38692.222:
> [CMS-concurrent-abortable-preclean-start]
> 2010-08-26T15:44:54.894+0200: 38692.573: [GC 38692.573: [ParNew:
> 235520K->0K(235712K), 0.0186037 secs] 1161216K->926149K(1179456K) icms_dc=42
> , 0.0187843 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:55.581+0200: 38693.250:
> [CMS-concurrent-abortable-preclean: 0.050/1.028 secs] [Times: user=1.05
> sys=0.01, real=1.03 secs]
> 2010-08-26T15:44:55.581+0200: 38693.251: [GC[YG occupancy: 140686 K (235712
> K)]38693.251: [Rescan (non-parallel) 38693.251: [grey object rescan,
> 0.0043953 secs]38693.255: [root rescan, 0.1209735 secs], 0.1254680
> secs]38693.376: [weak refs processing, 0.0013433 secs]38693.378: [class
> unloading, 0.0142815 secs]38693.392: [scrub symbol & string tables,
> 0.0160091 secs] [1 CMS-remark: 926149K(943744K)] 1066835K(1179456K),
> 0.1624119 secs] [Times: user=0.16 sys=0.00, real=0.16 secs]
> 2010-08-26T15:44:55.737+0200: 38693.414: [CMS-concurrent-sweep-start]
> 2010-08-26T15:44:56.205+0200: 38693.879: [GC 38693.880: [ParNew:
> 235520K->0K(235712K), 0.0188069 secs] 1161648K->926599K(1179456K) icms_dc=42
> , 0.0189967 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:56.891+0200: 38694.563: [CMS-concurrent-sweep: 0.474/1.149
> secs] [Times: user=1.64 sys=0.02, real=1.15 secs]
> 2010-08-26T15:44:56.891+0200: 38694.563: [CMS-concurrent-reset-start]
> 2010-08-26T15:44:56.891+0200: 38694.569: [CMS-concurrent-reset: 0.006/0.006
> secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-08-26T15:44:57.343+0200: 38695.018: [GC 38695.018: [ParNew:
> 235520K->0K(235712K), 0.0187471 secs] 1156567K->921554K(1179456K) icms_dc=32
> , 0.0189332 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:57.718+0200: 38695.402: [GC [1 CMS-initial-mark:
> 921554K(943744K)] 1001837K(1179456K), 0.0572313 secs] [Times: user=0.06
> sys=0.00, real=0.06 secs]
> 2010-08-26T15:44:57.780+0200: 38695.460: [CMS-concurrent-mark-start]
> 2010-08-26T15:44:58.545+0200: 38696.224: [GC 38696.225: [ParNew:
> 235520K->0K(235712K), 0.0183404 secs] 1157074K->922009K(1179456K) icms_dc=32
> , 0.0185097 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
> 2010-08-26T15:45:00.167+0200: 38697.850: [GC 38697.850: [ParNew:
> 235520K->0K(235712K), 0.0196135 secs] 1157529K->922547K(1179456K) icms_dc=32
> , 0.0197977 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
> 2010-08-26T15:45:01.321+0200: 38698.992: [GC 38698.992: [ParNew:
> 235520K->0K(235712K), 0.0187905 secs] 1158067K->923023K(1179456K) icms_dc=32
> , 0.0189818 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
> 2010-08-26T15:45:02.476+0200: 38700.146: [GC 38700.146: [ParNew:
> 235520K->0K(235712K), 0.0180207 secs] 1158543K->923471K(1179456K) icms_dc=32
> , 0.0181892 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
> 2010-08-26T15:45:03.615+0200: 38701.284: [GC 38701.284: [ParNew:
> 235520K->0K(235712K), 0.0186408 secs] 1158991K->923976K(1179456K) icms_dc=32
> , 0.0188287 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:45:04.051+0200: 38701.735: [CMS-concurrent-mark: 1.899/6.275
> secs] [Times: user=8.16 sys=0.06, real=6.27 secs]
> 2010-08-26T15:45:04.051+0200: 38701.735: [CMS-concurrent-preclean-start]
> 2010-08-26T15:45:04.083+0200: 38701.756: [CMS-concurrent-preclean:
> 0.016/0.021 secs] [Times: user=0.05 sys=0.02, real=0.03 secs]
> 2010-08-26T15:45:04.083+0200: 38701.756:
> [CMS-concurrent-abortable-preclean-start]
> 2010-08-26T15:45:04.753+0200: 38702.434: [GC 38702.435: [ParNew:
> 235520K->0K(235712K), 0.0184004 secs] 1159496K->924427K(1179456K) icms_dc=32
> , 0.0185851 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:45:05.409+0200: 38703.087:
> [CMS-concurrent-abortable-preclean: 0.063/1.331 secs] [Times: user=1.39
> sys=0.03, real=1.33 secs]
> 2010-08-26T15:45:05.409+0200: 38703.088: [GC[YG occupancy: 141725 K (235712
> K)]38703.088: [Rescan (non-parallel) 38703.088: [grey object rescan,
> 0.0041240 secs]38703.093: [root rescan, 0.1200361 secs], 0.1242579
> secs]38703.213: [weak refs processing, 0.0010203 secs]38703.214: [class
> unloading, 0.0142923 secs]38703.228: [scrub symbol & string tables,
> 0.0170438 secs] [1 CMS-remark: 924427K(943744K)] 1066152K(1179456K),
> 0.1618864 secs] [Times: user=0.17 sys=0.00, real=0.17 secs]
> 2010-08-26T15:45:05.580+0200: 38703.250: [CMS-concurrent-sweep-start]
> 2010-08-26T15:45:06.017+0200: 38703.687: [GC 38703.687: [ParNew:
> 235520K->0K(235712K), 0.0425827 secs] 1159944K->929053K(1179456K) icms_dc=32
> , 0.0427672 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
> 2010-08-26T15:45:07.202+0200: 38704.873: [GC 38704.873: [ParNew:
> 235520K->0K(235712K), 0.0179344 secs] 1161562K->926484K(1179456K) icms_dc=32
> , 0.0181045 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
> 2010-08-26T15:45:07.608+0200: 38705.282: [CMS-concurrent-sweep: 0.472/2.032
> secs] [Times: user=2.53 sys=0.08, real=2.03 secs]
> 2010-08-26T15:45:07.608+0200: 38705.282: [CMS-concurrent-reset-start]
> 2010-08-26T15:45:07.608+0200: 38705.288: [CMS-concurrent-reset: 0.005/0.005
> secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-08-26T15:45:08.341+0200: 38706.012: [GC 38706.012: [ParNew:
> 235520K->0K(235712K), 0.0180667 secs] 1162004K->926949K(1179456K) icms_dc=24
> , 0.0182417 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:45:08.747+0200: 38706.425: [GC [1 CMS-initial-mark:
> 926949K(943744K)] 1019208K(1179456K), 0.0677621 secs] [Times: user=0.06
> sys=0.00, real=0.06 secs]
> 2010-08-26T15:45:08.809+0200: 38706.493: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:09.480+0200: 38707.158: [GC 38707.159: [ParNew:
> 235520K->0K(235712K), 0.0477314 secs] 1162469K->932300K(1179456K) icms_dc=39
> , 0.0479217 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
> 2010-08-26T15:45:10.463+0200: 38708.133: [GC 38708.133: [ParNew:
> 235520K->0K(235712K), 0.0371965 secs] 1167820K->936491K(1179456K) icms_dc=54
> , 0.0373840 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
> 2010-08-26T15:45:12.085+0200: 38709.770: [GC 38709.770: [ParNew:
> 235520K->0K(235712K), 0.0185757 secs] 1172011K->936953K(1179456K) icms_dc=69
> , 0.0187606 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
> 2010-08-26T15:45:13.130+0200: 38710.806: [GC 38710.806: [ParNew (promotion
> failed): 235520K->235520K(235712K), 1.0952192 secs]38711.901:
> [CMS2010-08-26T15:45:14.222+0200: 38711.903: [CMS-concurrent-mark:
> 1.934/5.410 secs] [Times: user=8.20 sys=0.25, real=5.41 secs]
>  (concurrent mode failure): 943043K->933853K(943744K), 4.0906398 secs]
> 1172473K->933853K(1179456K), [CMS Perm : 97361K->97361K(131072K)] icms_dc=86
> , 5.1861184 secs] [Times: user=5.18 sys=0.00, real=5.18 secs]
> 2010-08-26T15:45:18.372+0200: 38716.052: [GC [1 CMS-initial-mark:
> 933853K(943744K)] 950390K(1179456K), 0.0155216 secs] [Times: user=0.01
> sys=0.00, real=0.02 secs]
> 2010-08-26T15:45:18.387+0200: 38716.068: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:19.355+0200: 38717.035: [GC 38717.035: [ParNew:
> 235520K->0K(235712K), 0.0623713 secs] 1169373K->942749K(1179456K)
> icms_dc=100 , 0.0625660 secs] [Times: user=0.06 sys=0.00, real=0.06 secs]
> 2010-08-26T15:45:20.509+0200: 38718.182: [CMS-concurrent-mark: 1.964/2.114
> secs] [Times: user=4.57 sys=0.11, real=2.12 secs]
> 2010-08-26T15:45:20.509+0200: 38718.182: [CMS-concurrent-preclean-start]
> 2010-08-26T15:45:20.525+0200: 38718.202: [CMS-concurrent-preclean:
> 0.015/0.020 secs] [Times: user=0.02 sys=0.02, real=0.02 secs]
> 2010-08-26T15:45:20.525+0200: 38718.202:
> [CMS-concurrent-abortable-preclean-start]
> 2010-08-26T15:45:20.540+0200: 38718.222: [GC 38718.222: [ParNew:
> 235520K->235520K(235712K), 0.0000592 secs]38718.222:
> [CMS2010-08-26T15:45:20.540+0200: 38718.223:
> [CMS-concurrent-abortable-preclean: 0.004/0.021 secs] [Times: user=0.02
> sys=0.00, real=0.01 secs]
>  (concurrent mode failure): 942749K->936038K(943744K), 4.0650806 secs]
> 1178269K->936038K(1179456K), [CMS Perm : 97371K->97371K(131072K)]
> icms_dc=100 , 4.0653451 secs] [Times: user=4.07 sys=0.00, real=4.07 secs]
> 2010-08-26T15:45:24.612+0200: 38722.289: [GC [1 CMS-initial-mark:
> 936038K(943744K)] 936042K(1179456K), 0.0023357 secs] [Times: user=0.00
> sys=0.00, real=0.00 secs]
> 2010-08-26T15:45:24.612+0200: 38722.291: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:26.141+0200: 38723.817: [GC 38723.817: [ParNew:
> 235520K->235520K(235712K), 0.0000588 secs]38723.817:
> [CMS2010-08-26T15:45:27.030+0200: 38724.709: [CMS-concurrent-mark:
> 1.935/2.418 secs] [Times: user=3.65 sys=0.19, real=2.42 secs]
>  (concurrent mode failure): 936038K->942227K(943744K), 4.9718511 secs]
> 1171558K->942227K(1179456K), [CMS Perm : 97393K->97393K(131072K)]
> icms_dc=100 , 4.9721115 secs] [Times: user=4.98 sys=0.00, real=4.98 secs]
> 2010-08-26T15:45:31.117+0200: 38728.790: [GC [1 CMS-initial-mark:
> 942227K(943744K)] 942270K(1179456K), 0.0025839 secs] [Times: user=0.00
> sys=0.00, real=0.00 secs]
> 2010-08-26T15:45:31.117+0200: 38728.793: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:32.084+0200: 38729.758: [GC 38729.759: [ParNew:
> 235520K->235520K(235712K), 0.0000623 secs]38729.759:
> [CMS2010-08-26T15:45:33.051+0200: 38730.736: [CMS-concurrent-mark:
> 1.934/1.943 secs] [Times: user=3.28 sys=0.19, real=1.93 secs]
>  (concurrent mode failure): 942227K->943743K(943744K), 4.9442127 secs]
> 1177747K->948024K(1179456K), [CMS Perm : 97396K->97396K(131072K)]
> icms_dc=100 , 4.9446265 secs] [Times: user=4.91 sys=0.02, real=4.94 secs]
> 2010-08-26T15:45:37.029+0200: 38734.704: [GC [1 CMS-initial-mark:
> 943743K(943744K)] 948026K(1179456K), 0.0067733 secs] [Times: user=0.00
> sys=0.00, real=0.00 secs]
> 2010-08-26T15:45:37.029+0200: 38734.711: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:38.745+0200: 38736.419: [Full GC 38736.419:
> [CMS2010-08-26T15:45:39.463+0200: 38737.138: [CMS-concurrent-mark:
> 1.952/2.427 secs] [Times: user=3.95 sys=0.16, real=2.43 secs]
>  (concurrent mode failure): 943743K->943743K(943744K), 4.7856092 secs]
> 1179455K->949021K(1179456K), [CMS Perm : 97406K->97406K(131072K)]
> icms_dc=100 , 4.7857927 secs] [Times: user=4.79 sys=0.00, real=4.79 secs]
> 2010-08-26T15:45:43.534+0200: 38741.206: [GC [1 CMS-initial-mark:
> 943743K(943744K)] 949287K(1179456K), 0.0080127 secs] [Times: user=0.00
> sys=0.00, real=0.00 secs]
> 2010-08-26T15:45:43.534+0200: 38741.214: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:44.455+0200: 38742.136: [Full GC 38742.136:
> [CMS2010-08-26T15:45:45.484+0200: 38743.168: [CMS-concurrent-mark:
> 1.952/1.954 secs] [Times: user=3.26 sys=0.05, real=1.95 secs]
>  (concurrent mode failure): 943743K->943743K(943744K), 4.9692415 secs]
> 1179440K->949759K(1179456K), [CMS Perm : 97406K->97406K(131072K)]
> icms_dc=100 , 4.9694250 secs] [Times: user=4.96 sys=0.02, real=4.98 secs]
> 2010-08-26T15:45:49.883+0200: 38747.569: [GC [1 CMS-initial-mark:
> 943743K(943744K)] 953372K(1179456K), 0.0086197 secs] [Times: user=0.02
> sys=0.00, real=0.02 secs]
> 2010-08-26T15:45:49.899+0200: 38747.578: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:50.960+0200: 38748.634: [Full GC 38748.634:
> [CMS2010-08-26T15:45:51.833+0200: 38749.515: [CMS-concurrent-mark:
> 1.927/1.938 secs] [Times: user=3.23 sys=0.11, real=1.93 secs]
>  (concurrent mode failure): 943743K->937800K(943744K), 4.9818453 secs]
> 1179455K->937800K(1179456K), [CMS Perm : 97407K->97407K(131072K)]
> icms_dc=100 , 4.9820296 secs] [Times: user=4.95 sys=0.02, real=4.98 secs]
> 2010-08-26T15:45:55.936+0200: 38753.617: [GC [1 CMS-initial-mark:
> 937800K(943744K)] 938528K(1179456K), 0.0024408 secs] [Times: user=0.00
> sys=0.00, real=0.00 secs]
> 2010-08-26T15:45:55.936+0200: 38753.620: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:56.950+0200: 38754.630: [GC 38754.631: [ParNew:
> 235520K->235520K(235712K), 0.0000597 secs]38754.631:
> [CMS2010-08-26T15:45:57.995+0200: 38755.679: [CMS-concurrent-mark:
> 2.057/2.059 secs] [Times: user=3.29 sys=0.23, real=2.06 secs]
>  (concurrent mode failure): 937800K->936173K(943744K), 5.1274815 secs]
> 1173320K->936173K(1179456K), [CMS Perm : 97407K->97407K(131072K)]
> icms_dc=100 , 5.1277524 secs] [Times: user=5.13 sys=0.00, real=5.13 secs]
> 2010-08-26T15:46:02.535+0200: 38760.215: [GC [1 CMS-initial-mark:
> 936173K(943744K)] 936184K(1179456K), 0.0024186 secs] [Times: user=0.00
> sys=0.00, real=0.00 secs]
> 2010-08-26T15:46:02.535+0200: 38760.218: [CMS-concurrent-mark-start]
> 2010-08-26T15:46:03.611+0200: 38761.290: [GC 38761.290: [ParNew (promotion
> failed): 235520K->235520K(235712K), 0.1302956 secs]38761.420:
> [CMS2010-08-26T15:46:04.609+0200: 38762.289: [CMS-concurrent-mark:
> 1.938/2.071 secs] [Times: user=3.54 sys=0.11, real=2.07 secs]
>  (concurrent mode failure): 943229K->939271K(943744K), 4.9629545 secs]
> 1171693K->939271K(1179456K), [CMS Perm : 97407K->97407K(131072K)]
> icms_dc=100 , 5.0934938 secs] [Times: user=5.09 sys=0.00, real=5.10 secs]
>
>  Thanks again for any help,
>
> Adam
>
> --
> Adam Hawthorne
> Software Engineer
> BASIS International Ltd.
> www.basis.com
> +1.505.345.5232 Phone
>
> ------------------------------
>
> _______________________________________________
> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100831/e3c298df/attachment-0001.html 

From y.s.ramakrishna at oracle.com  Tue Aug 31 15:27:25 2010
From: y.s.ramakrishna at oracle.com (Y. S. Ramakrishna)
Date: Tue, 31 Aug 2010 15:27:25 -0700
Subject: Long GC pause time
In-Reply-To: <4C7D49E4.6060908@oracle.com>
References: <SNT108-W11C7575D28055DFA3A6544E38A0@phx.gbl>
	<4C7D49E4.6060908@oracle.com>
Message-ID: <4C7D81CD.5050706@oracle.com>

Noam, also please provision and use the survivor spaces to age
objects and reduce pressure on CMS. The premature
promotion and subsequent death is probably exacerbating the
proliferating Ref object deaths (if i may be permitted an
oxymoron) in the old gen.

-- ramki

On 08/31/10 11:28, Jon Masamitsu wrote:
> Try -XX+ParallelRefProcEnabled
> 
> Your spending much of the pause time doing Reference processing
> 
> 89.867: [GC[YG occupancy: 420 K (274752 K)]89.867: [Rescan (parallel) , 
> 0.0242770 secs]89.892: [weak refs processing, 1.3860230 secs] [1 
> CMS-remark: 4198190K(6635136K)] 4198611K(6909888K), 1.4164670 secs] 
> [Times: user=1.68 sys=0.01, real=1.41 secs]
> 
> (see the "[weak refs processing, 1.3860230 secs]") so doing it in 
> parallel might
> help.  ParallelRefProcEnabled isn't always a win so it's not "on" by 
> default.
> 
> You might also try -XX:+CMSScavengeBeforeRemark.  That's a just a guess.
> Sometimes doing a minor collection before the remark helps.
> 
> On 08/31/10 08:22, noam katzowicz wrote:
>> Hello.
>>  
>> We are running JBoss AS over RH5, using java 6u21, with the following 
>> GC params:
>>  
>> -Xms6750m -Xmx6750m
>> -XX:+UseConcMarkSweepGC
>> -XX:SurvivorRatio=128
>> -XX:MaxTenuringThreshold=0
>> -XX:+UseCompressedOops
>> -XX:CMSInitiatingOccupancyFraction=60
>> -Dsun.rmi.dgc.client.gcInterval=3600000
>> -Dsun.rmi.dgc.server.gcInterval=3600000
>> -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+PrintTenuringDistribution
>>  
>> We are experiencing relatively long GC pause time of over 1 second.
>> Our application allocates hundreds of MBs per second. We?ve tried 
>> different generation sizes but the above was the best we came up with. 
>> Your help is appreciated.
>>  
>>
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0596370 secs] 4422502K->4173928K(6909888K), 
>> 0.0606730 secs] [Times: user=0.44 sys=0.03, real=0.06 secs]
>> 89.486: [CMS-concurrent-preclean: 1.308/1.649 secs] [Times: user=10.32 
>> sys=2.30, real=1.65 secs]
>> 89.487: [CMS-concurrent-abortable-preclean-start]
>> 89.804: [GC 89.804: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0603670 secs] 4446568K->4198190K(6909888K), 
>> 0.0614310 secs] [Times: user=0.48 sys=0.03, real=0.07 secs]
>> 89.866: [CMS-concurrent-abortable-preclean: 0.293/0.379 secs] [Times: 
>> user=2.42 sys=0.54, real=0.38 secs]
>> 89.867: [GC[YG occupancy: 420 K (274752 K)]89.867: [Rescan (parallel) 
>> , 0.0242770 secs]89.892: [weak refs processing, 1.3860230 secs] [1 
>> CMS-remark: 4198190K(6635136K)] 4198611K(6909888K), 1.4164670 secs] 
>> [Times: user=1.68 sys=0.01, real=1.41 secs]
>> 91.284: [CMS-concurrent-sweep-start]
>> 91.590: [GC 91.591: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0675550 secs] 4371287K->4128966K(6909888K), 
>> 0.0683470 secs] [Times: user=0.56 sys=0.01, real=0.07 secs]
>> 91.860: [GC 91.860: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0604320 secs] 4308368K->4053829K(6909888K), 
>> 0.0612020 secs] [Times: user=0.41 sys=0.01, real=0.06 secs]
>> 92.786: [GC 92.787: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.1028730 secs] 3964957K->3753351K(6909888K), 
>> 0.1038490 secs] [Times: user=0.80 sys=0.02, real=0.10 secs]
>> 93.106: [GC 93.107: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0487000 secs] 3949085K->3699496K(6909888K), 
>> 0.0494470 secs] [Times: user=0.32 sys=0.01, real=0.05 secs]
>> 93.371: [GC 93.371: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0430620 secs] 3905862K->3652769K(6909888K), 
>> 0.0438800 secs] [Times: user=0.33 sys=0.01, real=0.05 secs]
>> 94.637: [GC 94.638: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0564930 secs] 3416325K->3164258K(6909888K), 
>> 0.0575000 secs] [Times: user=0.44 sys=0.02, real=0.06 secs]
>> 94.863: [CMS-concurrent-sweep: 3.107/3.578 secs] [Times: user=16.85 
>> sys=3.23, real=3.58 secs]
>> 94.863: [CMS-concurrent-reset-start]
>> 94.878: [CMS-concurrent-reset: 0.015/0.015 secs] [Times: user=0.03 
>> sys=0.01, real=0.02 secs]
>> 95.981: [GC 95.981: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0514980 secs] 3405940K->3152334K(6909888K), 
>> 0.0524610 secs] [Times: user=0.44 sys=0.01, real=0.05 secs]
>> 100.179: [GC 100.180: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0676510 secs] 3424974K->3178150K(6909888K), 
>> 0.0686240 secs] [Times: user=0.58 sys=0.00, real=0.07 secs]
>> 100.909: [GC 100.909: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0707060 secs] 3450790K->3202638K(6909888K), 
>> 0.0717030 secs] [Times: user=0.55 sys=0.01, real=0.07 secs]
>> 101.613: [GC 101.613: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272636K->0K(274752K), 0.0667270 secs] 3475274K->3224061K(6909888K), 
>> 0.0678240 secs] [Times: user=0.50 sys=0.01, real=0.07 secs]
>> 102.072: [GC 102.073: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272635K->0K(274752K), 0.0590350 secs] 3496696K->3241246K(6909888K), 
>> 0.0601190 secs] [Times: user=0.44 sys=0.00, real=0.06 secs]
>> 102.756: [GC 102.756: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272613K->0K(274752K), 0.0630060 secs] 3513860K->3262602K(6909888K), 
>> 0.0640320 secs] [Times: user=0.49 sys=0.00, real=0.07 secs]
>> 103.452: [GC 103.453: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0533390 secs] 3535242K->3280184K(6909888K), 
>> 0.0543760 secs] [Times: user=0.41 sys=0.01, real=0.06 secs]
>> 104.462: [GC 104.462: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0729890 secs] 3552824K->3299003K(6909888K), 
>> 0.0740260 secs] [Times: user=0.48 sys=0.01, real=0.07 secs]
>> 106.274: [GC 106.274: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0609880 secs] 3571643K->3314370K(6909888K), 
>> 0.0619510 secs] [Times: user=0.41 sys=0.00, real=0.06 secs]
>> 108.154: [GC 108.155: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.1251050 secs] 3587010K->3343996K(6909888K), 
>> 0.1261630 secs] [Times: user=0.71 sys=0.02, real=0.13 secs]
>> 108.882: [GC 108.883: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0952060 secs] 3616636K->3369490K(6909888K), 
>> 0.0962930 secs] [Times: user=0.64 sys=0.01, real=0.10 secs]
>> 109.482: [GC 109.482: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272936K->0K(274752K), 0.0791270 secs] 3642426K->3392040K(6909888K), 
>> 0.0800830 secs] [Times: user=0.53 sys=0.01, real=0.08 secs]
>> 109.948: [GC 109.949: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 273038K->0K(274752K), 0.0835410 secs] 3665079K->3417813K(6909888K), 
>> 0.0845680 secs] [Times: user=0.57 sys=0.00, real=0.08 secs]
>> 110.035: [GC 110.035: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 64K->0K(274752K), 0.0088490 secs] 3417877K->3417828K(6909888K), 
>> 0.0093860 secs] [Times: user=0.09 sys=0.00, real=0.01 secs]
>> 110.484: [GC 110.485: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0880810 secs] 3690468K->3443727K(6909888K), 
>> 0.0890560 secs] [Times: user=0.59 sys=0.00, real=0.09 secs]
>> 110.910: [GC 110.910: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0860790 secs] 3716367K->3475451K(6909888K), 
>> 0.0870070 secs] [Times: user=0.58 sys=0.02, real=0.08 secs]
>> 111.276: [GC 111.277: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272639K->0K(274752K), 0.0772230 secs] 3748091K->3503686K(6909888K), 
>> 0.0782780 secs] [Times: user=0.47 sys=0.02, real=0.08 secs]
>> 111.658: [GC 111.659: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0754700 secs] 3776326K->3532682K(6909888K), 
>> 0.0763440 secs] [Times: user=0.54 sys=0.01, real=0.07 secs]
>> 112.068: [GC 112.068: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0705680 secs] 3805322K->3559619K(6909888K), 
>> 0.0714610 secs] [Times: user=0.47 sys=0.02, real=0.07 secs]
>> 112.497: [GC 112.498: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0779750 secs] 3832259K->3589461K(6909888K), 
>> 0.0788830 secs] [Times: user=0.50 sys=0.01, real=0.08 secs]
>> 112.911: [GC 112.911: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0728900 secs] 3862101K->3616744K(6909888K), 
>> 0.0738560 secs] [Times: user=0.52 sys=0.00, real=0.07 secs]
>> 113.362: [GC 113.363: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0609030 secs] 3889384K->3638477K(6909888K), 
>> 0.0619050 secs] [Times: user=0.44 sys=0.01, real=0.06 secs]
>> 113.829: [GC 113.829: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272614K->0K(274752K), 0.0641940 secs] 3911091K->3659305K(6909888K), 
>> 0.0653490 secs] [Times: user=0.48 sys=0.01, real=0.06 secs]
>> 114.311: [GC 114.311: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0618170 secs] 3931945K->3681619K(6909888K), 
>> 0.0626970 secs] [Times: user=0.50 sys=0.01, real=0.06 secs]
>> 114.841: [GC 114.842: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0670890 secs] 3954259K->3711036K(6909888K), 
>> 0.0680540 secs] [Times: user=0.52 sys=0.02, real=0.07 secs]
>> 115.356: [GC 115.357: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0623700 secs] 3983676K->3737945K(6909888K), 
>> 0.0633630 secs] [Times: user=0.49 sys=0.00, real=0.07 secs]
>> 115.829: [GC 115.830: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0665900 secs] 4010585K->3766876K(6909888K), 
>> 0.0676240 secs] [Times: user=0.51 sys=0.01, real=0.06 secs]
>> 116.271: [GC 116.271: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272630K->0K(274752K), 0.0594280 secs] 4039506K->3796861K(6909888K), 
>> 0.0603620 secs] [Times: user=0.46 sys=0.02, real=0.06 secs]
>> 116.716: [GC 116.717: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0724150 secs] 4069501K->3825191K(6909888K), 
>> 0.0733430 secs] [Times: user=0.52 sys=0.01, real=0.08 secs]
>> 117.145: [GC 117.146: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0664990 secs] 4097831K->3853057K(6909888K), 
>> 0.0675760 secs] [Times: user=0.48 sys=0.00, real=0.07 secs]
>> 117.543: [GC 117.545: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0650350 secs] 4125697K->3880562K(6909888K), 
>> 0.0673450 secs] [Times: user=0.49 sys=0.01, real=0.07 secs]
>> 117.956: [GC 117.956: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0664410 secs] 4153202K->3907935K(6909888K), 
>> 0.0673980 secs] [Times: user=0.49 sys=0.02, real=0.07 secs]
>> 118.385: [GC 118.386: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272631K->0K(274752K), 0.0655070 secs] 4180566K->3934925K(6909888K), 
>> 0.0664120 secs] [Times: user=0.50 sys=0.01, real=0.07 secs]
>> 118.826: [GC 118.827: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272608K->0K(274752K), 0.0605390 secs] 4207533K->3964211K(6909888K), 
>> 0.0616050 secs] [Times: user=0.50 sys=0.02, real=0.06 secs]
>> 119.233: [GC 119.234: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0656870 secs] 4236851K->3991065K(6909888K), 
>> 0.0666900 secs] [Times: user=0.50 sys=0.02, real=0.07 secs]
>> 119.301: [GC [1 CMS-initial-mark: 3991065K(6635136K)] 
>> 3991071K(6909888K), 0.0091650 secs] [Times: user=0.01 sys=0.00, 
>> real=0.01 secs]
>> 119.311: [CMS-concurrent-mark-start]
>> 119.699: [GC 119.699: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0647090 secs] 4263705K->4019961K(6909888K), 
>> 0.0656290 secs] [Times: user=0.53 sys=0.01, real=0.06 secs]
>> 120.081: [GC 120.082: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0641110 secs] 4292601K->4048334K(6909888K), 
>> 0.0649070 secs] [Times: user=0.50 sys=0.01, real=0.06 secs]
>> 120.538: [GC 120.539: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0677200 secs] 4320974K->4075491K(6909888K), 
>> 0.0687990 secs] [Times: user=0.49 sys=0.03, real=0.07 secs]
>> 120.621: [CMS-concurrent-mark: 1.076/1.310 secs] [Times: user=10.66 
>> sys=1.73, real=1.31 secs]
>> 120.621: [CMS-concurrent-preclean-start]
>> 120.942: [GC 120.943: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0736770 secs] 4348131K->4104969K(6909888K), 
>> 0.0747430 secs] [Times: user=0.51 sys=0.03, real=0.07 secs]
>> 121.337: [GC 121.338: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0569980 secs] 4377609K->4126623K(6909888K), 
>> 0.0579070 secs] [Times: user=0.48 sys=0.03, real=0.06 secs]
>> 121.766: [GC 121.767: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272615K->0K(274752K), 0.0590140 secs] 4399238K->4148770K(6909888K), 
>> 0.0599510 secs] [Times: user=0.44 sys=0.03, real=0.06 secs]
>> 122.174: [GC 122.175: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0531420 secs] 4421410K->4167895K(6909888K), 
>> 0.0540310 secs] [Times: user=0.45 sys=0.03, real=0.05 secs]
>> 122.257: [CMS-concurrent-preclean: 1.306/1.636 secs] [Times: 
>> user=10.25 sys=2.21, real=1.63 secs]
>> 122.257: [CMS-concurrent-abortable-preclean-start]
>> 122.631: [GC 122.631: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0528220 secs] 4440535K->4187654K(6909888K), 
>> 0.0537160 secs] [Times: user=0.46 sys=0.02, real=0.05 secs]
>> 122.696: [CMS-concurrent-abortable-preclean: 0.355/0.439 secs] [Times: 
>> user=2.50 sys=0.52, real=0.44 secs]
>> 122.698: [GC[YG occupancy: 7874 K (274752 K)]122.698: [Rescan 
>> (parallel) , 0.0253430 secs]122.723: [weak refs processing, 1.1038100 
>> secs] [1 CMS-remark: 4187654K(6635136K)] 4195529K(6909888K), 1.1357970 
>> secs] [Times: user=1.42 sys=0.01, real=1.14 secs]
>> 123.834: [CMS-concurrent-sweep-start]
>> 124.060: [GC 124.061: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0690300 secs] 4399395K->4167257K(6909888K), 
>> 0.0698010 secs] [Times: user=0.52 sys=0.02, real=0.07 secs]
>>  
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>   
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

From jon.masamitsu at oracle.com  Tue Aug 31 16:14:36 2010
From: jon.masamitsu at oracle.com (Jon Masamitsu)
Date: Tue, 31 Aug 2010 16:14:36 -0700
Subject: iCMS question
In-Reply-To: <AANLkTinGk8S7hZ6OKq-B3Y+dqypWYJA+D+1quhbQaVXi@mail.gmail.com>
References: <AANLkTi=D23ym9Nj7mv68oucB1wKfcUrh=b64A8N4Oqyu@mail.gmail.com>
	<4C7D6C0B.9070103@oracle.com>
	<AANLkTinGk8S7hZ6OKq-B3Y+dqypWYJA+D+1quhbQaVXi@mail.gmail.com>
Message-ID: <4C7D8CDC.3040301@oracle.com>

If a larger heap helps, consider using the survivor spaces again.
Most of the time the survivor spaces help filter out short-lived
objects from the CMS gen and keeps fragmentation down.

Good luck.

On 08/31/10 14:05, Adam Hawthorne wrote:
> Thanks, somehow I missed the obvious answer.
>
> Adam
>
> --
> Adam Hawthorne
> Software Engineer
> BASIS International Ltd.
> www.basis.com <http://www.basis.com>
> +1.505.345.5232 Phone
>
>
> On Tue, Aug 31, 2010 at 16:54, Jon Masamitsu <jon.masamitsu at oracle.com 
> <mailto:jon.masamitsu at oracle.com>> wrote:
>
>     I think the short answer is that your heap is full.
>
>     This is a minor collection that turns into a full collection (so you
>     don't see the telltale "Full GC" message).
>
>
>     2010-08-26T15:45:13.130+0200: 38710.806: [GC 38710.806: [ParNew
>     (promotion failed): 235520K->235520K(235712K), 1.0952192
>     secs]38711.901: [CMS2010-08-26T15:45:14.222+0200: 38711.903:
>     [CMS-concurrent-mark: 1.934/5.410 secs] [Times: user=8.20
>     sys=0.25, real=5.41 secs] 
>      (concurrent mode failure): 943043K->933853K(943744K), 4.0906398
>     secs] 1172473K->933853K(1179456K), [CMS Perm :
>     97361K->97361K(131072K)] icms_dc=86 , 5.1861184 secs] [Times:
>     user=5.18 sys=0.00, real=5.18 secs]
>
>     After the full collection the heap is still mostly full.
>
>     943043K->933853K(943744K)
>
>     I think your advice to increase the size of the should will help.
>
>
>     On 08/31/10 12:36, Adam Hawthorne wrote:
>>     Hi all,
>>
>>     I'm looking at an i-cms log, and I don't understand why I'm
>>     seeing what I'm seeing.  Any help is appreciated.  At about 
>>     38718.035 in the log below, icms_dc begins to report 100, and
>>     just after that I see several back-to-back concurrent mode
>>     failures.  During that time, the ParNew collections are reporting
>>     that they're doing no work.  This happens on and off for about 20
>>     minutes in the log.  Obviously the collections are taking
>>     seconds, but I don't see a Full GC every time.  What is it doing
>>     when it's not doing a full GC?  Is the continual CMS collection
>>     preventing the young generation from being collected (or perhaps
>>     collecting it simultaneously)?  When icms_dc=100, does that imply
>>     the application is stopped?
>>
>>     There was some uninformed command line twiddling going on here,
>>     so the GC parameters were:
>>     -Xmx1152m
>>     -Xms256m
>>     -XX:NewRatio=4
>>     -XX:MaxPermSize=128m
>>     -XX:+UseParNewGC
>>     -XX:-CMSParallelRemarkEnabled
>>     -XX:+UseConcMarkSweepGC
>>     -XX:+CMSPermGenSweepingEnabled
>>     -XX:+UseCMSCompactAtFullCollection
>>     -XX:CMSFullGCsBeforeCompaction=1
>>     -XX:+CMSClassUnloadingEnabled
>>     -XX:+CMSIncrementalMode
>>     -XX:+CMSIncrementalPacing
>>     -XX:CMSInitiatingOccupancyFraction=50
>>     -XX:ParallelGCThreads=1
>>     -XX:MaxTenuringThreshold=0
>>     -XX:SurvivorRatio=1024
>>     -XX:SoftRefLRUPolicyMSPerMB=1000
>>     -XX:+DisableExplicitGC
>>     -XX:+PrintGCDetails
>>     -XX:+PrintGCTimeStamps
>>     -XX:+PrintGCDateStamps
>>
>>     I had them
>>     remove -XX:ParallelGCThreads=1, -XX:MaxTenuringThreshold=0,
>>     and -XX:SurvivorRatio=1024 , thinking that since the duty cycle
>>     was approaching 100%, and because the ParNew pause times were so
>>     low, we could probably afford to spend more time doing young gc's
>>     instead of promoting.  I'm thinking I should probably tell them
>>     to increase their -Xmx and decrease their NewRatio as well.  Is
>>     there anything else?  I haven't done much tuning of i-cms in
>>     general, but the standard i-cms safety options don't seem like
>>     they would help here.
>>
>>     Here's the log excerpt:
>>
>>     2010-08-26T15:44:42.836+0200: 38680.510: [GC 38680.510: [ParNew:
>>     235520K->0K(235712K), 0.0187551 secs] 1158126K->923156K(1179456K)
>>     icms_dc=41 , 0.0189305 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:43.179+0200: 38680.861: [GC [1 CMS-initial-mark:
>>     923156K(943744K)] 996964K(1179456K), 0.0526219 secs] [Times:
>>     user=0.06 sys=0.00, real=0.06 secs] 
>>     2010-08-26T15:44:43.241+0200: 38680.914: [CMS-concurrent-mark-start]
>>     2010-08-26T15:44:44.021+0200: 38681.695: [GC 38681.695: [ParNew:
>>     235520K->0K(235712K), 0.0196054 secs] 1158676K->923705K(1179456K)
>>     icms_dc=41 , 0.0197785 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:45.160+0200: 38682.831: [GC 38682.831: [ParNew:
>>     235520K->0K(235712K), 0.0199663 secs] 1159225K->924310K(1179456K)
>>     icms_dc=41 , 0.0201448 secs] [Times: user=0.01 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:46.127+0200: 38683.797: [GC 38683.797: [ParNew:
>>     235520K->0K(235712K), 0.0549963 secs] 1159830K->930710K(1179456K)
>>     icms_dc=56 , 0.0551800 secs] [Times: user=0.05 sys=0.00,
>>     real=0.05 secs] 
>>     2010-08-26T15:44:47.562+0200: 38685.246: [CMS-concurrent-mark:
>>     1.903/4.332 secs] [Times: user=6.77 sys=0.25, real=4.32 secs] 
>>     2010-08-26T15:44:47.562+0200: 38685.246:
>>     [CMS-concurrent-preclean-start]
>>     2010-08-26T15:44:47.828+0200: 38685.506: [GC 38685.507: [ParNew:
>>     235520K->0K(235712K), 0.0180149 secs] 1166230K->931150K(1179456K)
>>     icms_dc=56 , 0.0182051 secs] [Times: user=0.02 sys=0.00,
>>     real=0.01 secs] 
>>     2010-08-26T15:44:48.108+0200: 38685.790:
>>     [CMS-concurrent-preclean: 0.023/0.543 secs] [Times: user=0.55
>>     sys=0.05, real=0.55 secs] 
>>     2010-08-26T15:44:48.108+0200: 38685.790:
>>     [CMS-concurrent-abortable-preclean-start]
>>     2010-08-26T15:44:48.483+0200: 38686.158:
>>     [CMS-concurrent-abortable-preclean: 0.040/0.368 secs] [Times:
>>     user=0.41 sys=0.02, real=0.38 secs] 
>>     2010-08-26T15:44:48.483+0200: 38686.159: [GC[YG occupancy: 136856
>>     K (235712 K)]38686.159: [Rescan (non-parallel) 38686.159: [grey
>>     object rescan, 0.0041031 secs]38686.163: [root rescan, 0.1153851
>>     secs], 0.1195876 secs]38686.279: [weak refs processing, 0.0012088
>>     secs]38686.280: [class unloading, 0.0143079 secs]38686.294:
>>     [scrub symbol & string tables, 0.0159790 secs] [1 CMS-remark:
>>     931150K(943744K)] 1068006K(1179456K), 0.1563910 secs] [Times:
>>     user=0.16 sys=0.00, real=0.16 secs] 
>>     2010-08-26T15:44:48.639+0200: 38686.316: [CMS-concurrent-sweep-start]
>>     2010-08-26T15:44:49.138+0200: 38686.808: [GC 38686.808: [ParNew:
>>     235520K->0K(235712K), 0.0183463 secs] 1166540K->931466K(1179456K)
>>     icms_dc=56 , 0.0185256 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:49.637+0200: 38687.307: [CMS-concurrent-sweep:
>>     0.470/0.992 secs] [Times: user=1.47 sys=0.02, real=1.00 secs] 
>>     2010-08-26T15:44:49.637+0200: 38687.307: [CMS-concurrent-reset-start]
>>     2010-08-26T15:44:49.637+0200: 38687.313: [CMS-concurrent-reset:
>>     0.005/0.005 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
>>     2010-08-26T15:44:50.277+0200: 38687.948: [GC 38687.948: [ParNew:
>>     235520K->0K(235712K), 0.0177987 secs] 1158207K->923110K(1179456K)
>>     icms_dc=42 , 0.0179840 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:50.604+0200: 38688.280: [GC [1 CMS-initial-mark:
>>     923110K(943744K)] 992484K(1179456K), 0.0509414 secs] [Times:
>>     user=0.05 sys=0.00, real=0.05 secs] 
>>     2010-08-26T15:44:50.651+0200: 38688.331: [CMS-concurrent-mark-start]
>>     2010-08-26T15:44:51.478+0200: 38689.157: [GC 38689.157: [ParNew:
>>     235520K->0K(235712K), 0.0178124 secs] 1158630K->923567K(1179456K)
>>     icms_dc=42 , 0.0179821 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:52.632+0200: 38690.306: [GC 38690.306: [ParNew:
>>     235520K->0K(235712K), 0.0180970 secs] 1159087K->924012K(1179456K)
>>     icms_dc=42 , 0.0182661 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:53.740+0200: 38691.422: [GC 38691.422: [ParNew:
>>     235520K->0K(235712K), 0.0267182 secs] 1159532K->925696K(1179456K)
>>     icms_dc=42 , 0.0269020 secs] [Times: user=0.03 sys=0.00,
>>     real=0.03 secs] 
>>     2010-08-26T15:44:54.520+0200: 38692.201: [CMS-concurrent-mark:
>>     1.894/3.870 secs] [Times: user=5.74 sys=0.13, real=3.87 secs] 
>>     2010-08-26T15:44:54.520+0200: 38692.201:
>>     [CMS-concurrent-preclean-start]
>>     2010-08-26T15:44:54.551+0200: 38692.222:
>>     [CMS-concurrent-preclean: 0.015/0.020 secs] [Times: user=0.05
>>     sys=0.02, real=0.03 secs] 
>>     2010-08-26T15:44:54.551+0200: 38692.222:
>>     [CMS-concurrent-abortable-preclean-start]
>>     2010-08-26T15:44:54.894+0200: 38692.573: [GC 38692.573: [ParNew:
>>     235520K->0K(235712K), 0.0186037 secs] 1161216K->926149K(1179456K)
>>     icms_dc=42 , 0.0187843 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:55.581+0200: 38693.250:
>>     [CMS-concurrent-abortable-preclean: 0.050/1.028 secs] [Times:
>>     user=1.05 sys=0.01, real=1.03 secs] 
>>     2010-08-26T15:44:55.581+0200: 38693.251: [GC[YG occupancy: 140686
>>     K (235712 K)]38693.251: [Rescan (non-parallel) 38693.251: [grey
>>     object rescan, 0.0043953 secs]38693.255: [root rescan, 0.1209735
>>     secs], 0.1254680 secs]38693.376: [weak refs processing, 0.0013433
>>     secs]38693.378: [class unloading, 0.0142815 secs]38693.392:
>>     [scrub symbol & string tables, 0.0160091 secs] [1 CMS-remark:
>>     926149K(943744K)] 1066835K(1179456K), 0.1624119 secs] [Times:
>>     user=0.16 sys=0.00, real=0.16 secs] 
>>     2010-08-26T15:44:55.737+0200: 38693.414: [CMS-concurrent-sweep-start]
>>     2010-08-26T15:44:56.205+0200: 38693.879: [GC 38693.880: [ParNew:
>>     235520K->0K(235712K), 0.0188069 secs] 1161648K->926599K(1179456K)
>>     icms_dc=42 , 0.0189967 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:56.891+0200: 38694.563: [CMS-concurrent-sweep:
>>     0.474/1.149 secs] [Times: user=1.64 sys=0.02, real=1.15 secs] 
>>     2010-08-26T15:44:56.891+0200: 38694.563: [CMS-concurrent-reset-start]
>>     2010-08-26T15:44:56.891+0200: 38694.569: [CMS-concurrent-reset:
>>     0.006/0.006 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
>>     2010-08-26T15:44:57.343+0200: 38695.018: [GC 38695.018: [ParNew:
>>     235520K->0K(235712K), 0.0187471 secs] 1156567K->921554K(1179456K)
>>     icms_dc=32 , 0.0189332 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:57.718+0200: 38695.402: [GC [1 CMS-initial-mark:
>>     921554K(943744K)] 1001837K(1179456K), 0.0572313 secs] [Times:
>>     user=0.06 sys=0.00, real=0.06 secs] 
>>     2010-08-26T15:44:57.780+0200: 38695.460: [CMS-concurrent-mark-start]
>>     2010-08-26T15:44:58.545+0200: 38696.224: [GC 38696.225: [ParNew:
>>     235520K->0K(235712K), 0.0183404 secs] 1157074K->922009K(1179456K)
>>     icms_dc=32 , 0.0185097 secs] [Times: user=0.02 sys=0.00,
>>     real=0.01 secs] 
>>     2010-08-26T15:45:00.167+0200: 38697.850: [GC 38697.850: [ParNew:
>>     235520K->0K(235712K), 0.0196135 secs] 1157529K->922547K(1179456K)
>>     icms_dc=32 , 0.0197977 secs] [Times: user=0.03 sys=0.00,
>>     real=0.03 secs] 
>>     2010-08-26T15:45:01.321+0200: 38698.992: [GC 38698.992: [ParNew:
>>     235520K->0K(235712K), 0.0187905 secs] 1158067K->923023K(1179456K)
>>     icms_dc=32 , 0.0189818 secs] [Times: user=0.01 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:45:02.476+0200: 38700.146: [GC 38700.146: [ParNew:
>>     235520K->0K(235712K), 0.0180207 secs] 1158543K->923471K(1179456K)
>>     icms_dc=32 , 0.0181892 secs] [Times: user=0.01 sys=0.00,
>>     real=0.01 secs] 
>>     2010-08-26T15:45:03.615+0200: 38701.284: [GC 38701.284: [ParNew:
>>     235520K->0K(235712K), 0.0186408 secs] 1158991K->923976K(1179456K)
>>     icms_dc=32 , 0.0188287 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:45:04.051+0200: 38701.735: [CMS-concurrent-mark:
>>     1.899/6.275 secs] [Times: user=8.16 sys=0.06, real=6.27 secs] 
>>     2010-08-26T15:45:04.051+0200: 38701.735:
>>     [CMS-concurrent-preclean-start]
>>     2010-08-26T15:45:04.083+0200: 38701.756:
>>     [CMS-concurrent-preclean: 0.016/0.021 secs] [Times: user=0.05
>>     sys=0.02, real=0.03 secs] 
>>     2010-08-26T15:45:04.083+0200: 38701.756:
>>     [CMS-concurrent-abortable-preclean-start]
>>     2010-08-26T15:45:04.753+0200: 38702.434: [GC 38702.435: [ParNew:
>>     235520K->0K(235712K), 0.0184004 secs] 1159496K->924427K(1179456K)
>>     icms_dc=32 , 0.0185851 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:45:05.409+0200: 38703.087:
>>     [CMS-concurrent-abortable-preclean: 0.063/1.331 secs] [Times:
>>     user=1.39 sys=0.03, real=1.33 secs] 
>>     2010-08-26T15:45:05.409+0200: 38703.088: [GC[YG occupancy: 141725
>>     K (235712 K)]38703.088: [Rescan (non-parallel) 38703.088: [grey
>>     object rescan, 0.0041240 secs]38703.093: [root rescan, 0.1200361
>>     secs], 0.1242579 secs]38703.213: [weak refs processing, 0.0010203
>>     secs]38703.214: [class unloading, 0.0142923 secs]38703.228:
>>     [scrub symbol & string tables, 0.0170438 secs] [1 CMS-remark:
>>     924427K(943744K)] 1066152K(1179456K), 0.1618864 secs] [Times:
>>     user=0.17 sys=0.00, real=0.17 secs] 
>>     2010-08-26T15:45:05.580+0200: 38703.250: [CMS-concurrent-sweep-start]
>>     2010-08-26T15:45:06.017+0200: 38703.687: [GC 38703.687: [ParNew:
>>     235520K->0K(235712K), 0.0425827 secs] 1159944K->929053K(1179456K)
>>     icms_dc=32 , 0.0427672 secs] [Times: user=0.03 sys=0.00,
>>     real=0.03 secs] 
>>     2010-08-26T15:45:07.202+0200: 38704.873: [GC 38704.873: [ParNew:
>>     235520K->0K(235712K), 0.0179344 secs] 1161562K->926484K(1179456K)
>>     icms_dc=32 , 0.0181045 secs] [Times: user=0.01 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:45:07.608+0200: 38705.282: [CMS-concurrent-sweep:
>>     0.472/2.032 secs] [Times: user=2.53 sys=0.08, real=2.03 secs] 
>>     2010-08-26T15:45:07.608+0200: 38705.282: [CMS-concurrent-reset-start]
>>     2010-08-26T15:45:07.608+0200: 38705.288: [CMS-concurrent-reset:
>>     0.005/0.005 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
>>     2010-08-26T15:45:08.341+0200: 38706.012: [GC 38706.012: [ParNew:
>>     235520K->0K(235712K), 0.0180667 secs] 1162004K->926949K(1179456K)
>>     icms_dc=24 , 0.0182417 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:45:08.747+0200: 38706.425: [GC [1 CMS-initial-mark:
>>     926949K(943744K)] 1019208K(1179456K), 0.0677621 secs] [Times:
>>     user=0.06 sys=0.00, real=0.06 secs] 
>>     2010-08-26T15:45:08.809+0200: 38706.493: [CMS-concurrent-mark-start]
>>     2010-08-26T15:45:09.480+0200: 38707.158: [GC 38707.159: [ParNew:
>>     235520K->0K(235712K), 0.0477314 secs] 1162469K->932300K(1179456K)
>>     icms_dc=39 , 0.0479217 secs] [Times: user=0.05 sys=0.00,
>>     real=0.05 secs] 
>>     2010-08-26T15:45:10.463+0200: 38708.133: [GC 38708.133: [ParNew:
>>     235520K->0K(235712K), 0.0371965 secs] 1167820K->936491K(1179456K)
>>     icms_dc=54 , 0.0373840 secs] [Times: user=0.03 sys=0.00,
>>     real=0.03 secs] 
>>     2010-08-26T15:45:12.085+0200: 38709.770: [GC 38709.770: [ParNew:
>>     235520K->0K(235712K), 0.0185757 secs] 1172011K->936953K(1179456K)
>>     icms_dc=69 , 0.0187606 secs] [Times: user=0.03 sys=0.00,
>>     real=0.03 secs] 
>>     2010-08-26T15:45:13.130+0200: 38710.806: [GC 38710.806: [ParNew
>>     (promotion failed): 235520K->235520K(235712K), 1.0952192
>>     secs]38711.901: [CMS2010-08-26T15:45:14.222+0200: 38711.903:
>>     [CMS-concurrent-mark: 1.934/5.410 secs] [Times: user=8.20
>>     sys=0.25, real=5.41 secs] 
>>      (concurrent mode failure): 943043K->933853K(943744K), 4.0906398
>>     secs] 1172473K->933853K(1179456K), [CMS Perm :
>>     97361K->97361K(131072K)] icms_dc=86 , 5.1861184 secs] [Times:
>>     user=5.18 sys=0.00, real=5.18 secs] 
>>     2010-08-26T15:45:18.372+0200: 38716.052: [GC [1 CMS-initial-mark:
>>     933853K(943744K)] 950390K(1179456K), 0.0155216 secs] [Times:
>>     user=0.01 sys=0.00, real=0.02 secs] 
>>     2010-08-26T15:45:18.387+0200: 38716.068: [CMS-concurrent-mark-start]
>>     2010-08-26T15:45:19.355+0200: 38717.035: [GC 38717.035: [ParNew:
>>     235520K->0K(235712K), 0.0623713 secs] 1169373K->942749K(1179456K)
>>     icms_dc=100 , 0.0625660 secs] [Times: user=0.06 sys=0.00,
>>     real=0.06 secs] 
>>     2010-08-26T15:45:20.509+0200: 38718.182: [CMS-concurrent-mark:
>>     1.964/2.114 secs] [Times: user=4.57 sys=0.11, real=2.12 secs] 
>>     2010-08-26T15:45:20.509+0200: 38718.182:
>>     [CMS-concurrent-preclean-start]
>>     2010-08-26T15:45:20.525+0200: 38718.202:
>>     [CMS-concurrent-preclean: 0.015/0.020 secs] [Times: user=0.02
>>     sys=0.02, real=0.02 secs] 
>>     2010-08-26T15:45:20.525+0200: 38718.202:
>>     [CMS-concurrent-abortable-preclean-start]
>>     2010-08-26T15:45:20.540+0200: 38718.222: [GC 38718.222: [ParNew:
>>     235520K->235520K(235712K), 0.0000592 secs]38718.222:
>>     [CMS2010-08-26T15:45:20.540+0200: 38718.223:
>>     [CMS-concurrent-abortable-preclean: 0.004/0.021 secs] [Times:
>>     user=0.02 sys=0.00, real=0.01 secs] 
>>      (concurrent mode failure): 942749K->936038K(943744K), 4.0650806
>>     secs] 1178269K->936038K(1179456K), [CMS Perm :
>>     97371K->97371K(131072K)] icms_dc=100 , 4.0653451 secs] [Times:
>>     user=4.07 sys=0.00, real=4.07 secs] 
>>     2010-08-26T15:45:24.612+0200: 38722.289: [GC [1 CMS-initial-mark:
>>     936038K(943744K)] 936042K(1179456K), 0.0023357 secs] [Times:
>>     user=0.00 sys=0.00, real=0.00 secs] 
>>     2010-08-26T15:45:24.612+0200: 38722.291: [CMS-concurrent-mark-start]
>>     2010-08-26T15:45:26.141+0200: 38723.817: [GC 38723.817: [ParNew:
>>     235520K->235520K(235712K), 0.0000588 secs]38723.817:
>>     [CMS2010-08-26T15:45:27.030+0200: 38724.709:
>>     [CMS-concurrent-mark: 1.935/2.418 secs] [Times: user=3.65
>>     sys=0.19, real=2.42 secs] 
>>      (concurrent mode failure): 936038K->942227K(943744K), 4.9718511
>>     secs] 1171558K->942227K(1179456K), [CMS Perm :
>>     97393K->97393K(131072K)] icms_dc=100 , 4.9721115 secs] [Times:
>>     user=4.98 sys=0.00, real=4.98 secs] 
>>     2010-08-26T15:45:31.117+0200: 38728.790: [GC [1 CMS-initial-mark:
>>     942227K(943744K)] 942270K(1179456K), 0.0025839 secs] [Times:
>>     user=0.00 sys=0.00, real=0.00 secs] 
>>     2010-08-26T15:45:31.117+0200: 38728.793: [CMS-concurrent-mark-start]
>>     2010-08-26T15:45:32.084+0200: 38729.758: [GC 38729.759: [ParNew:
>>     235520K->235520K(235712K), 0.0000623 secs]38729.759:
>>     [CMS2010-08-26T15:45:33.051+0200: 38730.736:
>>     [CMS-concurrent-mark: 1.934/1.943 secs] [Times: user=3.28
>>     sys=0.19, real=1.93 secs] 
>>      (concurrent mode failure): 942227K->943743K(943744K), 4.9442127
>>     secs] 1177747K->948024K(1179456K), [CMS Perm :
>>     97396K->97396K(131072K)] icms_dc=100 , 4.9446265 secs] [Times:
>>     user=4.91 sys=0.02, real=4.94 secs] 
>>     2010-08-26T15:45:37.029+0200: 38734.704: [GC [1 CMS-initial-mark:
>>     943743K(943744K)] 948026K(1179456K), 0.0067733 secs] [Times:
>>     user=0.00 sys=0.00, real=0.00 secs] 
>>     2010-08-26T15:45:37.029+0200: 38734.711: [CMS-concurrent-mark-start]
>>     2010-08-26T15:45:38.745+0200: 38736.419: [Full GC 38736.419:
>>     [CMS2010-08-26T15:45:39.463+0200: 38737.138:
>>     [CMS-concurrent-mark: 1.952/2.427 secs] [Times: user=3.95
>>     sys=0.16, real=2.43 secs] 
>>      (concurrent mode failure): 943743K->943743K(943744K), 4.7856092
>>     secs] 1179455K->949021K(1179456K), [CMS Perm :
>>     97406K->97406K(131072K)] icms_dc=100 , 4.7857927 secs] [Times:
>>     user=4.79 sys=0.00, real=4.79 secs] 
>>     2010-08-26T15:45:43.534+0200: 38741.206: [GC [1 CMS-initial-mark:
>>     943743K(943744K)] 949287K(1179456K), 0.0080127 secs] [Times:
>>     user=0.00 sys=0.00, real=0.00 secs] 
>>     2010-08-26T15:45:43.534+0200: 38741.214: [CMS-concurrent-mark-start]
>>     2010-08-26T15:45:44.455+0200: 38742.136: [Full GC 38742.136:
>>     [CMS2010-08-26T15:45:45.484+0200: 38743.168:
>>     [CMS-concurrent-mark: 1.952/1.954 secs] [Times: user=3.26
>>     sys=0.05, real=1.95 secs] 
>>      (concurrent mode failure): 943743K->943743K(943744K), 4.9692415
>>     secs] 1179440K->949759K(1179456K), [CMS Perm :
>>     97406K->97406K(131072K)] icms_dc=100 , 4.9694250 secs] [Times:
>>     user=4.96 sys=0.02, real=4.98 secs] 
>>     2010-08-26T15:45:49.883+0200: 38747.569: [GC [1 CMS-initial-mark:
>>     943743K(943744K)] 953372K(1179456K), 0.0086197 secs] [Times:
>>     user=0.02 sys=0.00, real=0.02 secs] 
>>     2010-08-26T15:45:49.899+0200: 38747.578: [CMS-concurrent-mark-start]
>>     2010-08-26T15:45:50.960+0200: 38748.634: [Full GC 38748.634:
>>     [CMS2010-08-26T15:45:51.833+0200: 38749.515:
>>     [CMS-concurrent-mark: 1.927/1.938 secs] [Times: user=3.23
>>     sys=0.11, real=1.93 secs] 
>>      (concurrent mode failure): 943743K->937800K(943744K), 4.9818453
>>     secs] 1179455K->937800K(1179456K), [CMS Perm :
>>     97407K->97407K(131072K)] icms_dc=100 , 4.9820296 secs] [Times:
>>     user=4.95 sys=0.02, real=4.98 secs] 
>>     2010-08-26T15:45:55.936+0200: 38753.617: [GC [1 CMS-initial-mark:
>>     937800K(943744K)] 938528K(1179456K), 0.0024408 secs] [Times:
>>     user=0.00 sys=0.00, real=0.00 secs] 
>>     2010-08-26T15:45:55.936+0200: 38753.620: [CMS-concurrent-mark-start]
>>     2010-08-26T15:45:56.950+0200: 38754.630: [GC 38754.631: [ParNew:
>>     235520K->235520K(235712K), 0.0000597 secs]38754.631:
>>     [CMS2010-08-26T15:45:57.995+0200: 38755.679:
>>     [CMS-concurrent-mark: 2.057/2.059 secs] [Times: user=3.29
>>     sys=0.23, real=2.06 secs] 
>>      (concurrent mode failure): 937800K->936173K(943744K), 5.1274815
>>     secs] 1173320K->936173K(1179456K), [CMS Perm :
>>     97407K->97407K(131072K)] icms_dc=100 , 5.1277524 secs] [Times:
>>     user=5.13 sys=0.00, real=5.13 secs] 
>>     2010-08-26T15:46:02.535+0200: 38760.215: [GC [1 CMS-initial-mark:
>>     936173K(943744K)] 936184K(1179456K), 0.0024186 secs] [Times:
>>     user=0.00 sys=0.00, real=0.00 secs] 
>>     2010-08-26T15:46:02.535+0200: 38760.218: [CMS-concurrent-mark-start]
>>     2010-08-26T15:46:03.611+0200: 38761.290: [GC 38761.290: [ParNew
>>     (promotion failed): 235520K->235520K(235712K), 0.1302956
>>     secs]38761.420: [CMS2010-08-26T15:46:04.609+0200: 38762.289:
>>     [CMS-concurrent-mark: 1.938/2.071 secs] [Times: user=3.54
>>     sys=0.11, real=2.07 secs] 
>>      (concurrent mode failure): 943229K->939271K(943744K), 4.9629545
>>     secs] 1171693K->939271K(1179456K), [CMS Perm :
>>     97407K->97407K(131072K)] icms_dc=100 , 5.0934938 secs] [Times:
>>     user=5.09 sys=0.00, real=5.10 secs] 
>>
>>     Thanks again for any help,
>>
>>     Adam
>>
>>     --
>>     Adam Hawthorne
>>     Software Engineer
>>     BASIS International Ltd.
>>     www.basis.com <http://www.basis.com>
>>     +1.505.345.5232 Phone
>>     ------------------------------------------------------------------------
>>
>>     _______________________________________________
>>     hotspot-gc-use mailing list
>>     hotspot-gc-use at openjdk.java.net <mailto:hotspot-gc-use at openjdk.java.net>
>>     http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>       
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100831/1bc6f0cb/attachment-0001.html 

From y.s.ramakrishna at oracle.com  Tue Aug 31 16:57:36 2010
From: y.s.ramakrishna at oracle.com (Y. S. Ramakrishna)
Date: Tue, 31 Aug 2010 16:57:36 -0700
Subject: Long GC pause time
In-Reply-To: <4C7D81CD.5050706@oracle.com>
References: <SNT108-W11C7575D28055DFA3A6544E38A0@phx.gbl>	<4C7D49E4.6060908@oracle.com>
	<4C7D81CD.5050706@oracle.com>
Message-ID: <4C7D96F0.1060600@oracle.com>

oh, and -XX:+PrintReferenceGC to get more detail on
what kind of Reference objects are causing the issue.

-- ramki

On 08/31/10 15:27, Y. S. Ramakrishna wrote:
> Noam, also please provision and use the survivor spaces to age
> objects and reduce pressure on CMS. The premature
> promotion and subsequent death is probably exacerbating the
> proliferating Ref object deaths (if i may be permitted an
> oxymoron) in the old gen.
> 
> -- ramki
> 
> On 08/31/10 11:28, Jon Masamitsu wrote:
>> Try -XX+ParallelRefProcEnabled
>>
>> Your spending much of the pause time doing Reference processing
>>
>> 89.867: [GC[YG occupancy: 420 K (274752 K)]89.867: [Rescan (parallel) , 
>> 0.0242770 secs]89.892: [weak refs processing, 1.3860230 secs] [1 
>> CMS-remark: 4198190K(6635136K)] 4198611K(6909888K), 1.4164670 secs] 
>> [Times: user=1.68 sys=0.01, real=1.41 secs]
>>
>> (see the "[weak refs processing, 1.3860230 secs]") so doing it in 
>> parallel might
>> help.  ParallelRefProcEnabled isn't always a win so it's not "on" by 
>> default.
>>
>> You might also try -XX:+CMSScavengeBeforeRemark.  That's a just a guess.
>> Sometimes doing a minor collection before the remark helps.
>>
>> On 08/31/10 08:22, noam katzowicz wrote:
>>> Hello.
>>>  
>>> We are running JBoss AS over RH5, using java 6u21, with the following 
>>> GC params:
>>>  
>>> -Xms6750m -Xmx6750m
>>> -XX:+UseConcMarkSweepGC
>>> -XX:SurvivorRatio=128
>>> -XX:MaxTenuringThreshold=0
>>> -XX:+UseCompressedOops
>>> -XX:CMSInitiatingOccupancyFraction=60
>>> -Dsun.rmi.dgc.client.gcInterval=3600000
>>> -Dsun.rmi.dgc.server.gcInterval=3600000
>>> -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+PrintTenuringDistribution
>>>  
>>> We are experiencing relatively long GC pause time of over 1 second.
>>> Our application allocates hundreds of MBs per second. We?ve tried 
>>> different generation sizes but the above was the best we came up with. 
>>> Your help is appreciated.
>>>  
>>>
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0596370 secs] 4422502K->4173928K(6909888K), 
>>> 0.0606730 secs] [Times: user=0.44 sys=0.03, real=0.06 secs]
>>> 89.486: [CMS-concurrent-preclean: 1.308/1.649 secs] [Times: user=10.32 
>>> sys=2.30, real=1.65 secs]
>>> 89.487: [CMS-concurrent-abortable-preclean-start]
>>> 89.804: [GC 89.804: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0603670 secs] 4446568K->4198190K(6909888K), 
>>> 0.0614310 secs] [Times: user=0.48 sys=0.03, real=0.07 secs]
>>> 89.866: [CMS-concurrent-abortable-preclean: 0.293/0.379 secs] [Times: 
>>> user=2.42 sys=0.54, real=0.38 secs]
>>> 89.867: [GC[YG occupancy: 420 K (274752 K)]89.867: [Rescan (parallel) 
>>> , 0.0242770 secs]89.892: [weak refs processing, 1.3860230 secs] [1 
>>> CMS-remark: 4198190K(6635136K)] 4198611K(6909888K), 1.4164670 secs] 
>>> [Times: user=1.68 sys=0.01, real=1.41 secs]
>>> 91.284: [CMS-concurrent-sweep-start]
>>> 91.590: [GC 91.591: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0675550 secs] 4371287K->4128966K(6909888K), 
>>> 0.0683470 secs] [Times: user=0.56 sys=0.01, real=0.07 secs]
>>> 91.860: [GC 91.860: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0604320 secs] 4308368K->4053829K(6909888K), 
>>> 0.0612020 secs] [Times: user=0.41 sys=0.01, real=0.06 secs]
>>> 92.786: [GC 92.787: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.1028730 secs] 3964957K->3753351K(6909888K), 
>>> 0.1038490 secs] [Times: user=0.80 sys=0.02, real=0.10 secs]
>>> 93.106: [GC 93.107: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0487000 secs] 3949085K->3699496K(6909888K), 
>>> 0.0494470 secs] [Times: user=0.32 sys=0.01, real=0.05 secs]
>>> 93.371: [GC 93.371: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0430620 secs] 3905862K->3652769K(6909888K), 
>>> 0.0438800 secs] [Times: user=0.33 sys=0.01, real=0.05 secs]
>>> 94.637: [GC 94.638: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0564930 secs] 3416325K->3164258K(6909888K), 
>>> 0.0575000 secs] [Times: user=0.44 sys=0.02, real=0.06 secs]
>>> 94.863: [CMS-concurrent-sweep: 3.107/3.578 secs] [Times: user=16.85 
>>> sys=3.23, real=3.58 secs]
>>> 94.863: [CMS-concurrent-reset-start]
>>> 94.878: [CMS-concurrent-reset: 0.015/0.015 secs] [Times: user=0.03 
>>> sys=0.01, real=0.02 secs]
>>> 95.981: [GC 95.981: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0514980 secs] 3405940K->3152334K(6909888K), 
>>> 0.0524610 secs] [Times: user=0.44 sys=0.01, real=0.05 secs]
>>> 100.179: [GC 100.180: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0676510 secs] 3424974K->3178150K(6909888K), 
>>> 0.0686240 secs] [Times: user=0.58 sys=0.00, real=0.07 secs]
>>> 100.909: [GC 100.909: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0707060 secs] 3450790K->3202638K(6909888K), 
>>> 0.0717030 secs] [Times: user=0.55 sys=0.01, real=0.07 secs]
>>> 101.613: [GC 101.613: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272636K->0K(274752K), 0.0667270 secs] 3475274K->3224061K(6909888K), 
>>> 0.0678240 secs] [Times: user=0.50 sys=0.01, real=0.07 secs]
>>> 102.072: [GC 102.073: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272635K->0K(274752K), 0.0590350 secs] 3496696K->3241246K(6909888K), 
>>> 0.0601190 secs] [Times: user=0.44 sys=0.00, real=0.06 secs]
>>> 102.756: [GC 102.756: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272613K->0K(274752K), 0.0630060 secs] 3513860K->3262602K(6909888K), 
>>> 0.0640320 secs] [Times: user=0.49 sys=0.00, real=0.07 secs]
>>> 103.452: [GC 103.453: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0533390 secs] 3535242K->3280184K(6909888K), 
>>> 0.0543760 secs] [Times: user=0.41 sys=0.01, real=0.06 secs]
>>> 104.462: [GC 104.462: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0729890 secs] 3552824K->3299003K(6909888K), 
>>> 0.0740260 secs] [Times: user=0.48 sys=0.01, real=0.07 secs]
>>> 106.274: [GC 106.274: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0609880 secs] 3571643K->3314370K(6909888K), 
>>> 0.0619510 secs] [Times: user=0.41 sys=0.00, real=0.06 secs]
>>> 108.154: [GC 108.155: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.1251050 secs] 3587010K->3343996K(6909888K), 
>>> 0.1261630 secs] [Times: user=0.71 sys=0.02, real=0.13 secs]
>>> 108.882: [GC 108.883: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0952060 secs] 3616636K->3369490K(6909888K), 
>>> 0.0962930 secs] [Times: user=0.64 sys=0.01, real=0.10 secs]
>>> 109.482: [GC 109.482: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272936K->0K(274752K), 0.0791270 secs] 3642426K->3392040K(6909888K), 
>>> 0.0800830 secs] [Times: user=0.53 sys=0.01, real=0.08 secs]
>>> 109.948: [GC 109.949: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 273038K->0K(274752K), 0.0835410 secs] 3665079K->3417813K(6909888K), 
>>> 0.0845680 secs] [Times: user=0.57 sys=0.00, real=0.08 secs]
>>> 110.035: [GC 110.035: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 64K->0K(274752K), 0.0088490 secs] 3417877K->3417828K(6909888K), 
>>> 0.0093860 secs] [Times: user=0.09 sys=0.00, real=0.01 secs]
>>> 110.484: [GC 110.485: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0880810 secs] 3690468K->3443727K(6909888K), 
>>> 0.0890560 secs] [Times: user=0.59 sys=0.00, real=0.09 secs]
>>> 110.910: [GC 110.910: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0860790 secs] 3716367K->3475451K(6909888K), 
>>> 0.0870070 secs] [Times: user=0.58 sys=0.02, real=0.08 secs]
>>> 111.276: [GC 111.277: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272639K->0K(274752K), 0.0772230 secs] 3748091K->3503686K(6909888K), 
>>> 0.0782780 secs] [Times: user=0.47 sys=0.02, real=0.08 secs]
>>> 111.658: [GC 111.659: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0754700 secs] 3776326K->3532682K(6909888K), 
>>> 0.0763440 secs] [Times: user=0.54 sys=0.01, real=0.07 secs]
>>> 112.068: [GC 112.068: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0705680 secs] 3805322K->3559619K(6909888K), 
>>> 0.0714610 secs] [Times: user=0.47 sys=0.02, real=0.07 secs]
>>> 112.497: [GC 112.498: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0779750 secs] 3832259K->3589461K(6909888K), 
>>> 0.0788830 secs] [Times: user=0.50 sys=0.01, real=0.08 secs]
>>> 112.911: [GC 112.911: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0728900 secs] 3862101K->3616744K(6909888K), 
>>> 0.0738560 secs] [Times: user=0.52 sys=0.00, real=0.07 secs]
>>> 113.362: [GC 113.363: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0609030 secs] 3889384K->3638477K(6909888K), 
>>> 0.0619050 secs] [Times: user=0.44 sys=0.01, real=0.06 secs]
>>> 113.829: [GC 113.829: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272614K->0K(274752K), 0.0641940 secs] 3911091K->3659305K(6909888K), 
>>> 0.0653490 secs] [Times: user=0.48 sys=0.01, real=0.06 secs]
>>> 114.311: [GC 114.311: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0618170 secs] 3931945K->3681619K(6909888K), 
>>> 0.0626970 secs] [Times: user=0.50 sys=0.01, real=0.06 secs]
>>> 114.841: [GC 114.842: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0670890 secs] 3954259K->3711036K(6909888K), 
>>> 0.0680540 secs] [Times: user=0.52 sys=0.02, real=0.07 secs]
>>> 115.356: [GC 115.357: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0623700 secs] 3983676K->3737945K(6909888K), 
>>> 0.0633630 secs] [Times: user=0.49 sys=0.00, real=0.07 secs]
>>> 115.829: [GC 115.830: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0665900 secs] 4010585K->3766876K(6909888K), 
>>> 0.0676240 secs] [Times: user=0.51 sys=0.01, real=0.06 secs]
>>> 116.271: [GC 116.271: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272630K->0K(274752K), 0.0594280 secs] 4039506K->3796861K(6909888K), 
>>> 0.0603620 secs] [Times: user=0.46 sys=0.02, real=0.06 secs]
>>> 116.716: [GC 116.717: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0724150 secs] 4069501K->3825191K(6909888K), 
>>> 0.0733430 secs] [Times: user=0.52 sys=0.01, real=0.08 secs]
>>> 117.145: [GC 117.146: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0664990 secs] 4097831K->3853057K(6909888K), 
>>> 0.0675760 secs] [Times: user=0.48 sys=0.00, real=0.07 secs]
>>> 117.543: [GC 117.545: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0650350 secs] 4125697K->3880562K(6909888K), 
>>> 0.0673450 secs] [Times: user=0.49 sys=0.01, real=0.07 secs]
>>> 117.956: [GC 117.956: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0664410 secs] 4153202K->3907935K(6909888K), 
>>> 0.0673980 secs] [Times: user=0.49 sys=0.02, real=0.07 secs]
>>> 118.385: [GC 118.386: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272631K->0K(274752K), 0.0655070 secs] 4180566K->3934925K(6909888K), 
>>> 0.0664120 secs] [Times: user=0.50 sys=0.01, real=0.07 secs]
>>> 118.826: [GC 118.827: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272608K->0K(274752K), 0.0605390 secs] 4207533K->3964211K(6909888K), 
>>> 0.0616050 secs] [Times: user=0.50 sys=0.02, real=0.06 secs]
>>> 119.233: [GC 119.234: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0656870 secs] 4236851K->3991065K(6909888K), 
>>> 0.0666900 secs] [Times: user=0.50 sys=0.02, real=0.07 secs]
>>> 119.301: [GC [1 CMS-initial-mark: 3991065K(6635136K)] 
>>> 3991071K(6909888K), 0.0091650 secs] [Times: user=0.01 sys=0.00, 
>>> real=0.01 secs]
>>> 119.311: [CMS-concurrent-mark-start]
>>> 119.699: [GC 119.699: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0647090 secs] 4263705K->4019961K(6909888K), 
>>> 0.0656290 secs] [Times: user=0.53 sys=0.01, real=0.06 secs]
>>> 120.081: [GC 120.082: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0641110 secs] 4292601K->4048334K(6909888K), 
>>> 0.0649070 secs] [Times: user=0.50 sys=0.01, real=0.06 secs]
>>> 120.538: [GC 120.539: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0677200 secs] 4320974K->4075491K(6909888K), 
>>> 0.0687990 secs] [Times: user=0.49 sys=0.03, real=0.07 secs]
>>> 120.621: [CMS-concurrent-mark: 1.076/1.310 secs] [Times: user=10.66 
>>> sys=1.73, real=1.31 secs]
>>> 120.621: [CMS-concurrent-preclean-start]
>>> 120.942: [GC 120.943: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0736770 secs] 4348131K->4104969K(6909888K), 
>>> 0.0747430 secs] [Times: user=0.51 sys=0.03, real=0.07 secs]
>>> 121.337: [GC 121.338: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0569980 secs] 4377609K->4126623K(6909888K), 
>>> 0.0579070 secs] [Times: user=0.48 sys=0.03, real=0.06 secs]
>>> 121.766: [GC 121.767: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272615K->0K(274752K), 0.0590140 secs] 4399238K->4148770K(6909888K), 
>>> 0.0599510 secs] [Times: user=0.44 sys=0.03, real=0.06 secs]
>>> 122.174: [GC 122.175: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0531420 secs] 4421410K->4167895K(6909888K), 
>>> 0.0540310 secs] [Times: user=0.45 sys=0.03, real=0.05 secs]
>>> 122.257: [CMS-concurrent-preclean: 1.306/1.636 secs] [Times: 
>>> user=10.25 sys=2.21, real=1.63 secs]
>>> 122.257: [CMS-concurrent-abortable-preclean-start]
>>> 122.631: [GC 122.631: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0528220 secs] 4440535K->4187654K(6909888K), 
>>> 0.0537160 secs] [Times: user=0.46 sys=0.02, real=0.05 secs]
>>> 122.696: [CMS-concurrent-abortable-preclean: 0.355/0.439 secs] [Times: 
>>> user=2.50 sys=0.52, real=0.44 secs]
>>> 122.698: [GC[YG occupancy: 7874 K (274752 K)]122.698: [Rescan 
>>> (parallel) , 0.0253430 secs]122.723: [weak refs processing, 1.1038100 
>>> secs] [1 CMS-remark: 4187654K(6635136K)] 4195529K(6909888K), 1.1357970 
>>> secs] [Times: user=1.42 sys=0.01, real=1.14 secs]
>>> 123.834: [CMS-concurrent-sweep-start]
>>> 124.060: [GC 124.061: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0690300 secs] 4399395K->4167257K(6909888K), 
>>> 0.0698010 secs] [Times: user=0.52 sys=0.02, real=0.07 secs]
>>>  
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>   
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use