<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi,<div><br></div><div>CMS failures occur as a result of a trend over time. It's almost impossible to recommend a correction from a single incident. that said, Jon's blog entry explains CMS failure very clearly. This the record you've sent suggests that young gen is way too small.. but again, I can't say anything with a single record.</div><div><br></div><div>Regards,</div><div>Kirk</div><div><br></div><div><div>On 2012-01-12, at 1:09 PM, Li Li wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>yesterday, we set the maxNewSize to 256mb. And it works as we expected. but an hours ago, there is a promotion failure and a concurrent mode failure which cost 14s! could anyone explain the gc logs for me?</div><div>
or any documents for the gc log format explanation?</div><div><div><br></div><div>1. Desired survivor size 3342336 bytes, new threshold 5 (max 5) </div><div> it says survivor size is 3mb</div></div><div>2. 58282K->57602K(59072K), 0.0543930 secs]</div>
<div> it says before young gc the memory used is 58282K, after young gc, there are 57602K live objects and the total young space is 59072K</div><div>3. (concurrent mode failure): 7907405K->3086848K(7929856K), 14.3005340 secs] 7961046K->3086848K(7988928K), [CMS Perm : 32296K->31852K(53932K)], 14.3552450 secs] [Times: user=14.53 sys=0.01, real=14.35 secs]</div>
<div> before old gc, 7.9GB is used. after old gc 3GB is alive. total old space is 7.9GB</div><div><br></div><div>in which situation will occur promotion failure and concurrent mode failure?</div><div>from <a href="http://www.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-1/">http://www.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-1/</a></div>
<div>the author says when CMS is doing concurrent work and JVM is asked for more memory. if there isn't any space for new allocation. then it will occur concurrent mode failure and it will stop the world and do a serial old gc.</div>
<div>if there exist enough space but they are fragemented, then a promotion failure will occur.</div><div>am I right?</div><div><br></div><div>2012-01-12T18:27:32.582+0800: [GC [ParNew</div><div>Desired survivor size 3342336 bytes, new threshold 1 (max 5)</div>
<div>- age 1: 4594648 bytes, 4594648 total</div><div>- age 2: 569200 bytes, 5163848 total</div><div>: 58548K->5738K(59072K), 0.0159400 secs] 7958648K->7908502K(7984352K), 0.0160610 secs] [Times: user=0.17 sys=0.00, real=0.02 secs]</div>
<div>2012-01-12T18:27:32.609+0800: [GC [ParNew (promotion failed)</div><div>Desired survivor size 3342336 bytes, new threshold 5 (max 5)</div><div>- age 1: 1666376 bytes, 1666376 total</div><div>: 58282K->57602K(59072K), 0.0543930 secs][CMS2012-01-12T18:27:33.804+0800: [CMS-concurrent-preclean: 14.098/34.323 secs] [Times: user=370.28 sys=5.65, real=34.31 secs]</div>
<div> (concurrent mode failure): 7907405K->3086848K(7929856K), 14.3005340 secs] 7961046K->3086848K(7988928K), [CMS Perm : 32296K->31852K(53932K)], 14.3552450 secs] [Times: user=14.53 sys=0.01, real=14.35 secs]</div>
<br><div class="gmail_quote">On Wed, Jan 11, 2012 at 5:48 PM, Kirk Pepperdine <span dir="ltr"><<a href="mailto:kirk.pepperdine@gmail.com">kirk.pepperdine@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
CMS is not adaptive. To reconfigure heap, for many reasons, you need a full GC to occur. The response to a concurrent mode failure is always a full GC. That gave the JVM the opportunity to resize heap space. If this behaviour isn't happening when it should or is cause other problems it's time to either set the young gen size directly with NewSize or switch to the parallel collector with the adaptive sizing policy turned on. Logic here is that you want to avoid long pauses, use CMS. If CMS is giving you long pauses, than the parallel collector might be a better choice.<br>
<br>
Regards,<br>
Kirk<br>
<div class="HOEnZb"><div class="h5"><br>
On 2012-01-11, at 10:32 AM, Li Li wrote:<br>
<br>
> after a concurrent mode failure. the young generation changed from about 50MB to 1.8GB<br>
> What's the logic behind this?<br>
><br>
> 2012-01-10T22:23:54.544+0800: [GC [ParNew: 55389K->6528K(59072K), 0.0175440 secs] 5886124K->5839323K(6195204K), 0.0177480 secs] [Times: user=0.20 sys=0.00, real=0.01 secs]<br>
> 2012-01-10T22:23:54.575+0800: [GC [ParNew: 59072K->6528K(59072K), 0.0234040 secs] 5891867K->5845823K(6201540K), 0.0236070 secs] [Times: user=0.24 sys=0.00, real=0.02 secs]<br>
> 2012-01-10T22:23:54.612+0800: [GC [ParNew (promotion failed): 59072K->58862K(59072K), 2.3119860 secs][CMS2012-01-10T22:23:57.153+0800: [CMS-concurrent-preclean: 10.999/28.245 secs] [Times: user=290.41 sys=4.65, real=28.24 secs]<br>
> (concurrent mode failure): 5841457K->2063142K(6144000K), 8.8971660 secs] 5898367K->2063142K(6203072K), [CMS Perm : 31369K->31131K(52316K)], 11.2110080 secs] [Times: user=11.73 sys=0.51, real=11.21 secs]<br>
> 2012-01-10T22:24:06.125+0800: [GC [ParNew: 1638400K->46121K(1843200K), 0.0225800 secs] 3701542K->2109263K(7987200K), 0.0228190 secs] [Times: user=0.26 sys=0.02, real=0.02 secs]<br>
> 2012-01-10T22:24:06.357+0800: [GC [ParNew: 1684521K->111262K(1843200K), 0.0381370 secs] 3747663K->2174404K(7987200K), 0.0383860 secs] [Times: user=0.44 sys=0.04, real=0.04 secs]<br>
><br>
</div></div><div class="HOEnZb"><div class="h5">> _______________________________________________<br>
> hotspot-gc-use mailing list<br>
> <a href="mailto:hotspot-gc-use@openjdk.java.net">hotspot-gc-use@openjdk.java.net</a><br>
> <a href="http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use" target="_blank">http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use</a><br>
<br>
</div></div></blockquote></div><br>
_______________________________________________<br>hotspot-gc-use mailing list<br><a href="mailto:hotspot-gc-use@openjdk.java.net">hotspot-gc-use@openjdk.java.net</a><br>http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use<br></blockquote></div><br></body></html>