<div dir="ltr"><div>Well thank you both </div>
<div>I will try these recommendations hoping for the best</div>
<div>Thanks</div>
<div>Dori<br><br></div>
<div class="gmail_quote">On Mon, Nov 8, 2010 at 7:11 PM, Y. S. Ramakrishna <span dir="ltr"><<a href="mailto:y.s.ramakrishna@oracle.com">y.s.ramakrishna@oracle.com</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote">I agree completely with Jon's recommendation. As regards, your question:-
<div class="im"><br><br>
<blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote">
<blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote">
<blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote">I didn't quiet understand what exactly you meant by "scale back the # parallel threads used by the CMS..." is there a parameter I need to set for that ? if yes what should be its value ?<br>
</blockquote></blockquote></blockquote><br></div>On a 4-processor machine, you get a single concurrent marking thread, so<br>you do not need to do anything specific since you are already at the minimum<br># concurrent worker threads used by CMS.<br>
<br>-- ramki
<div class="im"><br><br>On 11/07/10 20:04, Jon Masamitsu wrote:<br></div>
<blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote">
<div class="im">The incremental mode of CMS was implemented for running CMS on machines<br>with a single hardware thread. The idea was that CMS does work concurrently<br>with the application but with a single hardware thread, when CMS ran in<br>
a concurrent phase, it would still be stopping the application (CMS would<br>use the single hardware thread and there would be nothing for the application<br>to run on). So the incremental mode of CMS does the concurrent phases<br>
in increments and gives up the hardware thread to the application on<br>a regular basis. On a 4 processor machine I would not recommend the incremental<br>mode. If you have not tried the regular CMS (no incremental mode),<br>
I recommend that you try it. Overall it is more efficient.<br><br>On 11/6/10 11:32 PM, Dori Rabin wrote:<br></div>
<blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote">
<div class="im">Thanks fo your replies...<br>I am running on jdk1.6.0_04 and OS is Linux 2.4.21-37.ELsmp, I have 4 processors on the machine<br>About sharing the log file it might be a little problematic now so let's discuss it if no other option I also thought of getting rid of the incremental mode but I am afraid of the effect of long pauses on our realtime application due to the stop the world phase of CMS....<br>
I didn't quiet understand what exactly you meant by "scale back the # parallel threads used by the CMS..." is there a parameter I need to set for that ? if yes what should be its value ?<br>Thanks<br>Dori<br>
</div>
<div>
<div></div>
<div class="h5"> On Thu, Nov 4, 2010 at 6:51 PM, Y. S. Ramakrishna <<a href="mailto:y.s.ramakrishna@oracle.com" target="_blank">y.s.ramakrishna@oracle.com</a> <mailto:<a href="mailto:y.s.ramakrishna@oracle.com" target="_blank">y.s.ramakrishna@oracle.com</a>>> wrote:<br>
<br> Hi Dori --<br><br> What's the version of JDK you are running? Can you share a<br> complete log?<br> It appears as though the iCMS "auto-pacing" is, for some reason, not<br> kicking in "soon enough"; one workaround is to use turn off<br>
auto-pacing<br> and use an explicit duty-cycle (which has its own disadvantages).<br><br> I'd suggest filing a bug, and including a complete log file showing<br> the problem.<br><br> thanks.<br> -- ramki<br>
<br> On 11/04/10 06:27, Rabin Dori wrote:<br> > Hi,<br> ><br> > Once in a while and for a reason I cannot understand the CMS<br> kicks up<br> > too late which causes a promotion failure and full GC that takes<br>
very<br> > long (more than 2 minutes which causes other problems)…<br> ><br> > My question is how to tune the gc flags in order to make sure<br> that the<br> > concurrent sweep will always occur in parallel (incremental mode)<br>
> without long pause stop the world but also without reaching its<br> maximum<br> > capacity ?<br> ><br> ><br> ><br> > (I know that in my case the *CMSInitiatingOccupancyFraction*=60 is<br>
> ignored because of the CMSIncrementalMode<br> ><br> > And from looking in the log file we can see that the old generation<br> > reaches size of 835’959K out of 843’000K at the time the concurrent<br>
> failure (I marked this line in red)<br> ><br> ><br> ><br> > *_I am running the jvm with the following parameters :_*<br> ><br> > wrapper.java.additional.4=-XX:NewSize=200m<br> ><br>
> wrapper.java.additional.5=-XX:SurvivorRatio=6<br> ><br> > wrapper.java.additional.6=-XX:MaxTenuringThreshold=4<br> ><br> > wrapper.java.additional.7=-XX:+CMSIncrementalMode<br> ><br> > wrapper.java.additional.8=-XX:+CMSIncrementalPacing<br>
><br> > wrapper.java.additional.9=-XX:+DisableExplicitGC<br> ><br> > wrapper.java.additional.10=-XX:+UseConcMarkSweepGC<br> ><br> > wrapper.java.additional.11=-XX:+CMSClassUnloadingEnabled<br>
><br> > wrapper.java.additional.12=-XX:+PrintGCDetails<br> ><br> > wrapper.java.additional.13=-XX:+PrintGCTimeStamps<br> ><br> > wrapper.java.additional.14=-XX:-TraceClassUnloading<br> ><br>
> wrapper.java.additional.15=-XX:+HeapDumpOnOutOfMemoryError<br> ><br> > wrapper.java.additional.16=-verbose:gc<br> ><br> > wrapper.java.additional.17=-XX:CMSInitiatingOccupancyFraction=60<br> ><br>
> wrapper.java.additional.18=-XX:+UseCMSInitiatingOccupancyOnly<br> ><br> > wrapper.java.additional.19=-XX:+PrintTenuringDistribution<br> ><br> ><br> ><br> ><br> ><br> > *_Extracts from the log file:_*<br>
><br> > INFO | jvm 1 | 2010/11/02 04:54:33 | Desired survivor size<br> 13107200<br> > bytes, new threshold 4 (max 4)<br> ><br> > INFO | jvm 1 | 2010/11/02 04:54:33 | - age 1: 544000<br>
> bytes, 544000 total<br> ><br> > INFO | jvm 1 | 2010/11/02 04:54:33 | - age 2: 346320<br> > bytes, 890320 total<br> ><br> > INFO | jvm 1 | 2010/11/02 04:54:33 | - age 3: 262800<br>
> bytes, 1153120 total<br> ><br> > INFO | jvm 1 | 2010/11/02 04:54:33 | - age 4: 238528<br> > bytes, 1391648 total<br> ><br> > INFO | jvm 1 | 2010/11/02 04:54:33 | : 155621K->2065K(179200K),<br>
> 0.1046330 secs] 988712K->835373K(1022976K) icms_dc=0 , 0.1047500<br> secs]<br> > [Times: user=0.00 sys=0.00, real=0.11 secs]<br> ><br> > INFO | jvm 1 | 2010/11/02 04:55:54 | 422097.583: [GC<br>
422097.583:<br> > [ParNew<br> ><br> > INFO | jvm 1 | 2010/11/02 04:55:54 |<br> ><br> > INFO | jvm 1 | 2010/11/02 04:55:54 | Desired survivor size<br> 13107200<br> > bytes, new threshold 4 (max 4)<br>
><br> > INFO | jvm 1 | 2010/11/02 04:55:54 | - age 1: 577104<br> > bytes, 577104 total<br> ><br> > INFO | jvm 1 | 2010/11/02 04:55:54 | - age 2: 261856<br> > bytes, 838960 total<br>
><br> > INFO | jvm 1 | 2010/11/02 04:55:54 | - age 3: 298832<br> > bytes, 1137792 total<br> ><br> > INFO | jvm 1 | 2010/11/02 04:55:54 | - age 4: 259176<br> > bytes, 1396968 total<br>
><br> > INFO | jvm 1 | 2010/11/02 04:55:54 | : 155664K->2386K(179200K),<br> > 0.0498010 secs] 988972K->835920K(1022976K) icms_dc=0 , 0.0499370<br> secs]<br> > [Times: user=0.00 sys=0.00, real=0.05 secs]<br>
><br> > INFO | jvm 1 | 2010/11/02 04:57:27 | 422190.993: [GC<br> 422190.993:<br> > [ParNew<br> ><br> > INFO | jvm 1 | 2010/11/02 04:57:28 |<br> ><br> > INFO | jvm 1 | 2010/11/02 04:57:28 | Desired survivor size<br>
13107200<br> > bytes, new threshold 4 (max 4)<br> ><br> > INFO | jvm 1 | 2010/11/02 04:57:28 | - age 1: 676656<br> > bytes, 676656 total<br> ><br> > INFO | jvm 1 | 2010/11/02 04:57:28 | - age 2: 283376<br>
> bytes, 960032 total<br> ><br> > INFO | jvm 1 | 2010/11/02 04:57:28 | - age 3: 239472<br> > bytes, 1199504 total<br> ><br> > INFO | jvm 1 | 2010/11/02 04:57:28 | - age 4: 264960<br>
> bytes, 1464464 total<br> ><br> > INFO | jvm 1 | 2010/11/02 04:57:28 | : 155986K->1918K(179200K),<br> > 0.0652010 secs] 989520K->835699K(1022976K) icms_dc=0 , 0.0653200<br> secs]<br> > [Times: user=0.01 sys=0.00, real=0.07 secs]<br>
><br> > INFO | jvm 1 | 2010/11/02 04:58:54 | 422277.406: [GC<br> 422277.406:<br> > [ParNew<br> ><br> > INFO | jvm 1 | 2010/11/02 04:58:54 | Desired survivor size<br> 13107200<br> > bytes, new threshold 4 (max 4)<br>
><br> > INFO | jvm 1 | 2010/11/02 04:58:54 | - age 1: 615944<br> > bytes, 615944 total<br> ><br> > INFO | jvm 1 | 2010/11/02 04:58:54 | - age 2: 334120<br> > bytes, 950064 total<br>
><br> > INFO | jvm 1 | 2010/11/02 04:58:54 | - age 3: 276736<br> > bytes, 1226800 total<br> ><br> > INFO | jvm 1 | 2010/11/02 04:58:54 | - age 4: 236424<br> > bytes, 1463224 total<br>
><br> > INFO | jvm 1 | 2010/11/02 04:58:54 | : 155518K->1928K(179200K),<br> > 0.0378730 secs] 989299K->835959K(1022976K) icms_dc=0 , 0.0379920<br> secs]<br> > [Times: user=0.00 sys=0.00, real=0.04 secs]<br>
><br> > INFO | jvm 1 | 2010/11/02 05:00:23 | 422366.439: [GC<br> 422366.439:<br> > [ParNew<br> ><br> > INFO | jvm 1 | 2010/11/02 05:00:23 | (promotion failed)<br> ><br> > INFO | jvm 1 | 2010/11/02 05:00:23 | Desired survivor size<br>
13107200<br> > bytes, new threshold 4 (max 4)<br> ><br> > INFO | jvm 1 | 2010/11/02 05:00:23 | - age 1: 574000<br> > bytes, 574000 total<br> ><br> > INFO | jvm 1 | 2010/11/02 05:00:23 | - age 2: 315432<br>
> bytes, 889432 total<br> ><br> > INFO | jvm 1 | 2010/11/02 05:00:23 | - age 3: 281216<br> > bytes, 1170648 total<br> ><br> > INFO | jvm 1 | 2010/11/02 05:00:23 | - age 4: 271776<br>
> bytes, 1442424 total<br> ><br> > INFO | jvm 1 | 2010/11/02 05:00:23 | :<br> 155528K->155689K(179200K),<br> > 0.1007840 secs]422366.540: [CMS<br> ><br> > INFO | jvm 1 | 2010/11/02 05:01:46 | [Unloading class<br>
> sun.reflect.GeneratedMethodAccessor121]<br> ><br> > INFO | jvm 1 | 2010/11/02 05:01:46 | [Unloading class<br> > sun.reflect.GeneratedMethodAccessor119]<br> ><br> > INFO | jvm 1 | 2010/11/02 05:01:46 | [Unloading class<br>
> sun.reflect.GeneratedMethodAccessor124]<br> ><br> > INFO | jvm 1 | 2010/11/02 05:01:46 | [Unloading class<br> > sun.reflect.GeneratedMethodAccessor123]<br> ><br> > INFO | jvm 1 | 2010/11/02 05:01:46 | [Unloading class<br>
> sun.reflect.GeneratedMethodAccessor120]<br> ><br> > INFO | jvm 1 | 2010/11/02 05:01:46 | [Unloading class<br> > sun.reflect.GeneratedMethodAccessor122]<br> ><br> > ERROR | wrapper | 2010/11/02 05:02:37 | JVM appears hung:<br>
Timed out<br> > waiting for signal from JVM.<br> ><br> ><br> ><br> ><br> ><br> > *Dori Rabin*<br> ><br> > *cid:image001.gif@01CB69E7.E5E45760*<br> ><br> ><br> ><br>
> *cid:image002.jpg@01CB69E7.E5E45760*<br> ><br> > T. +972-3-123-4567 F. +972-3- 766-3559 M. +972-54- 4232-706<br> ><br></div></div> > Email: mailto:<a href="mailto:Dori" target="_blank">Dori</a> <mailto:<a href="mailto:Dori" target="_blank">Dori</a>> <mailto:<a href="mailto:your-email-address" target="_blank">your-email-address</a>
<div class="im"><br> <mailto:<a href="mailto:your-email-address" target="_blank">your-email-address</a>>>.<a href="mailto:Rabin@starhome.com" target="_blank">Rabin@starhome.com</a><br></div> <mailto:<a href="mailto:Rabin@starhome.com" target="_blank">Rabin@starhome.com</a>>
<div class="im"><br> ><br> ><br> ><br> ><br> ><br> > *cid:image003.gif@01CB69E7.E5E45760* <<a href="http://www.starhome.com/" target="_blank">http://www.starhome.com/</a>><br> > *cid:image004.gif@01CB69E7.E5E45760*<br>
> <<a href="http://blog.starhome.com/" target="_blank">http://blog.starhome.com/</a>> *cid:image005.gif@01CB69E7.E5E45760*<br> > <<a href="http://bit.ly/9SbzNs" target="_blank">http://bit.ly/9SbzNs</a>> *cid:image006.gif@01CB69E7.E5E45760*<br>
> <<a href="http://bit.ly/aoU2m3" target="_blank">http://bit.ly/aoU2m3</a>> *cid:image007.gif@01CB69E7.E5E45760*<br> > <<a href="http://linkd.in/bjscKL" target="_blank">http://linkd.in/bjscKL</a>><br>
><br> > This email contains proprietary and/or confidential information of<br> > Starhome. If you<br> ><br> > have received this email in error, please delete all copies without<br> > delay and do not<br>
><br> > copy, distribute, or rely on any information contained in this<br> email.<br> > Thank you!<br> ><br> ><br> ><br> ><br> ><br> ><br> ><br> ><br> ><br> ------------------------------------------------------------------------<br>
><br> > _______________________________________________<br> > hotspot-gc-use mailing list<br> > <a href="mailto:hotspot-gc-use@openjdk.java.net" target="_blank">hotspot-gc-use@openjdk.java.net</a><br>
</div> <mailto:<a href="mailto:hotspot-gc-use@openjdk.java.net" target="_blank">hotspot-gc-use@openjdk.java.net</a>>
<div class="im"><br> > <a href="http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use" target="_blank">http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use</a><br> _______________________________________________<br>
hotspot-gc-use mailing list<br> <a href="mailto:hotspot-gc-use@openjdk.java.net" target="_blank">hotspot-gc-use@openjdk.java.net</a><br></div> <mailto:<a href="mailto:hotspot-gc-use@openjdk.java.net" target="_blank">hotspot-gc-use@openjdk.java.net</a>>
<div class="im"><br> <a href="http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use" target="_blank">http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use</a><br><br><br></div></blockquote><br></blockquote>
</blockquote></div><br></div>