Thanks for the information Srinivas.<br><br><div class="gmail_quote">2012/11/4 Srinivas Ramakrishna <span dir="ltr"><<a href="mailto:ysr1729@gmail.com" target="_blank">ysr1729@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
CMS might help if you are able to tune it to avoid the full gc pauses entirely by doing all the<br>collection work mostly concurrently. With CMS, a few cpu's will be busy for a while as CMS runs<br>concurrently with the application in certain phases. If CMS "loses the race" (or there is excessive<br>
fragmentation), the stop-world full gc will be pretty long and will be single-threaded.<br><br>$ /usr/lib/jvm/jdk1.7.0_05/bin/java -XX:+PrintFlagsFinal -version | grep Compact<br> bool CMSCompactWhenClearAllSoftRefs = true {product} <br>
uintx CMSFullGCsBeforeCompaction = 0 {product} <br> bool CompactFields = true {product} <br> uintx HeapFirstMaximumCompactionCount = 3 {product} <br>
uintx HeapMaximumCompactionInterval = 20 {product} <br> intx MarkSweepAlwaysCompactCount = 4 {product} <br> bool UseCMSCompactAtFullCollection = true {product} <br>
bool UseMaximumCompactionOnSystemGC = true {product} <br>java version "1.7.0_05"<br>Java(TM) SE Runtime Environment (build 1.7.0_05-b06)<br>Java HotSpot(TM) 64-Bit Server VM (build 23.1-b03, mixed mode)<br>
<br>The options we used were HeapFirstMaximumCompactionCount and HeapMaximumCompactionInterval, both set to very large values.<br>For our use case, this worked reasonably well, but I have seen cases of other applications where this isn't effective because<br>
of the demographics of large oop-rich objects (such as large hash maps and such, for example) as they "churn through" the<br>old generation rather than sedimenting in the dense prefix. We didn't need to, and I think you won't need to either, worry<br>
about space lost to deadwood in the dense prefix, because the dense prefix computation doesn't waste much space. I don't<br>recall the computation, but John would be able to help.<br><br>-- ramki<div class="HOEnZb">
<div class="h5"><br><br><div class="gmail_quote">
On Sun, Nov 4, 2012 at 11:34 AM, lohit <span dir="ltr"><<a href="mailto:lohit.vijayarenu@gmail.com" target="_blank">lohit.vijayarenu@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Thanks Srinivas and Krystal for the pointers and information.<div>Would CMS help in this case? Because I see similar behavior in CMS as well, where in one cpu gets busy for quite some time.</div><div>I could try to grab similar stack trace if that helps. </div>
<div>From what I read there is no way around it other than disabling max compaction (I could not find any switch to deferring it forever, yet not disable it). </div><div>Downside of that is the memory gets fragmented and application performance degrade or causes OOM. </div>
<div>Any further thoughts on this?</div><div><br></div><div>Thanks again!</div><div><div><div><div><br><div class="gmail_quote">2012/11/4 Srinivas Ramakrishna <span dir="ltr"><<a href="mailto:ysr1729@gmail.com" target="_blank">ysr1729@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">This is the deferred pointer updates phase which is unfortunately single-threaded, and occasionally greatly dominates the pause times.<br>
I called in a bug for this late last year and there was some discussion on how this could be fixed. I was planning to work on<br>
a patch, but never quite got around to it. We worked around it (although the workaround may not always work, see a parallel (hmm)<br>ongoing discussion) by deferring the "maximum compaction" forever, always relying on the dense prefix to demarcate the<br>
boundary below which we would never choose to compact. In typical cases I have seen large oop-rich objects (which in our<br>case sedimented to the bottom of the heap) were the cause of this slow down.<br><br>I don't recall the bug id tracking this, but someone on the list may know. I'll try and find the old discussion of this, and send a pointer.<br>
<br>-- ramki<div><div><br><br><div class="gmail_quote">On Sun, Nov 4, 2012 at 9:54 AM, lohit <span dir="ltr"><<a href="mailto:lohit.vijayarenu@gmail.com" target="_blank">lohit.vijayarenu@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello Devs,<div><br></div><div>We are trying to profile Full GC performance on our Java server. </div><div>Heap Size is about 80G, running on CentOS 5.5. As of now we use ParallelOldGC for old generation compactions. </div>
<div>When we trigger fullGC by hand we see that it takes more than 2 minutes. </div><div>Running mpstat showed that for about half of the time only one CPU was spinning 100% and all others were idle. (Box has 24 CPUs)</div>
<div>While this was happening we took stack trace and see that all threads are waiting behind one thread whose trace look like below.</div><div><br></div><div>Question is the below expected. Even though documentation says parallelOldGC uses multiple threads, are there cases when this kind of serialization happens where all threads wait behind single thread.</div>
<div>I might have given very little information about problem, but let me know if I could add any more information to know about this</div><div><br></div><div><pre style="margin-top:0px;margin-bottom:0px;padding:0px;font-size:13px;line-height:1.3em;max-height:30em;overflow:auto;word-wrap:normal">
Thread 18 (Thread 0x419f2940 (LWP 55314)):^M
#0 0x00007f051735e3e0 in BitMap::get_next_one_offset_inline_aligned_right(unsigned long, unsigned long) const ()^M
from /usr/java/jdk1.6.0_24/jre/lib/amd64/server/libjvm.so^M
#1 0x00007f051735e09e in ParMarkBitMap::live_words_in_range(HeapWord*, oopDesc*) const ()^M
from /usr/java/jdk1.6.0_24/jre/lib/amd64/server/libjvm.so^M
#2 0x00007f0517398848 in ParallelCompactData::calc_new_pointer(HeapWord*) ()^M
from /usr/java/jdk1.6.0_24/jre/lib/amd64/server/libjvm.so^M
#3 0x00007f05173413ec in objArrayKlass::oop_update_pointers(ParCompactionManager*, oopDesc*) ()^M
from /usr/java/jdk1.6.0_24/jre/lib/amd64/server/libjvm.so^M
#4 0x00007f051739d0a7 in PSParallelCompact::update_deferred_objects(ParCompactionManager*, PSParallelCompact::SpaceId) ()^M
from /usr/java/jdk1.6.0_24/jre/lib/amd64/server/libjvm.so^M
#5 0x00007f051739c6c7 in PSParallelCompact::compact() () from /usr/java/jdk1.6.0_24/jre/lib/amd64/server/libjvm.so^M
#6 0x00007f051739aeca in PSParallelCompact::invoke_no_policy(bool) ()^M
from /usr/java/jdk1.6.0_24/jre/lib/amd64/server/libjvm.so^M
#7 0x00007f051739a845 in PSParallelCompact::invoke(bool) () from /usr/java/jdk1.6.0_24/jre/lib/amd64/server/libjvm.so^M
#8 0x00007f05174a0bb0 in VM_ParallelGCSystemGC::doit() () from /usr/java/jdk1.6.0_24/jre/lib/amd64/server/libjvm.so^M
#9 0x00007f05174ada5a in VM_Operation::evaluate() () from /usr/java/jdk1.6.0_24/jre/lib/amd64/server/libjvm.so^M
(More stack frames follow...)^M</pre><span><font color="#888888"><div><br></div>-- <br>Have a Nice Day!<br>Lohit<br>
</font></span></div>
</blockquote></div><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Have a Nice Day!<br>Lohit<br>
</div></div>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Have a Nice Day!<br>Lohit<br>