<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix"><br>
Hi Michal,<br>
<br>
I had a quick look at the log files as well. Monica is correct
that both of them are from G1 so it would be interesting to have
an iCMS log to compare with.<br>
<br>
I don't think you hit the 20% min young limit. The reason that
your young gen is not shrinking enough to meet the pause target is
that you have set NewSize on the command line. You have
-XX:NewSize=128m, which puts an limit on how small the young gen
can be.<br>
<br>
Could you try to remove that from the command line and run again?<br>
<br>
Thanks,<br>
Bengt<br>
<br>
On 1/31/13 7:44 PM, Monica Beckwith wrote:<br>
</div>
<blockquote cite="mid:510ABB9E.6050105@oracle.com" type="cite">
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
Hi Michal (and John),<br>
<br>
I did look at Michal's log files (BTW, both are the same G1 logs).
And I can confirm that scan RS is the issue... here's the plot:<br>
<img src="cid:part1.08090708.01080105@oracle.com" alt=""
height="360" width="914"><br>
<br>
The plot above only shows the max obj copy times (for all pauses),
max RS Scan times and Parallel Time. So yes, scan RS is the
culprit. <br>
Also, looking at your logs, it seems that for mixedGCs the
reclaimable bytes don't cross 6%. So can you please try increasing
the G1HeapWastePercent to 10? (the latest builds will have 10 as
the default value).<br>
<br>
Please let me know if that improves your RT.<br>
<br>
-Monica<br>
<br>
<br>
On 1/31/2013 11:56 AM, John Cuthbertson wrote:
<blockquote cite="mid:510AB061.6010403@oracle.com" type="cite">Hi
Michal, <br>
<br>
I haven't looked at the logs yet but from your description it
sounds like a large number of the RSets have been coarsened
and/or the fine grain tables have gotten very dense. <br>
<br>
The RSet for a heap region tracks incoming references to objects
in that region. There are three levels on granularity: sparse,
fine, and coarse. The sparse and fine entries are encoded in
open hash tables based upon the region containing the
references. As the number of references from region A that point
into region B increase, the number of cards in the hash table
entry for region A increase (it's actually a bitmap with one bit
per card) and as the number of regions that contain references
that point into B increase, the number of region entries in fine
grain table increase. Once we run out of space in the fine grain
table (i.e. we can't add another bitmap for another region) we
evict one of the densest region bitmaps and say "coarsen" that
entry. When we have a coarsened entry we have to search the
entire referencing region looking for incoming references
compared to searching specific cards with the sparse and fine
entries. <br>
<br>
I'll take a look at your logs to see if I can confirm. <br>
<br>
JohnC <br>
<br>
<br>
On 1/31/2013 6:12 AM, Michal Frajt wrote: <br>
<blockquote type="cite">Hi all, <br>
<br>
After the iCMS got officially deprecated we decided to compare
the G1 collector with our best tuned (i)CMS setup.
Unfortunately we are not able to make the G1 young collection
running any closer to the ParNew. Actually we wanted to
compare the G1 concurrent marking STW pauses with the CMS
initial-mark and remark STW pauses but already incredibly long
running G1 young collections are unacceptable for us. <br>
<br>
We were able to recognize that the very long G1 young
collections are caused by the scanning remembered sets. There
is not much documentation about G1 internals but we were able
to understand that the size of the remembered sets is related
to the amount of mutating references from old regions (cards)
to young regions. Unfortunately all our applications mutate
permanently thousands references from old objects to young
objects. <br>
<br>
We are testing with the latest OpenJDK7u extended by the
7189971 patch and CMSTriggerInterval implementation. The
attached GC log files represent two very equal applications
processing very similar data sets, one running the G1, second
running the CMS collector. The OpenJDK7u has an extra output
of _pending_cards (when G1TraceConcRefinement activated) which
somehow relates to the remembered sets size. <br>
<br>
Young Comparison (both 128m, survivor ratio 5, max tenuring
15) <br>
CMS - invoked every ~20 sec, avg. stop 60ms <br>
G1 - invoked every ~16 sec, avg. stop 410ms !!! <br>
<br>
It there anything what could help us to reduce the Scan RS
time or the G1 is simply not targeted for applications
mutating heavily old region objects? <br>
<br>
CMS parameters <br>
-Xmx8884m -Xms2048m -XX:NewSize=128m -XX:MaxNewSize=128m
-XX:PermSize=128m -XX:SurvivorRatio=5
-XX:MaxTenuringThreshold=15 -XX:CMSMarkStackSize=8M
-XX:CMSMarkStackSizeMax=32M -XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled -XX:CMSWaitDuration=60000
-XX:+CMSScavengeBeforeRemark -XX:CMSTriggerInterval=600000
-XX:+UseParNewGC -XX:ParallelGCThreads=8
-XX:ParallelCMSThreads=2 <br>
<br>
G1 parameters (mind MaxNewSize not specified) <br>
-Xmx8884m -Xms2048m -XX:NewSize=128m -XX:PermSize=128m
-XX:SurvivorRatio=5 -XX:MaxTenuringThreshold=15 -XX:+UseG1GC
-XX:MaxGCPauseMillis=200 -XX:G1MixedGCCountTarget=16
-XX:ParallelGCThreads=8 -XX:ConcGCThreads=2 <br>
<br>
G1 log file GC young pause <br>
[GC pause (young) [G1Ergonomics (CSet Construction) start
choosing CSet, _pending_cards: 23697, predicted base time:
32.88 ms, remaining <br>
time: 167.12 ms, target pause time: 200.00 ms] <br>
[Parallel Time: 389.8 ms, GC Workers: 8] <br>
>>>> <br>
[Scan RS (ms): Min: 328.8, Avg: 330.4, Max: 332.6, Diff:
3.8, Sum: 2642.9] <br>
<<<< <br>
[Eden: 119.0M(119.0M)->0.0B(118.0M) Survivors:
9216.0K->10.0M Heap: 1801.6M(2048.0M)->1685.7M(2048.0M)]
<br>
<br>
Regards, <br>
Michal <br>
<br>
<br>
</blockquote>
<br>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<a moz-do-not-send="true" href="http://www.oracle.com"
target="_blank"><img
src="cid:part2.02020502.06020804@oracle.com" alt="Oracle"
border="0" height="26" width="114"></a><br>
<font color="#666666" face="Verdana, Arial, Helvetica,
sans-serif" size="2">Monica Beckwith | Java Performance
Engineer<br>
VOIP: <a href="tel:+1%20512%20401%201274"
moz-do-not-send="true">+1 512 401 1274</a> <br>
Texas </font> <br>
<a moz-do-not-send="true"
href="http://www.oracle.com/commitment" target="_blank"><img
src="cid:part5.01030500.05090706@oracle.com" alt="Green
Oracle" align="absmiddle" border="0" height="28" width="44"></a>
<font color="#4B7D42" face="Verdana, Arial, Helvetica,
sans-serif" size="1">Oracle is committed to developing
practices and products that help protect the environment</font>
<!-- This signature was generated by the MyDesktop Oracle Business Signature utility version 4.0 -->
</div>
</blockquote>
<br>
</body>
</html>