<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Michal,<br>
<br>
Apologies for the delay.<br>
<br>
You can confirm whether RSet coarsening is the cause of your long RSet
scan times using the flags:<br>
<br>
-XX:+G1SummarizeRSetStats -XX:G1SummarizeRSetStatsPeriod=n<br>
<br>
Basically every n-th GC you'll get a summary of the RSet activity -
including total size, region with the largest RSet, concurrent
activity, and the number of coarsenings. These are diagnostic flags so
you'll need -XX:+UnlockDiagnosticVMOptions.<br>
<br>
If the number of coarsenings is high then you increase the number of
entries in the fine-grain RSet table using by setting
G1RSetRegionEntries. By default this gets set ergonomically using a
base number (256) per Mb of region size. So for the region size in your
log it will be set to 256. So this means that we can hold references
from 256 distinct regions in each fine grain table before we start
coarsening references from regions. This equates to 12.5% of your
heap. I would suggest bumping that value up to 512.<br>
<br>
Alternatively you can also try changing the heap region size by
explicitly setting -XX:G1HeapRegionSize=2m. In addition to increasing
the capacity of the fine grain tables, this should reduce the number of
entries in the RSets since there should be fewer cross-region
references.<br>
<br>
Regards,<br>
<br>
JohnC<br>
<br>
On 02/01/13 09:15, Michal Frajt wrote:
<blockquote cite="mid:004b01ce009f$be826620$3b873260$@frajt.eu"
type="cite">
<meta http-equiv="Content-Type" content="text/html; ">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
span.StylE-mailovZprvy17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:6562026;
mso-list-type:hybrid;
mso-list-template-ids:337512586 1712620822 67436547 67436549 67436545 67436547 67436549 67436545 67436547 67436549;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:18.0pt;
text-indent:-18.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:54.0pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:90.0pt;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:126.0pt;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:162.0pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:198.0pt;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:234.0pt;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:270.0pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:306.0pt;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:973873353;
mso-list-type:hybrid;
mso-list-template-ids:-1733514506 -1696530872 67436547 67436549 67436545 67436547 67436549 67436545 67436547 67436549;}
@list l1:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l2
{mso-list-id:1688630056;
mso-list-type:hybrid;
mso-list-template-ids:-1759187868 -200532592 67436547 67436549 67436545 67436547 67436549 67436545 67436547 67436549;}
@list l2:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l2:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l2:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l2:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l2:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l2:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l2:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l2:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l3
{mso-list-id:2039042386;
mso-list-type:hybrid;
mso-list-template-ids:459697548 1415744510 67436547 67436549 67436545 67436547 67436549 67436545 67436547 67436549;}
@list l3:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l3:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l3:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l3:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l3:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l3:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l3:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l3:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l3:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US">Monica,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US">Today morning I sent an email with the correct log files.
The attachment size was over the limit, it is still waiting for an
approval to be added to the posting list.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US">Some points<o:p></o:p></span></p>
<p class="MsoListParagraph"
style="margin-left: 18pt; text-indent: -18pt;"><!--[if !supportLists]--><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US"><span style="">-<span
style="font-family: "Times New Roman"; font-style: normal; font-variant: normal; font-weight: normal; font-size: 7pt; line-height: normal; font-size-adjust: none; font-stretch: normal;">
</span></span></span><!--[endif]--><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US">we compare G1 with CMS-ParNew (not iCMS, even we are
using iCMS everywhere else)<o:p></o:p></span></p>
<p class="MsoListParagraph"
style="margin-left: 18pt; text-indent: -18pt;"><!--[if !supportLists]--><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US"><span style="">-<span
style="font-family: "Times New Roman"; font-style: normal; font-variant: normal; font-weight: normal; font-size: 7pt; line-height: normal; font-size-adjust: none; font-stretch: normal;">
</span></span></span><!--[endif]--><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US">currently we evaluate the young gen processing only<o:p></o:p></span></p>
<p class="MsoListParagraph"
style="margin-left: 18pt; text-indent: -18pt;"><!--[if !supportLists]--><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US"><span style="">-<span
style="font-family: "Times New Roman"; font-style: normal; font-variant: normal; font-weight: normal; font-size: 7pt; line-height: normal; font-size-adjust: none; font-stretch: normal;">
</span></span></span><!--[endif]--><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US">now we set the G1RSetRegionEntries to 1024, it reduced
Scan RS time but it is still 3 times slower than ParNew<o:p></o:p></span></p>
<p class="MsoListParagraph"
style="margin-left: 18pt; text-indent: -18pt;"><!--[if !supportLists]--><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US"><span style="">-<span
style="font-family: "Times New Roman"; font-style: normal; font-variant: normal; font-weight: normal; font-size: 7pt; line-height: normal; font-size-adjust: none; font-stretch: normal;">
</span></span></span><!--[endif]--><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US">we do set NewSize to 128 for both the G1 and ParNew to
keep young collection invoked on same amount of garbage<o:p></o:p></span></p>
<p class="MsoListParagraph"
style="margin-left: 18pt; text-indent: -18pt;"><!--[if !supportLists]--><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US"><span style="">-<span
style="font-family: "Times New Roman"; font-style: normal; font-variant: normal; font-weight: normal; font-size: 7pt; line-height: normal; font-size-adjust: none; font-stretch: normal;">
</span></span></span><!--[endif]--><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US">setting G1 NewSize to 16GB and the target time to 60ms is
able to keep the track with the ParNew but the G1 young gen is invoked
3 times more frequently as adapted Eden size is around 30MB (ParNew has
128MB-survivor spaces)<o:p></o:p></span></p>
<p class="MsoListParagraph"
style="margin-left: 18pt; text-indent: -18pt;"><!--[if !supportLists]--><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US"><span style="">-<span
style="font-family: "Times New Roman"; font-style: normal; font-variant: normal; font-weight: normal; font-size: 7pt; line-height: normal; font-size-adjust: none; font-stretch: normal;">
</span></span></span><!--[endif]--><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US">I will be able to provide you the new log files and more
details on Tuesday<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"
lang="EN-US">Michal<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<div>
<div
style="border-style: solid none none; border-color: rgb(181, 196, 223) -moz-use-text-color -moz-use-text-color; border-width: 1pt medium medium; padding: 3pt 0cm 0cm;">
<p class="MsoNormal"><b><span
style="font-size: 10pt; font-family: "Tahoma","sans-serif"; color: windowtext;">From:</span></b><span
style="font-size: 10pt; font-family: "Tahoma","sans-serif"; color: windowtext;">
Monica Beckwith [<a class="moz-txt-link-freetext" href="mailto:monica.beckwith@oracle.com">mailto:monica.beckwith@oracle.com</a>] <br>
<b>Sent:</b> Friday, February 01, 2013 4:39 PM<br>
<b>To:</b> Michal Frajt<br>
<b>Cc:</b> <a class="moz-txt-link-abbreviated" href="mailto:hotspot-gc-dev@openjdk.java.net">hotspot-gc-dev@openjdk.java.net</a>; John Cuthbertson<br>
<b>Subject:</b> Re: CMS vs G1 - Scan RS very long<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks, Michal for providing us with the logs.
John (Cuthbertson) and I had a great discussion yesterday pertaining to
your G1 log and we think we understand what's going on with your
workload and have a experiment(s) plan for you. John is going to send
in more details (with explanation). Your workload's pattern is very
interesting to me/us and I have started a "work folder" for it, so that
I can document our experiment plan and the results and keep track of
the progress.<br>
<br>
Thanks for sharing your logs!<br>
<br>
-Monica<br>
<br>
On 2/1/2013 2:12 AM, Michal Frajt wrote: <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">Sorry, forgot to attach the archive in the last
message. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Od: <a moz-do-not-send="true"
href="mailto:hotspot-gc-dev-bounces@openjdk.java.net">hotspot-gc-dev-bounces@openjdk.java.net</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Komu: <a moz-do-not-send="true"
href="mailto:monica.beckwith@oracle.com">monica.beckwith@oracle.com</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Kopie: <a moz-do-not-send="true"
href="mailto:hotspot-gc-dev@openjdk.java.net">hotspot-gc-dev@openjdk.java.net</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Datum: Fri, 1 Feb 2013 09:05:58 +0100<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Předmet: Re: CMS vs G1 - Scan RS very long<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal">> Hi Monica / John,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">> Find the new archive which correctly
contains both the CMS and G1 log files. Yesterday I somehow managed to
overwrite the CMS log file when creating the tar archive. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">> Node2 - running CMS<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">> mmreutersservernode2_20130131105521_gc.log<br>
> mmreutersservernode2_jinfo.txt<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">> Node3 - running G1<br>
> mmreutersservernode3_20130131105525_gc.log<br>
> mmreutersservernode3_jinfo.txt<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">> Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">> Michal <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">> Od: "Monica Beckwith" <a
moz-do-not-send="true" href="mailto:monica.beckwith@oracle.com">monica.beckwith@oracle.com</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">> Komu: "John Cuthbertson" <a
moz-do-not-send="true" href="mailto:john.cuthbertson@oracle.com">john.cuthbertson@oracle.com</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">> Kopie: <a moz-do-not-send="true"
href="mailto:hotspot-gc-dev@openjdk.java.net">hotspot-gc-dev@openjdk.java.net</a>,"Michal
Frajt" <a moz-do-not-send="true" href="mailto:michal@frajt.eu">michal@frajt.eu</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">> Datum: Thu, 31 Jan 2013 12:44:46 -0600<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">> Předmet: Re: CMS vs G1 - Scan RS very long<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">> > Hi Michal (and John),<br>
> > <br>
> > I did look at Michal's log files (BTW, both are the same G1
logs). And I can confirm that scan RS is the issue... here's the plot:<br>
> > <img moz-do-not-send="true" id="_x0000_i1025"
src="cid:part1.05060004.07020808@oracle.com" border="0" height="360"
width="914"><br>
> > <br>
> > The plot above only shows the max obj copy times (for all
pauses), max RS Scan times and Parallel Time. So yes, scan RS is the
culprit. <br>
> > Also, looking at your logs, it seems that for mixedGCs the
reclaimable bytes don't cross 6%. So can you please try increasing the
G1HeapWastePercent to 10? (the latest builds will have 10 as the
default value).<br>
> > <br>
> > Please let me know if that improves your RT.<br>
> > <br>
> > -Monica<br>
> > <br>
> > <br>
> > On 1/31/2013 11:56 AM, John Cuthbertson wrote: <o:p></o:p></p>
<p class="MsoNormal">Hi Michal, <br>
> > <br>
> > I haven't looked at the logs yet but from your description it
sounds like a large number of the RSets have been coarsened and/or the
fine grain tables have gotten very dense. <br>
> > <br>
> > The RSet for a heap region tracks incoming references to
objects in that region. There are three levels on granularity: sparse,
fine, and coarse. The sparse and fine entries are encoded in open hash
tables based upon the region containing the references. As the number
of references from region A that point into region B increase, the
number of cards in the hash table entry for region A increase (it's
actually a bitmap with one bit per card) and as the number of regions
that contain references that point into B increase, the number of
region entries in fine grain table increase. Once we run out of space
in the fine grain table (i.e. we can't add another bitmap for another
region) we evict one of the densest region bitmaps and say "coarsen"
that entry. When we have a coarsened entry we have to search the entire
referencing region looking for incoming references compared to
searching specific cards with the sparse and fine entries. <br>
> > <br>
> > I'll take a look at your logs to see if I can confirm. <br>
> > <br>
> > JohnC <br>
> > <br>
> > <br>
> > On 1/31/2013 6:12 AM, Michal Frajt wrote: <br>
> > <o:p></o:p></p>
<p class="MsoNormal">Hi all, <br>
> > <br>
> > After the iCMS got officially deprecated we decided to
compare the G1 collector with our best tuned (i)CMS setup.
Unfortunately we are not able to make the G1 young collection running
any closer to the ParNew. Actually we wanted to compare the G1
concurrent marking STW pauses with the CMS initial-mark and remark STW
pauses but already incredibly long running G1 young collections are
unacceptable for us. <br>
> > <br>
> > We were able to recognize that the very long G1 young
collections are caused by the scanning remembered sets. There is not
much documentation about G1 internals but we were able to understand
that the size of the remembered sets is related to the amount of
mutating references from old regions (cards) to young regions.
Unfortunately all our applications mutate permanently thousands
references from old objects to young objects. <br>
> > <br>
> > We are testing with the latest OpenJDK7u extended by the
7189971 patch and CMSTriggerInterval implementation. The attached GC
log files represent two very equal applications processing very similar
data sets, one running the G1, second running the CMS collector. The
OpenJDK7u has an extra output of _pending_cards (when
G1TraceConcRefinement activated) which somehow relates to the
remembered sets size. <br>
> > <br>
> > Young Comparison (both 128m, survivor ratio 5, max tenuring
15) <br>
> > CMS - invoked every ~20 sec, avg. stop 60ms <br>
> > G1 - invoked every ~16 sec, avg. stop 410ms !!! <br>
> > <br>
> > It there anything what could help us to reduce the Scan RS
time or the G1 is simply not targeted for applications mutating heavily
old region objects? <br>
> > <br>
> > CMS parameters <br>
> > -Xmx8884m -Xms2048m -XX:NewSize=128m -XX:MaxNewSize=128m
-XX:PermSize=128m -XX:SurvivorRatio=5 -XX:MaxTenuringThreshold=15
-XX:CMSMarkStackSize=8M -XX:CMSMarkStackSizeMax=32M
-XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled
-XX:CMSWaitDuration=60000 -XX:+CMSScavengeBeforeRemark
-XX:CMSTriggerInterval=600000 -XX:+UseParNewGC -XX:ParallelGCThreads=8
-XX:ParallelCMSThreads=2 <br>
> > <br>
> > G1 parameters (mind MaxNewSize not specified) <br>
> > -Xmx8884m -Xms2048m -XX:NewSize=128m -XX:PermSize=128m
-XX:SurvivorRatio=5 -XX:MaxTenuringThreshold=15 -XX:+UseG1GC
-XX:MaxGCPauseMillis=200 -XX:G1MixedGCCountTarget=16
-XX:ParallelGCThreads=8 -XX:ConcGCThreads=2 <br>
> > <br>
> > G1 log file GC young pause <br>
> > [GC pause (young) [G1Ergonomics (CSet Construction) start
choosing CSet, _pending_cards: 23697, predicted base time: 32.88 ms,
remaining <br>
> > time: 167.12 ms, target pause time: 200.00 ms] <br>
> > [Parallel Time: 389.8 ms, GC Workers: 8] <br>
> > >>>> <br>
> > [Scan RS (ms): Min: 328.8, Avg: 330.4, Max: 332.6, Diff:
3.8, Sum: 2642.9] <br>
> > <<<< <br>
> > [Eden: 119.0M(119.0M)->0.0B(118.0M) Survivors:
9216.0K->10.0M Heap: 1801.6M(2048.0M)->1685.7M(2048.0M)] <br>
> > <br>
> > Regards, <br>
> > Michal <br>
> > <br>
> > <br>
> > <o:p></o:p></p>
<p class="MsoNormal"><br>
> > <o:p></o:p></p>
<p class="MsoNormal"><br>
> > <o:p></o:p></p>
<div>
<p class="MsoNormal">> > -- <br>
> > <a moz-do-not-send="true" href="http://www.oracle.com/"
target="_blank"><span style="text-decoration: none;"><img
moz-do-not-send="true" id="_x0000_i1026"
src="cid:part2.07060601.09070600@oracle.com" alt="Oracle" border="0"
height="26" width="114"></span></a><br>
> > <span
style="font-size: 10pt; font-family: "Verdana","sans-serif"; color: rgb(102, 102, 102);">Monica
Beckwith | Java Performance Engineer<br>
> > VOIP: <a moz-do-not-send="true"
href="tel:+1%20512%20401%201274">+1 512 401 1274</a> <br>
> > Texas </span><br>
> > <a moz-do-not-send="true"
href="http://www.oracle.com/commitment" target="_blank"><span
style="text-decoration: none;"><img moz-do-not-send="true"
id="_x0000_i1027" src="cid:part5.09000504.01050906@oracle.com"
alt="Green Oracle" border="0" height="28" width="44"></span></a><span
style="font-size: 7.5pt; font-family: "Verdana","sans-serif"; color: rgb(75, 125, 66);">Oracle
is committed to developing practices and products that help protect the
environment</span> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">-- <br>
<a moz-do-not-send="true" href="http://www.oracle.com" target="_blank"><span
style="text-decoration: none;"><img id="_x0000_i1028"
src="cid:part1.07070408.00060306@oracle.com" alt="Oracle" border="0"
height="26" width="114"></span></a><br>
<span
style="font-size: 10pt; font-family: "Verdana","sans-serif"; color: rgb(102, 102, 102);">Monica
Beckwith | Java Performance Engineer<br>
VOIP: <a moz-do-not-send="true" href="tel:+1%20512%20401%201274">+1
512 401 1274</a> <br>
Texas </span><br>
<a moz-do-not-send="true" href="http://www.oracle.com/commitment"
target="_blank"><span style="text-decoration: none;"><img
id="_x0000_i1029" src="cid:part2.00020409.00070109@oracle.com"
alt="Green
Oracle" border="0" height="28" width="44"></span></a><span
style="font-size: 7.5pt; font-family: "Verdana","sans-serif"; color: rgb(75, 125, 66);">Oracle
is committed to developing practices and products that help protect the
environment</span> <o:p></o:p></p>
</div>
</div>
</blockquote>
<br>
</body>
</html>