<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Arial Unicode MS";
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@Arial Unicode MS";
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:#17365D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:130513786;
mso-list-type:hybrid;
mso-list-template-ids:2008947168 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1
{mso-list-id:172114544;
mso-list-type:hybrid;
mso-list-template-ids:1023149700 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l1:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='color:#17365D'>Hello<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#17365D'>We are trying to fine tune our high impact application and would appreciate a little help in doing so using G1GC. Following are the application requirements<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#17365D'>Application Requirements<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'>• Expected load: 40 concurrent users with 1 Txns/user per second.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'>• 99th percentile – 250 ms .Desired all response times to be less than 1 second<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'>• 15 GB RAM per JVM for one set(set1) and 12 GB RAM for another set(set 2). Both jvms contain different kinds of data and the application interacts with both to fetch desired output. </span><span style='color:red'>Both JVMS have the same GC setting. But the kind of data in set 2 is more operation intensive and thus gets more activity that set 1 jvms.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'>• 2x load capacity. Hopefully we can get to 2X.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#17365D'>We implemented G1GC replacing the CMS as <o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'><o:p> </o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo1'><![if !supportLists]><span style='color:#17365D'><span style='mso-list:Ignore'>1)<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#17365D'>We have large heaps. Ranging from 8 Gb to 25 GB. The usual problem comes in with the set2 JVMS. Set1 generally does not crash.<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo1'><![if !supportLists]><span style='color:#17365D'><span style='mso-list:Ignore'>2)<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#17365D'>CMS was crashing the JVM frequently. A full GC cycle would crash the JVM. It used to happen during the nightly batch run for the application.<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo1'><![if !supportLists]><span style='color:#17365D'><span style='mso-list:Ignore'>3)<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#17365D'>We had Xmn upto 50% of the heap. <o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#17365D'>Once we switched from CMS to G1GC, the set2 jvms stopped crashing frequently. However we did have 2 incidents where the crash did happen. The G1GC parametrs that are being used are as follows:<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#17365D'>JAVA_ARGS="$JAVA_ARGS --J=-Xss${STACK_SIZE} \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'> --J=-XX:+UseG1GC \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'> --J=-XX:MaxGCPauseMillis=200 \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'> --J=-XX:ParallelGCThreads=20 \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'> --J=-XX:InitiatingHeapOccupancyPercent=60 \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'> --J=-XX:SurvivorRatio=2 \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'> --J=-XX:ConcGCThreads=5 \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'> --J=-Xmx$SERVER_HEAP \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'> --J=-Xms$SERVER_HEAP \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'> --J=-DDistributionManager.DISCONNECT_WAIT=$DISCONNECT_WAIT_TIME \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'> --J=-XX:+HeapDumpOnOutOfMemoryError \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'> --J=-XX:HeapDumpPath=${GF_LOG}/jvmdumps \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'> --J=-DgemfireSecurityPropertyFile=$DIR/$cluster_name/runtime/servers/$SERVER_NAME/gfsecurity.properties<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'> --J=-verbose:gc \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'> --J=-Xloggc:${GF_LOG}/logs/$SERVER_NAME/gc.log \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'> --J=-XX:+UseGCLogFileRotation --J=-XX:NumberOfGCLogFiles=10 --J=-XX:GCLogFileSize=1m \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'> --J=-XX:+PrintGCDateStamps --J=-XX:+PrintGCDetails --J=-XX:+PrintTenuringDistribution --J=-XX:+PrintAdaptiveSizePolicy"<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#17365D'>We tried starting from scratch and worked our way up to the above config with a series of load tests and other activities. Following are the observations that we have seen so far:<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'><o:p> </o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo2'><![if !supportLists]><span style='font-family:Symbol;color:#17365D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><i><span style='color:#17365D'>When the SurvivorRatio element was removed, then we observed a degraded performance</span></i><span style='color:#17365D'>. <i>Thus we kept it at 2 instead of default 8.</i><o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo2'><![if !supportLists]><span style='font-family:Symbol;color:#17365D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><i><span style='color:#17365D'>There are a handful of humongous allocations(maybe 3 -4 over a day) that are requested during the batch run.The number of those is not that high as you can see from the attached logs. Should I be changing the region size for them?</span></i><span style='color:#17365D'><o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#17365D'>Please let me know what can be changed and tested so that we do not encounter the jvm crash during batch times and also maintain the current sla. I have attached all the gc log files for the cluster.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#17365D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#17365D'>Thanks and Regards,<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'>Rohit Chaubey<o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'>Email: <a href="mailto:rohit.chaubey@broadridge.com">rohit.chaubey@broadridge.com</a><o:p></o:p></span></p><p class=MsoNormal><span style='color:#17365D'>Work: 201-714-3379, BB: 201-618-9230<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div><br clear="both">
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.<BR>
</body></html>