Here's a patch of all my local changes against a checkout from a couple months back (may not completely apply against JDK7 trunk). This has the fixes mentioned as well as a few other experiments I'd done.<div><br></div>
<div>I expressly grant permission to the JDK team to include this patch or parts thereof in the JDK should they find the code useful.</div><div><br></div><div>-Todd<br><div><br><div class="gmail_quote">On Fri, Apr 1, 2011 at 2:07 PM, Alex Aisinzon <span dir="ltr"><<a href="mailto:aaisinzon@guidewire.com">aaisinzon@guidewire.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Todd</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Please share the patch. I will see if I can build it.</span></p><p class="MsoNormal">
<span style="font-size:11.0pt;color:#1F497D"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Regards</span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p><p class="MsoNormal">
<span style="font-size:11.0pt;color:#1F497D">Alex Aisinzon</span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p><div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt">From:</span></b><span style="font-size:10.0pt"> Todd Lipcon [mailto:<a href="mailto:todd@cloudera.com" target="_blank">todd@cloudera.com</a>] <br><b>Sent:</b> Friday, April 01, 2011 1:55 PM</span></p>
<div><div></div><div class="h5"><br><b>To:</b> Alex Aisinzon<br><b>Cc:</b> <a href="mailto:hotspot-gc-use@openjdk.java.net" target="_blank">hotspot-gc-use@openjdk.java.net</a><br><b>Subject:</b> Re: G1 feedback</div></div>
<p></p></div><div><div></div><div class="h5"><p class="MsoNormal"> </p><p class="MsoNormal">On Fri, Apr 1, 2011 at 1:52 PM, Alex Aisinzon <<a href="mailto:aaisinzon@guidewire.com" target="_blank">aaisinzon@guidewire.com</a>> wrote:</p>
<div><blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in"><div><div><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Hi Todd</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">This is very interesting. I feel I need to read some additional material about G1 to fully understand your explanation. I guess the research paper on it would help. I will plan to read it.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">In any case, I am happy to give your patch a try. The challenge is that I am not well set to rebuild G1. Is there a way I could get the java binary/executable with the patch included?</span></p>
<div><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p></div></div></div></blockquote><div><p class="MsoNormal"> </p></div><div><p class="MsoNormal">Sorry, I'm not well equipped to distribute a binary (and there might be licensing issues, I'm not even sure).</p>
</div><div><p class="MsoNormal"> </p></div><div><p class="MsoNormal">-Todd</p></div><div><p class="MsoNormal"> </p></div><blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div><div><div><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p></div><div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal"><b><span style="font-size:10.0pt">From:</span></b><span style="font-size:10.0pt"> Todd Lipcon [mailto:<a href="mailto:todd@cloudera.com" target="_blank">todd@cloudera.com</a>] <br>
<b>Sent:</b> Friday, April 01, 2011 12:05 PM<br><b>To:</b> Alex Aisinzon<br><b>Cc:</b> <a href="mailto:hotspot-gc-use@openjdk.java.net" target="_blank">hotspot-gc-use@openjdk.java.net</a><br><b>Subject:</b> Re: G1 feedback</span></p>
</div><div><div><p class="MsoNormal"> </p><p class="MsoNormal">Hi Alex,</p><div><p class="MsoNormal"> </p></div><div><p class="MsoNormal">I've had similar results - see my threads from a few months back on this mailing list.</p>
</div><div><p class="MsoNormal"> </p></div><div><p class="MsoNormal">The summary is that, when there is a tight pause bound, some regions will accumulate which have estimates that are "stuck" higher than the goal. As these accumulate, they're never collected, which means that memory usage slowly grows until a full GC is required.</p>
</div><div><p class="MsoNormal"> </p></div><div><p class="MsoNormal">The two situations that caused this in my experiments were:</p></div><div><p class="MsoNormal">1) The JVM got context-switched out for several scheduling quanta during the "other" portion of a non-young region collection. Because "other" time is considered constant overhead, even mostly-garbage regions were carrying this as part of their estimate, causing all non-young regions to be deemed "too expensive". I fixed this with a patch to notice when the "other time" estimate was greater than the pause goal and decay it back towards 0.</p>
</div><div><p class="MsoNormal"> </p></div><div><p class="MsoNormal">2) A bad region accumulates many inter-region references in its "remember set", overflowing into the coarse rset. Once the coarse rset has been used, there's no facility to "uncoarsen" the rset entries even after all the referring regions are dead. If the number of coarse entries is high enough that the time estimate is greater than the pause time goal, then again, this region will never be collected, and memory will fill up until a full GC. I wrote a patch to improve the time estimation for coarse rset entries based on the liveness info in the coarse regions. This helped somewhat for my application.</p>
</div><div><p class="MsoNormal"> </p></div><div><p class="MsoNormal">Let me know if you'd like to try these patches out, I can dig them up again (or you might find them in the mailing list archives).</p></div><div><p class="MsoNormal">
</p></div><div><p class="MsoNormal" style="margin-bottom:12.0pt">-Todd</p><div><p class="MsoNormal">On Fri, Apr 1, 2011 at 11:48 AM, Alex Aisinzon <<a href="mailto:aaisinzon@guidewire.com" target="_blank">aaisinzon@guidewire.com</a>> wrote:</p>
<div><div><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Hi all</span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Thoughts on this feedback about G1?</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Take care</span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Alex A</span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p><div><div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Alex Aisinzon <br><b>Sent:</b> Saturday, March 26, 2011 6:46 AM<br><b>To:</b> <a href="mailto:hotspot-gc-use@openjdk.java.net" target="_blank">hotspot-gc-use@openjdk.java.net</a><br><b>Subject:</b> G1 feedback</p>
</div></div><p class="MsoNormal"> </p><p class="MsoNormal"><span style="font-size:11.0pt">Hi all</span></p><p class="MsoNormal"><span style="font-size:11.0pt"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt">I experimented with G1 and Sun JDK 1.6 update 24 and ran two long running tests (8 hours) with it:</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">With "</span> <span style="font-size:11.0pt">-server -XX:+UseG1GC -XX:+UseCompressedOops -Xms24576m -Xmx24576m", most pauses were very short. 4 pauses were around 7 seconds and 4 at 34 seconds.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I then added the objective of keeping the longest pause around 1 second and used "-server -XX:+UseG1GC -XX:MaxGCPauseMillis=1000 -XX:+UseCompressedOops -Xms24576m -Xmx24576m". Most pauses were a little above 1 second except for one pause at 8 seconds and one at 57 seconds.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The server is a dual X5570 (8 cores total) and has 48GB of RAM. Its average CPU utilization was around 60-65% so it was not over-used.</span></p><p class="MsoNormal"><span style="font-size:11.0pt">Everything would be perfect if it were not for the 7, 34 and 8, 57 seconds pauses.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">What would you recommend I do to either reduce these longer pauses or give insights into what happened so that G1 can avoid these very rare but pretty long pauses in the future?</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt">Thanks in advance</span></p><p class="MsoNormal"><span style="font-size:11.0pt"> </span></p><p class="MsoNormal">
<span style="font-size:11.0pt">Alex A</span></p></div></div><p class="MsoNormal" style="margin-bottom:12.0pt"><br>_______________________________________________<br>hotspot-gc-use mailing list<br><a href="mailto:hotspot-gc-use@openjdk.java.net" target="_blank">hotspot-gc-use@openjdk.java.net</a><br>
<a href="http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use" target="_blank">http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use</a></p></div><p class="MsoNormal"><br><br clear="all"><br>-- <br>Todd Lipcon<br>
Software Engineer, Cloudera</p></div></div></div></div></div></blockquote></div><p class="MsoNormal"><br><br clear="all"><br>-- <br>Todd Lipcon<br>Software Engineer, Cloudera</p></div></div></div></div></blockquote></div>
<br><br clear="all"><br>-- <br>Todd Lipcon<br>Software Engineer, Cloudera<br>
</div></div>