<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
Hi Jungwoo,<br>
<br>
<br>
<div class="moz-cite-prefix">On 2014-10-22 23:03, Jungwoo Ha wrote:<br>
</div>
<blockquote
cite="mid:CA+n_jhjMkK6jrQ2zubM=uNp3DZf+NLQ1bA=kdVim7_FwkS7yJA@mail.gmail.com"
type="cite">
<div dir="ltr">Bug: <a moz-do-not-send="true"
href="https://bugs.openjdk.java.net/browse/JDK-8061259">https://bugs.openjdk.java.net/browse/JDK-8061259</a><br>
<div><br>
</div>
<div>hotspot code: <a moz-do-not-send="true"
href="http://cr.openjdk.java.net/%7Erasbold/8061259/webrev.00/"
target="_blank"
style="font-size:13px;font-family:arial,sans-serif">http://cr.openjdk.java.net/~<span
class=""
style="color:rgb(34,34,34);background:rgb(255,255,204)">rasbold</span>/<span
class=""
style="color:rgb(34,34,34);background:rgb(255,255,204)">8061259</span>/webrev.00/</a></div>
<div><br>
</div>
<div>I created the patch for JDK9, but I can also observe this
on JDK7 and 8.</div>
<div><br>
</div>
<div>
<div style="font-family:arial,sans-serif;font-size:13px">We
are seeing several cases where GC worker threads are
serialized on GCRareEvent_lock</div>
<div style="font-family:arial,sans-serif;font-size:13px">causing
2 digit seconds pause on moderate sized heap.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">I
have a test code that can reproduce it and shows it is
solved.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px"><font
face="courier new, monospace">import java.util.LinkedList;</font></div>
<div style="font-family:arial,sans-serif;font-size:13px">
<div><font face="courier new, monospace">class PromoFail {</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace"> static class
Container {</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace"> Container p;</font></div>
<div><font face="courier new, monospace"> byte[] a;</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace"> public
Container(int size) {</font></div>
<div><font face="courier new, monospace"> if (size >
0) {</font></div>
<div><font face="courier new, monospace"> p = new
Container(size / 2);</font></div>
<div><font face="courier new, monospace"> } else {</font></div>
<div><font face="courier new, monospace"> p = null;</font></div>
<div><font face="courier new, monospace"> }</font></div>
<div><font face="courier new, monospace"> a = new
byte[size];</font></div>
<div><font face="courier new, monospace"> }</font></div>
<div><font face="courier new, monospace"> }</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace"> public static
void main(String args[]) {</font></div>
<div><font face="courier new, monospace"> if (args.length
< 1) {</font></div>
<div><font face="courier new, monospace">
System.err.println("@ 1st argument must be size in
MB.");</font></div>
<div><font face="courier new, monospace">
System.exit(1);</font></div>
<div><font face="courier new, monospace"> }</font></div>
<div><font face="courier new, monospace"> int size = 0;</font></div>
<div><font face="courier new, monospace"> try {</font></div>
<div><font face="courier new, monospace"> size =
Integer.parseInt(args[0]) * 1024 * 1024;</font></div>
<div><font face="courier new, monospace"> } catch
(NumberFormatException e) {</font></div>
<div><font face="courier new, monospace">
System.err.println("@ Cannot parse the size(=" + args[0]
+ ")");</font></div>
<div><font face="courier new, monospace">
System.exit(1);</font></div>
<div><font face="courier new, monospace"> }</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace"> // LinkedList
will have more unbalanced workload.</font></div>
<div><font face="courier new, monospace">
LinkedList<Container> list = new
LinkedList<Container>();</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace"> // 1st
iteration adds element without removal.</font></div>
<div><font face="courier new, monospace"> // These are
all live objects.</font></div>
<div><font face="courier new, monospace"> for (int i = 0;
i < size / 4; i++) {</font></div>
<div><font face="courier new, monospace"> list.add(new
Container(1));</font></div>
<div><font face="courier new, monospace"> }</font></div>
<div><font face="courier new, monospace"> // Promote to
the old gen.</font></div>
<div><font face="courier new, monospace"> System.gc();</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace"> for (int
containerSize = 2; container_size < 512;
container_size *= 3) {</font></div>
<div><font face="courier new, monospace"> for (int i =
0; i < size / 4; i++) {</font></div>
<div><font face="courier new, monospace"> // Most
likely removing an old object due to System.gc() from
previous iteration.</font></div>
<div><font face="courier new, monospace"> // This
will cause fragmentation.</font></div>
<div><font face="courier new, monospace">
list.remove();</font></div>
<div><font face="courier new, monospace">
list.add(new Container(containerSize));</font></div>
<div><font face="courier new, monospace"> }</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace"> {</font></div>
<div><font face="courier new, monospace">
System.gc();</font></div>
<div><font face="courier new, monospace"> Runtime
runtime = Runtime.getRuntime();</font></div>
<div><font face="courier new, monospace">
System.out.println("@ Current Used: "</font></div>
<div><font face="courier new, monospace"> +
(runtime.totalMemory() - runtime.freeMemory()) / 1024 /
1024);</font></div>
<div><font face="courier new, monospace"> }</font></div>
<div><font face="courier new, monospace"> }</font></div>
<div><font face="courier new, monospace"> }</font></div>
<div><font face="courier new, monospace">}</font></div>
</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">You
can run it with the following parameters.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px"><font
face="courier new, monospace">$ java -Xmx2g -Xms2g -Xmn1g
-XX:+UseCMSFastPromotionFailure -XX:+PrintGCDetails
-XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=6 PromoFail
4</font><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">Without
UseCMSFastPromotionFailure </div>
<div style="font-family:arial,sans-serif;font-size:13px">
<div><font face="courier new, monospace">#7: [GC (Allocation
Failure) #7: [ParNew#6:
[CMS-concurrent-abortable-preclean: 0.003/0.203 secs]
[Times: user=0.20 sys=0.20 real=0.20 secs] </font></div>
<div><font face="courier new, monospace"> (promotion
failed): 838912K->943744K(943744K), 62.0419534
secs]#8: [CMS (concurrent mode failure):
1048441K->1048575K(1048576K), 1.7731336 secs]
1609551K->1170596K(1992320K), [Metaspace:
3547K->3547K(1056768K)], <b>63.8151607</b>secs]
[Times: user=93.50 sys=22.12 real=63.82 secs] </font></div>
</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">With
UseCMSFastPromotionFailure</div>
<div style="font-family:arial,sans-serif;font-size:13px">
<div><font face="courier new, monospace">#7: [GC (Allocation
Failure) #7: [ParNew#6:
[CMS-concurrent-abortable-preclean: 0.004/0.204 secs]
[Times: user=0.30 sys=0.02 real=0.20 secs] </font></div>
<div><font face="courier new, monospace"> (promotion
failed): 838912K->943744K(943744K), 2.0949545
secs]#8: [CMS (concurrent mode failure):
1048363K->1048575K(1048576K), 1.7517250 secs]
1609551K->1170595K(1992320K), [Metaspace:
3546K->3546K(1056768K)], <b>3.8467384</b>secs]
[Times: user=10.85 sys=1.04 real=3.85 secs] <br>
</font></div>
</div>
</div>
</div>
</blockquote>
<br>
Thanks for fixing this and for providing the nice reproducer. I get
similar results when I run the reproducer on my workstation. Nice
work!<br>
<br>
I've looked a bit at the webrev. A couple of comments:<br>
<br>
Why do you use OrderAccess methods for writing and reading the
_has_promo_failed flag in has_promo_failed() and set_promot_failed()
?<br>
<br>
Can we write out the full word "promotion" instead of just "promo"
in the variables and methods?<br>
<br>
Can we change the name of the flag from UseCMSFastPromotionFailure
to CMSFastPromotionFailure? Most CMS flags start with CMS and I
don't think we need the "Use" prefix.<br>
<br>
What do you think about making the flag true by default? At least
for JDK 9. If we decide to backport to JDK 8 or 7 it might be a good
idea to keep the default value as false.<br>
<br>
Did you find the information provided by
_fast_promo_failure_hitcount useful for debugging? If it not too
useful I would consider removing it since it is cluttering up the
code a bit.<br>
<br>
<br>
<blockquote
cite="mid:CA+n_jhjMkK6jrQ2zubM=uNp3DZf+NLQ1bA=kdVim7_FwkS7yJA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">I
also ran it on Dacapo benchmarks. Please see attached
results.</div>
<div style="font-family:arial,sans-serif;font-size:13px">Those
are subset of DaCapo that shows any promotion failed pause.</div>
<div style="font-family:arial,sans-serif;font-size:13px">You
can see that some speed ups and no performance regressions.</div>
</div>
</div>
</blockquote>
<br>
<br>
Thanks for doing some performance testing of this!<br>
<br>
Bengt<br>
</body>
</html>