<div dir="ltr">I don't think it has anything to do with the user threads.<div>The crash was during the VM termination. The launcher initiated the termination (I guess through DestroyJavaVM or similar call), and crashed inside CMS. Following is the stack trace.</div><div><br></div><div><div>*** SIGSEGV (@(nil)), see go/cppdebug received by PID 782912 (TID 783045); stack trace: ***</div><div>PC: @ 0x7fc61e2e1e6c (unknown) CMSCollector::collect_in_background()</div><div> @ 0x29f09a0 1312 FailureSignalHandler()</div><div> @ 0x7fc61e6fd147 208 os::Linux::chained_handler()</div><div> @ 0x7fc61e701994 336 JVM_handle_linux_signal</div><div> @ 0x7fc61e6f64b8 64 signalHandler()</div><div> @ 0x7fc621578340 1888 __restore_rt</div><div> @ 0x7fc61e2e92f9 48 ConcurrentMarkSweepThread::run()</div><div> @ 0x7fc61e6f78a2 64 java_start()</div><div> @ 0x7fc62156e800 176 start_thread</div><div> @ 0x7fc6211d22ed (unknown) clone</div><div><br></div></div><div>tcmalloc writes 0xcdcdcdcd when it deallocates the memory (in this case, Java heap), and core dump show that CMS thread was interpreting that word as an oop and crashed.</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 4, 2015 at 9:51 AM, Jon Masamitsu <span dir="ltr"><<a href="mailto:jon.masamitsu@oracle.com" target="_blank">jon.masamitsu@oracle.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><span class="">
<br>
<br>
<div>On 11/4/2015 9:33 AM, Jungwoo Ha wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Unfortunately, we cannot provide the test publicly.
<div>But we do have internal tests that triggers this bug, and
got fixed with this patch.</div>
</div>
</blockquote>
<br></span>
Can you describe the circumstances that provoke the bug? Lots of
concurrent threads?<br>
Lots of concurrent work in progress?<span class="HOEnZb"><font color="#888888"><br>
<br>
Jon</font></span><div><div class="h5"><br>
<blockquote type="cite">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Nov 4, 2015 at 9:25 AM, Jon
Masamitsu <span dir="ltr"><<a href="mailto:jon.masamitsu@oracle.com" target="_blank">jon.masamitsu@oracle.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><span> <br>
<br>
<div>On 11/3/2015 5:32 PM, Jungwoo Ha wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Somehow I didn't receive Jon's email.
<div>We've first noticed this on JDK8, but I think
the bug is more triggered with the tcmalloc +
custom launcher.</div>
<div><br>
</div>
</div>
</blockquote>
<br>
</span> Jungwoo,<br>
<br>
Do you have a test that exhibits the bug?<span><font color="#888888"><br>
<br>
Jon</font></span><span><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Nov 3, 2015 at 4:14
PM, Carsten Varming <span dir="ltr"><<a href="mailto:varming@gmail.com" target="_blank"></a><a href="mailto:varming@gmail.com" target="_blank">varming@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Dear Jon,
<div><br>
</div>
<div>I am pretty sure this has been that way
for a long time. I have never seen a
problem, but I would like to see Hotspot
shut down its threads as well.</div>
<div><br>
</div>
<div>Carsten</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Nov 3, 2015
at 5:40 PM, Jon Masamitsu <span dir="ltr"><<a href="mailto:jon.masamitsu@oracle.com" target="_blank"></a><a href="mailto:jon.masamitsu@oracle.com" target="_blank">jon.masamitsu@oracle.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <font face="Times New Roman, Times, serif">Jungwoo,<br>
<br>
Did you first see this bug in jdk8?
I'm wondering if it<br>
was always broken.<span><font color="#888888"><br>
<br>
Jon<br>
</font></span></font>
<div>
<div>
<div>
<div><br>
<div>On 11/03/2015 01:14 PM,
Jungwoo Ha wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">BUG: <a href="https://bugs.openjdk.java.net/browse/JDK-8141356" target="_blank"></a><a href="https://bugs.openjdk.java.net/browse/JDK-8141356" target="_blank">https://bugs.openjdk.java.net/browse/JDK-8141356</a>
<div>Webrev: <a href="http://cr.openjdk.java.net/%7Ejwha/8141356/webrev.00/" target="_blank"></a><a href="http://cr.openjdk.java.net/~jwha/8141356/webrev.00/" target="_blank">http://cr.openjdk.java.net/~jwha/8141356/webrev.00/</a></div>
<div><br>
</div>
<div>
<div>Inside before_exit(),
it says </div>
<div><br>
</div>
<div><font face="monospace,
monospace"> // Stop
concurrent GC threads </font></div>
<div><font face="monospace,
monospace">
Universe::heap()->stop(); </font></div>
<div><br>
</div>
<div>but GenCollectedHeap
never implemented stop()
method, and thus calling
empty SharedHeap::stop()
method. </div>
<div>This causes CMS
threads to run during
the VM termination and
ends up crashing. <br>
</div>
<div><span style="color:rgb(0,0,0);white-space:pre-wrap">ConcurrentMarkSweepThread::stop() is currently a dead code and never gets called.</span><br>
</div>
<div><span style="color:rgb(0,0,0);white-space:pre-wrap">The patch just implemented GenCollectedHeap::stop() to call </span><span style="color:rgb(0,0,0);white-space:pre-wrap">ConcurrentMarkSweepThread::stop().</span></div>
<div><br>
</div>
<div>Can someone sponsor
this patch?</div>
<div><br>
</div>
-- <br>
<div>
<div dir="ltr">
<div><span style="font-size:12.8px">Jungwoo
Ha | Google | <a href="mailto:jwha@google.com" target="_blank"></a><a href="mailto:jwha@google.com" target="_blank">jwha@google.com</a></span><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div>
<div dir="ltr">
<div><span style="font-size:12.8000001907349px">Jungwoo
Ha | Java Platform Team | <a href="mailto:jwha@google.com" target="_blank"></a><a href="mailto:jwha@google.com" target="_blank">jwha@google.com</a></span><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</span></div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div>
<div dir="ltr">
<div><span style="font-size:12.8000001907349px">Jungwoo Ha |
Java Platform Team | <a href="mailto:jwha@google.com" target="_blank">jwha@google.com</a></span><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><span style="font-size:12.8000001907349px">Jungwoo Ha | Java Platform Team | <a href="mailto:jwha@google.com" target="_blank">jwha@google.com</a></span><br></div><div><br></div></div></div>
</div>