<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=gb2312">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks Andrew! </div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I just reproduced it with the fastdebug build. Now it reports error at PromotionInfo:verify(). More specifically, it found numDisplaceHdrs NOT equal numObjWithDisplaceHdrs. It fails at below. </div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<p style="margin: 0px; font: 12px Menlo; background-color: rgb(255, 255, 255); margin: 0px; background-color: rgb(255, 255, 255)">
guarantee(numDisplacedHdrs == numObjsWithDisplacedHdrs, "Displaced hdr count”);</p>
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Here is the debug info. It would be great if somebody can help to guide how to instrument the code to find out the root cause. </div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">(gdb) where</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">#0</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);">
</span></span><span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">0x00007f0b98256067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">#1</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);">
</span></span><span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">0x00007f0b98257448 in __GI_abort () at abort.c:89</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">#2</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);">
</span></span><span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">0x00007f0b973ae0bc in os::abort (dump_core=<optimized out>, siginfo=<optimized out>, context=<optimized out>) at /home/shangx/jdk/src/hotspot/os/linux/os_linux.cpp:1380</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">#3</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);">
</span></span><span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">0x00007f0b9786dac7 in VMError::report_and_die (id=id@entry=-536870912, message=message@entry=0x7f0b97ad9bd8 "guarantee(numDisplacedHdrs == numObjsWithDisplacedHdrs)
failed", detail_fmt=detail_fmt@entry=0x7f0b97ad9dde "Displaced hdr count",</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);"> </span></span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);">
</span></span><span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">detail_args=detail_args@entry=0x7ed8791a6b80, thread=0x7ed888375800, pc=pc@entry=0x0, siginfo=0x0, context=0x7f0b98190180 <g_stored_assertion_context>,
filename=0x7f0b97ad9370 "/home/shangx/jdk/src/hotspot/share/gc/cms/promotionInfo.cpp", lineno=275, size=0)</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);">
</span></span><span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">at /home/shangx/jdk/src/hotspot/share/utilities/vmError.cpp:1571</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">#4</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);">
</span></span><span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">0x00007f0b9786e8af in VMError::report_and_die (thread=<optimized out>, context=context@entry=0x7f0b98190180 <g_stored_assertion_context>, filename=filename@entry=0x7f0b97ad9370
"/home/shangx/jdk/src/hotspot/share/gc/cms/promotionInfo.cpp", lineno=lineno@entry=275,</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);"> </span></span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);">
</span></span><span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">message=message@entry=0x7f0b97ad9bd8 "guarantee(numDisplacedHdrs == numObjsWithDisplacedHdrs) failed", detail_fmt=detail_fmt@entry=0x7f0b97ad9dde "Displaced
hdr count", detail_args=0x7ed8791a6b80) at /home/shangx/jdk/src/hotspot/share/utilities/vmError.cpp:1287</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">#5</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);">
</span></span><span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">0x00007f0b96a478ee in report_vm_error (file=file@entry=0x7f0b97ad9370 "/home/shangx/jdk/src/hotspot/share/gc/cms/promotionInfo.cpp", line=line@entry=275,
error_msg=error_msg@entry=0x7f0b97ad9bd8 "guarantee(numDisplacedHdrs == numObjsWithDisplacedHdrs) failed",</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);"> </span></span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);">
</span></span><span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">detail_fmt=detail_fmt@entry=0x7f0b97ad9dde "Displaced hdr count") at /home/shangx/jdk/src/hotspot/share/utilities/debug.cpp:264</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">#6</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);">
</span></span><span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">0x00007f0b9746901a in PromotionInfo::verify (this=this@entry=0x7f0b90125020) at /home/shangx/jdk/src/hotspot/share/gc/cms/promotionInfo.cpp:275</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">#7</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);">
</span></span><span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">0x00007f0b969b6012 in promoted_oops_iterate<ParScanWithoutBarrierClosure> (cl=0x0, this=0x7f0b90125020) at /home/shangx/jdk/src/hotspot/share/gc/cms/promotionInfo.inline.hpp:46</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">#8</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);">
</span></span><span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">ConcurrentMarkSweepGeneration::par_oop_since_save_marks_iterate_done (this=<optimized out>, thread_num=<optimized out>) at /home/shangx/jdk/src/hotspot/share/gc/cms/concurrentMarkSweepGeneration.cpp:1104</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">#9</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);">
</span></span><span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">0x00007f0b973e24c8 in ParNewGenTask::work (this=0x7ed88c5811a0, worker_id=16) at /home/shangx/jdk/src/hotspot/share/gc/cms/parNewGeneration.cpp:622</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">#10 0x00007f0b978df7e0 in run_task (data=..., this=0x7ed888375800) at /home/shangx/jdk/src/hotspot/share/gc/shared/workgroup.cpp:335</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">#11 GangWorker::loop (this=0x7ed888375800) at /home/shangx/jdk/src/hotspot/share/gc/shared/workgroup.cpp:345</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">#12 0x00007f0b977a996b in Thread::call_run (this=0x7ed888375800) at /home/shangx/jdk/src/hotspot/share/runtime/thread.cpp:400</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">#13 0x00007f0b973b8e45 in thread_native_entry (thread=0x7ed888375800) at /home/shangx/jdk/src/hotspot/os/linux/os_linux.cpp:714</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">#14 0x00007f0b987d8064 in start_thread (arg=0x7ed8791a7700) at pthread_create.c:309</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">#15 0x00007f0b9830962d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">(gdb)
</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);"> </span></span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">(gdb)</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);"> </span></span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">(gdb)</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);"> </span></span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">(gdb)</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);"> </span></span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">(gdb) info args</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">sig = 6</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">(gdb) select-frame 6</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">(gdb)</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);"> </span></span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">(gdb)</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);"> </span></span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">(gdb)</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);"> </span></span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">(gdb)</span><span style="font-variant-ligatures: no-common-ligatures; font-variant-ligatures: no-common-ligatures"><span style="background-color: rgb(0, 0, 0);"> </span></span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">(gdb) info local</span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">numObjsWithDisplacedHdrs = <optimized out></span></p>
<p style="margin: 0px; font: 12px "Andale Mono"; color: rgb(23, 131, 5); background-color: rgb(26, 26, 26); margin: 0px; background-color: rgb(26, 26, 26)">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(0, 0, 0);">numDisplacedHdrs = 390463</span></p>
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Xinli </div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>发件人:</b> Andrew Haley <aph@redhat.com><br>
<b>发送时间:</b> 2019年4月6日 8:34<br>
<b>收件人:</b> shang xinli; Kim Barrett; shade@redhat.com<br>
<b>抄送:</b> hotspot-gc-dev@openjdk.java.net<br>
<b>主题:</b> Re: 回复: JVM crashes constantly when High GC happens</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText">On 4/6/19 5:37 AM, shang xinli wrote:<br>
> Does anybody know some tools can detect memory corruption at run time? In the Windows world, we can use GFLAGS<<a href="https://blogs.msdn.microsoft.com/webdav_101/2010/06/22/detecting-heap-corruption-using-gflags-and-dumps/">https://blogs.msdn.microsoft.com/webdav_101/2010/06/22/detecting-heap-corruption-using-gflags-and-dumps/</a>>,
is there any equivalent tool in Linux I can use?<br>
<br>
Yes, a fastdebug build adds a lot of checking.<br>
<br>
-- <br>
Andrew Haley<br>
Java Platform Lead Engineer<br>
Red Hat UK Ltd. <<a href="https://www.redhat.com">https://www.redhat.com</a>><br>
EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671<br>
</div>
</span></font></div>
</div>
</div>
</body>
</html>