<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:΢ÈíÑźÚ
}
--></style></head>
<body class='hmmessage'><div dir='ltr'>
<div style="direction: ltr; color: rgb(0, 0, 0); "><div style="color: rgb(0, 0, 0); "><div id="ecxdivRpF191303" style="direction: ltr; "><font face="Times New Roman" size="2">Hi all,</font></div><div><div style="direction: ltr; color: rgb(0, 0, 0); "><div><font face="Times New Roman" size="2"><br>
</font></div>
<div><font face="Times New Roman" size="2">This is Hal Mo<kungu.mjh@taobao.com> from Alibaba Group(with OCA).</font></div>
<div><font face="Times New Roman" size="2"><br>
</font></div>
<div><font face="Times New Roman" size="2">Our hadoop namenode crashed, when we set the heap size to 135G using CMS GC.</font></div>
<div><font face="Times New Roman" size="2">Attached please find the crash log(hs_err_pid.log).</font></div>
<div><font face="Times New Roman" size="2"><br>
</font></div>
<div><font face="Times New Roman" size="2">I can steadily reproduce the crash on a test machine with 190G physical memory, by a simple command:</font></div>
<div><i><font face="Times New Roman" size="2">$ java -Xmx135g -XX:+UseConcMarkSweepGC</font></i></div>
<div><font face="Times New Roman" size="2"><br>
</font></div>
<div><font face="Times New Roman" size="2">Then I build a debug jvm and use gdb to debug the problem.</font></div>
<div><font face="Times New Roman" size="2"><br>
</font></div>
<div>
<div><font face="Times New Roman" size="2">call stack</font></div>
<div><font face="Times New Roman" size="2"><br>
</font></div>
<div>
<div><i><font face="Times New Roman" size="2">C [libc.so.6+0x7a9b0] memset+0x40</font></i></div>
<div><i><font face="Times New Roman" size="2">V [libjvm.so+0x2b6c42] BlockOffsetArray::set_remainder_to_point_to_start_incl(unsigned long, unsigned long, bool)+0xce</font></i></div>
<div><i><font face="Times New Roman" size="2">V [libjvm.so+0x2b7043] BlockOffsetArray::set_remainder_to_point_to_start(HeapWord*, HeapWord*, bool)+0x71</font></i></div>
<div><i><font face="Times New Roman" size="2">V [libjvm.so+0x2b728d] BlockOffsetArray::BlockOffsetArray(BlockOffsetSharedArray*, MemRegion, bool)+0x9f</font></i></div>
<div><i><font face="Times New Roman" size="2">V [libjvm.so+0x3c089f] BlockOffsetArrayNonContigSpace::BlockOffsetArrayNonContigSpace(BlockOffsetSharedArray*, MemRegion)+0x37</font></i></div>
<div><i><font face="Times New Roman" size="2">V [libjvm.so+0x3be56f] CompactibleFreeListSpace::CompactibleFreeListSpace(BlockOffsetSharedArray*, MemRegion, bool, FreeBlockDictionary::DictionaryChoice)+0x9b</font></i></div>
<div><i><font face="Times New Roman" size="2">V [libjvm.so+0x3fd2e1] ConcurrentMarkSweepGeneration::ConcurrentMarkSweepGeneration(ReservedSpace, unsigned long, int, CardTableRS*, bool, FreeBlockDictionary::DictionaryChoice)+0x1df</font></i></div>
<div><i><font face="Times New Roman" size="2">V [libjvm.so+0x4dc03e] GenerationSpec::init(ReservedSpace, int, GenRemSet*)+0x37c</font></i></div>
<div><i><font face="Times New Roman" size="2">V [libjvm.so+0x4ced40] GenCollectedHeap::initialize()+0x510</font></i></div>
<div><i><font face="Times New Roman" size="2">V [libjvm.so+0x7c23c3] Universe::initialize_heap()+0x31d</font></i></div>
<div><i><font face="Times New Roman" size="2">V [libjvm.so+0x7c27ec] universe_init()+0xa6</font></i></div>
<div><i><font face="Times New Roman" size="2">V [libjvm.so+0x5056e2] init_globals()+0x34</font></i></div>
<div><i><font face="Times New Roman" size="2">V [libjvm.so+0x7ac926] Threads::create_vm(JavaVMInitArgs*, bool*)+0x23a</font></i></div>
<div><i><font face="Times New Roman" size="2">V [libjvm.so+0x53f3d4] JNI_CreateJavaVM+0x7a</font></i></div>
</div>
</div>
<div><font face="Times New Roman" size="2"><br>
</font></div>
<div><font face="Times New Roman" size="2">in function BlockOffsetArray::set_remainder_to_point_to_start_inc, inside the for loop:</font></div>
<div>
<div><i><font face="Times New Roman" size="2"> size_t reach = start_card - 1 + (power_to_cards_back(i+1) - 1);</font></i></div>
</div>
<div><font face="Times New Roman" size="2">when i = 7, the value of reach was 0. then the loop could not break, and </font></div>
<div>
<div><i><font face="Times New Roman" size="2"> _array->set_offset_array(start_card_for_region, reach, offset, reducing);</font></i></div>
</div>
<div><font face="Times New Roman" size="2">accessed the wrong address, and crashed.</font></div>
<div><font face="Times New Roman" size="2"><br>
</font></div>
<div><font face="Times New Roman" size="2">the root cause was </font></div>
<div><i><font face="Times New Roman" size="2">static size_t power_to_cards_back(uint i) {</font></i></div>
<div><i><font face="Times New Roman" size="2"> return (size_t)(1 << (LogBase * i));</font></i></div>
<div><i><font face="Times New Roman" size="2">}</font></i></div>
<div><font face="Times New Roman" size="2">the literal 1 is a 32bit int, and 1<<32 overflow. </font></div>
<div><font face="Times New Roman" size="2"><br>
</font></div>
<div><font face="Times New Roman" size="2"><br>
</font></div>
<div><font face="Times New Roman" size="2">Here was my fix(has been tested), also found in attached file cms_large_heap_crash.patch</font></div>
<div><font face="Times New Roman" size="2"><br>
</font></div>
<div><i><font face="Times New Roman" size="2">+++ b/src/share/vm/memory/blockOffsetTable.hpp<br>
@@ -289,7 +289,7 @@<br>
};<br>
<br>
static size_t power_to_cards_back(uint i) {<br>
- return (size_t)(1 << (LogBase * i));<br>
+ return (size_t)1 << (LogBase * i);<br>
}<br>
static size_t power_to_words_back(uint i) {<br>
return power_to_cards_back(i) * N_words;</font></i></div>
<div><i><font face="Times New Roman" size="2"><br>
</font></i></div>
<div><font face="Times New Roman" size="2">Contributed-by: Hal Mo <<a href="mailto:kungu.mjh@taobao.com">kungu.mjh@taobao.com</a>>
</font></div>
<div><font face="Times New Roman" size="2"><br>
</font></div>
<div><font face="Times New Roman" size="2">Similar situation also found in G1, but the size is mega(2^20) based. 2^(32+20) is too large to overflow.</font></div>
<div><font face="Times New Roman" size="2"><br>
</font></div>
<div><font face="Times New Roman" size="2">Krystal remind me, this changeset cover the same code, <a href="http://hg.openjdk.java.net/hsx/hotspot-comp/hotspot/rev/c3a720eefe82" target="_blank">http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/c3a720eefe82</a>
.</font></div>
<div><font face="Times New Roman" size="2">I do not build it on visual studio, someone please help to review the compatibility with VS.</font></div>
<div><font face="Times New Roman" size="2"><br>
</font></div>
<div><font face="Times New Roman" size="2">Regards,</font></div>
<div><font face="Times New Roman" size="2"><br>
</font></div>
<div><font face="Times New Roman" size="2">Hal</font></div></div></div></div></div> </div></body>
</html>