<div dir="ltr"><div>From the hex dump, I think it possible that you try to delete a HandleMark, and that crashes.<br><br>This corresponds with the code location in the stack, which is <br><br>```<br>V [libjvm.so+0x1a6d77a] Thread::~Thread()+0x8a<br>```<br></div><div><br></div><div>if gdb does not lie to me, this is near:</div><div><br></div><div>```</div><div>0x7ffff74f91a9 <Thread::~Thread()+137>: callq 0x7ffff6843210 <HandleMark::operator delete(void*)> </div><div>```<br><br>He complains about a broken NMT header at 0x00007fba3f56ca90. So this is the assumed NMT header: <br><br></div><div>```<br><font face="monospace">0x00007fba3f56ca90: 00 00 00 00 00 00 00 00 98 d0 37 3f ba 7f 00 00<br></font>```<br><br>But this is what the caller actually freed. Since NMT header precedes the allocation:</div><div><br>```<br><font face="monospace">0x00007fba3f56caa0: 90 98 02 38 ba 7f 00 00 80 a3 02 38 ba 7f 00 00<br>0x00007fba3f56cab0: e0 a3 02 38 ba 7f 00 00 f0 a3 02 38 ba 7f 00 00<br>0x00007fba3f56cac0: c8 a4 02 38 ba 7f 00 00 d8 00 00 00 00 00 00 00<br>0x00007fba3f56cad0: 60 a6 02 38 ba 7f 00 00 a0 e4 89 3e ba 7f 00 00<br>0x00007fba3f56cae0: 90 98 02 38 ba 7f 00 00 90 ac 02 38 ba 7f 00 00<br>0x00007fba3f56caf0: 78 05 40 f0 b9 7f 00 00 00 00 00 00 00 00 00 00<br>0x00007fba3f56cb00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00</font><br>```<br><br>HandleMark has seven pointer-sized members:<br><br>```<br><font face="monospace"> Thread *_thread; // thread that owns this mark<br> HandleArea *_area; // saved handle area<br> Chunk *_chunk; // saved arena chunk<br> char *_hwm, *_max; // saved arena info<br> size_t _size_in_bytes; // size of handle area<br> // Link to previous active HandleMark in thread<br> HandleMark* _previous_handle_mark;</font><br>```<br> <br>If I interprete 0x00007fba3f56caa0 as HandleMark*:<br><br>```<br><font face="monospace">0x00007fba3f56caa0: 90 98 02 38 ba 7f 00 00 80 a3 02 38 ba 7f 00 00<br> _thread _area<br>0x00007fba3f56cab0: e0 a3 02 38 ba 7f 00 00 f0 a3 02 38 ba 7f 00 00<br> _chunk _hwm<br>0x00007fba3f56cac0: c8 a4 02 38 ba 7f 00 00 d8 00 00 00 00 00 00 00<br> _max _size_in_bytes<br>0x00007fba3f56cad0: 60 a6 02 38 ba 7f 00 00<br> _previous_handle_mark</font><br>```<br><br>Note how this fits:<br><br>- The only 8-byte datum in this range is a size-like value for _size_in_bytes (d8 = 216). All other slots look like pointers.<br><br>- _chunk (0x7fba3802a3e0), _hwm (0x7fba3802a3f0) and _max (0x7fba3802a4c8) look like they fit an Arena of size 216 perfectly:<br>_max - _chunk = 232. A Chunk looks like this:<br><br><font face="monospace">```<br> //<br> // +-----------+--+--------------------------------------------+<br> // | |g | |<br> // | Chunk |a | Payload |<br> // | |p | |<br> // +-----------+--+--------------------------------------------+<br> // A B C D<br>```</font><br><br>and it has two pointer-sized members, so sizeof(Chunk) = 16. (232 - 16) gives you the arena size of 216.<br><br>Also, _hwm points to the beginning of the Arena (_chunk + 16). So this looks like a mark for a HandleArea that has been cleared or never used.<br><br>-------------<br><br>I see two possibilities. Either someone overwrote the NMT header of the block. Or, this HandleMark has never been allocated on the C-heap but lives on the stack. In that case, something with `Thread::thread->last_handle_mark()` could have gone wrong. Most HandleMark instances live on the stack, the only ones that don't are the initial marks created in `Thread::Thread()`.<br><br>-----<br><br>This is all pure hypotheses, since I don't have an hs-err file or any information other than the stack and the hex dump.<br><br>Cheers, Thomas<br> <br>On Mon, Aug 15, 2022 at 4:02 PM <<a href="mailto:daniel.daugherty@oracle.com">daniel.daugherty@oracle.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">This discussion reminded me of one of the bugs that I filed last week:<br>
<br>
JDK-8292318 <br>
runtime/cds/appcds/dynamicArchive/methodHandles/MethodHandlesAsCollectorTest.java <br>
fails with "fatal error: NMT corruption: Block at <addr>: block address <br>
is unaligned"<br>
<a href="https://bugs.openjdk.org/browse/JDK-8292318" rel="noreferrer" target="_blank">https://bugs.openjdk.org/browse/JDK-8292318</a><br>
<br>
So there might some overlap here...<br>
<br>
Dan<br>
<br>
<br>
<br>
On 8/15/22 9:17 AM, David Holmes wrote:<br>
> On 15/08/2022 7:21 pm, Thomas Stüfe wrote:<br>
>><br>
>><br>
>> On Mon, Aug 15, 2022 at 11:16 AM David Holmes <br>
>> <<a href="mailto:david.holmes@oracle.com" target="_blank">david.holmes@oracle.com</a> <mailto:<a href="mailto:david.holmes@oracle.com" target="_blank">david.holmes@oracle.com</a>>> wrote:<br>
>><br>
>> On 15/08/2022 6:28 pm, Thomas Stüfe wrote:<br>
>> > On Mon, Aug 15, 2022 at 10:27 AM Thomas Stüfe<br>
>> <<a href="mailto:thomas.stuefe@gmail.com" target="_blank">thomas.stuefe@gmail.com</a> <mailto:<a href="mailto:thomas.stuefe@gmail.com" target="_blank">thomas.stuefe@gmail.com</a>><br>
>> > <mailto:<a href="mailto:thomas.stuefe@gmail.com" target="_blank">thomas.stuefe@gmail.com</a><br>
>> <mailto:<a href="mailto:thomas.stuefe@gmail.com" target="_blank">thomas.stuefe@gmail.com</a>>>> wrote:<br>
>> ><br>
>> ><br>
>> > On Mon, Aug 15, 2022 at 10:08 AM David Holmes<br>
>> > <<a href="mailto:david.holmes@oracle.com" target="_blank">david.holmes@oracle.com</a> <mailto:<a href="mailto:david.holmes@oracle.com" target="_blank">david.holmes@oracle.com</a>><br>
>> <mailto:<a href="mailto:david.holmes@oracle.com" target="_blank">david.holmes@oracle.com</a> <mailto:<a href="mailto:david.holmes@oracle.com" target="_blank">david.holmes@oracle.com</a>>>><br>
>> wrote:<br>
>> ><br>
>> > Hi Thomas,<br>
>> ><br>
>> > On 15/08/2022 3:42 pm, Thomas Stüfe wrote:<br>
>> > > ..The only way I could see this happening is if tty<br>
>> itself is<br>
>> > not valid.<br>
>> > > Is this very early during initialization?<br>
>> ><br>
>> > Quite the opposite, it is during VM termination.<br>
>> ><br>
>> ><br>
>> > Yes, that makes sense too. tty is defaultStream::instance,<br>
>> which we<br>
>> > delete in ostream_exit() in DestroyVM.<br>
>> ><br>
>> > Since this has bugging me for a while (more during<br>
>> > pre-initialization, but its the same problem) I opened<br>
>> > <a href="https://bugs.openjdk.org/browse/JDK-8292351" rel="noreferrer" target="_blank">https://bugs.openjdk.org/browse/JDK-8292351</a><br>
>> <<a href="https://bugs.openjdk.org/browse/JDK-8292351" rel="noreferrer" target="_blank">https://bugs.openjdk.org/browse/JDK-8292351</a>><br>
>> > <<a href="https://bugs.openjdk.org/browse/JDK-8292351" rel="noreferrer" target="_blank">https://bugs.openjdk.org/browse/JDK-8292351</a><br>
>> <<a href="https://bugs.openjdk.org/browse/JDK-8292351" rel="noreferrer" target="_blank">https://bugs.openjdk.org/browse/JDK-8292351</a>>> to track this. Should<br>
>> > be trivial to fix.<br>
>><br>
>> Perhaps trivial to not crash, but it means we are attempting various<br>
>> "logging" actions after they can't be done - which is a pain. <br>
>> Afterall<br>
>> we do want to see the real error here.<br>
>><br>
>><br>
>> That can happen though. As in your case, NMT telling us that the <br>
>> C-heap is corrupted during VM cleanup. So it is a valid scenario.<br>
><br>
> Sure, I guess we can try and do the stream/tty deletion a bit later ...<br>
><br>
>> ><br>
>> > As for your crash, if you comment out "delete<br>
>> defaultStream::instance;"<br>
>> > in ostream.cpp, you should be able to see the real issue.<br>
>><br>
>> Thanks - needed to do a little more than just that but I got the <br>
>> idea.<br>
>> So that leads me to:<br>
>><br>
>> # Internal Error<br>
>> (/scratch/users/daholme/jdk-dev3.git/open/src/hotspot/share/services/mallocHeader.inline.hpp:61),<br>
>><br>
>> pid=2666, tid=2668<br>
>> # fatal error: NMT corruption: Block at 0x00007fba3f56ca90: header<br>
>> canary broken<br>
>><br>
>><br>
>> No hex dump?<br>
><br>
> stdout: [NMT Block at 0x00007fba3f56ca90, corruption at: <br>
> 0x00007fba3f56ca90:<br>
> 0x00007fba3f56ca10: e0 ca 56 3f ba 7f 00 00 f8 cc 56 3f ba 7f 00 00<br>
> 0x00007fba3f56ca20: 0a 00 00 00 00 00 00 00 78 05 40 f0 b9 7f 00 00<br>
> 0x00007fba3f56ca30: 40 03 7e 27 ba 7f 00 00 08 cc 56 3f ba 7f 00 00<br>
> 0x00007fba3f56ca40: 50 cb 56 3f ba 7f 00 00 f0 27 6f 3d ba 7f 00 00<br>
> 0x00007fba3f56ca50: 01 00 00 00 00 00 00 00 90 98 02 38 ba 7f 00 00<br>
> 0x00007fba3f56ca60: 02 00 00 00 00 00 00 00 a0 e4 89 3e ba 7f 00 00<br>
> 0x00007fba3f56ca70: a0 ca 56 3f ba 7f 00 00 0a 00 00 00 0e 00 00 00<br>
> 0x00007fba3f56ca80: 40 03 7e 27 ba 7f 00 00 f8 cc 56 3f ba 7f 00 00<br>
> 0x00007fba3f56ca90: 00 00 00 00 00 00 00 00 98 d0 37 3f ba 7f 00 00<br>
> 0x00007fba3f56caa0: 90 98 02 38 ba 7f 00 00 80 a3 02 38 ba 7f 00 00<br>
> 0x00007fba3f56cab0: e0 a3 02 38 ba 7f 00 00 f0 a3 02 38 ba 7f 00 00<br>
> 0x00007fba3f56cac0: c8 a4 02 38 ba 7f 00 00 d8 00 00 00 00 00 00 00<br>
> 0x00007fba3f56cad0: 60 a6 02 38 ba 7f 00 00 a0 e4 89 3e ba 7f 00 00<br>
> 0x00007fba3f56cae0: 90 98 02 38 ba 7f 00 00 90 ac 02 38 ba 7f 00 00<br>
> 0x00007fba3f56caf0: 78 05 40 f0 b9 7f 00 00 00 00 00 00 00 00 00 00<br>
> 0x00007fba3f56cb00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00<br>
><br>
> David<br>
> -----<br>
><br>
>><br>
>> ><br>
>> > Cheers,<br>
>> > David<br>
>> ><br>
>> > > On Mon, Aug 15, 2022 at 7:40 AM Thomas Stüfe<br>
>> > <<a href="mailto:thomas.stuefe@gmail.com" target="_blank">thomas.stuefe@gmail.com</a> <mailto:<a href="mailto:thomas.stuefe@gmail.com" target="_blank">thomas.stuefe@gmail.com</a>><br>
>> <mailto:<a href="mailto:thomas.stuefe@gmail.com" target="_blank">thomas.stuefe@gmail.com</a> <mailto:<a href="mailto:thomas.stuefe@gmail.com" target="_blank">thomas.stuefe@gmail.com</a>>><br>
>> > > <mailto:<a href="mailto:thomas.stuefe@gmail.com" target="_blank">thomas.stuefe@gmail.com</a><br>
>> <mailto:<a href="mailto:thomas.stuefe@gmail.com" target="_blank">thomas.stuefe@gmail.com</a>><br>
>> > <mailto:<a href="mailto:thomas.stuefe@gmail.com" target="_blank">thomas.stuefe@gmail.com</a><br>
>> <mailto:<a href="mailto:thomas.stuefe@gmail.com" target="_blank">thomas.stuefe@gmail.com</a>>>>> wrote:<br>
>> > ><br>
>> > > Hi David,<br>
>> > ><br>
>> > > NMT detected a heap corruption or the pointer to<br>
>> be freed is<br>
>> > > invalid. And then the reporter itself crashed <br>
>> while<br>
>> > printing the<br>
>> > > report. That is strange, since the reporter <br>
>> should be<br>
>> > resilient<br>
>> > > against crashes. It uses os::print_hex_dump, which<br>
>> should use<br>
>> > > SafeFetch to check if the to-be-dumped info is<br>
>> accessible.<br>
>> > ><br>
>> > > Any more information?<br>
>> > ><br>
>> > > Cheers, Thomas<br>
>> > ><br>
>> > > On Mon, Aug 15, 2022 at 7:18 AM David Holmes<br>
>> > > <<a href="mailto:david.holmes@oracle.com" target="_blank">david.holmes@oracle.com</a><br>
>> <mailto:<a href="mailto:david.holmes@oracle.com" target="_blank">david.holmes@oracle.com</a>> <mailto:<a href="mailto:david.holmes@oracle.com" target="_blank">david.holmes@oracle.com</a><br>
>> <mailto:<a href="mailto:david.holmes@oracle.com" target="_blank">david.holmes@oracle.com</a>>><br>
>> > <mailto:<a href="mailto:david.holmes@oracle.com" target="_blank">david.holmes@oracle.com</a><br>
>> <mailto:<a href="mailto:david.holmes@oracle.com" target="_blank">david.holmes@oracle.com</a>><br>
>> > <mailto:<a href="mailto:david.holmes@oracle.com" target="_blank">david.holmes@oracle.com</a><br>
>> <mailto:<a href="mailto:david.holmes@oracle.com" target="_blank">david.holmes@oracle.com</a>>>>> wrote:<br>
>> > ><br>
>> > > I'm getting the below crash:<br>
>> > ><br>
>> > > #<br>
>> > > # A fatal error has been detected by the Java<br>
>> Runtime<br>
>> > Environment:<br>
>> > > #<br>
>> > > # SIGSEGV (0xb) at pc=0x00007f3e3e24b417,<br>
>> pid=4537,<br>
>> > tid=4538<br>
>> > > #<br>
>> > > # JRE version: Java(TM) SE Runtime Environment<br>
>> (20.0)<br>
>> > (fastdebug<br>
>> > > build<br>
>> > > 20-internal-2022-06-10-0001400.daholme...)<br>
>> > > # Java VM: Java HotSpot(TM) 64-Bit Server VM<br>
>> (fastdebug<br>
>> > > 20-internal-2022-06-10-0001400.daholme..., mixed<br>
>> > mode, sharing,<br>
>> > > tiered,<br>
>> > > compressed oops, compressed class ptrs, g1 gc,<br>
>> > linux-amd64)<br>
>> > > # Problematic frame:<br>
>> > > # V [libjvm.so+0x174b417]<br>
>> > outputStream::print_cr(char const*,<br>
>> > > ...)+0x67<br>
>> > ><br>
>> > > --------------- T H R E A D ---------------<br>
>> > ><br>
>> > > Current thread (0x00007f3e38029890): Thread<br>
>> "Unknown<br>
>> > thread"<br>
>> > > [stack:<br>
>> > > 0x00007f3e3f8a9000,0x00007f3e3f9aa000] [id=4538]<br>
>> > ><br>
>> > > Stack: <br>
>> [0x00007f3e3f8a9000,0x00007f3e3f9aa000],<br>
>> > > sp=0x00007f3e3f9a84b0,<br>
>> > > free space=1021k<br>
>> > > Native frames: (J=compiled Java code,<br>
>> j=interpreted,<br>
>> > Vv=VM code,<br>
>> > > C=native code)<br>
>> > > V [libjvm.so+0x174b417] <br>
>> outputStream::print_cr(char<br>
>> > const*,<br>
>> > > ...)+0x67<br>
>> > > V [libjvm.so+0x157d708]<br>
>> > > MallocHeader::print_block_on_error(outputStream*,<br>
>> > unsigned<br>
>> > > char*) const+0x38<br>
>> > > V [libjvm.so+0x157f2ef]<br>
>> > MallocHeader::assert_block_integrity()<br>
>> > > const+0x10f<br>
>> > > V [libjvm.so+0x157eb6b]<br>
>> > MallocTracker::record_free(void*)+0x3b<br>
>> > > V [libjvm.so+0x172d3ef] os::free(void*)+0xbf<br>
>> > > V [libjvm.so+0x1a6d77a] <br>
>> Thread::~Thread()+0x8a<br>
>> > > V [libjvm.so+0x1060292] <br>
>> JavaThread::~JavaThread()+0x12<br>
>> > > V [libjvm.so+0x1a8b174] <br>
>> Threads::destroy_vm()+0x264<br>
>> > ><br>
>> > > Any tips on trying to figure out what is going<br>
>> wrong?<br>
>> > ><br>
>> > > Thanks,<br>
>> > > David<br>
>> > ><br>
>> ><br>
>><br>
<br>
</blockquote></div></div>