<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 16, 2022 at 12:52 AM David Holmes <<a href="mailto:david.holmes@oracle.com">david.holmes@oracle.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 16/08/2022 2:04 am, Thomas Stüfe wrote:<br>
> From the hex dump, I think it possible that you try to delete a <br>
> HandleMark, and that crashes.<br>
> <br>
> This corresponds with the code location in the stack, which is<br>
> <br>
> ```<br>
> V [libjvm.so+0x1a6d77a] Thread::~Thread()+0x8a<br>
> ```<br>
> <br>
> if gdb does not lie to me, this is near:<br>
> <br>
> ```<br>
> 0x7ffff74f91a9 <Thread::~Thread()+137>: callq 0x7ffff6843210 <br>
> <HandleMark::operator delete(void*)><br>
> ```<br>
> <br>
> He complains about a broken NMT header at 0x00007fba3f56ca90. So this is <br>
> the assumed NMT header:<br>
> <br>
> ```<br>
> 0x00007fba3f56ca90: 00 00 00 00 00 00 00 00 98 d0 37 3f ba 7f 00 00<br>
> ```<br>
> <br>
> But this is what the caller actually freed. Since NMT header precedes <br>
> the allocation:<br>
> <br>
> ```<br>
> 0x00007fba3f56caa0: 90 98 02 38 ba 7f 00 00 80 a3 02 38 ba 7f 00 00<br>
> 0x00007fba3f56cab0: e0 a3 02 38 ba 7f 00 00 f0 a3 02 38 ba 7f 00 00<br>
> 0x00007fba3f56cac0: c8 a4 02 38 ba 7f 00 00 d8 00 00 00 00 00 00 00<br>
> 0x00007fba3f56cad0: 60 a6 02 38 ba 7f 00 00 a0 e4 89 3e ba 7f 00 00<br>
> 0x00007fba3f56cae0: 90 98 02 38 ba 7f 00 00 90 ac 02 38 ba 7f 00 00<br>
> 0x00007fba3f56caf0: 78 05 40 f0 b9 7f 00 00 00 00 00 00 00 00 00 00<br>
> 0x00007fba3f56cb00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00<br>
> ```<br>
> <br>
> HandleMark has seven pointer-sized members:<br>
> <br>
> ```<br>
> Thread *_thread; // thread that owns this mark<br>
> HandleArea *_area; // saved handle area<br>
> Chunk *_chunk; // saved arena chunk<br>
> char *_hwm, *_max; // saved arena info<br>
> size_t _size_in_bytes; // size of handle area<br>
> // Link to previous active HandleMark in thread<br>
> HandleMark* _previous_handle_mark;<br>
> ```<br>
> <br>
> If I interprete 0x00007fba3f56caa0 as HandleMark*:<br>
> <br>
> ```<br>
> 0x00007fba3f56caa0: 90 98 02 38 ba 7f 00 00 80 a3 02 38 ba 7f 00 00<br>
> _thread _area<br>
> 0x00007fba3f56cab0: e0 a3 02 38 ba 7f 00 00 f0 a3 02 38 ba 7f 00 00<br>
> _chunk _hwm<br>
> 0x00007fba3f56cac0: c8 a4 02 38 ba 7f 00 00 d8 00 00 00 00 00 00 00<br>
> _max _size_in_bytes<br>
> 0x00007fba3f56cad0: 60 a6 02 38 ba 7f 00 00<br>
> _previous_handle_mark<br>
> ```<br>
> <br>
> Note how this fits:<br>
> <br>
> - The only 8-byte datum in this range is a size-like value for <br>
> _size_in_bytes (d8 = 216). All other slots look like pointers.<br>
> <br>
> - _chunk (0x7fba3802a3e0), _hwm (0x7fba3802a3f0) and _max <br>
> (0x7fba3802a4c8) look like they fit an Arena of size 216 perfectly:<br>
> _max - _chunk = 232. A Chunk looks like this:<br>
> <br>
> ```<br>
> //<br>
> // +-----------+--+--------------------------------------------+<br>
> // | |g | |<br>
> // | Chunk |a | Payload |<br>
> // | |p | |<br>
> // +-----------+--+--------------------------------------------+<br>
> // A B C D<br>
> ```<br>
> <br>
> and it has two pointer-sized members, so sizeof(Chunk) = 16. (232 - 16) <br>
> gives you the arena size of 216.<br>
> <br>
> Also, _hwm points to the beginning of the Arena (_chunk + 16). So this <br>
> looks like a mark for a HandleArea that has been cleared or never used.<br>
> <br>
> -------------<br>
> <br>
> I see two possibilities. Either someone overwrote the NMT header of the <br>
> block. Or, this HandleMark has never been allocated on the C-heap but <br>
> lives on the stack. In that case, something with <br>
> `Thread::thread->last_handle_mark()` could have gone wrong. Most <br>
> HandleMark instances live on the stack, the only ones that don't are the <br>
> initial marks created in `Thread::Thread()`.<br>
> <br>
> -----<br>
> <br>
> This is all pure hypotheses, since I don't have an hs-err file or any <br>
> information other than the stack and the hex dump.<br>
<br>
Thanks Thomas! That was way more than I expected! I suspect there is a <br>
HandleMark on the stack. I'm doing something unusual, but not prohibited.<br>
<br>
Cheers,<br>
David<br>
<br>
<br></blockquote><div><br></div><div>Happy to help :)</div><div><br></div><div>I wish the NMT error texts were a bit more informative, but its difficult to come up with better heuristics (same is true for <a href="https://bugs.openjdk.org/browse/JDK-8292318">https://bugs.openjdk.org/browse/JDK-8292318</a>, where the free'd pointer is really no pointer at all).</div><div><br></div><div>Cheers, Thomas</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
</blockquote></div></div>