Crash in the MallocTracker::record_free
David Holmes
david.holmes at oracle.com
Mon Aug 15 13:17:32 UTC 2022
On 15/08/2022 7:21 pm, Thomas Stüfe wrote:
>
>
> On Mon, Aug 15, 2022 at 11:16 AM David Holmes <david.holmes at oracle.com
> <mailto:david.holmes at oracle.com>> wrote:
>
> On 15/08/2022 6:28 pm, Thomas Stüfe wrote:
> > On Mon, Aug 15, 2022 at 10:27 AM Thomas Stüfe
> <thomas.stuefe at gmail.com <mailto:thomas.stuefe at gmail.com>
> > <mailto:thomas.stuefe at gmail.com
> <mailto:thomas.stuefe at gmail.com>>> wrote:
> >
> >
> > On Mon, Aug 15, 2022 at 10:08 AM David Holmes
> > <david.holmes at oracle.com <mailto:david.holmes at oracle.com>
> <mailto:david.holmes at oracle.com <mailto:david.holmes at oracle.com>>>
> wrote:
> >
> > Hi Thomas,
> >
> > On 15/08/2022 3:42 pm, Thomas Stüfe wrote:
> > > ..The only way I could see this happening is if tty
> itself is
> > not valid.
> > > Is this very early during initialization?
> >
> > Quite the opposite, it is during VM termination.
> >
> >
> > Yes, that makes sense too. tty is defaultStream::instance,
> which we
> > delete in ostream_exit() in DestroyVM.
> >
> > Since this has bugging me for a while (more during
> > pre-initialization, but its the same problem) I opened
> > https://bugs.openjdk.org/browse/JDK-8292351
> <https://bugs.openjdk.org/browse/JDK-8292351>
> > <https://bugs.openjdk.org/browse/JDK-8292351
> <https://bugs.openjdk.org/browse/JDK-8292351>> to track this. Should
> > be trivial to fix.
>
> Perhaps trivial to not crash, but it means we are attempting various
> "logging" actions after they can't be done - which is a pain. Afterall
> we do want to see the real error here.
>
>
> That can happen though. As in your case, NMT telling us that the C-heap
> is corrupted during VM cleanup. So it is a valid scenario.
Sure, I guess we can try and do the stream/tty deletion a bit later ...
> >
> > As for your crash, if you comment out "delete
> defaultStream::instance;"
> > in ostream.cpp, you should be able to see the real issue.
>
> Thanks - needed to do a little more than just that but I got the idea.
> So that leads me to:
>
> # Internal Error
> (/scratch/users/daholme/jdk-dev3.git/open/src/hotspot/share/services/mallocHeader.inline.hpp:61),
>
> pid=2666, tid=2668
> # fatal error: NMT corruption: Block at 0x00007fba3f56ca90: header
> canary broken
>
>
> No hex dump?
stdout: [NMT Block at 0x00007fba3f56ca90, corruption at:
0x00007fba3f56ca90:
0x00007fba3f56ca10: e0 ca 56 3f ba 7f 00 00 f8 cc 56 3f ba 7f 00 00
0x00007fba3f56ca20: 0a 00 00 00 00 00 00 00 78 05 40 f0 b9 7f 00 00
0x00007fba3f56ca30: 40 03 7e 27 ba 7f 00 00 08 cc 56 3f ba 7f 00 00
0x00007fba3f56ca40: 50 cb 56 3f ba 7f 00 00 f0 27 6f 3d ba 7f 00 00
0x00007fba3f56ca50: 01 00 00 00 00 00 00 00 90 98 02 38 ba 7f 00 00
0x00007fba3f56ca60: 02 00 00 00 00 00 00 00 a0 e4 89 3e ba 7f 00 00
0x00007fba3f56ca70: a0 ca 56 3f ba 7f 00 00 0a 00 00 00 0e 00 00 00
0x00007fba3f56ca80: 40 03 7e 27 ba 7f 00 00 f8 cc 56 3f ba 7f 00 00
0x00007fba3f56ca90: 00 00 00 00 00 00 00 00 98 d0 37 3f ba 7f 00 00
0x00007fba3f56caa0: 90 98 02 38 ba 7f 00 00 80 a3 02 38 ba 7f 00 00
0x00007fba3f56cab0: e0 a3 02 38 ba 7f 00 00 f0 a3 02 38 ba 7f 00 00
0x00007fba3f56cac0: c8 a4 02 38 ba 7f 00 00 d8 00 00 00 00 00 00 00
0x00007fba3f56cad0: 60 a6 02 38 ba 7f 00 00 a0 e4 89 3e ba 7f 00 00
0x00007fba3f56cae0: 90 98 02 38 ba 7f 00 00 90 ac 02 38 ba 7f 00 00
0x00007fba3f56caf0: 78 05 40 f0 b9 7f 00 00 00 00 00 00 00 00 00 00
0x00007fba3f56cb00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
David
-----
>
> >
> > Cheers,
> > David
> >
> > > On Mon, Aug 15, 2022 at 7:40 AM Thomas Stüfe
> > <thomas.stuefe at gmail.com <mailto:thomas.stuefe at gmail.com>
> <mailto:thomas.stuefe at gmail.com <mailto:thomas.stuefe at gmail.com>>
> > > <mailto:thomas.stuefe at gmail.com
> <mailto:thomas.stuefe at gmail.com>
> > <mailto:thomas.stuefe at gmail.com
> <mailto:thomas.stuefe at gmail.com>>>> wrote:
> > >
> > > Hi David,
> > >
> > > NMT detected a heap corruption or the pointer to
> be freed is
> > > invalid. And then the reporter itself crashed while
> > printing the
> > > report. That is strange, since the reporter should be
> > resilient
> > > against crashes. It uses os::print_hex_dump, which
> should use
> > > SafeFetch to check if the to-be-dumped info is
> accessible.
> > >
> > > Any more information?
> > >
> > > Cheers, Thomas
> > >
> > > On Mon, Aug 15, 2022 at 7:18 AM David Holmes
> > > <david.holmes at oracle.com
> <mailto:david.holmes at oracle.com> <mailto:david.holmes at oracle.com
> <mailto:david.holmes at oracle.com>>
> > <mailto:david.holmes at oracle.com
> <mailto:david.holmes at oracle.com>
> > <mailto:david.holmes at oracle.com
> <mailto:david.holmes at oracle.com>>>> wrote:
> > >
> > > I'm getting the below crash:
> > >
> > > #
> > > # A fatal error has been detected by the Java
> Runtime
> > Environment:
> > > #
> > > # SIGSEGV (0xb) at pc=0x00007f3e3e24b417,
> pid=4537,
> > tid=4538
> > > #
> > > # JRE version: Java(TM) SE Runtime Environment
> (20.0)
> > (fastdebug
> > > build
> > > 20-internal-2022-06-10-0001400.daholme...)
> > > # Java VM: Java HotSpot(TM) 64-Bit Server VM
> (fastdebug
> > > 20-internal-2022-06-10-0001400.daholme..., mixed
> > mode, sharing,
> > > tiered,
> > > compressed oops, compressed class ptrs, g1 gc,
> > linux-amd64)
> > > # Problematic frame:
> > > # V [libjvm.so+0x174b417]
> > outputStream::print_cr(char const*,
> > > ...)+0x67
> > >
> > > --------------- T H R E A D ---------------
> > >
> > > Current thread (0x00007f3e38029890): Thread
> "Unknown
> > thread"
> > > [stack:
> > > 0x00007f3e3f8a9000,0x00007f3e3f9aa000] [id=4538]
> > >
> > > Stack: [0x00007f3e3f8a9000,0x00007f3e3f9aa000],
> > > sp=0x00007f3e3f9a84b0,
> > > free space=1021k
> > > Native frames: (J=compiled Java code,
> j=interpreted,
> > Vv=VM code,
> > > C=native code)
> > > V [libjvm.so+0x174b417]
> outputStream::print_cr(char
> > const*,
> > > ...)+0x67
> > > V [libjvm.so+0x157d708]
> > > MallocHeader::print_block_on_error(outputStream*,
> > unsigned
> > > char*) const+0x38
> > > V [libjvm.so+0x157f2ef]
> > MallocHeader::assert_block_integrity()
> > > const+0x10f
> > > V [libjvm.so+0x157eb6b]
> > MallocTracker::record_free(void*)+0x3b
> > > V [libjvm.so+0x172d3ef] os::free(void*)+0xbf
> > > V [libjvm.so+0x1a6d77a] Thread::~Thread()+0x8a
> > > V [libjvm.so+0x1060292]
> JavaThread::~JavaThread()+0x12
> > > V [libjvm.so+0x1a8b174]
> Threads::destroy_vm()+0x264
> > >
> > > Any tips on trying to figure out what is going
> wrong?
> > >
> > > Thanks,
> > > David
> > >
> >
>
More information about the hotspot-runtime-dev
mailing list