Need help on debugging JVM crash

[Sundara Mohan M]

Hi Andrew,
    Attaching hs_err* from multiple hosts where both java thread top frame
is same.

Thanks
Sundar

On Tue, Mar 3, 2020 at 1:03 PM Andrew Haley <aph at redhat.com> wrote:

> On 3/3/20 5:39 PM, Sundara Mohan M wrote:
> >> Questions
> >> 1. When i looked at source code for printing stack trace i see
> followinghttps://
> github.com/openjdk/jdk11u/blob/master/src/hotspot/share/utilities/vmError.cpp#L696
> >> (Prints native stack trace)
> https://github.com/openjdk/jdk11u/blob/master/src/hotspot/share/utilities/vmError.cpp#L718
> >> (printing Java thread stack trace if it is involved in GC crash)
> >>   a. How do you know this java thread was involved in jvm crash?
>
> The top thread -- the first in the file -- is the one that crashed.
>
> >> When GC processes thread stack as root, the java thread first was
> >> recorded. This is why at crash, the java thread was printed out.
> >>
> >>   b. Can i assume the java thread printed after native stack trace was
> the
> >> culprit?
>
> Certainly not.
>
> >> Please check this thread stack frames, when GC is doing marking work, I
> >> think, it encountered a bad oop. Check:
> >>
> >> If it is a compiled frame, if so, it may related to compiled code.
> >>
> >>   c. Since i am seeing the same frame (~RuntimeStub::_new_array_Java, J
> >> 54174 c2 ch.qos.logback.classic.spi.ThrowableProxy.<init>..) but
> different
> >> stack trace in both crashes can this be the root cause?
> >>
> >> It is a C2 compiled frame. The bad oop could be a result of compiler.
> >>
> > Actually the top two frame are always same in different crashes
> > v ~RuntimeStub::_new_array_Java
> > J 54174 c2
> > ch.qos.logback.classic.spi.ThrowableProxy.<init>(Ljava/lang/Throwable;)V
> > (207 bytes) @ 0x00007f6687d92678 [0x00007f6687d8c700+0x0000000000005f78]
> > In this case do you think JVM code(frame 1) or C2 compiler code(frame 2)
> > might be issue?
>
> Probably not. My money would be on a bad library using Unsafe to do
> something unwise. But there are many other possibilities.
>
> > Is there any way to identify that and what kind of debug flags/settings
> > might give us this information?
> >
> >> It also needs detail debug information to make the conclusion.
> >>
> > Do you think any of the information dumped in hs_err* file might give us
> > more info (like registers content/Instructions/core file)?
> >
> > Can you please let me know what additional details might help to make the
> > conclusion? Also how to get those information?
>
> Let's see the complete hs_err file.
>
> --
> Andrew Haley  (he/him)
> Java Platform Lead Engineer
> Red Hat UK Ltd. <https://www.redhat.com>
> https://keybase.io/andrewhaley
> EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671
>
>