Need help on debugging JVM crash

[Sundara Mohan M]

Another crash file.

Thanks
Sundar

On Tue, Mar 3, 2020 at 2:00 PM Sundara Mohan M <m.sundar85 at gmail.com> wrote:

> Attaching 1 file as a work around!
>
> Thanks
> Sundar
>
> On Tue, Mar 3, 2020 at 1:57 PM Stefan Karlsson <stefan.karlsson at oracle.com>
> wrote:
>
>> I have approved the message, but it isn't arriving. As a workaround,
>> could you try send one hs_err file at a time, and cut the rest of the
>> message? Each hs_err file is < 500 KB, so maybe that will work.
>>
>> StefanK
>>
>> On 2020-03-03 19:13, Sundara Mohan M wrote:
>> > Waiting for moderator approval to get my hs_err* files sent.
>> >
>> > Is being held until the list moderator can review it for approval.
>> >
>> > The reason it is being held:
>> >
>> >      Message body is too big: 1048807 bytes with a limit of 500 KB
>> >
>> > Thanks
>> > Sundar
>> >
>> > On Tue, Mar 3, 2020 at 1:07 PM Sundara Mohan M <m.sundar85 at gmail.com>
>> wrote:
>> >
>> >> Hi Andrew,
>> >>      Attaching hs_err* from multiple hosts where both java thread top
>> frame
>> >> is same.
>> >>
>> >> Thanks
>> >> Sundar
>> >>
>> >> On Tue, Mar 3, 2020 at 1:03 PM Andrew Haley <aph at redhat.com> wrote:
>> >>
>> >>> On 3/3/20 5:39 PM, Sundara Mohan M wrote:
>> >>>>> Questions
>> >>>>> 1. When i looked at source code for printing stack trace i see
>> >>> followinghttps://
>> >>>
>> github.com/openjdk/jdk11u/blob/master/src/hotspot/share/utilities/vmError.cpp#L696
>> >>>>> (Prints native stack trace)
>> >>>
>> https://github.com/openjdk/jdk11u/blob/master/src/hotspot/share/utilities/vmError.cpp#L718
>> >>>>> (printing Java thread stack trace if it is involved in GC crash)
>> >>>>>    a. How do you know this java thread was involved in jvm crash?
>> >>> The top thread -- the first in the file -- is the one that crashed.
>> >>>
>> >>>>> When GC processes thread stack as root, the java thread first was
>> >>>>> recorded. This is why at crash, the java thread was printed out.
>> >>>>>
>> >>>>>    b. Can i assume the java thread printed after native stack trace
>> was
>> >>> the
>> >>>>> culprit?
>> >>> Certainly not.
>> >>>
>> >>>>> Please check this thread stack frames, when GC is doing marking
>> work, I
>> >>>>> think, it encountered a bad oop. Check:
>> >>>>>
>> >>>>> If it is a compiled frame, if so, it may related to compiled code.
>> >>>>>
>> >>>>>    c. Since i am seeing the same frame
>> (~RuntimeStub::_new_array_Java, J
>> >>>>> 54174 c2 ch.qos.logback.classic.spi.ThrowableProxy.<init>..) but
>> >>> different
>> >>>>> stack trace in both crashes can this be the root cause?
>> >>>>>
>> >>>>> It is a C2 compiled frame. The bad oop could be a result of
>> compiler.
>> >>>>>
>> >>>> Actually the top two frame are always same in different crashes
>> >>>> v ~RuntimeStub::_new_array_Java
>> >>>> J 54174 c2
>> >>>>
>> ch.qos.logback.classic.spi.ThrowableProxy.<init>(Ljava/lang/Throwable;)V
>> >>>> (207 bytes) @ 0x00007f6687d92678
>> [0x00007f6687d8c700+0x0000000000005f78]
>> >>>> In this case do you think JVM code(frame 1) or C2 compiler
>> code(frame 2)
>> >>>> might be issue?
>> >>> Probably not. My money would be on a bad library using Unsafe to do
>> >>> something unwise. But there are many other possibilities.
>> >>>
>> >>>> Is there any way to identify that and what kind of debug
>> flags/settings
>> >>>> might give us this information?
>> >>>>
>> >>>>> It also needs detail debug information to make the conclusion.
>> >>>>>
>> >>>> Do you think any of the information dumped in hs_err* file might
>> give us
>> >>>> more info (like registers content/Instructions/core file)?
>> >>>>
>> >>>> Can you please let me know what additional details might help to make
>> >>> the
>> >>>> conclusion? Also how to get those information?
>> >>> Let's see the complete hs_err file.
>> >>>
>> >>> --
>> >>> Andrew Haley  (he/him)
>> >>> Java Platform Lead Engineer
>> >>> Red Hat UK Ltd. <https://www.redhat.com>
>> >>> https://keybase.io/andrewhaley
>> >>> EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671
>> >>>
>> >>>
>>
>>