Need help on debugging JVM crash

[Sundara Mohan M]

Waiting for moderator approval to get my hs_err* files sent.

Is being held until the list moderator can review it for approval.

The reason it is being held:

    Message body is too big: 1048807 bytes with a limit of 500 KB

Thanks
Sundar

On Tue, Mar 3, 2020 at 1:07 PM Sundara Mohan M <m.sundar85 at gmail.com> wrote:

> Hi Andrew,
>     Attaching hs_err* from multiple hosts where both java thread top frame
> is same.
>
> Thanks
> Sundar
>
> On Tue, Mar 3, 2020 at 1:03 PM Andrew Haley <aph at redhat.com> wrote:
>
>> On 3/3/20 5:39 PM, Sundara Mohan M wrote:
>> >> Questions
>> >> 1. When i looked at source code for printing stack trace i see
>> followinghttps://
>> github.com/openjdk/jdk11u/blob/master/src/hotspot/share/utilities/vmError.cpp#L696
>> >> (Prints native stack trace)
>> https://github.com/openjdk/jdk11u/blob/master/src/hotspot/share/utilities/vmError.cpp#L718
>> >> (printing Java thread stack trace if it is involved in GC crash)
>> >>   a. How do you know this java thread was involved in jvm crash?
>>
>> The top thread -- the first in the file -- is the one that crashed.
>>
>> >> When GC processes thread stack as root, the java thread first was
>> >> recorded. This is why at crash, the java thread was printed out.
>> >>
>> >>   b. Can i assume the java thread printed after native stack trace was
>> the
>> >> culprit?
>>
>> Certainly not.
>>
>> >> Please check this thread stack frames, when GC is doing marking work, I
>> >> think, it encountered a bad oop. Check:
>> >>
>> >> If it is a compiled frame, if so, it may related to compiled code.
>> >>
>> >>   c. Since i am seeing the same frame (~RuntimeStub::_new_array_Java, J
>> >> 54174 c2 ch.qos.logback.classic.spi.ThrowableProxy.<init>..) but
>> different
>> >> stack trace in both crashes can this be the root cause?
>> >>
>> >> It is a C2 compiled frame. The bad oop could be a result of compiler.
>> >>
>> > Actually the top two frame are always same in different crashes
>> > v ~RuntimeStub::_new_array_Java
>> > J 54174 c2
>> > ch.qos.logback.classic.spi.ThrowableProxy.<init>(Ljava/lang/Throwable;)V
>> > (207 bytes) @ 0x00007f6687d92678 [0x00007f6687d8c700+0x0000000000005f78]
>> > In this case do you think JVM code(frame 1) or C2 compiler code(frame 2)
>> > might be issue?
>>
>> Probably not. My money would be on a bad library using Unsafe to do
>> something unwise. But there are many other possibilities.
>>
>> > Is there any way to identify that and what kind of debug flags/settings
>> > might give us this information?
>> >
>> >> It also needs detail debug information to make the conclusion.
>> >>
>> > Do you think any of the information dumped in hs_err* file might give us
>> > more info (like registers content/Instructions/core file)?
>> >
>> > Can you please let me know what additional details might help to make
>> the
>> > conclusion? Also how to get those information?
>>
>> Let's see the complete hs_err file.
>>
>> --
>> Andrew Haley  (he/him)
>> Java Platform Lead Engineer
>> Red Hat UK Ltd. <https://www.redhat.com>
>> https://keybase.io/andrewhaley
>> EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671
>>
>>