Need help on debugging JVM crash

[Stefan Karlsson]

I have approved the message, but it isn't arriving. As a workaround, 
could you try send one hs_err file at a time, and cut the rest of the 
message? Each hs_err file is < 500 KB, so maybe that will work.

StefanK

On 2020-03-03 19:13, Sundara Mohan M wrote:
> Waiting for moderator approval to get my hs_err* files sent.
>
> Is being held until the list moderator can review it for approval.
>
> The reason it is being held:
>
>      Message body is too big: 1048807 bytes with a limit of 500 KB
>
> Thanks
> Sundar
>
> On Tue, Mar 3, 2020 at 1:07 PM Sundara Mohan M <m.sundar85 at gmail.com> wrote:
>
>> Hi Andrew,
>>      Attaching hs_err* from multiple hosts where both java thread top frame
>> is same.
>>
>> Thanks
>> Sundar
>>
>> On Tue, Mar 3, 2020 at 1:03 PM Andrew Haley <aph at redhat.com> wrote:
>>
>>> On 3/3/20 5:39 PM, Sundara Mohan M wrote:
>>>>> Questions
>>>>> 1. When i looked at source code for printing stack trace i see
>>> followinghttps://
>>> github.com/openjdk/jdk11u/blob/master/src/hotspot/share/utilities/vmError.cpp#L696
>>>>> (Prints native stack trace)
>>> https://github.com/openjdk/jdk11u/blob/master/src/hotspot/share/utilities/vmError.cpp#L718
>>>>> (printing Java thread stack trace if it is involved in GC crash)
>>>>>    a. How do you know this java thread was involved in jvm crash?
>>> The top thread -- the first in the file -- is the one that crashed.
>>>
>>>>> When GC processes thread stack as root, the java thread first was
>>>>> recorded. This is why at crash, the java thread was printed out.
>>>>>
>>>>>    b. Can i assume the java thread printed after native stack trace was
>>> the
>>>>> culprit?
>>> Certainly not.
>>>
>>>>> Please check this thread stack frames, when GC is doing marking work, I
>>>>> think, it encountered a bad oop. Check:
>>>>>
>>>>> If it is a compiled frame, if so, it may related to compiled code.
>>>>>
>>>>>    c. Since i am seeing the same frame (~RuntimeStub::_new_array_Java, J
>>>>> 54174 c2 ch.qos.logback.classic.spi.ThrowableProxy.<init>..) but
>>> different
>>>>> stack trace in both crashes can this be the root cause?
>>>>>
>>>>> It is a C2 compiled frame. The bad oop could be a result of compiler.
>>>>>
>>>> Actually the top two frame are always same in different crashes
>>>> v ~RuntimeStub::_new_array_Java
>>>> J 54174 c2
>>>> ch.qos.logback.classic.spi.ThrowableProxy.<init>(Ljava/lang/Throwable;)V
>>>> (207 bytes) @ 0x00007f6687d92678 [0x00007f6687d8c700+0x0000000000005f78]
>>>> In this case do you think JVM code(frame 1) or C2 compiler code(frame 2)
>>>> might be issue?
>>> Probably not. My money would be on a bad library using Unsafe to do
>>> something unwise. But there are many other possibilities.
>>>
>>>> Is there any way to identify that and what kind of debug flags/settings
>>>> might give us this information?
>>>>
>>>>> It also needs detail debug information to make the conclusion.
>>>>>
>>>> Do you think any of the information dumped in hs_err* file might give us
>>>> more info (like registers content/Instructions/core file)?
>>>>
>>>> Can you please let me know what additional details might help to make
>>> the
>>>> conclusion? Also how to get those information?
>>> Let's see the complete hs_err file.
>>>
>>> --
>>> Andrew Haley  (he/him)
>>> Java Platform Lead Engineer
>>> Red Hat UK Ltd. <https://www.redhat.com>
>>> https://keybase.io/andrewhaley
>>> EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671
>>>
>>>