JVM Crashes after few hours

Wed Nov 23 09:28:59 UTC 2016

Are there any tools to analyze the core dump file and what key information one should look for to debug the root cause of such crashes?

Thanks
Rajesh Agarwal

-----Original Message-----
From: Tobias Hartmann [mailto:tobias.hartmann at oracle.com] 
Sent: Wednesday, November 23, 2016 2:08 PM
To: Agarwal, Rajesh <Rajesh.Agarwal at snapon.com>; kirk at kodewerk.com
Cc: hotspot-compiler-dev at openjdk.java.net
Subject: Re: JVM Crashes after few hours

On 23.11.2016 09:24, Agarwal, Rajesh wrote:
> How can we get the core file?

A core file is automatically generated by the VM if a crash occurs. According to the hs_err file you provided:
"# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again"

So you need to enable core dumping on your machine.

> And is it possible to download the JDK 8 fastdebug VM for linux 64 bit? If yes, please provide the link.

Unfortunately, it's not possible to download a fastdebug VM. You would have to build it on your own or go through the official bug reporting process on http://bugreport.java.com.

Thanks,
Tobias

> 
> Thanks
> Rajesh Agarwal
> 
> 
> -----Original Message-----
> From: Tobias Hartmann [mailto:tobias.hartmann at oracle.com]
> Sent: Wednesday, November 23, 2016 1:50 PM
> To: Agarwal, Rajesh <Rajesh.Agarwal at snapon.com>; kirk at kodewerk.com
> Cc: hotspot-compiler-dev at openjdk.java.net
> Subject: Re: JVM Crashes after few hours
> 
> 
> On 23.11.2016 03:59, Agarwal, Rajesh wrote:
>> Apart from fastdebug VM, how can be debug the root cause of such issue? Do you think using Java Flight Recorder will help?
> 
> Since this is a bug in the VM, the Java Flight Recorder does not help. We would need a reproducer or at least a core file to investigate. Running with a fastdebug VM may also trigger an assert and generates a replay compilation file.
> 
> Best regards,
> Tobias
> 
>> Thanks
>> Rajesh Agarwal
>>
>>
>> -----Original Message-----
>> From: Tobias Hartmann [mailto:tobias.hartmann at oracle.com]
>> Sent: Wednesday, November 23, 2016 12:08 AM
>> To: Agarwal, Rajesh <Rajesh.Agarwal at snapon.com>; kirk at kodewerk.com
>> Cc: hotspot-compiler-dev at openjdk.java.net
>> Subject: Re: JVM Crashes after few hours
>>
>>
>> On 22.11.2016 19:18, Agarwal, Rajesh wrote:
>>> We are able to reproduce the issue everytime during load testing process of our app.
>>>
>>> Do you see any link with kryo which uses sun.misc.unsafe class for serializing the Java objects?
>>
>> It's very hard to say. The problem with JDK-8050079 was that the customers framework changed the common class hierarchy by replacing java.lang.Object and that triggered a bug in the compiler. This looks very similar so I would guess that this is caused by some unusual class hierarchy.
>>
>> If your time allows, you could try to build and run with a fastdebug VM build and check if we hit an assert. Also, a replay compilation file should be generated.
>>
>> Just out of curiosity, is there a specific reason you are using -XX:-ReduceInitialCardMarks?
>>
>> Best regards,
>> Tobias
>>
>>>
>>> Thanks
>>> Rajesh Agarwal
>>>
>>>
>>>
>>>
>>> On Tue, Nov 22, 2016 at 11:17 PM +0530, "Tobias Hartmann" <tobias.hartmann at oracle.com <mailto:tobias.hartmann at oracle.com>> wrote:
>>>
>>> Sorry, I got confused by the version string. Seems like you are using JDK 8u112 which includes the fix for 8050079.
>>>
>>> Do you happen to have a reproducer for this?
>>>
>>> Best regards,
>>> Tobias
>>>
>>> On 22.11.2016 18:25, Tobias Hartmann wrote:
>>>> Hi,
>>>>
>>>> the stack trace looks like:
>>>> https://bugs.openjdk.java.net/browse/JDK-8050079
>>>>
>>>> This bug was fixed in 8u40 and 9. Could you please check if you are able to reproduce the problem with 8u40 or later?
>>>>
>>>> Thanks,
>>>> Tobias
>>>>
>>>> On 22.11.2016 16:39, Agarwal, Rajesh wrote:
>>>>> Attached is the complete log. 
>>>>>
>>>>> Thanks
>>>>> Rajesh Agarwal
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: kirk at kodewerk.com [mailto:kirk at kodewerk.com]
>>>>> Sent: Tuesday, November 22, 2016 8:55 PM
>>>>> To: Agarwal, Rajesh <Rajesh.Agarwal at snapon.com>
>>>>> Cc: hotspot-compiler-dev at openjdk.java.net
>>>>> Subject: Re: JVM Crashes after few hours
>>>>>
>>>>> Hi,
>>>>>
>>>>> Can we see the rest of the log? This view of the log has cut off important information.
>>>>>
>>>>> Kind regards,
>>>>> Kirk Pepperdine
>>>>>
>>>>>> On Nov 22, 2016, at 8:43 AM, Agarwal, Rajesh <Rajesh.Agarwal at snapon.com> wrote:
>>>>>>
>>>>>> The JVM either crashes or hangs after running for few hours with following error.
>>>>>>
>>>>>> #
>>>>>> # A fatal error has been detected by the Java Runtime Environment:
>>>>>> #
>>>>>> #  SIGSEGV (0xb) at pc=0x00007f93aa58ed32, pid=24048,
>>>>>> tid=0x00007f9368a68700 # # JRE version: Java(TM) SE Runtime 
>>>>>> Environment (8.0_112-b15) (build 1.8.0_112-b15) # Java VM: Java
>>>>>> HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode linux-amd64 
>>>>>> compressed oops) # Problematic frame:
>>>>>> # V  [libjvm.so+0x640d32]
>>>>>> InstanceKlass::find_method_index(Array<Method*>*, Symbol*, 
>>>>>> Symbol*, bool, bool)+0x42 # # Failed to write core dump. Core 
>>>>>> dumps have been disabled. To enable core dumping, try "ulimit -c unlimited"
>>>>>> before starting Java again # # If you would like to submit a bug report, please visit:
>>>>>> #   http://bugreport.java.com/bugreport/crash.jsp
>>>>>> #
>>>>>>
>>>>>> ---------------  T H R E A D  ---------------
>>>>>>
>>>>>> Current thread (0x00007f93a4a50800):  JavaThread "C2 CompilerThread0" 
>>>>>> daemon [_thread_in_vm, id=24067,
>>>>>> stack(0x00007f9368968000,0x00007f9368a69000)]
>>>>>>
>>>>>> siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 
>>>>>> 0x00007f9326bd3218
>>>>>>
>>>>>> vm_info: Java HotSpot(TM) 64-Bit Server VM (25.112-b15) for
>>>>>> linux-amd64 JRE (1.8.0_112-b15), built on Sep 22 2016 21:10:53 by 
>>>>>> "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
>>>>>>
>>>>>> VM Arguments:
>>>>>> jvm_args: -Xms2560m -Xmx2560m -XX:-UseAdaptiveSizePolicy
>>>>>> -XX:NewRatio=2 -XX:SurvivorRatio=2 -XX:TargetSurvivorRatio=80 
>>>>>> -Xnoclassgc -verbose:gc -XX:+PrintGCDateStamps 
>>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:-ReduceInitialCardMarks 
>>>>>> -XX:+UseG1GC -XX:MaxGCPauseMillis=400 -XX:CICompilerCount=2 
>>>>>> -XX:+UnlockDiagnosticVMOptions -XX:+LogCompilation Thanks Rajesh 
>>>>>> Agarwal
>>>>>>
>>>>>>
>>>>>