JVM Crashes after few hours

[Agarwal, Rajesh]

Apart from fastdebug VM, how can be debug the root cause of such issue? Do you think using Java Flight Recorder will help?

Thanks
Rajesh Agarwal


-----Original Message-----
From: Tobias Hartmann [mailto:tobias.hartmann at oracle.com] 
Sent: Wednesday, November 23, 2016 12:08 AM
To: Agarwal, Rajesh <Rajesh.Agarwal at snapon.com>; kirk at kodewerk.com
Cc: hotspot-compiler-dev at openjdk.java.net
Subject: Re: JVM Crashes after few hours


On 22.11.2016 19:18, Agarwal, Rajesh wrote:
> We are able to reproduce the issue everytime during load testing process of our app.
> 
> Do you see any link with kryo which uses sun.misc.unsafe class for serializing the Java objects?

It's very hard to say. The problem with JDK-8050079 was that the customers framework changed the common class hierarchy by replacing java.lang.Object and that triggered a bug in the compiler. This looks very similar so I would guess that this is caused by some unusual class hierarchy.

If your time allows, you could try to build and run with a fastdebug VM build and check if we hit an assert. Also, a replay compilation file should be generated.

Just out of curiosity, is there a specific reason you are using -XX:-ReduceInitialCardMarks?

Best regards,
Tobias

> 
> Thanks
> Rajesh Agarwal
> 
> 
> 
> 
> On Tue, Nov 22, 2016 at 11:17 PM +0530, "Tobias Hartmann" <tobias.hartmann at oracle.com <mailto:tobias.hartmann at oracle.com>> wrote:
> 
> Sorry, I got confused by the version string. Seems like you are using JDK 8u112 which includes the fix for 8050079.
> 
> Do you happen to have a reproducer for this?
> 
> Best regards,
> Tobias
> 
> On 22.11.2016 18:25, Tobias Hartmann wrote:
>> Hi,
>> 
>> the stack trace looks like:
>> https://bugs.openjdk.java.net/browse/JDK-8050079
>> 
>> This bug was fixed in 8u40 and 9. Could you please check if you are able to reproduce the problem with 8u40 or later?
>> 
>> Thanks,
>> Tobias
>> 
>> On 22.11.2016 16:39, Agarwal, Rajesh wrote:
>>> Attached is the complete log. 
>>>
>>> Thanks
>>> Rajesh Agarwal
>>>
>>>
>>> -----Original Message-----
>>> From: kirk at kodewerk.com [mailto:kirk at kodewerk.com]
>>> Sent: Tuesday, November 22, 2016 8:55 PM
>>> To: Agarwal, Rajesh <Rajesh.Agarwal at snapon.com>
>>> Cc: hotspot-compiler-dev at openjdk.java.net
>>> Subject: Re: JVM Crashes after few hours
>>>
>>> Hi,
>>>
>>> Can we see the rest of the log? This view of the log has cut off important information.
>>>
>>> Kind regards,
>>> Kirk Pepperdine
>>>
>>>> On Nov 22, 2016, at 8:43 AM, Agarwal, Rajesh <Rajesh.Agarwal at snapon.com> wrote:
>>>>
>>>> The JVM either crashes or hangs after running for few hours with following error.
>>>>
>>>> #
>>>> # A fatal error has been detected by the Java Runtime Environment:
>>>> #
>>>> #  SIGSEGV (0xb) at pc=0x00007f93aa58ed32, pid=24048,
>>>> tid=0x00007f9368a68700 # # JRE version: Java(TM) SE Runtime 
>>>> Environment (8.0_112-b15) (build 1.8.0_112-b15) # Java VM: Java
>>>> HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode linux-amd64 
>>>> compressed oops) # Problematic frame:
>>>> # V  [libjvm.so+0x640d32]
>>>> InstanceKlass::find_method_index(Array<Method*>*, Symbol*, Symbol*, 
>>>> bool, bool)+0x42 # # Failed to write core dump. Core dumps have 
>>>> been disabled. To enable core dumping, try "ulimit -c unlimited" 
>>>> before starting Java again # # If you would like to submit a bug report, please visit:
>>>> #   http://bugreport.java.com/bugreport/crash.jsp
>>>> #
>>>>
>>>> ---------------  T H R E A D  ---------------
>>>>
>>>> Current thread (0x00007f93a4a50800):  JavaThread "C2 CompilerThread0" 
>>>> daemon [_thread_in_vm, id=24067,
>>>> stack(0x00007f9368968000,0x00007f9368a69000)]
>>>>
>>>> siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 
>>>> 0x00007f9326bd3218
>>>>
>>>> vm_info: Java HotSpot(TM) 64-Bit Server VM (25.112-b15) for
>>>> linux-amd64 JRE (1.8.0_112-b15), built on Sep 22 2016 21:10:53 by 
>>>> "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
>>>>
>>>> VM Arguments:
>>>> jvm_args: -Xms2560m -Xmx2560m -XX:-UseAdaptiveSizePolicy
>>>> -XX:NewRatio=2 -XX:SurvivorRatio=2 -XX:TargetSurvivorRatio=80 
>>>> -Xnoclassgc -verbose:gc -XX:+PrintGCDateStamps 
>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:-ReduceInitialCardMarks 
>>>> -XX:+UseG1GC -XX:MaxGCPauseMillis=400 -XX:CICompilerCount=2 
>>>> -XX:+UnlockDiagnosticVMOptions -XX:+LogCompilation Thanks Rajesh 
>>>> Agarwal
>>>>
>>>>
>>>