JDK-9058853 OpenJDK 11 crash with Shenandoah GC on CentOS 7

Thu Jan 10 19:33:50 UTC 2019

Thanks Roman, that is indeed the related OpenJDK public bug. Sorry about my
comments around the 9058853 bug being non-public. I can test out a patch
against shenandoah/jdk11 though we've moved the production workloads back
to G1 for the time being.

On Thu, Jan 10, 2019 at 2:15 PM Roman Kennke <rkennke at redhat.com> wrote:

> Hello David,
>
> I believe this is the same bug as:
> https://bugs.openjdk.java.net/browse/JDK-8216364
>
> we're currently investigating and working on a fix.
>
> We can use the reproducer provided in the bug report above. If you can
> build shenandoah/jdk11 with a patch, you might want to test a fix, as
> soon as we have it?
>
> Best regards,
> Roman
>
>
> > Hi Shenandoah-dev,
> >
> > I wanted to flag a JIT crash that we encountered running OpenJDK 11 crash
> > with Shenandoah GC on CentOS 7. There was a bug report filed as
> JDK-9058853
> > [1], though that ticket appears to no longer publicly visible (seems like
> > there's maybe a separate discussion as that seems counter to the OpenJDK
> > JBS [2] though I assume it relates to Oracle not wanting to handle triage
> > support tickets for Shenandoah per JDK-8215030 [3])
> >
> > In the interest of hardening Shenandoah GC, I wanted to see if folks here
> > are interested in fixing this as we were able to create a minimal repro
> > that quickly triggers the JIT crash consistently [4]. Note that both the
> > original bug sighting and repro are using LMAX disruptor 3.4.2 which
> relies
> > heavily on sun.misc.unsafe so had filed an issue there [5]. Dropping in
> the
> > VarHandles based disruptor version [6] does not trigger the crash.
> >
> > I've included some of the detail inline below, but full
> hs_err_pid94107.log
> > [7] and replay_pid94107.log [8] are linked with the repro.
> >
> > Thanks,
> > Dave
> >
> > [1] https://bugs.openjdk.java.net/browse/JDK-9058853
> > [2] https://wiki.openjdk.java.net/display/general/JBS+Overview
> > [3] https://bugs.openjdk.java.net/browse/JDK-8215030
> > [4] https://github.com/schlosna/shenandoah
> > [5] https://github.com/LMAX-Exchange/disruptor/issues/251
> > [6] https://github.com/LMAX-Exchange/disruptor/tree/jdk9-varhandles
> > [7]
> >
> https://github.com/schlosna/shenandoah/blob/develop/doc/hs_err_pid94107.log
> > [8]
> >
> https://github.com/schlosna/shenandoah/blob/develop/doc/replay_pid94107.log
> >
> > ```
> > Current thread (0x00007f2bb41fb800):  JavaThread "C2 CompilerThread0"
> > daemon [_thread_in_native, id=94124,
> > stack(0x00007f2bb88f5000,0x00007f2bb89f6000)]
> >
> > Current CompileTask:
> > C2:    245  301 %     4
> >  com.lmax.disruptor.MultiProducerSequencer::initialiseAvailableBuffer @ 8
> > (31 bytes)
> > ```
> >
> > gdb backtrace from core dump:
> > ```
> > (gdb) bt
> > #0  0x00007fd2d0654207 in __GI_raise (sig=sig at entry=6) at
> > ../nptl/sysdeps/unix/sysv/linux/raise.c:55
> > #1  0x00007fd2d06558f8 in __GI_abort () at abort.c:90
> > #2  0x00007fd2cfc75c99 in os::abort (dump_core=<optimized out>,
> > siginfo=<optimized out>, context=<optimized out>) at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/os/linux/os_linux.cpp:1411
> > #3  0x00007fd2cff5f559 in VMError::report_and_die (id=<optimized out>,
> > message=message at entry=0x0, detail_fmt=detail_fmt at entry=0x7fd2d0013b60
> "%s",
> > detail_args=detail_args at entry=0x7fd2b0481020, thread=thread at entry
> > =0x7fd2c81fb800,
> >     pc=pc at entry=
> >     0x7fd2cfcd23cd <NodeHash::hash_delete(Node const*)+13>
> >
> "H\213\006H\211\363\377PX\211\302\061\300\205\322tJA\213|$\bM\213D$\030\215w\377!\362\211\321A\211\321I\215\f\310A\203\311\001H\213\071H\205\377t&H9\373u\016\353'\017\037\200",
> >     siginfo=siginfo at entry=0x7fd2b0481430, context=context at entry
> =0x7fd2b0481300,
> > filename=filename at entry=0x0, lineno=lineno at entry=0, size=size at entry=0)
> >     at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/share/utilities/vmError.cpp:1541
> > #4  0x00007fd2cff5ff53 in VMError::report_and_die
> > (thread=thread at entry=0x7fd2c81fb800,
> > sig=sig at entry=11,
> >     pc=pc at entry=0x7fd2cfcd23cd <NodeHash::hash_delete(Node const*)+13>
> >
> "H\213\006H\211\363\377PX\211\302\061\300\205\322tJA\213|$\bM\213D$\030\215w\377!\362\211\321A\211\321I\215\f\310A\203\311\001H\213\071H\205\377t&H9\373u\016\353'\017\037\200",
> > siginfo=siginfo at entry=0x7fd2b0481430, context=context at entry
> =0x7fd2b0481300,
> > detail_fmt=detail_fmt at entry=0x7fd2d0013b60 "%s")
> >     at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/share/utilities/vmError.cpp:1241
> > #5  0x00007fd2cff5ffa1 in VMError::report_and_die
> > (thread=thread at entry=0x7fd2c81fb800,
> > sig=sig at entry=11,
> >     pc=pc at entry=0x7fd2cfcd23cd <NodeHash::hash_delete(Node const*)+13>
> >
> "H\213\006H\211\363\377PX\211\302\061\300\205\322tJA\213|$\bM\213D$\030\215w\377!\362\211\321A\211\321I\215\f\310A\203\311\001H\213\071H\205\377t&H9\373u\016\353'\017\037\200",
> > siginfo=siginfo at entry=0x7fd2b0481430, context=context at entry
> =0x7fd2b0481300)
> > at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/share/utilities/vmError.cpp:1247
> > #6  0x00007fd2cfc8070a in JVM_handle_linux_signal (sig=11,
> > info=0x7fd2b0481430, ucVoid=0x7fd2b0481300,
> > abort_if_unrecognized=<optimized out>)
> >     at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/os_cpu/linux_x86/os_linux_x86.cpp:620
> > #7  0x00007fd2cfc743a8 in signalHandler (sig=11, info=0x7fd2b0481430,
> > uc=0x7fd2b0481300) at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/os/linux/os_linux.cpp:4497
> > #8  <signal handler called>
> > #9  NodeHash::hash_delete (this=this at entry=0x7fd2b04835b8, n=n at entry
> =0x0)
> > at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/share/opto/phaseX.cpp:235
> > #10 0x00007fd2cfdd2520 in hash_delete (n=n at entry=0x0,
> this=0x7fd2b0482c30)
> > at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/share/opto/phaseX.hpp:375
> > #11 rehash_node_delayed (n=n at entry=0x0, this=0x7fd2b0482c30) at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/share/opto/phaseX.hpp:520
> > #12 PhaseIterGVN::replace_input_of (this=0x7fd2b0482c30, n=n at entry=0x0,
> > in=0x7fd2800a51e8, i=0) at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/share/opto/phaseX.hpp:530
> > #13 0x00007fd2cfde5c47 in ShenandoahWriteBarrierNode::pin_and_expand
> > (phase=phase at entry=0x7fd2b0481fe0) at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/share/gc/shenandoah/c2/shenandoahSupport.cpp:3053
> > #14 0x00007fd2cfb1ade0 in PhaseIdealLoop::build_and_optimize
> > (this=this at entry=0x7fd2b0481fe0, mode=mode at entry
> =LoopOptsShenandoahExpand)
> >     at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/share/opto/loopnode.cpp:2898
> > #15 0x00007fd2cfdd49f4 in PhaseIdealLoop (mode=LoopOptsShenandoahExpand,
> > igvn=..., this=0x7fd2b0481fe0) at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/share/opto/loopnode.hpp:945
> > #16 ShenandoahWriteBarrierNode::expand (C=0x7fd2b0484dc0, igvn=...,
> > loop_opts_cnt=@0x7fd2b0482ac0: 37) at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/share/gc/shenandoah/c2/shenandoahSupport.cpp:545
> > #17 0x00007fd2cf6dc616 in Compile::Optimize (this=0x7fd2b0484dc0) at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/share/opto/compile.cpp:2416
> > #18 0x00007fd2cf6dd74c in Compile::Compile (this=0x7fd2b0484dc0,
> > ci_env=<optimized out>, compiler=0x7fd2c81fafa0, target=<optimized out>,
> > osr_bci=<optimized out>, subsume_loads=<optimized out>,
> > do_escape_analysis=true,
> >     eliminate_boxing=true, directive=0x7fd2c81e16d0) at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/share/opto/compile.cpp:882
> > #19 0x00007fd2cf5fc496 in C2Compiler::compile_method
> (this=0x7fd2c81fafa0,
> > env=0x7fd2b0485a20, target=0x7fd2781809e0, entry_bci=8,
> > directive=0x7fd2c81e16d0)
> >     at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/share/opto/c2compiler.cpp:109
> > #20 0x00007fd2cf6e62cb in CompileBroker::invoke_compiler_on_method
> > (task=task at entry=0x7fd2c82037c0) at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/share/compiler/compileBroker.cpp:2112
> > #21 0x00007fd2cf6e7d78 in CompileBroker::compiler_thread_loop () at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/share/compiler/compileBroker.cpp:1808
> > #22 0x00007fd2cfefa472 in JavaThread::thread_main_inner
> > (this=this at entry=0x7fd2c81fb800)
> > at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/share/runtime/thread.cpp:1752
> > #23 0x00007fd2cfefa7d8 in JavaThread::run (this=0x7fd2c81fb800) at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/share/runtime/thread.cpp:1732
> > #24 0x00007fd2cfc7e482 in thread_native_entry (thread=0x7fd2c81fb800) at
> >
> /usr/src/debug/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/openjdk/src/hotspot/os/linux/os_linux.cpp:698
> > #25 0x00007fd2d0e07dd5 in start_thread (arg=0x7fd2b0486700) at
> > pthread_create.c:307
> > #26 0x00007fd2d071bead in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
> > ```
> >
>
>