Shenandoah JDK 8 backport crashes when running Cassandra

Wed Feb 5 14:23:09 UTC 2020

Hi all,
After deploying Cassandra 3.11.4 with Shenandoah's backport to JDK 8, the JVM occasionally crashes.
I can share the full crash log privately, here's the start:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f2fd6a1a559, pid=102562, tid=0x00007f28885f7700
#
# JRE version: OpenJDK Runtime Environment (8.0) (build 1.8.0-builds.shipilev.net-openjdk-shenandoah-jdk8-b517-20191231)
# Java VM: OpenJDK 64-Bit Server VM (25.71-b517-20191231 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J 10739 C1 org.caffinitas.ohc.linked.OffHeapMap$Table.removeLinkInternal(JJJJ)V (83 bytes) @ 0x00007f2fd6a1a559 [0x00007f2fd6a19e00+0x759]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
---------------  T H R E A D  ---------------
Current thread (0x00007f27d8007620):  JavaThread "ReadStage-2" daemon [_thread_in_Java, id=103716, stack(0x00007f28885b7000,0x00007f28885f8000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x000000000000000f
Registers:
RAX=0x0000000000000000, RBX=0x000000000000000d, RCX=0x0000000000000008, RDX=0x0000000000000010
RSP=0x00007f28885f6090, RBP=0x00007f28885f63e8, RSI=0xffffffffffffffff, RDI=0x0000000000000010
R8 =0x000000000000000f, R9 =0x0000000000000000, R10=0x00007f2fed1b5360, R11=0x00007f2f4c6f7d18
R12=0x0000000000000000, R13=0x0000000000560040, R14=0x00007f28885f6408, R15=0x00007f27d8007620
RIP=0x00007f2fd6a1a559, EFLAGS=0x0000000000010207, CSGSFS=0x0000000000000033, ERR=0x0000000000000006
  TRAPNO=0x000000000000000e
Top of Stack: (sp=0x00007f28885f6090)
0x00007f28885f6090:   0000000000000005 00007f2fd7a1e42c
0x00007f28885f60a0:   0000000700000001 00000007bb964570
0x00007f28885f60b0:   00000007bb964700 0000000600000005
0x00007f28885f60c0:   00000007beb6bb20 00007f2f00000001
0x00007f28885f60d0:   00000003f5d8e200 00000007bb964570
0x00007f28885f60e0:   0000000000000001 00007f2fd74214b0
0x00007f28885f60f0:   0000000000000004 00007f2fd6312118
0x00007f28885f6100:   f80914d8e6dd26ab e312c4a300000000
0x00007f28885f6110:   00000003f5d8e200 00007f2fd9335344
0x00007f28885f6120:   0000000000000014 00007f2fd6a1a764
0x00007f28885f6130:   00000003f808eaf3 00007f2f00000000
0x00007f28885f6140:   00000007baaa4830 00007f292b45b2c0
0x00007f28885f6150:   0000000000329b88 00007f2fe6432320
0x00007f28885f6160:   ffffffffffffffff 0000000000000000
0x00007f28885f6170:   00000006e3f4c168 5205e883464e5371
0x00007f28885f6180:   00000003f5d8d868 0000000000000005
0x00007f28885f6190:   ffffffffffffffff 0000000000000010
0x00007f28885f61a0:   00000003f5d8ce58 00000007bea177b0
0x00007f28885f61b0:   00000003f5d8e0d8 00007f2fd60fca74
0x00007f28885f61c0:   00007f28885f63e8 00007f2fd8b5225c
0x00007f28885f61d0:   00000003f5d8d7e0 00000003f5d8e1f0
0x00007f28885f61e0:   0000000400000004 00007f292b45b2c0
0x00007f28885f61f0:   0000000000165ae8 00007f25398fff50
0x00007f28885f6200:   0000000000000000 0000000000000000
0x00007f28885f6210:   00000006e3f4c168 518c1e395d6acb5d
0x00007f28885f6220:   0000000700000005 0000000000000014
0x00007f28885f6230:   00000003f5d8d5d8 00000003f5d8d840
0x00007f28885f6240:   00007f292b45b2c0 0000000000165ae8
0x00007f28885f6250:   00007f28885f62b8 00007f2fd5007a40
0x00007f28885f6260:   00007f2fd5007a40 0000000000000000
0x00007f28885f6270:   0000000000000010 00007f28885f6278
0x00007f28885f6280:   00007f2f4d0430ed 00007f28885f62d0
Instructions: (pc=0x00007f2fd6a1a559)
0x00007f2fd6a1a539:   01 00 00 48 8b bc 24 08 01 00 00 4c 03 c7 48 b8
0x00007f2fd6a1a549:   00 00 00 00 00 00 00 00 4c 8b 8c 24 d8 00 00 00
0x00007f2fd6a1a559:   4e 89 0c 00 48 81 c4 30 01 00 00 5d 85 05 95 7b
0x00007f2fd6a1a569:   5c 17 c3 48 8b 94 24 e8 00 00 00 48 8b b4 24 e0
Register to memory mapping:
RAX=0x0000000000000000 is an unknown value
RBX=0x000000000000000d is an unknown value
RCX=0x0000000000000008 is an unknown value
RDX=0x0000000000000010 is an unknown value
RSP=0x00007f28885f6090 is pointing into the stack for thread: 0x00007f27d8007620
RBP=0x00007f28885f63e8 is pointing into the stack for thread: 0x00007f27d8007620
RSI=0xffffffffffffffff is an unknown value
RDI=0x0000000000000010 is an unknown value
R8 =0x000000000000000f is an unknown value
R9 =0x0000000000000000 is an unknown value
R10=0x00007f2fed1b5360: <offset 0x1008360> in /usr/lib/jvm/jdk1.8.0_builds.shipilev.net-openjdk-shenandoah-jdk8/jre/lib/amd64/server/libjvm.so at 0x00007f2fec1ad000
R11=0x00007f2f4c6f7d18 is pointing into metadata
R12=0x0000000000000000 is an unknown value
R13=0x0000000000560040 is an unknown value
R14=0x00007f28885f6408 is pointing into the stack for thread: 0x00007f27d8007620
R15=0x00007f27d8007620 is a thread
Stack: [0x00007f28885b7000,0x00007f28885f8000],  sp=0x00007f28885f6090,  free space=252k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 10739 C1 org.caffinitas.ohc.linked.OffHeapMap$Table.removeLinkInternal(JJJJ)V (83 bytes) @ 0x00007f2fd6a1a559 [0x00007f2fd6a19e00+0x759]
J 19594 C1 org.caffinitas.ohc.linked.OffHeapMap.removeInternal(JJZ)V (129 bytes) @ 0x00007f2fd8b5225c [0x00007f2fd8b51e40+0x41c]
j  org.caffinitas.ohc.linked.OffHeapMap.removeEldest()J+19
J 27364 C1 org.caffinitas.ohc.linked.OffHeapMap.putEntry(JJJJZJJJJ)Z (604 bytes) @ 0x00007f2fd95d177c [0x00007f2fd95d0140+0x163c]
J 15835 C2 org.caffinitas.ohc.linked.OHCacheImpl.putInternal(Ljava/lang/Object;Ljava/lang/Object;ZLjava/lang/Object;J)Z (317 bytes) @ 0x00007f2fd8722fc8 [0x00007f2fd8722a00+0x5c8]
J 20313 C2 org.apache.cassandra.db.SinglePartitionReadCommand.getThroughCache(Lorg/apache/cassandra/db/ColumnFamilyStore;Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator; (604 bytes) @ 0x00007f2fd6a74388 [0x00007f2fd6a70d00+0x3688]
J 11197 C2 org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator; (258 bytes) @ 0x00007f2fd7b8a538 [0x00007f2fd7b8a0a0+0x498]
J 15994 C2 org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V (356 bytes) @ 0x00007f2fd6116774 [0x00007f2fd6116620+0x154]
J 15992 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V (51 bytes) @ 0x00007f2fd79ab02c [0x00007f2fd79aaf80+0xac]
J 24963% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (306 bytes) @ 0x00007f2fd9aefa10 [0x00007f2fd9aeeba0+0xe70]
J 6632 C1 java.lang.Thread.run()V (17 bytes) @ 0x00007f2fd5329564 [0x00007f2fd5329400+0x164]
v  ~StubRoutines::call_stub
V  [libjvm.so+0x67f2c0]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0xfb0
V  [libjvm.so+0x67c7c7]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x277
V  [libjvm.so+0x67cda7]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x57
V  [libjvm.so+0x6bac95]  thread_entry(JavaThread*, Thread*)+0x75
V  [libjvm.so+0xaa5c7f]  JavaThread::thread_main_inner()+0xdf
V  [libjvm.so+0xaa5fd9]  JavaThread::run()+0x329
V  [libjvm.so+0x8cdcb2]  java_start(Thread*)+0xf2
C  [libpthread.so.0+0x8184]  start_thread+0xc4