Shenandoah crash

Thu Feb 1 14:33:36 UTC 2018

1 feb. 2018 kl. 14:53 skrev Aleksey Shipilev <shade at redhat.com>:
> 
> Thanks for the bug report!
> 
> On 02/01/2018 01:05 PM, Lennart Börjeson wrote:
>> #
>> # A fatal error has been detected by the Java Runtime Environment:
>> #
>> #  Internal Error (/Users/lennartb/RaT/openJDK/shenandoah-jdk10/src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp:184), pid=37194, tid=11267
>> #  Error: Shenandoah verification failed; After Mark, Roots: Must be marked in complete bitmap
>> 
>> Referenced from:
>>  interior location: 0x00007faf6cd0d3c0
>>  outside of Java heap
>>  0x00007faf6cd0d3c0 is a weak global jni handle
> 
>                            ^^^^^^^^^^^^^^^^^^^^^^
> 
> This one seems enough of the smoking gun to chase the bug for now. It must be the bug in handling
> the weak global JNI handles: we apparently miss that root during mark?
> 
> If we cannot find any obvious bugs there and/or fail to write a reproducer, we would need to work to
> understanding the failing application better. We are going to FOSDEM for the rest of the week, so --
> barring heroics of anyone here -- we would be able to look at it next week.

And then you'll be at JFokus, won't you? CU!

> 
>> Time: Thu Feb  1 12:36:33 2018 CET elapsed time: 1471 seconds (0d 0h 24m 31s)
> 
> You waited for 24 minutes for this bug to appear, right? I wonder if you can make it more frequent
> with -XX:ShenandoahGCHeuristics=aggressive.
> 

Somewhat faster crash (but another cause?) with aggressive (top of hs-log):

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/Users/lennartb/RaT/openJDK/shenandoah-jdk10/src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp:184), pid=4735, tid=11011
#  Error: Shenandoah verification failed; After Full GC, Marked: Object klass pointer must go to metaspace

Referenced from:
  interior location: 0x00000007be78012c
  0x00000007be780120 - klass 0x00000007c0251f58 java.time.OffsetDateTime
        allocated after complete mark start
        allocated after next mark start
        marked complete
        marked next
    not in collection set
  region: |0x00007fa60df79b00| 4089|P  |BTE 0x00000007be400000, 0x00000007be800000, 0x00000007be800000|U 100%|T   0%|G   0%|S 100%|L 100%|FTS           36086|LTS           59995| |CP   1|TAMS 0x00000007be400000, 0x00000007be780018|

Object:
  0x00000007bc5c0f00 - safe print, no details
  region: |0x00007fa60df78f80| 4081|EC |BTE 0x00000007bc400000, 0x00000007bc400000, 0x00000007bc800000|U   0%|T   0%|G   0%|S   0%|L   0%|FTS               0|LTS               0| |CP   0|TAMS 0x00000007bc400000, 0x00000007bc400000|

#
# JRE version: OpenJDK Runtime Environment (10.0) (fastdebug build 10-internal+0-adhoc.lennartb.shenandoah-jdk10)
# Java VM: OpenJDK 64-Bit Server VM (fastdebug 10-internal+0-adhoc.lennartb.shenandoah-jdk10, mixed mode, tiered, compressed oops, Shenandoah gc, bsd-amd64)
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  S U M M A R Y ------------

Command Line: --add-modules=java.xml.bind -Xmx16G -XX:+UseShenandoahGC -XX:+ShenandoahVerify -Xlog:gc=off,gc+ergo=off -DrunningInEclipse=true -XX:ShenandoahGCHeuristics=aggressive -Dfile.encoding=UTF-8 com.cinnober.bmf.gui.Main --console=false

Host: cft757eris.local, MacBookPro10,1 x86_64 2400 MHz, 8 cores, 16G, Darwin 17.4.0
Time: Thu Feb  1 15:27:41 2018 CET elapsed time: 605 seconds (0d 0h 10m 5s)

---------------  T H R E A D  ---------------

Current thread (0x00007fa60f805000):  GCTaskThread "Shenandoah GC Threads#2" [stack: 0x0000700002a7a000,0x0000700002b7a000] [id=11011]

Stack: [0x0000700002a7a000,0x0000700002b7a000],  sp=0x0000700002b755c0,  free space=1005k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.dylib+0xbff15e]  VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x4d0
V  [libjvm.dylib+0xbff858]  VMError::report_and_die(Thread*, char const*, int, char const*, char const*, __va_list_tag*)+0x3c
V  [libjvm.dylib+0x44f942]  report_vm_error(char const*, int, char const*, char const*, ...)+0xc8
V  [libjvm.dylib+0xb0b266]  ShenandoahVerifyOopClosure::print_failure(ShenandoahVerifyOopClosure::SafeLevel, oop, char const*)+0x1018
V  [libjvm.dylib+0xb0a22e]  ShenandoahVerifyOopClosure::verify(ShenandoahVerifyOopClosure::SafeLevel, oop, bool, char const*)+0x4c
V  [libjvm.dylib+0xb08f0d]  ShenandoahVerifyOopClosure::verify_oop(oop)+0x247
V  [libjvm.dylib+0xb0bbe3]  void ShenandoahVerifyOopClosure::verify_oop_at<unsigned int>(unsigned int*, oop)+0x3b
V  [libjvm.dylib+0xb0bac5]  void ShenandoahVerifyOopClosure::do_oop_work<unsigned int>(unsigned int*)+0x111
V  [libjvm.dylib+0x61b7c4]  InstanceKlass::oop_oop_iterate_v(oop, ExtendedOopClosure*)+0x106
V  [libjvm.dylib+0xb08566]  ShenandoahVerifyOopClosure::verify_oops_from(oop)+0x6a
V  [libjvm.dylib+0xb0c3ca]  ShenandoahVerifierMarkedRegionTask::verify_and_follow(HeapWord*, Stack<ShenandoahVerifierTask, (MemoryType)5>&, ShenandoahVerifyOopClosure&, unsigned long*)+0x154
V  [libjvm.dylib+0xb0c1f9]  ShenandoahVerifierMarkedRegionTask::work_regular(ShenandoahHeapRegion*, Stack<ShenandoahVerifierTask, (MemoryType)5>&, ShenandoahVerifyOopClosure&)+0xcf
V  [libjvm.dylib+0xb0bfb8]  ShenandoahVerifierMarkedRegionTask::work(unsigned int)+0x1d6
V  [libjvm.dylib+0xc58f46]  GangWorker::run_task(WorkData)+0x62
V  [libjvm.dylib+0xc59009]  GangWorker::loop()+0x25
V  [libjvm.dylib+0x958d5e]  thread_native_entry(Thread*)+0x135

>> # Java VM: OpenJDK 64-Bit Server VM (fastdebug 10-internal+0-adhoc.lennartb.shenandoah-jdk10)
> 
> Also, you don't need fastdebug build to run with -XX:+ShenandoahVerify, and the failure like this
> would be caught in release build as well.
> 
> Thanks,
> -Aleksey
> 

Best regards,

/Lennart