Fatal Error on PPC64 and PPC64LE for both JDK7 and JDK8
Tiago Sturmer Daitx
tdaitx at br.ibm.com
Thu Apr 16 23:00:11 UTC 2015
I'm trying to debug a problem when building and testing the Apache
Oozie, but as I'm yet familiar with the Hotspot internals I can't
pinpoint exactly what is going on. I would appreciate any help and
guidance I can get to understand and fix this.
I have been able to reproduce the issue in both RHEL 7.1 (PPC64 and
PPC64LE) and Ubuntu 14.04 (PPC64LE) with the distros OpenJDK 1.7 and 1.8
default install. I have also built jdk7u from ppc-aix-port repository
and noticed that it fails a bit differently.
To reproduce:
$ yum install git openssh-clients cmake jpackage-utils.noarch \
zlib zlib-devel openssl openssl-devel gcc gcc-c++ maven
$ export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk
$ export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:/opt/apache-maven-3.2.1/bin/:$PATH
$ export MAVEN_OPTS="-Xms512m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=512m"
$ git clone -b branch-4.1 https://github.com/apache/oozie.git
$ cd oozie
$ ./bin/mkdistro.sh -DskipTests -Phadoop-2
$ ./bin/mkdistro.sh -Phadoop-2 -X -fn -l oozie.stdout
Here is a summary of what I have obtained so far, please let me know if
you need the core files or hs_error reports.
OpenJDK 7 from distros
----------------------
The stdout shows that various components are failing:
$ grep -e "Core dump" -e "hs_error" oozie.stdout
# Core dump written. Default location: /home/tdaitx/oozie/core/core or core.9933
# /tmp/jvm-9933/hs_error.log
# Core dump written. Default location: /home/tdaitx/oozie/sharelib/pig/core or core.10610
# /tmp/jvm-10610/hs_error.log
# Core dump written. Default location: /home/tdaitx/oozie/sharelib/hive/core or core.10647
# /tmp/jvm-10647/hs_error.log
# Core dump written. Default location: /home/tdaitx/oozie/sharelib/sqoop/core or core.11294
# /tmp/jvm-11294/hs_error.log
# Core dump written. Default location: /home/tdaitx/oozie/sharelib/streaming/core or core.11331
# /tmp/jvm-11331/hs_error.log
# Core dump written. Default location: /home/tdaitx/oozie/sharelib/distcp/core or core.11368
# /tmp/jvm-11368/hs_error.log
# Core dump written. Default location: /home/tdaitx/oozie/examples/core or core.11432
# /tmp/jvm-11432/hs_error.log
# Core dump written. Default location: /home/tdaitx/oozie/tools/core or core.12239
# /tmp/jvm-12239/hs_error.log
# Core dump written. Default location: /home/tdaitx/oozie/minitest/core or core.12335
# /tmp/jvm-12335/hs_error.log
# Core dump written. Default location: /home/tdaitx/oozie/zookeeper-security-tests/core or core.12437
# /tmp/jvm-12437/hs_error.log
The hs_error sample shows:
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00003fffa51515bc, pid=10610, tid=70366847889872
#
# JRE version: OpenJDK Runtime Environment (7.0_79-b14) (build 1.7.0_79-mockbuild_2015_04_10_10_48-b00)
# Java VM: OpenJDK 64-Bit Server VM (24.79-b02 mixed mode linux-ppc64 compressed oops)
# Derivative: IcedTea 2.5.5
# Distribution: Built on Red Hat Enterprise Linux Server release 7.1 (Maipo) (Fri Apr 10 10:48:01 EDT 2015)
# Problematic frame:
# V [libjvm.so+0x8115bc]
Debugging it with GDB (and OpenJDK debug symbols installed), backtrace
shows:
$ gdb /usr/lib/jvm/java-1.7.0-openjdk/bin/java /home/tdaitx/oozie/sharelib/pig/core.10610
(gdb) bt
<snip>
Core was generated by `java -Xmx1024m -da -XX:MaxPermSize=512m -jar /home/tdaitx/oozie/sharelib/pig/ta'.
Program terminated with signal 6, Aborted.
#0 0x00003fffa544e4d8 in raise () from /lib64/power8/libc.so.6
#0 0x00003fffa544e4d8 in raise () from /lib64/power8/libc.so.6
#1 0x00003fffa5450658 in abort () from /lib64/power8/libc.so.6
#2 0x00003fffa50cedcc in os::abort (dump_core=<optimized out>) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/
hotspot/src/os/linux/vm/os_linux.cpp:1630
#3 0x00003fffa52a977c in VMError::report_and_die (this=0x3fff8ef8b5d8) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/
openjdk/hotspot/src/share/vm/utilities/vmError.cpp:1073
#4 0x00003fffa52a9f1c in crash_handler (sig=<optimized out>, info=0x3fff8ef8c4d8, ucVoid=0x3fff8ef8b760) at /usr/src/debug/java-1.7.0-openjdk
-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/os/linux/vm/vmError_linux.cpp:106
#5 <signal handler called>
#6 ParallelScavengeHeap::block_start (this=<optimized out>, addr=0xc0d8ff58) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.pp
c64le/openjdk/hotspot/src/share/vm/oops/oop.inline.hpp:431
#7 0x00003fffa50c8920 in os::print_location (st=0x3fff8ef8d058, x=3235446616, verbose=<optimized out>) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/share/vm/runtime/os.cpp:905
#8 0x00003fffa50dce10 in os::print_register_info (st=0x3fff8ef8d058, context=0x3fff8ef8d330) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp:605
#9 0x00003fffa52a81c8 in VMError::report (this=0x3fff8ef8d148, st=0x3fff8ef8d058) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/share/vm/utilities/vmError.cpp:558
#10 0x00003fffa52a91ec in VMError::report_and_die (this=0x3fff8ef8d148) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/share/vm/utilities/vmError.cpp:1007
#11 0x00003fffa50dc264 in JVM_handle_linux_signal (sig=<optimized out>, info=0x3fff8ef8e0a8, ucVoid=0x3fff8ef8d330, abort_if_unrecognized=<optimized out>) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp:437
#12 0x00003fffa50cb7dc in signalHandler (sig=<optimized out>, info=<optimized out>, uc=<optimized out>) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:4361
#13 <signal handler called>
#14 update_contents (cm=0x3fffa016b0c0, this=<optimized out>) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/share/vm/oops/oop.pcgc.inline.hpp:53
#15 MoveAndUpdateClosure::do_addr (this=0x3fff8ef8e440, addr=<optimized out>, words=1) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/share/vm/gc_implementation/parallelScavenge/psParallelCompact.cpp:3616
#16 0x00003fffa50f1fc8 in ParMarkBitMap::iterate (this=0x3fffa53b0760 <PSParallelCompact::_mark_bitmap>, live_closure=0x3fff8ef8e440, range_beg=<optimized out>, range_end=71172096) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/share/vm/gc_implementation/parallelScavenge/parMarkBitMap.cpp:181
#17 0x00003fffa5154388 in iterate (this=0x3fffa53b0760 <PSParallelCompact::_mark_bitmap>, range_end=0xc1f00000, range_beg=<optimized out>, live_closure=0x3fff8ef8e440) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/share/vm/gc_implementation/parallelScavenge/parMarkBitMap.hpp:361
#18 PSParallelCompact::fill_region (cm=0x3fffa016b0c0, region_idx=<optimized out>) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/share/vm/gc_implementation/parallelScavenge/psParallelCompact.cpp:3399
#19 0x00003fffa51429c4 in fill_and_update_region (region=<optimized out>, cm=0x3fffa016b0c0) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/share/vm/gc_implementation/parallelScavenge/psParallelCompact.hpp:1304
#20 ParCompactionManager::drain_region_stacks (this=0x3fffa016b0c0) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/share/vm/gc_implementation/parallelScavenge/psCompactionManager.cpp:219
#21 0x00003fffa51223b4 in DrainStacksCompactionTask::do_it (this=<optimized out>, manager=<optimized out>, which=<optimized out>) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/share/vm/gc_implementation/parallelScavenge/pcTasks.cpp:341
#22 0x00003fffa4d7b25c in GCTaskThread::run (this=0x3fffa002a000) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/share/vm/gc_implementation/parallelScavenge/gcTaskThread.cpp:144
#23 0x00003fffa50cd710 in java_start (thread=0x3fffa002a000) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:910
#24 0x00003fffa5658944 in start_thread () from /lib64/power8/libpthread.so.0
#25 0x00003fffa5527640 in clone () from /lib64/power8/libc.so.6
(gdb) frame 6
#6 ParallelScavengeHeap::block_start (this=<optimized out>, addr=0xc0d8ff58)
at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/share/vm/oops/oop.inline.hpp:431
431 inline int oopDesc::size_given_klass(Klass* klass) {
(gdb) frame 14
#14 update_contents (cm=0x3fffa016b0c0, this=<optimized out>)
at /usr/src/debug/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/openjdk/hotspot/src/share/vm/oops/oop.pcgc.inline.hpp:53
53 if (!new_klass->oop_is_typeArray()) {
All other core files show a very similar backtrace (signal at
update_contents and sometimes another at block_start). Example:
$ sdiff -BWs -I '\[New LWP.*' core.12239.bt core.10610.bt
Core was generated by `java -Xmx1024m -da -XX:MaxPermSize=512 | Core was generated by `java -Xmx1024m -da -XX:MaxPermSize=512
#0 0x00003fff9e4ae4d8 in raise () from /lib64/power8/libc.so | #0 0x00003fffa544e4d8 in raise () from /lib64/power8/libc.so
#0 0x00003fff9e4ae4d8 in raise () from /lib64/power8/libc.so | #0 0x00003fffa544e4d8 in raise () from /lib64/power8/libc.so
#1 0x00003fff9e4b0658 in abort () from /lib64/power8/libc.so | #1 0x00003fffa5450658 in abort () from /lib64/power8/libc.so
#2 0x00003fff9e12edcc in os::abort (dump_core=<optimized out | #2 0x00003fffa50cedcc in os::abort (dump_core=<optimized out
#3 0x00003fff9e30977c in VMError::report_and_die (this=0x3ff | #3 0x00003fffa52a977c in VMError::report_and_die (this=0x3ff
#4 0x00003fff9e309f1c in crash_handler (sig=<optimized out>, | #4 0x00003fffa52a9f1c in crash_handler (sig=<optimized out>,
#6 JavaThread::get_thread_name_string (this=0x3fff14a13800, | #6 ParallelScavengeHeap::block_start (this=<optimized out>,
#7 0x00003fff9e2a4f50 in JavaThread::print_on_error (this=0x | #7 0x00003fffa50c8920 in os::print_location (st=0x3fff8ef8d0
#8 0x00003fff9e2abd64 in Threads::print_on_error (st=0x3fff8 | #8 0x00003fffa50dce10 in os::print_register_info (st=0x3fff8
#9 0x00003fff9e307968 in VMError::report (this=0x3fff87ffd17 | #9 0x00003fffa52a81c8 in VMError::report (this=0x3fff8ef8d14
#10 0x00003fff9e3091ec in VMError::report_and_die (this=0x3ff | #10 0x00003fffa52a91ec in VMError::report_and_die (this=0x3ff
#11 0x00003fff9e309f1c in crash_handler (sig=<optimized out>, | #11 0x00003fffa50dc264 in JVM_handle_linux_signal (sig=<optim
#12 <signal handler called> | #12 0x00003fffa50cb7dc in signalHandler (sig=<optimized out>,
#13 ParallelScavengeHeap::block_start (this=<optimized out>, | #13 <signal handler called>
#14 0x00003fff9e128920 in os::print_location (st=0x3fff87ffd0 | #14 update_contents (cm=0x3fffa016b0c0, this=<optimized out>)
#15 0x00003fff9e13ce10 in os::print_register_info (st=0x3fff8 | #15 MoveAndUpdateClosure::do_addr (this=0x3fff8ef8e440, addr=
#16 0x00003fff9e3081c8 in VMError::report (this=0x3fff87ffd17 | #16 0x00003fffa50f1fc8 in ParMarkBitMap::iterate (this=0x3fff
#17 0x00003fff9e3091ec in VMError::report_and_die (this=0x3ff | #17 0x00003fffa5154388 in iterate (this=0x3fffa53b0760 <PSPar
#18 0x00003fff9e13c264 in JVM_handle_linux_signal (sig=<optim | #18 PSParallelCompact::fill_region (cm=0x3fffa016b0c0, region
#19 0x00003fff9e12b7dc in signalHandler (sig=<optimized out>, | #19 0x00003fffa51429c4 in fill_and_update_region (region=<opt
#20 <signal handler called> | #20 ParCompactionManager::drain_region_stacks (this=0x3fffa01
#21 update_contents (cm=0x3fff9816b0a0, this=<optimized out>) | #21 0x00003fffa51223b4 in DrainStacksCompactionTask::do_it (t
#22 MoveAndUpdateClosure::do_addr (this=0x3fff87ffe470, addr= | #22 0x00003fffa4d7b25c in GCTaskThread::run (this=0x3fffa002a
#23 0x00003fff9e151fc8 in ParMarkBitMap::iterate (this=0x3fff | #23 0x00003fffa50cd710 in java_start (thread=0x3fffa002a000)
#24 0x00003fff9e1b4388 in iterate (this=0x3fff9e410760 <PSPar | #24 0x00003fffa5658944 in start_thread () from /lib64/power8/
#25 PSParallelCompact::fill_region (cm=0x3fff9816b0a0, region | #25 0x00003fffa5527640 in clone () from /lib64/power8/libc.so
#26 0x00003fff9e183320 in StealRegionCompactionTask::do_it (t <
#27 0x00003fff9dddb25c in GCTaskThread::run (this=0x3fff9802a <
#28 0x00003fff9e12d710 in java_start (thread=0x3fff9802a000) <
#29 0x00003fff9e6b8944 in start_thread () from /lib64/power8/ <
#30 0x00003fff9e587640 in clone () from /lib64/power8/libc.so <
JDK7u from PPC-AIX-Port repository
----------------------------------------------
$ grep -e "Core dump" -e "hs_error" oozie-ppc-aix-port.stdout
# Core dump written. Default location: /home/tdaitx/oozie/client/target/site/apidocs/core or core.26317
# Core dump written. Default location: /home/tdaitx/oozie/core/core or core.29406
# Core dump written. Default location: /home/tdaitx/oozie/sharelib/pig/core or core.30071
# Core dump written. Default location: /home/tdaitx/oozie/sharelib/hive/core or core.30108
No debugging symbols in my custom build, but the backtraces are
equivalent for the core files above.
(gdb) bt
#0 0x00003fffae1be4d8 in raise () from /lib64/power8/libc.so.6
#1 0x00003fffae1c0658 in abort () from /lib64/power8/libc.so.6
#2 0x00003fffadbc1ba8 in os::abort(bool) ()
from /home/tdaitx/ppc-aix-port-jdk7u/output-jdk7u-fastdebug/j2sdk-image/jre/lib/ppc64/server/libjvm.so
#3 0x00003fffade956bc in VMError::report_and_die() ()
from /home/tdaitx/ppc-aix-port-jdk7u/output-jdk7u-fastdebug/j2sdk-image/jre/lib/ppc64/server/libjvm.so
#4 0x00003fffad588ef4 in report_vm_error(char const*, int, char const*, char const*) ()
from /home/tdaitx/ppc-aix-port-jdk7u/output-jdk7u-fastdebug/j2sdk-image/jre/lib/ppc64/server/libjvm.so
#5 0x00003fffadbf7bbc in ParMarkBitMap::verify_clear() const ()
from /home/tdaitx/ppc-aix-port-jdk7u/output-jdk7u-fastdebug/j2sdk-image/jre/lib/ppc64/server/libjvm.so
#6 0x00003fffadc8c930 in PSParallelCompact::pre_compact(PreGCValues*) ()
from /home/tdaitx/ppc-aix-port-jdk7u/output-jdk7u-fastdebug/j2sdk-image/jre/lib/ppc64/server/libjvm.so
#7 0x00003fffadc9b668 in PSParallelCompact::invoke_no_policy(bool) ()
from /home/tdaitx/ppc-aix-port-jdk7u/output-jdk7u-fastdebug/j2sdk-image/jre/lib/ppc64/server/libjvm.so
#8 0x00003fffadcac444 in PSScavenge::invoke() ()
from /home/tdaitx/ppc-aix-port-jdk7u/output-jdk7u-fastdebug/j2sdk-image/jre/lib/ppc64/server/libjvm.so
#9 0x00003fffadc08d58 in ParallelScavengeHeap::failed_mem_allocate(unsigned long) ()
from /home/tdaitx/ppc-aix-port-jdk7u/output-jdk7u-fastdebug/j2sdk-image/jre/lib/ppc64/server/libjvm.so
#10 0x00003fffade981b4 in VM_ParallelGCFailedAllocation::doit() ()
from /home/tdaitx/ppc-aix-port-jdk7u/output-jdk7u-fastdebug/j2sdk-image/jre/lib/ppc64/server/libjvm.so
#11 0x00003fffadebfb78 in VM_Operation::evaluate() ()
from /home/tdaitx/ppc-aix-port-jdk7u/output-jdk7u-fastdebug/j2sdk-image/jre/lib/ppc64/server/libjvm.so
#12 0x00003fffadebca10 in VMThread::evaluate_operation(VM_Operation*) ()
from /home/tdaitx/ppc-aix-port-jdk7u/output-jdk7u-fastdebug/j2sdk-image/jre/lib/ppc64/server/libjvm.so
#13 0x00003fffadebd6fc in VMThread::loop() ()
from /home/tdaitx/ppc-aix-port-jdk7u/output-jdk7u-fastdebug/j2sdk-image/jre/lib/ppc64/server/libjvm.so
#14 0x00003fffadebd9d4 in VMThread::run() ()
from /home/tdaitx/ppc-aix-port-jdk7u/output-jdk7u-fastdebug/j2sdk-image/jre/lib/ppc64/server/libjvm.so
#15 0x00003fffadbc01b0 in java_start(Thread*) ()
from /home/tdaitx/ppc-aix-port-jdk7u/output-jdk7u-fastdebug/j2sdk-image/jre/lib/ppc64/server/libjvm.so
#16 0x00003fffae3d8944 in start_thread () from /lib64/power8/libpthread.so.0
#17 0x00003fffae297640 in clone () from /lib64/power8/libc.so.6
Regards,
Tiago
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/ppc-aix-port-dev/attachments/20150416/50782237/attachment-0001.html>
More information about the ppc-aix-port-dev
mailing list