JVM hangs beyond recovery

John Cuthbertson john.cuthbertson at oracle.com
Fri Apr 16 16:55:00 PDT 2010 

Hi Stas,

The thread that requested the safepoint operation is:

> Thread 11 (Thread 0x42c61940 (LWP 20879)):
> #0  0x0000003c7160ad09 in pthread_cond_wait@@GLIBC_2.3.2 ()
>    from /lib64/libpthread.so.0
> #1  0x00002ba3704859f7 in os::PlatformEvent::park ()
>    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #2  0x00002ba37045ea92 in Monitor::IWait ()
>    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #3  0x00002ba37045f0fa in Monitor::wait ()
>    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #4  0x00002ba3705b9f79 in VMThread::execute ()
>    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #5  0x00002ba3700ecc7d in BiasedLocking::revoke_and_rebias ()
>    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #6  0x00002ba3705376d1 in ObjectSynchronizer::fast_enter ()
>    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #7  0x00002ba37029061a in InterpreterRuntime::monitorenter ()
>    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #8  0x00002aaaaaf8003b in ?? ()
> #9  0x00002aaaaaf7ffe9 in ?? ()
> #10 0x00002aaaaf3208d8 in ?? ()
> #11 0x00002aaaaf3208d8 in ?? ()
> #12 0x00000036a0681609 in ?? ()
> #13 0x00002aaaaf320768 in ?? ()
> #14 0x0000000042c603d8 in ?? ()
> #15 0x00002aaabea33dd7 in ?? ()
> #16 0x0000000042c60458 in ?? ()
> #17 0x00002aaabeba65a8 in ?? ()
> #18 0x0000000000000000 in ?? ()
which is trying to clean up some meta-data associated with biased 
locking while attempting to get a monitor. So you could, as a 
workaround, try turning off UseBiasedLocking.

The thread I think the safepoint is waiting for is:
> Thread 7 (Thread 0x43065940 (LWP 20886)):
> #0  0x0000003c7160d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1  0x0000003c71608c55 in _L_lock_1127 () from /lib64/libpthread.so.0
> #2  0x0000003c71608b53 in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3  0x0000003c712010ab in dlsym () from /lib64/libdl.so.2
> #4  0x00002aaaaad0b923 in sysFindLibraryEntry ()
>    from /usr/java/jdk1.6.0_16/jre/lib/amd64/native_threads/libhpi.so
> #5  0x00002ba37046203a in NativeLookup::lookup_style ()
>    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #6  0x00002ba3704620f7 in NativeLookup::lookup_entry ()
>    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #7  0x00002ba370462493 in NativeLookup::lookup_base ()
>    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #8  0x00002ba3704625ae in NativeLookup::lookup ()
>    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #9  0x00002ba370294290 in InterpreterRuntime::prepare_native_call ()
>    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #10 0x00002aaaaaf73d6c in ?? ()
> #11 0x00002aaacc448500 in ?? ()
> #12 0x00002aaaaaf73d2d in ?? ()
> #13 0x0000000043064570 in ?? ()
> #14 0x00002aaaae6b8608 in ?? ()
> #15 0x0000000043ad4800 in ?? ()
> #16 0x0000000043064498 in ?? ()
> #17 0x00002aaabeb785b0 in ?? ()
> #18 0x0000000043064508 in ?? ()
> #19 0x00002aaabeb7bf80 in ?? ()
> #20 0x0000000000000000 in ?? ()
It's thread state will be _thread_in_vm (it's trying to lookup the name 
of a jni routine in a shared library).  Note: _thread_in_vm is different 
from _thread_in_native. With the latter, a thread is considered at a 
safepoint and the vm operation can proceed. Since this thread is not at 
a safepoint, it will be allowed to roll forward until it reaches a state 
that is safepoint-safe (_thread-in_native, _thread_blocked) - which 
should be just before InterpreterRuntime::prepare_native_call returns 
back to its caller.

 From the stack trace it looks like thread #7 is waiting on a lock 
that's internal to libdl - presumably the one that makes the dl* 
routines MT-safe - and that's what seems to be stopping the thread from 
making forward progress.

JohnC

Y. Srinivas Ramakrishna wrote:
> On 04/16/10 15:51, Stas Oskin wrote:
>   
>> Hi.
>>
>> [Make sure to provide them with the JDK version string.]
>>
>> Right, totally forgot.
>> The Java version is:
>> java version "1.6.0_16"
>> Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
>> Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode)
>>     
>
> I suggest using 6u18 (with -XX:-ReduceInitialCardMarks as in release
> notes; this is independent of the hang you are seeing) to see
> if the hang reproduces; just on the off-chance this may have been
> fixed in the interim (i must state that such a hang does not sound
> familiar to me though). Or if you have a reproducible test
> case that would be best. If you have a support contract (and even
> if you don't), you should probably file a bug (through support, if
> you have a support contract), including the test case.
>
>   
>> Running on Linux x86_64.
>>
>>
>> The most import point in my opinion, is that JVM both compiles the Java 
>> code in two concurrent threads, and runs JNI code at a same time. So I 
>> would if this might be what causing the hangs?
>>     
>
> I don't think so. The compiler threads have stopped for the safepoint
> (see below), and (JNI) native code (outside of the JVM) is allowed to run.
>
> Thread 29 (Thread 0x40fe6940 (LWP 20859)):
> #0  0x0000003c7160ad09 in pthread_cond_wait@@GLIBC_2.3.2 ()
>     from /lib64/libpthread.so.0
> #1  0x00002ba3704859f7 in os::PlatformEvent::park ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #2  0x00002ba37045e6f9 in Monitor::ILock ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #3  0x00002ba37045ee30 in Monitor::lock_without_safepoint_check ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #4  0x00002ba3704efbfa in SafepointSynchronize::block ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #5  0x00002ba37056c89b in JavaThread::check_safepoint_and_suspend_for_native_trans () from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #6  0x00002ba37012f642 in ThreadStateTransition::transition_from_native ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #7  0x00002ba37015833b in BufferBlob::create ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #8  0x00002ba37016bbd9 in Compile::init_scratch_buffer_blob ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #9  0x00002ba37048ab2c in Compile::Output ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #10 0x00002ba370170d4d in Compile::Code_Gen ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #11 0x00002ba37016d364 in Compile::Compile ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #12 0x00002ba370101dee in C2Compiler::compile_method ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #13 0x00002ba370176028 in CompileBroker::invoke_compiler_on_method ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #14 0x00002ba3701759c6 in CompileBroker::compiler_thread_loop ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #15 0x00002ba3705711e9 in compiler_thread_entry ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #16 0x00002ba37056acb1 in JavaThread::run ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #17 0x00002ba37048626f in java_start ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #18 0x0000003c71606617 in start_thread () from /lib64/libpthread.so.0
> #19 0x0000003c70ad3c2d in clone () from /lib64/libc.so.6
>
> Thread 28 (Thread 0x416ab940 (LWP 20860)):
> #0  0x0000003c7160ad09 in pthread_cond_wait@@GLIBC_2.3.2 ()
>     from /lib64/libpthread.so.0
> #1  0x00002ba3704859f7 in os::PlatformEvent::park ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #2  0x00002ba37045e6f9 in Monitor::ILock ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #3  0x00002ba37045ee30 in Monitor::lock_without_safepoint_check ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #4  0x00002ba3704efbfa in SafepointSynchronize::block ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #5  0x00002ba37056c89b in JavaThread::check_safepoint_and_suspend_for_native_trans () from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #6  0x00002ba37012b066 in ciObject::klass ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #7  0x00002ba370441fb8 in LoadNode::Value ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #8  0x00002ba3704b814b in PhaseIterGVN::transform_old ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #9  0x00002ba3704b7cad in PhaseIterGVN::optimize ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #10 0x00002ba37042bbc9 in PhaseMacroExpand::expand_macro_nodes ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #11 0x00002ba37017062e in Compile::Optimize ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #12 0x00002ba37016d313 in Compile::Compile ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #13 0x00002ba370101dee in C2Compiler::compile_method ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #14 0x00002ba370176028 in CompileBroker::invoke_compiler_on_method ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #15 0x00002ba3701759c6 in CompileBroker::compiler_thread_loop ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #16 0x00002ba3705711e9 in compiler_thread_entry ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #17 0x00002ba37056acb1 in JavaThread::run ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #18 0x00002ba37048626f in java_start ()
>     from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
> #19 0x0000003c71606617 in start_thread () from /lib64/libpthread.so.0
> #20 0x0000003c70ad3c2d in clone () from /lib64/libc.so.6
>
>
> I am not familiar with this part of the runtime or its debugging,
> so I will copy the alias and let someone more familiar with such
> issues to help you. (Recall also the suggestions for the
> [Print]Safepoint[Statistics|[Timeout*]] flags in previous email
> to help with debugging.)
>
> hope that helps; 'bye!
> -- ramki
>
>   
>> Thanks again.
>>     
>
>

More information about the hotspot-runtime-dev mailing list