JVM hangs beyond recovery

Fri Apr 16 16:46:00 PDT 2010

Thanks for the initial analysis Ramki,

Y. Srinivas Ramakrishna said the following on 04/17/10 07:19:
> The JVM appears to be hung trying to reach a safepoint:-

And it seem likely this thread is the one that's causing the problem:

Thread 7 (Thread 0x43065940 (LWP 20886)):
#0  0x0000003c7160d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x0000003c71608c55 in _L_lock_1127 () from /lib64/libpthread.so.0
#2  0x0000003c71608b53 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000003c712010ab in dlsym () from /lib64/libdl.so.2
#4  0x00002aaaaad0b923 in sysFindLibraryEntry ()
    from /usr/java/jdk1.6.0_16/jre/lib/amd64/native_threads/libhpi.so
#5  0x00002ba37046203a in NativeLookup::lookup_style ()
    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
#6  0x00002ba3704620f7 in NativeLookup::lookup_entry ()
    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
#7  0x00002ba370462493 in NativeLookup::lookup_base ()
    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
#8  0x00002ba3704625ae in NativeLookup::lookup ()
    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
#9  0x00002ba370294290 in InterpreterRuntime::prepare_native_call ()
    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so

As this is a call to the runtime the thread state is _thread_in_vm and 
so the VMThread will be waiting for this thread to inform it that it has 
reached the safepoint. But for some reason the dlsym lock is not 
available and so this thread blocks before it can reach the safepoint 
and so the VM comes to a grinding halt.

So who owns the dlsym lock? Any chance you're using some library loading 
hook to call into the VM while in the midst of a dlsym related routine?

David Holmes

> On 04/16/10 13:31, Stas Oskin wrote:
>> Hi.
>>
>> I hit a possible deadlock in JVM, which causes my JNI application to 
>> hang.
>>
>> It happens usually on start-up, and the JVM just stuck, jstack and 
>> other tools are not working and the only option is to get a threads 
>> dump via GDB.
>>
>> Attached below is a sample of such dump.
>> As you can see, there are two concurrent compiling threads, while the 
>> JNI code executes.
>>
>> Any chance that this causes the hang?
>> Can anyone tell if this is a known issue, and there is any workaround?
>>
>> Thanks in advance!
>>
>> http://pastie.org/923725
>>
>>
>> 0x0000003c71607955 in pthread_join () from /lib64/libpthread.so.0
>> 40 Thread 0x41234940 (LWP 20847) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 39 Thread 0x40640940 (LWP 20849) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 38 Thread 0x41345940 (LWP 20850) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 37 Thread 0x41446940 (LWP 20851) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 36 Thread 0x41ab2940 (LWP 20852) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 35 Thread 0x42055940 (LWP 20853) 0x0000003c7160af70 in 
>> pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 34 Thread 0x41970940 (LWP 20854) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 33 Thread 0x41bb3940 (LWP 20855) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 32 Thread 0x41cb4940 (LWP 20856) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 31 Thread 0x403e2940 (LWP 20857) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 30 Thread 0x40ee5940 (LWP 20858) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 29 Thread 0x40fe6940 (LWP 20859) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 28 Thread 0x416ab940 (LWP 20860) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 27 Thread 0x417ac940 (LWP 20861) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 26 Thread 0x41db5940 (LWP 20862) 0x0000003c7160af70 in 
>> pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 25 Thread 0x410e9940 (LWP 20863) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 24 Thread 0x41eb6940 (LWP 20865) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 23 Thread 0x42156940 (LWP 20866) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 22 Thread 0x42257940 (LWP 20867) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 21 Thread 0x42358940 (LWP 20868) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 20 Thread 0x42459940 (LWP 20869) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 19 Thread 0x41547940 (LWP 20870) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 18 Thread 0x4255a940 (LWP 20871) 0x0000003c7160d95b in accept ()
>> from /lib64/libpthread.so.0
>> 17 Thread 0x4265b940 (LWP 20872) 0x0000003c7160d95b in accept ()
>> from /lib64/libpthread.so.0
>> 16 Thread 0x40cc0940 (LWP 20873) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 15 Thread 0x4275c940 (LWP 20874) 0x0000003c70ad4018 in epoll_wait ()
>> from /lib64/libc.so.6
>> 14 Thread 0x4285d940 (LWP 20875) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 13 Thread 0x4295e940 (LWP 20876) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 12 Thread 0x42b60940 (LWP 20878) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 11 Thread 0x42c61940 (LWP 20879) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 10 Thread 0x42d62940 (LWP 20880) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 9 Thread 0x42e63940 (LWP 20881) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 8 Thread 0x42f64940 (LWP 20882) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 7 Thread 0x43065940 (LWP 20886) 0x0000003c7160d2e4 in __lll_lock_wait ()
>> from /lib64/libpthread.so.0
>> 6 Thread 0x43166940 (LWP 20887) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 5 Thread 0x43267940 (LWP 20888) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 4 Thread 0x43368940 (LWP 20889) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 3 Thread 0x43469940 (LWP 20890) 0x0000003c7160ad09 in 
>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> 2 Thread 0x44a19940 (LWP 20891) 0x0000003c7160af70 in 
>> pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> * 1 Thread 0x2ba36feee5c0 (LWP 20843) 0x0000003c71607955 in 
>> pthread_join ()
>> from /lib64/libpthread.so.0
> 