JVM hangs beyond recovery

Fri Apr 16 14:19:15 PDT 2010

The JVM appears to be hung trying to reach a safepoint:-

Thread 34 (Thread 0x41970940 (LWP 20854)):
#0  0x0000003c7160ad09 in pthread_cond_wait@@GLIBC_2.3.2 ()
    from /lib64/libpthread.so.0
#1  0x00002ba3704859f7 in os::PlatformEvent::park ()
    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
#2  0x00002ba37045ea92 in Monitor::IWait ()
    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
#3  0x00002ba37045f25e in Monitor::wait ()
    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
#4  0x00002ba3704ef56d in SafepointSynchronize::begin ()
    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
#5  0x00002ba3705b9968 in VMThread::loop ()
    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
#6  0x00002ba3705b946e in VMThread::run ()
    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
#7  0x00002ba37048626f in java_start ()
    from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so
#8  0x0000003c71606617 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003c70ad3c2d in clone () from /lib64/libc.so.6

There are a couple of safepoint debugging options
that you can try (check globals.hpp, especially [Print]Safepoint[Timeout*]).
I'll let the safepoint cognoscenti help with more detailed debugging advice etc.
[Make sure to provide them with the JDK version string.]

over and out.
-- ramki

On 04/16/10 13:31, Stas Oskin wrote:
> Hi.
> 
> I hit a possible deadlock in JVM, which causes my JNI application to hang.
> 
> It happens usually on start-up, and the JVM just stuck, jstack and other 
> tools are not working and the only option is to get a threads dump via GDB.
> 
> Attached below is a sample of such dump.
> As you can see, there are two concurrent compiling threads, while the 
> JNI code executes.
> 
> Any chance that this causes the hang?
> Can anyone tell if this is a known issue, and there is any workaround?
> 
> Thanks in advance!
> 
> http://pastie.org/923725
> 
> 
> 0x0000003c71607955 in pthread_join () from /lib64/libpthread.so.0
> 40 Thread 0x41234940 (LWP 20847) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 39 Thread 0x40640940 (LWP 20849) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 38 Thread 0x41345940 (LWP 20850) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 37 Thread 0x41446940 (LWP 20851) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 36 Thread 0x41ab2940 (LWP 20852) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 35 Thread 0x42055940 (LWP 20853) 0x0000003c7160af70 in 
> pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 34 Thread 0x41970940 (LWP 20854) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 33 Thread 0x41bb3940 (LWP 20855) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 32 Thread 0x41cb4940 (LWP 20856) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 31 Thread 0x403e2940 (LWP 20857) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 30 Thread 0x40ee5940 (LWP 20858) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 29 Thread 0x40fe6940 (LWP 20859) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 28 Thread 0x416ab940 (LWP 20860) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 27 Thread 0x417ac940 (LWP 20861) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 26 Thread 0x41db5940 (LWP 20862) 0x0000003c7160af70 in 
> pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 25 Thread 0x410e9940 (LWP 20863) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 24 Thread 0x41eb6940 (LWP 20865) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 23 Thread 0x42156940 (LWP 20866) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 22 Thread 0x42257940 (LWP 20867) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 21 Thread 0x42358940 (LWP 20868) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 20 Thread 0x42459940 (LWP 20869) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 19 Thread 0x41547940 (LWP 20870) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 18 Thread 0x4255a940 (LWP 20871) 0x0000003c7160d95b in accept ()
> from /lib64/libpthread.so.0
> 17 Thread 0x4265b940 (LWP 20872) 0x0000003c7160d95b in accept ()
> from /lib64/libpthread.so.0
> 16 Thread 0x40cc0940 (LWP 20873) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 15 Thread 0x4275c940 (LWP 20874) 0x0000003c70ad4018 in epoll_wait ()
> from /lib64/libc.so.6
> 14 Thread 0x4285d940 (LWP 20875) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 13 Thread 0x4295e940 (LWP 20876) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 12 Thread 0x42b60940 (LWP 20878) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 11 Thread 0x42c61940 (LWP 20879) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 10 Thread 0x42d62940 (LWP 20880) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 9 Thread 0x42e63940 (LWP 20881) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 8 Thread 0x42f64940 (LWP 20882) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 7 Thread 0x43065940 (LWP 20886) 0x0000003c7160d2e4 in __lll_lock_wait ()
> from /lib64/libpthread.so.0
> 6 Thread 0x43166940 (LWP 20887) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 5 Thread 0x43267940 (LWP 20888) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 4 Thread 0x43368940 (LWP 20889) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 3 Thread 0x43469940 (LWP 20890) 0x0000003c7160ad09 in 
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 2 Thread 0x44a19940 (LWP 20891) 0x0000003c7160af70 in 
> pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> * 1 Thread 0x2ba36feee5c0 (LWP 20843) 0x0000003c71607955 in pthread_join ()
> from /lib64/libpthread.so.0