JVM hangs beyond recovery

[David Holmes]

I can't shed any further light on this as there are too many 
unidentified frames in the stack traces. From a quick read the MySQL 
connector is supposed to be pure Java, so that would rule it out.

One thing odd I did notice in the stack dump is thread 83:

#7  0x00002b4b2f9d4e2e in JVM_MonitorWait ()
    from /usr/java/jdk1.6.0_20/jre/lib/amd64/server/libjvm.so
#8  0x00002aaaaaf73986 in ?? ()
#9  0x0000000000000003 in ?? ()
#10 0x00002b4b2fb6a023 in os::javaTimeMillis ()
    from /usr/java/jdk1.6.0_20/jre/lib/amd64/server/libjvm.so
#11 0x00002aaaaaf67a42 in ?? ()
#12 0x0000000000000000 in ?? ()

There's no way to get from os::javaTimeMillis into JVM_MonitorWait.

David Holmes

Stas Oskin said the following on 06/09/10 18:36:
> Hi.
> 
> Thanks for the explanation.
> 
>     The underlying problem looks familiar to me though I can't find the
>     culprit thread in the gdb thread dump. This problem occurs when a
>     library uses a native dynamic loader hook (I'll call it dlsym) to
>     make a call into the JVM. This is done while holding an internal
>     dlsym mutex lock and the end result is a deadlock if any of the
>     other threads in the Java application need to perform a dlsym
>     related operation (which you can see in the thread dump that they do
>     and are all blocked on that internal mutex).
> 
> 
> I noticed this happening mostly in DB related operations. I'm using 
> mysql-connector-java-5.1.6 which according to MySQL documentation, is "a 
> native Java driver", and indeed doesn't seem to contain any native code 
> library, but I wonder if this related to what you experienced in past.
> 
>     The culprit is whomever calls into the VM from inside the dlsym
>     code. As I said I can't see that culprit in the stack traces, but
>     typically it happens when someone installs an "on-load" hook or
>     something of that nature.
> 
> 
> Is there any good way to detect the exact library causing this deadlock?
> This happens quite rarely and spontaneously, so just disabling native 
> libraries one by one and trying to find the cause didn't work for me.
> 
> Thanks again for all the help.