jdk deadlock bug in JDK 1.6.0_05?

[David Holmes - Sun Microsystems]

Hello Keith,

There was bug whereby a thread that was blocked trying to acquire the 
monitor used for class initialization inside a synchronized 
block/method, would report that it was blocked acquiring the monitor 
associated with that synchronized block/method. But that was fixed in 
6u2 and this is 6u5. Also note that monitors used for class 
initialization do not show "- locked XXXX" entries in the dump.

I don't suppose this is on Solaris is it? If so a pstack dump (or jstack 
-ml) wold be more informative than the Java-level stack dump.

David Holmes
-----------

Keith McNeill said the following on 08/28/08 06:34:
> A customer of ours is reporting a hang problem.  They took a thread dump 
> during the hang and the thread dump reported a deadlock.  The relavent 
> portion of the thread dump is  (there are approx 500 threads running in 
> the application):
> 
> ......
> Found one Java-level deadlock:
> =============================
> "Thread-135314":
>       waiting to lock monitor 0x08ab51e0 (object 0x78c478e0, a 
> java.util.ArrayList),
>       which is held by "Thread-997"
> "Thread-997":
>       waiting to lock monitor 0x08abc554 (object 0x78089d38, a 
> com.streambase.sb.runtime.ParallelModule$ParallelLockee),
>       which is held by "Thread-11620"
> "Thread-11620":
>       waiting to lock monitor 0x08ab51e0 (object 0x78c478e0, a 
> java.util.ArrayList),
>       which is held by "Thread-997"
> 
> Java stack information for the threads listed above:
> ===================================================
> "Thread-135314":
> 
>     at
>     com.streambase.sb.runtime.sbd.OutputQueue.subscribe(OutputQueue.java:361)
> 
>     - waiting to lock <0x78c478e0> (a java.util.ArrayList)
>     at
>     com.streambase.sb.sbd.InnerDequeueHandle.subscribe(InnerDequeueHandle.java:310)
> 
>     at
>     com.streambase.sb.sbd.XmlRpcHandlerManager$SUBSCRIBEHandler.process(XmlRpcHandlerManager.java:181)
> 
>     at
>     com.streambase.sb.xmlrpc.XmlRpcManager.process(XmlRpcManager.java:208)
>     at
>     com.streambase.sb.sbd.XmlRpcHandlerManager.processXmlRpcRequest(XmlRpcHandlerManager.java:119)
> 
>     at
>     com.streambase.sb.sbd.StreamBaseServer.processXmlRpcRequest(StreamBaseServer.java:174)
> 
>     at
>     com.streambase.sb.sbd.StreamBaseServer.processXmlRpcRequestJni(StreamBaseServer.java:164)
> 
> 
> "Thread-997":
> 
>     at
>     com.streambase.sb.sbd.StreamBaseNode.enqueue(StreamBaseNode.java:797)
>     - waiting to lock <0x78089d38> (a
>     com.streambase.sb.runtime.ParallelModule$ParallelLockee)
> 
> "Thread-11620":
> 
>     at cm._default_.sbd_1219465761228_1.s___Monitor_Input_O_S_(Unknown
>     Source)
>     - waiting to lock <0x78c478e0> (a java.util.ArrayList)
>     at
>     cm._default_.sbd_1219465761228_1.op___Monitoring_Routing__0(Unknown
>     Source)
>     at
>     cm._default_.sbd_1219465761228_1.op___Monitor_Input_Monitor_Filter__0(Unknown
>     Source)
>     at cm._default_.sbd_1219465761228_1.s___Monitor_Input_(Unknown Source)
>     at
>     cm._default_.sbd_1219465761228_1._enqueue_Tuples_1_Monitor_Input_(Unknown
>     Source)
>     at cm._default_.sbd_1219465761228_1.enqueueTuples(Unknown Source)
>     at
>     com.streambase.sb.runtime.MainModule.enqueueTuples(MainModule.java:232)
>     at
>     com.streambase.sb.sbd.StreamBaseNode.enqueue(StreamBaseNode.java:799)
>     - locked <0x78089d38> (a
>     com.streambase.sb.runtime.ParallelModule$ParallelLockee)
> 
> 
> Found 1 deadlock.
> --------------------
> 
> The key is that the thread dump thinks that Thread-997 holds monitor 
> 0x08ab51e0 (object 0x78c478e0).
> 
> If I believe the thread dump output then that thread doesn't hold that 
> monitor/lock:
> --------------
> "Thread-997":
> 
>     at
>     com.streambase.sb.sbd.StreamBaseNode.enqueue(StreamBaseNode.java:797)
>     - waiting to lock <0x78089d38> (a
>     com.streambase.sb.runtime.ParallelModule$ParallelLockee)
> 
> -------------------
> There is no locked <0x78c478e0> in that thread. In fact there is no 
> locked <0x78c478e0> in the entire stack dump. So no one is holding that 
> lock
> 
> I could believe that in the past Thread-997 held that lock. But it 
> doesn't now....at least according to the thread dump.
> 
> Looking through the code the thread stack traces seem quite reasonable. 
> All of the locked and waiting on locks seem correct except for the 
> threads waiting on Thread-997 for a lock.
> 
> Note that "Thread-997" is coming in via JNI
> 
> I can see a couple of possibilities:
> 
> 1) Thread dump is lying and the dead lock is happening somewhere else
> 2) JVM has a bug in that Thread-997 hasn't released the lock for some 
> reason.
> 
> Any suggestions on how to debug the problem with the customer?  It 
> doesn't seem like that the thread dump output is giving me good information.
> 
> Thanks,
> 
> Keith
>