jdk deadlock bug in JDK 1.6.0_05?

[Tom Rodriguez]

Note that every lock held by a thread isn't shown in a thread dump.   
It doesn't show locks which are acquired by native code using  
MonitorEnter.  Is there any native code that might be acquiring locks?

tom

On Aug 27, 2008, at 1:34 PM, Keith McNeill wrote:

> A customer of ours is reporting a hang problem.  They took a thread  
> dump during the hang and the thread dump reported a deadlock.  The  
> relavent portion of the thread dump is  (there are approx 500  
> threads running in the application):
>
> ......
> Found one Java-level deadlock:
> =============================
> "Thread-135314":
>       waiting to lock monitor 0x08ab51e0 (object 0x78c478e0, a  
> java.util.ArrayList),
>       which is held by "Thread-997"
> "Thread-997":
>       waiting to lock monitor 0x08abc554 (object 0x78089d38, a  
> com.streambase.sb.runtime.ParallelModule$ParallelLockee),
>       which is held by "Thread-11620"
> "Thread-11620":
>       waiting to lock monitor 0x08ab51e0 (object 0x78c478e0, a  
> java.util.ArrayList),
>       which is held by "Thread-997"
>
> Java stack information for the threads listed above:
> ===================================================
> "Thread-135314":
> at  
> com.streambase.sb.runtime.sbd.OutputQueue.subscribe(OutputQueue.java: 
> 361)
> - waiting to lock <0x78c478e0> (a java.util.ArrayList)
> at  
> com 
> .streambase 
> .sb.sbd.InnerDequeueHandle.subscribe(InnerDequeueHandle.java:310)
> at com.streambase.sb.sbd.XmlRpcHandlerManager 
> $SUBSCRIBEHandler.process(XmlRpcHandlerManager.java:181)
> at com.streambase.sb.xmlrpc.XmlRpcManager.process(XmlRpcManager.java: 
> 208)
> at  
> com 
> .streambase 
> .sb 
> .sbd 
> .XmlRpcHandlerManager.processXmlRpcRequest(XmlRpcHandlerManager.java: 
> 119)
> at  
> com 
> .streambase 
> .sb.sbd.StreamBaseServer.processXmlRpcRequest(StreamBaseServer.java: 
> 174)
> at  
> com 
> .streambase 
> .sb 
> .sbd.StreamBaseServer.processXmlRpcRequestJni(StreamBaseServer.java: 
> 164)
> "Thread-997":
> at com.streambase.sb.sbd.StreamBaseNode.enqueue(StreamBaseNode.java: 
> 797)
> - waiting to lock <0x78089d38> (a  
> com.streambase.sb.runtime.ParallelModule$ParallelLockee)
> "Thread-11620":
> at cm._default_.sbd_1219465761228_1.s___Monitor_Input_O_S_(Unknown  
> Source)
> - waiting to lock <0x78c478e0> (a java.util.ArrayList)
> at  
> cm._default_.sbd_1219465761228_1.op___Monitoring_Routing__0(Unknown  
> Source)
> at  
> cm 
> ._default_ 
> .sbd_1219465761228_1.op___Monitor_Input_Monitor_Filter__0(Unknown  
> Source)
> at cm._default_.sbd_1219465761228_1.s___Monitor_Input_(Unknown Source)
> at  
> cm 
> ._default_ 
> .sbd_1219465761228_1._enqueue_Tuples_1_Monitor_Input_(Unknown Source)
> at cm._default_.sbd_1219465761228_1.enqueueTuples(Unknown Source)
> at  
> com.streambase.sb.runtime.MainModule.enqueueTuples(MainModule.java: 
> 232)
> at com.streambase.sb.sbd.StreamBaseNode.enqueue(StreamBaseNode.java: 
> 799)
> - locked <0x78089d38> (a com.streambase.sb.runtime.ParallelModule 
> $ParallelLockee)
>
> Found 1 deadlock.
> --------------------
>
> The key is that the thread dump thinks that Thread-997 holds monitor  
> 0x08ab51e0 (object 0x78c478e0).
>
> If I believe the thread dump output then that thread doesn't hold  
> that monitor/lock:
> --------------
> "Thread-997":
> at com.streambase.sb.sbd.StreamBaseNode.enqueue(StreamBaseNode.java: 
> 797)
> - waiting to lock <0x78089d38> (a  
> com.streambase.sb.runtime.ParallelModule$ParallelLockee)
> -------------------
> There is no locked <0x78c478e0> in that thread. In fact there is no  
> locked <0x78c478e0> in the entire stack dump. So no one is holding  
> that lock
>
> I could believe that in the past Thread-997 held that lock. But it  
> doesn't now....at least according to the thread dump.
>
> Looking through the code the thread stack traces seem quite  
> reasonable. All of the locked and waiting on locks seem correct  
> except for the threads waiting on Thread-997 for a lock.
>
> Note that "Thread-997" is coming in via JNI
>
> I can see a couple of possibilities:
>
> 1) Thread dump is lying and the dead lock is happening somewhere else
> 2) JVM has a bug in that Thread-997 hasn't released the lock for  
> some reason.
>
> Any suggestions on how to debug the problem with the customer?  It  
> doesn't seem like that the thread dump output is giving me good  
> information.
>
> Thanks,
>
> Keith
>