jdk deadlock bug in JDK 1.6.0_05?

[Keith McNeill]

There is native code. But we don't acquire any locks in the native code.

Keith

Tom Rodriguez wrote:
> Note that every lock held by a thread isn't shown in a thread dump. It 
> doesn't show locks which are acquired by native code using 
> MonitorEnter. Is there any native code that might be acquiring locks?
>
> tom
>
> On Aug 27, 2008, at 1:34 PM, Keith McNeill wrote:
>
>> A customer of ours is reporting a hang problem. They took a thread 
>> dump during the hang and the thread dump reported a deadlock. The 
>> relavent portion of the thread dump is (there are approx 500 threads 
>> running in the application):
>>
>> ......
>> Found one Java-level deadlock:
>> =============================
>> "Thread-135314":
>> waiting to lock monitor 0x08ab51e0 (object 0x78c478e0, a 
>> java.util.ArrayList),
>> which is held by "Thread-997"
>> "Thread-997":
>> waiting to lock monitor 0x08abc554 (object 0x78089d38, a 
>> com.streambase.sb.runtime.ParallelModule$ParallelLockee),
>> which is held by "Thread-11620"
>> "Thread-11620":
>> waiting to lock monitor 0x08ab51e0 (object 0x78c478e0, a 
>> java.util.ArrayList),
>> which is held by "Thread-997"
>>
>> Java stack information for the threads listed above:
>> ===================================================
>> "Thread-135314":
>> at 
>> com.streambase.sb.runtime.sbd.OutputQueue.subscribe(OutputQueue.java:361) 
>>
>> - waiting to lock <0x78c478e0> (a java.util.ArrayList)
>> at 
>> com.streambase.sb.sbd.InnerDequeueHandle.subscribe(InnerDequeueHandle.java:310) 
>>
>> at 
>> com.streambase.sb.sbd.XmlRpcHandlerManager$SUBSCRIBEHandler.process(XmlRpcHandlerManager.java:181) 
>>
>> at 
>> com.streambase.sb.xmlrpc.XmlRpcManager.process(XmlRpcManager.java:208)
>> at 
>> com.streambase.sb.sbd.XmlRpcHandlerManager.processXmlRpcRequest(XmlRpcHandlerManager.java:119) 
>>
>> at 
>> com.streambase.sb.sbd.StreamBaseServer.processXmlRpcRequest(StreamBaseServer.java:174) 
>>
>> at 
>> com.streambase.sb.sbd.StreamBaseServer.processXmlRpcRequestJni(StreamBaseServer.java:164) 
>>
>> "Thread-997":
>> at com.streambase.sb.sbd.StreamBaseNode.enqueue(StreamBaseNode.java:797)
>> - waiting to lock <0x78089d38> (a 
>> com.streambase.sb.runtime.ParallelModule$ParallelLockee)
>> "Thread-11620":
>> at cm._default_.sbd_1219465761228_1.s___Monitor_Input_O_S_(Unknown 
>> Source)
>> - waiting to lock <0x78c478e0> (a java.util.ArrayList)
>> at 
>> cm._default_.sbd_1219465761228_1.op___Monitoring_Routing__0(Unknown 
>> Source)
>> at 
>> cm._default_.sbd_1219465761228_1.op___Monitor_Input_Monitor_Filter__0(Unknown 
>> Source)
>> at cm._default_.sbd_1219465761228_1.s___Monitor_Input_(Unknown Source)
>> at 
>> cm._default_.sbd_1219465761228_1._enqueue_Tuples_1_Monitor_Input_(Unknown 
>> Source)
>> at cm._default_.sbd_1219465761228_1.enqueueTuples(Unknown Source)
>> at 
>> com.streambase.sb.runtime.MainModule.enqueueTuples(MainModule.java:232)
>> at com.streambase.sb.sbd.StreamBaseNode.enqueue(StreamBaseNode.java:799)
>> - locked <0x78089d38> (a 
>> com.streambase.sb.runtime.ParallelModule$ParallelLockee)
>>
>> Found 1 deadlock.
>> --------------------
>>
>> The key is that the thread dump thinks that Thread-997 holds monitor 
>> 0x08ab51e0 (object 0x78c478e0).
>>
>> If I believe the thread dump output then that thread doesn't hold 
>> that monitor/lock:
>> --------------
>> "Thread-997":
>> at com.streambase.sb.sbd.StreamBaseNode.enqueue(StreamBaseNode.java:797)
>> - waiting to lock <0x78089d38> (a 
>> com.streambase.sb.runtime.ParallelModule$ParallelLockee)
>> -------------------
>> There is no locked <0x78c478e0> in that thread. In fact there is no 
>> locked <0x78c478e0> in the entire stack dump. So no one is holding 
>> that lock
>>
>> I could believe that in the past Thread-997 held that lock. But it 
>> doesn't now....at least according to the thread dump.
>>
>> Looking through the code the thread stack traces seem quite 
>> reasonable. All of the locked and waiting on locks seem correct 
>> except for the threads waiting on Thread-997 for a lock.
>>
>> Note that "Thread-997" is coming in via JNI
>>
>> I can see a couple of possibilities:
>>
>> 1) Thread dump is lying and the dead lock is happening somewhere else
>> 2) JVM has a bug in that Thread-997 hasn't released the lock for some 
>> reason.
>>
>> Any suggestions on how to debug the problem with the customer? It 
>> doesn't seem like that the thread dump output is giving me good 
>> information.
>>
>> Thanks,
>>
>> Keith
>>
>