jdk deadlock bug in JDK 1.6.0_05?

[Keith McNeill]

Nope, not solaris, it is on linux.  But will try jstack -ml next time it 
happens.

Thanks,

Keith

David Holmes - Sun Microsystems wrote:
> Hello Keith,
>
> There was bug whereby a thread that was blocked trying to acquire the 
> monitor used for class initialization inside a synchronized 
> block/method, would report that it was blocked acquiring the monitor 
> associated with that synchronized block/method. But that was fixed in 
> 6u2 and this is 6u5. Also note that monitors used for class 
> initialization do not show "- locked XXXX" entries in the dump.
>
> I don't suppose this is on Solaris is it? If so a pstack dump (or 
> jstack -ml) wold be more informative than the Java-level stack dump.
>
> David Holmes
> -----------
>
> Keith McNeill said the following on 08/28/08 06:34:
>> A customer of ours is reporting a hang problem.  They took a thread 
>> dump during the hang and the thread dump reported a deadlock.  The 
>> relavent portion of the thread dump is  (there are approx 500 threads 
>> running in the application):
>>
>> ......
>> Found one Java-level deadlock:
>> =============================
>> "Thread-135314":
>>       waiting to lock monitor 0x08ab51e0 (object 0x78c478e0, a 
>> java.util.ArrayList),
>>       which is held by "Thread-997"
>> "Thread-997":
>>       waiting to lock monitor 0x08abc554 (object 0x78089d38, a 
>> com.streambase.sb.runtime.ParallelModule$ParallelLockee),
>>       which is held by "Thread-11620"
>> "Thread-11620":
>>       waiting to lock monitor 0x08ab51e0 (object 0x78c478e0, a 
>> java.util.ArrayList),
>>       which is held by "Thread-997"
>>
>> Java stack information for the threads listed above:
>> ===================================================
>> "Thread-135314":
>>
>>     at
>>     
>> com.streambase.sb.runtime.sbd.OutputQueue.subscribe(OutputQueue.java:361) 
>>
>>
>>     - waiting to lock <0x78c478e0> (a java.util.ArrayList)
>>     at
>>     
>> com.streambase.sb.sbd.InnerDequeueHandle.subscribe(InnerDequeueHandle.java:310) 
>>
>>
>>     at
>>     
>> com.streambase.sb.sbd.XmlRpcHandlerManager$SUBSCRIBEHandler.process(XmlRpcHandlerManager.java:181) 
>>
>>
>>     at
>>     
>> com.streambase.sb.xmlrpc.XmlRpcManager.process(XmlRpcManager.java:208)
>>     at
>>     
>> com.streambase.sb.sbd.XmlRpcHandlerManager.processXmlRpcRequest(XmlRpcHandlerManager.java:119) 
>>
>>
>>     at
>>     
>> com.streambase.sb.sbd.StreamBaseServer.processXmlRpcRequest(StreamBaseServer.java:174) 
>>
>>
>>     at
>>     
>> com.streambase.sb.sbd.StreamBaseServer.processXmlRpcRequestJni(StreamBaseServer.java:164) 
>>
>>
>>
>> "Thread-997":
>>
>>     at
>>     
>> com.streambase.sb.sbd.StreamBaseNode.enqueue(StreamBaseNode.java:797)
>>     - waiting to lock <0x78089d38> (a
>>     com.streambase.sb.runtime.ParallelModule$ParallelLockee)
>>
>> "Thread-11620":
>>
>>     at cm._default_.sbd_1219465761228_1.s___Monitor_Input_O_S_(Unknown
>>     Source)
>>     - waiting to lock <0x78c478e0> (a java.util.ArrayList)
>>     at
>>     cm._default_.sbd_1219465761228_1.op___Monitoring_Routing__0(Unknown
>>     Source)
>>     at
>>     
>> cm._default_.sbd_1219465761228_1.op___Monitor_Input_Monitor_Filter__0(Unknown 
>>
>>     Source)
>>     at cm._default_.sbd_1219465761228_1.s___Monitor_Input_(Unknown 
>> Source)
>>     at
>>     
>> cm._default_.sbd_1219465761228_1._enqueue_Tuples_1_Monitor_Input_(Unknown 
>>
>>     Source)
>>     at cm._default_.sbd_1219465761228_1.enqueueTuples(Unknown Source)
>>     at
>>     
>> com.streambase.sb.runtime.MainModule.enqueueTuples(MainModule.java:232)
>>     at
>>     
>> com.streambase.sb.sbd.StreamBaseNode.enqueue(StreamBaseNode.java:799)
>>     - locked <0x78089d38> (a
>>     com.streambase.sb.runtime.ParallelModule$ParallelLockee)
>>
>>
>> Found 1 deadlock.
>> --------------------
>>
>> The key is that the thread dump thinks that Thread-997 holds monitor 
>> 0x08ab51e0 (object 0x78c478e0).
>>
>> If I believe the thread dump output then that thread doesn't hold 
>> that monitor/lock:
>> --------------
>> "Thread-997":
>>
>>     at
>>     
>> com.streambase.sb.sbd.StreamBaseNode.enqueue(StreamBaseNode.java:797)
>>     - waiting to lock <0x78089d38> (a
>>     com.streambase.sb.runtime.ParallelModule$ParallelLockee)
>>
>> -------------------
>> There is no locked <0x78c478e0> in that thread. In fact there is no 
>> locked <0x78c478e0> in the entire stack dump. So no one is holding 
>> that lock
>>
>> I could believe that in the past Thread-997 held that lock. But it 
>> doesn't now....at least according to the thread dump.
>>
>> Looking through the code the thread stack traces seem quite 
>> reasonable. All of the locked and waiting on locks seem correct 
>> except for the threads waiting on Thread-997 for a lock.
>>
>> Note that "Thread-997" is coming in via JNI
>>
>> I can see a couple of possibilities:
>>
>> 1) Thread dump is lying and the dead lock is happening somewhere else
>> 2) JVM has a bug in that Thread-997 hasn't released the lock for some 
>> reason.
>>
>> Any suggestions on how to debug the problem with the customer?  It 
>> doesn't seem like that the thread dump output is giving me good 
>> information.
>>
>> Thanks,
>>
>> Keith
>>