jdk deadlock bug in JDK 1.6.0_05?

[Keith McNeill]

A customer of ours is reporting a hang problem.  They took a thread dump 
during the hang and the thread dump reported a deadlock.  The relavent 
portion of the thread dump is  (there are approx 500 threads running in 
the application):

......
Found one Java-level deadlock:
=============================
"Thread-135314":
      waiting to lock monitor 0x08ab51e0 (object 0x78c478e0, a 
java.util.ArrayList),
      which is held by "Thread-997"
"Thread-997":
      waiting to lock monitor 0x08abc554 (object 0x78089d38, a 
com.streambase.sb.runtime.ParallelModule$ParallelLockee),
      which is held by "Thread-11620"
"Thread-11620":
      waiting to lock monitor 0x08ab51e0 (object 0x78c478e0, a 
java.util.ArrayList),
      which is held by "Thread-997"

Java stack information for the threads listed above:
===================================================
"Thread-135314":

    at
    com.streambase.sb.runtime.sbd.OutputQueue.subscribe(OutputQueue.java:361)

    - waiting to lock <0x78c478e0> (a java.util.ArrayList)
    at
    com.streambase.sb.sbd.InnerDequeueHandle.subscribe(InnerDequeueHandle.java:310)

    at
    com.streambase.sb.sbd.XmlRpcHandlerManager$SUBSCRIBEHandler.process(XmlRpcHandlerManager.java:181)

    at
    com.streambase.sb.xmlrpc.XmlRpcManager.process(XmlRpcManager.java:208)
    at
    com.streambase.sb.sbd.XmlRpcHandlerManager.processXmlRpcRequest(XmlRpcHandlerManager.java:119)

    at
    com.streambase.sb.sbd.StreamBaseServer.processXmlRpcRequest(StreamBaseServer.java:174)

    at
    com.streambase.sb.sbd.StreamBaseServer.processXmlRpcRequestJni(StreamBaseServer.java:164)


"Thread-997":

    at
    com.streambase.sb.sbd.StreamBaseNode.enqueue(StreamBaseNode.java:797)
    - waiting to lock <0x78089d38> (a
    com.streambase.sb.runtime.ParallelModule$ParallelLockee)

"Thread-11620":

    at cm._default_.sbd_1219465761228_1.s___Monitor_Input_O_S_(Unknown
    Source)
    - waiting to lock <0x78c478e0> (a java.util.ArrayList)
    at
    cm._default_.sbd_1219465761228_1.op___Monitoring_Routing__0(Unknown
    Source)
    at
    cm._default_.sbd_1219465761228_1.op___Monitor_Input_Monitor_Filter__0(Unknown
    Source)
    at cm._default_.sbd_1219465761228_1.s___Monitor_Input_(Unknown Source)
    at
    cm._default_.sbd_1219465761228_1._enqueue_Tuples_1_Monitor_Input_(Unknown
    Source)
    at cm._default_.sbd_1219465761228_1.enqueueTuples(Unknown Source)
    at
    com.streambase.sb.runtime.MainModule.enqueueTuples(MainModule.java:232)
    at
    com.streambase.sb.sbd.StreamBaseNode.enqueue(StreamBaseNode.java:799)
    - locked <0x78089d38> (a
    com.streambase.sb.runtime.ParallelModule$ParallelLockee)


Found 1 deadlock.
--------------------

The key is that the thread dump thinks that Thread-997 holds monitor 
0x08ab51e0 (object 0x78c478e0).

If I believe the thread dump output then that thread doesn't hold that 
monitor/lock:
--------------
"Thread-997":

    at
    com.streambase.sb.sbd.StreamBaseNode.enqueue(StreamBaseNode.java:797)
    - waiting to lock <0x78089d38> (a
    com.streambase.sb.runtime.ParallelModule$ParallelLockee)

-------------------
There is no locked <0x78c478e0> in that thread. In fact there is no 
locked <0x78c478e0> in the entire stack dump. So no one is holding that 
lock

I could believe that in the past Thread-997 held that lock. But it 
doesn't now....at least according to the thread dump.

Looking through the code the thread stack traces seem quite reasonable. 
All of the locked and waiting on locks seem correct except for the 
threads waiting on Thread-997 for a lock.

Note that "Thread-997" is coming in via JNI

I can see a couple of possibilities:

1) Thread dump is lying and the dead lock is happening somewhere else
2) JVM has a bug in that Thread-997 hasn't released the lock for some 
reason.

Any suggestions on how to debug the problem with the customer?  It 
doesn't seem like that the thread dump output is giving me good information.

Thanks,

Keith

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-dev/attachments/20080827/c924b4cd/attachment.html