[concurrency-interest] ClassLoader deadlock

Peter jini at zeus.net.au
Sat Feb 6 21:32:58 UTC 2016


The 0x040ebee8 monitor is most likely being held by native code.

Regards,

Peter Firmstone.


On 7/02/2016 4:27 AM, Peter Levart wrote:
>
>
> On 02/06/2016 01:17 PM, Peter Firmstone wrote:
>> The security manager is non blocking, unfortunately java system 
>> classes aren't.
>
> It doesn't seem so. At least, I can't find where main thread locks the 
> 0x040ebee8 monitor:
>
> "Thread-1":
>   waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
>   which is held by "main"
>
>
> "main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for 
> monitor entry [0x0185e000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>         - waiting to lock <0x03f624b8> (a java.lang.Object)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         at 
> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>         at 
> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>         at 
> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>         at 
> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>         at 
> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>         at 
> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>         at 
> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>         at 
> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>         at 
> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>         at 
> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>         at 
> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>         at java.lang.System.checkIO(System.java:253)
>         at java.lang.System.setErr(System.java:199)
>         at 
> org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>
>
> ...no mention of 0x040ebee8 in main thread...?!
>
> Peter
>
>>
>> The policy provider in use thread confines PermissionCollection 
>> instances, which never leave the cpu cache, they are discarded 
>> immediately after use.
>>
>> The problem here is that two different threads are attempting to load 
>> the same class at the same time.
>>
>> I'll have to make sure that all classes are loaded before the 
>> security manager becomes the system security manager, either that or 
>> eliminate the AccessControlContext check permission cache.
>>
>> While the standard java security manager and policy provider consume 
>> around 10% cpu, this consumes less than 1%.
>>
>> On 6/02/2016 9:57 PM, David Holmes wrote:
>>> On 6/02/2016 9:39 PM, Peter Firmstone wrote:
>>>> Hmm, thought the new parallel lock stategy in ClassLoader wasn't
>>>> deadlock prone?  Guess I was wrong.
>>>
>>> The deadlock is introduced by a custom security manager. SM's play a 
>>> critical role in many parts of the core libraries so if they utilize 
>>> synchronization then they can easily introduce deadlock.
>>>
>>>> Observation:  On an unrelated occassion, I had a URLClassLoader
>>>> synchronization hotspot (well not standard URLClassLoader, but a high
>>>> performance RFC3986 URL ClassLoader, that use normalized RFC3986 URI
>>>> instead of URL DNS lookup), I solved that problem by thread confining
>>>> class loading.  I've never experienced deadlock using thread
>>>> confinement, initially I was worried that there would be
>>>> interdependencies, however I didn't experience a problem, as the 
>>>> single
>>>> thread would always load the classes it needed.
>>>>
>>>> I'm wondering if multiple locks for ClassLoader's might be the wrong
>>>> strategy, I found thread confinement was very performant.
>>>
>>> Are you suggesting that a class-load request is actually transmitted 
>>> to a single class-loading thread, while the initial thread blocks 
>>> until the loading is complete? That would still seem prone to the 
>>> same SM related deadlock scenario. And it would not generally be 
>>> particularly performant.
>>
>> I thought that too, until I tried it.
>>
>> Would you like to see some performance figures?
>>
>> Cheers,
>>
>> Peter.
>>
>>>
>>> David
>>>
>>>> Regards,
>>>>
>>>> Peter.
>>>>
>>>> 2016-02-06 21:06:07
>>>> Full thread dump Java HotSpot(TM) Client VM (25.0-b70 mixed mode):
>>>>
>>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x14387400 nid=0xc54
>>>> runnable [0x00000000]
>>>>     java.lang.Thread.State: RUNNABLE
>>>>
>>>> "C1 CompilerThread0" #8 daemon prio=9 os_prio=2 tid=0x14332c00
>>>> nid=0x111c waiting on condition [0x00000000]
>>>>     java.lang.Thread.State: RUNNABLE
>>>>
>>>> "Attach Listener" #7 daemon prio=5 os_prio=2 tid=0x14331c00 nid=0x13d8
>>>> waiting on condition [0x00000000]
>>>>     java.lang.Thread.State: RUNNABLE
>>>>
>>>> "Signal Dispatcher" #6 daemon prio=9 os_prio=2 tid=0x14331400 
>>>> nid=0x10b8
>>>> runnable [0x00000000]
>>>>     java.lang.Thread.State: RUNNABLE
>>>>
>>>> "Thread-1" #5 prio=10 os_prio=2 tid=0x14318800 nid=0x4a8 waiting for
>>>> monitor entry [0x03ded000]
>>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>>          at
>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65) 
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44) 
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57) 
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244) 
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202) 
>>>>
>>>>
>>>>          at
>>>> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447) 
>>>>
>>>>          at
>>>> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158) 
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137) 
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526) 
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266) 
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070) 
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535) 
>>>>
>>>>
>>>>          at 
>>>> java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>>>>          - locked <0x03f624b8> (a java.lang.Object)
>>>>          at 
>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>          at
>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167) 
>>>>
>>>>
>>>>          at
>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166) 
>>>>
>>>>
>>>>          - locked <0x03ef8b30> (a
>>>> org.cliffc.high_scale_lib.NonBlockingHashMap)
>>>>          at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
>>>>
>>>>          at
>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>>          at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j 
>>>>
>>>>
>>>> ava:180)
>>>>          at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294 
>>>>
>>>>
>>>> )
>>>>          at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
>>>>
>>>>
>>>>          at java.lang.Thread.run(Thread.java:744)
>>>>
>>>> "Finalizer" #3 daemon prio=8 os_prio=1 tid=0x14277800 nid=0x15cc in
>>>> Object.wait() [0x03cff000]
>>>>     java.lang.Thread.State: WAITING (on object monitor)
>>>>          at java.lang.Object.wait(Native Method)
>>>>          - waiting on <0x03e056d8> (a 
>>>> java.lang.ref.ReferenceQueue$Lock)
>>>>          at 
>>>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
>>>>          - locked <0x03e056d8> (a java.lang.ref.ReferenceQueue$Lock)
>>>>          at 
>>>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
>>>>          at 
>>>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>>
>>>> "Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x14271c00
>>>> nid=0x1398 in Object.wait() [0x144cf000]
>>>>     java.lang.Thread.State: WAITING (on object monitor)
>>>>          at java.lang.Object.wait(Native Method)
>>>>          - waiting on <0x03e05200> (a java.lang.ref.Reference$Lock)
>>>>          at java.lang.Object.wait(Object.java:502)
>>>>          at
>>>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>>          - locked <0x03e05200> (a java.lang.ref.Reference$Lock)
>>>>
>>>> "main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for 
>>>> monitor
>>>> entry [0x0185e000]
>>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>>>>          - waiting to lock <0x03f624b8> (a java.lang.Object)
>>>>          at 
>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>          at
>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65) 
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44) 
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57) 
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244) 
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202) 
>>>>
>>>>
>>>>          at java.lang.System.checkIO(System.java:253)
>>>>          at java.lang.System.setErr(System.java:199)
>>>>          at 
>>>> org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>>
>>>> "VM Thread" os_prio=2 tid=0x1426e400 nid=0x16a8 runnable
>>>>
>>>> "VM Periodic Task Thread" os_prio=2 tid=0x14388400 nid=0x17a8 
>>>> waiting on
>>>> condition
>>>>
>>>> JNI global references: 19
>>>>
>>>>
>>>> Found one Java-level deadlock:
>>>> =============================
>>>> "Thread-1":
>>>>    waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
>>>>    which is held by "main"
>>>> "main":
>>>>    waiting to lock monitor 0x14274a3c (object 0x03f624b8, a
>>>> java.lang.Object),
>>>>    which is held by "Thread-1"
>>>>
>>>> Java stack information for the threads listed above:
>>>> ===================================================
>>>> "Thread-1":
>>>>          at
>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65) 
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44) 
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57) 
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244) 
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202) 
>>>>
>>>>
>>>>          at
>>>> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447) 
>>>>
>>>>          at
>>>> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158) 
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137) 
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526) 
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266) 
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070) 
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535) 
>>>>
>>>>
>>>>          at 
>>>> java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>>>>          - locked <0x03f624b8> (a java.lang.Object)
>>>>          at 
>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>          at
>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167) 
>>>>
>>>>
>>>>          at
>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166) 
>>>>
>>>>
>>>>          - locked <0x03ef8b30> (a
>>>> org.cliffc.high_scale_lib.NonBlockingHashMap)
>>>>          at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
>>>>
>>>>          at
>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>>          at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j 
>>>>
>>>>
>>>> ava:180)
>>>>          at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294 
>>>>
>>>>
>>>> )
>>>>          at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
>>>>
>>>>
>>>>          at java.lang.Thread.run(Thread.java:744)
>>>> "main":
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>>>>          - waiting to lock <0x03f624b8> (a java.lang.Object)
>>>>          at 
>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>          at
>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65) 
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44) 
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57) 
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244) 
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261) 
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202) 
>>>>
>>>>
>>>>          at java.lang.System.checkIO(System.java:253)
>>>>          at java.lang.System.setErr(System.java:199)
>>>>          at 
>>>> org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>>
>>>> Found 1 deadlock.
>>
>
>
> _______________________________________________
> Concurrency-interest mailing list
> Concurrency-interest at cs.oswego.edu
> http://cs.oswego.edu/mailman/listinfo/concurrency-interest




More information about the core-libs-dev mailing list