ClassLoader deadlock

David Holmes david.holmes at oracle.com
Sat Feb 6 21:31:22 UTC 2016 

On 7/02/2016 4:27 AM, Peter Levart wrote:
>
>
> On 02/06/2016 01:17 PM, Peter Firmstone wrote:
>> The security manager is non blocking, unfortunately java system
>> classes aren't.
>
> It doesn't seem so. At least, I can't find where main thread locks the
> 0x040ebee8 monitor:
>
> "Thread-1":
>    waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
>    which is held by "main"

I think this is an internal monitor acquisition by the VM in the class 
initialization code. It won't show up on the thread dump - at least in 
the version of the VM being run here. Latest JDK 9 may behave differently.

Cheers,
David

>
> "main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for monitor
> entry [0x0185e000]
>     java.lang.Thread.State: BLOCKED (on object monitor)
>          at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>          - waiting to lock <0x03f624b8> (a java.lang.Object)
>          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>          at
> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>          at
> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>          at
> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>          at
> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>          at
> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>          at
> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>          at
> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>          at
> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>          at
> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>          at
> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>          at
> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>          at java.lang.System.checkIO(System.java:253)
>          at java.lang.System.setErr(System.java:199)
>          at org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>
>
> ...no mention of 0x040ebee8 in main thread...?!
>
> Peter
>
>>
>> The policy provider in use thread confines PermissionCollection
>> instances, which never leave the cpu cache, they are discarded
>> immediately after use.
>>
>> The problem here is that two different threads are attempting to load
>> the same class at the same time.
>>
>> I'll have to make sure that all classes are loaded before the security
>> manager becomes the system security manager, either that or eliminate
>> the AccessControlContext check permission cache.
>>
>> While the standard java security manager and policy provider consume
>> around 10% cpu, this consumes less than 1%.
>>
>> On 6/02/2016 9:57 PM, David Holmes wrote:
>>> On 6/02/2016 9:39 PM, Peter Firmstone wrote:
>>>> Hmm, thought the new parallel lock stategy in ClassLoader wasn't
>>>> deadlock prone?  Guess I was wrong.
>>>
>>> The deadlock is introduced by a custom security manager. SM's play a
>>> critical role in many parts of the core libraries so if they utilize
>>> synchronization then they can easily introduce deadlock.
>>>
>>>> Observation: On an unrelated occassion, I had a URLClassLoader
>>>> synchronization hotspot (well not standard URLClassLoader, but a high
>>>> performance RFC3986 URL ClassLoader, that use normalized RFC3986 URI
>>>> instead of URL DNS lookup), I solved that problem by thread confining
>>>> class loading.  I've never experienced deadlock using thread
>>>> confinement, initially I was worried that there would be
>>>> interdependencies, however I didn't experience a problem, as the single
>>>> thread would always load the classes it needed.
>>>>
>>>> I'm wondering if multiple locks for ClassLoader's might be the wrong
>>>> strategy, I found thread confinement was very performant.
>>>
>>> Are you suggesting that a class-load request is actually transmitted
>>> to a single class-loading thread, while the initial thread blocks
>>> until the loading is complete? That would still seem prone to the
>>> same SM related deadlock scenario. And it would not generally be
>>> particularly performant.
>>
>> I thought that too, until I tried it.
>>
>> Would you like to see some performance figures?
>>
>> Cheers,
>>
>> Peter.
>>
>>>
>>> David
>>>
>>>> Regards,
>>>>
>>>> Peter.
>>>>
>>>> 2016-02-06 21:06:07
>>>> Full thread dump Java HotSpot(TM) Client VM (25.0-b70 mixed mode):
>>>>
>>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x14387400 nid=0xc54
>>>> runnable [0x00000000]
>>>>     java.lang.Thread.State: RUNNABLE
>>>>
>>>> "C1 CompilerThread0" #8 daemon prio=9 os_prio=2 tid=0x14332c00
>>>> nid=0x111c waiting on condition [0x00000000]
>>>>     java.lang.Thread.State: RUNNABLE
>>>>
>>>> "Attach Listener" #7 daemon prio=5 os_prio=2 tid=0x14331c00 nid=0x13d8
>>>> waiting on condition [0x00000000]
>>>>     java.lang.Thread.State: RUNNABLE
>>>>
>>>> "Signal Dispatcher" #6 daemon prio=9 os_prio=2 tid=0x14331400
>>>> nid=0x10b8
>>>> runnable [0x00000000]
>>>>     java.lang.Thread.State: RUNNABLE
>>>>
>>>> "Thread-1" #5 prio=10 os_prio=2 tid=0x14318800 nid=0x4a8 waiting for
>>>> monitor entry [0x03ded000]
>>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>>          at
>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>>          at
>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>>
>>>>
>>>>          at
>>>> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
>>>>          at
>>>> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
>>>>
>>>>
>>>>          at
>>>> java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>>>>          - locked <0x03f624b8> (a java.lang.Object)
>>>>          at
>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>          at
>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167)
>>>>
>>>>
>>>>          at
>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166)
>>>>
>>>>
>>>>          - locked <0x03ef8b30> (a
>>>> org.cliffc.high_scale_lib.NonBlockingHashMap)
>>>>          at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>          at
>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>>          at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j
>>>>
>>>>
>>>> ava:180)
>>>>          at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294
>>>>
>>>>
>>>> )
>>>>          at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>
>>>>
>>>>          at java.lang.Thread.run(Thread.java:744)
>>>>
>>>> "Finalizer" #3 daemon prio=8 os_prio=1 tid=0x14277800 nid=0x15cc in
>>>> Object.wait() [0x03cff000]
>>>>     java.lang.Thread.State: WAITING (on object monitor)
>>>>          at java.lang.Object.wait(Native Method)
>>>>          - waiting on <0x03e056d8> (a
>>>> java.lang.ref.ReferenceQueue$Lock)
>>>>          at
>>>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
>>>>          - locked <0x03e056d8> (a java.lang.ref.ReferenceQueue$Lock)
>>>>          at
>>>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
>>>>          at
>>>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>>
>>>> "Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x14271c00
>>>> nid=0x1398 in Object.wait() [0x144cf000]
>>>>     java.lang.Thread.State: WAITING (on object monitor)
>>>>          at java.lang.Object.wait(Native Method)
>>>>          - waiting on <0x03e05200> (a java.lang.ref.Reference$Lock)
>>>>          at java.lang.Object.wait(Object.java:502)
>>>>          at
>>>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>>          - locked <0x03e05200> (a java.lang.ref.Reference$Lock)
>>>>
>>>> "main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for
>>>> monitor
>>>> entry [0x0185e000]
>>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>>>>          - waiting to lock <0x03f624b8> (a java.lang.Object)
>>>>          at
>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>          at
>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>>          at
>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>>
>>>>
>>>>          at java.lang.System.checkIO(System.java:253)
>>>>          at java.lang.System.setErr(System.java:199)
>>>>          at
>>>> org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>>
>>>> "VM Thread" os_prio=2 tid=0x1426e400 nid=0x16a8 runnable
>>>>
>>>> "VM Periodic Task Thread" os_prio=2 tid=0x14388400 nid=0x17a8
>>>> waiting on
>>>> condition
>>>>
>>>> JNI global references: 19
>>>>
>>>>
>>>> Found one Java-level deadlock:
>>>> =============================
>>>> "Thread-1":
>>>>    waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
>>>>    which is held by "main"
>>>> "main":
>>>>    waiting to lock monitor 0x14274a3c (object 0x03f624b8, a
>>>> java.lang.Object),
>>>>    which is held by "Thread-1"
>>>>
>>>> Java stack information for the threads listed above:
>>>> ===================================================
>>>> "Thread-1":
>>>>          at
>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>>          at
>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>>
>>>>
>>>>          at
>>>> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
>>>>          at
>>>> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
>>>>
>>>>
>>>>          at
>>>> java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>>>>          - locked <0x03f624b8> (a java.lang.Object)
>>>>          at
>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>          at
>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167)
>>>>
>>>>
>>>>          at
>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166)
>>>>
>>>>
>>>>          - locked <0x03ef8b30> (a
>>>> org.cliffc.high_scale_lib.NonBlockingHashMap)
>>>>          at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>          at
>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>>          at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j
>>>>
>>>>
>>>> ava:180)
>>>>          at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294
>>>>
>>>>
>>>> )
>>>>          at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>
>>>>
>>>>          at java.lang.Thread.run(Thread.java:744)
>>>> "main":
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>>>>          - waiting to lock <0x03f624b8> (a java.lang.Object)
>>>>          at
>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>          at
>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>>          at
>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>>
>>>>
>>>>          at java.lang.System.checkIO(System.java:253)
>>>>          at java.lang.System.setErr(System.java:199)
>>>>          at
>>>> org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>>
>>>> Found 1 deadlock.
>>
>

More information about the core-libs-dev mailing list