ClassLoader deadlock
Peter Levart
peter.levart at gmail.com
Sat Feb 6 18:27:29 UTC 2016
On 02/06/2016 01:17 PM, Peter Firmstone wrote:
> The security manager is non blocking, unfortunately java system
> classes aren't.
It doesn't seem so. At least, I can't find where main thread locks the
0x040ebee8 monitor:
"Thread-1":
waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
which is held by "main"
"main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for monitor
entry [0x0185e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
- waiting to lock <0x03f624b8> (a java.lang.Object)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
at
org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
at
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
at
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
at
org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
at
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
at
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
at
org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
at
org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
at
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
at
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
at java.lang.System.checkIO(System.java:253)
at java.lang.System.setErr(System.java:199)
at org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
...no mention of 0x040ebee8 in main thread...?!
Peter
>
> The policy provider in use thread confines PermissionCollection
> instances, which never leave the cpu cache, they are discarded
> immediately after use.
>
> The problem here is that two different threads are attempting to load
> the same class at the same time.
>
> I'll have to make sure that all classes are loaded before the security
> manager becomes the system security manager, either that or eliminate
> the AccessControlContext check permission cache.
>
> While the standard java security manager and policy provider consume
> around 10% cpu, this consumes less than 1%.
>
> On 6/02/2016 9:57 PM, David Holmes wrote:
>> On 6/02/2016 9:39 PM, Peter Firmstone wrote:
>>> Hmm, thought the new parallel lock stategy in ClassLoader wasn't
>>> deadlock prone? Guess I was wrong.
>>
>> The deadlock is introduced by a custom security manager. SM's play a
>> critical role in many parts of the core libraries so if they utilize
>> synchronization then they can easily introduce deadlock.
>>
>>> Observation: On an unrelated occassion, I had a URLClassLoader
>>> synchronization hotspot (well not standard URLClassLoader, but a high
>>> performance RFC3986 URL ClassLoader, that use normalized RFC3986 URI
>>> instead of URL DNS lookup), I solved that problem by thread confining
>>> class loading. I've never experienced deadlock using thread
>>> confinement, initially I was worried that there would be
>>> interdependencies, however I didn't experience a problem, as the single
>>> thread would always load the classes it needed.
>>>
>>> I'm wondering if multiple locks for ClassLoader's might be the wrong
>>> strategy, I found thread confinement was very performant.
>>
>> Are you suggesting that a class-load request is actually transmitted
>> to a single class-loading thread, while the initial thread blocks
>> until the loading is complete? That would still seem prone to the
>> same SM related deadlock scenario. And it would not generally be
>> particularly performant.
>
> I thought that too, until I tried it.
>
> Would you like to see some performance figures?
>
> Cheers,
>
> Peter.
>
>>
>> David
>>
>>> Regards,
>>>
>>> Peter.
>>>
>>> 2016-02-06 21:06:07
>>> Full thread dump Java HotSpot(TM) Client VM (25.0-b70 mixed mode):
>>>
>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x14387400 nid=0xc54
>>> runnable [0x00000000]
>>> java.lang.Thread.State: RUNNABLE
>>>
>>> "C1 CompilerThread0" #8 daemon prio=9 os_prio=2 tid=0x14332c00
>>> nid=0x111c waiting on condition [0x00000000]
>>> java.lang.Thread.State: RUNNABLE
>>>
>>> "Attach Listener" #7 daemon prio=5 os_prio=2 tid=0x14331c00 nid=0x13d8
>>> waiting on condition [0x00000000]
>>> java.lang.Thread.State: RUNNABLE
>>>
>>> "Signal Dispatcher" #6 daemon prio=9 os_prio=2 tid=0x14331400
>>> nid=0x10b8
>>> runnable [0x00000000]
>>> java.lang.Thread.State: RUNNABLE
>>>
>>> "Thread-1" #5 prio=10 os_prio=2 tid=0x14318800 nid=0x4a8 waiting for
>>> monitor entry [0x03ded000]
>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>> at
>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>> at
>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>
>>> at
>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>> at
>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>
>>>
>>> at
>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>
>>>
>>> at
>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>
>>>
>>> at
>>> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
>>> at
>>> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)
>>>
>>>
>>> at
>>> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
>>>
>>> at
>>> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)
>>>
>>>
>>> at
>>> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)
>>>
>>>
>>> at
>>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
>>>
>>> at
>>> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
>>>
>>>
>>> at
>>> java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>>> - locked <0x03f624b8> (a java.lang.Object)
>>> at
>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>> at
>>> org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167)
>>>
>>>
>>> at
>>> org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200)
>>>
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166)
>>>
>>>
>>> - locked <0x03ef8b30> (a
>>> org.cliffc.high_scale_lib.NonBlockingHashMap)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> at
>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j
>>>
>>>
>>> ava:180)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294
>>>
>>>
>>> )
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>
>>>
>>> at java.lang.Thread.run(Thread.java:744)
>>>
>>> "Finalizer" #3 daemon prio=8 os_prio=1 tid=0x14277800 nid=0x15cc in
>>> Object.wait() [0x03cff000]
>>> java.lang.Thread.State: WAITING (on object monitor)
>>> at java.lang.Object.wait(Native Method)
>>> - waiting on <0x03e056d8> (a
>>> java.lang.ref.ReferenceQueue$Lock)
>>> at
>>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
>>> - locked <0x03e056d8> (a java.lang.ref.ReferenceQueue$Lock)
>>> at
>>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
>>> at
>>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>
>>> "Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x14271c00
>>> nid=0x1398 in Object.wait() [0x144cf000]
>>> java.lang.Thread.State: WAITING (on object monitor)
>>> at java.lang.Object.wait(Native Method)
>>> - waiting on <0x03e05200> (a java.lang.ref.Reference$Lock)
>>> at java.lang.Object.wait(Object.java:502)
>>> at
>>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>> - locked <0x03e05200> (a java.lang.ref.Reference$Lock)
>>>
>>> "main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for
>>> monitor
>>> entry [0x0185e000]
>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>>> - waiting to lock <0x03f624b8> (a java.lang.Object)
>>> at
>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>> at
>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>> at
>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>
>>> at
>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>> at
>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>
>>>
>>> at
>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>
>>>
>>> at
>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>
>>>
>>> at java.lang.System.checkIO(System.java:253)
>>> at java.lang.System.setErr(System.java:199)
>>> at
>>> org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>
>>> "VM Thread" os_prio=2 tid=0x1426e400 nid=0x16a8 runnable
>>>
>>> "VM Periodic Task Thread" os_prio=2 tid=0x14388400 nid=0x17a8
>>> waiting on
>>> condition
>>>
>>> JNI global references: 19
>>>
>>>
>>> Found one Java-level deadlock:
>>> =============================
>>> "Thread-1":
>>> waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
>>> which is held by "main"
>>> "main":
>>> waiting to lock monitor 0x14274a3c (object 0x03f624b8, a
>>> java.lang.Object),
>>> which is held by "Thread-1"
>>>
>>> Java stack information for the threads listed above:
>>> ===================================================
>>> "Thread-1":
>>> at
>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>> at
>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>
>>> at
>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>> at
>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>
>>>
>>> at
>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>
>>>
>>> at
>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>
>>>
>>> at
>>> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
>>> at
>>> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)
>>>
>>>
>>> at
>>> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
>>>
>>> at
>>> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)
>>>
>>>
>>> at
>>> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)
>>>
>>>
>>> at
>>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
>>>
>>> at
>>> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
>>>
>>>
>>> at
>>> java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>>> - locked <0x03f624b8> (a java.lang.Object)
>>> at
>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>> at
>>> org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167)
>>>
>>>
>>> at
>>> org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200)
>>>
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166)
>>>
>>>
>>> - locked <0x03ef8b30> (a
>>> org.cliffc.high_scale_lib.NonBlockingHashMap)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> at
>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j
>>>
>>>
>>> ava:180)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294
>>>
>>>
>>> )
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>
>>>
>>> at java.lang.Thread.run(Thread.java:744)
>>> "main":
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>>> - waiting to lock <0x03f624b8> (a java.lang.Object)
>>> at
>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>> at
>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>> at
>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>
>>> at
>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>
>>>
>>> at
>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>> at
>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>
>>>
>>> at
>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>
>>>
>>> at
>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>
>>>
>>> at java.lang.System.checkIO(System.java:253)
>>> at java.lang.System.setErr(System.java:199)
>>> at
>>> org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>
>>> Found 1 deadlock.
>
More information about the core-libs-dev
mailing list