ClassLoader deadlock

David Holmes david.holmes at oracle.com
Sat Feb 6 11:57:00 UTC 2016 

On 6/02/2016 9:39 PM, Peter Firmstone wrote:
> Hmm, thought the new parallel lock stategy in ClassLoader wasn't
> deadlock prone?  Guess I was wrong.

The deadlock is introduced by a custom security manager. SM's play a 
critical role in many parts of the core libraries so if they utilize 
synchronization then they can easily introduce deadlock.

> Observation:  On an unrelated occassion, I had a URLClassLoader
> synchronization hotspot (well not standard URLClassLoader, but a high
> performance RFC3986 URL ClassLoader, that use normalized RFC3986 URI
> instead of URL DNS lookup), I solved that problem by thread confining
> class loading.  I've never experienced deadlock using thread
> confinement, initially I was worried that there would be
> interdependencies, however I didn't experience a problem, as the single
> thread would always load the classes it needed.
>
> I'm wondering if multiple locks for ClassLoader's might be the wrong
> strategy, I found thread confinement was very performant.

Are you suggesting that a class-load request is actually transmitted to 
a single class-loading thread, while the initial thread blocks until the 
loading is complete? That would still seem prone to the same SM related 
deadlock scenario. And it would not generally be particularly performant.

David

> Regards,
>
> Peter.
>
> 2016-02-06 21:06:07
> Full thread dump Java HotSpot(TM) Client VM (25.0-b70 mixed mode):
>
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x14387400 nid=0xc54
> runnable [0x00000000]
>     java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread0" #8 daemon prio=9 os_prio=2 tid=0x14332c00
> nid=0x111c waiting on condition [0x00000000]
>     java.lang.Thread.State: RUNNABLE
>
> "Attach Listener" #7 daemon prio=5 os_prio=2 tid=0x14331c00 nid=0x13d8
> waiting on condition [0x00000000]
>     java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #6 daemon prio=9 os_prio=2 tid=0x14331400 nid=0x10b8
> runnable [0x00000000]
>     java.lang.Thread.State: RUNNABLE
>
> "Thread-1" #5 prio=10 os_prio=2 tid=0x14318800 nid=0x4a8 waiting for
> monitor entry [0x03ded000]
>     java.lang.Thread.State: BLOCKED (on object monitor)
>          at
> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>
>          at
> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>          at
> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>          at
> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>          at
> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>
>          at
> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>
>          at
> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>
>          at
> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>          at
> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>
>          at
> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>
>          at
> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>
>          at
> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
>          at
> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)
>
>          at
> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
>          at
> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)
>
>          at
> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)
>
>          at
> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
>          at
> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
>
>          at java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
>          at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
>          at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>          - locked <0x03f624b8> (a java.lang.Object)
>          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>          at
> org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167)
>
>          at
> org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200)
>
>          at
> org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166)
>
>          - locked <0x03ef8b30> (a
> org.cliffc.high_scale_lib.NonBlockingHashMap)
>          at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>          at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j
>
> ava:180)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294
>
> )
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
>          at java.lang.Thread.run(Thread.java:744)
>
> "Finalizer" #3 daemon prio=8 os_prio=1 tid=0x14277800 nid=0x15cc in
> Object.wait() [0x03cff000]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          - waiting on <0x03e056d8> (a java.lang.ref.ReferenceQueue$Lock)
>          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
>          - locked <0x03e056d8> (a java.lang.ref.ReferenceQueue$Lock)
>          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
>          at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x14271c00
> nid=0x1398 in Object.wait() [0x144cf000]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          - waiting on <0x03e05200> (a java.lang.ref.Reference$Lock)
>          at java.lang.Object.wait(Object.java:502)
>          at
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>          - locked <0x03e05200> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for monitor
> entry [0x0185e000]
>     java.lang.Thread.State: BLOCKED (on object monitor)
>          at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>          - waiting to lock <0x03f624b8> (a java.lang.Object)
>          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>          at
> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>
>          at
> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>          at
> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>          at
> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>          at
> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>
>          at
> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>
>          at
> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>
>          at
> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>          at
> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>
>          at
> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>
>          at
> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>
>          at java.lang.System.checkIO(System.java:253)
>          at java.lang.System.setErr(System.java:199)
>          at org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>
> "VM Thread" os_prio=2 tid=0x1426e400 nid=0x16a8 runnable
>
> "VM Periodic Task Thread" os_prio=2 tid=0x14388400 nid=0x17a8 waiting on
> condition
>
> JNI global references: 19
>
>
> Found one Java-level deadlock:
> =============================
> "Thread-1":
>    waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
>    which is held by "main"
> "main":
>    waiting to lock monitor 0x14274a3c (object 0x03f624b8, a
> java.lang.Object),
>    which is held by "Thread-1"
>
> Java stack information for the threads listed above:
> ===================================================
> "Thread-1":
>          at
> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>
>          at
> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>          at
> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>          at
> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>          at
> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>
>          at
> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>
>          at
> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>
>          at
> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>          at
> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>
>          at
> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>
>          at
> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>
>          at
> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
>          at
> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)
>
>          at
> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
>          at
> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)
>
>          at
> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)
>
>          at
> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
>          at
> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
>
>          at java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
>          at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
>          at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>          - locked <0x03f624b8> (a java.lang.Object)
>          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>          at
> org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167)
>
>          at
> org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200)
>
>          at
> org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166)
>
>          - locked <0x03ef8b30> (a
> org.cliffc.high_scale_lib.NonBlockingHashMap)
>          at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>          at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j
>
> ava:180)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294
>
> )
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
>          at java.lang.Thread.run(Thread.java:744)
> "main":
>          at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>          - waiting to lock <0x03f624b8> (a java.lang.Object)
>          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>          at
> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>
>          at
> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>          at
> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>          at
> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>          at
> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>
>          at
> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>
>          at
> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>
>          at
> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>          at
> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>
>          at
> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>
>          at
> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>
>          at java.lang.System.checkIO(System.java:253)
>          at java.lang.System.setErr(System.java:199)
>          at org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>
> Found 1 deadlock.

More information about the core-libs-dev mailing list