RFR: 8231584: Deadlock with ClassLoader.findLibrary and System.loadLibrary call

[Mandy Chung]

I will need to look at this closer.   Skimming through the 
implementation, it seems to me that the Runtime::loadLibrary0 does not 
need to be synchronized.   ClassLoader::loadLibrary0 should ensure that 
a native library of a given name is loaded and registered only once.   
Can you explain why you think ClassLoader::findLibrary needs to be 
called with a lock?

Mandy

On 9/27/19 9:37 AM, Anton Kozlov wrote:
> Hi,
>
> I think that JDK-8194653 [0] affects jdk/jdk and it doesn't specific to FileSystems.getDefault.
>
> I'm starting a new thread/bug as the original issue marked as resolved ([3])
>
> Ryan, thanks for investigation and providing a test [1]!
>
> But the test fails in the same way if the FileSystems.getDefault replaced with anything that calls System.loadLibrary in an initializer. For jdk/jdk it may be jdk.net.ExtendedSocketOptions.SO_FLOW_SLA and the issue will arise.
>
> I don't think it's possible to provide a general fix for that. We are unable to prevent a user to take the Runtime's monitor and do initialization and loadLibrary, which make a room for deadlock. Taking the lock is incorrect action at first. We can only prevent doing that by accident.
>
>  From the original deadlock report [2], initialization and loadLibrary under the monitor caused not by java.* code, but class loader's findLibrary (sun.misc.ExtClassLoader to be specific). A workaround for ExtClassLoader is possible, to ensure all necessary classes are inited before findLibrary called, i.e. in the constructor. But then we should require the same for any custom class loader. Chances that it would be implemented correctly are not high since even ExtClassLoader is flawed.
>
> I propose to call a ClassLoader.findLibrary with Runtime's monitor unlocked:
>
> http://cr.openjdk.java.net/~akozlov/8231584/webrev.00
>
> Thanks,
> Anton
>
> [0]: https://bugs.openjdk.java.net/browse/JDK-8194653
> [1]: https://mail.openjdk.java.net/pipermail/core-libs-dev/2019-April/059811.html
> [2]: http://mail.openjdk.java.net/pipermail/core-libs-dev/2018-January/050819.html
> [3]: https://bugs.openjdk.java.net/browse/JDK-8231584
>