RFR: 8231584: Deadlock with ClassLoader.findLibrary and System.loadLibrary call

[Anton Kozlov]

Hi,

I think that JDK-8194653 [0] affects jdk/jdk and it doesn't specific to FileSystems.getDefault.

I'm starting a new thread/bug as the original issue marked as resolved ([3])

Ryan, thanks for investigation and providing a test [1]! 

But the test fails in the same way if the FileSystems.getDefault replaced with anything that calls System.loadLibrary in an initializer. For jdk/jdk it may be jdk.net.ExtendedSocketOptions.SO_FLOW_SLA and the issue will arise.

I don't think it's possible to provide a general fix for that. We are unable to prevent a user to take the Runtime's monitor and do initialization and loadLibrary, which make a room for deadlock. Taking the lock is incorrect action at first. We can only prevent doing that by accident.

>From the original deadlock report [2], initialization and loadLibrary under the monitor caused not by java.* code, but class loader's findLibrary (sun.misc.ExtClassLoader to be specific). A workaround for ExtClassLoader is possible, to ensure all necessary classes are inited before findLibrary called, i.e. in the constructor. But then we should require the same for any custom class loader. Chances that it would be implemented correctly are not high since even ExtClassLoader is flawed.

I propose to call a ClassLoader.findLibrary with Runtime's monitor unlocked:

http://cr.openjdk.java.net/~akozlov/8231584/webrev.00

Thanks,
Anton

[0]: https://bugs.openjdk.java.net/browse/JDK-8194653
[1]: https://mail.openjdk.java.net/pipermail/core-libs-dev/2019-April/059811.html
[2]: http://mail.openjdk.java.net/pipermail/core-libs-dev/2018-January/050819.html
[3]: https://bugs.openjdk.java.net/browse/JDK-8231584