RFR: 8266310: deadlock while loading the JNI code

[Aleksei Voitylov]

Hi David, Mandy,

In the updated PR I removed the lock held during the load/unload calls.
Our testing confirmed that without that removal the deadlock can easily
be reproduced, even without signed jars. Now the lock is only held to
prevent races during access to "libraries" and "loadedLibraryNames".

Thank you,
-Aleksei

On 14/05/2021 03:28, David Holmes wrote:
> On 14/05/2021 7:20 am, Mandy Chung wrote:
>>
>>
>> On 5/13/21 6:05 AM, David Holmes wrote:
>>> Not every problem has a solution :) If JNI_OnLoad has to execute
>>> atomically with respect to loading a library then there will always
>>> be a deadlock potential. The only complete solution would not hold a
>>> lock while JNI_OnLoad is executed, but that completely changes the
>>> semantics of native library loading. 
>>
>> I have been giving some thought on this issue.  I agree with David
>> that we should look into a solution not holding a lock while
>> JNI_OnLoad is executed.  It might be possible to put all threads that
>> trying to load the same native library that is being loaded to wait
>> and only one thread is loading it and executing JNI_OnLoad (e.g. add
>> a state in NativeLibrary to indicate it is being loaded, already
>> loaded, being unloaded).  I haven't had the cycle to think through it
>> in details though.
>
> Any mechanism that blocks threads from accessing the library while
> another thread is performing JNI_OnLoad, suffers from the same
> deadlock problem.
>
> David
>
>> Mandy