JDK8: Unloading native JNI library

[Vemund Ostgaard]

Hi David,

On 25. aug. 2017 04:18, David Holmes wrote:
> Hi Vemund,
>
> What OS do you see this on? Are you able to test different OS?

Excellent question. Earlier in my struggles to get this working I had 
tested on both an RHEL7 and Ubuntu 16.04 and seen the same problems, so 
I did further testing only on the Ubuntu machine. Since I resolved a few 
issues since then, I went back to RHEL7 and rebuilt everything there to 
confirm both OS were still giving the same behavior.

Turned out that on RHEL7 I did no longer see any issue, the unloading 
worked fine also on the first attempt. To be sure, I rebuilt the native 
library from scratch also on Ubuntu 16.04, and to my surprise the issue 
was gone there too.

Scratching my head I eventually worked out how I had hit the problem 
originally. At some point I must have mixed up the native library 
compiled on RHEL7 and the one compiled on Ubuntu by mistake. I can now 
only reproduce the issue if I try to load the native library compiled on 
RHEL7 on the Ubuntu 16.04 machine. In that case, everything behaved 
exactly as before, the first library loaded can not be unloaded, but 
subsequent libraries can.

So, it is all a mistake on my part basically, when loading and unloading 
the native libraries on the same Linux flavor and version they were 
compiled on, it works.

Thanks for the reply, it led me to finding my mistake.

Vemund

>
> Thanks,
> David
>
> On 24/08/2017 10:37 PM, Vemund Ostgaard wrote:
>> Hello,
>>
>> I am struggling with a problem related to unloading (and reloading) a
>> native JNI library that has been loaded with System.load(). I have
>> been referred to this mailing list as the right place to ask, so
>> seeking your advice on the issue.
>>
>> My goal was to load a JNI library from a JVM process and be able to
>> reload an updated copy of the same library later without restarting
>> the JVM. The library might be loaded from the exact same path as
>> before or from a different path.
>>
>> The testing I have done (more details below) indicates to me that the
>> first JNI library I load with System.load() does not get unloaded
>> completely when its loading classloader is garbage collected,
>> subsequent attempts to load a library from the same path indicates
>> that the native library kept the state it had from the first loaded copy.
>>
>> Surprisingly though, any other copy of the native library that is
>> loaded from other file system locations than the first path are
>> apparently unloaded when their loading classloader is garbage
>> collected, and subsequent attempts to load from those paths indicate
>> that those libraries do not keep their state.
>>
>> So, I am curious whether there is something special about the first
>> time System.load() is used from application code in a new JVM, and
>> whether there is some way I can avoid that behavior.
>>
>>
>> Details about my testing:
>>
>> I have added these methods to my native JNI library:
>>
>> JNIEXPORT jint JNICALL JNI_OnLoad(JavaVM *jvm, void *reserved)
>> {
>>      counter++;
>>      printf("JNI_OnLoad called: %d\n", counter);
>>      (void)jvm;
>>      (void)reserved;
>>      return JNI_VERSION_1_4;
>> }
>>
>> JNIEXPORT void JNICALL JNI_OnUnload(JavaVM *jvm, void *reserved)
>> {
>>      printf( "JNI_OnUnload called: %d\n", counter);
>>      (void)jvm;
>>      (void)reserved;
>>      return;
>> }
>>
>> The counter is a global variable in the same file, and it is
>> initialized to 0. The plan was to see if the counter was initialized
>> to 0 again after the library is unloaded and loaded again, and also to
>> see that these methods were called at all. I also recompiled the
>> native library in two other versions with the counter initialized to
>> 100 and 200 respectively, to be able to tell the difference between
>> different copies of the library being loaded/unloaded.
>>
>> I wrote a small java class that calls System.load() to load my native
>> JNI library, and a small custom classloader that extends URL
>> classloader. To both of these classes I added overrides for finalize()
>> just to see it being called and printing out a unique ID to know what
>> copy of each class was being finalized.
>>
>> The third Java class is just a main() method that calls a test()
>> method with the test logic using the other two classes to test loading
>> and unloading of the native library. When the test() method returns to
>> the main() method, the classloader object goes out of scope and I call
>> System.gc() a few times to get everything garbage collected. The
>> test() method also returns a WeakReference() to the classloader object
>> so I can call WeakReference.get() before and after the System.gc()
>> calls to verify that this becomes null. I interpret the WeakReference
>> becoming null as a sign that my classloader is getting garbage collected.
>>
>> My testing shows that in all cases:
>> - JNI_OnLoad() and JNI_OnUnload() gets called as expected
>> - The finalizer() methods for my two classes are called as expected
>> - The WeakReference.get() for my custom classloader returns null after
>> a few System.gc() calls and sleeping a couple seconds total.
>>
>> As already mentioned, the first JNI library loaded keeps its state,
>> the counter printed from JNI_OnLoad() and JNI_OnUnload() keeps
>> increasing for every repeated load+unload from that file location,
>> while for all other libraries loaded from other file locations, the
>> state is apparently reset. This holds true also if using multiple
>> custom classloaders or identical copies of the JNI library in
>> different locations. The key factor seems to only be loading from the
>> same file location as used the first time.
>>
>> I can't see that my code behaves any different for the multiple times
>> I try to load and unload the JNI libraries, so I am wondering if there
>> is some JNI bug or known behavior that explains what I see?
>>
>>
>> Thank you,
>> Vemund