[foreign] Library loading

[Maurizio Cimadamore]

Hi,
lately, we’ve been thinking about our abstraction for loading native 
library, namely LibraryLookup. LibraryLookup is a cute little interface 
which lets you load libraries by name or by full path (similar to 
System::loadLibrary vs. System::load) - after you obtain a lookup 
object, you can then search for specific symbols and get back an address.

As many of you know, library loading with JNI is limited in two ways:

1. library loading is tied to the class loader
2. there is no way to get the “address” of a symbol in JNI (the lookup 
process, is hidden in VM and JDK code)

To be able to create downcall handles targeting specific library 
symbols, we need at least a solution for (2). And, since we were there, 
we thought that using the new abstraction to also address (1) was a good 
move.

But we started to have second thoughts about where we have landed. In 
many cases (see [1]), bindings distributed via Maven will attempt the 
standard trick of bundling native libraries with jars, extracting native 
library on a temp folder, from where they are loaded (using a full 
path). Unfortunately, tricks such as this _no longer works_  because the 
new LibraryLookup abstraction is completely orthogonal w.r.t. 
System::loadLibrary. This creates a new migration problem for bindings 
that want to gradually move away from JNI; as [1] demonstrates, when the 
old tricks are mixed with the new APIs, things simply don't work, as 
it's not possible to side-effect the set of libraries visible to the VM 
in the same way it was possible using System::loadLibrary.

Another issue is that the behavior of LibraryLookup::ofDefault is 
platform dependent (this is a known issue, see [2]). On some systems 
(e.g. MacOS), it “leaks” symbols from other, unrelated libraries 
(allowing, in this case, the bindings to surprisingly work!). In other 
systems, default lookup does not leak. Moreover (and we already knew 
this) default lookup on Windows is an hack that has been implemented by 
using a library which allows to walk all loaded libraries in current 
process (this is a functionality mostly intended for debugging 
purposes). In other words, the semantics of LibraryLookup::ofDefault is 
unstable. We have tried to explore what it would take to make this 
default lookup “stable”. And the answers weren't pretty, in the sense 
that we would need to parse output of system specific commands (e.g. 
`ldconfig -p`), or take the existing default lookup and "sanitize" it, 
by filtering out (either in Java, or at lower level, using linker map 
files) symbols that belong to the C standard library. While this seems 
like a reasonable task, that path is also full of thorns: certain 
symbols (errno, atexit, to name a few) are implemented in 
platform/compiler dependent ways, so it is difficult to give the user a 
view of a “standard C library” just in terms of a library lookup. In 
other words, putting together a C standard library is not just a mere 
matter of exposing the lookup which can be used to find its symbols; it 
probably involves building a full high-level, portable standard C API, 
which, while useful, is not something we're willing to tackle at this stage.

But, wait, there’s more; while inventing a new, classloader-neutral 
library mechanism sounds good on paper, as it allows to remove 
restrictions in the loading mechanism (JNI does not allow the same 
library to be loaded by multiple loaders), the classloader dependency 
also gave us a pretty nice way to manage the lifecycle of a native 
library: the library cannot be unloaded unless the owning classloader 
has been garbage collected. This avoids many catastrophic situations 
where a library is unloaded just before (or while) a native call on that 
library takes place. To address these issues, LibraryLookup is left 
reinventing much of the same reachability-based machinery (see all the 
javadoc in LibraryLookup, where terms like “X keeps a strong reference 
to Y” are employed throughout). One could argue that, now that we have 
ResourceScope, it would be possible, in principle, to build a 
ResourceScope-backed library loader. While this is possible, it must be 
noted the current lookup mechanism isn’t that, sadly.

So, what we’re left with is a LibraryLookup abstraction which looks nice 
and simple on paper, but that, in practice, is difficult to work with, 
has convoluted lifecycle properties, and does not provide a backward 
compatible option for clients that want to keep using the same JNI 
“hacks” (e.g. extracting libraries from jars, see above). In other 
words, LibraryLookup is probably not the primitive abstraction we want 
here. Luckily, a much simpler approach is possible:

MemoryAddress addr = MethodHandles.lookup().findNative("strlen");

Users will load libraries the usual way (System::load/loadLibrary) and 
findNative will look into the method handle lookup class’ classloader to 
lookup into all the libraries associated with that classloader. Turns 
out that a simple method like this is enough to address 95% of the use 
cases addressed by the current LibraryLookup abstraction --- basically 
everything except from default lookup, which is also the 5% which works 
unreliably.

Of course, since we’re still incubating, we would need to place this 
method somewhere else. e.g. a static method in CLinker (and maybe make 
it @CallerSensitive, to retrieve the classloader associated with the 
caller) - but that all seems doable enough. Note that this does not 
close the door, in the future, to building a more sophisticated library 
loading mechanism, where libraries are loaded and unloaded using a 
ResourceScope. Or, maybe ad-hoc library loading wrappers will naturally 
emerge (after all, it is trivial to write a custom library loader using 
dlopen/dlsym together with the Foreign Linker API). A (sharp) prototype 
of this approach is available at [3].

For all the reasons listed above, we plan to switch to the new, simpler, 
loading mechanism at some point after integrating the current Panama 
work into upstream jdk [2]. We believe that the proposed loading 
mechanism is more primitive, and provides a much better migration path 
for existing frameworks having to deal with distribution of native 
libraries via Maven or Gradle.

Maurizio

[1] - https://github.com/openjdk/panama-foreign/pull/509
[2] - https://bugs.openjdk.java.net/browse/JDK-8265222
[3] - 
https://github.com/openjdk/panama-foreign/compare/foreign-jextract...sundararajana:panama_uses_jni_loading?expand=1
​