library loading - continued

Mon May 10 22:25:42 UTC 2021

It's not possible to create bindings for dlfcn, the header that provides 
the function declarations dlopen, dlclose, dlsym, etc, it seems. 
CLinker.systemLookup() fails to provide the symbol, anyway, and 
attempting to load libc fails. The header says that it's part of the GNU 
C library but maybe I'm barking up the wrong tree or something.

Is System.load()/System.loadLibrary() really the best approach? Not only 
does it have issues loading libc(and others?) but it isn't exactly all 
that primitive of a API either. Both dlopen(Linux) and 
LoadLibrary(Windows) both return pointers(which can be user managed) 
when loading libraries and dlopen has modes that can be set, but 
System.load()/System.loadLibrary() both register the loaded libraries 
with a ClassLoader, a JVM construct. How does one get a pointer of a 
loaded library from a ClassLoader and then use that pointer to unload a 
library?

In addition to all that, it's no longer clear(compared to the old way 
Panama did things) where a particular symbol is coming from. Yes, we 
know it's from a ClassLoader(if you bother reading javadoc, which 
doesn't work on Netbeans) but there isn't a visual representation of 
this in code.

On 5/10/21 7:49 AM, Maurizio Cimadamore wrote:
> Hi,
> in my email [1] I described a plan to simplify the library loading 
> mechanism based on the `LibraryLookup` abstraction. The motivations 
> behind the restructuring were three-fold:
>
> * Semantics of `LibraryLookup::ofDefault` is unstable; on some 
> platforms, default lookup leaks other loaded library symbols; on other 
> platforms (e.g. Windows) implementing it required use of debugging APIs.
> * `LibraryLookup` has its own concept of lifecycle which is 
> incompatible with `ResourceScope`, and also with that typically 
> associated with JNI loaded libraries (whose lifecycle is associated 
> with that of the owning classloader)
> * `LibraryLookup` does not offer a migration path for JNI-based 
> frameworks. That is, many frameworks are written with the assumption 
> that `System::loadLibrary` will affect the set of available symbols 
> _everywhere_ else (at least within the same classloader context)
>
> For these reasons, we decided to remove `LibraryLookup` and to provide 
> a more basic lookup capability instead, as a static method 
> (`CLinker::findNative`). This method would lookup symbols in libraries 
> loaded by the classloader used by the caller; in doing so, the new 
> method addresses the migration issues which affected `LibraryLookup`.
>
> When writing this patch [2, 3] we have realized that this new scheme 
> was not without its own issues:
>
> * Loading symbols in the standard C library is now incredibly 
> difficult, sometimes requiring surprising workarounds [4, 5]
> * There is no way to e.g. implement a lookup which filters the set of 
> symbols available to users
>
> After staring at these issues, we realized that having a lookup 
> abstraction was not, in itself, the root cause of the problems we were 
> seeing in [1]. These problems were, rather, caused by the fact that 
> the lookup abstraction we had was introducing its own concept of 
> library loading which had nothing to do with JNI library loading. In 
> other words, `LibraryLookup`, despite its name, wasn't a *pure* lookup 
> abstraction, it was attempting to do a little more (e.g. keeping 
> libraries loaded using GC reachability), without too much success.
>
> Having realized that, we now believe we can safely reintroduce some 
> kind of lookup abstraction, in the form of `SymbolLookup`: a simple 
> functional interface which, given a symbol name, gives us the address 
> of that symbol (if available). The use or a more neutral name here 
> (`SymbolLookup` instead of `LibraryLookup`) is deliberate: not always 
> a lookup is a result of library loading; lookup objects might in fact 
> be created (or composed) by users, using existing resources.
>
> How do developers obtain a `SymbolLookup` ? Two ways:
>
> * They can obtain a `SymbolLookup` for a given class loader - which 
> allows to search symbols in all the libraries loaded by that class loader
> * They can ask `CLinker` a so called *system lookup* - a lookup which 
> allows to search for basic C symbols (such as `strlen` and `qsort`)
>
> Note that the first mechanism allows us to solve the migration 
> problem, by exposing the lookup abstraction associated with a specific 
> class loader. This means frameworks will still be able to call 
> `System::loadLibrary` to *side-effect* the results of a class 
> loader-based symbol lookup. Note that, since `SymbolLookup` is a 
> simple functional interface, it would be possible for developers to 
> set up symbol lookup chains e.g. where lookup for parent class loader 
> is consulted first (e.g. following classloader delegation). This 
> flexibility will certainly come in handy in real world use cases.
>
> The second mechanism gives us something similar to the previous 
> concept of *default lookup* - but without the messy bits: the system 
> lookup is simply a system-dependent symbol lookup object, which might 
> help in retrieving common C symbols. The API makes no promises as to 
> *which* libraries will be consulted by this lookup (this is an 
> implementation detail). That said, the implementation (unlike before) 
> will not make use of `RTLD_DEFAULT` whose semantics is brittle and not 
> implementable across all OSs.
>
> It is possible that, in the future, we might add more ways to obtain a 
> symbol lookup - for instance:
>
> ```
> SymbolLookup.ofLibrary(String libName, ResourceScope scope)
>
> ```
>
> This is not too different from what we had in the original 
> `LibraryLookup` abstraction, and would allow developers to load a 
> library and associate its lifecycle with a `ResourceScope` (rather 
> than a class loader). That is, when the scope is closed, the library 
> will be unloaded. However, adding these new mode will require some 
> additional foundational work on the `CLinker` support - as we need to 
> make sure that the memory address used by a downcall method handle 
> cannot be unloaded while the downcall method handle is being invoked. 
> This means that, at the very minimum, the linker will need to 
> acquire/release the scope associated with the address of the native 
> function, to prevent premature closing of said scope.
>
> Summing up, we believe that while investigating for a more minimal 
> library lookup mechanism we have found a way to provide most (all?) of 
> the functionalities available before, in a more disciplined and 
> compositional manner.
>
> Cheers
> Maurizio
>
> [1] - 
> https://mail.openjdk.java.net/pipermail/panama-dev/2021-April/013577.html
> [2] - https://git.openjdk.java.net/panama-foreign/pull/526
> [3] - https://git.openjdk.java.net/panama-foreign/pull/529
> [4] - 
> https://github.com/sundararajana/panama-foreign/blob/197db28d5097b1689b4befcd7129eb7af13f41c2/test/jdk/java/foreign/libStdLibTest.c
> [5] - https://github.com/openjdk/panama-foreign/pull/527
>
>
>