library loading - continued

Tue May 11 10:07:29 UTC 2021

On 10/05/2021 23:25, Ty Young wrote:
> It's not possible to create bindings for dlfcn, the header that 
> provides the function declarations dlopen, dlclose, dlsym, etc, it 
> seems. CLinker.systemLookup() fails to provide the symbol, anyway, and 
> attempting to load libc fails. The header says that it's part of the 
> GNU C library but maybe I'm barking up the wrong tree or something.
dlopen/dlclose, at least on my linux system are in the dl library, as 
you later discovered. The good thing about the system lookup approach is 
that we can expand the set of "system" symbols to e.g. include POSIX (on 
Linux). After all, msvcrt.dll also contains more stuff than just 
standard C library.
>
>
> Is System.load()/System.loadLibrary() really the best approach? Not 
> only does it have issues loading libc(and others?) but it isn't 
> exactly all that primitive of a API either. Both dlopen(Linux) and 
> LoadLibrary(Windows) both return pointers(which can be user managed) 
> when loading libraries and dlopen has modes that can be set, but 
> System.load()/System.loadLibrary() both register the loaded libraries 
> with a ClassLoader, a JVM construct. How does one get a pointer of a 
> loaded library from a ClassLoader and then use that pointer to unload 
> a library?

Let's sort through the arguments a bit here:

* System.load doesn't have any issue loading the C library - _assuming 
you can find where it is_ :-) While the location of such library is 
relatively stable on Windows and MacOS, on Linux is completely distro 
dependent. Most distributions nowadays embrace the "multiarch" 
philosophy, so they have a different libc for different archs. For 
instance, in my machine, libc is here:

/usr/lib/x86_64-linux-gnu/libc.so

Well, it *seems* to be there - if you look at that file you discover 
it's a text file (!!), used by ldconfig:

```
$ cat /usr/lib/x86_64-linux-gnu/libc.so
/* GNU ld script
    Use the shared library, but some functions are only in
    the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf64-x86-64)
GROUP ( /lib/x86_64-linux-gnu/libc.so.6 
/usr/lib/x86_64-linux-gnu/libc_nonshared.a  AS_NEEDED ( 
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 ) )
```

Which gives away the real location of the library:

/lib/x86_64-linux-gnu/libc.so.6

We've been looking around for a "standard" way to discover the location 
of these system libraries and came up pretty empty handed. The best way 
seems to be to run:

ldconfig -p

And then filter the output in some way. Is the output format stable? Of 
course not - see [1]. So, while there's nothing wrong with 
System.loadLibrary in terms of "library loading" ability, on linux there 
is simply no great way to tell where libc is.

* Registering a library with a classloader is a pretty pragmatic way to 
make sure the library doesn't get unloaded when you don't want to (e.g. 
_while_ a native method is being executed, or just before it is called). 
If you drop that dependency, you will probably have to reinvent 
something similar (which we did in our LibraryLookup construct). As I 
write in my email, we do want a ResourceScope-based library loading - 
but to get there there's other stuff that needs to be done first. In 
other words, calling dlopen/LoadLibrary is the "easy" part of the 
problem - the hard part is to guarantee that library is not unloaded 
prematurely - System.loadLibrary, for better or worse, already has 
guarantees in that direction.

* System.loadLibrary is side-effecting. That is, you can have a big 
static initializer which does a bunch of stuff - e.g. extracts native 
libraries from a jar, copy the library in a temp folder and then loads 
it from there. All the symbols will be visible from the rest of the 
program (if the initializer is ran early enough). These tricks no longer 
works when working with libraries whose loading is fully encapsulated in 
a LibraryLookup instance - and these tricks are pretty universal in 
Maven-land. We'd like to offer something that offers a gentle migration 
path for those developers too.

>
>
> In addition to all that, it's no longer clear(compared to the old way 
> Panama did things) where a particular symbol is coming from. Yes, we 
> know it's from a ClassLoader(if you bother reading javadoc, which 
> doesn't work on Netbeans) but there isn't a visual representation of 
> this in code.

True - this is because of the side-effecting described above. Turns out, 
a lot of real code depends on that, for better or worse.

To me the important question is this: does providing a classloader-based 
lookup _now_ prevents us from adding better library loading _in the 
future_ ? As discussed in my email, the answer is no, and in fact we 
already have plans to offer a more minimal wrapper around 
dlopen/dlsym/LoadLibrary (and ResourceScope). For code that is rewritten 
from scratch, that more minimal wrapper might well be the preferred 
solution - but for some code, esp. that distributed via maven/gradle, it 
might still be preferrable to go through the class loader lookup.

Maurizio

>
>
> On 5/10/21 7:49 AM, Maurizio Cimadamore wrote:
>> Hi,
>> in my email [1] I described a plan to simplify the library loading 
>> mechanism based on the `LibraryLookup` abstraction. The motivations 
>> behind the restructuring were three-fold:
>>
>> * Semantics of `LibraryLookup::ofDefault` is unstable; on some 
>> platforms, default lookup leaks other loaded library symbols; on 
>> other platforms (e.g. Windows) implementing it required use of 
>> debugging APIs.
>> * `LibraryLookup` has its own concept of lifecycle which is 
>> incompatible with `ResourceScope`, and also with that typically 
>> associated with JNI loaded libraries (whose lifecycle is associated 
>> with that of the owning classloader)
>> * `LibraryLookup` does not offer a migration path for JNI-based 
>> frameworks. That is, many frameworks are written with the assumption 
>> that `System::loadLibrary` will affect the set of available symbols 
>> _everywhere_ else (at least within the same classloader context)
>>
>> For these reasons, we decided to remove `LibraryLookup` and to 
>> provide a more basic lookup capability instead, as a static method 
>> (`CLinker::findNative`). This method would lookup symbols in 
>> libraries loaded by the classloader used by the caller; in doing so, 
>> the new method addresses the migration issues which affected 
>> `LibraryLookup`.
>>
>> When writing this patch [2, 3] we have realized that this new scheme 
>> was not without its own issues:
>>
>> * Loading symbols in the standard C library is now incredibly 
>> difficult, sometimes requiring surprising workarounds [4, 5]
>> * There is no way to e.g. implement a lookup which filters the set of 
>> symbols available to users
>>
>> After staring at these issues, we realized that having a lookup 
>> abstraction was not, in itself, the root cause of the problems we 
>> were seeing in [1]. These problems were, rather, caused by the fact 
>> that the lookup abstraction we had was introducing its own concept of 
>> library loading which had nothing to do with JNI library loading. In 
>> other words, `LibraryLookup`, despite its name, wasn't a *pure* 
>> lookup abstraction, it was attempting to do a little more (e.g. 
>> keeping libraries loaded using GC reachability), without too much 
>> success.
>>
>> Having realized that, we now believe we can safely reintroduce some 
>> kind of lookup abstraction, in the form of `SymbolLookup`: a simple 
>> functional interface which, given a symbol name, gives us the address 
>> of that symbol (if available). The use or a more neutral name here 
>> (`SymbolLookup` instead of `LibraryLookup`) is deliberate: not always 
>> a lookup is a result of library loading; lookup objects might in fact 
>> be created (or composed) by users, using existing resources.
>>
>> How do developers obtain a `SymbolLookup` ? Two ways:
>>
>> * They can obtain a `SymbolLookup` for a given class loader - which 
>> allows to search symbols in all the libraries loaded by that class 
>> loader
>> * They can ask `CLinker` a so called *system lookup* - a lookup which 
>> allows to search for basic C symbols (such as `strlen` and `qsort`)
>>
>> Note that the first mechanism allows us to solve the migration 
>> problem, by exposing the lookup abstraction associated with a 
>> specific class loader. This means frameworks will still be able to 
>> call `System::loadLibrary` to *side-effect* the results of a class 
>> loader-based symbol lookup. Note that, since `SymbolLookup` is a 
>> simple functional interface, it would be possible for developers to 
>> set up symbol lookup chains e.g. where lookup for parent class loader 
>> is consulted first (e.g. following classloader delegation). This 
>> flexibility will certainly come in handy in real world use cases.
>>
>> The second mechanism gives us something similar to the previous 
>> concept of *default lookup* - but without the messy bits: the system 
>> lookup is simply a system-dependent symbol lookup object, which might 
>> help in retrieving common C symbols. The API makes no promises as to 
>> *which* libraries will be consulted by this lookup (this is an 
>> implementation detail). That said, the implementation (unlike before) 
>> will not make use of `RTLD_DEFAULT` whose semantics is brittle and 
>> not implementable across all OSs.
>>
>> It is possible that, in the future, we might add more ways to obtain 
>> a symbol lookup - for instance:
>>
>> ```
>> SymbolLookup.ofLibrary(String libName, ResourceScope scope)
>>
>> ```
>>
>> This is not too different from what we had in the original 
>> `LibraryLookup` abstraction, and would allow developers to load a 
>> library and associate its lifecycle with a `ResourceScope` (rather 
>> than a class loader). That is, when the scope is closed, the library 
>> will be unloaded. However, adding these new mode will require some 
>> additional foundational work on the `CLinker` support - as we need to 
>> make sure that the memory address used by a downcall method handle 
>> cannot be unloaded while the downcall method handle is being invoked. 
>> This means that, at the very minimum, the linker will need to 
>> acquire/release the scope associated with the address of the native 
>> function, to prevent premature closing of said scope.
>>
>> Summing up, we believe that while investigating for a more minimal 
>> library lookup mechanism we have found a way to provide most (all?) 
>> of the functionalities available before, in a more disciplined and 
>> compositional manner.
>>
>> Cheers
>> Maurizio
>>
>> [1] - 
>> https://mail.openjdk.java.net/pipermail/panama-dev/2021-April/013577.html
>> [2] - https://git.openjdk.java.net/panama-foreign/pull/526
>> [3] - https://git.openjdk.java.net/panama-foreign/pull/529
>> [4] - 
>> https://github.com/sundararajana/panama-foreign/blob/197db28d5097b1689b4befcd7129eb7af13f41c2/test/jdk/java/foreign/libStdLibTest.c
>> [5] - https://github.com/openjdk/panama-foreign/pull/527
>>
>>
>>