library loading - continued

[Ty Young]

Someone off-list pointed out that dl* functions are provided by a 
different library than libc: dl.


Also, does System.load()/System.loadLibrary() affect 
CLinker.systemLookup()? Just noticed that CLinker.systemLookup()'s name 
is rather problematic if it doesn't.



On 5/10/21 5:25 PM, Ty Young wrote:
> It's not possible to create bindings for dlfcn, the header that 
> provides the function declarations dlopen, dlclose, dlsym, etc, it 
> seems. CLinker.systemLookup() fails to provide the symbol, anyway, and 
> attempting to load libc fails. The header says that it's part of the 
> GNU C library but maybe I'm barking up the wrong tree or something.
>
>
> Is System.load()/System.loadLibrary() really the best approach? Not 
> only does it have issues loading libc(and others?) but it isn't 
> exactly all that primitive of a API either. Both dlopen(Linux) and 
> LoadLibrary(Windows) both return pointers(which can be user managed) 
> when loading libraries and dlopen has modes that can be set, but 
> System.load()/System.loadLibrary() both register the loaded libraries 
> with a ClassLoader, a JVM construct. How does one get a pointer of a 
> loaded library from a ClassLoader and then use that pointer to unload 
> a library?
>
>
> In addition to all that, it's no longer clear(compared to the old way 
> Panama did things) where a particular symbol is coming from. Yes, we 
> know it's from a ClassLoader(if you bother reading javadoc, which 
> doesn't work on Netbeans) but there isn't a visual representation of 
> this in code.
>
>
> On 5/10/21 7:49 AM, Maurizio Cimadamore wrote:
>> Hi,
>> in my email [1] I described a plan to simplify the library loading 
>> mechanism based on the `LibraryLookup` abstraction. The motivations 
>> behind the restructuring were three-fold:
>>
>> * Semantics of `LibraryLookup::ofDefault` is unstable; on some 
>> platforms, default lookup leaks other loaded library symbols; on 
>> other platforms (e.g. Windows) implementing it required use of 
>> debugging APIs.
>> * `LibraryLookup` has its own concept of lifecycle which is 
>> incompatible with `ResourceScope`, and also with that typically 
>> associated with JNI loaded libraries (whose lifecycle is associated 
>> with that of the owning classloader)
>> * `LibraryLookup` does not offer a migration path for JNI-based 
>> frameworks. That is, many frameworks are written with the assumption 
>> that `System::loadLibrary` will affect the set of available symbols 
>> _everywhere_ else (at least within the same classloader context)
>>
>> For these reasons, we decided to remove `LibraryLookup` and to 
>> provide a more basic lookup capability instead, as a static method 
>> (`CLinker::findNative`). This method would lookup symbols in 
>> libraries loaded by the classloader used by the caller; in doing so, 
>> the new method addresses the migration issues which affected 
>> `LibraryLookup`.
>>
>> When writing this patch [2, 3] we have realized that this new scheme 
>> was not without its own issues:
>>
>> * Loading symbols in the standard C library is now incredibly 
>> difficult, sometimes requiring surprising workarounds [4, 5]
>> * There is no way to e.g. implement a lookup which filters the set of 
>> symbols available to users
>>
>> After staring at these issues, we realized that having a lookup 
>> abstraction was not, in itself, the root cause of the problems we 
>> were seeing in [1]. These problems were, rather, caused by the fact 
>> that the lookup abstraction we had was introducing its own concept of 
>> library loading which had nothing to do with JNI library loading. In 
>> other words, `LibraryLookup`, despite its name, wasn't a *pure* 
>> lookup abstraction, it was attempting to do a little more (e.g. 
>> keeping libraries loaded using GC reachability), without too much 
>> success.
>>
>> Having realized that, we now believe we can safely reintroduce some 
>> kind of lookup abstraction, in the form of `SymbolLookup`: a simple 
>> functional interface which, given a symbol name, gives us the address 
>> of that symbol (if available). The use or a more neutral name here 
>> (`SymbolLookup` instead of `LibraryLookup`) is deliberate: not always 
>> a lookup is a result of library loading; lookup objects might in fact 
>> be created (or composed) by users, using existing resources.
>>
>> How do developers obtain a `SymbolLookup` ? Two ways:
>>
>> * They can obtain a `SymbolLookup` for a given class loader - which 
>> allows to search symbols in all the libraries loaded by that class 
>> loader
>> * They can ask `CLinker` a so called *system lookup* - a lookup which 
>> allows to search for basic C symbols (such as `strlen` and `qsort`)
>>
>> Note that the first mechanism allows us to solve the migration 
>> problem, by exposing the lookup abstraction associated with a 
>> specific class loader. This means frameworks will still be able to 
>> call `System::loadLibrary` to *side-effect* the results of a class 
>> loader-based symbol lookup. Note that, since `SymbolLookup` is a 
>> simple functional interface, it would be possible for developers to 
>> set up symbol lookup chains e.g. where lookup for parent class loader 
>> is consulted first (e.g. following classloader delegation). This 
>> flexibility will certainly come in handy in real world use cases.
>>
>> The second mechanism gives us something similar to the previous 
>> concept of *default lookup* - but without the messy bits: the system 
>> lookup is simply a system-dependent symbol lookup object, which might 
>> help in retrieving common C symbols. The API makes no promises as to 
>> *which* libraries will be consulted by this lookup (this is an 
>> implementation detail). That said, the implementation (unlike before) 
>> will not make use of `RTLD_DEFAULT` whose semantics is brittle and 
>> not implementable across all OSs.
>>
>> It is possible that, in the future, we might add more ways to obtain 
>> a symbol lookup - for instance:
>>
>> ```
>> SymbolLookup.ofLibrary(String libName, ResourceScope scope)
>>
>> ```
>>
>> This is not too different from what we had in the original 
>> `LibraryLookup` abstraction, and would allow developers to load a 
>> library and associate its lifecycle with a `ResourceScope` (rather 
>> than a class loader). That is, when the scope is closed, the library 
>> will be unloaded. However, adding these new mode will require some 
>> additional foundational work on the `CLinker` support - as we need to 
>> make sure that the memory address used by a downcall method handle 
>> cannot be unloaded while the downcall method handle is being invoked. 
>> This means that, at the very minimum, the linker will need to 
>> acquire/release the scope associated with the address of the native 
>> function, to prevent premature closing of said scope.
>>
>> Summing up, we believe that while investigating for a more minimal 
>> library lookup mechanism we have found a way to provide most (all?) 
>> of the functionalities available before, in a more disciplined and 
>> compositional manner.
>>
>> Cheers
>> Maurizio
>>
>> [1] - 
>> https://mail.openjdk.java.net/pipermail/panama-dev/2021-April/013577.html
>> [2] - https://git.openjdk.java.net/panama-foreign/pull/526
>> [3] - https://git.openjdk.java.net/panama-foreign/pull/529
>> [4] - 
>> https://github.com/sundararajana/panama-foreign/blob/197db28d5097b1689b4befcd7129eb7af13f41c2/test/jdk/java/foreign/libStdLibTest.c
>> [5] - https://github.com/openjdk/panama-foreign/pull/527
>>
>>
>>