Using SymbolLookup#libraryLookup with fallback of SymbolLookup#loaderLookup

[Jorn Vernee]

I want to point out another issue with 
System::load(Library)/java.library.path. It doesn't handle library 
dependencies, which are instead handled automatically by the OS loader. 
This is fine for a JNI library, which can statically link all its 
dependencies to get around this limitation, but not great for existing 
native libraries which jextract would interact with, as they are unaware 
of this limitation.

 > A use cases which is not covered is when you expect that the user 
normally provides the library at an OS-dependent path, but for custom 
use cases allow them to load the library from a custom location.

But this use case is already covered. A user can simply amend the OS 
dependent library search locations. Typically through amending PATH 
(Windows), LD_LIBRARY_PATH (Linux), or DYLD_LIBRARY_PATH (mac), but 
there are other options. This will work for a library's dependencies as 
well, and is more or less as much effort as setting java.library.path, I 
think?

Jorn

On 30-9-2024 11:36, Maurizio Cimadamore wrote:
>
> Hi
>
> On 28/09/2024 15:26, 
> some-java-user-99206970363698485155 at vodafonemail.de wrote:
>
>> Hello,
>>
>> it seems currently the various `--library` options of jextract can be 
>> summarized as:
>>
>>  1. The library is at an OS-dependent library path, and the name is
>>     mapped in an OS-specific way (`--library libname`)
>>  2. The library is at a specific relative or absolute file path
>>     (`--library :filepath`); no OS-specific file name mapping is
>>     performed, so you limit which OS you support, or you have to omit
>>     the `.dll` or `.so` extension to avoid confusion (in case that works)
>>  3. The library is loaded with System#loadLibrary
>>     (`--use-system-load-library`); this is similar to (1.) except
>>     that it also supports `java.library.path` (related: JDK-8311090),
>>     but might have limitations for the dynamic linker (CODETOOLS-7903654)
>>  4. It is assumed that the user has already loaded the library with
>>     System#load or System#loadLibrary (no `--library`)
>>
>>
> Your description of the options is correct. Not sure I see (3) as 
> similar to (1) in the sense that (1) uses the OS to do the library 
> lookup (e.g. OS dynamic linker), wherease (3) is oblivious to all that 
> and uses java.library.path (plus a bunch of hardcoded paths in the VM 
> impl - which are not even relevant in modern Linux distros).
>
>> A use cases which is not covered is when you expect that the user 
>> normally provides the library at an OS-dependent path, but for custom 
>> use cases allow them to load the library from a custom location. The 
>> context of this use case is not where the user generates the code 
>> with jextract themselves, but where they consume some third-party 
>> bindings which include jextract generated code.
>>
> I believe I can see how that might be useful with the tree-sitter 
> library, as you have two components there: the library and the 
> user-defined “plugins” :-)
>
> That said, maybe in such a setting you want to load the main library 
> with (1)/(2) and leave the “plugins” to “loaderLookup” ? Note that 
> “loaderLookup” is always added by jextract to the chain of lookups 
> generated for a given set of bindings. But maybe this is not exactly 
> the use case you have in mind.
>
>> Ideally this would be solved with a user-provided SymbolLookup (see 
>> also [1] and CODETOOLS-7903186), but that might not be that easily 
>> achievable or complicate the API of jextract-based bindings.
>>
> Yes, this is the general solution to this kind of problems.
>
>> A potential solution to address part of the problem might be to have 
>> a (opt-in) combination of the library loading variants mentioned 
>> above: (1.) respectively (3.) with a fallback of (4.) [^2]
>> This is just a rough general idea. However, if System#load / 
>> #loadLibrary is considered legacy and should be avoided, then maybe 
>> this approach should be avoided? What do you think?
>>
> So, let me try to uplevel your observation a bit. The behavior of the 
> various symbol lookup factories is not completely symmetric:
>
>   * SymbolLookup::loaderLookup always succeeds
>   * Linker::defaultLookup always succeeds
>   * Symbol::libraryLookup can fail (e.g. if the specified library
>     cannot be loaded)
>
> Because of the differences in the exception behavior, these things 
> can’t compose nicely if there’s a library load failure. A possible way 
> to address this, would be for jextract to try and wrap a library 
> lookup, so that either:
>
>   * we return the library lookup successfully created
>   * we return a dummy lookup (one that always fails to find symbols),
>     in case creation failed
>
> This would allow jextract to “keep going” even in the face of library 
> loading errors. My question is whether this behavior would be useful: 
> IMHO |--library| is a strong sign to jextract that you know where 
> things are located. So, if things aren’t where you expected, should 
> that issue be ignored? Or should we report the failure to the user? My 
> feeling is that 99% you want that to be an error - e.g. because that 
> will be the result of a wrong name being passed to the jextract 
> command line.
>
> So, while swallowing exceptions is a possibility, I’m not sure we’ll 
> make (all) users a service?
>
> The general solution for this kind of cornery lookup situation is 
> CODETOOLS-7903186 - adding even more library-related flags is surely 
> technically possible, but strikes me as a bad way to get there.
>
> Maurizio
>
> ​
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20241001/476f257a/attachment-0001.htm>