Using SymbolLookup#libraryLookup with fallback of SymbolLookup#loaderLookup

[Maurizio Cimadamore]

Hi

On 28/09/2024 15:26, some-java-user-99206970363698485155 at vodafonemail.de 
wrote:

> Hello,
>
> it seems currently the various `--library` options of jextract can be 
> summarized as:
>
>  1. The library is at an OS-dependent library path, and the name is
>     mapped in an OS-specific way (`--library libname`)
>  2. The library is at a specific relative or absolute file path
>     (`--library :filepath`); no OS-specific file name mapping is
>     performed, so you limit which OS you support, or you have to omit
>     the `.dll` or `.so` extension to avoid confusion (in case that works)
>  3. The library is loaded with System#loadLibrary
>     (`--use-system-load-library`); this is similar to (1.) except that
>     it also supports `java.library.path` (related: JDK-8311090), but
>     might have limitations for the dynamic linker (CODETOOLS-7903654)
>  4. It is assumed that the user has already loaded the library with
>     System#load or System#loadLibrary (no `--library`)
>
>
Your description of the options is correct. Not sure I see (3) as 
similar to (1) in the sense that (1) uses the OS to do the library 
lookup (e.g. OS dynamic linker), wherease (3) is oblivious to all that 
and uses java.library.path (plus a bunch of hardcoded paths in the VM 
impl - which are not even relevant in modern Linux distros).

> A use cases which is not covered is when you expect that the user 
> normally provides the library at an OS-dependent path, but for custom 
> use cases allow them to load the library from a custom location. The 
> context of this use case is not where the user generates the code with 
> jextract themselves, but where they consume some third-party bindings 
> which include jextract generated code.
>
I believe I can see how that might be useful with the tree-sitter 
library, as you have two components there: the library and the 
user-defined “plugins” :-)

That said, maybe in such a setting you want to load the main library 
with (1)/(2) and leave the “plugins” to “loaderLookup” ? Note that 
“loaderLookup” is always added by jextract to the chain of lookups 
generated for a given set of bindings. But maybe this is not exactly the 
use case you have in mind.

> Ideally this would be solved with a user-provided SymbolLookup (see 
> also [1] and CODETOOLS-7903186), but that might not be that easily 
> achievable or complicate the API of jextract-based bindings.
>
Yes, this is the general solution to this kind of problems.

> A potential solution to address part of the problem might be to have a 
> (opt-in) combination of the library loading variants mentioned above: 
> (1.) respectively (3.) with a fallback of (4.) [^2]
> This is just a rough general idea. However, if System#load / 
> #loadLibrary is considered legacy and should be avoided, then maybe 
> this approach should be avoided? What do you think?
>
So, let me try to uplevel your observation a bit. The behavior of the 
various symbol lookup factories is not completely symmetric:

  * SymbolLookup::loaderLookup always succeeds
  * Linker::defaultLookup always succeeds
  * Symbol::libraryLookup can fail (e.g. if the specified library cannot
    be loaded)

Because of the differences in the exception behavior, these things can’t 
compose nicely if there’s a library load failure. A possible way to 
address this, would be for jextract to try and wrap a library lookup, so 
that either:

  * we return the library lookup successfully created
  * we return a dummy lookup (one that always fails to find symbols), in
    case creation failed

This would allow jextract to “keep going” even in the face of library 
loading errors. My question is whether this behavior would be useful: 
IMHO |--library| is a strong sign to jextract that you know where things 
are located. So, if things aren’t where you expected, should that issue 
be ignored? Or should we report the failure to the user? My feeling is 
that 99% you want that to be an error - e.g. because that will be the 
result of a wrong name being passed to the jextract command line.

So, while swallowing exceptions is a possibility, I’m not sure we’ll 
make (all) users a service?

The general solution for this kind of cornery lookup situation is 
CODETOOLS-7903186 - adding even more library-related flags is surely 
technically possible, but strikes me as a bad way to get there.

Maurizio

​
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20240930/94ebd233/attachment-0001.htm>