Providing your own SymbolLookup / ServiceLoader for SymbolLookup
Hello, first of all thank you for your work on the FFM API and jextract! It is really great that interoperability with native libraries becomes easier. One issue I currently have with jextract is that none of its `--library` or `--use-system-load-library` options seem to offer a way for users to provide their own SymbolLookup. Why is this an issue? - The header class generated by jextract and possibly additional bindings classes might be generated by a third-party project, so you cannot easily modify the source code to change the SymbolLookup creation (unless you want to fork that project). - For convenience Java projects often bundle native libraries, extract them to a temporary file and then load them, for example even JavaFX does this (see its `NativeLibLoader` class). This avoids requiring users to configure `java.library.path`. There are probably advantages and disadvantages with this approach, but nonetheless it would be useful if developers had the choice, by providing their own SymbolLookup and specifying the path of the library. - `SymbolLookup#libraryLookup` seems to be currently the only API which allows manually unloading the library (by closing the `Arena`). All other approaches either keep the library loaded until JVM exit, or until the classloader is garbage collected which might be JVM exit as well (?). Especially on Windows this is a problem because it prevents the deletion of the file while the library is loaded, so `File#deleteOnExit` or similar won't have any effect. Ideally a solution would put the choice of how the library is loaded in the hands of the consumers of the generated headers, and not the users of jextract (since they might be a different person, e.g. if a third-party project provides bindings). Otherwise if this is something users of jextract have to manually enable, then most of them will likely not do it. Therefore maybe one approach could be to offer some kind of `SymbolLookupProvider` which is used with a ServiceLoader: ``` interface SymbolLookupProvider { Optional<SymbolLookup> getSymbolLookup(Class<?> caller, String libName); } ``` The `caller` here is whoever calls `SymbolLookup#libraryLookup(String, Arena)`. This might help if someone is unlucky enough to have multiple dependencies which expect a different library of the same name. If possible it would be useful if `System#loadLibrary` could use SymbolLookupProvider as well. And only in case no SymbolLookupProvider provided a SymbolLookup, the existing default logic is used. This would be a bit similar to the existing `ClassLoader#findLibrary`, except that using that method for this purpose seems difficult (you need a custom class loading setup) respectively impossible (when `SymbolLookup#libraryLookup` is used). There are a few open questions though: - `SymbolLookup#libraryLookup(String, Arena)` already provides an Arena. Should that be ignored if a SymbolLookupProvider can provide a lookup, or should the SymbolLookupProvider have the choice to use the given Arena or use its own. But then the guarantee of `SymbolLookup#libraryLookup` that closing the Arena unloads the library might no longer hold. - `System#loadLibrary` seems to not use SymbolLookup internally at the moment, so supporting that would not be easily possible probably. What do you think? Do you have any other ideas for solving the current issues with jextract mentioned above? Was there already an discussion about this or a similar issue in the past? (I tried to search the jextract archives but could not find anything) Kind regards (Maybe the proposal here is not specific to jextract and applies to the FFM API in general. Please let me know if I should forward this mail to panama-dev.)
On 20/08/2024 23:42, some-java-user-99206970363698485155@vodafonemail.de wrote:
Hello,
first of all thank you for your work on the FFM API and jextract! It is really great that interoperability with native libraries becomes easier.
One issue I currently have with jextract is that none of its `--library` or `--use-system-load-library` options seem to offer a way for users to provide their own SymbolLookup. Why is this an issue? - The header class generated by jextract and possibly additional bindings classes might be generated by a third-party project, so you cannot easily modify the source code to change the SymbolLookup creation (unless you want to fork that project). - For convenience Java projects often bundle native libraries, extract them to a temporary file and then load them, for example even JavaFX does this (see its `NativeLibLoader` class). This avoids requiring users to configure `java.library.path`. There are probably advantages and disadvantages with this approach, but nonetheless it would be useful if developers had the choice, by providing their own SymbolLookup and specifying the path of the library. - `SymbolLookup#libraryLookup` seems to be currently the only API which allows manually unloading the library (by closing the `Arena`). All other approaches either keep the library loaded until JVM exit, or until the classloader is garbage collected which might be JVM exit as well (?). Especially on Windows this is a problem because it prevents the deletion of the file while the library is loaded, so `File#deleteOnExit` or similar won't have any effect.
Well, jextract generates code. If you have your own lookup, jextract cannot know what code to generate. One option would be to use a service loader (as you suggest), but this would require a jextract runtime library, which we don’t want (unless the service is in the JDK, which we also don’t want). An option that often works in these cases is to NOT use any —library option. Jextract will generate nothing, but will use the loader lookup. So you are free to “influence” what’s in the loader lookup as you wish (e.g. by calling System::loadLibrary). This tend to work for native libraries distributed via maven inside jars (but with the limitation that you cannot unload). Another option, which we are considering for the long term, is to change the bindings so that jextract will generate a “default” instantiation of a binding, but then you can also use the provided constructor to specify your own lookup. Something like this: |class OpenGL { int GL_VERSION(); // this is an _instance method_ ... // other bindings OpenGL(SymbolLookup lookup) { ... } static final OpenGL LIB = new OpenGL(/* lookup selected from command line option */); } | This gives you bindings that are “ready to use”, pretty much as today - but in addition, you can create your own “LIB” instance with whatever lookup you want. I believe this is where we want to go (and this doesn’t suffer from the pesky problems you described with the service provider interface having to decide in which arena to work). Maurizio
participants (2)
-
Maurizio Cimadamore
-
some-java-user-99206970363698485155@vodafonemail.de