Notes on packaging of native code in libraries

[Maurizio Cimadamore]

On 24/02/2023 14:45, Mike Hearn wrote:
> Especially once Panama ships the Java ecosystem could strongly benefit 
> from a standardization of how native components are bundled into and 
> loaded from libraries, as current build systems and the JVM tooling 
> don't have much to say on the topic. The result is a lot of wheel 
> reinvention across the ecosystem in the form of 
> NativeLibraryLoader.java classes, always unique per project, and a 
> bunch of bugs / developer friction that doesn't need to happen.

I tend to agree with the overall assessment. Shipping native libraries 
with Java projects is a known pain point, and it would be nice to have 
some solution for that. That being said, while I'm aware that the best 
way to make things work in today's world is by shipping native libraries 
in a jar, and then extract them _somewhere_, so that they can be loaded 
with `System::loadLibrary`, I'm not sure how much that can be viewed as 
a full solution, rather than a workaround. I can imagine cases where 
extracting libraries into a custom folder is not feasible (e.g. because 
of missing permissions). My general feeling is that, with jars and 
native libraries it's like trying to fit a round peg in a square hole: 
surely you can devise some pragmatic solution which makes things sort of 
work, but what you get is always a little brittle.

If you look at what we did for jextract [1], the approach we used was 
different: jextract is written entirely in Java, but has a dependency 
(via Foreign Function & Memory API) on libclang. When we build jextract, 
we create a jmod [2] for jextract, with the native library for libclang 
in the right place. We then create a JDK image which contains 
jdk.compiler, java.base and the newly created jextract module. The 
resulting JDK will have the libraries in the right place. This means 
that we can provide a launcher simply by calling jextract's entry point 
using the custom JDK image. You can run jextract and run all jextract 
tests against this custom image, which then requires zero extra custom 
arguments passed on the command line (because the native libraries are 
added in the right place already).

An approach such as this seems more promising than doing heroics with 
jarfiles, at least for applications, and one that could be more amenable 
to things like code signing (in jextract we don't do this, but I don't 
see why that could not be added).

Maurizio

[1] - https://jdk.java.net/jextract/
[2] - https://github.com/openjdk/jextract/blob/master/build.gradle

>
> The good news is that all this would be very cheap to improve. All 
> that's needed is:
>
>   * A defined layout for JARs (or JMODs) that standardizes where to
>     place native libraries given an OS and CPU architecture.
>
>   * Tooling that extracts native code to a user-specified directory
>     that's then appended to the java.library.path at runtime (e.g. a
>     flag to the java launcher?), so that once build systems learn to
>     pass this flag or do the extraction themselves library authors can
>     just deprecate and eventually remove all their custom loader code
>     (which is large, complex, copy/pasted between projects and
>     inconsistent).
>
>   * Support for that mechanism in jlink.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/leyden-dev/attachments/20230224/d591806d/attachment.htm>