Notes on packaging of native code in libraries

[Mike Hearn]

Thanks! Yes creating JMODs is one way to do it, but if you were to
distribute jextract as a library and not just an application you'd get user
complaints because it'd take more work to consume:

   - JMODs are platform specific but build tools don't make it easy to
   select the right artifacts based on platforms. You can do it but it often
   requires custom Maven/Gradle plugins and such.

   - You can't put JMODs on the module path so users would now need to run
   jlink to get a JDK they could use, but build tools don't invoke jlink
   during the normal development cycle and don't easily support switching
   which JDK they use half way through based on the output of build tasks
   (maybe they should but last time I tried, they don't). You could write
   extra plugins to teach them to do that maybe, but then you'd hit
   performance problems - jlinking is a fairly heavy and slow operation, and
   doesn't cache anything. All this complexity is why some vendor JDKs
   pre-jlink JavaFX.

   - It requires libraries to be modular and jlinkable. Sadly this is often
   quite challenging :( e.g. out of the box a vanilla Spring Boot Web app
   can't be linked because the module graph is invalid. The Spring guys know
   and don't plan to fix it. See
   https://github.com/spring-projects/spring-boot/issues/33942

So this runs into a fundamental question that Jigsaw never really resolved
and maybe Leyden needs to: is jlink meant to be a last step optimization
process you run before distribution (today, yes) or is it meant to be an
integrated part of the compile-and-test cycle? If it's the former you can
get bugs that only appear at distribution time. If it's the latter then it
needs a different performance and compatibility model. GraalVM Native Image
faces the same problem: it's slow to compile and introduces new bugs, so
it's tough to integrate it into the standard development and testing
process. Ordinary HotSpot with a classpath/module path is so great for
development because changing things about your app is just so darn fast,
users are loath to lose that. Hence the proliferation of libraries that
extract code on the fly. It's inelegant and creates other problems but it
preserves the ultra-fast build loops that people love so much, and doesn't
require special support from build systems.

Sponsored Link: maybe jextract should be distributed with Conveyor ;) It'd
be convenient to have installs that can keep themselves up to date, added
to the path automatically etc.


On Fri, 24 Feb 2023 at 16:09, Maurizio Cimadamore <
maurizio.cimadamore at oracle.com> wrote:

>
> On 24/02/2023 14:45, Mike Hearn wrote:
>
> Especially once Panama ships the Java ecosystem could strongly benefit
> from a standardization of how native components are bundled into and loaded
> from libraries, as current build systems and the JVM tooling don't have
> much to say on the topic. The result is a lot of wheel reinvention across
> the ecosystem in the form of NativeLibraryLoader.java classes, always
> unique per project, and a bunch of bugs / developer friction that doesn't
> need to happen.
>
> I tend to agree with the overall assessment. Shipping native libraries
> with Java projects is a known pain point, and it would be nice to have some
> solution for that. That being said, while I'm aware that the best way to
> make things work in today's world is by shipping native libraries in a jar,
> and then extract them _somewhere_, so that they can be loaded with
> `System::loadLibrary`, I'm not sure how much that can be viewed as a full
> solution, rather than a workaround. I can imagine cases where extracting
> libraries into a custom folder is not feasible (e.g. because of missing
> permissions). My general feeling is that, with jars and native libraries
> it's like trying to fit a round peg in a square hole: surely you can devise
> some pragmatic solution which makes things sort of work, but what you get
> is always a little brittle.
>
> If you look at what we did for jextract [1], the approach we used was
> different: jextract is written entirely in Java, but has a dependency (via
> Foreign Function & Memory API) on libclang. When we build jextract, we
> create a jmod [2] for jextract, with the native library for libclang in the
> right place. We then create a JDK image which contains jdk.compiler,
> java.base and the newly created jextract module. The resulting JDK will
> have the libraries in the right place. This means that we can provide a
> launcher simply by calling jextract's entry point using the custom JDK
> image. You can run jextract and run all jextract tests against this custom
> image, which then requires zero extra custom arguments passed on the
> command line (because the native libraries are added in the right place
> already).
>
> An approach such as this seems more promising than doing heroics with
> jarfiles, at least for applications, and one that could be more amenable to
> things like code signing (in jextract we don't do this, but I don't see why
> that could not be added).
>
> Maurizio
>
> [1] - https://jdk.java.net/jextract/
> [2] - https://github.com/openjdk/jextract/blob/master/build.gradle
>
>
> The good news is that all this would be very cheap to improve. All that's
> needed is:
>
>    - A defined layout for JARs (or JMODs) that standardizes where to
>    place native libraries given an OS and CPU architecture.
>
>    - Tooling that extracts native code to a user-specified directory
>    that's then appended to the java.library.path at runtime (e.g. a flag to
>    the java launcher?), so that once build systems learn to pass this flag or
>    do the extraction themselves library authors can just deprecate and
>    eventually remove all their custom loader code (which is large, complex,
>    copy/pasted between projects and inconsistent).
>
>    - Support for that mechanism in jlink.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/leyden-dev/attachments/20230224/8afb8531/attachment.htm>