February 2023 - leyden-dev

Notes on packaging of native code in libraries
by Mike Hearn 24 Feb '23

24 Feb '23

Hello, Recently there was a discussion of hermetic Java and how native libraries are packaged in Google's solution. The approach is Linux specific but once you leave that OS new issues emerge. We've just published a blog post that explores the problems that arise when distributing Java apps to platforms that require code signing: https://hydraulic.software/blog/11-in-jar-signing.html Conveyor is a build tool that layers on top of jlink and it does extensive work to turn JARs into something that can be shipped to Mac/Windows users successfully. The blog post discusses some of these challenges, which any approach to static Java will also face (assuming it's supported on more than just Linux of course). The core issues are related to code signing, and the practice of extracting libraries on the fly from JARs to caches or temp directories. Conveyor has two modes, one which does this extraction ahead of time (but many libraries don't expect this and must be explicitly configured via system properties, or they just break), and the other which code signs libraries inside the JARs. The latter is less likely to break apps but yields other problems. Neither approach is perfect. Especially once Panama ships the Java ecosystem could strongly benefit from a standardization of how native components are bundled into and loaded from libraries, as current build systems and the JVM tooling don't have much to say on the topic. The result is a lot of wheel reinvention across the ecosystem in the form of NativeLibraryLoader.java classes, always unique per project, and a bunch of bugs / developer friction that doesn't need to happen. The good news is that all this would be very cheap to improve. All that's needed is: - A defined layout for JARs (or JMODs) that standardizes where to place native libraries given an OS and CPU architecture. - Tooling that extracts native code to a user-specified directory that's then appended to the java.library.path at runtime (e.g. a flag to the java launcher?), so that once build systems learn to pass this flag or do the extraction themselves library authors can just deprecate and eventually remove all their custom loader code (which is large, complex, copy/pasted between projects and inconsistent). - Support for that mechanism in jlink. The current JMOD mechanism improved things over the prior situation, but unfortunately doesn't quite supply everything needed. A solution that works with JARs and supports multiple libraries in one JAR would be easily adopted.

2 3

Hermetic Java (static image packaging/formatting) investigation and proposal
by Jiangli Zhou 20 Feb '23

20 Feb '23

Hi, During the last one and a half years, Google has done some extensive research on linux-x64 with Java static image, as project Hermetic Java [1 <http://cr.openjdk.java.net/~jiangli/hermetic_java.pdf>]. We would like to share our experiences/results with the community and present our approach for discussion under the Leyden project. We hope to contribute the work to OpenJDK through project Leyden, via the JEP [2 <http://cr.openjdk.java.net/~mr/jep/jep-2.0-02.html>] process as needed. With Hermetic Java, our main goal is to create a single executable image including the Java runtime environment, Java application and the dependencies. This addresses some real-world Java deployment issues and challenges that we have encountered over the years. We believe it fits very well with the overall goal of project Leyden in the following aspects: - Provide a build-time created static image derived from an application and JDK; Image executes as a standalone program. - Satisfy closed-world constraints. - Is built on top of OpenJDK and can utilize existing OpenJDK components including the Hotspot VM, runtime JIT compiler (C1, C2), CDS, etc. Our focus has been on the image packaging and formatting part. This works roughly as follows: 1. The executable image (see slide <http://cr.openjdk.java.net/~jiangli/hermetic_java.pdf> #10 of [1 <http://cr.openjdk.java.net/~jiangli/hermetic_java.pdf>]) consists of three sections: the ELF executable section (see slide <http://cr.openjdk.java.net/~jiangli/hermetic_java.pdf> #14), the JDK runtime section (see slide <http://cr.openjdk.java.net/~jiangli/hermetic_java.pdf> #20, #21) and the JAR section (see slide <http://cr.openjdk.java.net/~jiangli/hermetic_java.pdf> #22). The ELF section is at the beginning of the image and contains the Java launcher executable, which allows the image to work as a native executable. The JDK runtime section contains the JDK lib/modules image starting at a page-aligned file offset. This section can include other data that requires special alignment, such as the CDS archive. The JAR section holds the Java application classes, dependent library classes, and resources. JDK runtime resource files, such as java.security and java.policy are also packaged within the JAR section. 1. The Java launcher executable is statically linked with Hotspot/JDK natives and application JNI natives (see slide <http://cr.openjdk.java.net/~jiangli/hermetic_java.pdf> #15 - #18). For static native library support, we enhance and complete existing OpenJDK work [3 <https://bugs.openjdk.org/browse/JDK-8005716>, 4 <https://bugs.openjdk.org/browse/JDK-8136556>, 5 <https://bugs.openjdk.org/browse/JDK-8232748>]. It provides a flexible solution for loading built-in (static) native libraries while still allowing dynamically loading shared JNI libraries (if desired). 1. With a single executable image, we define the image file path as the java.home (see slide <http://cr.openjdk.java.net/~jiangli/hermetic_java.pdf> #23). A JavaHome class is used to provide uniform APIs for accessing JDK resources in traditional and Hermetic Java (single image) execution modes. Hermetic Java is an accumulation of wisdom that Google obtained from real-world production deployments over many years (years before the current project research/experiments). We would love to gather feedback from community members. Any input and feedback are welcome and appreciated! We are happy to provide additional information and answer questions (open to discussions in any form). [1] http://cr.openjdk.java.net/~jiangli/hermetic_java.pdf [2] http://cr.openjdk.java.net/~mr/jep/jep-2.0-02.html [3] https://bugs.openjdk.org/browse/JDK-8005716 [4] https://bugs.openjdk.org/browse/JDK-8136556 [5] https://bugs.openjdk.org/browse/JDK-8232748 Best regards, Jiangli

7 24

File Format Investigation: jimage
by Severin Gehwolf 17 Feb '23

17 Feb '23

Hi, We have been looking at lib/modules (a.k.a jimage) as a potential candidate format for Project Leyden. Along the way, I've documented how the file is structured[1] and used in its current form. The document discussing jimage in the context of Leyden is here: https://cr.openjdk.org/~sgehwolf/leyden/jimage_file_format_investigation_le… Looking forward to your feedback! Thanks, Severin [1] https://cr.openjdk.org/~sgehwolf/leyden/jimage_visualization_1.png

4 7