Hermetic Java (static image packaging/formatting) investigation and proposal

[Magnus Ihse Bursie]

On 2023-02-14 00:52, Jiangli Zhou wrote:

> Hi Magnus,
>
> Thanks for the thoughts! Please see comments inlined below.
>
> On Mon, Feb 13, 2023 at 5:43 AM Magnus Ihse Bursie 
> <magnus.ihse.bursie at oracle.com> wrote:
>
>     Hi Jiangli,
>
>     On 2023-02-08 03:08, Jiangli Zhou wrote:
>     > Hi Brian,
>     >
>     > Here are the main buckets of the changes discovered in JDK/VM to
>     > support the proposed hermetic image:
>     >
>     > 1) Resolve symbol conflicts to fully support JDK static builds.
>     Those
>     > are mainly caused by duplicated symbols defined in different native
>     > libraries or VM code.
>     >
>     > 2) Complete the built-in native library support in JDK. For
>     easier and
>     > more reliable testing/release/deployment, we wanted to support JDK
>     > dynamic and static builds with the same set of object files (.o).
>     > We've changed to use unique names for
>     >
>     JNI_OnLoad|JNI_OnUnload|Agent_OnLoad|Agent_OnUnload|Agent_OnAttach in
>     > different JDK JNI libraries by default. For both dynamic linked and
>     > static linked JDK builds, we use unique symbols for JNI_OnLoad
>     > function and friends. However, non-builtin application JNI libraries
>     > can still have the default JNI_OnLoad|... naming. We still properly
>     > support application JNI libraries using the default JNI_OnLoad (and
>     > friends) naming.
>     >
>     > As we wanted to produce dynamic and static builds from the same
>     set of
>     > object files, we've moved away from using the STATIC_BUILD macro.
>     >
>     > We've also done some makefile work to build both dynamic shared
>     > libraries (DSOs) and static libraries, within one JDK build.
>
>     This sounds like interesting work indeed. However, I am inclined to
>     agree with Andrew and wonder how much it relates to Project
>     Leyden. It
>     might be that Leyden will need some kind of packaging story, and that
>     this can have a role to play in that. But it is not immediately clear
>     that it does fit in, and indeed, I think this is not one of Leyden
>     main
>     problem areas at the time.
>
>     But your code sounds very much interesting from a pure build
>     perspective! For at least this part of the code, I think you should
>     ignore Leyden for now, and just see if the static build changes
>     you have
>     made could be fit for inclusion in OpenJDK.
>
>     The static build part of the build system has been sadly neglected
>     due
>     to resource limitations, for a long time. :( The rudimentary system
>     (actually, more like two separate systems) we have was put in place
>     mostly due to external requirements from Project Mobile and the Graal
>     integration, and was tacked on mostly as an after-thought. It is not
>     regularly tested, and I'd frankly be surprised if it actually works
>     right now. So I fully understand if you have been staying away from
>     STATIC_BUILD. :)
>
>     It sounds like you have created a more dynamic system to be able to
>     select per library, if it should be compiled statically or
>     dynamically.
>     Do I understand you correctly? If done correctly, it can probably
>     help
>     bring a better abstraction to the build process.
>
>
> For JDK/hotspot natives, our experiment/prototype builds all the JDK 
> regular artifacts (e.g. lib/.../*.so) that the existing JDK build 
> produces. Additionally, it also creates the JDK static libraries (e.g. 
> lib_static/*.a) and a bin/javastatic (with most of the needed JDK 
> static libraries statically linked into the launcher, for testing 
> purposes only, such as running jtreg tests) in the binary, as part of 
> the single JDK build. The hermetic image creation is done as a post 
> process, which takes the needed pre-built JDK static libraries for 
> linking into the final executable. The post process is done outside 
> the JDK build. The JDK runtime support is enhanced to be able to 
> support both built-in libraries and dynamically loaded shared libraries.
>
> It doesn't seem to be problematic to get the JDK static support into 
> OpenJDK first. It's especially helpful for you or erikj@ to look at 
> the makefiles changes and help massage the changes according to the 
> JDK build standard.
>
>
>     If you are willing to contribute your work to OpenJDK, I would
>     definitely be interested in studying it in detail. 
>
>
> Thanks!
>
>     As you might be
>     aware, contributions to OpenJDK must be done on the OpenJDK
>     infrastructure. One way to do this is to create a branch in the
>     sandbox
>     repo[1], and push your changes there.
>
>
> Will get back to you on this, after some explorations on open sourcing 
> the changes.

Let me know when you are done with that process.

/Magnus

>
>     If it turns out to be of use for Project Leyden, all the better if
>     it is
>     already in place. And if it turns out that this is orthogonal to
>     Project
>     Leyden, I still think a cleanup in this area might be beneficial
>     for all
>     of the JDK.
>
>
> All sounds good!
>
> Best,
> Jiangli
>
>
>     /Magnus
>
>     [1] https://github.com/openjdk/jdk-sandbox
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/build-dev/attachments/20230214/3e783c3d/attachment.htm>