Hermetic Java (static image packaging/formatting) investigation and proposal

[Jiangli Zhou]

Hi Magnus,

Thanks for the thoughts! Please see comments inlined below.

On Mon, Feb 13, 2023 at 5:43 AM Magnus Ihse Bursie <
magnus.ihse.bursie at oracle.com> wrote:

> Hi Jiangli,
>
> On 2023-02-08 03:08, Jiangli Zhou wrote:
> > Hi Brian,
> >
> > Here are the main buckets of the changes discovered in JDK/VM to
> > support the proposed hermetic image:
> >
> > 1) Resolve symbol conflicts to fully support JDK static builds. Those
> > are mainly caused by duplicated symbols defined in different native
> > libraries or VM code.
> >
> > 2) Complete the built-in native library support in JDK. For easier and
> > more reliable testing/release/deployment, we wanted to support JDK
> > dynamic and static builds with the same set of object files (.o).
> > We've changed to use unique names for
> > JNI_OnLoad|JNI_OnUnload|Agent_OnLoad|Agent_OnUnload|Agent_OnAttach in
> > different JDK JNI libraries by default. For both dynamic linked and
> > static linked JDK builds, we use unique symbols for JNI_OnLoad
> > function and friends. However, non-builtin application JNI libraries
> > can still have the default JNI_OnLoad|... naming. We still properly
> > support application JNI libraries using the default JNI_OnLoad (and
> > friends) naming.
> >
> > As we wanted to produce dynamic and static builds from the same set of
> > object files, we've moved away from using the STATIC_BUILD macro.
> >
> > We've also done some makefile work to build both dynamic shared
> > libraries (DSOs) and static libraries, within one JDK build.
>
> This sounds like interesting work indeed. However, I am inclined to
> agree with Andrew and wonder how much it relates to Project Leyden. It
> might be that Leyden will need some kind of packaging story, and that
> this can have a role to play in that. But it is not immediately clear
> that it does fit in, and indeed, I think this is not one of Leyden main
> problem areas at the time.
>
> But your code sounds very much interesting from a pure build
> perspective! For at least this part of the code, I think you should
> ignore Leyden for now, and just see if the static build changes you have
> made could be fit for inclusion in OpenJDK.
>
> The static build part of the build system has been sadly neglected due
> to resource limitations, for a long time. :( The rudimentary system
> (actually, more like two separate systems) we have was put in place
> mostly due to external requirements from Project Mobile and the Graal
> integration, and was tacked on mostly as an after-thought. It is not
> regularly tested, and I'd frankly be surprised if it actually works
> right now. So I fully understand if you have been staying away from
> STATIC_BUILD. :)
>
> It sounds like you have created a more dynamic system to be able to
> select per library, if it should be compiled statically or dynamically.
> Do I understand you correctly? If done correctly, it can probably help
> bring a better abstraction to the build process.
>

For JDK/hotspot natives, our experiment/prototype builds all the JDK
regular artifacts (e.g. lib/.../*.so) that the existing JDK build produces.
Additionally, it also creates the JDK static libraries (e.g.
lib_static/*.a) and a bin/javastatic (with most of the needed JDK static
libraries statically linked into the launcher, for testing purposes only,
such as running jtreg tests) in the binary, as part of the single JDK
build. The hermetic image creation is done as a post process, which takes
the needed pre-built JDK static libraries for linking into the final
executable. The post process is done outside the JDK build. The JDK runtime
support is enhanced to be able to support both built-in libraries and
dynamically loaded shared libraries.

It doesn't seem to be problematic to get the JDK static support into
OpenJDK first. It's especially helpful for you or erikj@ to look at the
makefiles changes and help massage the changes according to the JDK build
standard.


>
> If you are willing to contribute your work to OpenJDK, I would
> definitely be interested in studying it in detail.


Thanks!


> As you might be
> aware, contributions to OpenJDK must be done on the OpenJDK
> infrastructure. One way to do this is to create a branch in the sandbox
> repo[1], and push your changes there.
>

Will get back to you on this, after some explorations on open sourcing the
changes.


>
> If it turns out to be of use for Project Leyden, all the better if it is
> already in place. And if it turns out that this is orthogonal to Project
> Leyden, I still think a cleanup in this area might be beneficial for all
> of the JDK.
>

All sounds good!

Best,
Jiangli


>
> /Magnus
>
> [1] https://github.com/openjdk/jdk-sandbox
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/leyden-dev/attachments/20230213/01541088/attachment.htm>