Hermetic Java (static image packaging/formatting) investigation and proposal

[Jiangli Zhou]

Hi Magnus,

Follow up on your suggestion regarding contributing the minor JDK static
build fixes/enhancements (branched from the hermetic Java discussion [1]).
Since JEP 178 [2] has covered specification related issues already, we can
create individual bugs/enhancements and PRs for the related changes on an
as-needed basis. I've created JDK-8303796 [3] as a starting one. Will work
with you and others on contributing our changes. Thank you!

[1] -
https://mail.openjdk.org/pipermail/leyden-dev/2023-February/000119.html
[2] - https://openjdk.org/jeps/178
[3] - https://bugs.openjdk.org/browse/JDK-8303796

Best,
Jiangli

On Tue, Feb 14, 2023 at 12:55 AM Magnus Ihse Bursie <
magnus.ihse.bursie at oracle.com> wrote:

> On 2023-02-14 00:52, Jiangli Zhou wrote:
>
> Hi Magnus,
>
> Thanks for the thoughts! Please see comments inlined below.
>
> On Mon, Feb 13, 2023 at 5:43 AM Magnus Ihse Bursie <
> magnus.ihse.bursie at oracle.com> wrote:
>
>> Hi Jiangli,
>>
>> On 2023-02-08 03:08, Jiangli Zhou wrote:
>> > Hi Brian,
>> >
>> > Here are the main buckets of the changes discovered in JDK/VM to
>> > support the proposed hermetic image:
>> >
>> > 1) Resolve symbol conflicts to fully support JDK static builds. Those
>> > are mainly caused by duplicated symbols defined in different native
>> > libraries or VM code.
>> >
>> > 2) Complete the built-in native library support in JDK. For easier and
>> > more reliable testing/release/deployment, we wanted to support JDK
>> > dynamic and static builds with the same set of object files (.o).
>> > We've changed to use unique names for
>> > JNI_OnLoad|JNI_OnUnload|Agent_OnLoad|Agent_OnUnload|Agent_OnAttach in
>> > different JDK JNI libraries by default. For both dynamic linked and
>> > static linked JDK builds, we use unique symbols for JNI_OnLoad
>> > function and friends. However, non-builtin application JNI libraries
>> > can still have the default JNI_OnLoad|... naming. We still properly
>> > support application JNI libraries using the default JNI_OnLoad (and
>> > friends) naming.
>> >
>> > As we wanted to produce dynamic and static builds from the same set of
>> > object files, we've moved away from using the STATIC_BUILD macro.
>> >
>> > We've also done some makefile work to build both dynamic shared
>> > libraries (DSOs) and static libraries, within one JDK build.
>>
>> This sounds like interesting work indeed. However, I am inclined to
>> agree with Andrew and wonder how much it relates to Project Leyden. It
>> might be that Leyden will need some kind of packaging story, and that
>> this can have a role to play in that. But it is not immediately clear
>> that it does fit in, and indeed, I think this is not one of Leyden main
>> problem areas at the time.
>>
>> But your code sounds very much interesting from a pure build
>> perspective! For at least this part of the code, I think you should
>> ignore Leyden for now, and just see if the static build changes you have
>> made could be fit for inclusion in OpenJDK.
>>
>> The static build part of the build system has been sadly neglected due
>> to resource limitations, for a long time. :( The rudimentary system
>> (actually, more like two separate systems) we have was put in place
>> mostly due to external requirements from Project Mobile and the Graal
>> integration, and was tacked on mostly as an after-thought. It is not
>> regularly tested, and I'd frankly be surprised if it actually works
>> right now. So I fully understand if you have been staying away from
>> STATIC_BUILD. :)
>>
>> It sounds like you have created a more dynamic system to be able to
>> select per library, if it should be compiled statically or dynamically.
>> Do I understand you correctly? If done correctly, it can probably help
>> bring a better abstraction to the build process.
>>
>
> For JDK/hotspot natives, our experiment/prototype builds all the JDK
> regular artifacts (e.g. lib/.../*.so) that the existing JDK build produces.
> Additionally, it also creates the JDK static libraries (e.g.
> lib_static/*.a) and a bin/javastatic (with most of the needed JDK static
> libraries statically linked into the launcher, for testing purposes only,
> such as running jtreg tests) in the binary, as part of the single JDK
> build. The hermetic image creation is done as a post process, which takes
> the needed pre-built JDK static libraries for linking into the final
> executable. The post process is done outside the JDK build. The JDK runtime
> support is enhanced to be able to support both built-in libraries and
> dynamically loaded shared libraries.
>
> It doesn't seem to be problematic to get the JDK static support into
> OpenJDK first. It's especially helpful for you or erikj@ to look at the
> makefiles changes and help massage the changes according to the JDK build
> standard.
>
>
>>
>> If you are willing to contribute your work to OpenJDK, I would
>> definitely be interested in studying it in detail.
>
>
> Thanks!
>
>
>> As you might be
>> aware, contributions to OpenJDK must be done on the OpenJDK
>> infrastructure. One way to do this is to create a branch in the sandbox
>> repo[1], and push your changes there.
>>
>
> Will get back to you on this, after some explorations on open sourcing the
> changes.
>
> Let me know when you are done with that process.
>
> /Magnus
>
>
>
>>
>> If it turns out to be of use for Project Leyden, all the better if it is
>> already in place. And if it turns out that this is orthogonal to Project
>> Leyden, I still think a cleanup in this area might be beneficial for all
>> of the JDK.
>>
>
> All sounds good!
>
> Best,
> Jiangli
>
>
>>
>> /Magnus
>>
>> [1] https://github.com/openjdk/jdk-sandbox
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/build-dev/attachments/20230307/4fbfff95/attachment.htm>