Questions about the Hermetic Java project

Tue Apr 16 15:30:04 UTC 2024

Jiangli,

First of all: I tried building the leyden branch "hermetic-java-runtime" 
but failed. :-( I tried some various attempts to work around the 
failures, but the javastatic file I ended up with were unable to start.

What kind of environment are you using to build this branch? (OS, 
compiler, libc version, etc) Do you have any special configure flags 
that are required?

I would very much like to test it out myself to check this static java 
launcher that you are creating.

On 2024-04-16 03:01, Jiangli Zhou wrote:

> The main purpose of StaticLink.gmk is to support the static-java-image
> make target, which can be used to perform the actual static linking
> step using libjvm.a and JDK static libraries. That currently doesn't
> exist in the JDK mainline. Creating a "fully" statically linked Java
> launcher is the first step (out of many) towards supporting
> static/hermetic Java.

I am still not sure it is the best design to have this in a separate 
file. You are in some way "just" creating a new launcher.

> As part of cleaning/refactoring/integrating for the static linking
> step, we want to agree and decide/accept on the following:
>
> - Support the "fully" statically linked java launcher for testing and
> demoing the capability of static JDK support, e.g.
>    - Support running jtreg testing using the "fully" statically linked
> Java launcher

So how do you want to test native jtreg tests? By compiling the jtreg 
tests to native .a files and statically link with those as well, or to 
make this a hybrid mode where a statically linked launcher loads the 
jtreg tests dynamically?

>    - Set up tests in github workflow to help detect any breaking
> changes for static support, e.g. new symbol issues introduced by any
> changes. There were some earlier discussions on this with Ron and Alan
> during the zoom meetings.

If we do this correct, we should not have to worry about symbol 
conflicts. (Local symbols should all be hidden before creation of the 
static libraries.) That said, creating static libraries needs to be 
regularly tested or it will break.

> - Which JDK native libraries to be statically linked with the new
> launcher target? E.g. StaticLink.gmk currently excludes libjsound.a,
> libawt_xawt.a, etc from statically linked with the launcher.
That sounds rather arbitrarily. I would assume that a statically linked 
launcher should include native libraries from all included modules. If 
you are not interested in java.desktop native libraries, then you'd need 
to exclude java.desktop from the build. But my assumption here is that 
we need to build a solution that works with all modules included in the 
JDK, and that is what must be tested in GHA.
> - Do we want more than one statically linked launcher target, based on
> the set of linked native libraries?

Do you mean that you want like "static-java-java.base-only", 
"static-java-eveything-but-java.desktop" etc? That does not sound like 
something that belongs in the makefiles. If you want to build a JDK that 
can only support a certain subset of modules, you need to specify those 
module in your build scripts as input to configure.

> (II) What missing:
> - Static linking step as mentioned above
>
> (III) What needs to be improved (require cleanups and refactoring, and
> you mentioned some of those in your response as well):
> - Support building both the static libraries and dynamic libraries
> using the same set of .o files, instead of separately compiled .o
> files. That helps improve build speed and reduce memory overhead for
> building JDK. Your current refactoring work aims to help that.
> - Clean up the usages of STATIC_BUILD macro. Most of the usages are in
> test code.
> - Other runtime fixes/enhancements in the leyden
> https://github.com/openjdk/leyden/tree/hermetic-java-runtime  branch
>
> I think most work mentioned in III has dependencies on II. We need a
> workable base to be able to build the "fully" statically linked
> launcher for building and testing the work mentioned in III, when
> integrating any of those to the JDK mainline. The makefile refactoring
> work can be done in parallel but does not need to be completed before
> we add the static linking step in JDK mainline.

I'm not sure I understand why you consider (II) to be a prerequisite for 
the (III) issues. From my point of view, they are all mutually 
independent changes that needs to be done, and if anything, I think 
getting in code to create a statically linked launcher will probably be 
easier if we get further along on the other issues.

In particular, I'd like to see changes to the STATIC_BUILD macro come in 
first. Ideally, we should not send in such a macro at all. The 
JNI_OnLoad stuff is bugging me mostly.

> As of today, the leyden
> https://github.com/openjdk/leyden/tree/hermetic-java-runtime  branch
> can build a "fully" statically linked Java launcher. The issue of
> compiling the dynamic and static libraries .o files separately is not
> a blocker. It's good to have it resolved at some point of time.

Weeeell, yes and no... It is not *technically* a blocker, but unless we 
can reuse the .o files, including static builds in the standard testing 
on GHA will effectively double the build time, and that is definitely 
not acceptable. And the alternative is to bring in static builds without 
having it regularly tested, which I don't think is acceptable either.

So the conclusion is that getting static and dynamic libraries built 
from the same .o files is in effect a blocker for Hermetic Java.

>> This, in turn, require several changes:
>>
>> 1) The linking code needs to be cleaned up, and all technical debt needs to be resolved. This is what I have been doing since I started working on static builds for Hermetic Java. JDK-8329704 (which was integrated yesterday) was the first major milestone of this cleanup. Now, the path were to find a library created by the JDK (static or dynamic) is encapsulated in ResolveLibPath. This is currently a monster, but at least all knowledge is collected in a single location, instead of spread over the code base. Getting this simplified is the next step.
>>
>> 2) We need to stop passing the STATIC_BUILD define when compiling. This is partially addressed in your PR, where you have replaced #ifdef STATIC_BUILD with a dynamic lookup. But there is also the problem of JNI/JVMTI entry points. I have been pondering how we can compile the code in a way so we support both dynamic and static name resolution, and I think I have a solution.
>>
>> This is unfortunately quite complex, and I have started a discussion with Alan if it is possible to update the JNI spec so that both static and dynamic entry points can have the form "JNI_OnLoad_<library-name>". Ideally, I'd like to see us push for this with as much effort as possible. If we got this in place, static builds would be much easier, and the changes required for Hermetic Java even smaller.
> Thumbs up! That seems to be a good direction. Currently in the leyden
> branch, it first looks up the unique
> JNI_OnLoad<_lib_name>|Agent_OnLoad<_lib_name> etc for built-in
> libraries, then search for the dynamic libraries using the
> conventional naming when necessary. e.g.:
>
> https://github.com/openjdk/leyden/commit/a5c886d2e85a0ff0c3712a5488ae61d8c9d7ba1a
> https://github.com/openjdk/leyden/commit/1da8e3240e0bd27366d19f2e7dde386e46015135
>
> When spec supports JNI_OnLoad_<library-name> and etc. for dynamic
> libraries, we may still need to support the conventional naming
> without the <_lib_name> part for existing libraries out there.

That looks interesting. Basically, this is very similar to what I 
imagine needs to be done in the mainline. However, I don't think we can 
just change this code like that without a CSR? Or are we allowed to 
treat JDK-internal libraries different than the specification states?

I would say getting this part into mainline should be the main focus 
right now, not the StaticLink.gmk file. And that includes getting any 
CSR approvals etc.

> Thank you for taking this on! Potentially we could consider taking the
> objcopy to localizing hotspot symbols on unix-like platforms, based on
> https://github.com/openjdk/jdk/pull/17456  discussions. Additional
> testing is still needed to verify the solution.
We don't need any additional testing on that; proof of concept shows it 
works. It just needs to be implemented. With JDK-8330261 (pushed today), 
the ground is now prepared to get it done. It is next on my todo-list.
> It might be a good idea to follow up on the static linking discussion
> in tomorrow's zoom meeting (hope you'll be able to join tomorrow).

I'll join but might be somewhat late due to family commitments.

/Magnus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/leyden-dev/attachments/20240416/dc85a939/attachment.htm>