Hermetic Java project meeting

[Jiangli Zhou]

Magnus, thanks for the update! Looks like you've made some good progress.
Please see my comments below.

On Tue, Apr 23, 2024 at 3:42 AM Magnus Ihse Bursie <
magnus.ihse.bursie at oracle.com> wrote:

> I will not be able to participate in the meeting today.
>
> Let me report a bit on my work this week.
>
> I have made a proof of concept branch which can properly compile static
> java on macos with clang and linux with gcc. (With "properly" I mean
> hiding non-exported symbols). I still have problems replicating the
> build with clang on linux. I suspect my linux workstation has a broken
> installation of clang. That machine has grown more erratic over time,
> and I need to reinstall it, but I'm procrastinating spending the time
> doing that... So for now I'm just skipping the clang on linux part.
>

Just to be more clear, that's with using `objcopy` to localize non-exported
symbols for all JDK static libraries and libjvm.a, not just libjvm.a right?

Can you please include the compiler or linker errors on linux/clang?


>
> I have also made great progress on windows. Julian's hint about objcopy
> working fine on COFF, and being present in cygwin, got me to realize
> that this was in effect a possible way forward. My PoC currently manages
> to extract a list of exported symbols from the static libs using
> dumpbin. This is necessary since --localize-hidden does not work on COFF
> (the concept of "hidden" symbols is an ELF-only concept), and instead we
> need to use the --keep-global-symbols option, which (despite the name)
> converts all symbols in the given list to global and all other to local.
> I am currently working actively with getting the next steps done in this
> PoC.
>
> I have initiated talks with SAP, and they are interested in helping out
> getting static linking working on AIX (given that it is possible to do
> with not too much effort.)
>
> I have also tried to extract all the changes (and only the changes)
> related to static build from the hermetic-java-runtime branch (ignoring
> the JavaHome/resource loading changes), to see if I could get something
> like StaticLink.gmk in mainline. I thought I was doing quite fine, but
> after a while I realized my testing was botched since the launcher had
> actually loaded the libraries dynamically instead, even though they were
> statically linked. :-( I am currently trying to bisect my way thought my
> repo to understand where things went wrong.
>

Did you run with `bin/javastatic`? The system automatically detects if the
binary contains statically linked native libraries and avoids loading the
dynamic libraries. Can you please share which test(s) ran into the library
loading issue? I'll see if I can reproduce the problem that you are running
into.

For your work to get StaticLink.gmk into the mainline, I suggest taking all
changes in the hermetic-java-runtime branch for testing to simplify things
on your side. Then you can just extract the makefile part to integrate with
the mainline. As we've discussed in the zoom meeting, I'll work on getting
the runtime changes for static support into the mainline incrementally.


> This problem was exaggerated by the fact that we cannot build *only*
> static libs, so the dynamic ones where there alongside to be
> (improperly) picked up. I might want to spend some time on fixing this
> first. That will help both with speeding up the build/test cycle for
> static builds, and help avoid that kind of issue repeating. That will
> require some more refactoring in the core build/link code though.
>
> Experimenting with the static launcher on linux/gcc and macos made me
> realize that we will need to know the set of external libraries needed
> by each individual JDK library. When building a dynamic library, this
> knowledge (e.g. -liconv -framework Application) is encoded into the
> lib*.so file by the linker. But not so for a static library. Instead, we
> need to store this information for each JDK library, and then in the
> end, when we want to pick up all static libraries and link them together
> to form the "javastatic" executable, we need to pass this set of
> external libraries to the linker.
>
> This was done haphazardly in StaticLink.gmk in the hermetic-java-runtime
> branch, where an arbitrary subset of external libraries were hard-coded.
> Before integration in mainline can be possible, this information needs
> to be collected correctly and automatically for all included JDK
> libraries. Fortunately, it is not likely to be too hard. I basically
> just need to store the information from the LIBS provided to the
> NativeCompilation, and pick that up for all static libraries we include
> in the static launcher. (A complication is that we need to de-duplicate
> the list, and that some libraries are specified using two words, like
> "-framework Application" on macos, so it will take some care getting it
> right.)
>

Right, currently the hermetic-java-runtime branch specifies a list of
hard-coded dependency libraries for linking. One of the goals of the
hermetic prototype was avoiding/reducing refactoring/restructuring the
existing code whenever possible. The reason is to reduce merge overhead
when integrating with new changes from the mainline. We can do the proper
refactoring and cleanups when getting the changes into the mainline.


>
> I have also been thinking about possible ways that we can share compiled
> obj files between static and dynamic libraries, even if we cannot do it
> fully. Most files do not need the STATIC_BUILD define and will thus be
> identical for both static and dynamic builds. It might be possible to
> just hard-code the exact files that needs to be different. It's ugly,
> and I still would like to make sure we press forward with the spec
> changes to JNI/JVMTI, but it would work as a stop-gap measure.
>

We have only briefly touched on the spec change topic (for the naming of
native libraries) during the zoom meetings. I also agree that we should get
that part started now. It's unclear to me if there's any existing blocker
for that.

Best,
Jiangli


>
> /Magnus
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/leyden-dev/attachments/20240424/613a18e7/attachment.htm>