Questions about the Hermetic Java project

David Holmes david.holmes at oracle.com
Mon Jun 2 02:55:36 UTC 2025 

On 31/05/2025 7:20 am, Jiangli Zhou wrote:
> On Thu, May 29, 2025 at 11:54 PM David Holmes <david.holmes at oracle.com> wrote:
>>
>> On 30/05/2025 9:26 am, Jiangli Zhou wrote:
>>>
>>> I just thought of one more thing related to the discussion now. Any
>>> concern if the implementation does not ignore JNI_OnLoad_L and etc if
>>> they are defined application's dynamically linked native libraries? Or
>>> that's unspecified behavior and it's up to the implement to decide?
>>
>> For Internal libraries or external? For external you have to follow the
>> spec - if both methods exist you only want to execute one of them.
> 
> It's for the external (non-JDK) library that I'm a bit more cautious.
> 
> In the existing code in JDK mainline,
> https://github.com/openjdk/jdk/blob/3cc630985d47be6ba4cf991698e999f17dbde203/src/java.base/share/classes/jdk/internal/loader/NativeLibraries.java#L117,
> loadLibrary() first tries to find the built-in library using
> JNI_OnLoad_L symbol (L is the library name). When dlsym is called to
> find the symbol from the main process, any of the already loaded
> shared libraries are also searched, as described by the dlsym man page
> (included related part below).
> 
> https://man7.org/linux/man-pages/man3/dlsym.3.html:
>         RTLD_DEFAULT
>                Find the first occurrence of the desired symbol using the
>                default shared object search order.  The search will
>                include global symbols in the executable and its
>                dependencies, as well as symbols in shared objects that
>                were dynamically loaded with the RTLD_GLOBAL flag.
> 
> I think it would be rare, it is possible to construct such case:
> 
> There are user JNI libraries A and B, with B is built as a dependency
> of A. A defines JNI_OnLoad_A and JNI_OnLoad. B defines JNI_OnLoad_B
> and JNI_OnLoad. When A is being loaded using loadLibrary(),
> loadLibrary() tries first to lookup JNI_OnLoad_A, which is not found.
> A is then loaded dynamically, which causes B being loaded implicitly
> as a dependency of A. Later when loadLibrary() is called for B,
> JNI_OnLoad_B would be found and then called. This is an existing
> behavior. I think it's an unspecified behavior and we don't need to
> add any additional checks to prevent JNI_OnLoad_B from being called.

That sounds like a significant design flaw to me. You can't specify that 
JNI_OnLoad_L will only be called if L is statically linked, if the 
existence of JNI_OnLoad_L is used to infer that L is statically linked! 
I would expect libraries to have both versions of the OnLoad functions 
to allow for them being statically or dynamically linked - which the 
spec allows for by saying the alternate variant is ignored. But then the 
JDK will execute the wrong method if it finds JNI_OnLoad_L in a 
dynamically linked library.

David
-----

> Thanks,
> Jiangli
> 
>>
>> David
>> -----
>>
>>> Thanks!
>>> Jiangli
>>>>
>>>> David
>>>>
>>>>> Best,
>>>>> Jiangli
>>>>>
>>>>>
>>>>>>
>>>>>> David
>>>>>> -----
>>>>>>
>>>>>>
>>>>>>>     From JNI spec [3]:
>>>>>>>
>>>>>>> - JNI_OnLoad/JNI_OnUnload
>>>>>>>       Optional function defined by dynamically linked libraries.
>>>>>>>
>>>>>>>       LINKAGE:
>>>>>>>       Exported from dynamically linked native libraries that contain
>>>>>>> native method implementations.
>>>>>>>
>>>>>>> - JNI_OnLoad_L
>>>>>>>       Mandatory function that must be defined by statically linked libraries .
>>>>>>>
>>>>>>>       LINKAGE:
>>>>>>>       Exported from statically linked native libraries that contain native
>>>>>>> method implementations.
>>>>>>>
>>>>>>> - JNI_OnUnload_L
>>>>>>>       Optional function defined by statically linked libraries.
>>>>>>>
>>>>>>>     From JVMTI spec [4]:
>>>>>>>
>>>>>>> An agent L whose image has been combined with the VM is defined as
>>>>>>> statically linked if and only if the agent exports a function called
>>>>>>> Agent_OnLoad_L.
>>>>>>>
>>>>>>> [1]: https://bugs.openjdk.org/browse/JDK-8350450
>>>>>>> [2]: https://github.com/openjdk/leyden/tree/hermetic-java-runtime
>>>>>>> [3]: https://docs.oracle.com/en/java/javase/21/docs/specs/jni/
>>>>>>> [4]: https://docs.oracle.com/en/java/javase/24/docs/specs/jvmti.html
>>>>>>>
>>>>>>> Best,
>>>>>>> Jiangli
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> And finally, on top of all of this, is the question of widening the platform support. To support linux/gcc with objcopy is trivial, but the question about Windows still remain. I have two possible ways forward, one is to check if there is alternative tooling to use (the prime candidate is the clang-ldd), and the other is to try to "fake" a partial linking by concatenating all source code before compiling. This is not ideal, though, for many reasons, and I am not keen on implementing it, not even for testing. And at this point, I have not had time to investigate any of these options much further, since I have been focusing on 1) above.
>>>>>>>>>
>>>>>>>>> A third option is of course to just say that due to toolchain limitations, static linking is not available on Windows.
>>>>>>>>
>>>>>>>> Thank you for taking this on! Potentially we could consider taking the
>>>>>>>> objcopy to localizing hotspot symbols on unix-like platforms, based on
>>>>>>>> https://github.com/openjdk/jdk/pull/17456 discussions. Additional
>>>>>>>> testing is still needed to verify the solution.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> My recommendation is that you keep on working to resolve the (much more thorny) issues of resource access in Hermetic Java in your branch, where you have a prototype static build that works for you. In the meantime, I will make sure that there will be a functioning, stable and robust way of creating static builds in the mainline, that can be regularly tested and not bit-rot, like the static build hacks that has gone in before.
>>>>>>>>
>>>>>>>> Most of the JDK resources are now supported as hermetic jimage
>>>>>>>> (lib/modules) bundled in the
>>>>>>>> https://github.com/openjdk/leyden/tree/hermetic-java-runtime branch.
>>>>>>>> The remaining sound.properties, ct.sym and .jfc files can be handled
>>>>>>>> later. Overally, that part of the work has confirmed the hermetic
>>>>>>>> jimage bundled solution is robust and helps resolve some of the
>>>>>>>> difficult start-up sequence issues observed when the hermetic resource
>>>>>>>> was implemented using JAR file based solution.
>>>>>>>>
>>>>>>>> It might be a good idea to follow up on the static linking discussion
>>>>>>>> in tomorrow's zoom meeting (hope you'll be able to join tomorrow).
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> Jiangli
>>>>>>>>>
>>>>>>>>> /Magnus
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>> Jiangli
>>>>>>>>>
>>>>>>>>> On Thu, Feb 15, 2024 at 12:01 PM Jiangli Zhou <jianglizhou at google.com> wrote:
>>>>>>>>>>
>>>>>>>>>> On Wed, Feb 14, 2024 at 5:07 PM Jiangli Zhou <jianglizhou at google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Magnus,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for looking into this from the build perspective.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Feb 14, 2024 at 1:00 AM Magnus Ihse Bursie
>>>>>>>>>>> <magnus.ihse.bursie at oracle.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> First some background for build-dev: I have spent some time looking at
>>>>>>>>>>>> the build implications of the Hermetic Java effort, which is part of
>>>>>>>>>>>> Project Leyden. A high-level overview is available here:
>>>>>>>>>>>> https://cr.openjdk.org/~jiangli/hermetic_java.pdf and the current source
>>>>>>>>>>>> code is here: https://github.com/openjdk/leyden/tree/hermetic-java-runtime.
>>>>>>>>>>>
>>>>>>>>>>> Some additional hermetic Java related references that are also useful:
>>>>>>>>>>>
>>>>>>>>>>> - https://bugs.openjdk.org/browse/JDK-8303796 is an umbrella bug that
>>>>>>>>>>> links to the issues for resolving static linking issues so far
>>>>>>>>>>> - https://github.com/openjdk/jdk21/pull/26 is the enhancement for
>>>>>>>>>>> building the complete set of static libraries in JDK/VM, particularly
>>>>>>>>>>> including libjvm.a
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hermetic Java faces several challenges, but the part that is relevant
>>>>>>>>>>>> for the build system is the ability to create static libraries. We've
>>>>>>>>>>>> had this functionality (in three different ways...) for some time, but
>>>>>>>>>>>> it is rather badly implemented.
>>>>>>>>>>>>
>>>>>>>>>>>> As a result of my investigations, I have a bunch of questions. :-) I
>>>>>>>>>>>> have gotten some answers in private discussion, but for the sake of
>>>>>>>>>>>> transparency I will repeat them here, to foster an open dialogue.
>>>>>>>>>>>>
>>>>>>>>>>>> 1. Am I correct in understanding that the ultimate goal of this exercise
>>>>>>>>>>>> is to be able to have jmods which include static libraries (*.a) of the
>>>>>>>>>>>> native code which the module uses, and that the user can then run a
>>>>>>>>>>>> special jlink command to have this linked into a single executable
>>>>>>>>>>>> binary (which also bundles the *.class files and any additional
>>>>>>>>>>>> resources needed)?
>>>>>>>>>>>>
>>>>>>>>>>>> 2. If so, is the idea to create special kinds of static jmods, like
>>>>>>>>>>>> java.base-static.jmod, that contains *.a files instead of lib*.so files?
>>>>>>>>>>>> Or is the idea that the normal jmod should contain both?
>>>>>>>>>>>>
>>>>>>>>>>>> 3. Linking .o and .a files into an executable is a formidable task. Is
>>>>>>>>>>>> the intention to have jlink call a system-provided ld, or to bundle ld
>>>>>>>>>>>> with jlink, or to reimplement this functionality in Java?
>>>>>>>>>>>
>>>>>>>>>>> I have a similar view as Alan responded in your other email thread.
>>>>>>>>>>> Things are still in the early stage for the general solution.
>>>>>>>>>>>
>>>>>>>>>>> In the https://github.com/openjdk/leyden/tree/hermetic-java-runtime
>>>>>>>>>>> branch, when configuring JDK with --with-static-java=yes, the JDK
>>>>>>>>>>> binary contains the following extra artifacts:
>>>>>>>>>>>
>>>>>>>>>>> - static-libs/*.a: The complete set of JDK/VM static libraries
>>>>>>>>>>> - jdk/bin/javastatic: A demo Java launcher fully statically linked
>>>>>>>>>>> with the selected JDK .a libraries (e.g. it currently statically link
>>>>>>>>>>> with the headless) and libjvm.a. It's the standard Java launcher
>>>>>>>>>>> without additional work for hermetic Java.
>>>>>>>>>>>
>>>>>>>>>>> In our prototype for hermetic Java, we build the hermetic executable
>>>>>>>>>>> image (a single image) from the following input (see description on
>>>>>>>>>>> singlejar packaging tool in
>>>>>>>>>>> https://cr.openjdk.org/~jiangli/hermetic_java.pdf):
>>>>>>>>>>>
>>>>>>>>>>> - A customized launcher (with additional work for hermetic) executable
>>>>>>>>>>> fully statically linked with JDK/VM static libraries (.a files),
>>>>>>>>>>> application natives and dependencies (e.g. in .a static libraries)
>>>>>>>>>>> - JDK lib/modules, JDK resource files
>>>>>>>>>>> - Application classes and resource files
>>>>>>>>>>>
>>>>>>>>>>> Including a JDK library .a into the corresponding .jmod would require
>>>>>>>>>>> extracting the .a for linking with the executable. In some systems
>>>>>>>>>>> that may cause memory overhead due to the extracted copy of the .a
>>>>>>>>>>> files. I think we should consider the memory overhead issue.
>>>>>>>>>>>
>>>>>>>>>>> One possibility (as Alan described in his response) is for jlink to
>>>>>>>>>>> invoke the ld on the build system. jlink could pass the needed JDK
>>>>>>>>>>> static libraries and libjvm.a (provided as part of the JDK binary) to
>>>>>>>>>>> ld based on the modules required for the application.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I gave a bit more thoughts on this one. For jlink to trigger ld, it
>>>>>>>>>> would need to know the complete linker options and inputs. Those
>>>>>>>>>> include options and inputs related to the application part as well. In
>>>>>>>>>> some usages, it might be easier to handle native linking separately
>>>>>>>>>> and pass the linker output, the executable to jlink directly. Maybe we
>>>>>>>>>> could consider supporting different modes for various usages
>>>>>>>>>> requirements, from static libraries and native linking point of view:
>>>>>>>>>>
>>>>>>>>>> Mode #1
>>>>>>>>>> Support .jmod packaged natives static libraries, for both JDK/VM .a
>>>>>>>>>> and application natives and dependencies. If the inputs to jlink
>>>>>>>>>> include .jmods, jlink can extract the .a libraries and pass the
>>>>>>>>>> information to ld to link the executable.
>>>>>>>>>>
>>>>>>>>>> Mode #2
>>>>>>>>>> Support separate .a as jlink input. Jlink could pass the path
>>>>>>>>>> information to the .a libraries and other linker options to ld to
>>>>>>>>>> create the executable.
>>>>>>>>>>
>>>>>>>>>> For both mode #1 and #2, jlink would then use the linker output
>>>>>>>>>> executable to create the final hermetic image.
>>>>>>>>>>
>>>>>>>>>> Mode #3
>>>>>>>>>> Support a fully linked executable as a jlink input. When a linked
>>>>>>>>>> executable is given to jlink, it can process it directly with other
>>>>>>>>>> JDK data/files to create the final image, without native linking step.
>>>>>>>>>>
>>>>>>>>>> Any other thoughts and considerations?
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Jiangli
>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 4. Is the intention is to allow users to create their own jmods with
>>>>>>>>>>>> static libraries, and have these linked in as well? This seems to be the
>>>>>>>>>>>> case.
>>>>>>>>>>>
>>>>>>>>>>> An alternative with less memory overhead could be using application
>>>>>>>>>>> modular JAR and separate .a as the input for jlink.
>>>>>>>>>>>
>>>>>>>>>>>> If that is so, then there will always be the risk for name
>>>>>>>>>>>> collisions, and we can only minimize the risk by making sure any global
>>>>>>>>>>>> names are as unique as possible.
>>>>>>>>>>>
>>>>>>>>>>> Part of the current effort includes resolving the discovered symbol
>>>>>>>>>>> collision issues with static linking. Will respond to your other email
>>>>>>>>>>> on the symbol issue separately later.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 5. The original implementation of static builds in the JDK, created for
>>>>>>>>>>>> the Mobile project, used a configure flag, --enable-static-builds, to
>>>>>>>>>>>> change the entire behavior of the build system to only produce *.a files
>>>>>>>>>>>> instead of lib*.so. In contrast, the current system is using a special
>>>>>>>>>>>> target instead.
>>>>>>>>>>>
>>>>>>>>>>> I think we would need both configure flag and special target for the
>>>>>>>>>>> static builds.
>>>>>>>>>>>
>>>>>>>>>>>> In my eyes, this is a much worse solution. Apart from
>>>>>>>>>>>> the conceptual principle (if the build should generate static or dynamic
>>>>>>>>>>>> libraries is definitely a property of what a "configuration" means),
>>>>>>>>>>>> this makes it much harder to implement efficiently, since we cannot make
>>>>>>>>>>>> changes in NativeCompilation.gmk, where they are needed.
>>>>>>>>>>>
>>>>>>>>>>> For the potential objcopy work to resolve symbol issues, we can add
>>>>>>>>>>> that conditionally in NativeCompilation.gmk if STATIC_LIBS is true. We
>>>>>>>>>>> have an internal prototype (not included in
>>>>>>>>>>> https://github.com/openjdk/leyden/tree/hermetic-java-runtime yet) done
>>>>>>>>>>> by one of colleagues for localizing symbols in libfreetype using
>>>>>>>>>>> objcopy.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> That was not as much a question as a statement. 🙂 But here is the
>>>>>>>>>>>> question: Do you think it would be reasonable to restore the old
>>>>>>>>>>>> behavior but with the new methods, so that we don't use special targets,
>>>>>>>>>>>> but instead tells configure to generate static libraries? I'm thinking
>>>>>>>>>>>> we should have a flag like "--with-library-type=" that can have values
>>>>>>>>>>>> "dynamic" (which is default), "static" or "both".
>>>>>>>>>>>
>>>>>>>>>>> If we want to also build a fully statically linked launcher, maybe
>>>>>>>>>>> --with-static-java? Being able to configure either dynamic, static or
>>>>>>>>>>> both as you suggested also seems to be a good idea.
>>>>>>>>>>>
>>>>>>>>>>>> I am not sure if "both" are needed, but if we want to bundle both lib*.so and *.a files
>>>>>>>>>>>> into a single jmod file (see question 2 above), then it definitely is.
>>>>>>>>>>>> In general, the cost of producing two kinds of libraries are quite
>>>>>>>>>>>> small, compared to the cost of compiling the source code to object files.
>>>>>>>>>>>
>>>>>>>>>>> Completely agree. It would be good to avoid recompiling the .o file
>>>>>>>>>>> for static and dynamic builds. As proposed in
>>>>>>>>>>> https://bugs.openjdk.org/browse/JDK-8303796:
>>>>>>>>>>>
>>>>>>>>>>> It's beneficial to be able to build both .so and .a from the same set
>>>>>>>>>>> of .o files. That would involve some changes to handle the dynamic JDK
>>>>>>>>>>> and static JDK difference at runtime, instead of relying on the
>>>>>>>>>>> STATIC_BUILD macro.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Finally, I have looked at how to manipulate symbol visibility. There
>>>>>>>>>>>> seems many ways forward, so I feel confident that we can find a good
>>>>>>>>>>>> solution.
>>>>>>>>>>>>
>>>>>>>>>>>> One way forward is to use objcopy to manipulate symbol status
>>>>>>>>>>>> (global/local). There is an option --localize-symbol in objcopy, that
>>>>>>>>>>>> has been available in objcopy since at least 2.15, which was released
>>>>>>>>>>>> 2004, so it should be safe to use. But ideally we should avoid using
>>>>>>>>>>>> objcopy and do this as part of the linking process. This should be
>>>>>>>>>>>> possible to do, given that we make changes in NativeCompilation.gmk --
>>>>>>>>>>>> see question 5 above.
>>>>>>>>>>>>
>>>>>>>>>>>> As a fallback, it is also possible to rename symbols, either piecewise
>>>>>>>>>>>> or wholesale, using objcopy. There are many ways to do this, using
>>>>>>>>>>>> --prefix-symbols, --redefine-sym or --redefine-syms (note the -s, this
>>>>>>>>>>>> takes a file with a list of symbols). Thus we can always introduce a
>>>>>>>>>>>> "post factum namespace" by renaming symbols.
>>>>>>>>>>>
>>>>>>>>>>> Renaming or redefining the symbol at build time could cause confusions
>>>>>>>>>>> with debugging. That's a concern raised in
>>>>>>>>>>> https://github.com/openjdk/jdk/pull/17456 discussions.
>>>>>>>>>>>
>>>>>>>>>>> Additionally, redefining symbols using tools like objcopy may not
>>>>>>>>>>> handle member names referenced in string literals. For example, in
>>>>>>>>>>> https://github.com/openjdk/jdk/pull/17456 additional changes are
>>>>>>>>>>> needed in assembling and SA to reflect the symbol change.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> So in the end, I think it will be fully possible to produce .a files
>>>>>>>>>>>> that only has global symbols for the functions that are part of the API
>>>>>>>>>>>> exposed by that library, and have all other symbols local, and make this
>>>>>>>>>>>> is in a way that is consistent with the rest of the build system.
>>>>>>>>>>>>
>>>>>>>>>>>> Finally, a note on Hotspot. Due to debugging reasons, we export
>>>>>>>>>>>> basically all symbols in hotspot as global. This is not reasonable to do
>>>>>>>>>>>> for a static build. The effect of not exporting those symbols will be
>>>>>>>>>>>> that SA will not function to 100%. On the other hand, I have no idea if
>>>>>>>>>>>> SA works at all with a static build. Have you tested this? Is this part
>>>>>>>>>>>> of the plan to support, or will it be officially dropped for Hermetic Java?
>>>>>>>>>>>
>>>>>>>>>>> We have done some testing with jtreg SA related tests for the fully
>>>>>>>>>>> statically linked `javastatic`.
>>>>>>>>>>>
>>>>>>>>>>> If we use objcopy to localize symbols in hotspot, it's not yet clear
>>>>>>>>>>> what's the impact on SA. We could do some tests. The other question
>>>>>>>>>>> that I raised is the supported gcc versions (for partial linking)
>>>>>>>>>>> related to the solution.
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Jiangli
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> /Magnus
>>>>>>>>>>>>
>>>>>>
>>>>
>>

More information about the leyden-dev mailing list