February 2024 - leyden-dev

Questions about the Hermetic Java project
by Magnus Ihse Bursie 09 Jun '25

09 Jun '25

First some background for build-dev: I have spent some time looking at the build implications of the Hermetic Java effort, which is part of Project Leyden. A high-level overview is available here: https://cr.openjdk.org/~jiangli/hermetic_java.pdf and the current source code is here: https://github.com/openjdk/leyden/tree/hermetic-java-runtime. Hermetic Java faces several challenges, but the part that is relevant for the build system is the ability to create static libraries. We've had this functionality (in three different ways...) for some time, but it is rather badly implemented. As a result of my investigations, I have a bunch of questions. :-) I have gotten some answers in private discussion, but for the sake of transparency I will repeat them here, to foster an open dialogue. 1. Am I correct in understanding that the ultimate goal of this exercise is to be able to have jmods which include static libraries (*.a) of the native code which the module uses, and that the user can then run a special jlink command to have this linked into a single executable binary (which also bundles the *.class files and any additional resources needed)? 2. If so, is the idea to create special kinds of static jmods, like java.base-static.jmod, that contains *.a files instead of lib*.so files? Or is the idea that the normal jmod should contain both? 3. Linking .o and .a files into an executable is a formidable task. Is the intention to have jlink call a system-provided ld, or to bundle ld with jlink, or to reimplement this functionality in Java? 4. Is the intention is to allow users to create their own jmods with static libraries, and have these linked in as well? This seems to be the case. If that is so, then there will always be the risk for name collisions, and we can only minimize the risk by making sure any global names are as unique as possible. 5. The original implementation of static builds in the JDK, created for the Mobile project, used a configure flag, --enable-static-builds, to change the entire behavior of the build system to only produce *.a files instead of lib*.so. In contrast, the current system is using a special target instead. In my eyes, this is a much worse solution. Apart from the conceptual principle (if the build should generate static or dynamic libraries is definitely a property of what a "configuration" means), this makes it much harder to implement efficiently, since we cannot make changes in NativeCompilation.gmk, where they are needed. That was not as much a question as a statement. 🙂 But here is the question: Do you think it would be reasonable to restore the old behavior but with the new methods, so that we don't use special targets, but instead tells configure to generate static libraries? I'm thinking we should have a flag like "--with-library-type=" that can have values "dynamic" (which is default), "static" or "both". I am not sure if "both" are needed, but if we want to bundle both lib*.so and *.a files into a single jmod file (see question 2 above), then it definitely is. In general, the cost of producing two kinds of libraries are quite small, compared to the cost of compiling the source code to object files. Finally, I have looked at how to manipulate symbol visibility. There seems many ways forward, so I feel confident that we can find a good solution. One way forward is to use objcopy to manipulate symbol status (global/local). There is an option --localize-symbol in objcopy, that has been available in objcopy since at least 2.15, which was released 2004, so it should be safe to use. But ideally we should avoid using objcopy and do this as part of the linking process. This should be possible to do, given that we make changes in NativeCompilation.gmk -- see question 5 above. As a fallback, it is also possible to rename symbols, either piecewise or wholesale, using objcopy. There are many ways to do this, using --prefix-symbols, --redefine-sym or --redefine-syms (note the -s, this takes a file with a list of symbols). Thus we can always introduce a "post factum namespace" by renaming symbols. So in the end, I think it will be fully possible to produce .a files that only has global symbols for the functions that are part of the API exposed by that library, and have all other symbols local, and make this is in a way that is consistent with the rest of the build system. Finally, a note on Hotspot. Due to debugging reasons, we export basically all symbols in hotspot as global. This is not reasonable to do for a static build. The effect of not exporting those symbols will be that SA will not function to 100%. On the other hand, I have no idea if SA works at all with a static build. Have you tested this? Is this part of the plan to support, or will it be officially dropped for Hermetic Java? /Magnus

7 38

Statistics about CDS effectiveness
by Stephane Nicoll 11 Jun '24

11 Jun '24

Hey, I hope this mailing list is the right place for this. I am working with Sébastien Deleuze on the Spring team and we've been focusing quite a bit on CDS lately and how it can be easily leveraged by our user base. One thing we quickly found out was that the cache can be more or less effective depending on the way you start the app (classloaders, non deterministic list of jar files, etc). Initially, we were testing the startup time and comparing when CDS wasn't enabled. I think there must be a better way and, for the lack of finding one, I've developed a (very) pragmatic approach of parsing the JVM logs and producing a report[1]. The tool has improved a bit for our own use case, creating the cache and providing a report of the warnings that were issued (to spot packages that couldn't be managed and why). Parsing the logs is good enough for our little testing there but we were wondering if there was an appetite (or need) to productize this. We were thinking that JFR events could be an option. Thanks, S. [1] https://github.com/snicoll/cds-log-parser -- This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.

2 2

git: openjdk/leyden: premain: Improve CompileTrainingData::_init_deps_left handling
by duke 29 Feb '24

29 Feb '24

Changeset: 4e9483db Author: Vladimir Ivanov <vladimir.x.ivanov(a)oracle.com> Date: 2024-02-28 19:44:38 +0000 URL: https://git.openjdk.org/leyden/commit/4e9483db83e5ba9055b14a566cee7a98fc87d… Improve CompileTrainingData::_init_deps_left handling ! src/hotspot/share/cds/cds_globals.hpp ! src/hotspot/share/classfile/compactHashtable.hpp ! src/hotspot/share/code/SCCache.cpp ! src/hotspot/share/code/SCCache.hpp ! src/hotspot/share/compiler/compilationPolicy.cpp ! src/hotspot/share/oops/instanceKlass.cpp ! src/hotspot/share/oops/trainingData.cpp ! src/hotspot/share/oops/trainingData.hpp ! src/hotspot/share/runtime/java.cpp

1 0

git: openjdk/leyden: premain: 2 new changesets
by duke 28 Feb '24

28 Feb '24

Changeset: 69027b63 Author: Igor Veresov <igor.veresov(a)oracle.com> Date: 2024-02-09 23:35:10 +0000 URL: https://git.openjdk.org/leyden/commit/69027b63c2cf2923835ba4c2a179929180d2d… Fix allocation types ! src/hotspot/share/oops/trainingData.cpp Changeset: 5012dd1b Author: Igor Veresov <igor.veresov(a)oracle.com> Date: 2024-02-28 09:54:22 +0000 URL: https://git.openjdk.org/leyden/commit/5012dd1b839694b465d85e9b01d91611185d5… Remove init tracking ! src/hotspot/share/oops/instanceKlass.cpp ! src/hotspot/share/oops/instanceKlass.hpp ! src/hotspot/share/oops/objArrayKlass.cpp ! src/hotspot/share/oops/trainingData.cpp ! src/hotspot/share/oops/trainingData.hpp ! src/hotspot/share/prims/jni.cpp ! src/hotspot/share/prims/unsafe.cpp

1 0

git: openjdk/leyden: premain: 3 new changesets
by duke 22 Feb '24

22 Feb '24

Changeset: e22e1cb3 Author: Vladimir Ivanov <vladimir.x.ivanov(a)oracle.com> Date: 2024-02-21 12:12:27 +0000 URL: https://git.openjdk.org/leyden/commit/e22e1cb375f1fed4340ebe1d3813074b35eeb… Trace method invocation from interpreter ! src/hotspot/cpu/aarch64/interp_masm_aarch64.cpp ! src/hotspot/share/runtime/sharedRuntime.cpp Changeset: c3984a7c Author: Vladimir Ivanov <vladimir.x.ivanov(a)oracle.com> Date: 2024-02-22 14:52:56 +0000 URL: https://git.openjdk.org/leyden/commit/c3984a7ccba08d4cbaeb38404cfc99a6343bb… relocInfo et al tracing improvements ! src/hotspot/share/asm/codeBuffer.cpp ! src/hotspot/share/asm/codeBuffer.hpp ! src/hotspot/share/code/SCCache.cpp ! src/hotspot/share/code/nmethod.cpp ! src/hotspot/share/code/nmethod.hpp ! src/hotspot/share/code/relocInfo.cpp ! src/hotspot/share/code/relocInfo.hpp Changeset: 1303cece Author: Vladimir Ivanov <vladimir.x.ivanov(a)oracle.com> Date: 2024-02-22 15:19:15 +0000 URL: https://git.openjdk.org/leyden/commit/1303cece6c2ee2bc41574ee36eab166537eb6… Merge remote-tracking branch 'origin/premain' into premain ! src/hotspot/share/code/SCCache.cpp ! src/hotspot/share/code/SCCache.cpp

1 0

git: openjdk/leyden: premain: fix Windows build failure
by duke 22 Feb '24

22 Feb '24

Changeset: e2d3cd2b Author: Calvin Cheung <calvin.cheung(a)oracle.com> Date: 2024-02-22 23:03:34 +0000 URL: https://git.openjdk.org/leyden/commit/e2d3cd2b0a604be9753e39b36943a81393485… fix Windows build failure ! src/hotspot/share/cds/archiveBuilder.cpp ! src/hotspot/share/cds/archiveBuilder.hpp ! src/hotspot/share/cds/cdsAccess.cpp ! src/hotspot/share/cds/cdsAccess.hpp ! src/hotspot/share/cds/cdsConfig.cpp ! src/hotspot/share/cds/classListWriter.cpp ! src/hotspot/share/cds/heapShared.cpp ! src/hotspot/share/cds/heapShared.hpp ! src/hotspot/share/classfile/modules.cpp ! src/hotspot/share/code/SCCache.cpp ! src/hotspot/share/code/SCCache.hpp

1 0

git: openjdk/leyden: premain: 2 new changesets
by duke 22 Feb '24

22 Feb '24

Changeset: 240a6782 Author: Ashutosh Mehra <asmehra(a)redhat.com> Date: 2024-02-22 15:53:50 +0000 URL: https://git.openjdk.org/leyden/commit/240a6782d61015b34f7defbe296ba52a692a0… jmeter test plan used by WarmupMakefile for springboot-petclinic application This test plan is derived from the test plan in springboot-petclinic github repo: https://github.com/spring-projects/spring-petclinic/blob/main/src/test/jmet… Signed-off-by: Ashutosh Mehra <asmehra(a)redhat.com> + test/hotspot/jtreg/premain/spring-petclinic/petclinic_test_plan.jmx Changeset: 50ea3bb5 Author: Ashutosh Mehra <asmehra(a)redhat.com> Date: 2024-02-22 15:57:28 +0000 URL: https://git.openjdk.org/leyden/commit/50ea3bb504409eec5a80e76736b941a926a04… Minor improvements to WarmupMakefile Signed-off-by: Ashutosh Mehra <asmehra(a)redhat.com> ! test/hotspot/jtreg/premain/spring-petclinic/WarmupMakefile

1 0

git: openjdk/leyden: hermetic-java-runtime: Co-locate hermetic net.properties in the `modules` image (as /java.base/sun/net/net.properties) with sun.net.NetProperties.
by duke 22 Feb '24

22 Feb '24

Changeset: f0efc4c9 Author: Jiangli Zhou <jianglizhou(a)google.com> Date: 2024-02-21 16:59:10 +0000 URL: https://git.openjdk.org/leyden/commit/f0efc4c949f86bc3c247c835f254b601d7074… Co-locate hermetic net.properties in the `modules` image (as /java.base/sun/net/net.properties) with sun.net.NetProperties. This can be reverted if we decide not to provide net.properties by default. ! make/Images.gmk ! src/java.base/share/classes/sun/net/NetProperties.java

1 0

git: openjdk/leyden: premain: 8318064: [premain] CDS regenerated LambdaForm holder classes miss some methods
by duke 22 Feb '24

22 Feb '24

Changeset: d67819c6 Author: iklam <ilam3388(a)gmail.com> Date: 2024-02-21 16:36:43 +0000 URL: https://git.openjdk.org/leyden/commit/d67819c6d8401e2c532e26772ac220827e9a1… 8318064: [premain] CDS regenerated LambdaForm holder classes miss some methods ! make/Images.gmk

1 0

git: openjdk/leyden: premain: 2 new changesets
by duke 21 Feb '24

21 Feb '24

Changeset: 5ce5a63d Author: iklam <ilam3388(a)gmail.com> Date: 2024-02-21 15:41:39 +0000 URL: https://git.openjdk.org/leyden/commit/5ce5a63d4b3831b850c87dc3833f1193b2eaa… Refactored code for deciding what classes can be pre-inited ! src/hotspot/share/cds/classPrelinker.cpp ! src/hotspot/share/cds/classPrelinker.hpp ! src/hotspot/share/cds/dumpTimeClassInfo.hpp ! src/hotspot/share/cds/metaspaceShared.cpp ! src/hotspot/share/classfile/systemDictionaryShared.cpp ! src/hotspot/share/classfile/systemDictionaryShared.hpp Changeset: 3cdb6324 Author: iklam <ilam3388(a)gmail.com> Date: 2024-02-21 15:45:01 +0000 URL: https://git.openjdk.org/leyden/commit/3cdb6324c09f260abd7626c3fb37a59bd2d88… Fixed test failure in ArchiveHeapTestClass.java ! test/hotspot/jtreg/runtime/cds/appcds/cacheObject/ArchiveHeapTestClass.java

1 0