April 2025 - leyden-dev

error writing the shared archive file with large number of loaded classes
by Olivier Bourgain 18 Apr '25

18 Apr '25

Hello, I have been experimenting with Leyden for a pretty large Spring Boot application (>1M loc, fat jar is about 240MB) and I think I hit a limitation of the current implementation of Leyden. I got the following error message at the end of the training run: [31,832s][error ][cds,heap ] [ 0] {0x000000060080be70} jdk.internal.loader.ArchivedClassLoaders::appLoader (offset = 16) [31,832s][error ][cds,heap ] [ 1] {0x0000000601400000} jdk.internal.loader.ClassLoaders$AppClassLoader::parallelLockMap (offset = 36) [31,832s][error ][cds,heap ] [ 2] {0x000000060095c5a8} java.util.concurrent.ConcurrentHashMap [31,832s][error ][cds,heap ] Cannot archive the sub-graph referenced from [Ljava.util.concurrent.ConcurrentHashMap$Node; object (0x000000060354bc88) size 262160, skipped. [31,832s][error ][cds ] An error has occurred while writing the shared archive file. So it looks like some classloader has a very large ConcurrentHashMap that ends up with a backing array of length 65536. Tracing back to Leyden’s sources, I found that the the archived objects size is limited - https://github.com/openjdk/leyden/blob/master/src/hotspot/share/cds/heapSha… - https://github.com/openjdk/leyden/blob/master/src/hotspot/share/cds/archive… <https://github.com/openjdk/leyden/blob/master/src/hotspot/share/cds/archive…> - https://github.com/openjdk/leyden/blob/83c7d3bbe860656ee403fa29df6a9e0aae96… By adding a --add-opens java.base/java.util=ALL-UNNAMED to the command line, it disables the optimized module handling [0.001s][info][cds] optimized module handling: disabled due to incompatible property: jdk.module.addopens=java.base/java.util=ALL-UNNAMED And the error goes away. I tried to make my case work without adding some add-opens that are not required by the app. As the limit in object size comes from MIN_GC_REGION_ALIGNMENT, which is limited by Shenandoah’s region size, and I don’t use that GC, I simply modified is_too_large_to_archive() to always return false. Then the archiving with optimized module handling succeeds. I created a Github repo to reproduce the issue, see the readme there: https://github.com/obourgain/leyden-classloader With regard to the improvements in startup time (on a M1 mac), here is what is get: - base line, exploded Spring Boot jar = 11s - spring AOT = 10.5s - leyden with optimized module handling disabled = 7.8s - leyden with optimized module handling enabled = 7.1s - leyden with optimized module handling disabled and Spring AOT = 7.3s - leyden with optimized module handling enabled and Spring AOT = 6.3s Overall, I think that’s pretty impressive improvements for this app’s startup which was pretty efficient already. Moreover, I did that on a project that started 12 or 13 years ago, so it shows that Leyden is applicable for that kind of project with substantial gains. I plan to try the premain branch soon. Let me know if I can help with more information. (this message was first submitted before I subscribed to the list. A copy of it may be pending review by the admin. Feel free to delete the pending copy)

2 2

git: openjdk/leyden: premain: Added test for diagnostic flags [+-]AOTRecordTraining and [+-]AOTReplayTraining
by Ioi Lam 18 Apr '25

18 Apr '25

Changeset: 2bf0da39 Branch: premain Author: Ioi Lam <iklam(a)openjdk.org> Date: 2025-04-17 20:43:15 +0000 URL: https://git.openjdk.org/leyden/commit/2bf0da39a480c8a5bb12ac067cb5d384be119… Added test for diagnostic flags [+-]AOTRecordTraining and [+-]AOTReplayTraining + test/hotspot/jtreg/runtime/cds/appcds/aotProfile/AOTProfileFlags.java

1 0

git: openjdk/leyden: hermetic-java-runtime: Add experimental native linking (for ld or lld on Linux) part. I added two experimental options to the --link-hermetic-image flag:
by duke 17 Apr '25

17 Apr '25

Changeset: ad613c30 Branch: hermetic-java-runtime Author: Jiangli Zhou <jianglizhou(a)google.com> Date: 2025-04-08 18:13:59 +0000 URL: https://git.openjdk.org/leyden/commit/ad613c3044454e7b479ade017ef919ee8d13d… Add experimental native linking (for ld or lld on Linux) part. I added two experimental options to the --link-hermetic-image flag: - `link-command=<link_cmd>`: Specify the complete native linking command - `pre-linked-exe=<pre_linked_launcher_executable>`: Specify the prelinked launcher executable, no native linking is performed in this case If the native linking command is specified via the `link-command=<link_cmd>` option (see below example), jlink invokes the native linker and performs the linking step. It finds the output file name/path from the native linking command and the output file is used to create the final hermetic Java image. ``` jlink --hermetic --link-hermetic-image link-command="/usr/bin/g++ ..." ... ``` The current change explicitly looks for the `-o` linker option, which only works with `ld` and `lld`. We need a better solution to support various linkers on different platforms. Following is an jlink command example to create hermetic Java image by including native linking part, with the current change: ``` In JDK build/linux-x86_64-server-fastdebug dir: $ /<snip>/jianglizhou/github/hermetic-linking/build/linux-x86_64-server-fastdebug/jdk/bin/jlink --hermetic --link-hermetic-image link-command="/usr/bin/g++ -Wl,-z,defs -Wl,-z,relro -Wl,-z,now -Wl,--no-as-needed -Wl,-z,noexecstack -m64 -pie -Wl,-z,defs -Wl,-z,relro -Wl,-z,now -Wl,--no-as-needed -Wl,-z,noexecstack -m64 -Wl,-rpath,\$ORIGIN -Wl,--disable-new-dtags -Wl,-rpath,\$ORIGIN/../lib -Wl,--disable-new-dtags -o /tmp/java support/static-native/launcher/main.o -Wl,--export-dynamic -Wl,--whole-archive support/native/java.base/libverify/static/libverify.a support/native/java.base/libjava/static/libjava.a support/native/java.base/libzip/static/libzip.a support/native/java.base/libjimage/static/libjimage.a support/native/java.base/libjli/static/libjli.a support/native/java.base/libnet/static/libnet.a support/native/java.base/libnio/static/libnio.a support/native/java.base/libsyslookup/static/libsyslookup.a support/native/java.base/libsimdsort/static/libsimdsort.a support/native/java.desk top/libawt/static/libawt.a support/native/java.desktop/libawt_xawt/static/libawt_xawt.a support/native/java.desktop/libjawt/static/libjawt.a support/native/java.desktop/libmlib_image/static/libmlib_image.a support/native/java.desktop/liblcms/static/liblcms.a support/native/java.desktop/libjavajpeg/static/libjavajpeg.a support/native/java.desktop/libfontmanager/static/libfontmanager.a support/native/java.desktop/libjsound/static/libjsound.a support/native/java.instrument/libinstrument/static/libinstrument.a support/native/java.management/libmanagement/static/libmanagement.a support/native/java.prefs/libprefs/static/libprefs.a support/native/java.rmi/librmi/static/librmi.a support/native/java.security.jgss/libj2gss/static/libj2gss.a support/native/java.smartcardio/libj2pcsc/static/libj2pcsc.a support/native/jdk.attach/libattach/static/libattach.a support/native/jdk.crypto.cryptoki/libj2pkcs11/static/libj2pkcs11.a support/native/jdk.hotspot.agent/libsaproc/static/libsaproc.a support/na tive/jdk.incubator.vector/libjsvml/static/libjsvml.a support/native/jdk.jdwp.agent/libdt_socket/static/libdt_socket.a support/native/jdk.jdwp.agent/libjdwp/static/libjdwp.a support/native/jdk.management/libmanagement_ext/static/libmanagement_ext.a support/native/jdk.management.agent/libmanagement_agent/static/libmanagement_agent.a support/native/jdk.net/libextnet/static/libextnet.a support/native/jdk.sctp/libsctp/static/libsctp.a support/native/jdk.security.auth/libjaas/static/libjaas.a hotspot/variant-server/libjvm/objs/static/libjvm.a -Wl,--no-whole-archive -lX11 -lXext -lXi -lXrender -lXtst -lasound -ldl -lfreetype -lm -lpthread -lrt -lz -static-libgcc -static-libstdc++ -static-libstdc++ -static-libgcc" --module-path "/<snip/jianglizhou/github/hermetic-linking/build/linux-x86_64-server-fastdebug/images/jmods" --endian little --hermetic-resources /java.base/conf/security/java.security=/java.base/java/security/java.security,/java.logging/conf/logging.properties=/java.logging/java/u til/logging/logging.properties,/java.base/conf/net.properties=/java.base/sun/net/net.properties,/java.desktop/lib/psfont.properties.ja=/java.desktop/sun/print/psfont.properties.ja,/java.desktop/lib/psfontj2d.properties=/java.desktop/sun/print/psfontj2d.properties,/java.base/lib/security/blocked.certs=/java.base/sun/security/util/blocked.certs,/java.base/lib/security/cacerts=/java.base/sun/security/util/cacerts,/java.base/lib/security/public_suffix_list.dat=/java.base/sun/security/util/public_suffix_list.dat,/java.base/lib/security/default.policy=/java.base/sun/security/provider/default.policy,/java.base/conf/security/java.policy=/java.base/sun/security/provider/java.policy,/java.base/conf/security/policy/limited/default_US_export.policy=/java.base/javax/crypto/limited_default_US_export.policy,/java.base/conf/security/policy/limited/default_local.policy=/java.base/javax/crypto/limited_default_local.policy,/java.base/conf/security/policy/limited/exempt_local.policy=/java.base/javax/cr ypto/limited_exempt_local.policy,/java.base/conf/security/policy/unlimited/default_US_export.policy=/java.base/javax/crypto/unlimited_default_US_export.policy,/java.base/conf/security/policy/unlimited/default_local.policy=/java.base/javax/crypto/unlimited_default_local.policy --generate-jli-classes=@/<snip>/jianglizhou/github/hermetic-linking/build/linux-x86_64-server-fastdebug/support/link_opt/default_jli_trace.txt --add-modules demo --keep-packaged-modules /<snip>/jianglizhou/github/hermetic-linking/build/linux-x86_64-server-fastdebug/images/jdk/jmods1 --output demo-image ``` ! src/jdk.jlink/share/classes/jdk/tools/jlink/internal/plugins/LinkHermeticImagePlugin.java

3 2

RFR: 8354675: Create separate image builder for hermetic
by Severin Gehwolf 17 Apr '25

17 Apr '25

This is a start of a simple refactoring for moving hermetic handling into dedicated implementation classes. In this case I cover `ImageBuilder` and `ExecutableImage`. A similar change should be done for `ImageFileCreator` but that's for another PR. @jianglizhou Please take a look. Thanks! While at it I've observed the following: 1. `<hermetic-image-file> --list-modules` doesn't seem to work. Shows [1] for me. 2. `strip -g <hermetic-image-file>` will destroy the image. Looks like [2]. Probably a drawback for tucking on the offset after the modules file? We might want to consider building in debuginfo handling into `jlink` itself. Like the `--strip-native-debug-symbols` plugin does (but doesn't work for hermetic, yet). Thoughts? [1] $ ./demo-image -m mod.example/com.example.demo.Main [0.017s][warning][cds] Failed to open shared archive file (Not a directory) Hi $ ./demo-image --list-modules [0.020s][warning][cds] Failed to open shared archive file (Not a directory) Exception in thread "main" java.lang.InternalError: Unable to detect the run-time image at java.base/jdk.internal.module.SystemModuleFinders.ofSystem(Unknown Source) at java.base/java.lang.module.ModuleFinder.ofSystem(Unknown Source) at java.base/jdk.internal.module.ModuleBootstrap.unlimitedFinder(Unknown Source) at java.base/jdk.internal.module.ModuleBootstrap.limitedFinder(Unknown Source) at java.base/sun.launcher.LauncherHelper.listModules(Unknown Source) [2] This is a build with `--with-native-debug-symbols=internal` (fastdebug) $ ll -lh ./demo-image -rwxr-xr-x. 1 sgehwolf sgehwolf 909M Apr 10 17:36 ./demo-image $ strip -g ./demo-image $ ll -lh ./demo-image -rwxr-xr-x. 1 sgehwolf sgehwolf 50M Apr 10 18:07 ./demo-image $ ./demo-image -m mod.example/com.example.demo.Main Error occurred during initialization of VM Failed setting boot class path. ------------- Commit messages: - Create a separate image builder for hermetic Changes: https://git.openjdk.org/leyden/pull/55/files Webrev: https://webrevs.openjdk.org/?repo=leyden&pr=55&range=00 Issue: https://bugs.openjdk.org/browse/JDK-8354675 Stats: 165 lines in 3 files changed: 138 ins; 21 del; 6 mod Patch: https://git.openjdk.org/leyden/pull/55.diff Fetch: git fetch https://git.openjdk.org/leyden.git pull/55/head:pull/55 PR: https://git.openjdk.org/leyden/pull/55

5 12

About the name "Hermetic Java"...
by Magnus Ihse Bursie 17 Apr '25

17 Apr '25

The name "Hermetic Java" has been bugging me a long time. I don't think there is much of a correlation between the general concept of hermeticity and the goal of the Hermetic Java project. The dictionary definition of "hermetic" is: * (specialist) tightly closed so that no air can escape or enter (synonym: airtight) * (formal, disapproving) closed and difficult to become a part of I don't think this name signals the purpose or functionality to the users. Also, it does not help that the pre-existing terminology of "hermetic builds" mean that you download all dependencies ahead of time, and can then build on an air-gapped machine. (This is, imho, a much better use of "hermetic".) I've been thinking about alternative names for a while, and I think that just a plain and clear "Single-Binary Java" would be the best. No-one can misunderstand it. It aligns with the overall tendency in Java to actually call things what they are. And it is just one syllable longer than "hermetic". :) (My runner-up suggestion is "Monolithic Java". But things can be monolithic in so many way, so I don't think that's nearly as helpful.) I'm not saying we need to rename the branch or project or anything, but the artifacts generated by the project, like documentation and end-user fronting information, should use "Single-Binary Java" instead. And methods like "JVM_UseHermeticJDKImage" should be called "JVM_UseSingleBinaryJDKImage" instead. /Magnus

1 0

git: openjdk/leyden: premain: 8353298: AOT cache creation asserts with _array_klasses in an unregistered InstanceKlass
by Ioi Lam 17 Apr '25

17 Apr '25

Changeset: a285fdad Branch: premain Author: Ioi Lam <iklam(a)openjdk.org> Date: 2025-04-16 18:38:14 +0000 URL: https://git.openjdk.org/leyden/commit/a285fdad2876dae6c33ddffa7aa9c53b0adcb… 8353298: AOT cache creation asserts with _array_klasses in an unregistered InstanceKlass ! src/hotspot/share/oops/klass.cpp + test/hotspot/jtreg/runtime/cds/appcds/aotCache/AOTCacheSupportForCustomLoaders.java

1 0

Prototype for new AOT command-line ergonomics
by ioi.lam＠oracle.com 17 Apr '25

17 Apr '25

I have implemented part of the draft JEP "Ahead-of-time Command Line Ergonomics" (https://openjdk.org/jeps/8350022) in the Leyden premain branch: https://github.com/openjdk/leyden/blob/premain/README.md#prototype-for-jep-… Please check it out and let us know what you think. Here's an example that creates an AOT cache with a single command: $ java -XX:AOTCacheOutput=JavacBenchApp.aot -cp JavacBenchApp.jar \ JavacBenchApp 50 Generated source code for 51 classes and compiled them in 2212 ms [3.720s][warning][cds] Skipping Sanity: Unsupported location Temporary AOTConfiguration recorded: JavacBenchApp.aot.config Launching child process /bld/images/jdk/bin/java to assemble AOT cache JavacBenchApp.aot using configuration JavacBenchApp.aot.config Picked up JAVA_TOOL_OPTIONS: -Djava.class.path=JavacBenchApp.jar -XX:AOTCacheOutput=JavacBenchApp.aot -XX:AOTConfiguration=JavacBenchApp.aot.config -XX:AOTMode=create Reading AOTConfiguration JavacBenchApp.aot.config and writing AOTCache JavacBenchApp.aot AOTCache creation is complete: JavacBenchApp.aot 39956480 bytes Removed temporary AOT configuration file JavacBenchApp.aot.config Thanks - Ioi

1 0

git: openjdk/leyden: premain: 2 new changesets
by duke 17 Apr '25

17 Apr '25

Changeset: e0f4b049 Branch: premain Author: Igor Veresov <igor.veresov(a)oracle.com> Date: 2025-04-16 16:26:09 +0000 URL: https://git.openjdk.org/leyden/commit/e0f4b049e52693aacccb9e9ef63b29d66d612… Prefix training options with AOT ! src/hotspot/share/cds/aotLinkedClassBulkLoader.cpp ! src/hotspot/share/cds/cdsConfig.cpp ! src/hotspot/share/cds/cds_globals.hpp ! src/hotspot/share/cds/metaspaceShared.cpp ! src/hotspot/share/cds/methodProfiler.cpp ! src/hotspot/share/compiler/compilationPolicy.cpp ! src/hotspot/share/compiler/compileBroker.cpp ! src/hotspot/share/compiler/compilerDefinitions.cpp ! src/hotspot/share/compiler/compiler_globals.hpp ! src/hotspot/share/compiler/recompilationPolicy.cpp ! src/hotspot/share/oops/recompilationSchedule.cpp ! src/hotspot/share/oops/trainingData.cpp ! src/hotspot/share/oops/trainingData.hpp ! src/hotspot/share/runtime/init.cpp ! src/hotspot/share/runtime/java.cpp ! src/hotspot/share/utilities/ostream.cpp ! src/hotspot/share/utilities/xmlstream.cpp Changeset: d170f211 Branch: premain Author: Igor Veresov <igor.veresov(a)oracle.com> Date: 2025-04-16 18:37:31 +0000 URL: https://git.openjdk.org/leyden/commit/d170f2114f4904c6ebcee461e87b6f46ae39a… Fix flag usage in tests ! test/hotspot/jtreg/premain/javac/javac-test.sh ! test/hotspot/jtreg/premain/javac_new_workflow/README.md ! test/hotspot/jtreg/premain/javac_new_workflow/run.sh ! test/hotspot/jtreg/premain/jmh/run.sh ! test/hotspot/jtreg/premain/lib/bench-lib.sh ! test/hotspot/jtreg/premain/lib/premain-run.sh ! test/hotspot/jtreg/premain/spring-petclinic/WarmupMakefile ! test/hotspot/jtreg/premain/spring-petclinic/bench.sh ! test/hotspot/jtreg/premain/spring-petclinic/result.sh

1 0

git: openjdk/leyden: premain: Added info about 8350022: Ahead-of-time Command Line Ergonomics in README.md
by Ioi Lam 16 Apr '25

16 Apr '25

Changeset: 63454751 Branch: premain Author: Ioi Lam <iklam(a)openjdk.org> Date: 2025-04-15 20:56:05 +0000 URL: https://git.openjdk.org/leyden/commit/634547513c2a2b707ae43a735dc24fd1977da… Added info about 8350022: Ahead-of-time Command Line Ergonomics in README.md ! README.md

1 0

git: openjdk/leyden: hermetic-java-runtime: 8354675: Create separate image builder for hermetic (#55)
by duke 15 Apr '25

15 Apr '25

Changeset: 2260f973 Branch: hermetic-java-runtime Author: Severin Gehwolf <sgehwolf(a)redhat.com> Committer: GitHub <noreply(a)github.com> Date: 2025-04-15 18:42:54 +0000 URL: https://git.openjdk.org/leyden/commit/2260f973c3aa910ebe8ad543d77e1d2552b4d… 8354675: Create separate image builder for hermetic (#55) ! src/jdk.jlink/share/classes/jdk/tools/jlink/builder/DefaultImageBuilder.java + src/jdk.jlink/share/classes/jdk/tools/jlink/builder/HermeticImageBuilder.java ! src/jdk.jlink/share/classes/jdk/tools/jlink/internal/TaskHelper.java

1 0