Is it better for JDK to distribute jmod files independently of JDK itself?

Wed Feb 9 02:58:19 UTC 2022

Thank Alan and Mike for their replies. There are problems in this work that
I didn't expect before, but I don't think it is difficult to solve.

My current idea is to allow jmod to include such a manifest file:

fb9264533c96fd3589d5afa4e9736f7351d938b6ce7e6a142a32345842b85690  bin/java
4e3373188e828751cd64a4bac5ecc08967335cfc72d1a7272eaba35be056b0d7
 bin/keytool
57bc8681c5525ff907f0ca6bd59fa20e5ed31c36d96bcac299136b54002acb5e
 classes/META-INF/services/java.nio.file.spi.FileSystemProvider
6eef3965f7348ef15f81b2b8358be7cd68b3cf0902fb19d4f7f475db27cc3b88
 classes/com/sun/crypto/provider/AESCipher$AES128_CBC_NoPadding.class
d926525e4ef6fa86c48c2bdda05ccb5f469e56394b9932310eeb145a015a327d
 classes/com/sun/crypto/provider/AESCipher$AES128_CFB_NoPadding.class
c0636edd475a650fbeda67b296f21b3524ba5f8587ea8e7b01d1bf6b67cc6f63
 classes/com/sun/crypto/provider/AESCipher$AES128_ECB_NoPadding.class
d41123cab33597437eb71ef0ffcbd7545ed50615d8f594afdf672826e4a8abd4
 classes/com/sun/crypto/provider/AESCipher$AES128_GCM_NoPadding.class
...
f4ecb9abe6831695d478d4395bc82e93b5244c5dbaddc4948ca49e3efd81eae1
 classes/sun/util/resources/cldr/CurrencyNames_en.class
319683701a1b539ff93479fd63b102bfd01064f26f481f6665e52a7e747acd45
 classes/sun/util/resources/cldr/LocaleNames.class
c63870397963307f56a3189e7bc5c18426814f0943e265df5447899c641c081f
 classes/sun/util/resources/cldr/LocaleNamesProvider.class
f0294177b22606daf29663784a06b4fc6942e2bc131568f9144178b95389efa3
 classes/sun/util/resources/cldr/LocaleNames_en.class
fe71ede9dbca736478356f2e9905a28af45dddc17efedcf2b94e5763e65b89f3
 classes/sun/util/resources/cldr/TimeZoneNames.class
7e1b1967a91bd8f3f348e39f0bf2440c0b523afdcb59e28c49949317a698e95e
 classes/sun/util/resources/cldr/TimeZoneNamesProvider.class
8d3258fa4a145ef87b4084a8dab56934d4b0ef57b4a14630f96fc83f1cabc4b8
 classes/sun/util/resources/cldr/TimeZoneNames_en.class
701982a49e37d15a9bef6faf5ccf91284a7bd565167c2af35b9232e43580ddd1
 classes/sun/util/spi/CalendarProvider.class
a88eeb56c61c0df79c8d000a43de10dba48c77130c8ada1af942c9a8ce29a787
 conf/net.properties
e9441e51f0d29e25313d33a9005640f11ad93c27f01bc46a34bf02f2a98ffd6b
 conf/sdp/sdp.conf.template
f2a00a1dec3b7a097f0815f338a84717ba1017d5d7aae96d842d2188d67c3250
 conf/security/java.policy
d10da8fc5bdcb2771024c99cd7a5f8ce84954c21124d27a90a670c32dc1002ee
 conf/security/java.security
6da0747334b0fea7592fd92614b2bbc8b126535e129b1fee483774d914e98eb5
 conf/security/policy/README.txt
758b930a526fc670ab7537f8c26321527050a31f5f42149a2dda623c56a0a1a9
 conf/security/policy/limited/default_US_export.policy
2b2627548e61316150d47ffc3e6cad465ca05b3cccd4785eb7d21aa7baa0f441
 conf/security/policy/limited/default_local.policy
8c3d7648abcd95a272ce12db870082937f4d7f6878d730d83cb7fbb31eb8b2c9
 conf/security/policy/limited/exempt_local.policy
758b930a526fc670ab7537f8c26321527050a31f5f42149a2dda623c56a0a1a9
 conf/security/policy/unlimited/default_US_export.policy
8d8a318e6d90dfd7e26612d2b6385aa704f686ca6134c551f8928418d92b851a
 conf/security/policy/unlimited/default_local.policy
915edc29d63cf2323575c8070d6f0d27d7ff30904aba58ac8ec6571b56d4f48c
 include/classfile_constants.h
1266aea5b9f5d5db1cb6f8e5c6c43cfa7f80bc4f72d7fe42c6131bb939dc70f4
 include/jni.h
56467aef065d402cefbfcff80578ea68568270561a0ee210f8db3154bf7702b1
 include/jvmti.h
6ee3e52d24bdb4f4d0312dfbab3d47bd524cbdac5540a4d790cac0620c59b3c8
 include/jvmticmlr.h
88cb5c33e306900dd35a78d5a439087123b8e91b0986bb5acb42cc9bd2fcc42e
 include/linux/jni_md.h
a69bce275ba7a3570af6579cb0f55682cd75fedfcd49e0e8e9022270c447c916
 legal/ADDITIONAL_LICENSE_INFO
a44eb7b5caf5534c6ef536b21edb40b4d6babf91bf97d9d45596868618b2c6fb
 legal/ASSEMBLY_EXCEPTION
4b9abebc4338048a7c2dc184e9f800deb349366bdf28eb23c2677a77b4c87726
 legal/LICENSE
45c6d4da48325edfbff3dcf71c704e504c057904435ed23c6d57046d551eb69d
 legal/aes.md
1ff912740e84e024711def5fa482ffbb46eff64559760c467352dfa7c39a3307
 legal/asm.md
bef40679922d6fdfb7e4ddb223ad6722300f6054ba737bbf6188d60fcec517f9
 legal/c-libutl.md
4491d38ccd79fcda6d14871a3551bced8db26ab2f2232b9d9db2e1eabae25d2f
 legal/cldr.md
af0057e8553906083f69c2fb9fe9ed4ae8bc2340a0b1e376a424702f00300b29
 legal/icu.md
d9ed58c3132c2c8e82b095eb4ce24cafd1f20531c16a7c9d01f2134843904db7
 legal/public_suffix.md
5e4e6623a21a63f9bc16ea54af4133b8038e490c0d499a74676f9e5a61b9c5b2
 legal/unicode.md
97551969e2a0f0ebfad1a56f2d6de68a1550790f4004c0fe1bb38ff9c9531bab
 lib/classlist
a54aef23cf04b363c8d960f52221ef5615bc324c94cbe0731459fd46012092e8  lib/jexec
4b90aa83dcfa5dda3088b3305a8d6eb10665ed0e20b556961171503d0e5bb2f6
 lib/jrt-fs.jar
3a1a2ec8b5bc95a2b90ff0e997a6267505d8a5959bd5eea39e1a95d3fc2c1553
 lib/jspawnhelper
aa9efb969444c1484e29adecab55a122458090616e766b2f1230ef05bc3867e0
 lib/jvm.cfg
70fd9b3b282196212ad376706b3a994034cf045eeb5c8bab1d4c39b71c6b93df
 lib/libjava.so
549db6b79b5c0da247cead07e4f1c6186ef13f143fda7d59a727edcf8004d9d9
 lib/libjimage.so
00940eec1c6c3aefab64a925bbe350c6da6741c8b659cb871af519450665fd1d
 lib/libjli.so
aadeaddf9a98f7c083a8b8f8308dd9c9d3cee24ec6cd212a58c2fdc8ca09b1ef
 lib/libjsig.so
e64c81046cc02db85778af58ed6232dd77196606f7e87416ab68f5bdb983fcd1
 lib/libnet.so
13641e5cbf6fd61fced7311147ccb6e52c064353594ec60270d28ba3f4cd07b0
 lib/libnio.so
6be92fbef15868208bbabc839b3314257f1370ea09f66d651a71990822061155
 lib/libverify.so
bb9e17f34e87a9325e3ea07b8ae6aad8c92603c9b9ecbffe5e7aaafc47086130
 lib/libzip.so
96572f243f31c2ef81a6e627542e596f6a9295cff3c7ae095c1b595cb1457ded
 lib/security/blacklisted.certs
509753061b403271d6cd3b29897cbfbf416f56b10de9b82a8c7cc2cdcd9f10ef
 lib/security/cacerts
638d2f75d11b7100158ac4135e68ddc58d1662ce1afaaf35cdd57363deb94a00
 lib/security/default.policy
1200ae5fff5720e797186b4afcae436c27a43c1e46d2ae44cec435461727fbbb
 lib/security/public_suffix_list.dat
aadeaddf9a98f7c083a8b8f8308dd9c9d3cee24ec6cd212a58c2fdc8ca09b1ef
 lib/server/libjsig.so
703d12633d0b2e0b200053b0642b35f8339769532144f210cf232f47699d4817
 lib/server/libjvm.so
0fd049816c99a6d1dfb517fbcf32ca345dd134e971415859057e45e28d031d50
 lib/tzdb.dat

Each file in the jmod file and its hash value are recorded here.

When such a manifest file exists, this jmod becomes a "fallback jmod", it
allows some of these files not to exist in jmod.
When using "fallback jmod", you need to specify a runtime path (or use the
current 'java.home' by default).

I'm trying to develop a tool to generate a "fallback jmod" from existing
jmod file,
you may need to use it this way later (use '--replace' to specify that it
overwrites the source jmod file):

vjmod reduce --replace --runtime-path $JAVA_HOME ./java.base.jmod

It traverses the entries in the jmod file and adds a manifest file to the
jmod file.
For the files in the 'conf' directory, do not modify them;
For other files, the tool searches for them in the runtime path, if they
exist in the runtime path and are consistent with the file in the jmod,
delete it from the jmod file. (I may add command line options that allow
users to specify additional files list that should not be deleted or forced
to be deleted)
By deleting only the files whose contents exactly match, the problem
described by Alan can be solved.

It should also provide a way for users to restore jmod:

vjmod restore --replace --runtime-path $JAVA_HOME ./java.base.jmod

In the coming period of time, I will also explore this possibility and try
to implement a jlink plugin.
(If a jlink plugin is not enough to implement it, I will try the javaagent
or other methods)
When jlink handles a "fallback jmod", when it finds a file recorded in the
manifest but does not exist, it will follow the rules of jmod try to
extract the file from the runtime path,
and verify it with the hash value of the record; If this file already
exists in the jmod file, use it directly.

Thank you for your help. Do you have any suggestions for this idea?

Glavo <zjx001202 at gmail.com> 于2022年2月8日周二 22:17写道：

> Since Java 9, JDK has been distributed with jmod files bundled with all
> modules. I am confused by this release strategy.
>
> One important problem is that this expands the size of JDK's compressed
> package by 60% (~ 70MB), and the decompressed size expands by 30%.
> This takes up a lot of extra hard disk space, and more importantly, it
> causes a lot of bandwidth waste.
> Many times we only want a JDK and don't need jlink (especially CI
> environment), but unfortunately I haven't seen any openjdk distribution
> provide such things.
>
> Conversely, when I use jlink, it also annoys me. I want to use a dedicated
> Linux machine to compile the program and use jlink to generate the runtime
> for each platform.
> On this machine, I only need a JDK of Linux AMD64 and jmod files of JDK of
> various platforms,
> but I have never seen any provider provide JDK's jmod files separately, so
> I have to download one complete JDK after another,
> even if I can't use them at all except jmod files. When the number of
> target platforms increases, the waste caused by this increases sharply.
>
> These wastes seem to be less important for a single JDK, but in some
> scenarios (CI, container, virtual machine, etc), a large number of jdks may
> need to be downloaded and stored,
> and these wastes cannot be ignored.
>
> Now I'm confused about the current distribution method. Would it be a
> better choice to distribute JDK and jmod separately?
>
> In addition, I think there are other better options for the out-of-the-box
> availability of jlink.
> I noticed that a copy of everything in jmod already exists in the JDK
> folder.
> In fact, jmod of JDK core module only needs to store a file list (List of
> paths of all native libraries, executable files and legal files),
> original content backup of some modifiable content (e.g. conf files), and
> the name of module.
> We can re extract the complete jmod file from JDK through these meta
> information, the size of these meta information is completely negligible.
> I'm spending some free time trying to implement such a tool, and I don't
> think it's difficult.
>
> I think if jlink can recognize this kind of meta information, JDK only
> needs to bundle these trivial meta information contents,
> and the complete jmod file is distributed separately from JDK, which may
> be the best choice.
>
> These are my immature ideas. I hope I can get your advice. Thank you.
>