RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v8]

Thu Nov 16 18:33:33 UTC 2023

On Tue, 14 Nov 2023 15:21:47 GMT, Severin Gehwolf <sgehwolf at openjdk.org> wrote:

>> Please review this patch which adds a "jmodless" jlink mode to the JDK. Fundamentally this patch adds an option to use `jlink` even though your JDK install might not come with the packaged modules (directory `jmods`). This is particularly useful to further reduce the size of a jlinked runtime. After the removal of the concept of a JRE, a common distribution mechanism is still the full JDK with all modules and packaged modules. However, packaged modules can incur an additional size tax. For example in a container scenario it could be useful to have a base JDK container including all modules, but without also delivering the packaged modules. This comes at a size advantage of `~25%`. Such a base JDK container could then be used to `jlink` application specific runtimes, further reducing the size of the application runtime image (App + JDK runtime; as a single image *or* separate bundles, depending on the app being modularized).
>> 
>> The basic design of this approach is to add a jlink plugin for tracking non-class and non-resource files of a JDK install. I.e. files which aren't present in the jimage (`lib/modules`). This enables producing a `JmodLessArchive` class which has all the info of what constitutes the final jlinked runtime.
>> 
>> Basic usage example:
>> 
>> $ diff -u <(./bin/java --list-modules --limit-modules java.se) <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules --limit-modules java.se)
>> $ diff -u <(./bin/java --list-modules --limit-modules java.se) <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules --limit-modules jdk.jlink)
>> $ ls ../linux-x86_64-server-release/images/jdk/jmods
>> java.base.jmod            java.net.http.jmod       java.sql.rowset.jmod      jdk.crypto.ec.jmod         jdk.internal.opt.jmod                     jdk.jdi.jmod         jdk.management.agent.jmod  jdk.security.auth.jmod
>> java.compiler.jmod        java.prefs.jmod          java.transaction.xa.jmod  jdk.dynalink.jmod          jdk.internal.vm.ci.jmod                   jdk.jdwp.agent.jmod  jdk.management.jfr.jmod    jdk.security.jgss.jmod
>> java.datatransfer.jmod    java.rmi.jmod            java.xml.crypto.jmod      jdk.editpad.jmod           jdk.internal.vm.compiler.jmod             jdk.jfr.jmod         jdk.management.jmod        jdk.unsupported.desktop.jmod
>> java.desktop.jmod         java.scripting.jmod      java.xml.jmod             jdk.hotspot.agent.jmod     jdk.internal.vm.compiler.management.jmod  jdk.jlink.jmod       jdk.naming.dns.j...
>
> Severin Gehwolf has updated the pull request incrementally with four additional commits since the last revision:
> 
>  - First pass at 'run-image' => 'run-time image'
>  - Remove 'Please double check!' phrase.
>  - Don't show --add-run-image-resources plugin in listing
>  - Rename resource file to jdk_internal_runimage

Thanks.

> How can the user know what plugins are applied to `image2`? i.e. what is the jlink command to produce `image2` if running from `jdk22` with the packaged modules present?

The only way to know is by knowing the chain of `jlink` commands yielding up to the final image. Let `jlink'` be the jlink using packaged modules. Currently this can be at most two. In addition, the contents of `argfile` needs to be known. That doesn't seem to be very different to the status quo, though. See below.

> Reading the changes, I'm not sure but I think it's not equivalent to:
> 
> ```
> jdk22/bin/jlink --add-modules jdk.jlink --output image2 --vendor-bug.url https://xyz.com/bugs --save-jlink-argfiles argfile --generate-jli-classes jli_trace.txt --strip-debug --add-options "-Dcom.foo.XYZ=true" --add-options "-Dcom.acme.name=BAR"
> ```

Note that plugins like `--add-options` have been modified so that only the options passed at the current CLI will propagate to the final image. So in this case, `image1` would have property `com.foo.XYZ` set, but not `image2`. Incidentally, what you probably intended to use was `--add-options "-Dcom.foo.XYZ=true -Dcom.acme.name=BAR".

Having said that, depending on the contents of `argfile`, those could be equivalent. In fact, they are with `--unlock-run-image`, and an empty `argfile`. `--unlock-run-image` avoids adding the marker file, which is the only difference when we extract the image.

`--save-jlink-argfiles` brings a strange angle to this discussion, but it's conceivable to get a similarly different image, even with --keep-packaged-modules. Consider:

echo '--add-options=-XX:+UseParallelGC' > argfile
./jdk22/bin/jlink --save-jlink-argfiles argfile --strip-debug --add-modules jdk.jlink --keep-packaged-modules ./image_a/jmods --output image_a
./image_a/bin/jlink --add-modules java.base --output image_b

It's not very apparent that `image_b` will have `-XX:+UseParallelGC`. So you need some sort of track record what you've used previously already in order to know how `image_b` got produced (short of `--save-opts`)?

> I think this behavior should be documented.

Makes a lot of sense. Where? In the CSR?

> > Note that plugins like `--add-options` have been modified so that only the options passed at the current CLI will propagate to the final image. So in this case, `image1` would have property `com.foo.XYZ` set, but not `image2`. Incidentally, what you probably intended to use was `--add-options "-Dcom.foo.XYZ=true -Dcom.acme.name=BAR".
> > Having said that, depending on the contents of `argfile`, those could be equivalent. In fact, they are with `--unlock-run-image`, and an empty `argfile`. `--unlock-run-image` avoids adding the marker file, which is the only difference when we extract the image.
> > `--save-jlink-argfiles` brings a strange angle to this discussion, but it's conceivable to get a similarly different image, even with --keep-packaged-modules.
> 
> Right. This example intends to show that the behavior is not straight-forward for users to follow and also subject to the implementation of each of the plugins. I think we need an easy-to-understand model for developers to understand. Some possible options:
> 
> Option 1: all plugins applied in `image1` are _auto-applied_ to `image2` by default

Option 1 seems most appealing. I can massage `--add-options` to concatenate. The exception of the rule seems `--system-modules` for obvious reasons.

> Option 2: all plugins applied in `image1` are _not applied_ to `image2` by default
> 
> When there is an exception, it should be documented clearly by the plugin (possibly in the output from `--list-plugins`). I also think option 1 may be more useful to the developers. I'm not sure how many plugins can undo the transformation done in `image1` when creating `image2`.
> 
> For example, with option 1,
> 
> ```
> $ jdk22/bin/jlink --add-modules jdk.compiler,jdk.jlink --output image1 --vendor-bug.url https://xyz.com/bugs --save-jlink-argfiles argfile --generate-jli-classes jli_trace.txt --strip-debug --add-options "-Dcom.foo.XYZ=true"
> 
> $ image1/bin/jlink --add-modules jdk.jlink --output image2 --add-options "-Dcom.acme.name=BAR"
> # equivalent to:
> $ jdk22/bin/jlink --add-modules jdk.jlink --output image2 --vendor-bug.url https://xyz.com/bugs --save-jlink-argfiles argfile --generate-jli-classes jli_trace.txt --strip-debug --add-options "-Dcom.foo.XYZ=true  -Dcom.acme.name=BAR"
> 
> $ image1/bin/jlink --add-modules java.base --output image3 --vendor-bug.url https://com.acme/bugs
> # equivalent to:
> $ jdk22/bin/jlink --add-modules java.base --output image3  --generate-jli-classes jli_trace.txt --strip-debug --add-options "-Dcom.foo.XYZ=true" --vendor-bug.url https://com.acme/bugs
> ```
> 
> Discussion points are:
> 
>     * `--save-jlink-argfiles` is only applicable when `jdk.jlink` is added to the custom module.  I think this one is not an issue.

For the runtime image link that's a requirement: `jdk.jlink` part of the image that is being used to create the image to perform the runtime image link. But I agree that this use-case seems rather expert.

>     * `--add-options` concatenates the options?

Sure, that can be done.

>     * `-vendor-bug.url https://xyz.com/bugs --vendor-bug.url https://com.acme/bugs` last one wins?

Yes. That's also how it works if the build got configured with specific vendor and then a jlink is being performed using packaged modules.

> We need to go through each plugin and decide on its behavior.

OK. I'll add documentation to the `--list-plugins` help should there be deviations.

> I'm also pondering how the Plugin API should support this run-time image based linking.

It could, but doesn't need to IMO. OTOH, `transform()` could grow an argument to indicate packaged-modules vs runtime image link.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-1815011717