From ethan at mccue.dev Tue Jun 13 14:16:56 2023 From: ethan at mccue.dev (Ethan McCue) Date: Tue, 13 Jun 2023 10:16:56 -0400 Subject: Condensers and Shading Message-ID: Hi All, I just read through the early discussion on the condenser concept. A brain-worm I can't dispose of[1] is that the concept of time shifting could be used to allow language level shading. That is, if a user can say that none of the classes in a module will be reflected upon using dynamically constructed values.[2] Class.forName(f()); // X Then it would be a "safe" operation to change the package name of all of those classes and bundle them inside of a module which requires them.[3] shadeable module org.example { exports org.example; exports org.example.abc; } module org.other { requires shaded org.example; } Now, this is not in Leyden's mission statement. Nothing about startup time is improved by renaming packages and classes. But the set of information you'd need to allow shading seems to overlap with the set of information you'd need to allow for making a static executable. So in the same way Valhalla is converging towards T! + value + implicit constructor = inline-ability there might be a "can make static exe" that falls out of other features like this. [1]: And one I feel I should bring up considering that the set of condensers might start out or remain closed to keep a consistent platform definition. [2]: + some other restrictions I am sure [3]: There might be a place other than modules which could work, but module-info declarations are currently the only place where users explicitly mark reflect-ability constraints today, so it -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.pressler at oracle.com Tue Jun 13 14:32:16 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Tue, 13 Jun 2023 14:32:16 +0000 Subject: Condensers and Shading In-Reply-To: References: Message-ID: <4F23C98C-0A68-4826-8062-341A225FE3BD@oracle.com> There is no general way to know whether multiple instances of a library can coexist in the same process or in what mysterious ways attempting that will break (the instances could write to the same files, bind to the same port etc.). This is true for all languages. However, to the extent that is possible, modules already support this with layers; i.e. you already have the best possible ?language level shading? you can hope for. The reason layers can only be configured programmatically is that the best possible is not good enough to recommend for wide use, but there are tools out there (e.g. https://github.com/moditect/layrry) that offer configuring layers with configuration files. ? Ron > On 13 Jun 2023, at 10:16, Ethan McCue wrote: > > Hi All, > > I just read through the early discussion on the condenser concept. > > A brain-worm I can't dispose of[1] is that the concept of time shifting could be used to allow language level shading. > > That is, if a user can say that none of the classes in a module will be reflected upon using dynamically constructed values.[2] > > Class.forName(f()); // X > > Then it would be a "safe" operation to change the package name of all of those classes and bundle them inside of a module which requires them.[3] > > shadeable module org.example { > exports org.example; > exports org.example.abc; > } > > module org.other { > requires shaded org.example; > } > > Now, this is not in Leyden's mission statement. Nothing about startup time is improved by renaming packages and classes. But the set of information you'd need to allow shading seems to overlap with the set of information you'd need to allow for making a static executable. > > So in the same way Valhalla is converging towards T! + value + implicit constructor = inline-ability there might be a "can make static exe" that falls out of other features like this. > > [1]: And one I feel I should bring up considering that the set of condensers might start out or remain closed to keep a consistent platform definition. > [2]: + some other restrictions I am sure > [3]: There might be a place other than modules which could work, but module-info declarations are currently the only place where users explicitly mark reflect-ability constraints today, so it > From brian.goetz at oracle.com Tue Jun 13 14:33:51 2023 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 13 Jun 2023 10:33:51 -0400 Subject: Condensers and Shading In-Reply-To: References: Message-ID: <8d467266-0ef3-1ddb-dbd6-5554d424f198@oracle.com> > A brain-worm I can't dispose of[1] is that the concept of time > shifting could be used to allow language level shading. It's quite possible; program transformation is a pretty general tool.? However, I want to caution that Leyden's design center is around _semantic-preserving_ transformations.? That is, a Java program has a meaning, and the transformed program should have the same meaning as the original. As you point out, the kind of information needed to make this transformation has a lot of overlap with the kind of information you need for dead code stripping, AOT compilation, reflection elimination, etc. From sgehwolf at redhat.com Thu Jun 15 08:50:55 2023 From: sgehwolf at redhat.com (Severin Gehwolf) Date: Thu, 15 Jun 2023 10:50:55 +0200 Subject: jmods-less jlinking prototype In-Reply-To: <7c67f4f2-a794-8852-5aca-8bb4055803b5@oracle.com> References: <5a81ad40b9b20d92a73ddedf437187c7f141f9ce.camel@redhat.com> <7c67f4f2-a794-8852-5aca-8bb4055803b5@oracle.com> Message-ID: <4a2cb94a868dac427e2e33997f406f599b9018f3.camel@redhat.com> Hi Alan, On Wed, 2023-03-15 at 14:51 +0000, Alan Bateman wrote: > > > > The Problem: > > > > ------------ > > > > Why do we need the 'jmods' directory when running jlink when 'java -- > > > > list-modules' lists all the modules and all the files from the JDK > > > > build are in (only 'jmods' directory is > > > > missing)[1]? > > > > Just some history around this. During JDK 9, there was a suggestion that > > the packaged modules should be a separate download but that had its own > > set of issues.? There were also suggestions that jlink should use the > > classes/resources from the current run-time image to avoid needing to > > include the complete contents of all packaged modules, it just wasn't > > interesting to pursue at the time. That was 6+ years ago and there is a > > lot of experience with jlink since then, plus we've now on a road where > > jlink may be working with condensers in the future. So it could be > > interesting to explore to see how it might fit in. > > > > More background is that generating a run-time image may involve a number > > of transformations, e.g jlink --strip-debug may remove debugging symbols > > and strip debug related class file attributes, other plugins, like > > generate-jli-classes and system-modules, generate code and/or modify a > > number of classes.? Post generation, there may also be changes to > > configuration files in conf/**. I think the point is that all types of > > resources (native libs, executables, classes/resources, conf) can be > > filtered out, stripped, or irreversibly transformed. Even if the > > run-time image has the jdk.jlink module, it might not have many of the > > modules that were in the original JDK builds so it's not really a good > > starting point for stamping out further images. Of course the starting point needs to be suitable for the individual use-cases. > > On container environments, we see a lot of examples where jlink is run > > to produce a run-time image with a subset of the modules and with some > > stripping and compression to get a smaller run-time image. Smaller > > container means faster to transport across network and deploy, great! > > Most of the recipes that I've seen published don't allow for further > > jlink-ing, either because they only have a small subset of the modules > > or they don't have the jdk.jlink module so they don't have the tool. > > Maybe this effort might change that. Exactly! > > So I think it could be interesting to explore to at least see how far it > > can get. I'd like to restart this conversation in order to ultimately arrive at a go/no-go decision about this patch. Hope that's OK. We do agree it's interesting to explore further. Since last time we spoke the patch has been improved further and I think it's viable. It supports the three main platforms and passes the basic jtreg testing (including the added test; see below for the GHA runs). So if we were to contribute this into a shared repo, which one should this go? a) openjdk/jdk (a.k.a JDK mainline) or b) leyden/jdk c) something else? Would you have any thoughts as to which repo would be most suitable at this point to discuss a PR? We are reasonably confident that it works well for a majority of use- cases (also outside of leyden) so it might be useful to have in mainline JDK. But maybe it makes sense to first add it to leyden? I'm not sure. In the leyden context it would be a great first building block to further explore composability with a simple enough patch[1]. Consider: j_A -> j_B -> j_C -> j_D where j_A is a full jlink from jmods (produced during the OpenJDK build), and then j_B..j_D would be further reductions (condensor runs if you will). Happy to document the caveats of this new jmod-less mode in a CSR if that helps. IMHO, there aren't that many in practice. Thanks, Severin [1] https://github.com/jerboaa/jdk/compare/jlink-jmods-base...jerboaa:jdk:jlink-jmods-less?expand=1 jdk/tools/jlink tests, see 'jdk/jlink' test runs: Before: https://github.com/jerboaa/jdk/actions/runs/5266908852 After: https://github.com/jerboaa/jdk/actions/runs/5269856679 From Alan.Bateman at oracle.com Thu Jun 15 17:03:59 2023 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 15 Jun 2023 18:03:59 +0100 Subject: jmods-less jlinking prototype In-Reply-To: <4a2cb94a868dac427e2e33997f406f599b9018f3.camel@redhat.com> References: <5a81ad40b9b20d92a73ddedf437187c7f141f9ce.camel@redhat.com> <7c67f4f2-a794-8852-5aca-8bb4055803b5@oracle.com> <4a2cb94a868dac427e2e33997f406f599b9018f3.camel@redhat.com> Message-ID: <98802f4a-82e0-1331-cb1e-543b0b22650b@oracle.com> On 15/06/2023 09:50, Severin Gehwolf wrote: > : > I'd like to restart this conversation in order to ultimately arrive at > a go/no-go decision about this patch. Hope that's OK. > > We do agree it's interesting to explore further. Since last time we > spoke the patch has been improved further and I think it's viable. It > supports the three main platforms and passes the basic jtreg testing > (including the added test; see below for the GHA runs). So if we were > to contribute this into a shared repo, which one should this go? a) > openjdk/jdk (a.k.a JDK mainline) or b) leyden/jdk c) something else? > Would you have any thoughts as to which repo would be most suitable at > this point to discuss a PR? > > We are reasonably confident that it works well for a majority of use- > cases (also outside of leyden) so it might be useful to have in > mainline JDK. But maybe it makes sense to first add it to leyden? I'm > not sure. In the leyden context it would be a great first building > block to further explore composability with a simple enough patch[1]. > Consider: > > j_A -> j_B -> j_C -> j_D > > where j_A is a full jlink from jmods (produced during the OpenJDK > build), and then j_B..j_D would be further reductions (condensor runs > if you will). > > Happy to document the caveats of this new jmod-less mode in a CSR if > that helps. IMHO, there aren't that many in practice. I think a nice property to have would be for a jlink command without packaged modules generate an identical run-time image as it would with the same command when using the packaged modules. In my view at least, if it can't guarantee to generate the same run-time image then it should fail. I can't tell how close you are to this but it means being able to reconstitute the original resources or detect that they have changed. So I think it means plugins to un-do transformations where feasible, maybe hashes to detect changes to resources. If we had that they I think it would be a useful change to have in the main line. Clearly if there is stripping of native debug symbols, and some other transformations, then you aren't going to be able to reconstitute the original natives with debug info. So if someone were to create a run-image with stripping and including jdk.jlink module then the resulting run-image could not be used to stamp down further run-images without specifying the location of the packaged modules. I'm less sure about connection to Leyden and condensers. -Alan From sgehwolf at redhat.com Fri Jun 16 12:48:13 2023 From: sgehwolf at redhat.com (Severin Gehwolf) Date: Fri, 16 Jun 2023 14:48:13 +0200 Subject: jmods-less jlinking prototype In-Reply-To: <98802f4a-82e0-1331-cb1e-543b0b22650b@oracle.com> References: <5a81ad40b9b20d92a73ddedf437187c7f141f9ce.camel@redhat.com> <7c67f4f2-a794-8852-5aca-8bb4055803b5@oracle.com> <4a2cb94a868dac427e2e33997f406f599b9018f3.camel@redhat.com> <98802f4a-82e0-1331-cb1e-543b0b22650b@oracle.com> Message-ID: <145f250bd9b2584e27191db6939910126af6b1ff.camel@redhat.com> On Thu, 2023-06-15 at 18:03 +0100, Alan Bateman wrote: > On 15/06/2023 09:50, Severin Gehwolf wrote: > > : > > I'd like to restart this conversation in order to ultimately arrive at > > a go/no-go decision about this patch. Hope that's OK. > > > > We do agree it's interesting to explore further. Since last time we > > spoke the patch has been improved further and I think it's viable. It > > supports the three main platforms and passes the basic jtreg testing > > (including the added test; see below for the GHA runs). So if we were > > to contribute this into a shared repo, which one should this go? a) > > openjdk/jdk (a.k.a JDK mainline) or b) leyden/jdk c) something else? > > Would you have any thoughts as to which repo would be most suitable at > > this point to discuss a PR? > > > > We are reasonably confident that it works well for a majority of use- > > cases (also outside of leyden) so it might be useful to have in > > mainline JDK. But maybe it makes sense to first add it to leyden? I'm > > not sure. In the leyden context it would be a great first building > > block to further explore composability with a simple enough patch[1]. > > Consider: > > > > j_A -> j_B -> j_C -> j_D > > > > where j_A is a full jlink from jmods (produced during the OpenJDK > > build), and then j_B..j_D would be further reductions (condensor runs > > if you will). > > > > Happy to document the caveats of this new jmod-less mode in a CSR if > > that helps. IMHO, there aren't that many in practice. > > I think a nice property to have would be for a jlink command without > packaged modules generate an identical run-time image as it would with > the same command when using the packaged modules. OK. Looking at java.se I get this (the jimage ends up different, but that's it): $ ./build/linux-x86_64-server-release/images/jdk/bin/jlink --add-modules java.se --output build/jmod-full-jlink-java.se --verbose $ ./build/linux-x86_64-server-release/images/jdk/bin/jlink --add-modules java.se,jdk.jlink --output build/jmod-less-jlink1 --verbose $ ./build/jmod-less-jlink1/bin/jlink --add-modules java.se --output build/jmod-less-jlink-java.se --verbose $ diff -r -u ./build/jmod-less-jlink-java.se/ jmod-full-jlink-java.se/ Binary files ./build/jmod-less-jlink-java.se/lib/modules and build/jmod-full-jlink-java.se/lib/modules differ $ ./build/linux-x86_64-server-release/images/jdk/bin/jimage extract --dir build/jimage-java.se-jmodfull ./build/jmod-full-jlink-java.se/lib/modules $ ./build/linux-x86_64-server-release/images/jdk/bin/jimage extract --dir build/jimage-java.se-jmodless ./build/jmod-less-jlink-java.se/lib/modules $ diff -r -u build/jimage-java.se-jmodfull build/jimage-java.se-jmodless $ diff -u <(./build/linux-x86_64-server-release/images/jdk//bin/jimage info build/jmod-less-jlink-java.se/lib/modules) <(./build/linux-x86_64-server-release/images/jdk//bin/jimage info build/jmod-full-jlink-java.se/lib/modules) --- /dev/fd/63 2023-06-16 14:45:48.800145724 +0200 +++ /dev/fd/62 2023-06-16 14:45:48.801145733 +0200 @@ -5,6 +5,6 @@ Table Length: 19474 Offsets Size: 77896 Redirects Size: 77896 - Locations Size: 391737 + Locations Size: 391733 Strings Size: 446073 - Index Size: 993630 + Index Size: 993626 So this might be a side-effect of how the jimage is serialized to disk. I'll keep looking but it seems the patch is already fairly close? > In my view at least, > if it can't guarantee to generate the same run-time image then it should > fail. I can't tell how close you are to this but it means being able to > reconstitute the original resources or detect that they have changed. > I'm not sure what you mean by reconstitute the original resources? Could you clarify, please? Given an unmodified JDK image, a jmod-less jlink will preserve the resources (doesn't change them). Is the idea that if somebody, say changed 'conf/net.properties' a subsequent jmod-less jlink should fail? Something like: T_1: jlink --add-modules ALL-MODULE-PATH --output build/jmod-less-jlink-all-mods T_2: Edit build/jmod-less-jlink-all-mods/conf/net.properties T_3: ./build/jmod-less-all-mods/bin/jlink --add-modules java.se --output build/jmod-less-modified-net Make the jlink at time T_3 fail? Is that the idea? Something else? > So I think it means plugins to un-do transformations where feasible, maybe > hashes to detect changes to resources. If we had that they I think it > would be a useful change to have in the main line. Glad to hear! Detecting modifications to files not in the jimage seems doable. Although, I'm less sure what transformation you think of to un-do?! > Clearly if there is > stripping of native debug symbols, and some other transformations, then > you aren't going to be able to reconstitute the original natives with > debug info. So if someone were to create a run-image with stripping and > including jdk.jlink module then the resulting run-image could not be > used to stamp down further run-images without specifying the location of > the packaged modules. Sounds good. > I'm less sure about connection to Leyden and condensers. Fair enough. Since leyden inherits from mainline that's a non-issue. Cheers, Severin From Alan.Bateman at oracle.com Sat Jun 17 16:05:16 2023 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 17 Jun 2023 17:05:16 +0100 Subject: jmods-less jlinking prototype In-Reply-To: <145f250bd9b2584e27191db6939910126af6b1ff.camel@redhat.com> References: <5a81ad40b9b20d92a73ddedf437187c7f141f9ce.camel@redhat.com> <7c67f4f2-a794-8852-5aca-8bb4055803b5@oracle.com> <4a2cb94a868dac427e2e33997f406f599b9018f3.camel@redhat.com> <98802f4a-82e0-1331-cb1e-543b0b22650b@oracle.com> <145f250bd9b2584e27191db6939910126af6b1ff.camel@redhat.com> Message-ID: <5c157608-d4bb-9de7-a492-dc85b8b9dc9e@oracle.com> On 16/06/2023 13:48, Severin Gehwolf wrote: > : > OK. Looking at java.se I get this (the jimage ends up different, but > that's it): > > $ ./build/linux-x86_64-server-release/images/jdk/bin/jlink --add-modules java.se --output build/jmod-full-jlink-java.se --verbose > $ ./build/linux-x86_64-server-release/images/jdk/bin/jlink --add-modules java.se,jdk.jlink --output build/jmod-less-jlink1 --verbose > $ ./build/jmod-less-jlink1/bin/jlink --add-modules java.se --output build/jmod-less-jlink-java.se --verbose > $ diff -r -u ./build/jmod-less-jlink-java.se/ jmod-full-jlink-java.se/ > Binary files ./build/jmod-less-jlink-java.se/lib/modules and build/jmod-full-jlink-java.se/lib/modules differ > $ ./build/linux-x86_64-server-release/images/jdk/bin/jimage extract --dir build/jimage-java.se-jmodfull ./build/jmod-full-jlink-java.se/lib/modules > $ ./build/linux-x86_64-server-release/images/jdk/bin/jimage extract --dir build/jimage-java.se-jmodless ./build/jmod-less-jlink-java.se/lib/modules > $ diff -r -u build/jimage-java.se-jmodfull build/jimage-java.se-jmodless > > > $ diff -u <(./build/linux-x86_64-server-release/images/jdk//bin/jimage info build/jmod-less-jlink-java.se/lib/modules) <(./build/linux-x86_64-server-release/images/jdk//bin/jimage info build/jmod-full-jlink-java.se/lib/modules) > --- /dev/fd/63 2023-06-16 14:45:48.800145724 +0200 > +++ /dev/fd/62 2023-06-16 14:45:48.801145733 +0200 > @@ -5,6 +5,6 @@ > Table Length: 19474 > Offsets Size: 77896 > Redirects Size: 77896 > - Locations Size: 391737 > + Locations Size: 391733 > Strings Size: 446073 > - Index Size: 993630 > + Index Size: 993626 > > > So this might be a side-effect of how the jimage is serialized to disk. > I'll keep looking but it seems the patch is already fairly close? Reproducible builds have been a bit of whack-a-mole that has included the image writer. I'm not aware of any remaining issues so what you are seeing may be the ordering that strings are hashed or it may be be contents (use `jimage extract --dir lib/modules` and compare). > : > I'm not sure what you mean by reconstitute the original resources? > Could you clarify, please? > > Given an unmodified JDK image, a jmod-less jlink will preserve the > resources (doesn't change them). > > Is the idea that if somebody, say changed 'conf/net.properties' a > subsequent jmod-less jlink should fail? Something like: > > T_1: jlink --add-modules ALL-MODULE-PATH --output build/jmod-less-jlink-all-mods > T_2: Edit build/jmod-less-jlink-all-mods/conf/net.properties > T_3: ./build/jmod-less-all-mods/bin/jlink --add-modules java.se --output build/jmod-less-modified-net > > Make the jlink at time T_3 fail? Is that the idea? Something else? The SystemModulesPlugin is one of complicated plugins as it replaces SystemModulesMap and generates classes that are specific to the set of modules that go into the target runtime. If the linking is "as if" the packaged modules are present then the resources that go through the pipeline shouldn't include the generated/modified classes. If T_3 is changed to generate a run-image with a subset of the modules then you'll see what I mean. In T_1, the target run-time image will include the jdk.jlink module. I think that is your signal to identify the resources that have been added, removed or modified in the plugin pipeline. There are many ways that the diffs can be expressed and saved so that the resources in the modules that go through the pipeline in T_3 are "as if" they came from the packaged modules used in T_1. A motivating example for user editable configuration is where conf/security/java.security at T_2 is changed to allow weak crypto. That's not something to silently propagate. It could be that T_3 fails because the hashes don't match or it has saved the original conf resource. For the native libs then I think the ModuleFinder can detect that the hashes have changed. There's a choice here on whether it just warns or fails. As you know, the only plugin that modifies native libs right now is the debug symbol stripping. There is also the VM selection plugin but that just works as a filter. -Alan From sgehwolf at redhat.com Tue Jun 20 12:16:26 2023 From: sgehwolf at redhat.com (Severin Gehwolf) Date: Tue, 20 Jun 2023 14:16:26 +0200 Subject: jmods-less jlinking prototype In-Reply-To: <5c157608-d4bb-9de7-a492-dc85b8b9dc9e@oracle.com> References: <5a81ad40b9b20d92a73ddedf437187c7f141f9ce.camel@redhat.com> <7c67f4f2-a794-8852-5aca-8bb4055803b5@oracle.com> <4a2cb94a868dac427e2e33997f406f599b9018f3.camel@redhat.com> <98802f4a-82e0-1331-cb1e-543b0b22650b@oracle.com> <145f250bd9b2584e27191db6939910126af6b1ff.camel@redhat.com> <5c157608-d4bb-9de7-a492-dc85b8b9dc9e@oracle.com> Message-ID: <54b562f602e428e6b2fd8f3025f863df031a986d.camel@redhat.com> On Sat, 2023-06-17 at 17:05 +0100, Alan Bateman wrote: > > On 16/06/2023 13:48, Severin Gehwolf wrote: > > > > : > > > > OK. Looking at java.se I get this (the jimage ends up different, but > > > > that's it): > > > > > > > > $ ./build/linux-x86_64-server-release/images/jdk/bin/jlink --add-modules java.se --output build/jmod-full-jlink-java.se --verbose > > > > $ ./build/linux-x86_64-server-release/images/jdk/bin/jlink --add-modules java.se,jdk.jlink --output build/jmod-less-jlink1 --verbose > > > > $ ./build/jmod-less-jlink1/bin/jlink --add-modules java.se --output build/jmod-less-jlink-java.se --verbose > > > > $ diff -r -u ./build/jmod-less-jlink-java.se/? jmod-full-jlink-java.se/ > > > > Binary files ./build/jmod-less-jlink-java.se/lib/modules and build/jmod-full-jlink-java.se/lib/modules differ > > > > $ ./build/linux-x86_64-server-release/images/jdk/bin/jimage extract --dir build/jimage-java.se-jmodfull ./build/jmod-full-jlink-java.se/lib/modules > > > > $ ./build/linux-x86_64-server-release/images/jdk/bin/jimage extract --dir build/jimage-java.se-jmodless ./build/jmod-less-jlink-java.se/lib/modules > > > > $ diff -r -u build/jimage-java.se-jmodfull build/jimage-java.se-jmodless > > > > > > > > > > > > $ diff -u <(./build/linux-x86_64-server-release/images/jdk//bin/jimage info build/jmod-less-jlink-java.se/lib/modules) <(./build/linux-x86_64-server-release/images/jdk//bin/jimage info build/jmod-full-jlink-java.se/lib/modules) > > > > --- /dev/fd/63??2023-06-16 14:45:48.800145724 +0200 > > > > +++ /dev/fd/62??2023-06-16 14:45:48.801145733 +0200 > > > > @@ -5,6 +5,6 @@ > > > > ?? Table Length:?? 19474 > > > > ?? Offsets Size:?? 77896 > > > > ?? Redirects Size: 77896 > > > > - Locations Size: 391737 > > > > + Locations Size: 391733 > > > > ?? Strings Size:?? 446073 > > > > - Index Size:???? 993630 > > > > + Index Size:???? 993626 > > > > > > > > > > > > So this might be a side-effect of how the jimage is serialized to disk. > > > > I'll keep looking but it seems the patch is already fairly close? > > Reproducible builds have been a bit of whack-a-mole that has included > > the image writer. I'm not aware of any remaining issues so what you are > > seeing may be the ordering that strings are hashed or it may be be > > contents (use `jimage extract --dir lib/modules` and compare). > > I've done the 'jimage extract' compare already (see above commands) and they're identical. So it's probably strings ordering. > > > > > > : > > > > I'm not sure what you mean by reconstitute the original resources? > > > > Could you clarify, please? > > > > > > > > Given an unmodified JDK image, a jmod-less jlink will preserve the > > > > resources (doesn't change them). > > > > > > > > Is the idea that if somebody, say changed 'conf/net.properties' a > > > > subsequent jmod-less jlink should fail? Something like: > > > > > > > > T_1: jlink --add-modules ALL-MODULE-PATH --output build/jmod-less-jlink-all-mods > > > > T_2: Edit build/jmod-less-jlink-all-mods/conf/net.properties > > > > T_3: ./build/jmod-less-all-mods/bin/jlink --add-modules java.se --output build/jmod-less-modified-net > > > > > > > > Make the jlink at time T_3 fail? Is that the idea? Something else? > > The SystemModulesPlugin is one of complicated plugins as it replaces > > SystemModulesMap and generates classes that are specific to the set of > > modules that go into the target runtime. If the linking is "as if" the > > packaged modules are present then the resources that go through the > > pipeline shouldn't include the generated/modified classes. If T_3 is > > changed to generate a run-image with a subset of the modules then you'll > > see what I mean. OK, thanks. Noted. > > In T_1, the target run-time image will include the jdk.jlink module. I > > think that is your signal to identify the resources that have been > > added, removed or modified in the plugin pipeline. There are many ways > > that the diffs can be expressed and saved so that the resources in the > > modules that go through the pipeline in T_3 are "as if" they came from > > the packaged modules used in T_1. > > > > A motivating example for user editable configuration is where > > conf/security/java.security at T_2 is changed to allow weak crypto. > > That's not something to silently propagate. It could be that T_3 fails > > because the hashes don't match or it has saved the original conf resource. OK, got it. > > For the native libs then I think the ModuleFinder can detect that the > > hashes have changed. There's a choice here on whether it just warns or > > fails. As you know, the only plugin that modifies native libs right now > > is the debug symbol stripping. There is also the VM selection plugin but > > that just works as a filter. Sounds good. I'll go for producing warnings first. Thanks, Severin From heidinga at redhat.com Mon Jun 26 15:18:46 2023 From: heidinga at redhat.com (Dan Heidinga) Date: Mon, 26 Jun 2023 11:18:46 -0400 Subject: jlink: pre-generate lambda classes Message-ID: I've updated my jlink patch to pregenerate lambda classes [0] so it works with the current Leyden repo and the updated classfile api (which is awesome - check it out if you haven't yet!). Previously discussed in the "Bytecode transformation investigation" thread [1]. I'd like to push it to an experimental branch here on the leyden repo so that others can freely pick up and experiment with the code. Would that be possible? My local branch is called "pregenerate-lambdas" but any name would work for me. The patch pre-generates during jlink time classes for non-Serializable lambda expressions and uses NestMates to handle access to the private implementation method. It's a good example of the kinds of changes needed to shift runtime-generated classes to jlink time. Still lots to update but worth getting into Leyden repo so the code is shareable. Mark / Brian - is it possible to get an experimental branch on the leyden repo for this patch? --Dan [0] https://github.com/openjdk/leyden/compare/master...DanHeidinga:leyden:pregenerate-lambdas?expand=1 [1] https://mail.openjdk.org/pipermail/leyden-dev/2022-August/000055.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Mon Jun 26 16:00:29 2023 From: brian.goetz at oracle.com (Brian Goetz) Date: Mon, 26 Jun 2023 12:00:29 -0400 Subject: jlink: pre-generate lambda classes In-Reply-To: References: Message-ID: Is there any mechanical impediment to pushing to an experimental branch (e.g., committer status, repo config), or are you just asking about how to coordinate work in progress?? If the latter, creating experimental branches is fair game, and clear naming and communication is really the only requirement. On 6/26/2023 11:18 AM, Dan Heidinga wrote: > I've updated my jlink patch to pregenerate lambda classes [0] so it > works with the current Leyden repo and the updated?classfile api > (which is awesome - check it out if you haven't yet!).? Previously > discussed in the "Bytecode transformation investigation" thread [1]. > > I'd like to push it to an experimental branch here on the leyden repo > so that others can freely pick?up and experiment with the code.? Would > that be possible? > > My local branch is called "pregenerate-lambdas" but any name would > work for me. > > The patch pre-generates during jlink time classes for non-Serializable > lambda expressions and uses NestMates to handle access to the private > implementation method.? It's a good example?of the kinds of changes > needed to shift runtime-generated classes to jlink time. > > Still lots to update but worth getting into Leyden repo so the code is > shareable. > > Mark / Brian - is it possible to get an experimental branch on the > leyden repo for this patch? > > --Dan > > [0] > https://github.com/openjdk/leyden/compare/master...DanHeidinga:leyden:pregenerate-lambdas?expand=1 > > [1] https://mail.openjdk.org/pipermail/leyden-dev/2022-August/000055.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.reinhold at oracle.com Mon Jun 26 16:06:58 2023 From: mark.reinhold at oracle.com (Mark Reinhold) Date: Mon, 26 Jun 2023 16:06:58 +0000 Subject: jlink: pre-generate lambda classes In-Reply-To: References: Message-ID: <20230626120655.731324181@eggemoggin.niobe.net> 2023/6/26 12:00:29 -0400, brian.goetz at oracle.com: > Is there any mechanical impediment to pushing to an experimental branch > (e.g., committer status, repo config), The Leyden repo exists, but requires further configuration before it?s ready for use. I?ll look into it ... > or are you just asking about how > to coordinate work in progress? If the latter, creating experimental > branches is fair game, and clear naming and communication is really the > only requirement. Agreed. - Mark From heidinga at redhat.com Mon Jun 26 16:09:16 2023 From: heidinga at redhat.com (Dan Heidinga) Date: Mon, 26 Jun 2023 12:09:16 -0400 Subject: jlink: pre-generate lambda classes In-Reply-To: References: Message-ID: On Mon, Jun 26, 2023 at 12:00?PM Brian Goetz wrote: > Is there any mechanical impediment to pushing to an experimental branch > (e.g., committer status, repo config), > I don't have access / permission to create experimental branches on the leyden repo. So yes to mechanical impediment. > or are you just asking about how to coordinate work in progress? If the > latter, creating experimental branches is fair game, and clear naming and > communication is really the only requirement. > And even if I had access, I wanted to confirm experimental branches were OK. Not looking to step out of bounds here =) --Dan > > > > On 6/26/2023 11:18 AM, Dan Heidinga wrote: > > I've updated my jlink patch to pregenerate lambda classes [0] so it works > with the current Leyden repo and the updated classfile api (which is > awesome - check it out if you haven't yet!). Previously discussed in the > "Bytecode transformation investigation" thread [1]. > > I'd like to push it to an experimental branch here on the leyden repo so > that others can freely pick up and experiment with the code. Would that be > possible? > > My local branch is called "pregenerate-lambdas" but any name would work > for me. > > The patch pre-generates during jlink time classes for non-Serializable > lambda expressions and uses NestMates to handle access to the private > implementation method. It's a good example of the kinds of changes needed > to shift runtime-generated classes to jlink time. > > Still lots to update but worth getting into Leyden repo so the code is > shareable. > > Mark / Brian - is it possible to get an experimental branch on the leyden > repo for this patch? > > --Dan > > [0] > https://github.com/openjdk/leyden/compare/master...DanHeidinga:leyden:pregenerate-lambdas?expand=1 > > [1] https://mail.openjdk.org/pipermail/leyden-dev/2022-August/000055.html > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jianglizhou at google.com Mon Jun 26 23:46:10 2023 From: jianglizhou at google.com (Jiangli Zhou) Date: Mon, 26 Jun 2023 16:46:10 -0700 Subject: Runtime fixes/enhancements for static JDK work Message-ID: Hello everyone, I'd like to get input on some runtime fixes and enhancements [1] for the static JDK support. The changes [1] include part of our hermetic Java [2] prototype work towards more complete support for static JDK, based on the existing JEP 178 [3] work. The changes mainly include the following besides some miscellaneous small fixes: 1) Support unique names for JNI_OnLoad|JNI_OnUnload|Agent_OnLoad|Agent_OnUnload|Agent_OnAttach in JDK JNI libraries by default. For both dynamic JDK and static JDK, this is to use unique symbols for JNI_OnLoad function and friends. The JNI_OnLoad|JNI_OnUnload|Agent_OnLoad|Agent_OnUnload|Agent_OnAttach naming is handled by existing macros in src/java.base/share/native/libjava/jni_util.h without additional changes. Non-builtin application JNI libraries can still have the default JNI_OnLoad|... naming. We've found that it is useful for testing during the transition from dynamic JDK to static JDK. 2) Use runtime checks to determine if the current execution is running on static JDK or dynamic JDK. This allows the removal of the STATIC_BUILD macro usages. With all STATIC_BUILD macro usages being cleaned up, the same set of .o files produced by JDK build can be used to create the shared libraries and static libraries (used for creating the final static image). That seems to be a cleaner approach. The changes [1] in the github branch currently have not cleaned up all STATIC_BUILD macro usages yet. The number of usages [4] in (non-test) JDK and hotspot is small (only in 11 files). We've already recently integrated some of the static JDK related changes, including duplicate symbol fixes and build changes for static libraries. The main tracking bug is JDK-8303796 [5] currently. We'd like your thoughts and opinions on if a new JEP would be needed/useful for the additional runtime fixes/enhancement. The question has been previously raised by Mark in JDK-8307858 [6] comments. NOTES: - For OS specific changes, the github branch currently only contains changes for unix/linux port. - I only ran HelloWorld using the bin/javastatic binary built from the branch as a sanity test. I haven't run other tests with the changes on top of the latest JDK head after porting (some of the porting included some rework). To build and try bin/javastatic on linux_x64: $ bash configure --with-boot-jdk= --with-static-java=yes $ make static-java-image Best, Jiangli [1] https://github.com/openjdk/jdk/compare/master...jianglizhou:jdk:static-java [2] https://mail.openjdk.org/pipermail/leyden-dev/2023-February/000107.html [3] https://openjdk.org/jeps/178 [4] https://github.com/search?q=repo%3Aopenjdk%2Fjdk+STATIC_BUILD+NOT+path%3A%22test%2F%22+NOT+path%3A%22make%2F%22&type=code [5] https://bugs.openjdk.org/browse/JDK-8303796 [6] https://bugs.openjdk.org/browse/JDK-8307858 From Alan.Bateman at oracle.com Tue Jun 27 13:06:58 2023 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 27 Jun 2023 14:06:58 +0100 Subject: Runtime fixes/enhancements for static JDK work In-Reply-To: References: Message-ID: <9cf84c23-d4db-412d-c4d4-c088f4800ace@oracle.com> On 27/06/2023 00:46, Jiangli Zhou wrote: > : > > We've already recently integrated some of the static JDK related > changes, including duplicate symbol fixes and build changes for static > libraries. The main tracking bug is JDK-8303796 [5] currently. We'd > like your thoughts and opinions on if a new JEP would be needed/useful > for the additional runtime fixes/enhancement. The question has been > previously raised by Mark in JDK-8307858 [6] comments. As I see it, this patch gets you to the point of creating bin/javastatic containing all native code needed for headless (and jpackage-less) use on Linux. It's as you say, something that someone working on the JDK could use for testing, it's not something that suitable for distribution as it's incomplete. It's probably next steps where a JEP will be important, maybe several JEPs depending on far it goes and whether the effort is split into into coherent steps. There are many reasons why a JEP (or JEPs) will be needed. It's a significant move to produce and consume a self-contained static image. Today, the JDK image is zipped up or wrapped in an installer for developers to unzip or install. It's a big decision for producers of builds as to whether they will publish static images as alternative downloads. For developers, or other consumers of builds, they it's a hugely significant packaging/deployment option for developers. A JEP for next steps will also have to answer a lot of questions about packaging and the deployment experience. Will a static image include all standard + JDK modules, how will be integrate with jlink??? Can developers use static image "as is" or do they have to run a tool that combines their modular or non-modular application with the static image. Is jpackage in this story? There are many topics to discuss here. A JEP is a great place to capture topics on all the issues that arise from moving to a self containing file. Can jpackage or some other tool change the user-editable configuration when combining the output from the JDK build? What should code do if it re-execs itself with ${java.home}/bin/java? Can a self-contained image be monitored/management when the troubleshooting/other tools don't exist. There are many more questions like this that a JEP could answer. -Alan From jianglizhou at google.com Tue Jun 27 22:56:06 2023 From: jianglizhou at google.com (Jiangli Zhou) Date: Tue, 27 Jun 2023 15:56:06 -0700 Subject: Runtime fixes/enhancements for static JDK work In-Reply-To: <9cf84c23-d4db-412d-c4d4-c088f4800ace@oracle.com> References: <9cf84c23-d4db-412d-c4d4-c088f4800ace@oracle.com> Message-ID: Hi Alan, Thanks a lot for the feedback. Please see some comments inline below. On Tue, Jun 27, 2023 at 6:07?AM Alan Bateman wrote: > > On 27/06/2023 00:46, Jiangli Zhou wrote: > > : > > > > We've already recently integrated some of the static JDK related > > changes, including duplicate symbol fixes and build changes for static > > libraries. The main tracking bug is JDK-8303796 [5] currently. We'd > > like your thoughts and opinions on if a new JEP would be needed/useful > > for the additional runtime fixes/enhancement. The question has been > > previously raised by Mark in JDK-8307858 [6] comments. > > As I see it, this patch gets you to the point of creating bin/javastatic > containing all native code needed for headless (and jpackage-less) use > on Linux. It's as you say, something that someone working on the JDK > could use for testing, it's not something that suitable for distribution > as it's incomplete. Indeed, the makefile change in the github branch (https://github.com/openjdk/jdk/compare/master...jianglizhou:jdk:static-java) for building the bin/javastatic is for demo and testing purposes (for the JDK static linking capability) only. Your comments above made me realize that I should have been more clear with the earlier questions. For the JDK/VM static linking related changes only (excluding the makefile part) in the github branch, any feedback on if it's still good with integrating them as individual bug fixes and minor enhancements? Or it might be better to have broader conversations on static images and start scoping out potential JEPs first? There are benefits with both ways indeed. > > It's probably next steps where a JEP will be important, maybe several > JEPs depending on far it goes and whether the effort is split into into > coherent steps. There are many reasons why a JEP (or JEPs) will be > needed. It's a significant move to produce and consume a self-contained > static image. Today, the JDK image is zipped up or wrapped in an > installer for developers to unzip or install. It's a big decision for > producers of builds as to whether they will publish static images as > alternative downloads. For developers, or other consumers of builds, > they it's a hugely significant packaging/deployment option for developers. > > A JEP for next steps will also have to answer a lot of questions about > packaging and the deployment experience. Will a static image include all > standard + JDK modules, how will be integrate with jlink? Can > developers use static image "as is" or do they have to run a tool that > combines their modular or non-modular application with the static image. > Is jpackage in this story? There are many topics to discuss here. > > A JEP is a great place to capture topics on all the issues that arise > from moving to a self containing file. Can jpackage or some other tool > change the user-editable configuration when combining the output from > the JDK build? What should code do if it re-execs itself with > ${java.home}/bin/java? Can a self-contained image be > monitored/management when the troubleshooting/other tools don't exist. > There are many more questions like this that a JEP could answer. Agreed, as you pointed out there are many open questions that need to be discussed. In the hermetic Java prototype, some of the questions (e.g. what we need to do with the tools that are normally packaged in JDK binary, etc) are left open. For some of the questions, we have only high-level thoughts, e.g. can the static image/executable be re-exec'ed in places where ${java.home}/bin/java would be used normally? To help make the potential JEP(s) successful, it's possibly a good time to start regular discussion meetings on scoping out more details including considerations on jlink part, as you and Ron have suggested. Best, Jiangli > > -Alan