From robin.bygrave at gmail.com Mon Apr 17 13:14:21 2023 From: robin.bygrave at gmail.com (Rob Bygrave) Date: Tue, 18 Apr 2023 01:14:21 +1200 Subject: provides and requires static ... runtime error Message-ID: I'm seeing a runtime error I was not expecting. *Error occurred during initialization of boot layerjava.lang.module.ResolutionException: Module io.avaje.config does not read a module that exports io.avaje.inject.spi* The module io.avaje.config has a requires static io.avaje.inject , and that io.avaje.inject module exports the interface used in the provides clause. module io.avaje.config { ... requires static io.avaje.inject; ... provides io.avaje.inject.spi.PropertyRequiresPlugin with io.avaje.config.InjectPropertiesPlugin; } When a module is run with module-path without the io.avaje.inject module then the error above occurs at runtime when the module is run using Java 20 (build 20+36-2344). So reading https://docs.oracle.com/javase/9/docs/api/java/lang/module/package-summary.html We see: *If any of the following conditions occur in the readability graph, then resolution fails * *- **A module M declares that it 'uses p.S' or 'provides p.S with ...' but package p is neither in module M nor exported to M by any module that M "reads"* So requires *static* io.avaje.inject means it does not "read" io.avaje.inject. To me that suggests we are effectively unable to use requires *static* for a provides type (it will compile successfully but we will observe runtime errors if the optional module is indeed not in the module-path). Have I missed something obvious? Is there any way to use a requires static with the provides type/package to provide a service that can optionally be used (based on if the module that exports the provides type is in the module path)? This works fine with classpath obviously. Thanks, Rob. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Mon Apr 17 13:23:40 2023 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 17 Apr 2023 14:23:40 +0100 Subject: provides and requires static ... runtime error In-Reply-To: References: Message-ID: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> On 17/04/2023 14:14, Rob Bygrave wrote: > I'm seeing a runtime error I was not expecting. > > /Error occurred during initialization of boot layer > java.lang.module.ResolutionException: Module io.avaje.config does not > read a module that exports io.avaje.inject.spi/ > JDK-8299504 [1] captures a number of comments on this scenario. -Alan [1] https://bugs.openjdk.org/browse/JDK-8299504 -------------- next part -------------- An HTML attachment was scrubbed... URL: From robin.bygrave at gmail.com Mon Apr 17 14:20:19 2023 From: robin.bygrave at gmail.com (Rob Bygrave) Date: Tue, 18 Apr 2023 02:20:19 +1200 Subject: provides and requires static ... runtime error In-Reply-To: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> Message-ID: *> really a request to allow more usage of `requires static` and specifically to allow `provides p.S with ...` * Yes, that's my case. *> If Foo isn't available, then (so the thinking goes) the `provides` is moot, and the implementation's dependence on Foo should be ignored.* Yes!!. *> However, this desire is not clearly communicated by the implementation module expressing `requires static ` -- the interface-module could export many packages, not just one package containing one interface Foo, so the `requires static` is not flagging anything about the service interface per se.* Hmmm, my thinking was going the other direction. With the provides *p.S* uses ... the p.S is *ONLY* visible to the module at compile time via the requires *static *[so at compile time we can infer that the p.S type is optional and might not exist at runtime based on the requires *static*]. *> Optional dependencies are meant to support annotations and other limited use cases.* Ok ... umm. *> All of this boils down to: if the implementation module says `provides`, then is it reasonable to consider that ServiceLoader is the *only* vector by which the implementation class will be instantiated? If yes, then module resolution should perhaps be tolerant of a `provides` that specifies a missing service interface. If no ...* Yes, I follow this and agree. > *is it reasonable to consider that ServiceLoader is the *only* vector by which the implementation class will be instantiated? * In my view this is expected and imo I get there by thinking in the opposite direction from the provides p.S type to the requires static rather than the other way around. That is, the p.S type is only available via requires static hence it is expected to potentially not exist at runtime in the module-path. That is, if p.S was expected to exist at runtime it would be "read" via a requires or requires transient clause and not via a requires static - *the use of requires static for this case is explicit and intentional*. In using requires static ... imo we are explicitly going out-of-our-way to say "the types here might not be available at runtime" and the classic case for this as I see it is this case of providing an optional service, that will only be service loaded if the user of that service is in the classpath / module-path. *IF* the module that is the user of a service is in the classpath / module-path then that module will ensure that the p.S type is in the module-path. Cheers, Rob. Would a real world example help? - avaje-inject has a Plugin interface, and will look to load Plugin implementations at runtime. - avaje-config is a module that has its own functions (external configuration) but can also provide a Plugin for avaje-inject (provide external configuration for dependency injection). - avaje-jsonb is a module that has its own functions (json marshalling) but can also provide a Plugin for avaje-inject (provide default json marshalling implementation for dependency injection). On Tue, 18 Apr 2023 at 01:23, Alan Bateman wrote: > On 17/04/2023 14:14, Rob Bygrave wrote: > > I'm seeing a runtime error I was not expecting. > > > *Error occurred during initialization of boot layer > java.lang.module.ResolutionException: Module io.avaje.config does not read > a module that exports io.avaje.inject.spi* > > > JDK-8299504 [1] captures a number of comments on this scenario. > > -Alan > > [1] https://bugs.openjdk.org/browse/JDK-8299504 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robin.bygrave at gmail.com Mon Apr 17 23:40:17 2023 From: robin.bygrave at gmail.com (Rob Bygrave) Date: Tue, 18 Apr 2023 11:40:17 +1200 Subject: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> Message-ID: *> Optional dependencies are meant to support annotations and other limited use cases.* FWIW to me *maven* *optional true* dependencies map to requires static ... *true* When we are in the mode of translating from *'classpath only'* to* 'classpath + module-path'* support, whenever we see maven optional dependencies we expect to use requires static. *> A class that `implements` an interface is not an optional dependency situation -- loading the class *always* causing loading of the interface, which will fail, so we make module resolution fail fast to ensure the loading failure never happens.* It is an optional dependency for classpath. In terms of classpath we do not have any fail fast for this, IF something tries to new up the service implementation AND the p.S interface type is not in the classpath we get an exception (with the cause being ClassNotFoundException for p.S). This is expected and all good. With module-path the *module fail fast resolution *actually prevents this pattern / use case that works with classpath from working. Code using this pattern with classpath will no longer work with module-path due to the fail fast resolution error. The effect of the fail fast resolution on p.S is that we have turned an optional dependency (in classpath) into a non-optional dependency (the p.S interface *MUST* be in the module-path). To be clear, the pattern with classpath and maven is that: We have a maven dependency with true that has the p.S interface type. The p.S type exists at compile time so we can compile the implementation of the service interface but we understand that p.S might not exist at runtime as we have explicitly stated that the dependency that has p.S is a maven optional dependency. We have an implementation of p.S that we expect to be optionally created via ServiceLoader. Yes this implementation is only expected to be created by ServiceLoader. We see an exception at runtime if code tries to instantiate the implementation when the p.S type is not in the classpath and that is expected and all good. With module-path this now does not work (due to the fail fast on p.S interface type). With module-path, IF module resolution was changed to tolerate a `provides` that specifies a missing service interface, I'd argue that we'd get the same behaviour that we get with classpath. That is, if this was allowed then if something tried to create a new instance of the implementation we would expect the same behaviour that we see in classpath and the same or similar error would be thrown as what is thrown when running with classpath (cause ClassNotFoundException for p.S). Cheers, Rob. On Tue, 18 Apr 2023 at 02:20, Rob Bygrave wrote: > *> really a request to allow more usage of `requires static` and > specifically to allow `provides p.S with ...` * > > Yes, that's my case. > > > *> If Foo isn't available, then (so the thinking goes) the `provides` is > moot, and the implementation's dependence on Foo should be ignored.* > > Yes!!. > > > *> However, this desire is not clearly communicated by the implementation > module expressing `requires static ` -- the > interface-module could export many packages, not just one package > containing one interface Foo, so the `requires static` is not flagging > anything about the service interface per se.* > > Hmmm, my thinking was going the other direction. With the provides *p.S* > uses ... the p.S is *ONLY* visible to the module at compile time via the > requires *static *[so at compile time we can infer that the p.S type is > optional and might not exist at runtime based on the requires *static*]. > > > *> Optional dependencies are meant to support annotations and other > limited use cases.* > > Ok ... umm. > > > *> All of this boils down to: if the implementation module says > `provides`, then is it reasonable to consider that ServiceLoader is the > *only* vector by which the implementation class will be instantiated? If > yes, then module resolution should perhaps be tolerant of a `provides` that > specifies a missing service interface. If no ...* > > Yes, I follow this and agree. > > > > *is it reasonable to consider that ServiceLoader is the *only* vector > by which the implementation class will be instantiated? * > > In my view this is expected and imo I get there by thinking in the > opposite direction from the provides p.S type to the requires static rather > than the other way around. That is, the p.S type is only available via > requires static hence it is expected to potentially not exist at runtime in > the module-path. That is, if p.S was expected to exist at runtime it would > be "read" via a requires or requires transient clause and not via a > requires static - *the use of requires static for this case is explicit > and intentional*. > > In using requires static ... imo we are explicitly going out-of-our-way to > say "the types here might not be available at runtime" and the classic case > for this as I see it is this case of providing an optional service, that > will only be service loaded if the user of that service is in the classpath > / module-path. *IF* the module that is the user of a service is in the > classpath / module-path then that module will ensure that the p.S type is > in the module-path. > > > Cheers, Rob. > > Would a real world example help? > > - avaje-inject has a Plugin interface, and will look to load Plugin > implementations at runtime. > - avaje-config is a module that has its own functions (external > configuration) but can also provide a Plugin for avaje-inject (provide > external configuration for dependency injection). > - avaje-jsonb is a module that has its own functions (json marshalling) > but can also provide a Plugin for avaje-inject (provide default json > marshalling implementation for dependency injection). > > > > On Tue, 18 Apr 2023 at 01:23, Alan Bateman > wrote: > >> On 17/04/2023 14:14, Rob Bygrave wrote: >> >> I'm seeing a runtime error I was not expecting. >> >> >> *Error occurred during initialization of boot layer >> java.lang.module.ResolutionException: Module io.avaje.config does not read >> a module that exports io.avaje.inject.spi* >> >> >> JDK-8299504 [1] captures a number of comments on this scenario. >> >> -Alan >> >> [1] https://bugs.openjdk.org/browse/JDK-8299504 >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Apr 18 08:05:06 2023 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 18 Apr 2023 09:05:06 +0100 Subject: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> Message-ID: <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> On 17/04/2023 15:20, Rob Bygrave wrote: > /:/ > > > > /is it reasonable to consider that ServiceLoader is the *only* vector > by which the implementation class will be instantiated?/ > > In my view this is expected and imo I get there by thinking in the > opposite direction from the provides p.S type to the requires static > rather than the other way around. That is, the p.S type is only > available via requires static hence it is expected to potentially not > exist at runtime in the module-path. That is, if p.S was expected to > exist at runtime it would be "read" via a requires or requires > transient clause and not via a requires static - /the use of requires > static for this case is explicit and intentional/. > > In using requires static ... imo we are explicitly going > out-of-our-way to say "the types here might not be available at > runtime" and the classic case for this as I see it is this case of > providing an optional service, that will only be service loaded if the > user of that service is in the classpath / module-path. /IF/ the > module that is the user of a service is in the classpath / module-path > then that module will ensure that the p.S type is in the module-path. > In your example, module io.avaje.config declares that it provides an implementation of io.avaje.inject.spi.PropertyRequiresPlugin. There is nothing to connect this to `requires static io.avaje.inject`. The module system would need to search "far and wide" for io.avaje.inject to see if exports io.avaje.inject.spi to io.avaje.config, otherwise there is no way for it to know that the "missing package" is in a module that is not required to be present at run-time. In other words, `requires static io.avaje.inject` does not convey to the module system that io.avaje.inject exports packages with service types. I understand there is a temptation to compare `requires static` with optional dependences in Maven but they are not the same thing. `requires static` is more for use-cases like annotations that do not need to be present at run-time. It could of course be extended but it's a slippery slope that ultimately amounts to giving up on reliability. In this case, it amounts to giving up on a post resolution check and allowing all service providers to have a dangling reference to a service type. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From robin.bygrave at gmail.com Tue Apr 18 10:50:30 2023 From: robin.bygrave at gmail.com (Rob Bygrave) Date: Tue, 18 Apr 2023 22:50:30 +1200 Subject: provides and requires static ... runtime error In-Reply-To: <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> Message-ID: *> `requires static` is more for use-cases like annotations that do not need to be present at run-time* I am aware that annotations have a retention policy but otherwise why are we specifically saying "like annotations" here as opposed to just saying "types"? *> I understand there is a temptation to compare `requires static` with optional dependences in Maven but they are not the same thing. `requires static` is more for use-cases like annotations that do not need to be present at run-time. It could of course be extended but it's a slippery slope that ultimately amounts to giving up on reliability.* Background: I also maintain Ebean ORM which is made up of 20+ modules (+ 3rd party dependencies) and works with classpath and module-path. A quick search for "requires static" there shows me 25 uses of requires static (192 requires clauses in total on the core modules, I'm suggesting this is a non-trivial use of module-path that uses a decent amount of requires static). Of those 25 requires static, 5 of those are dependencies on optional annotations and 20 are optional dependencies that are *NOT* annotations. They all match to maven optional true dependencies (self fulfilling I know). For all these cases requires static has worked exactly as I expect and exactly the same with classpath and module-path. There have been no issues. No issues when the requires static was for annotations and no issues when the requires static was for normal types (not annotations). > *do not need to be present at run-time.* So for ebean orm, it is using [requires static / optional at runtime] types that are both annotations and normal class types and they are all potentially not present at run-time. There is a suggestion there is a slippery slope for the requires static types that are not annotations? I'm wondering why? As in, optional dependencies have been reasonably extensively used and there has been no issue hit here with module-path and requires static and everything has worked as expected*. *Except the issue noted in this thread which imo isn't an issue with requires static. That is, I'm now aware of the issue with ServiceLoader in module-path not working when provides p.S is via requires static (unlike classpath) ... but in my mind that issue isn't really an issue with requires static per say but instead it is an issue with the runtime module resolution (by not allowing *provides p.S* to be optional). > *It could of course be extended * Extended? What is wrong with how requires static works now? How are you suggesting it could be extended and for what purpose? > *but it's a slippery slope that ultimately amounts to giving up on reliability.* Well, optional dependencies by their nature are sometimes dynamically determined at runtime which isn't ideal but also not difficult or new. Can you give an example of what you mean by this? Thanks, Rob. On Tue, 18 Apr 2023 at 20:05, Alan Bateman wrote: > > On 17/04/2023 15:20, Rob Bygrave wrote: > > *:* > > > > *is it reasonable to consider that ServiceLoader is the *only* vector > by which the implementation class will be instantiated? * > > In my view this is expected and imo I get there by thinking in the > opposite direction from the provides p.S type to the requires static rather > than the other way around. That is, the p.S type is only available via > requires static hence it is expected to potentially not exist at runtime in > the module-path. That is, if p.S was expected to exist at runtime it would > be "read" via a requires or requires transient clause and not via a > requires static - *the use of requires static for this case is explicit > and intentional*. > > In using requires static ... imo we are explicitly going out-of-our-way to > say "the types here might not be available at runtime" and the classic case > for this as I see it is this case of providing an optional service, that > will only be service loaded if the user of that service is in the classpath > / module-path. *IF* the module that is the user of a service is in the > classpath / module-path then that module will ensure that the p.S type is > in the module-path. > > > In your example, module io.avaje.config declares that it provides an > implementation of io.avaje.inject.spi.PropertyRequiresPlugin. There is > nothing to connect this to `requires static io.avaje.inject`. The module > system would need to search "far and wide" for io.avaje.inject to see if > exports io.avaje.inject.spi to io.avaje.config, otherwise there is no way > for it to know that the "missing package" is in a module that is not > required to be present at run-time. In other words, `requires static io.avaje.inject` > does not convey to the module system that io.avaje.inject exports > packages with service types. > > I understand there is a temptation to compare `requires static` with > optional dependences in Maven but they are not the same thing. `requires > static` is more for use-cases like annotations that do not need to be > present at run-time. It could of course be extended but it's a slippery > slope that ultimately amounts to giving up on reliability. In this case, it > amounts to giving up on a post resolution check and allowing all service > providers to have a dangling reference to a service type. > > -Alan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robin.bygrave at gmail.com Tue Apr 18 11:18:58 2023 From: robin.bygrave at gmail.com (Rob Bygrave) Date: Tue, 18 Apr 2023 23:18:58 +1200 Subject: jigsaw-dev Digest, Vol 151, Issue 3 In-Reply-To: References: Message-ID: *> In this case, it amounts to giving up on a post resolution check and allowing all service providers to have a dangling reference to a service type.* Noting that the effect of this is also that use of ServiceLoader works differently in module-path compared to classpath for this case. Use of ServiceLoader wrt optional service loading that works in classpath will not work in module-path (resolution error). I come back to Alex's comment in the bug report: *> Alex: All of this boils down to: if the implementation module says `provides`, then is it reasonable to consider that ServiceLoader is the *only* vector by which the implementation class will be instantiated? If yes, then module resolution should perhaps be tolerant of a `provides` that specifies a missing service interface. * Obviously I'm in the *"module resolution should be tolerant of a `provides` that specifies a missing service interface. "* camp. I'd happily trade the runtime resolution error (that fails fast) for the "consistent with classpath" only error if something else tries to instantiate the service (and p.S isn't in module-path). The workarounds at this point look like: - Don't use module-path, just stick to classpath - Re-design, likely extracting out the service interface into its own artifact and make it a hard dependency of the module implementing the service - Don't use ServiceLoader On Tue, 18 Apr 2023 at 22:50, wrote: > Send jigsaw-dev mailing list submissions to > jigsaw-dev at openjdk.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.openjdk.org/mailman/listinfo/jigsaw-dev > or, via email, send a message with subject or body 'help' to > jigsaw-dev-request at openjdk.org > > You can reach the person managing the list at > jigsaw-dev-owner at openjdk.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of jigsaw-dev digest..." > Today's Topics: > > 1. Re: provides and requires static ... runtime error (Alan Bateman) > 2. Re: provides and requires static ... runtime error (Rob Bygrave) > > > > ---------- Forwarded message ---------- > From: Alan Bateman > To: Rob Bygrave > Cc: jigsaw-dev at openjdk.java.net > Bcc: > Date: Tue, 18 Apr 2023 09:05:06 +0100 > Subject: Re: provides and requires static ... runtime error > > On 17/04/2023 15:20, Rob Bygrave wrote: > > *:* > > > > *is it reasonable to consider that ServiceLoader is the *only* vector > by which the implementation class will be instantiated? * > > In my view this is expected and imo I get there by thinking in the > opposite direction from the provides p.S type to the requires static rather > than the other way around. That is, the p.S type is only available via > requires static hence it is expected to potentially not exist at runtime in > the module-path. That is, if p.S was expected to exist at runtime it would > be "read" via a requires or requires transient clause and not via a > requires static - *the use of requires static for this case is explicit > and intentional*. > > In using requires static ... imo we are explicitly going out-of-our-way to > say "the types here might not be available at runtime" and the classic case > for this as I see it is this case of providing an optional service, that > will only be service loaded if the user of that service is in the classpath > / module-path. *IF* the module that is the user of a service is in the > classpath / module-path then that module will ensure that the p.S type is > in the module-path. > > > In your example, module io.avaje.config declares that it provides an > implementation of io.avaje.inject.spi.PropertyRequiresPlugin. There is > nothing to connect this to `requires static io.avaje.inject`. The module > system would need to search "far and wide" for io.avaje.inject to see if > exports io.avaje.inject.spi to io.avaje.config, otherwise there is no way > for it to know that the "missing package" is in a module that is not > required to be present at run-time. In other words, `requires static io.avaje.inject` > does not convey to the module system that io.avaje.inject exports > packages with service types. > > I understand there is a temptation to compare `requires static` with > optional dependences in Maven but they are not the same thing. `requires > static` is more for use-cases like annotations that do not need to be > present at run-time. It could of course be extended but it's a slippery > slope that ultimately amounts to giving up on reliability. In this case, it > amounts to giving up on a post resolution check and allowing all service > providers to have a dangling reference to a service type. > > -Alan > > > > ---------- Forwarded message ---------- > From: Rob Bygrave > To: Alan Bateman > Cc: jigsaw-dev at openjdk.java.net > Bcc: > Date: Tue, 18 Apr 2023 22:50:30 +1200 > Subject: Re: provides and requires static ... runtime error > *> `requires static` is more for use-cases like annotations that do not > need to be present at run-time* > > I am aware that annotations have a retention policy but otherwise why are > we specifically saying "like annotations" here as opposed to just saying > "types"? > > > *> I understand there is a temptation to compare `requires static` with > optional dependences in Maven but they are not the same thing. `requires > static` is more for use-cases like annotations that do not need to be > present at run-time. It could of course be extended but it's a slippery > slope that ultimately amounts to giving up on reliability.* > > Background: I also maintain Ebean ORM which is made up of 20+ modules (+ > 3rd party dependencies) and works with classpath and module-path. A quick > search for "requires static" there shows me 25 uses of requires static (192 > requires clauses in total on the core modules, I'm suggesting this is a > non-trivial use of module-path that uses a decent amount of requires > static). > > Of those 25 requires static, 5 of those are dependencies on optional > annotations and 20 are optional dependencies that are *NOT* annotations. > They all match to maven optional true dependencies (self fulfilling I know). > > For all these cases requires static has worked exactly as I expect and > exactly the same with classpath and module-path. There have been no > issues. No issues when the requires static was for annotations and no > issues when the requires static was for normal types (not annotations). > > > > *do not need to be present at run-time.* > > So for ebean orm, it is using [requires static / optional at runtime] > types that are both annotations and normal class types and they are all > potentially not present at run-time. There is a suggestion there is a > slippery slope for the requires static types that are not annotations? I'm > wondering why? As in, optional dependencies have been reasonably > extensively used and there has been no issue hit here with module-path and > requires static and everything has worked as expected*. > > *Except the issue noted in this thread which imo isn't an issue with > requires static. That is, I'm now aware of the issue with ServiceLoader in > module-path not working when provides p.S is via requires static (unlike > classpath) ... but in my mind that issue isn't really an issue with > requires static per say but instead it is an issue with the runtime module > resolution (by not allowing *provides p.S* to be optional). > > > > *It could of course be extended * > > Extended? What is wrong with how requires static works now? How > are you suggesting it could be extended and for what purpose? > > > > *but it's a slippery slope that ultimately amounts to giving up on > reliability.* > > Well, optional dependencies by their nature are sometimes dynamically > determined at runtime which isn't ideal but also not difficult or new. Can > you give an example of what you mean by this? > > > Thanks, Rob. > > On Tue, 18 Apr 2023 at 20:05, Alan Bateman > wrote: > >> >> On 17/04/2023 15:20, Rob Bygrave wrote: >> >> *:* >> >> >> > *is it reasonable to consider that ServiceLoader is the *only* vector >> by which the implementation class will be instantiated? * >> >> In my view this is expected and imo I get there by thinking in the >> opposite direction from the provides p.S type to the requires static rather >> than the other way around. That is, the p.S type is only available via >> requires static hence it is expected to potentially not exist at runtime in >> the module-path. That is, if p.S was expected to exist at runtime it would >> be "read" via a requires or requires transient clause and not via a >> requires static - *the use of requires static for this case is explicit >> and intentional*. >> >> In using requires static ... imo we are explicitly going out-of-our-way >> to say "the types here might not be available at runtime" and the classic >> case for this as I see it is this case of providing an optional service, >> that will only be service loaded if the user of that service is in the >> classpath / module-path. *IF* the module that is the user of a service >> is in the classpath / module-path then that module will ensure that the p.S >> type is in the module-path. >> >> >> In your example, module io.avaje.config declares that it provides an >> implementation of io.avaje.inject.spi.PropertyRequiresPlugin. There is >> nothing to connect this to `requires static io.avaje.inject`. The module >> system would need to search "far and wide" for io.avaje.inject to see if >> exports io.avaje.inject.spi to io.avaje.config, otherwise there is no >> way for it to know that the "missing package" is in a module that is not >> required to be present at run-time. In other words, `requires static io.avaje.inject` >> does not convey to the module system that io.avaje.inject exports >> packages with service types. >> >> I understand there is a temptation to compare `requires static` with >> optional dependences in Maven but they are not the same thing. `requires >> static` is more for use-cases like annotations that do not need to be >> present at run-time. It could of course be extended but it's a slippery >> slope that ultimately amounts to giving up on reliability. In this case, it >> amounts to giving up on a post resolution check and allowing all service >> providers to have a dangling reference to a service type. >> >> -Alan >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.pressler at oracle.com Tue Apr 18 12:46:47 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Tue, 18 Apr 2023 12:46:47 +0000 Subject: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> Message-ID: <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> An optional service is something that could be considered, but I think there?s a difference in expectations that underlies this entire discussion. Java?s classpath offers a very dynamic model of program configurations where class resolution is *expected* to possibly fail at runtime. This has some disadvantages but also some clear advantages. Modules are not meant to continue the same model by other means, but to offer a very different one, with its own pros and cons. Dialing back Java?s dynamism is their point, and in more ways than one (both strong encapsulation and reliable configuration are about restricting dynamism in favour of more static guarantees). When you configure the application on the command line you already know whether or not you wish to include a module that contains the service interface in the configuration. If you don?t, then don?t include modules that provide that service. There is no need to ?work around? the lesser dynamism. That is a different experience from the classpath?s model of ?I?ll throw some stuff in there, maybe it?s needed maybe it?s not; maybe it?s consistent, maybe not,? but it is different by design. In particular, `requires static` ? intended for compile-time dependencies ? also doesn?t work like the classpath. If you happen to reference a class that exists on the class path it will be resolved, but `requires static` is insufficient to resolve the module. Which makes me wonder, what is the root of the optionality in your code? I.e. how does io.avaje.inject come to be resolved? ? Ron On 18 Apr 2023, at 11:50, Rob Bygrave > wrote: > `requires static` is more for use-cases like annotations that do not need to be present at run-time I am aware that annotations have a retention policy but otherwise why are we specifically saying "like annotations" here as opposed to just saying "types"? > I understand there is a temptation to compare `requires static` with optional dependences in Maven but they are not the same thing. `requires static` is more for use-cases like annotations that do not need to be present at run-time. It could of course be extended but it's a slippery slope that ultimately amounts to giving up on reliability. Background: I also maintain Ebean ORM which is made up of 20+ modules (+ 3rd party dependencies) and works with classpath and module-path. A quick search for "requires static" there shows me 25 uses of requires static (192 requires clauses in total on the core modules, I'm suggesting this is a non-trivial use of module-path that uses a decent amount of requires static). Of those 25 requires static, 5 of those are dependencies on optional annotations and 20 are optional dependencies that are NOT annotations. They all match to maven optional true dependencies (self fulfilling I know). For all these cases requires static has worked exactly as I expect and exactly the same with classpath and module-path. There have been no issues. No issues when the requires static was for annotations and no issues when the requires static was for normal types (not annotations). > do not need to be present at run-time. So for ebean orm, it is using [requires static / optional at runtime] types that are both annotations and normal class types and they are all potentially not present at run-time. There is a suggestion there is a slippery slope for the requires static types that are not annotations? I'm wondering why? As in, optional dependencies have been reasonably extensively used and there has been no issue hit here with module-path and requires static and everything has worked as expected*. *Except the issue noted in this thread which imo isn't an issue with requires static. That is, I'm now aware of the issue with ServiceLoader in module-path not working when provides p.S is via requires static (unlike classpath) ... but in my mind that issue isn't really an issue with requires static per say but instead it is an issue with the runtime module resolution (by not allowing provides p.S to be optional). > It could of course be extended Extended? What is wrong with how requires static works now? How are you suggesting it could be extended and for what purpose? > but it's a slippery slope that ultimately amounts to giving up on reliability. Well, optional dependencies by their nature are sometimes dynamically determined at runtime which isn't ideal but also not difficult or new. Can you give an example of what you mean by this? Thanks, Rob. On Tue, 18 Apr 2023 at 20:05, Alan Bateman > wrote: On 17/04/2023 15:20, Rob Bygrave wrote: : > is it reasonable to consider that ServiceLoader is the *only* vector by which the implementation class will be instantiated? In my view this is expected and imo I get there by thinking in the opposite direction from the provides p.S type to the requires static rather than the other way around. That is, the p.S type is only available via requires static hence it is expected to potentially not exist at runtime in the module-path. That is, if p.S was expected to exist at runtime it would be "read" via a requires or requires transient clause and not via a requires static - the use of requires static for this case is explicit and intentional. In using requires static ... imo we are explicitly going out-of-our-way to say "the types here might not be available at runtime" and the classic case for this as I see it is this case of providing an optional service, that will only be service loaded if the user of that service is in the classpath / module-path. IF the module that is the user of a service is in the classpath / module-path then that module will ensure that the p.S type is in the module-path. In your example, module io.avaje.config declares that it provides an implementation of io.avaje.inject.spi.PropertyRequiresPlugin. There is nothing to connect this to `requires static io.avaje.inject`. The module system would need to search "far and wide" for io.avaje.inject to see if exports io.avaje.inject.spi to io.avaje.config, otherwise there is no way for it to know that the "missing package" is in a module that is not required to be present at run-time. In other words, `requires static io.avaje.inject` does not convey to the module system that io.avaje.inject exports packages with service types. I understand there is a temptation to compare `requires static` with optional dependences in Maven but they are not the same thing. `requires static` is more for use-cases like annotations that do not need to be present at run-time. It could of course be extended but it's a slippery slope that ultimately amounts to giving up on reliability. In this case, it amounts to giving up on a post resolution check and allowing all service providers to have a dangling reference to a service type. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From josiahnoel at gmail.com Tue Apr 18 15:01:03 2023 From: josiahnoel at gmail.com (Josiah Noel) Date: Tue, 18 Apr 2023 11:01:03 -0400 Subject: provides and requires static ... runtime error In-Reply-To: <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> Message-ID: On Tue, Apr 18, 2023 at 8:46?AM Ron Pressler wrote: > > Which makes me wonder, what is the root of the optionality in your code? > I.e. how does io.avaje.inject come to be resolved? > So avaje jsonb/config/http implements SPI interfaces exported by avaje inject(which is added as a maven optional dependency). The idea here is that the plugin implementation would be loaded by avaje-inject to add to the DI scope. Outside of avaje inject, these service implementation classes have no meaning and are not meant to be instantiated. In some cases, the service implementation package may not even be exported by the module, so even if you tried you couldn't instantiate outside of a service loader. Does this help answer your question? Or did I misread it? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.pressler at oracle.com Tue Apr 18 16:05:06 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Tue, 18 Apr 2023 16:05:06 +0000 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> Message-ID: <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> On 18 Apr 2023, at 16:01, Josiah Noel > wrote: On Tue, Apr 18, 2023 at 8:46?AM Ron Pressler > wrote: Which makes me wonder, what is the root of the optionality in your code? I.e. how does io.avaje.inject come to be resolved? So avaje jsonb/config/http implements SPI interfaces exported by avaje inject(which is added as a maven optional dependency). The idea here is that the plugin implementation would be loaded by avaje-inject to add to the DI scope. Outside of avaje inject, these service implementation classes have no meaning and are not meant to be instantiated. In some cases, the service implementation package may not even be exported by the module, so even if you tried you couldn't instantiate outside of a service loader. Does this help answer your question? Or did I misread it? I think so, thank you. But when the application runs in a particular configuration, the application deployer knows whether or not that configuration uses the avaje inject module. If it does not, why is the configuration including the service module? The class path has a loosy-goosey attitude, but modules are all about creating a proper configuration. Ideally, there shouldn?t be any modules that could not possibly be used. We could discuss making modules more laid back in some specific situations, but generally, they exist so that we could have a strict configuration. So I guess my question now is, why would you want to allow a particular application configuration to include a module that could not possibly be used in that configuration? I assume the answer is to avoid the need to document to the user that if they choose to use some capability offered by module X (avaje inject) they should also add module Y (the service provider). If we were more lenient, you could just give them module Y, and then all would work whether or not they choose to use X. But the flip side of that is that if they choose not to use X, Y would just sit there, unused. Best case scenario, it would just increase their image size; worst-case scenario is that it could preclude some future optimisations that may require a full program analysis. To do this properly, as Alan alluded, `requires static` is insufficient because the module doesn?t know where the service interface is supposed to come from. Rather, we would need a new specific mechanism that says ?if module X is readable, then provide this service?. To know how important such a feature is we need to understand the source of the problem: why is it hard to exclude unused modules? ? Ron -------------- next part -------------- An HTML attachment was scrubbed... URL: From robin.bygrave at gmail.com Tue Apr 18 22:00:30 2023 From: robin.bygrave at gmail.com (Rob Bygrave) Date: Wed, 19 Apr 2023 10:00:30 +1200 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> Message-ID: *> source of the problem:* I'll have a crack. As I see it, I think this case is that some modules have "split personalities" in that they don't exist to provide a service as their *main goal* but it happens that they can also additionally provide a service IF they are used in an app configuration that uses and loads implementations of that service [and that service implementation would be left unused if it is used in a standalone configuration that does not include the service interface]. Y (service provider) has a main goal of doing "Y good stuff". It really does not want a *hard* dependency to X, it can be used by itself completely standalone but IF it is used in an application configuration that includes X then ... it additionally happens to provide an implementation of an X service interface. Some people might call this "An optionally provided service" that might not be loaded. e.g. Module X has an interface x.spi.Plugin and will service load all the implementations. Module Y exists to do "Y good stuff", exports y; and can be used completely by itself standalone. An application configuration that only includes Y is good and expected. Module Y can also (as a secondary reason to exist) implement x.spi.Plugin to provide useful functionality to an application configuration that includes Y + X. module y { exports y; // Main reason to exist // As a secondary reason to exist requires static x; provides x.spi.Plugin with ... } Application configuration using just Y standalone fails at startup based on the existence of "provides x.spi.Plugin". *> if they choose not to use X, Y would just sit there, unused.* Y still provides its main reason to exist, it's "Y good stuff". What is not used is that 1 class which is the implementation of x.spi.Plugin (and with module-path this implementation can be hidden / not exported and not open to abuse / accidental usage - yay!!). Fixing a module with a split personality: ----------------------------------------------- We could split the Y module into 2 modules - Y and Y-X-Plugin. The only classes in Y-X-Plugin are the implementation of x.spi.Plugin + module-info with the service loader configuration etc. Now we have 3 application configurations to consider: - Y (now works) - X + Y (might be confusing for users) - X + Y + X-Y-Plugin (needs documentation) What is mooted is to be able to just have the 2 configurations to consider: - Y (standalone) - X + Y (y provides its "Y good stuff" + acts as a x.spi.Plugin for X) The reason why this could be considered acceptable is because the implementation of x.spi.Plugin that would exist in Y can be hidden / not exported and not open to abuse / accidental use. Cheers, Rob. On Wed, 19 Apr 2023 at 04:05, Ron Pressler wrote: > > > On 18 Apr 2023, at 16:01, Josiah Noel wrote: > > On Tue, Apr 18, 2023 at 8:46?AM Ron Pressler > wrote: > >> >> Which makes me wonder, what is the root of the optionality in your code? >> I.e. how does io.avaje.inject come to be resolved? >> > > So avaje jsonb/config/http implements SPI interfaces exported by avaje > inject(which is added as a maven optional dependency). The idea here is > that the plugin implementation would be loaded by avaje-inject to add to > the DI scope. > > Outside of avaje inject, these service implementation classes have no > meaning and are not meant to be instantiated. In some cases, the service > implementation package may not even be exported by the module, so even if > you tried you couldn't instantiate outside of a service loader. > > Does this help answer your question? Or did I misread it? > > > I think so, thank you. But when the application runs in a particular > configuration, the application deployer knows whether or not that > configuration uses the avaje inject module. If it does not, why is the > configuration including the service module? > > The class path has a loosy-goosey attitude, but modules are all about > creating a proper configuration. Ideally, there shouldn?t be any modules > that could not possibly be used. > > We could discuss making modules more laid back in some specific > situations, but generally, they exist so that we could have a strict > configuration. So I guess my question now is, why would you want to allow a > particular application configuration to include a module that could not > possibly be used in that configuration? > > I assume the answer is to avoid the need to document to the user that if > they choose to use some capability offered by module X (avaje inject) they > should also add module Y (the service provider). If we were more lenient, > you could just give them module Y, and then all would work whether or not > they choose to use X. But the flip side of that is that if they choose not > to use X, Y would just sit there, unused. Best case scenario, it would just > increase their image size; worst-case scenario is that it could preclude > some future optimisations that may require a full program analysis. > > To do this properly, as Alan alluded, `requires static` is insufficient > because the module doesn?t know where the service interface is supposed to > come from. Rather, we would need a new specific mechanism that says ?if > module X is readable, then provide this service?. To know how important > such a feature is we need to understand the source of the problem: why is > it hard to exclude unused modules? > > ? Ron > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.pressler at oracle.com Tue Apr 18 22:29:47 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Tue, 18 Apr 2023 22:29:47 +0000 Subject: JEP draft: Disallow the Dynamic Loading of Agents by Default Message-ID: <168352AB-C660-484F-BAA6-31A2B6F0D0C8@oracle.com> Hi. Following last month?s email about changing the default of EnableDynamicAgentLoading [1], we?ve now published two JEP drafts. The first is an informational JEP describing what integrity is and why we need it, and motivates what changes are required to get it (which include the restriction of dynamically loaded agents among others): https://openjdk.org/jeps/8305968 Integrity and Strong Encapsulation The second touches on the specifics of dynamically loaded agents and the proposed change: https://openjdk.org/jeps/8306275 Disallow the Dynamic Loading of Agents by Default [1]: https://mail.openjdk.org/pipermail/jigsaw-dev/2023-March/014816.html ? Ron From josiahnoel at gmail.com Thu Apr 20 13:55:54 2023 From: josiahnoel at gmail.com (Josiah Noel) Date: Thu, 20 Apr 2023 09:55:54 -0400 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> Message-ID: > > The reason why this could be considered acceptable is because the > implementation of x.spi.Plugin that would exist in Y can be hidden / not > exported and not open to abuse / accidental use. > This also helps resolve that concern on the bug report. All of this boils down to: if the implementation module says `provides`, > then is it reasonable to consider that ServiceLoader is the *only* vector > by which the implementation class will be instantiated? If yes, then module > resolution should perhaps be tolerant of a `provides` that specifies a > missing service interface. If no -- that is, we think arbitrary code might > instantiate (hence load) the implementation class -- then module resolution > is correct as-is. if we don't export the implementation class, then the ServiceLoader becomes the only mechanism by which it can be instantiated. In this case, if the user doesn't add the optional interface dependency, it becomes impossible to accidentally load the interface and get a ClassNotFoundException when using module Y's non-SPI packages. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Apr 20 14:27:35 2023 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 20 Apr 2023 15:27:35 +0100 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> Message-ID: <2683d96c-adff-98e2-c8e4-d7be023dcf1d@oracle.com> On 20/04/2023 14:55, Josiah Noel wrote: > : > if we don't export the implementation class, then the > ServiceLoader?becomes the only mechanism by which?it can be > instantiated. In this case, if the user doesn't add the optional > interface dependency, it becomes impossible to accidentally load the > interface and get a ClassNotFoundException when using module Y's > non-SPI packages. > There could be code in the module that references the implementation class, in which case you might get NCDFE at runtime. Requires-static might be okay during initial migration of older code that is okay with dangling references and uses reflection guards. For reliability, it is usually better to use services to implement optionality. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From josiahnoel at gmail.com Thu Apr 20 15:03:20 2023 From: josiahnoel at gmail.com (Josiah Noel) Date: Thu, 20 Apr 2023 11:03:20 -0400 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: <2683d96c-adff-98e2-c8e4-d7be023dcf1d@oracle.com> References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <2683d96c-adff-98e2-c8e4-d7be023dcf1d@oracle.com> Message-ID: > > There could be code in the module that references the implementation > class, in which case you might get NCDFE at runtime. Requires-static might > be okay during initial migration of older code that is okay with dangling > references and uses reflection guards. For reliability, it is usually > better to use services to implement optionality. > > If module Y required module X for normal runtime operation, would not the creator of Y know this and use requires instead of requires static? Since the dependency isn't actually optional at runtime? We're talking about a case where module Y's primary purpose is not to implement SPI from X, but to provide its own packages/functionality unrelated to X. The classes in Y that use X should not be exported, and should only be loaded when X service-loads them. Hence, the Y module will use requires static to clearly inform that the classes are required at build time, but not for Y's normal operation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.buckley at oracle.com Thu Apr 20 18:07:35 2023 From: alex.buckley at oracle.com (Alex Buckley) Date: Thu, 20 Apr 2023 11:07:35 -0700 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <2683d96c-adff-98e2-c8e4-d7be023dcf1d@oracle.com> Message-ID: <6c06f754-ef54-aa43-7a6c-9c7167e4ed58@oracle.com> On 4/20/2023 8:03 AM, Josiah Noel wrote: > We're talking about a case where module Y's primary purpose is not to > implement SPI from X, but to provide its own packages/functionality > unrelated to X. The classes in Y that use X should not be exported, and > should only be loaded when X service-loads?them. Hence, the Y module > will use requires static to clearly inform that the classes are required > at build time, but not for Y's normal operation. We understand the scenario, which involves the module system allowing optionality not only of service providers for a given service, but also of the service itself. However, as Alan has been saying, `requires static` doesn't have the precise, clear meaning that you think. When the module system sees the directive `requires static X;` in some module Z, the module system doesn't know the reason for the directive: - Perhaps code in Z is annotated using annotation interfaces from X. - Perhaps code in Z is going to reflect over the classes in X. - Perhaps code in Z is implementing X's exported interfaces and the owner of Z is really sure that whenever Z is resolved, X is also resolved. (No use of ServiceLoader is intended in this item.) You're going to say, "But that's Z, not Y! Y doesn't do any of those things! Y uses `requires static X;` to get a light touch on a service in X, and Y's service provision should be considered optional if X isn't available." But Y and Z are the same to the module system -- each directs the module system to allow itself to be resolved even if X is not resolved. The module system then sees that Y wants to provide a service, but the module system has no idea where the service is, so fails fast at startup. This is a feature, not a bug. The classpath behavior, of only discovering that a service provider can't provide a service during run time (because the service is missing), is not what we wanted for modular services. If `provides` directives were treated as optional, i.e., ignored when the service can't be found at startup, people would say "Why bother with `provides`? It's no more reliable than putting everything on the classpath, when ServiceLoader throws because it can't instantiate the service provider." So, I hope you can see why we are extremely cautious about acceding to requests to relax the checking of `provides`. In your scenario, Y is pulling double duty, which makes it harder to understand overall, and you'll benefit a lot more from breaking it up than from a more relaxed module system. Alex From josiahnoel at gmail.com Fri Apr 21 14:02:51 2023 From: josiahnoel at gmail.com (Josiah Noel) Date: Fri, 21 Apr 2023 10:02:51 -0400 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: <6c06f754-ef54-aa43-7a6c-9c7167e4ed58@oracle.com> References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <2683d96c-adff-98e2-c8e4-d7be023dcf1d@oracle.com> <6c06f754-ef54-aa43-7a6c-9c7167e4ed58@oracle.com> Message-ID: I see, I believe I fully understand your concerns. It makes the module version of ServiceLoader a lot less useful to me, but I see your main hesitation. In your scenario, Y is > pulling double duty, which makes it harder to understand overall, and > you'll benefit a lot more from breaking it up than from a more relaxed > module system. > Perhaps I'm looking at it from the wrong angle, but from a maintenance perspective, I'm not really seeing the benefits. Say you have 5 artifacts that provide an optional service for X, to support the module-path we now need to maintain 10. It also makes it somewhat cumbersome to add the extra dependencies to use them all with X. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.pressler at oracle.com Fri Apr 21 16:14:33 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Fri, 21 Apr 2023 16:14:33 +0000 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <2683d96c-adff-98e2-c8e4-d7be023dcf1d@oracle.com> <6c06f754-ef54-aa43-7a6c-9c7167e4ed58@oracle.com> Message-ID: I?m guessing that what?s bothering you isn?t so much the number of *modules* but the number of JARs. So I think a more general solution than adding a way to describe ?this service is conditional on the presence of X? would be to allow multiple modules in a single JAR that would contain both Y and the X-Y-plugin. That feature is something we could very much consider (subject to prioritisation, of course). Would you consider that a satisfactory solution? ? Ron P.S. In the meantime, here?s what you could do, and that ? in my opinion ? *is* an acceptable use for `requires static X`: In module Y (which doesn?t include the plugin) rely on `requires static X` to test, at runtime (say in a static initialiser of a class that?s part of Y?s main functionality), whether or not X is available, also whether the plugin is available. If X is available and the plugin isn?t, you could issue a message telling your users: please add the plugin module. On 21 Apr 2023, at 16:02, Josiah Noel > wrote: I see, I believe I fully understand your concerns. It makes the module version of ServiceLoader a lot less useful to me, but I see your main hesitation. In your scenario, Y is pulling double duty, which makes it harder to understand overall, and you'll benefit a lot more from breaking it up than from a more relaxed module system. Perhaps I'm looking at it from the wrong angle, but from a maintenance perspective, I'm not really seeing the benefits. Say you have 5 artifacts that provide an optional service for X, to support the module-path we now need to maintain 10. It also makes it somewhat cumbersome to add the extra dependencies to use them all with X. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josiahnoel at gmail.com Fri Apr 21 19:03:31 2023 From: josiahnoel at gmail.com (Josiah Noel) Date: Fri, 21 Apr 2023 15:03:31 -0400 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <2683d96c-adff-98e2-c8e4-d7be023dcf1d@oracle.com> <6c06f754-ef54-aa43-7a6c-9c7167e4ed58@oracle.com> Message-ID: > > I?m guessing that what?s bothering you isn?t so much the number of > *modules* but the number of JARs. So I think a more general solution than > adding a way to describe ?this service is conditional on the presence of X? > would be to allow multiple modules in a single JAR that would contain both > Y and the X-Y-plugin. > > That feature is something we could very much consider (subject to > prioritisation, of course). Would you consider that a satisfactory solution? > I would be satisfied by this, thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From johannes.spangenberg at hotmail.de Sat Apr 22 00:19:57 2023 From: johannes.spangenberg at hotmail.de (Johannes Spangenberg) Date: Sat, 22 Apr 2023 02:19:57 +0200 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: <6c06f754-ef54-aa43-7a6c-9c7167e4ed58@oracle.com> References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <2683d96c-adff-98e2-c8e4-d7be023dcf1d@oracle.com> <6c06f754-ef54-aa43-7a6c-9c7167e4ed58@oracle.com> Message-ID: Hello, I couldn't quite follow some of the arguments regarding this topic. I wasn't sure if I should interfere, but I hope it will be helpful now that I did. :/ > But the flip side of that is that if they choose not to use X, Y would > just sit there, unused. Best case scenario, it would just increase > their image size; worst-case scenario is that it could preclude some > future optimisations that may require a full program analysis. Not sure about the worst-case. The current implementation already checks during initialization whether the service is usable. After all, it throws an exception when it is not. Any optimization can just ignore the service provider in such case. There is no full program analysis required, which is not already part of every module initialization. > To do this properly, as Alan alluded, `requires static` is > insufficient because the module doesn?t know where the service > interface is supposed to come from. Rather, we would need a new > specific mechanism that says ?if module X is readable, then provide > this service?. Why do you need the information which module provides the service interface? I currently don't see how this information would be helpful. I could also imagine cases where you want to provide an optional service without using `requires static`. For example, imagine a framework introduces a new service interface. You may want to provide an implementation for the service interface while keeping support for previous versions of the framework. > The module system then sees that Y wants to provide a service, but the > module system has no idea where the service is, so fails fast at > startup. This is a feature, not a bug. The classpath behavior, of only > discovering that a service provider can't provide a service during run > time (because the service is missing), is not what we wanted for > modular services. Does this issue really exist? The module system would already discover that during initialization. When the service interface is not available, the service provider will be ignored, there is no error (or anything else) later during runtime. Besides that, if the service interface is not available, you are unable to invoke the service loader anyway. Invoking the service loader requires a class instance of the service interface, which is not available. If you would still try to invoke the service loader, you would get a NCDFE at the call site of the service loader, the provides clause wouldn't make a difference. I am not quite sure about the behavior if you stack multiple module layers on top of each other. I think the service loader should only load instances that implement the class from the same module layer as the class instance given to the service loader. Everything else would cause a runtime error independent of the issue we are currently discussing. In this case, ignoring the provides clause would be the right solution, even if X would be available on another module layer. I currently see only two reasons for keeping the current behavior. The first one is to discourage service providers to be casually added to modules. You could argue that service providers should have their own module, so that users can add and remove them individually. The second reason might be the desire to fail early when the service interface was renamed or removed after updating X. This could be addressed by adding a modifier like `provides static`, there is no need to correlate the provides cluase with a specific `requires` clause. > So I think a more general solution than adding a way to describe ?this > service is conditional on the presence of X? would be to allow > multiple modules in a single JAR I guess that would work as well, but it would probably be more complex to set up. Best regards Johannes From greggwon at cox.net Sat Apr 22 17:55:48 2023 From: greggwon at cox.net (Gregg Wonderly) Date: Sat, 22 Apr 2023 12:55:48 -0500 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <2683d96c-adff-98e2-c8e4-d7be023dcf1d@oracle.com> <6c06f754-ef54-aa43-7a6c-9c7167e4ed58@oracle.com> Message-ID: <7113F589-92F1-4BDB-AF59-DFB7B4A1F165@cox.net> The dependency injection, like this, performed loosely, at runtime, is a problem only when the structure of association is loose. Modularity has injected this detail by suggesting that the ?huge jar file? solution is a problem. Jars inside of Jars and other mechanisms have become forbidden due to licensing issues and other blind attempts at control of ?use? as something that provides value to the owner. Instead of providing sanity, we?ve opened to door to hugely complex paths of reference and association that are nearly impossible to understand and manage for large software systems. > On Apr 21, 2023, at 7:19 PM, Johannes Spangenberg wrote: > > Hello, I couldn't quite follow some of the arguments regarding this topic. I wasn't sure if I should interfere, but I hope it will be helpful now that I did. :/ > >> But the flip side of that is that if they choose not to use X, Y would just sit there, unused. Best case scenario, it would just increase their image size; worst-case scenario is that it could preclude some future optimisations that may require a full program analysis. > > Not sure about the worst-case. The current implementation already checks during initialization whether the service is usable. After all, it throws an exception when it is not. Any optimization can just ignore the service provider in such case. There is no full program analysis required, which is not already part of every module initialization. > >> To do this properly, as Alan alluded, `requires static` is insufficient because the module doesn?t know where the service interface is supposed to come from. Rather, we would need a new specific mechanism that says ?if module X is readable, then provide this service?. > > Why do you need the information which module provides the service interface? I currently don't see how this information would be helpful. I could also imagine cases where you want to provide an optional service without using `requires static`. For example, imagine a framework introduces a new service interface. You may want to provide an implementation for the service interface while keeping support for previous versions of the framework. What I would prefer the most, is a very structured mechanism akin to what .Net?s HostBuilder mechanism provides. Much like a docker-compose.yml file provides the details of how your application environment is assembled, the HostBuilder mechanism provides a very clean mechanism to allow a chained calling structure that would provide the opportunity for injection of services needed. In essence, it really feels liked we need some kind of IModuleContext and ContextBuilder that would provide the ability to inject the things that you intend to use in the way that you need them to be. A configuration file at the manifest level could allow outside specifications, but inside of the ContextBuilder use, there could be explicit choices that provide the intended modules and the context of their use so that static and dynamic assemblies could both happen. > >> The module system then sees that Y wants to provide a service, but the module system has no idea where the service is, so fails fast at startup. This is a feature, not a bug. The classpath behavior, of only discovering that a service provider can't provide a service during run time (because the service is missing), is not what we wanted for modular services. > > Does this issue really exist? The module system would already discover that during initialization. When the service interface is not available, the service provider will be ignored, there is no error (or anything else) later during runtime. Besides that, if the service interface is not available, you are unable to invoke the service loader anyway. Invoking the service loader requires a class instance of the service interface, which is not available. If you would still try to invoke the service loader, you would get a NCDFE at the call site of the service loader, the provides clause wouldn't make a difference. > > I am not quite sure about the behavior if you stack multiple module layers on top of each other. I think the service loader should only load instances that implement the class from the same module layer as the class instance given to the service loader. Everything else would cause a runtime error independent of the issue we are currently discussing. In this case, ignoring the provides clause would be the right solution, even if X would be available on another module layer. > > I currently see only two reasons for keeping the current behavior. The first one is to discourage service providers to be casually added to modules. You could argue that service providers should have their own module, so that users can add and remove them individually. The second reason might be the desire to fail early when the service interface was renamed or removed after updating X. This could be addressed by adding a modifier like `provides static`, there is no need to correlate the provides cluase with a specific `requires` clause. Service loader has a problem of not being able to readily stack dependencies around services/modules very readily. It seems like the struggle here is about being specific enough and the mechanisms of today are too much on the suggestion or guessing side instead of being specific enough. > >> So I think a more general solution than adding a way to describe ?this service is conditional on the presence of X? would be to allow multiple modules in a single JAR > > I guess that would work as well, but it would probably be more complex to set up. > > Best regards > Johannes Gregg Wonderly From robin.bygrave at gmail.com Sat Apr 22 21:05:52 2023 From: robin.bygrave at gmail.com (Rob Bygrave) Date: Sun, 23 Apr 2023 09:05:52 +1200 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> Message-ID: > Ron: I?m guessing that what?s bothering you isn?t so much the number of *modules* but the number of JARs. So I think a more general solution than adding a way to describe ?this service is conditional on the presence of X? would be to allow multiple modules in a single JAR that would contain both Y and the X-Y-plugin. How would this solve the issue? I can't see how having a second module on the same jar/artifact would work for this case. Bearing in mind this library needs to support both classpath and module-path. If a y.jar could include 2 module definitions say y and y.x what would be defined in y.x that solve this issue? Can you be more explicit Ron? (I can't see this working tbh) Instead of a second module my gut says that what would solve this is a: 'provides static' ... where that explicitly means that runtime module resolution should allow the service interface to not exist. Cheers, Rob. On Fri, 21 Apr 2023, 1:56 am Josiah Noel, wrote: > The reason why this could be considered acceptable is because the >> implementation of x.spi.Plugin that would exist in Y can be hidden / not >> exported and not open to abuse / accidental use. >> > > This also helps resolve that concern on the bug report. > > All of this boils down to: if the implementation module says `provides`, >> then is it reasonable to consider that ServiceLoader is the *only* vector >> by which the implementation class will be instantiated? If yes, then module >> resolution should perhaps be tolerant of a `provides` that specifies a >> missing service interface. If no -- that is, we think arbitrary code might >> instantiate (hence load) the implementation class -- then module resolution >> is correct as-is. > > > if we don't export the implementation class, then the > ServiceLoader becomes the only mechanism by which it can be instantiated. > In this case, if the user doesn't add the optional interface dependency, it > becomes impossible to accidentally load the interface and get a > ClassNotFoundException when using module Y's non-SPI packages. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josiahnoel at gmail.com Sat Apr 22 22:49:22 2023 From: josiahnoel at gmail.com (Josiah Noel) Date: Sat, 22 Apr 2023 18:49:22 -0400 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> Message-ID: > > How would this solve the issue? I can't see how having a second module on > the same jar/artifact would work for this case. > Here's how I envision it. In Jar Y, we'd have two modules. > module y { > > exports y.pkgs; > > } > and in the second module, we would have the SPI impl class > > module y.inject { > requires static x; > provides x.plugin with y.inject.PluginImpl; > > } > Now say some project adds the X Jar and the Y jar, the modules should resolve and X should service load the SPI from the y.inject module. > Instead of a second module my gut says that what would solve this is a: > 'provides static' > I'd like this too, but I don't think it's gonna happen. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robin.bygrave at gmail.com Sun Apr 23 05:32:17 2023 From: robin.bygrave at gmail.com (Rob Bygrave) Date: Sun, 23 Apr 2023 17:32:17 +1200 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> Message-ID: > Now say some project adds the X Jar and the Y jar, That is not the issue we have though, but instead the issue we have is if the application ONLY has y.jar (this fails module resolution at runtime on startup). If y.jar contains 2 modules but there is no x module in the module-path ... how does the second x.y module in y.jar work / how does it change the module resolution so that it does not produce the same error? On Sun, 23 Apr 2023, 10:50 am Josiah Noel, wrote: > > >> How would this solve the issue? I can't see how having a second module on >> the same jar/artifact would work for this case. >> > > Here's how I envision it. In Jar Y, we'd have two modules. > > >> module y { >> >> exports y.pkgs; >> >> } >> > > and in the second module, we would have the SPI impl class > >> >> module y.inject { >> requires static x; >> provides x.plugin with y.inject.PluginImpl; >> >> } >> > > Now say some project adds the X Jar and the Y jar, the modules should > resolve and X should service load the SPI from the y.inject module. > > >> Instead of a second module my gut says that what would solve this is a: >> 'provides static' >> > > I'd like this too, but I don't think it's gonna happen. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.pressler at oracle.com Sun Apr 23 08:43:06 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Sun, 23 Apr 2023 08:43:06 +0000 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> Message-ID: <9464FA4F-618C-47BF-B74F-8E5BFBC0B099@oracle.com> On 23 Apr 2023, at 00:49, Josiah Noel > wrote: How would this solve the issue? I can't see how having a second module on the same jar/artifact would work for this case. Here's how I envision it. In Jar Y, we'd have two modules. module y { exports y.pkgs; } and in the second module, we would have the SPI impl class module y.inject { requires static x; provides x.plugin with y.inject.PluginImpl; } Almost. Module y.inject `requires x`, not `requires static x`. If x is not required by any resolved module, there can be no resolved module that `uses x.plugin`. Now say some project adds the X Jar and the Y jar, the modules should resolve and X should service load the SPI from the y.inject module. Instead of a second module my gut says that what would solve this is a: 'provides static' I'd like this too, but I don't think it's gonna happen. There?s no need for that if you can get the desired behaviour without it. Since the difficulty is the number of artefacts rather than the number of modules, a multi-module JAR is a more elegant solution, as it doesn?t require a new construct. ? Ron -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.pressler at oracle.com Sun Apr 23 08:46:22 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Sun, 23 Apr 2023 08:46:22 +0000 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> Message-ID: <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> > On 22 Apr 2023, at 23:05, Rob Bygrave wrote: > > > > Ron: I?m guessing that what?s bothering you isn?t so much the number of *modules* but the number of JARs. So I think a more general solution than adding a way to describe ?this service is conditional on the presence of X? would be to allow multiple modules in a single JAR that would contain both Y and the X-Y-plugin. > > How would this solve the issue? I can't see how having a second module on the same jar/artifact would work for this case. See Josiah?s email and my response to him. > > Bearing in mind this library needs to support both classpath and module-path. Since we don?t yet have multi-module JARs, the question of how it would be used on the classpath is TBD. However, one possible approach is to treat a multi-module JAR on the classpath as if each module was in its own JAR, and all of them were placed on the classpath. ? Ron From ron.pressler at oracle.com Sun Apr 23 08:52:58 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Sun, 23 Apr 2023 08:52:58 +0000 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: <7113F589-92F1-4BDB-AF59-DFB7B4A1F165@cox.net> References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <2683d96c-adff-98e2-c8e4-d7be023dcf1d@oracle.com> <6c06f754-ef54-aa43-7a6c-9c7167e4ed58@oracle.com> <7113F589-92F1-4BDB-AF59-DFB7B4A1F165@cox.net> Message-ID: <755AADCA-9B23-45B7-BD08-C716FA30EB6B@oracle.com> On 22 Apr 2023, at 19:55, Gregg Wonderly > wrote: The dependency injection, like this, performed loosely, at runtime, is a problem only when the structure of association is loose. Modularity has injected this detail by suggesting that the ?huge jar file? solution is a problem. I don?t think it has. A multi-module JAR would be precisely addressed at allowing huge JARs on the one hand, while on the other allowing the application to construct a minimal image that contains only the relevant portions of the JAR with all that benefits to optimisation, maintainability, and security modules brings (see https://openjdk.org/jeps/8305968). Jars inside of Jars and other mechanisms have become forbidden due to licensing issues and other blind attempts at control of ?use? as something that provides value to the owner. I don?t see what has changed recently in regard to licensing. The impact of a library?s license does not change if it?s used on the classpath or the module path. Instead of providing sanity, we?ve opened to door to hugely complex paths of reference and association that are nearly impossible to understand and manage for large software systems. Quite the opposite. Modularity doesn?t change any "paths of reference and association?; it merely requires them to be explicit. ? Ron -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.pressler at oracle.com Sun Apr 23 11:32:21 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Sun, 23 Apr 2023 11:32:21 +0000 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <2683d96c-adff-98e2-c8e4-d7be023dcf1d@oracle.com> <6c06f754-ef54-aa43-7a6c-9c7167e4ed58@oracle.com> Message-ID: <41092574-68EF-428E-8DDF-56D9B1092802@oracle.com> > On 22 Apr 2023, at 02:19, Johannes Spangenberg wrote: > > Hello, I couldn't quite follow some of the arguments regarding this topic. I wasn't sure if I should interfere, but I hope it will be helpful now that I did. :/ > >> But the flip side of that is that if they choose not to use X, Y would just sit there, unused. Best case scenario, it would just increase their image size; worst-case scenario is that it could preclude some future optimisations that may require a full program analysis. > > Not sure about the worst-case. The current implementation already checks during initialization whether the service is usable. After all, it throws an exception when it is not. Any optimization can just ignore the service provider in such case. There is no full program analysis required, which is not already part of every module initialization. The module graph analysis doesn?t look inside modules. It cannot exclude the service implementation if it?s *part* of a module, as it doesn?t know whether other parts of the module rely on the classes needed for the service. > > Why do you need ... The runtime doesn?t know why Y `requires static X` or what parts of Y depend on X. This means that code goes back to the classpath?s loosey-goosey attitude and to relying on CNFE, losing one of modules? goals (reliable configuration). We want to *discourage* `requires static` when used for classes that are relied upon at runtime (as opposed to just compile-time) not encourage it. > > I am not quite sure about the behavior if you stack multiple module layers on top of each other. I think the service loader should only load instances that implement the class from the same module layer as the class instance given to the service loader. Everything else would cause a runtime error independent of the issue we are currently discussing. In this case, ignoring the provides clause would be the right solution, even if X would be available on another module layer. > > I currently see only two reasons for keeping the current behavior. The first one is to discourage service providers to be casually added to modules. You could argue that service providers should have their own module, so that users can add and remove them individually. The second reason might be the desire to fail early when the service interface was renamed or removed after updating X. Only the second one, which is another way of saying reliable configuration. > This could be addressed by adding a modifier like `provides static`, there is no need to correlate the provides cluase with a specific `requires` clause. It can be even better addressed by multi-module JARs, since with ?provides static? you'd still have an indeterminate portion of a module not subject to reliable configuration. In addition, it?s simpler and also benefits other use cases. ? Ron From greggwon at cox.net Sun Apr 23 17:16:46 2023 From: greggwon at cox.net (Gregg Wonderly) Date: Sun, 23 Apr 2023 12:16:46 -0500 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> Message-ID: <317567F8-2D71-4BCD-9FFA-F384D6526331@cox.net> > On Apr 23, 2023, at 3:46 AM, Ron Pressler wrote: >> On 22 Apr 2023, at 23:05, Rob Bygrave wrote >> >>> Ron: I?m guessing that what?s bothering you isn?t so much the number of *modules* but the number of JARs. So I think a more general solution than adding a way to describe ?this service is conditional on the presence of X? would be to allow multiple modules in a single JAR that would contain both Y and the X-Y-plugin. >> >> How would this solve the issue? I can't see how having a second module on the same jar/artifact would work for this case. > > See Josiah?s email and my response to him. > >> >> Bearing in mind this library needs to support both classpath and module-path. > > Since we don?t yet have multi-module JARs, the question of how it would be used on the classpath is TBD. However, one possible approach is to treat a multi-module JAR on the classpath as if each module was in its own JAR, and all of them were placed on the classpath. It seems ?bad? to somehow impose an order by taking a ?package? (jar) and turning it into a list (classpath). Again, I really think that order of reference and use should be explicit. If there is manifest data describing an ordering that is explicit there, that might work, but we still have the problem that if I need a different ordering to use a service or other interface implementation as an override of what the multi-module jar contains, we need to control that somehow very explicitly. Stacking so many ?orderings? to implicit details will be more problematic than a simplifying detail it seems to me. Gregg Wonderly From ron.pressler at oracle.com Sun Apr 23 17:47:59 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Sun, 23 Apr 2023 17:47:59 +0000 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: <317567F8-2D71-4BCD-9FFA-F384D6526331@cox.net> References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> <317567F8-2D71-4BCD-9FFA-F384D6526331@cox.net> Message-ID: On 23 Apr 2023, at 18:16, Gregg Wonderly > wrote: It seems ?bad? to somehow impose an order by taking a ?package? (jar) and turning it into a list (classpath). The modules in the JAR won?t be able to have split packages so order shouldn?t matter. Again, I really think that order of reference and use should be explicit. If there is manifest data describing an ordering that is explicit there, that might work, but we still have the problem that if I need a different ordering to use a service or other interface implementation as an override of what the multi-module jar contains, we need to control that somehow very explicitly. Stacking so many ?orderings? to implicit details will be more problematic than a simplifying detail it seems to me. Ordering shouldn?t (ideally) matter at all. If an application wants to ?override" A with B to ensure A isn?t used, it should not include A. The old way is to just throw everything into one big pile and let the runtime choose what gets used according to some rules; the new way is supposed to be different. The application specifies everything it uses and nothing but what it uses. Providing too much should, ideally, fail just as providing too little. ? Ron -------------- next part -------------- An HTML attachment was scrubbed... URL: From robin.bygrave at gmail.com Sun Apr 23 21:44:11 2023 From: robin.bygrave at gmail.com (Rob Bygrave) Date: Mon, 24 Apr 2023 09:44:11 +1200 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> Message-ID: > See Josiah?s email and my response to him. So if app module A uses Y it will have: requires y. If app module A uses X and Y it will have: requires x; requires y; requires y.inject; We now need everyone using x and y together to explicitly know that y also provides a plugin and that they ALSO must remember to add the requires x.inject; (and x.inject has no exports, it only has the provides). Have I understood correctly? Cheers, Rob. On Sun, 23 Apr 2023, 8:46 pm Ron Pressler, wrote: > > > > On 22 Apr 2023, at 23:05, Rob Bygrave wrote: > > > > > > > Ron: I?m guessing that what?s bothering you isn?t so much the number > of *modules* but the number of JARs. So I think a more general solution > than adding a way to describe ?this service is conditional on the presence > of X? would be to allow multiple modules in a single JAR that would contain > both Y and the X-Y-plugin. > > > > How would this solve the issue? I can't see how having a second module > on the same jar/artifact would work for this case. > > See Josiah?s email and my response to him. > > > > > Bearing in mind this library needs to support both classpath and > module-path. > > Since we don?t yet have multi-module JARs, the question of how it would be > used on the classpath is TBD. However, one possible approach is to treat a > multi-module JAR on the classpath as if each module was in its own JAR, and > all of them were placed on the classpath. > > ? Ron -------------- next part -------------- An HTML attachment was scrubbed... URL: From josiahnoel at gmail.com Sun Apr 23 23:04:19 2023 From: josiahnoel at gmail.com (Josiah Noel) Date: Sun, 23 Apr 2023 19:04:19 -0400 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> Message-ID: > We now need everyone using x and y together to explicitly know that y also > provides a plugin and that they ALSO must remember to add the requires > y.inject; (and y.inject has no exports, it only has the provides). > I thought it would just "work" without needing the `requires y.inject;` Am I wrong? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.pressler at oracle.com Mon Apr 24 08:06:08 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Mon, 24 Apr 2023 08:06:08 +0000 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> Message-ID: <34D03E42-56E7-4E37-A3EB-D4906D877DD9@oracle.com> On 23 Apr 2023, at 22:44, Rob Bygrave > wrote: So if app module A uses Y it will have: requires y. If app module A uses X and Y it will have: requires x; requires y; requires y.inject; We now need everyone using x and y together to explicitly know that y also provides a plugin and that they ALSO must remember to add the requires x.inject; (and x.inject has no exports, it only has the provides). Have I understood correctly? Not quite. If it uses X and Y it will have:: requires x; requires y; uses x.Plugin; which is exactly what you wanted, no? (You don?t even need `requires y` if Y?s other functionality is not used directly). The y.inject module can be on the module path whether or not the application has X. It?s just not resolved unless some module `uses x.Plugin`, and to do that it must also `requires x` since that?s the module that defines the service interface `x.Plugin`. ? Ron -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.pressler at oracle.com Mon Apr 24 08:58:19 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Mon, 24 Apr 2023 08:58:19 +0000 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> Message-ID: <6181E68C-72C0-498B-88F3-3EEDEF265670@oracle.com> On 24 Apr 2023, at 00:04, Josiah Noel > wrote: We now need everyone using x and y together to explicitly know that y also provides a plugin and that they ALSO must remember to add the requires y.inject; (and y.inject has no exports, it only has the provides). I thought it would just "work" without needing the `requires y.inject;` Am I wrong? Exactly. You?re not wrong. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robin.bygrave at gmail.com Mon Apr 24 09:31:06 2023 From: robin.bygrave at gmail.com (Rob Bygrave) Date: Mon, 24 Apr 2023 21:31:06 +1200 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: <34D03E42-56E7-4E37-A3EB-D4906D877DD9@oracle.com> References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> <34D03E42-56E7-4E37-A3EB-D4906D877DD9@oracle.com> Message-ID: > uses x.Plugin No it won't. The uses x.Plugin is in x. So the application does not have the uses clause. Hence the problem hete that it does not 'just work'. On Mon, 24 Apr 2023, 8:06 pm Ron Pressler, wrote: > > > On 23 Apr 2023, at 22:44, Rob Bygrave wrote: > > So if app module A uses Y it will have: > > requires y. > > If app module A uses X and Y it will have: > > requires x; > requires y; > requires y.inject; > > We now need everyone using x and y together to explicitly know that y also > provides a plugin and that they ALSO must remember to add the requires > x.inject; (and x.inject has no exports, it only has the provides). > > Have I understood correctly? > > > Not quite. If it uses X and Y it will have:: > > requires x; > requires y; > uses x.Plugin; > > which is exactly what you wanted, no? (You don?t even need `requires y` if > Y?s other functionality is not used directly). > > The y.inject module can be on the module path whether or not the > application has X. It?s just not resolved unless some module `uses > x.Plugin`, and to do that it must also `requires x` since that?s the module > that defines the service interface `x.Plugin`. > > ? Ron > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.pressler at oracle.com Mon Apr 24 09:51:51 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Mon, 24 Apr 2023 09:51:51 +0000 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> <34D03E42-56E7-4E37-A3EB-D4906D877DD9@oracle.com> Message-ID: > On 24 Apr 2023, at 10:31, Rob Bygrave wrote: > > > uses x.Plugin > > No it won't. > > The uses x.Plugin is in x. So the application does not have the uses clause. > > Hence the problem hete that it does not 'just work'. > I?m confused. To use the implementation of the `x.Plugin` service (defined in X) that you want Y to `provides` an implementation of, some module *must* have a `uses x.Plugin`. The runtime has to know that that service is needed by the program. Maybe the application `uses` it, but X itself could also declare that it `uses` an implementation of a service interface that it, itself, declares. So perhaps it is X that `uses X.plugin`, but someone has to actually say they want to use the service. ? Ron From josiahnoel at gmail.com Mon Apr 24 13:23:39 2023 From: josiahnoel at gmail.com (Josiah Noel) Date: Mon, 24 Apr 2023 09:23:39 -0400 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> <34D03E42-56E7-4E37-A3EB-D4906D877DD9@oracle.com> Message-ID: > I?m confused. To use the implementation of the `x.Plugin` service (defined > in X) that you want Y to `provides` an implementation of, some module > *must* have a `uses x.Plugin`. The runtime has to know that that service is > needed by the program. Maybe the application `uses` it, but X itself could > also declare that it `uses` an implementation of a service interface that > it, itself, declares. So perhaps it is X that `uses X.plugin`, but someone > has to actually say they want to use the service. > I believe I'm picking up what you're putting down, but let's review it again to be sure. In artifact X we have: > exports x.spi; > uses x.spi.Plugin; > in artifact Y we have two modules: y and y.inject. exports y.stuff; > And > requires x; > provides x.spi.Plugin with y.inject.PluginImpl; > With this in mind, we have two situations we want to support. we have an application Z that wants to use the Y artifact alone and a case where Z wants to use X and Y. With Y alone, the z module should look like this correct? requires y; > And in the case where Z wants both X and Y (with X doing the service-loading), we do: requires x; > requires y; > Am I right to understand that X will service load the y.inject plugin correctly? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.pressler at oracle.com Mon Apr 24 13:32:45 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Mon, 24 Apr 2023 13:32:45 +0000 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> <34D03E42-56E7-4E37-A3EB-D4906D877DD9@oracle.com> Message-ID: On 24 Apr 2023, at 14:23, Josiah Noel > wrote: I?m confused. To use the implementation of the `x.Plugin` service (defined in X) that you want Y to `provides` an implementation of, some module *must* have a `uses x.Plugin`. The runtime has to know that that service is needed by the program. Maybe the application `uses` it, but X itself could also declare that it `uses` an implementation of a service interface that it, itself, declares. So perhaps it is X that `uses X.plugin`, but someone has to actually say they want to use the service. I believe I'm picking up what you're putting down, but let's review it again to be sure. In artifact X we have: exports x.spi; uses x.spi.Plugin; in artifact Y we have two modules: y and y.inject. exports y.stuff; And requires x; provides x.spi.Plugin with y.inject.PluginImpl; With this in mind, we have two situations we want to support. we have an application Z that wants to use the Y artifact alone and a case where Z wants to use X and Y. With Y alone, the z module should look like this correct? requires y; And in the case where Z wants both X and Y (with X doing the service-loading), we do: requires x; requires y; Am I right to understand that X will service load the y.inject plugin correctly? Yes. This is also how it works today, only y and y.inject must be in two separate JARs. ? Ron -------------- next part -------------- An HTML attachment was scrubbed... URL: From robin.bygrave at gmail.com Tue Apr 25 01:39:14 2023 From: robin.bygrave at gmail.com (Rob Bygrave) Date: Tue, 25 Apr 2023 13:39:14 +1200 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> <34D03E42-56E7-4E37-A3EB-D4906D877DD9@oracle.com> Message-ID: *> only y and y.inject must be in two separate JARs.* Does this work without *any* module having a `requires y.inject`? If so, how is the y.inject module get included without a `requires y.inject` being specified? (There is an obvious follow up question if that is the case as it likely breaks the case where only y module is used). Cheers, Rob. On Tue, 25 Apr 2023 at 01:32, Ron Pressler wrote: > > > On 24 Apr 2023, at 14:23, Josiah Noel wrote: > > > >> I?m confused. To use the implementation of the `x.Plugin` service >> (defined in X) that you want Y to `provides` an implementation of, some >> module *must* have a `uses x.Plugin`. The runtime has to know that that >> service is needed by the program. Maybe the application `uses` it, but X >> itself could also declare that it `uses` an implementation of a service >> interface that it, itself, declares. So perhaps it is X that `uses >> X.plugin`, but someone has to actually say they want to use the service. >> > > I believe I'm picking up what you're putting down, but let's review it > again to be sure. > > > In artifact X we have: > > >> exports x.spi; >> uses x.spi.Plugin; >> > > in artifact Y we have two modules: y and y.inject. > > exports y.stuff; >> > > And > >> requires x; >> provides x.spi.Plugin with y.inject.PluginImpl; >> > > With this in mind, we have two situations we want to support. we have an > application Z that wants to use the Y artifact alone and a case where Z > wants to use X and Y. > > With Y alone, the z module should look like this correct? > > requires y; >> > > And in the case where Z wants both X and Y (with X doing the > service-loading), we do: > > requires x; >> requires y; >> > > Am I right to understand that X will service load the y.inject plugin > correctly? > > > Yes. > This is also how it works today, only y and y.inject must be in two > separate JARs. > > ? Ron > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josiahnoel at gmail.com Tue Apr 25 04:32:47 2023 From: josiahnoel at gmail.com (Josiah Noel) Date: Tue, 25 Apr 2023 00:32:47 -0400 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> <34D03E42-56E7-4E37-A3EB-D4906D877DD9@oracle.com> Message-ID: > > Does this work without *any* module having a `requires y.inject`? > Yes. If so, how is the y.inject module get included without a `requires > y.inject` being specified? > It will resolve when a module uses the x.spi.Plugin it provides. (x in this case) Hence, there is no need for any module to require y.inject. (as it doesn't export anything). If there is no module on the module path that uses the x.spi.Plugin, the y.inject module will not resolve, and no exception will be thrown. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michal at kleczek.org Tue Apr 25 06:41:42 2023 From: michal at kleczek.org (=?utf-8?Q?Micha=C5=82_K=C5=82eczek?=) Date: Tue, 25 Apr 2023 08:41:42 +0200 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> <34D03E42-56E7-4E37-A3EB-D4906D877DD9@oracle.com> Message-ID: <3DFFC925-A51A-4D40-99E5-4581CFFD6ED1@kleczek.org> As specified in https://help.intrexx.com/apidocs/jdk17/api/java.base/java/lang/module/Configuration.html#resolveAndBind(java.lang.module.ModuleFinder,java.util.List,java.lang.module.ModuleFinder,java.util.Collection) all modules reachable by a ModuleFinder are examined for uses-provides relationship and added to the set of resolved modules. It gives you the optionality: just add/remove artefacts to/from the module path to add/remove service providers. > On 25 Apr 2023, at 03:39, Rob Bygrave wrote: > > > only y and y.inject must be in two separate JARs. > > Does this work without any module having a `requires y.inject`? If so, how is the y.inject module get included without a `requires y.inject` being specified? (There is an obvious follow up question if that is the case as it likely breaks the case where only y module is used). > > > Cheers, Rob. > > On Tue, 25 Apr 2023 at 01:32, Ron Pressler > wrote: >> >> >>> On 24 Apr 2023, at 14:23, Josiah Noel > wrote: >>> >>> >>>> I?m confused. To use the implementation of the `x.Plugin` service (defined in X) that you want Y to `provides` an implementation of, some module *must* have a `uses x.Plugin`. The runtime has to know that that service is needed by the program. Maybe the application `uses` it, but X itself could also declare that it `uses` an implementation of a service interface that it, itself, declares. So perhaps it is X that `uses X.plugin`, but someone has to actually say they want to use the service. >>> >>> I believe I'm picking up what you're putting down, but let's review it again to be sure. >>> >>> >>> In artifact X we have: >>> >>>> >>>> exports x.spi; >>>> uses x.spi.Plugin; >>> >>> in artifact Y we have two modules: y and y.inject. >>> >>>> exports y.stuff; >>> >>> >>> And >>>> requires x; >>>> provides x.spi.Plugin with y.inject.PluginImpl; >>> >>> With this in mind, we have two situations we want to support. we have an application Z that wants to use the Y artifact alone and a case where Z wants to use X and Y. >>> >>> With Y alone, the z module should look like this correct? >>> >>>> requires y; >>> >>> And in the case where Z wants both X and Y (with X doing the service-loading), we do: >>> >>>> requires x; >>>> requires y; >>> >>> Am I right to understand that X will service load the y.inject plugin correctly? >> >> Yes. >> This is also how it works today, only y and y.inject must be in two separate JARs. >> >> ? Ron -------------- next part -------------- An HTML attachment was scrubbed... URL: From eirbjo at gmail.com Tue Apr 25 11:30:38 2023 From: eirbjo at gmail.com (=?UTF-8?B?RWlyaWsgQmrDuHJzbsO4cw==?=) Date: Tue, 25 Apr 2023 13:30:38 +0200 Subject: jlink --add-modules not applied to launcher Message-ID: Hi, I'm working on a Maven plugin which calls jlink to produce a runtime image. The launcher main method sets up a graph of ModuleLayers. The module jar files for these layers are added to runtime image folder lib/myframework/modules by the Maven plugin, along with some metadata defining the graph of ModuleLayers to be constructed. Since the "actual" module graph in this case isn't visible for jlink, the Maven plugin inspects all module descriptors and collects all "java.*" required modules which it adds to the "-add-modules" jlink option. This works fine in the sense that the these modules are included in the runtime image. When executing the custom launcher however, the required images are not added to the boot layer. The only way I've found so far to get around this is to add "--add-modules java.sql" etc to the JLINK_VM_OPTIONS environment variable in the launcher script. This file can of course be edited by my Maven plugin, but this feels a bit dirty. So far I haven't found a way to tell jlink which modules should be added to the launcher's boot modules. Am I missing some switch? PS: I know I could just add all modules via --add-modules instead of building my custom layer graph, but that graph is there for reasons and I don't think it will run well on a collapsed boot layer. Thanks, Eirik. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eirbjo at gmail.com Tue Apr 25 12:20:23 2023 From: eirbjo at gmail.com (=?UTF-8?B?RWlyaWsgQmrDuHJzbsO4cw==?=) Date: Tue, 25 Apr 2023 14:20:23 +0200 Subject: jlink --add-modules not applied to launcher In-Reply-To: References: Message-ID: tir. 25. apr. 2023 kl. 13:30 skrev Eirik Bj?rsn?s : > > So far I haven't found a way to tell jlink which modules should be added > to the launcher's boot modules. Am I missing some switch? > Found the ?add-options plugin, which probably does what I need. Eirik. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Apr 25 12:48:16 2023 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 25 Apr 2023 13:48:16 +0100 Subject: jlink --add-modules not applied to launcher In-Reply-To: References: Message-ID: On 25/04/2023 13:20, Eirik Bj?rsn?s wrote: > > > tir. 25. apr. 2023 kl. 13:30 skrev Eirik Bj?rsn?s : > > > So far I haven't found a way to tell jlink which modules should be > added to the launcher's boot modules. Am I missing some switch? > > > Found the ?add-options plugin, which probably does what I need. > The command line parsing in jlink doesn't currently allow the argument for a plugin use GNU style options. There are workarounds, e.g. `--add-options \ --add-modules=foo,bar` so you mileage might vary on this. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eirbjo at gmail.com Tue Apr 25 15:14:55 2023 From: eirbjo at gmail.com (=?UTF-8?B?RWlyaWsgQmrDuHJzbsO4cw==?=) Date: Tue, 25 Apr 2023 17:14:55 +0200 Subject: jlink --add-modules not applied to launcher In-Reply-To: References: Message-ID: On Tue, Apr 25, 2023 at 2:48?PM Alan Bateman wrote: > The command line parsing in jlink doesn't currently allow the argument for > a plugin use GNU style options. There are workarounds, e.g. `--add-options > \ --add-modules=foo,bar` so you mileage might vary on this. > I'm calling jlink through its ToolProvider and it does indeed work to simply add a space just before --add-modules=, like this: String[] options = new String[] {"--add-options", " --add-modules=" +String.join(",", addModules), ...}; The leading space makes following check of the --add-options param value in TaskHelper.OptionsHelper not match and throw BadArgs: if (param == null || param.isEmpty() > || (param.length() >= 2 && param.charAt(0) == '-' > && param.charAt(1) == '-')) { > throw new BadArgs("err.missing.arg", name). > showUsage(true); > } The logic here could probably be improved. Thanks, Eirik. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robin.bygrave at gmail.com Wed Apr 26 00:50:23 2023 From: robin.bygrave at gmail.com (Rob Bygrave) Date: Wed, 26 Apr 2023 12:50:23 +1200 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: <3DFFC925-A51A-4D40-99E5-4581CFFD6ED1@kleczek.org> References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> <34D03E42-56E7-4E37-A3EB-D4906D877DD9@oracle.com> <3DFFC925-A51A-4D40-99E5-4581CFFD6ED1@kleczek.org> Message-ID: *> As specified in ...* Awesome, thanks for that. Really nice to know that detail. *> just add/remove artefacts to/from the module path to add/remove service providers.* That is true and will work for some folks but also presents an issue for library authors with 000's of users who now have the ability to "stuff it up" (by not including the dependency). The ideal is to not provide users the option to "stuff it up" or at least have the default of "just works" and especially when those users number in 000's (and tend to reads documentation quickly, lightly, or not at all so we prefer to not lend on docs if we can avoid it). I'll put in a reply to Josiah that outlines a couple of options available to avoid it / default to "just works". Thanks again, Cheers, Rob. On Tue, 25 Apr 2023 at 18:41, Micha? K?eczek wrote: > As specified in > https://help.intrexx.com/apidocs/jdk17/api/java.base/java/lang/module/Configuration.html#resolveAndBind(java.lang.module.ModuleFinder,java.util.List,java.lang.module.ModuleFinder,java.util.Collection) > > all modules reachable by a ModuleFinder are examined for uses-provides > relationship and added to the set of resolved modules. > > It gives you the optionality: just add/remove artefacts to/from the module > path to add/remove service providers. > > On 25 Apr 2023, at 03:39, Rob Bygrave wrote: > > *> only y and y.inject must be in two separate JARs.* > > Does this work without *any* module having a `requires y.inject`? If so, > how is the y.inject module get included without a `requires y.inject` > being specified? (There is an obvious follow up question if that is the > case as it likely breaks the case where only y module is used). > > > Cheers, Rob. > > On Tue, 25 Apr 2023 at 01:32, Ron Pressler > wrote: > >> >> >> On 24 Apr 2023, at 14:23, Josiah Noel wrote: >> >> >> >>> I?m confused. To use the implementation of the `x.Plugin` service >>> (defined in X) that you want Y to `provides` an implementation of, some >>> module *must* have a `uses x.Plugin`. The runtime has to know that that >>> service is needed by the program. Maybe the application `uses` it, but X >>> itself could also declare that it `uses` an implementation of a service >>> interface that it, itself, declares. So perhaps it is X that `uses >>> X.plugin`, but someone has to actually say they want to use the service. >>> >> >> I believe I'm picking up what you're putting down, but let's review it >> again to be sure. >> >> >> In artifact X we have: >> >> >>> exports x.spi; >>> uses x.spi.Plugin; >>> >> >> in artifact Y we have two modules: y and y.inject. >> >> exports y.stuff; >>> >> >> And >> >>> requires x; >>> provides x.spi.Plugin with y.inject.PluginImpl; >>> >> >> With this in mind, we have two situations we want to support. we have an >> application Z that wants to use the Y artifact alone and a case where Z >> wants to use X and Y. >> >> With Y alone, the z module should look like this correct? >> >> requires y; >>> >> >> And in the case where Z wants both X and Y (with X doing the >> service-loading), we do: >> >> requires x; >>> requires y; >>> >> >> Am I right to understand that X will service load the y.inject plugin >> correctly? >> >> >> Yes. >> This is also how it works today, only y and y.inject must be in two >> separate JARs. >> >> ? Ron >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robin.bygrave at gmail.com Wed Apr 26 01:51:14 2023 From: robin.bygrave at gmail.com (Rob Bygrave) Date: Wed, 26 Apr 2023 13:51:14 +1200 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> <34D03E42-56E7-4E37-A3EB-D4906D877DD9@oracle.com> Message-ID: *> Yes > Does this work without any module having a `requires y.inject`? * Awesome. *> If there is no module on the module path that uses the x.spi.Plugin, the y.inject module will not resolve, and no exception will be thrown.* Awesome. *Summary: (as this thread has got massive)* The suggested future Java that supports multiple modules in a single jar would indeed solve this issue and "just works". To me at least, it looks like it solves it *perfectly* without any negative side effects or downsides. The only followup questions I can see regarding multiple modules in a single jar are: - Would this solution be intended to work for older versions of Java at runtime like Java 11? We probably can't answer this yet but open source libs tend towards supporting a minimum version based on LTS for better or worse (and will be keen to know if they can adopt this by using the latest build tooling but still support older/existing java runtimes. If so, I would think that would greatly help the adoption of the 2 modules in 1 jar solution). - For my own personal interest, what are the other motivations for supporting multiple modules in a single jar? (If these other motivations are high that improves our chances of seeing this solution happen) *Workarounds today* For myself as I see it the workarounds that are available today are: - *[1] Not using ServiceLoader. This is relatively simple and has only 1 downside which is that the implementation must be in an exported package. As such, this is still my preferred workaround (but it's certainly not the ideal longer term solution). - [2] Changing the design by "reversing the relationship" e.g x at runtime detects y. This will work in some cases but isn't going to work generally or at scale. - [3] Splitting the artifact / Have a separate jar for the optional service implementation - [3a] Make users aware of this and require the plugin dependency to be explicitly included in the path. This is not ideal for the case where there are 000s of users and now has "stuff it up" by default behaviour (people forget to include the dependency). - [3b] Use a maven composite dependency and document/promote that as the dependency to use (So 3 artifacts now, ok'ish). - **[3c] Use maven transitive dependency to by default transparently include the plugin dependency (most closely matches the 2 modules in 1 jar approach) For people interested I have put a skeleton of this at: https://github.com/rbygrave/skeleton-optional-service For people interested, the top 2 workarounds today as I see them are: *[1] Not using ServiceLoader. **[3c] Use maven transitive dependency ... which I have as a PR at: https://github.com/rbygrave/skeleton-optional-service/pull/1 This uses a slightly dirty build trick to workaround the maven cyclical dependency. This can be considered ok if the plugin implementation is very stable (which I expect to hold true). **[3c] To me this is the closest thing to the "2 modules in 1 jar" solution Cheers, Rob. On Tue, 25 Apr 2023 at 16:33, Josiah Noel wrote: > Does this work without *any* module having a `requires y.inject`? >> > > Yes. > > If so, how is the y.inject module get included without a `requires >> y.inject` being specified? >> > > It will resolve when a module uses the x.spi.Plugin it provides. (x in > this case) Hence, there is no need for any module to require y.inject. (as > it doesn't export anything). > > If there is no module on the module path that uses the x.spi.Plugin, the > y.inject module will not resolve, and no exception will be thrown. > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From webseiten.designer at googlemail.com Wed Apr 26 07:16:09 2023 From: webseiten.designer at googlemail.com (Tim Feuerbach) Date: Wed, 26 Apr 2023 09:16:09 +0200 Subject: [External] : Re: provides and requires static ... runtime error Message-ID: > - For my own personal interest, what are the other motivations for supporting multiple modules in a single jar? (If these other motivations are high that improves our chances of seeing this solution happen) One of the use cases is listed here: https://openjdk.org/projects/jigsaw/spec/issues/#MultiModuleExecutableJARs - " Provide a means to create an executable modular ?uber-JAR? that contains more than one module, preserving module identities and boundaries, so that an entire application can shipped as a single artifact." This is a common way of deploying an application built with a framework like Spring, however the question is if this is still relevant today when the trend goes towards shipping containers instead of jars, where you don't care about the granularity of files inside. In the end, the proper way of deploying a modular Java web application remains creating an application-specific image with jlink, which I'm sure of can be added to a container image in a way that the base JDK modules form a layer that is shared between multiple containers from different applications. My personal interest in multi-module jars would be enforcing architectural boundaries in a modular monolith at compile time. I could see using modules for different domains or even layers like API/core/persistence, allowing fine grained control over what gets exported to whom and which layers are allowed to require which dependency. For example, the web API layer would have no business importing JPA types. As a bonus you could also limit the attack surface for reflection gadgets. Instead of opening everything to the framework for deep reflection (which, as I've learned recently, is not preventable by using package based opens due to being able to define a class inside the target package at runtime), I could encapsulate high-security areas further, without having to change my build process. Of course, this comes down more to tools like Maven making it (seem) cumbersome to create multiple jars per dependency management unit, which is a weak motivation to change the JDK. -- Tim From andrejusc at yahoo.com Wed Apr 26 07:38:35 2023 From: andrejusc at yahoo.com (Andrejus Chaliapinas) Date: Wed, 26 Apr 2023 07:38:35 +0000 (UTC) Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: Message-ID: <177910942.115238.1682494715156@mail.yahoo.com> Would it be more appropriate in case of "uber-JAR" to introduce then "module feature" concept, i.e. the one which would allow grouping (and then enforcing of their needed presence within final "uber-JAR") of modules? That "module feature" then could have it's own handling mechanism and semantics and will not clash with separate module granularity/boundaries. Andrejus On Wednesday, April 26, 2023 at 08:21:25 AM GMT+1, Tim Feuerbach wrote: > - For my own personal interest, what are the other motivations for supporting multiple modules in a single jar? (If these other motivations are high that improves our chances of seeing this solution happen) One of the use cases is listed here: https://openjdk.org/projects/jigsaw/spec/issues/#MultiModuleExecutableJARs - " Provide a means to create an executable modular ?uber-JAR? that contains more than one module, preserving module identities and boundaries, so that an entire application can? shipped as a single artifact." This is a common way of deploying an application built with a framework like Spring, however the question is if this is still relevant today when the trend goes towards shipping containers instead of jars, where you don't care about the granularity of files inside. In the end, the proper way of deploying a modular Java web application remains creating an application-specific image with jlink, which I'm sure of can be added to a container image in a way that the base JDK modules form a layer that is shared between multiple containers from different applications. My personal interest in multi-module jars would be enforcing architectural boundaries in a modular monolith at compile time. I could see using modules for different domains or even layers like API/core/persistence, allowing fine grained control over what gets exported to whom and which layers are allowed to require which dependency. For example, the web API layer would have no business importing JPA types. As a bonus you could also limit the attack surface for reflection gadgets. Instead of opening everything to the framework for deep reflection (which, as I've learned recently, is not preventable by using package based opens due to being able to define a class inside the target package at runtime), I could encapsulate high-security areas further, without having to change my build process. Of course, this comes down more to tools like Maven making it (seem) cumbersome to create multiple jars per dependency management unit, which is a weak motivation to change the JDK. -- Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.pressler at oracle.com Wed Apr 26 09:56:21 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Wed, 26 Apr 2023 09:56:21 +0000 Subject: [External] : Re: provides and requires static ... runtime error In-Reply-To: References: <9bd26385-bbf9-7aad-9a3b-adeab04d3477@oracle.com> <89a89c7f-4c27-ee55-ab20-3a27da0c0020@oracle.com> <86FEE86A-581C-4071-9AE6-DDA15264DBED@oracle.com> <5DCA7A65-BF4A-4E3F-9F66-8F4F3F1A8DFF@oracle.com> <7164E689-EDD0-4CA6-B7D2-10FB09CAAEAC@oracle.com> <34D03E42-56E7-4E37-A3EB-D4906D877DD9@oracle.com> Message-ID: On 26 Apr 2023, at 02:51, Rob Bygrave > wrote: - Would this solution be intended to work for older versions of Java at runtime like Java 11? We probably can't answer this yet but open source libs tend towards supporting a minimum version based on LTS for better or worse (and will be keen to know if they can adopt this by using the latest build tooling but still support older/existing java runtimes. If so, I would think that would greatly help the adoption of the 2 modules in 1 jar solution). The answer to that is no. If and when we support multi-module JARs, that new feature will not be backported. Libraries that wish to support old versions have always had this problem, but the solution is to pick some minimal version and employ only the features available in that version. A new feature can be used once the baseline JDK version of the library is pushed past the versions that don?t have the feature. - For my own personal interest, what are the other motivations for supporting multiple modules in a single jar? (If these other motivations are high that improves our chances of seeing this solution happen) The freedom to pick the desired artefact-splitting rather than have it forced by module boundaries. If a library is made up of multiple modules, all of whom are needed for the library?s functionality, it could be distributed as a single JAR. Workarounds today - [3] Splitting the artifact / Have a separate jar for the optional service implementation This is what you should do today, and it isn?t a workaround but the correct way of doing things. The use of a build tool like Maven shouldn?t be a problem (and if it is, it?s something to be addressed by Maven). Maven can be configured to always add all artefacts to the module path, including the plugin. If no module declares that it `uses` the service, that module will simply not be resolved. The same considerations have always applied when the classpath is used. The only pain is that you, as the library?s author, have to build more artefacts, but since they?re handled by Maven on the user side, and because having many artefacts is already the norm, I don?t think the pain is too large. ? Ron -------------- next part -------------- An HTML attachment was scrubbed... URL: From volker.simonis at gmail.com Fri Apr 28 15:38:12 2023 From: volker.simonis at gmail.com (Volker Simonis) Date: Fri, 28 Apr 2023 17:38:12 +0200 Subject: JEP draft: Disallow the Dynamic Loading of Agents by Default In-Reply-To: <168352AB-C660-484F-BAA6-31A2B6F0D0C8@oracle.com> References: <168352AB-C660-484F-BAA6-31A2B6F0D0C8@oracle.com> Message-ID: On Wed, Apr 19, 2023 at 12:29?AM Ron Pressler wrote: > > Hi. > > Following last month?s email about changing the default of > EnableDynamicAgentLoading [1], we?ve now published two JEP drafts. > > The first is an informational JEP describing what integrity is and why we need it, > and motivates what changes are required to get it (which include the > restriction of dynamically loaded agents among others): > > https://openjdk.org/jeps/8305968 Integrity and Strong Encapsulation > > - Use sun.misc.Unsafe to access and modify private fields. > > - Load a native library that employs JNI to call private methods and > set private fields. (The JNI API is not subject to access checks.) > > - Load an agent that changes code in a running application, > using an API intended for tools only. > > To attain our goal of integrity by default, we will gradually restrict > these APIs and close all loopholes in a series of upcoming JEPs, ensuring > that no library can assume superpowers without the application's consent. > Libraries that rely on these APIs should spend the time remaining until > they are restricted to prepare their users for any necessary changes. I think it is a little unfortunate to put the usage of s.m.Unsafe and JNI/Instrumentation/JVMTI into the same category, especially when it comes to blaming developers for their usage. While s.m.Unsafe has always been an internal, undocumented and unsupported API, the latter three are part of the Java Platform (e.g. "native" is a Java keyword and Runtime.loadLibrary() is part of the Java API). Do you really plan to make JNI an optional feature which will have to be manually enabled at startup? What will be the benefit? I understand that in an ideal world where you had no user-supplied JNI libraries at all, you might be able to perform more/better optimizations. But as you'd have to support JNI anyway, wouldn't the maintenance of the resulting code become a nightmare. How many "if (JNI) {..} else {..}" would we get? And what would be the benefit of disabling it by default for the user except increased "integrity"? I.e. do you have some concrete examples of planned features X, Y, Z which will only work with disabled JNI? Will these features be Java SE features or implementation specific OpenJDK-only features? > The second touches on the specifics of dynamically loaded agents and the > proposed change: > > https://openjdk.org/jeps/8306275 Disallow the Dynamic Loading of Agents by Default > > Agents are used by profiling tools to instrument Java applications, > but agents can also be misused to undermine the integrity of the > Java Platform. I don't think it is fair to assume that profilers are the only "valid" use case for agents and imply that all other use cases are a mis-use of the API. > - It is not a goal to change the Attach API that allows a tool to > connect to a running JVM for monitoring and management purposes. I don't understand this "Non-Goal"? The Attach API [1] allows to dynamically attach to a running JVM and "Once a reference to a virtual machine is obtained, the loadAgent, loadAgentLibrary, and loadAgentPath methods are used to load agents into target virtual machine". So how can you achieve this JEP's goals without changing/restricting the Attach API? I therefore think this "Non-Goal" should be rephrased to explain which parts of the Attach API will be changed and moved to the "Goal" section instead. General comments: - You go into great detail to explain why a human-operated tool is "superior" (in the sense of trust and security) to a library and "would ideally not be subject to the integrity constraints imposed on the application". I can't follow this argument, because both, the decision to use a specific tool as well as the decision to rely on a library is taken by a human. I'd even argue that the decision to depend on a specific library which requires the dynmaic attach mechanism is taken by a more knowledgeable user (i.e. the developer himself). Of course both, a tool as well as a library can contain malicious code, but I don't see a fundamental difference between the two. - You may argue that users have to be protected from malicious libraries which gain their superpowers by secretly loading agents at runtime. But users who don't know and don't care about their library dependencies will just as easy and without reflection (pun intended :) add the -XX:+EnableDynamicAgentLoading to their command line arguments (making this the new, most often used command line option even surpassing the usage of --add-opens :) - I still can't understand the benefit of "only" changing the default behavior for dynamic agent loading. I could understand this if you'd do it with a plan to deprecate and completely remove the dynamic agent loading capability. But what are the benefits of changing the default if you'll have to support the functionality anyway? As mentioned in earlier discussions, my main concern with the proposed change is the impact it will have on the evolution of Java. Java's dynamic features are one of its biggest strength and a major reason for its success. Sacrificing some of them or making their usage increasingly expensive requires a broader discussion in the community and shouldn't happen "under the hood" of a discussion about the default setting of a command line flag. - I don't understand why this JEP has scope "SE". As you rightly mentioned, the Attach API is a "non-standard" API which can be changed at any time and without affecting the Java SE specification, so this JEP should rather have scope "JDK" instead. On the other hand, the fact that this functionality is not governed by the SE specification will allow different OpenJDK distributors to use a different default setting for -XX:EnableDynamicAgentLoading which has the potential to cause a lot of confusion if we can't sattle on a common strategy. - If doing this change at all, I think it would be better to do it in a non-LTS release first. Best regards, Volker [1] https://docs.oracle.com/en/java/javase/20/docs/api/jdk.attach/com/sun/tools/attach/VirtualMachine.html > [1]: https://mail.openjdk.org/pipermail/jigsaw-dev/2023-March/014816.html > > ? Ron From eirbjo at gmail.com Fri Apr 28 19:14:08 2023 From: eirbjo at gmail.com (=?UTF-8?B?RWlyaWsgQmrDuHJzbsO4cw==?=) Date: Fri, 28 Apr 2023 21:14:08 +0200 Subject: JEP draft: Disallow the Dynamic Loading of Agents by Default In-Reply-To: References: <168352AB-C660-484F-BAA6-31A2B6F0D0C8@oracle.com> Message-ID: > > > Agents are used by profiling tools to instrument Java applications, > > but agents can also be misused to undermine the integrity of the > > Java Platform. > > I don't think it is fair to assume that profilers are the only "valid" > use case for agents and imply that all other use cases are a mis-use > of the API. > First, I don't read the JEP as implying that all non-profiler use cases are misuse. Having said that, I do think that agents can in fact strengthen the integrity of the platform. Case in point is that when the Java serialization vulnerabilities hit around 2015, I could very quickly ( a few hours) whip together the "NotSoSerial" serialization firewall agent [1] to efficiently prevent exploits. I later got word that a large CMS vendor deployed it to their platform which included some of the world's busiest websites. I don't know if they used the attach mechanism to plug their serialization holes, but they surely could at the time. With microservices gaining popularity over the years, restarts are probably more common and automated now, including configuration of JVM options. So attaching to long-running instances to prevent restarts is probably becoming less useful over time. The agent misuse that the JEP is referring to here is perhaps mostly concerning libraries using the attach mechanism to get access they otherwise would not have in a running JVM? Perhaps the JEP could be updated to be more clear on this? Cheers, Eirik. [1] https://github.com/kantega/notsoserial/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From heidinga at redhat.com Sat Apr 29 02:30:01 2023 From: heidinga at redhat.com (Dan Heidinga) Date: Fri, 28 Apr 2023 22:30:01 -0400 Subject: JEP draft: Disallow the Dynamic Loading of Agents by Default Message-ID: Hi Ron, Thanks for writing up the JEP draft outlining the proposal to disallow dynamic loading of agents by default. The Red Hat Java team has continued to discuss this proposal internally and with our stakeholders. While there is a general agreement (or at least acceptance) with the overall direction of this proposal, we still see the concerns raised by Andrew [0] as needing to be addressed. So let?s start with the high-order bit: timing. The JEP draft states it ?intends to adopt the proposal [to disable agents by default] six years later, in 2023, for JDK 21.? We would like to see this targeted to JDK 22 instead as the change has not been widely evangelized yet and comes as a surprise to many, both to those actively developing OpenJDK and to those monitoring its development. We owe it to the community to give this proposal enough bake time, especially given that JDK 21 is an LTS release. Though the official position is that LTS releases are no different than any other release, the actions of JDK consumers make them special. Users will be surprised by this change when they migrate from JDK 17 to 21. If we delay till JDK 22, we give the ecosystem time to adapt to the change before the next LTS. Additionally, users who have tested with the -XX:-EnableDynamicAgentLoading option will have false expectations if they validated their use of jcmd to load the agent as the behaviour was not correct prior to JDK 21 [1]. The next concern is one you call out in the draft that ?Java's excellent serviceability has long been a source of pride for the platform.? We agree! Java currently has an excellent, prime position in Observability capabilities. For better or for worse, there are many Observability tools that have relied on dynamic attach to provide the necessary monitoring for workloads It?s important we give Java?s monitoring tools sufficient time to migrate comfortably without shocking the ecosystem by changing the default in an LTS. By delaying the change till JDK 22, we give the ecosystem 2 years to migrate and to prepare users for the change. Additionally, this provides the time for Java?s profiling tools to adapt as well. And for the ?ad-hoc troubleshooting? tools - like Byteman - to retrain their users. Finally, while it?s easy to agree with the principle that ?the application owner is given the final say on whether to allow dynamic loading of agents?, the argument can (and should!) be made that those application owners have made that final decision by deploying the libraries and tools that use dynamic attach. A JVM command line argument is not the only way to express their consent for code to run on their system. For many production uses, the reality is more complicated than a single ?application owner?. Take a Kubernetes cluster for example. Who is the application owner when deploying to a Kubernetes cluster? The dev team that develops the application? The operations team that manages the cluster? The monitoring team that monitors the applications? The Support team that troubleshoots issues with the deployment? We should be careful not to understate or misrepresent the complex web of ?owners? that are currently able (and, for business reasons, need) to apply agents dynamically. Downplaying the complexity many organizations experience when dealing with changes to command line options (as an example) weakens the argument for changing today?s status quo. We also know that in many cases customers and users may not be in a position to modify startup scripts (e.g. even to add in an extra parameter) as to do so may invalidate support contracts, etc. Dynamically attached agents have been a ?superpower? of Java and their use has greatly benefited the ecosystem as they?ve helped service and support use cases that otherwise aren?t possible, as they helped propel Java to the forefront of the Observability tooling, and allowed many other useful libraries to be developed. Let?s delay flipping the default until JDK 22 to give the breadth of the ecosystem time to absorb this change. ?Dan [0] https://mail.openjdk.org/pipermail/serviceability-dev/2023-March/047084.html [1] https://bugs.openjdk.org/browse/JDK-8304438 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.pressler at oracle.com Sun Apr 30 14:19:13 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Sun, 30 Apr 2023 14:19:13 +0000 Subject: [External] : Re: JEP draft: Disallow the Dynamic Loading of Agents by Default In-Reply-To: References: Message-ID: <1D74CB9A-1040-4992-954E-AAA1430FE4F0@oracle.com> Hi Dan! > On 29 Apr 2023, at 03:30, Dan Heidinga wrote: > > Hi Ron, > > Thanks for writing up the JEP draft outlining the proposal to disallow dynamic loading of agents by default. The Red Hat Java team has continued to discuss this proposal internally and with our stakeholders. > > While there is a general agreement (or at least acceptance) with the overall direction of this proposal, we still see the concerns raised by Andrew [0] as needing to be addressed. > > So let?s start with the high-order bit: timing. > > The JEP draft states it ?intends to adopt the proposal [to disable agents by default] six years later, in 2023, for JDK 21.? We would like to see this targeted to JDK 22 instead as the change has not been widely evangelized yet and comes as a surprise to many, both to those actively developing OpenJDK and to those monitoring its development. > > We owe it to the community to give this proposal enough bake time, especially given that JDK 21 is an LTS release. Though the official position is that LTS releases are no different than any other release, the actions of JDK consumers make them special. Users will be surprised by this change when they migrate from JDK 17 to 21. If we delay till JDK 22, we give the ecosystem time to adapt to the change before the next LTS. > > Additionally, users who have tested with the -XX:-EnableDynamicAgentLoading option will have false expectations if they validated their use of jcmd to load the agent as the behaviour was not correct prior to JDK 21 [1]. > > The next concern is one you call out in the draft that ?Java's excellent serviceability has long been a source of pride for the platform.? We agree! > > Java currently has an excellent, prime position in Observability capabilities. For better or for worse, there are many Observability tools that have relied on dynamic attach to provide the necessary monitoring for workloads > > It?s important we give Java?s monitoring tools sufficient time to migrate comfortably without shocking the ecosystem by changing the default in an LTS. By delaying the change till JDK 22, we give the ecosystem 2 years to migrate and to prepare users for the change. > > Additionally, this provides the time for Java?s profiling tools to adapt as well. And for the ?ad-hoc troubleshooting? tools - like Byteman - to retrain their users. That?s a fair point. Even though the change was announced some years ago, some strong encapsulation features had a transition period where they emitted warnings before changing defaults. Since we can reasonably expect 21 to see relatively high adoption, we could take that opportunity to educate more users and only emit a warning when an agent is loaded dynamically (otherwise, since many users unfortunately skip versions, they would be equally surprised and unprepared at the next version they adopt as they would be if the default change were made in 21). Would you find that reasonable? If so, we may perhaps be able to also emit warnings on JNI use in 21, thus bringing agents, JNI, and FFM to the same baseline in 21, i.e. they would all issue warnings unless sanctioned by the application. > > Finally, while it?s easy to agree with the principle that ?the application owner is given the final say on whether to allow dynamic loading of agents?, the argument can (and should!) be made that those application owners have made that final decision by deploying the libraries and tools that use dynamic attach. A JVM command line argument is not the only way to express their consent for code to run on their system. > > For many production uses, the reality is more complicated than a single ?application owner?. Take a Kubernetes cluster for example. > > Who is the application owner when deploying to a Kubernetes cluster? The dev team that develops the application? The operations team that manages the cluster? The monitoring team that monitors the applications? The Support team that troubleshoots issues with the deployment? > > We should be careful not to understate or misrepresent the complex web of ?owners? that are currently able (and, for business reasons, need) to apply agents dynamically. Downplaying the complexity many organizations experience when dealing with changes to command line options (as an example) weakens the argument for changing today?s status quo. Right. In this case, ?owner? means any person who has been given the sufficient OS privileges to attach a dynamic agent (and who then also has sufficient privileges to stop or start the process). Because the ideal is not to disrupt tools at all but rather to prevent libraries from escalating their powers without the application?s knowledge and consent, we?ve begun to explore means other than the flag to allow a tool to load an agent at runtime. Two ideas we?ve had so far are a challenge-response mechanism that would verify there?s a person in the loop or issuing certificates to tools that would be used by the VM to verify that it is an approved tool that?s loading an agent (revoking certificates that find their way to libraries). These mechanisms are, however, complex, so they (or perhaps some other alternative) may appear only later. > > Dynamically attached agents have been a ?superpower? of Java and their use has greatly benefited the ecosystem as they?ve helped service and support use cases that otherwise aren?t possible, as they helped propel Java to the forefront of the Observability tooling, and allowed many other useful libraries to be developed. > > Let?s delay flipping the default until JDK 22 to give the breadth of the ecosystem time to absorb this change. Very well. If a warning is acceptable, we can do that in 21 and delay the default change to 22. ? Ron P.S. > We also know that in many cases customers and users may not be in a position to modify startup scripts (e.g. even to add in an extra parameter) as to do so may invalidate support contracts, etc. Could you expand more on that? Even if the default change happens in 22, it would not apply retroactively. Upgrading to a new JDK requires changing startup scripts, as does adding/upgrading libraries, which happens at least as frequently as upgrading a JDK version. How can a Java application be developed and deployed without the ability to change the command line? I can?t see how an application is expected to change its runtime version and yet not be able to change the command line? I mean, I can imagine setups where that could sometimes *happen* to work, but not a way for this to be *expected* to work. Certainly since the JRE was removed it?s been the assumption that upgrading a JDK version may require changing the command line. There are more important mechanisms than loading agents dynamically that require setting VM options, such as selecting a GC/heap configuration tailored to the application?s particular needs. Even in third-party hosting situations, the applications needs some level of control over the command line and the host will appreciate more control that allows it to select what capabilities it offers hosted applications. From mike at plan99.net Sun Apr 30 18:59:17 2023 From: mike at plan99.net (Mike Hearn) Date: Sun, 30 Apr 2023 20:59:17 +0200 Subject: [External] : Re: JEP draft: Disallow the Dynamic Loading of Agents by Default In-Reply-To: <1D74CB9A-1040-4992-954E-AAA1430FE4F0@oracle.com> References: <1D74CB9A-1040-4992-954E-AAA1430FE4F0@oracle.com> Message-ID: > we?ve begun to explore means other than the flag to allow a tool to load an agent at runtime How about restricting access to the jcmd socket. For in-VM code it can be blocked at the filesystem implementation level, and for sub-processes by using the operating system APIs to determine if the other side of the socket is part of the same process tree at connect time. This would avoid the need for new UI to re-enable existing jcmd functionality, whilst preventing code loaded into the VM from connecting back to that same VM. Only truly external tools could trigger agent loading, or modules that had been given permission to do that. That is assuming native code is restricted of course but that's a whole other kettle of fish. I don't quite understand how the plan is meant to work when it reaches that stage. JNI is so widely used and so many libraries would break that you'd either just immediately see workarounds appear, like build systems concatenating lists of flags from META-INF files, or it'd just cause a lot of chaos and upgrade rejection. It'd also have to be dynamically controllable for plugins, so then you'd also get people running their apps inside classloaders that auto-grant any request to load native code. There must be a better way? Maybe the problem can be recast as one of build-time observability? One concept I experimented with for my own build system is the notion of "control reports", a generalization of lockfiles. The build generates text files describing things you care about, and these are then checked in to vcs. There is a build mode that doesn't write the generated reports on disk, instead it verifies there's no difference and exits if there are. By checking the reports into version control and using OWNERS files + code reviews various policies can be enforced. In this context it'd work like so: the JVM would be changed to look for a flags file in the JAR containing the main class/main module as part of its startup. JVM flags can thus be shipped as part of the app, formalizing an already existing convention that's today implemented with shell scripts. Build systems can generate this flag file from their existing lists of JVM flags, and/or compute it automatically by merging flag files found in any library JARs. For people who don't care about deprivileging libs (prototyping, learners, people who only use in-house code etc) this will keep things working as it does today. For people who wish to use this new security feature they can review the generated flag file and check it in. Now if they add a dependency on a library that needs to use panama, jni, an agent, opened packages etc, this will become visible as added lines in the flag file, can be pushed back on when the commit is reviewed, and by placing an OWNERS file in the directory containing the report tech leads can allow delegates to add dependencies without being able to change JVM flags. From ron.pressler at oracle.com Sun Apr 30 22:24:13 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Sun, 30 Apr 2023 22:24:13 +0000 Subject: [External] : Re: JEP draft: Disallow the Dynamic Loading of Agents by Default In-Reply-To: References: <1D74CB9A-1040-4992-954E-AAA1430FE4F0@oracle.com> Message-ID: <41BFCE0A-AE21-468B-8CF0-57710034845C@oracle.com> Hi Mike! On 30 Apr 2023, at 19:59, Mike Hearn > wrote: we?ve begun to explore means other than the flag to allow a tool to load an agent at runtime How about restricting access to the jcmd socket. For in-VM code it can be blocked at the filesystem implementation level, and for sub-processes by using the operating system APIs to determine if the other side of the socket is part of the same process tree at connect time. This would avoid the need for new UI to re-enable existing jcmd functionality, whilst preventing code loaded into the VM from connecting back to that same VM. Only truly external tools could trigger agent loading, or modules that had been given permission to do that. Determining the process on the other side and/or maintaining the integrity of the process tree is not easy on all OSes. That is assuming native code is restricted of course but that's a whole other kettle of fish. I don't quite understand how the plan is meant to work when it reaches that stage. JNI is so widely used and so many libraries would break that you'd either just immediately see workarounds appear, like build systems concatenating lists of flags from META-INF files, or it'd just cause a lot of chaos and upgrade rejection. It'd also have to be dynamically controllable for plugins, so then you'd also get people running their apps inside classloaders that auto-grant any request to load native code. No libraries will break and no workarounds would be possible once all loopholes are closed. All you?d need to do is grant the module permission to load/use native libraries, be it through JNI or FFM. Build tools may do whatever they want, but it?s important to remember that the low-integrity free-for-all worked more-or-less only while Java was relatively stagnant. Once the platform started evolving more rapidly we saw many applications break in the Java 8 -> 9 upgrade due to deep dependencies that relied on JDK internals in the days before strong encapsulation. There must be a better way? Before thinking about other ways, I think it?s best to first acknowledge the problem, and that is that we wish for two fundamentally contradictory things: On the one hand, we want it to be easy to break other people?s code?s encapsulation when we need to, and on the other we want features that fundamentally depend on encapsulation not being broken, such as smooth upgrades, robust security mechanisms, and future Leyden features. Both may be good, but they are in conflict. Because we can't to have our invariants and break them too, someone must choose between one and the other. But I also don?t think it?s a very complicated choice. The view of the Java ecosystem as a hotbed of encapsulation-breaking is gradually becoming outdated. Much of the necessity to break encapsulation came about due to the platform?s stagnation in the JDK 6-8 timeframe. While the ability to break encapsulation is being restricted, the problems that required it in the first place are also being addressed. The number of uses that require breaking strong encapsulation is shrinking. Maybe the problem can be recast as one of build-time observability? One concept I experimented with for my own build system is the notion of "control reports", a generalization of lockfiles. The build generates text files describing things you care about, and these are then checked in to vcs. There is a build mode that doesn't write the generated reports on disk, instead it verifies there's no difference and exits if there are. By checking the reports into version control and using OWNERS files + code reviews various policies can be enforced. In this context it'd work like so: the JVM would be changed to look for a flags file in the JAR containing the main class/main module as part of its startup. JVM flags can thus be shipped as part of the app, formalizing an already existing convention that's today implemented with shell scripts. Build systems can generate this flag file from their existing lists of JVM flags, and/or compute it automatically by merging flag files found in any library JARs. For people who don't care about deprivileging libs (prototyping, learners, people who only use in-house code etc) this will keep things working as it does today. A main JAR, when used, is already given control over some command-line flags via its manifest, including strong encapsulation controls. That?s fine, because it?s controlled by the application, not libraries. For people who wish to use this new security feature It?s not a security feature. It?s an integrity feature. they can review the generated flag file and check it in. Now if they add a dependency on a library that needs to use panama, jni, an agent, opened packages etc, this will become visible as added lines in the flag file, can be pushed back on when the commit is reviewed, and by placing an OWNERS file in the directory containing the report tech leads can allow delegates to add dependencies without being able to change JVM flags. I think what you?re describing is an automated generation of what the informational JEP calls the codebase map (i.e. the command-line) based on what libraries want. Such automated permission-management could perhaps be appropriate for a security mechanism for mobile phone apps ? requesting access to files, for example, is quite common ? but not so much for integrity, as the need to break encapsulation is supposed to be abnormal and rare; it is certainly discouraged, and we believe that it will become less and less necessary (currently, serialization is probably the biggest issue, but we?re getting closer to a point where it won?t be: https://openjdk.org/projects/amber/design-notes/towards-better-serialization). Of course, tools outside the JDK can offer such a mechanism, but if the number of libraries requiring bypassing of encapsulation continues to decline that may be a rather elaborate solution to a rather small problem. Personally, I think it might be best to see where the ecosystem is in a few years. ? Ron -------------- next part -------------- An HTML attachment was scrubbed... URL: