From pavelturk2000 at gmail.com Mon Jan 1 15:51:24 2024 From: pavelturk2000 at gmail.com (PavelTurk) Date: Mon, 1 Jan 2024 17:51:24 +0200 Subject: How to get reads from module? Message-ID: <54fb9136-c7f4-1d65-8ac7-3287c13633e9@gmail.com> Hello all. I've asked this question, however I got no answer. So I decided to ask the question again providing more details. Consider the?following?layer?graph: 1)?boot 2)?layerA?(parent?-?boot)?has moduleX 3)?layerB?(parent?-?boot)?has?moduleX 4)?layerC?(parents?-?layerA,?layerB)?has?moduleY As?I?understand?it?is?a?possible?situation.?ModuleY?requires?moduleX?that?is?present?in?two?parent?layers. Now, I have a reference to moduleY: var moduleY = ....; And I need to open foo package of moduleX to moduleY dynamically: ModuleLayer.Controller.addOpens?(moduleX, foo, moduleY)//I have foo and moduleY The?question?-?how?can?I?find?out?from?which?layer?moduleX is used for moduleY so I could use `addOpens(..)`? Best regards, Pavel From michal at kleczek.org Tue Jan 2 07:56:34 2024 From: michal at kleczek.org (=?utf-8?Q?Micha=C5=82_K=C5=82eczek?=) Date: Tue, 2 Jan 2024 08:56:34 +0100 Subject: How to get reads from module? In-Reply-To: <54fb9136-c7f4-1d65-8ac7-3287c13633e9@gmail.com> References: <54fb9136-c7f4-1d65-8ac7-3287c13633e9@gmail.com> Message-ID: How about moduleY.layer().findModule(?moduleX?)? Micha? > On 1 Jan 2024, at 16:52, PavelTurk wrote: > > ?Hello all. I've asked this question, however I got no answer. So I decided to ask the question again > providing more details. > > Consider the following layer graph: > > 1) boot > 2) layerA (parent - boot) has moduleX > 3) layerB (parent - boot) has moduleX > 4) layerC (parents - layerA, layerB) has moduleY > > As I understand it is a possible situation. ModuleY requires moduleX that is present in two parent layers. > Now, I have a reference to moduleY: var moduleY = ....; And I need to open foo package of moduleX to moduleY > dynamically: > > ModuleLayer.Controller.addOpens?(moduleX, foo, moduleY)//I have foo and moduleY > > The question - how can I find out from which layer moduleX is used for moduleY so I could use `addOpens(..)`? > > Best regards, Pavel From Alan.Bateman at oracle.com Tue Jan 2 08:03:19 2024 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 2 Jan 2024 08:03:19 +0000 Subject: A couple of questions In-Reply-To: <7214b7d4-4002-1e0d-d7ee-f00eed90382f@gmail.com> References: <7214b7d4-4002-1e0d-d7ee-f00eed90382f@gmail.com> Message-ID: <331fd5e4-660f-485e-bf4b-5eecf4fa3487@oracle.com> On 29/12/2023 11:47, PavelTurk wrote: > Hello all. Could anyone help me with the following questions: > > 1. Let's suppose I have a reference to an unnamed module: > module.getName() == null . But I need at least some information about it. > How can I get file name of this module - jar/war etc? If you have a reference to an unnamed module then you can get its class loader with Module::getClassLoader. There's unnamed module for each class loader, 1:1 relationship. You can get the set of packages for the classes defined in the unnamed module with Module::getPackages which may help get closer to what you are looking for. That said, there is no API to expose the class path or the locations where the class loader has loaded classes. The class bytes can come from anywhere. What is the context for this question? Agents/tooling can use APIs to enumerate all classes loaded in the VM and most of these classes will have a URL code source. I think that is the closest to what you are looking for (if I understand your question correctly). > > 2. Let's suppose I have the following layer graph: > > 1) boot > 2) layerA (parent - boot) has moduleX > 3) layerB (parent - boot) has moduleX > 4) layerC (parents - layerA, layerB) has moduleY > > As I understand it is a possible situation. ModuleY requires moduleX > that is present in two parent layers. > The question - how can I find out from which layer moduleX will be > used for moduleY? The Configuration objects for layerA,?layerB and?layerC encapsulate the readability graphs used to the these module layers. Use Configuration::modules to get the set the resolved modules in the configuration, and ResovledRead::read to get the set of modules that are read. This will allow you to see which modules in C read "moduleX" in A and which modules in C read "moduleX" in B. -Alan. From Alan.Bateman at oracle.com Tue Jan 2 11:40:34 2024 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 2 Jan 2024 11:40:34 +0000 Subject: How to get reads from module? In-Reply-To: References: <54fb9136-c7f4-1d65-8ac7-3287c13633e9@gmail.com> Message-ID: <6b06a360-0069-488d-8b95-920e42f0fc64@oracle.com> On 02/01/2024 07:56, Micha? K?eczek wrote: > How about moduleY.layer().findModule(?moduleX?)? > That does a depth first search to find the "moduleX". I think Pavel is looking for the moduleX that is read by moduleY. This can be done in model world with the Configuration objects as they have the readability graphs to answer the question. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From michal at kleczek.org Tue Jan 2 11:55:31 2024 From: michal at kleczek.org (=?utf-8?Q?Micha=C5=82_K=C5=82eczek?=) Date: Tue, 2 Jan 2024 12:55:31 +0100 Subject: How to get reads from module? In-Reply-To: <6b06a360-0069-488d-8b95-920e42f0fc64@oracle.com> References: <6b06a360-0069-488d-8b95-920e42f0fc64@oracle.com> Message-ID: Now I am lost. Wouldn?t it find the same module? Micha? > On 2 Jan 2024, at 12:40, Alan Bateman wrote: > > ? On 02/01/2024 07:56, Micha? K?eczek wrote: >> How about moduleY.layer().findModule(?moduleX?)? >> > That does a depth first search to find the "moduleX". I think Pavel is looking for the moduleX that is read by moduleY. This can be done in model world with the Configuration objects as they have the readability graphs to answer the question. > > -Alan From Alan.Bateman at oracle.com Tue Jan 2 12:43:13 2024 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 2 Jan 2024 12:43:13 +0000 Subject: How to get reads from module? In-Reply-To: References: <6b06a360-0069-488d-8b95-920e42f0fc64@oracle.com> Message-ID: <10ada413-a882-4883-b44f-a5514427f60f@oracle.com> On 02/01/2024 11:55, Micha? K?eczek wrote: > Now I am lost. Wouldn?t it find the same module? > This is a readability graph rather than a dependency graph. They are the same thing until you add implied readability ("requires transitive") to the picture. In that case, it's possible for a module in layerC to require (and thus read) a module in layer A. That module in layer A, in turn, requires-transitive another module in layer A and thus our module in layerC reads two modules in layerA. It doesn't matter if there is a module in layerB with the same name as the second module in layerA, it's okay as long as there isn't a scenario where a module reads two (or more) modules with the same name. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From michal at kleczek.org Tue Jan 2 12:45:23 2024 From: michal at kleczek.org (=?utf-8?Q?Micha=C5=82_K=C5=82eczek?=) Date: Tue, 2 Jan 2024 13:45:23 +0100 Subject: How to get reads from module? In-Reply-To: <10ada413-a882-4883-b44f-a5514427f60f@oracle.com> References: <6b06a360-0069-488d-8b95-920e42f0fc64@oracle.com> <10ada413-a882-4883-b44f-a5514427f60f@oracle.com> Message-ID: > On 2 Jan 2024, at 13:43, Alan Bateman wrote: > > On 02/01/2024 11:55, Micha? K?eczek wrote: >> Now I am lost. Wouldn?t it find the same module? >> > This is a readability graph rather than a dependency graph. They are the same thing until you add implied readability ("requires transitive") to the picture. Ahh, right. Thanks! ? Michal From pavelturk2000 at gmail.com Wed Jan 10 10:29:33 2024 From: pavelturk2000 at gmail.com (PavelTurk) Date: Wed, 10 Jan 2024 12:29:33 +0200 Subject: A couple of questions In-Reply-To: <331fd5e4-660f-485e-bf4b-5eecf4fa3487@oracle.com> References: <7214b7d4-4002-1e0d-d7ee-f00eed90382f@gmail.com> <331fd5e4-660f-485e-bf4b-5eecf4fa3487@oracle.com> Message-ID: >> >> 2. Let's suppose I have the following layer graph: >> >> 1) boot >> 2) layerA (parent - boot) has moduleX >> 3) layerB (parent - boot) has moduleX >> 4) layerC (parents - layerA, layerB) has moduleY >> >> As I understand it is a possible situation. ModuleY requires moduleX that is present in two parent layers. >> The question - how can I find out from which layer moduleX will be used for moduleY? > > The Configuration objects for layerA,?layerB and?layerC encapsulate the readability graphs used to the these module layers. Use Configuration::modules to get the set the resolved modules in the configuration, and ResovledRead::read to get the set of modules that are read. This will allow you to see which modules in C read "moduleX" in A and which modules in C read "moduleX" in B. > Still can't get reference to read modules: ??????? layerC.configuration().modules().stream().forEach((m) -> { ??????????? for (ResolvedModule r : m.reads()) { ??????????????? //LINE X ??????????? } ??????? }); At LINE X I need to get `Module` from `ResolvedModule` to use Module.getLayer() method. How to do it? Best regards, Pavel From Alan.Bateman at oracle.com Thu Jan 11 13:30:02 2024 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 11 Jan 2024 13:30:02 +0000 Subject: A couple of questions In-Reply-To: References: <7214b7d4-4002-1e0d-d7ee-f00eed90382f@gmail.com> <331fd5e4-660f-485e-bf4b-5eecf4fa3487@oracle.com> Message-ID: On 10/01/2024 10:29, PavelTurk wrote: > : > > Still can't get reference to read modules: > > ??????? layerC.configuration().modules().stream().forEach((m) -> { > ??????????? for (ResolvedModule r : m.reads()) { > ??????????????? //LINE X > ??????????? } > ??????? }); > > At LINE X I need to get `Module` from `ResolvedModule` to use > Module.getLayer() method. How to do it? Starting in model world (Configuration/ResolvedModule) is right and you first need to find the Configuration containing the moduleX that moduleY reads, this should do it: ??????? Configuration cf = layerC.configuration() ??????????????? .findModule("moduleY") ??????????????? .orElseThrow() ??????????????? .reads().stream() ??????????????? .filter(m -> m.name().equals("moduleX")) ??????????????? .map(ResolvedModule::configuration) ??????????????? .findAny() ??????????????? .orElseThrow(); Once you have the Configuration then you need to find the ModuleLayer that was reifed from that Configuration, this should do it: ??????? ModuleLayer layer = layerC.parents().stream() ??????????????? .filter(l -> l.configuration() == cf) ??????????????? .findAny() ??????????????? .orElseThrow(); Can you try it? -Alan From pavelturk2000 at gmail.com Thu Jan 11 21:57:05 2024 From: pavelturk2000 at gmail.com (PavelTurk) Date: Thu, 11 Jan 2024 23:57:05 +0200 Subject: A couple of questions In-Reply-To: References: <7214b7d4-4002-1e0d-d7ee-f00eed90382f@gmail.com> <331fd5e4-660f-485e-bf4b-5eecf4fa3487@oracle.com> Message-ID: > Starting in model world (Configuration/ResolvedModule) is right and you first need to find the Configuration containing the moduleX that moduleY reads, this should do it: > Yes, I understand it, I sent a piece of real application code, so it was a little different. > Once you have the Configuration then you need to find the ModuleLayer that was reifed from that Configuration, this should do it: > > ??????? ModuleLayer layer = layerC.parents().stream() > ??????????????? .filter(l -> l.configuration() == cf) > ??????????????? .findAny() > ??????????????? .orElseThrow(); > > Can you try it? > Yes, it worked. Thank you very much for your help. I didn't understand that we could map layer and modules of this layer by configuration. So I just created Map where as a key I used the configuration of all layers and after that for (var readModule : m.reads()) { ??? var layer = moduleLayersByConfiguration.get(readModule.configuration()); } Several notes: 1. Your code `ModuleLayer layer = layerC.parents().stream()...` didn't find layer if moduleX was in boot layer, but not in direct parents of layerC. In our example 4 layers: 1)?boot 2)?layerA?(parent?-?boot)?has?moduleX 3)?layerB?(parent?-?boot)?has?moduleX 4)?layerC?(parents?-?layerA,?layerB)?has?moduleY 2. Please see this diagram https://i.stack.imgur.com/ss2os.png that was generated for real application using reads and ResolvedModules (the topic of this thread). Here we have two layers - the right is a boot layer and the left is a webserver layer. The program must show reads of com.google.gson. Reads are shown using dashed arrows. a) we see that gson reads 5 modules. At the same time gson requires only 1 module: module com.google.gson { ??? exports com.google.gson; ??? exports com.google.gson.annotations; ??? exports com.google.gson.reflect; ??? exports com.google.gson.stream; ??? requires transitive java.sql; } Do I understand it correctly that extra 4 modules are read via java.sql? 3. Do I understand it correctly if some explicit module (with module-info) from webserver layer will require any automatic module from webserver layer then this explicit module will read ALL automatic modules from both layers? Best regards, Pavel From Alan.Bateman at oracle.com Fri Jan 12 14:23:14 2024 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 12 Jan 2024 14:23:14 +0000 Subject: A couple of questions In-Reply-To: References: <7214b7d4-4002-1e0d-d7ee-f00eed90382f@gmail.com> <331fd5e4-660f-485e-bf4b-5eecf4fa3487@oracle.com> Message-ID: <9d0bdf6e-1532-4cf7-af07-56811cf4407d@oracle.com> On 11/01/2024 21:57, PavelTurk wrote: > : > > a) we see that gson reads 5 modules. At the same time gson requires > only 1 module: > module com.google.gson { > ??? exports com.google.gson; > ??? exports com.google.gson.annotations; > ??? exports com.google.gson.reflect; > ??? exports com.google.gson.stream; > > ??? requires transitive java.sql; > } > > Do I understand it correctly that extra 4 modules are read via java.sql? In the API docs for java.sql you will see that it requires transitive java.logging, java.transaction.xa, and transitive java.xml. This means that com.google.gson will read java.sql, java.logging, java.transaction.xa, and transitive java.xml. In addition, every module reads java.base so that explains the edges you see in the readability graph. > > 3. Do I understand it correctly if some explicit module (with > module-info) from webserver layer will require any automatic module > from webserver layer then this explicit > module will read ALL automatic modules from both layers? If an automatic module is enumerated during resolution when all observable automatic modules are enumerated. If an explicit module requires an automatic module then the explicit module will read all enumerated automatic modules. That said, there is an open issue around readability when there is a stack or tree of configurations when there are modules with the same name in different configurations. The science is right when they are all explicit modules but there is an issue when there are automatic modules in configurations other than the configuration for the boot layer and where the module names aren't globally unique.? This is the issue we were discussing in other thread where it will require both specification and implementation changes to address. -Alan From pavelturk2000 at gmail.com Mon Jan 15 16:03:21 2024 From: pavelturk2000 at gmail.com (PavelTurk) Date: Mon, 15 Jan 2024 18:03:21 +0200 Subject: What is the practical use of JPMS layers? Message-ID: <68259665-8305-c594-3e15-b905c48dbaeb@gmail.com> Hello all. JPMS is not only about modules, but it also supports layers. What is the practical use of JPMS layers? Best regards, Pavel From sormuras at gmail.com Tue Jan 16 07:10:58 2024 From: sormuras at gmail.com (Christian Stein) Date: Tue, 16 Jan 2024 08:10:58 +0100 Subject: What is the practical use of JPMS layers? In-Reply-To: <68259665-8305-c594-3e15-b905c48dbaeb@gmail.com> References: <68259665-8305-c594-3e15-b905c48dbaeb@gmail.com> Message-ID: An exemplary use of module layers is documented and implemented here: https://github.com/moditect/layrry Copied from its "Why Layrry?" section: This is where Layrry comes in: utilizing the notion of module layers, it > provides a declarative approach as well as an API for assembling > modularized applications, organized in module layers. The JARs to be > included are described using Maven GAV (group id, artifact id, version) > coordinates, solving the issue of retrieving all required JARs in the right > version. Module layers allow to use different versions of one and the same module in > different layers of an application (as long as they are not exposed in a > conflicting way on module API boundaries). Cheers, Christian On Mon, Jan 15, 2024 at 6:58?PM PavelTurk wrote: > Hello all. > > JPMS is not only about modules, but it also supports layers. What is the > practical use of JPMS layers? > > Best regards, Pavel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pavelturk2000 at gmail.com Wed Jan 17 17:22:08 2024 From: pavelturk2000 at gmail.com (PavelTurk) Date: Wed, 17 Jan 2024 19:22:08 +0200 Subject: Is it possible to add JRE and JDK modules in child layers? Message-ID: Hello all. We use JPMS for our framework where we have the following layers: 1) boot layer: framework + JRE/JDK modules (to add them we use --add-modules ALL-DEFAULT) 2) child layers [with child layers..] where every child layer is dynamically added component. As I understand Layrry framework does the same.? I asked several days ago about practical use of JPMS layers and got only one response about Layrry. So, I can conclude that layers are used primarily for creating dynamic substems, plugins, components, extensions etc. However, we couldn't find a way to add JRE and JDK modules in child layers. Because of this we have to add all default modules to boot layer what is a very very serious flaw. Why do we need it? As components are created dynamically when/if they are required we don't know what JRE/JDK modules will be required we create boot layer. That's why add all of them to boot layer. Could anyone say if it is possible to add JRE/JDK modules to child layers? Is there any way to do it? Are there any plans to add such possibility? Best regards, Pavel From Alan.Bateman at oracle.com Thu Jan 18 09:04:54 2024 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 18 Jan 2024 09:04:54 +0000 Subject: Is it possible to add JRE and JDK modules in child layers? In-Reply-To: References: Message-ID: On 17/01/2024 17:22, PavelTurk wrote: > Hello all. > > We use JPMS for our framework where we have the following layers: > > 1) boot layer: framework + JRE/JDK modules (to add them we use > --add-modules ALL-DEFAULT) > 2) child layers [with child layers..] where every child layer is > dynamically added component. > > As I understand Layrry framework does the same.? I asked several days > ago about practical use > of JPMS layers and got only one response about Layrry. So, I can > conclude that layers are used > primarily for creating dynamic substems, plugins, components, > extensions etc. > > However, we couldn't find a way to add JRE and JDK modules in child > layers. Because of this we > have to add all default modules to boot layer what is a very very > serious flaw. > > Why do we need it? As components are created dynamically when/if they > are required we don't > know what JRE/JDK modules will be required we create boot layer. > That's why add all of them to > boot layer. > > Could anyone say if it is possible to add JRE/JDK modules to child > layers? Is there any way to do it? > Are there any plans to add such possibility? This is the topic that was on the original requirements wish list as "Run-time augmentation of the platform modules". There isn't any general support for this in the JDK. One reason is that module layers, and in this case the boot layer, are immutable. Another reason is that integrity of the platform requires that classes in the java.* packages be defined to the boot or platform class loaders. In other words, you can't have child layers containing modules with java.* classes that are mapped to custom class loaders. That said, there are two things to be aware of: 1. ALL-DEFAULT. You've already found that. As explained in JEP 261, this is there to support cases where a container module is in the boot layer but it does dynamic configuration with modules that require standard modules that might not be in the boot layer. 2. Java agents and the JMX/management agent. If an agent is started in a running VM, and the required java.instrument or java.management modules are not in the boot layer, then a child layer will magically comes into being with the required standard + JDK modules. This it possible because the starting of these agents is under the control of the VM with the original mapping of modules to class loaders. To your question, you can't use the API to do dynamic configuration and create module layers that contain the standard modules. The solution for containers is ALL-DEFAULT to ensure that all modules in the system image that export an API are in the boot layer. -Alan From pavelturk2000 at gmail.com Thu Jan 18 13:49:01 2024 From: pavelturk2000 at gmail.com (PavelTurk) Date: Thu, 18 Jan 2024 15:49:01 +0200 Subject: Is it possible to add JRE and JDK modules in child layers? In-Reply-To: References: Message-ID: <18b34e7d-dad8-e57d-5328-083bf6cb734e@gmail.com> Hi Alan, Thank you very much for your answer. You wrote: > This is the topic that was on the original requirements wish list as "Run-time augmentation of the platform modules". There isn't any general support for this in the JDK. One reason is that module layers, and in this case the boot layer, are immutable. Another reason is that integrity of the platform requires that classes in the java.* packages be defined to the boot or platform class loaders. In other words, you can't have child layers containing modules with java.* classes that are mapped?to?custom?class?loaders. I think that that all layers (including boot layer) must be immutable. What about "Another reason is that integrity of the platform requires that classes in the java.* packages be defined to the boot or platform class loaders". Do I understand it correctly that it is ABSOLUTELY, ABSOLUTELY impossible to do find a solution for this problem? I mean that all today's knowledge in software development are still not enough to find a way a) to provide integrity of the platform and b) to load standard (JRE/JDK) modules in child layers? Best regards, Pavel From tomas.langer at oracle.com Fri Jan 19 13:02:55 2024 From: tomas.langer at oracle.com (Tomas Langer) Date: Fri, 19 Jan 2024 13:02:55 +0000 Subject: Java extensibility and JPMS (ServiceLoader) Message-ID: Helidon currently has around 300 modules with module-info.java. In general, this has improved our module structure and design. Yet, we are now encountering some major issues related to extensibility. I will put down a few points that are problematic, and explain each in detail further in the e-mail (it is quite long, sorry about that). 1. provider implementations cannot be code generated without major problems 2. the provider interface module MUST be on module path, even if it could have `requires static` 3. the provider implementation must be public with public constructor 4. duality of definition between module path and class path I am trying to propose a solution within the bounds of the current service loader design. Of course there may be other solutions (both with current design, or even creating a brand new extensibility solution in Java, this is just to illustrate it). The first two issues are quite major, as they force us to recommend not to use JPMS to our users... We could design our own extensibility approach, though that would require the use of reflection to instantiate the services, and we would have to live with the limits of the module system (where Java ServiceLoader works around a few of them). I feel the right way is to use what Java provides, so I would welcome any help (and possibly changes in the language) to support our use cases. Thanks, Tomas Langer Architect, project Helidon Ad 1 - Code generation ------------------------------------- Problem: We cannot code generate service implementations (well we can, but the user must handcraft them in module-info.java, so we end up running the APT, generating a service, failing the compiler to tell the user to add the service, compiling again every time a new service is added). Possible solution: Provide extensibility to module-info.java that can be code generated Without JPMS: it just works, as `META-INF/services` files can be code generated without issues Details: What I do not see is how we are supposed to do extensibility through annotation processing. There are a lot of usecases for this, such as: - generating code for serializers/deserializers for objects that persist to JSON, XML, YAML - generating code for database entities - generating descriptor for services in a service registry We actually want to implement these three use cases, and it is a major pain for the user - I would not mind much if this hurt us, as framework developers, but we must force the user to take action by breaking the compilation, or come up with some really weird solution (such as source code modification using some preprocessor before compilation, or postprocessor running on bytecode to re-generate module-info.class) Ad 2 - Provider module cannot be optional dependency ------------------------------------- Problem: We cannot declare `requires static` on a module that has the ServiceLoader provider interface (or abstract class) Possible solution: Change the rules for JPMS to allow this Without JPMS: it just works, as `META-INF/services` to not impose any classpath structure Details: If a module (my.json) defines a provider interface (let's say `JsonSerializer`), and I create a module with `MyJsonSerializer` (provides JsonSerializer witih MyJsonSerializer), currently I MUST do the following "requires my.json". If I do a "requires static my.json" I fail to start the JVM if that module is not on module path. The service CANNOT be used unless the module is on module path (as anybody attempting to load it must declare `uses` in their module info with a proper `requires` on the `my.json` module. So what are my options right now? - have a "requires my.json" and just dump the module on all my users (not so good - people may want to use my library without JSON altogether, I may also provide support for XML, YAML - all of these would need to be on module path) - create a module for each (one for JSON, XML, YAML + my library) - resulting in 4 modules (and this may grow if I decide to support other format); this looks kind of OK on the first look, but with the number of modules we have, and the number of features we support, this gets out of hand really really quickly; this approach is also very user unfriendly, as now the user needs to understand 4 modules instead of just 1, and use the right ones at the right time). - considering the number of modules we already have, this would make our project unmaintainable (and unusable for users) As JPMS already allows "static" dependencies, there should be no reason not to allow it in this case as well. We can break the module system even now - just use a class from static dependency in a public class - this will fail at runtime only. The service loader is not different (this is a reaction to text in https://bugs.openjdk.org/browse/JDK-8299504). Ad 3 - Provider implementation must be public with public constructor ------------------------------------- Problem: This creates a new public API that we may not want to expose, or document Possible solution: Change the rules to allow package local service provider implementations with package local constructors Without JPMS: Same issue, even more problematic as there is no restrictions on package visibility Details: Provider implementations are not supposed to be visible to users - they are not public API of my module (the fact that I provide a service is part of my public API). Right now there is only one option to work around this, and it only works in JPMS, and in my opinion it brings in even more problems - put the provider implementation in an un-exported package. The problem with this approach is that now the provider implementation MUST use only public methods of my module, thus creating even more public APIs, where if I just put it in my exported package, I can use package private methods of my other classes to implement the service (so I pay the price of having one public class with one public constructor agains multiple public classes and public methods). Also the "hiding" in unexported package is lost when on classpath anyway... Ad 4 - Duality of definition between classpath and module path ------------------------------------- Problem: To support services, we MUST declare them twice - once in `provides` in module-info.java, once through META-INF/services Possible solution: Java could read module descriptors even when running in classpath mode to add service implementations and merge it with META-INF/services information Without JPMS: it just works, as `META-INF/services` is always honored on classpath and for non-JPMS modules on module path Details: This is again quite a pain for us as framework develoepers, and a pitfall for users. When we started with JPMS, we had both created manually, which obviously ended in a huge inconsistent mess. So now we have a custom Maven plugin, that creates META-INF/services files based on the content in module-info.java and fails on inconsistencies. I do not consider this a nice solution for us, and definitely not for end users. Also there is no way to find out that you forgot to add one (or the other), as JVM just does not care. So basically you end up with a runtime issue that is really hard to troubleshoot. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Fri Jan 19 14:30:20 2024 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 19 Jan 2024 14:30:20 +0000 Subject: Is it possible to add JRE and JDK modules in child layers? In-Reply-To: <18b34e7d-dad8-e57d-5328-083bf6cb734e@gmail.com> References: <18b34e7d-dad8-e57d-5328-083bf6cb734e@gmail.com> Message-ID: <29624b9e-910c-4193-8b01-5814d1cae6b4@oracle.com> On 18/01/2024 13:49, PavelTurk wrote: > : > > Do I understand it correctly that it is ABSOLUTELY, ABSOLUTELY > impossible to do find a solution for this problem? > I mean that all today's knowledge in software development are still > not enough to find a way > a) to provide integrity of the platform and b) to load standard > (JRE/JDK) modules in child layers? I didn't say it was impossible, instead I listed down some of the challenges with finding a workable solution to this type of use case. The science isn't too difficult, it's mostly about slicing a module graph and reifying the slices as a stack of ModuleLayers so that the ModuleLayer.Controller handed out to the container is for a ModueLayer that doesn't include the modules mapped to the boot or platform class loaders. The harder part is coming up with an API that looks like it has always been there. For now, deploying the container with --add-modules ALL-DEFAULT is not terrible. It ensures that all modules in the run-time image that export an API are in the boot layer so there shouldn't be any issues with modules in child layers requiring modules that the container doesn't transitively require. -Alan From josiahnoel at gmail.com Fri Jan 19 15:17:30 2024 From: josiahnoel at gmail.com (Josiah Noel) Date: Fri, 19 Jan 2024 10:17:30 -0500 Subject: Java extensibility and JPMS (ServiceLoader) In-Reply-To: References: Message-ID: I help maintain a few service-loader-based modular annotation processor libraries with the avaje framework, so I have also run into this and had to work around it. *Ad 1:* So this is indeed an inconvenience but in my experience, it's not a crazy one. When processing is over we check the module-info's directives to throw compilation warnings like: *[ERROR] /M:/Dev/avaje-helidon-nima-api-example/src/main/java/module-info.java:[1,1] Missing `provides io.ava* *je.http.client.HttpClient.GeneratedComponent with com.jojo.helidon.api.client.httpclient.GeneratedHttpCompon* *ent;`* *[ERROR] /M:/Dev/avaje-helidon-nima-api-example/src/main/java/module-info.java:[1,1] Missing `provides io.ava* *je.jsonb.Jsonb.GeneratedComponent with com.jojo.helidon.api.jsonb.GeneratedJsonComponent;`* *[ERROR] /M:/Dev/avaje-helidon-nima-api-example/src/main/java/module-info.java:[1,1] Missing `provides io.ava* *je.validation.Validator.GeneratedComponent with com.jojo.helidon.api.controller.valid.GeneratedValidatorComp* *onent;`* *[ERROR] /M:/Dev/avaje-helidon-nima-api-example/src/main/java/module-info.java:[1,1] Missing "provides io.ava* *je.inject.spi.Module with com.jojo.helidon.api.ApiModule;"* The *real problem *with checking the module info in a processor is that ModuleElement is bugged such that if you call *ModuleElement.getDirectives* on the project module It breaks compilation . (they called this one a duplicate but I still have the problem even in JDK 22-ea and JDK 23-ea) To get around this I have to use Filer to retrieve the module-info's sources file and parse it as a string to avoid calling *getDirectives. * I've built libraries to do this to make this easier, but the problem remains that if any other processor uses *getDirectives*, compilation will still break. *Ad 2:* I asked this question here a while back, and what I got out of it is that we needed to do some form of circular dependency if we truly wanted optional services, the tech for compiling multi-module jars isn't here yet. It's a pain to deploy when we change the plugins, but we rarely do so this has worked fine for us. Example: avaje-validator provides a plugin for avaje inject that itself depends on avaje-validator. *Ad 3: * Yeah I got nothing, we never had to do this. *Ad 4: * This one was the simplest, we can define an annotation to go on the user's service class and process them to generate META-INF files and validate the module-info. In this way, one cannot forget to add the proper module-info information. In some of the libs we do this with their processors, but we also have a dedicated library for handling this sort of thing. On Fri, Jan 19, 2024 at 8:04?AM Tomas Langer wrote: > Helidon currently has around 300 modules with module-info.java. In > general, this has improved our module structure and design. > Yet, we are now encountering some major issues related to extensibility. > I will put down a few points that are problematic, and explain each in > detail further in the e-mail (it is quite long, sorry about that). > > 1. provider implementations cannot be code generated without major problems > 2. the provider interface module MUST be on module path, even if it could > have `requires static` > 3. the provider implementation must be public with public constructor > 4. duality of definition between module path and class path > > I am trying to propose a solution within the bounds of the current service > loader design. Of course there may be other solutions (both with current > design, or even creating a brand new extensibility solution in Java, this > is just to illustrate it). > > The first two issues are quite major, as they force us to recommend not to > use JPMS to our users... > > We could design our own extensibility approach, though that would require > the use of reflection to instantiate the services, and we would have to > live with the limits of the module system (where Java ServiceLoader works > around a few of them). > I feel the right way is to use what Java provides, so I would welcome any > help (and possibly changes in the language) to support our use cases. > > Thanks, > Tomas Langer > Architect, project Helidon > > > *Ad 1 - Code generation* > ------------------------------------- > Problem: We cannot code generate service implementations (well we can, but > the user must handcraft them in module-info.java, so we end up running the > APT, generating a service, failing the compiler to tell the user to add the > service, compiling again every time a new service is added). > Possible solution: Provide extensibility to module-info.java that can be > code generated > Without JPMS: it just works, as `META-INF/services` files can be code > generated without issues > > Details: > What I do not see is how we are supposed to do extensibility through > annotation processing. > There are a lot of usecases for this, such as: > - generating code for serializers/deserializers for objects that persist > to JSON, XML, YAML > - generating code for database entities > - generating descriptor for services in a service registry > > We actually want to implement these three use cases, and it is a major > pain for the user - I would not mind much if this hurt us, as framework > developers, but we must force the user to take action by breaking the > compilation, or come up with some really weird solution (such as source > code modification using some preprocessor before compilation, or > postprocessor running on bytecode to re-generate module-info.class) > > > *Ad 2 - Provider module cannot be optional dependency* > ------------------------------------- > Problem: We cannot declare `requires static` on a module that has the > ServiceLoader provider interface (or abstract class) > Possible solution: Change the rules for JPMS to allow this > Without JPMS: it just works, as `META-INF/services` to not impose any > classpath structure > > Details: > If a module (my.json) defines a provider interface (let's say > `JsonSerializer`), and I create a module with `MyJsonSerializer` (provides > JsonSerializer witih MyJsonSerializer), currently I MUST do the following > "requires my.json". If I do a "requires static my.json" I fail to start the > JVM if that module is not on module path. > > The service CANNOT be used unless the module is on module path (as anybody > attempting to load it must declare `uses` in their module info with a > proper `requires` on the `my.json` module. > > So what are my options right now? > - have a "requires my.json" and just dump the module on all my users (not > so good - people may want to use my library without JSON altogether, I may > also provide support for XML, YAML - all of these would need to be on > module path) > - create a module for each (one for JSON, XML, YAML + my library) - > resulting in 4 modules (and this may grow if I decide to support other > format); this looks kind of OK on the first look, but with the number of > modules we have, and the number of features we support, this gets out of > hand really really quickly; this approach is also very user unfriendly, as > now the user needs to understand 4 modules instead of just 1, and use the > right ones at the right time). > - considering the number of modules we already have, this would make our > project unmaintainable (and unusable for users) > > As JPMS already allows "static" dependencies, there should be no reason > not to allow it in this case as well. We can break the module system even > now - just use a class from static dependency in a public class - this will > fail at runtime only. The service loader is not different (this is a > reaction to text in https://bugs.openjdk.org/browse/JDK-8299504). > > > *Ad 3 - Provider implementation must be public with public constructor* > ------------------------------------- > Problem: This creates a new public API that we may not want to expose, or > document > Possible solution: Change the rules to allow package local service > provider implementations with package local constructors > Without JPMS: Same issue, even more problematic as there is no > restrictions on package visibility > > Details: > Provider implementations are not supposed to be visible to users - they > are not public API of my module (the fact that I provide a service is part > of my public API). > Right now there is only one option to work around this, and it only works > in JPMS, and in my opinion it brings in even more problems - put the > provider implementation in an un-exported package. > The problem with this approach is that now the provider implementation > MUST use only public methods of my module, thus creating even more public > APIs, where if I just put it in my exported package, I can use package > private methods of my other classes to implement the service (so I pay the > price of having one public class with one public constructor agains > multiple public classes and public methods). Also the "hiding" in > unexported package is lost when on classpath anyway... > > > *Ad 4 - Duality of definition between classpath and module path* > ------------------------------------- > Problem: To support services, we MUST declare them twice - once in > `provides` in module-info.java, once through META-INF/services > Possible solution: Java could read module descriptors even when running in > classpath mode to add service implementations and merge it with > META-INF/services information > Without JPMS: it just works, as `META-INF/services` is always honored on > classpath and for non-JPMS modules on module path > > Details: > This is again quite a pain for us as framework develoepers, and a pitfall > for users. When we started with JPMS, we had both created manually, which > obviously ended in a huge inconsistent mess. > So now we have a custom Maven plugin, that creates META-INF/services files > based on the content in module-info.java and fails on inconsistencies. > I do not consider this a nice solution for us, and definitely not for end > users. Also there is no way to find out that you forgot to add one (or the > other), as JVM just does not care. So basically you end up with a runtime > issue that is really hard to troubleshoot. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From johannes.spangenberg at hotmail.de Sun Jan 21 01:19:02 2024 From: johannes.spangenberg at hotmail.de (Johannes Spangenberg) Date: Sun, 21 Jan 2024 02:19:02 +0100 Subject: Java extensibility and JPMS (ServiceLoader) In-Reply-To: References: Message-ID: > *Ad 2 - Provider module cannot be optional dependency* > ------------------------------------- > Problem: We cannot declare `requires static` on a module that has the > ServiceLoader provider interface (or abstract class) Note that there was a relative long discussion about this exact topic in April 2023: https://mail.openjdk.org/pipermail/jigsaw-dev/2023-April/014846.html I personally think that the current implementation is too restrictive as I see very little gain in this runtime error, but there was also a noticeable opposition. > *Ad 4 - Duality of definition between classpath and module path* > ------------------------------------- > Problem: To support services, we MUST declare them twice - once in > `provides` in module-info.java, once through META-INF/services > Possible solution: Java could read module descriptors even when > running in classpath mode to add service implementations and merge it > with META-INF/services information While I did also run into this issue, I would favor a solution during the build phase. Maybe some project still uses Java 8, where the JDK does not yet know about the module-info. The behavior should not change between different versions of the runtime. PS: I am not affiliated with the project, besides being subscribed to this mailing list. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josiahnoel at gmail.com Sun Jan 21 04:16:22 2024 From: josiahnoel at gmail.com (Josiah Noel) Date: Sat, 20 Jan 2024 23:16:22 -0500 Subject: Java extensibility and JPMS (ServiceLoader) In-Reply-To: References: Message-ID: > > Note that there was a relative long discussion about this exact topic in > April 2023: > Should multi-module jars be implemented, most issues with problem 2 would be resolved without needing to resort to circular dependencies or requiring the user to remember a bunch of extra service modules. Most apps deploying as a jar would have the optional services work essentially as if they were on the classpath. The one exception to this case I've found is that applications deployed as fully modular jlink runtime images require the optional service modules to be explicitly defined in the module-info or added to the jlink command. (which is a pain for me, as I like using jlink to add my application modules to the runtime) This too I've had to deal with by scanning the module info in the processor and giving compiler warnings if a service is detected in the processing environment and not specified in the module-info. It's unfortunate though that this limitation will continue to exist. While I did also run into this issue, I would favor a solution during the > build phase. Maybe some project still uses Java 8, where the JDK does not > yet know about the module-info. The behavior should not change between > different versions of the runtime. Well, Helidon's baseline is now JDK 21, so Java 8 compatibility shouldn't be a concern. I still think that auto-generating service files with an annotation processor is the way to go if an alternative easier solution cannot be found in this discussion. On Sat, Jan 20, 2024 at 8:19?PM Johannes Spangenberg < johannes.spangenberg at hotmail.de> wrote: > *Ad 2 - Provider module cannot be optional dependency* > ------------------------------------- > Problem: We cannot declare `requires static` on a module that has the > ServiceLoader provider interface (or abstract class) > > Note that there was a relative long discussion about this exact topic in > April 2023: > > https://mail.openjdk.org/pipermail/jigsaw-dev/2023-April/014846.html > > I personally think that the current implementation is too restrictive as I > see very little gain in this runtime error, but there was also a noticeable > opposition. > > *Ad 4 - Duality of definition between classpath and module path* > ------------------------------------- > Problem: To support services, we MUST declare them twice - once in > `provides` in module-info.java, once through META-INF/services > Possible solution: Java could read module descriptors even when running in > classpath mode to add service implementations and merge it with > META-INF/services information > > While I did also run into this issue, I would favor a solution during the > build phase. Maybe some project still uses Java 8, where the JDK does not > yet know about the module-info. The behavior should not change between > different versions of the runtime. > > PS: I am not affiliated with the project, besides being subscribed to this > mailing list. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pavelturk2000 at gmail.com Mon Jan 22 10:07:37 2024 From: pavelturk2000 at gmail.com (PavelTurk) Date: Mon, 22 Jan 2024 12:07:37 +0200 Subject: Is it possible to add JRE and JDK modules in child layers? In-Reply-To: <29624b9e-910c-4193-8b01-5814d1cae6b4@oracle.com> References: <18b34e7d-dad8-e57d-5328-083bf6cb734e@gmail.com> <29624b9e-910c-4193-8b01-5814d1cae6b4@oracle.com> Message-ID: <0c2552cc-895b-96e6-1524-5488ed420749@gmail.com> Alan, thank you very much for your explanations. I created an issue https://bugs.openjdk.org/browse/JDK-8324272 So, everyone who is interested in this topic can track it. Best regards, Pavel On 1/19/24 4:30 PM, Alan Bateman wrote: > On 18/01/2024 13:49, PavelTurk wrote: >> : >> >> Do I understand it correctly that it is ABSOLUTELY, ABSOLUTELY impossible to do find a solution for this problem? >> I mean that all today's knowledge in software development are still not enough to find a way >> a) to provide integrity of the platform and b) to load standard (JRE/JDK) modules in child layers? > > I didn't say it was impossible, instead I listed down some of the challenges with finding a workable solution to this type of use case. The science isn't too difficult, it's mostly about slicing a module graph and reifying the slices as a stack of ModuleLayers so that the ModuleLayer.Controller handed out to the container is for a ModueLayer that doesn't include the modules mapped to the boot or platform class loaders. The harder part is coming up with an API that looks like it has always been there. > > For now, deploying the container with --add-modules ALL-DEFAULT is not terrible. It ensures that all modules in the run-time image that export an API are in the boot layer so there shouldn't be any issues with modules in child layers requiring modules that the container doesn't transitively require. > > -Alan From alex.buckley at oracle.com Mon Jan 22 20:13:39 2024 From: alex.buckley at oracle.com (Alex Buckley) Date: Mon, 22 Jan 2024 12:13:39 -0800 Subject: Java extensibility and JPMS (ServiceLoader) In-Reply-To: References: Message-ID: Tomas, A clarification for point 3: On 1/19/2024 5:02 AM, Tomas Langer wrote: > *Ad 3 - Provider implementation must be public with public constructor* > ------------------------------------- > Details: > Provider implementations are not supposed to be visible to users - they > are not public API of my module (the fact that I provide a service is > part of my public API). > Right now there is only one option to work around this, and it only > works in JPMS, and in my opinion it brings in even more problems - put > the provider implementation in an un-exported package. > The problem with this approach is that now the provider implementation > MUST use only public methods of my module, thus creating even more > public APIs, where if I just put it in my exported package, I can use > package private methods of my other classes to implement the service (so > I pay the price of having one public class with one public constructor > agains multiple public classes and public methods). Also the "hiding" in > unexported package is lost when on classpath anyway... I don't understand what you mean by "the provider implementation MUST use only public methods of my module". You have an arrangement of modules and packages in mind that I simply cannot see -- please share it. Alex From alex.buckley at oracle.com Tue Jan 23 01:35:40 2024 From: alex.buckley at oracle.com (Alex Buckley) Date: Mon, 22 Jan 2024 17:35:40 -0800 Subject: Java extensibility and JPMS (ServiceLoader) In-Reply-To: References: Message-ID: <9d838c78-fe31-48ce-a9a7-30051c354439@oracle.com> Tomas, I realize that you want to allow your modular JARs to be deployed on the classpath and still enjoy some degree of encapsulation. However, mapping packages 1:1 to modules isn't the right way to use the module system. You get to formalize the system structure, and will be forced to break any legacy cycles between packages, but every module will export 100% of its packages, so there's no real information hiding. The maintainer of a module in one subsystem can freely require a module in another subsystem and access the exported-but-not-really-supported package of that other subsystem, so over time, a ball of mud will re-emerge. Having "internal" packages is a _feature_ of modules, not a problem. It's only a problem if those modules are deployed on the classpath, and we don't recommend doing that. On point 3: The requirement that a provider class must be public with a public ctor comes from the original design of ServiceLoader. This resulted in providers that were "too public", so we used the module system to solve that problem: you can take existing providers, "hide" them in unexported/internal packages of modules, and service consumers are not impacted at all. Except for the small addition of "provider methods" as an alternative to public ctors, we took ServiceLoader in Java 9 as we found it. Evolving ServiceLoader in Java 23+ to allow providers with very limited accessibility -- package-access classes / ctors / provider methods -- is not incompatible with anything in the module system. In fact, keeping the circle of code that can instantiate providers as tight as possible is a good thing. However, we haven't heard the request for package-access providers from many people, and working on that feature would mean not working on something else, so it's unlikely that the feature will appear in the foreseeable future. Alex On 1/22/2024 3:30 PM, Tomas Langer wrote: > We have a single module/single package approach, so all internal classes > are hidden by visibility (package local) and not by package (as that > cannot be enforced on classpath. > Using more than one package usually results in internal packages and in > requirement to have some methods public, so classes in other packages of > your module can access them (where we can live with package local only...) > > Tomas > > Zasl?no z Outlooku pro Android > ------------------------------------------------------------------------ > *From:* jigsaw-dev on behalf of Alex > Buckley > *Sent:* Monday, January 22, 2024 9:13:39 PM > *To:* jigsaw-dev at openjdk.org > *Subject:* Re: Java extensibility and JPMS (ServiceLoader) > Tomas, > > A clarification for point 3: > > On 1/19/2024 5:02 AM, Tomas Langer wrote: >> *Ad 3 - Provider implementation must be public with public constructor* >> ------------------------------------- >> Details: >> Provider implementations are not supposed to be visible to users - they >> are not public API of my module (the fact that I provide a service is >> part of my public API). >> Right now there is only one option to work around this, and it only >> works in JPMS, and in my opinion it brings in even more problems - put >> the provider implementation in an un-exported package. >> The problem with this approach is that now the provider implementation >> MUST use only public methods of my module, thus creating even more >> public APIs, where if I just put it in my exported package, I can use >> package private methods of my other classes to implement the service (so >> I pay the price of having one public class with one public constructor >> agains multiple public classes and public methods). Also the "hiding" in >> unexported package is lost when on classpath anyway... > > I don't understand what you mean by "the provider implementation MUST > use only public methods of my module". You have an arrangement of > modules and packages in mind that I simply cannot see -- please share it. > > Alex From tomas.langer at oracle.com Thu Jan 25 12:34:42 2024 From: tomas.langer at oracle.com (Tomas Langer) Date: Thu, 25 Jan 2024 12:34:42 +0000 Subject: Starting a module with main class from a different module Message-ID: Is there a way to define a main class on a module if it comes from a different one? Currently we need to start the JVM as follows: java ?module-path ... --add-modules my.modular.application --module io.helidon.microprofile.cdi/io.helidon.microprofile.cdi.Main What we do: * our microprofile implementation bootstraps a framework, that discovers its component using service loader, it also contains the Main class needed to start * the "my.modular.application" provides such components (such as an HTTP endpoint), but does not need a main class Ideally, I would like to run "my.modular.application" to have a simpler command line. I cannot do that, as the main class is from a different module (and I think that cannot be added to the module metadata). So I have to run the main class from the CDI module, and use `--add-modules`, as otherwise my application module would not end up on the module path (as it is not the "main" module started). Is there some way please to simplify this? Or do we need to create a Main class for each application, even if it would just delegate to the "cdi.Main"? When running on classpath, there is no problem, as the `Main-Class` attribute in manifest can point to a class in another jar, and the `Class-Path` attribute in the manifest can contain the jar that has the main class, so on classpath we can do: java -jar my.modular.application.jar Thanks Tomas Langer project Helidon -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Jan 25 15:11:51 2024 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 25 Jan 2024 15:11:51 +0000 Subject: Starting a module with main class from a different module In-Reply-To: References: Message-ID: On 25/01/2024 12:34, Tomas Langer wrote: > Is there a way to define a main class on a module if it comes from a > different one? > > Currently we need to start the JVM as follows: > *java ?module-path ... --add-modules my.modular.application --module > io.helidon.microprofile.cdi/io.helidon.microprofile.cdi.Main* > > What we do: > > * our microprofile implementation bootstraps a framework, that > discovers its component using service loader, it also contains the > Main class needed to start > * the "my.modular.application" provides such components (such as an > HTTP endpoint), but does not need a main class > > > Ideally, I would like to run "my.modular.application" to have a > simpler command line. > I cannot do that, as the main class is from a different module (and I > think that cannot be added to the module metadata). > So I have to run the main class from the CDI module, and use > `--add-modules`, as otherwise my application module would not end up > on the module path (as it is not the "main" module started). > > > Is there some way please to simplify this? Or do we need to create a > Main class for each application, even if it would just delegate to the > "cdi.Main"? > Is the application module discovered with service loader too? In that case, the --add-modules shouldn't be needed, instead just make it observable on the module path and service binding will ensure that it resolved. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From rfscholte at apache.org Fri Jan 26 10:46:51 2024 From: rfscholte at apache.org (Robert Scholte) Date: Fri, 26 Jan 2024 11:46:51 +0100 Subject: Custom Classloaders Message-ID: <034201da5044$f893b980$e9bb2c80$@apache.org> Hi, I'm working on a new application, using jlink to build it. One challenge is related to the simplelogger of SLF4J (as SLF4J-api is probably the most used logging api, the simplelogger implementation matches my requirements) There are 2 things that I would like to adjust: 1. Simplelogger uses the classloader to find the simplelogger.properties ( https://github.com/qos-ch/slf4j/blob/master/slf4j-simple/src/main/java/org/s lf4j/simple/SimpleLoggerConfiguration.java#L107-L130 ). However, I want to externalize this file to ${java.home}/conf/simplelogger.properties, so users can adjust it to their preferences. If I could change the classloader I could control where to find this file. When bundling simplelogger with jlink the jar will end up in the AppClassloader due to the SimpleServiceProvider. The only option to avoid this, is to move this jar to a separate directory and when making a new ModuleLayer, and use ModuleFinder with this path. But still, I see no way to change the classloader for this jar. 2. The simplelogger.properties is the default configuration, and I want to make it possible to override these values. The simplelogger offers the option to use SystemProperties to override values from simplelogger.properties. I want to avoid that the System.properties will be polluted with entities just for logging. As far as I know it is not possible to scope System.properties: they are global, available for every part of the application. Another idea I had was to extend the simplelogging.properties resources at runtime using the SequenceInputStream, and for this I also need to have control over the classloader, as it requires to override the ClassLoader.getResourceAsStream() It seems both challenges could be solved if could control the classloader (while still being protected by jdk.internal.loader.Loader). Am I missing some feature or it there another way to solve these issues? Best, Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Fri Jan 26 11:12:13 2024 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 26 Jan 2024 11:12:13 +0000 Subject: Custom Classloaders In-Reply-To: <034201da5044$f893b980$e9bb2c80$@apache.org> References: <034201da5044$f893b980$e9bb2c80$@apache.org> Message-ID: On 26/01/2024 10:46, Robert Scholte wrote: > > Hi, > > I?m working on a new application, using jlink to build it. > > One challenge is related to the simplelogger of SLF4J (as SLF4J-api is > probably the most used logging api, the simplelogger implementation > matches my requirements) > > There are 2 things that I would like to adjust: > > 1. Simplelogger uses the classloader to find the > simplelogger.properties ( > https://github.com/qos-ch/slf4j/blob/master/slf4j-simple/src/main/java/org/slf4j/simple/SimpleLoggerConfiguration.java#L107-L130 > ). > > However, I want to externalize this file to > ${java.home}/conf/simplelogger.properties, so users can adjust it to > their preferences. If I could change the classloader I could control > where to find this file. > > When bundling simplelogger with jlink the jar will end up in the > AppClassloader due to the SimpleServiceProvider. > > The only option to avoid this, is to move this jar to a separate > directory and when making a new ModuleLayer, and use ModuleFinder with > this path. > > But still, I see no way to change the classloader for this jar. > > 2. The simplelogger.properties is the default configuration, and I > want to make it possible to override these values. > > The simplelogger offers the option to use SystemProperties to override > values from simplelogger.properties. > > I want to avoid that the System.properties will be polluted with > entities just for logging. > > As far as I know it is not possible to scope System.properties: they > are global, available for every part of the application. > > Another idea I had was to extend the simplelogging.properties > resources at runtime using the SequenceInputStream, and for this I > also need to have control over the classloader, as it requires to > override the ClassLoader.getResourceAsStream() > > It seems both challenges could be solved if could control the > classloader (while still being protected by jdk.internal.loader.Loader). > > Am I missing some feature or it there another way to solve these issues? > > Hi Robert, The code you linked to using the current Thread's context ClassLoader (CCL) to find the resource. So not the same thing as the class loader that defined SimpleLoggerConfiguration. At a guess, that code look like it is intended to support bundling of logging configuration with an application. It means code can set the current Thread's CCL, initialize SL4J to so it find the resource (config file) with that class loader, and reset it after. To be editable in ${java.home}/conf would require changing?SimpleLoggerConfiguration.loadProperties to look for the properties file there. The defining classer for SimpleLoggerConfiguration won't change that as Class.getResource or ClassLoader.getResource doesn't find resources there. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From forax at univ-mlv.fr Fri Jan 26 11:27:29 2024 From: forax at univ-mlv.fr (Remi Forax) Date: Fri, 26 Jan 2024 12:27:29 +0100 (CET) Subject: Custom Classloaders In-Reply-To: <034201da5044$f893b980$e9bb2c80$@apache.org> References: <034201da5044$f893b980$e9bb2c80$@apache.org> Message-ID: <39023581.113292571.1706268449187.JavaMail.zimbra@univ-eiffel.fr> > From: "Robert Scholte" > To: "jigsaw-dev" > Sent: Friday, January 26, 2024 11:46:51 AM > Subject: Custom Classloaders > Hi, Hello Robert, you were miss at last Devoxx.be, > I?m working on a new application, using jlink to build it. > One challenge is related to the simplelogger of SLF4J (as SLF4J-api is probably > the most used logging api, the simplelogger implementation matches my > requirements) > There are 2 things that I would like to adjust: > 1. Simplelogger uses the classloader to find the simplelogger.properties ( [ > https://github.com/qos-ch/slf4j/blob/master/slf4j-simple/src/main/java/org/slf4j/simple/SimpleLoggerConfiguration.java#L107-L130 > | > https://github.com/qos-ch/slf4j/blob/master/slf4j-simple/src/main/java/org/slf4j/simple/SimpleLoggerConfiguration.java#L107-L130 > ] ). > However, I want to externalize this file to > ${java.home}/conf/simplelogger.properties, so users can adjust it to their > preferences. If I could change the classloader I could control where to find > this file. > When bundling simplelogger with jlink the jar will end up in the AppClassloader > due to the SimpleServiceProvider. > The only option to avoid this, is to move this jar to a separate directory and > when making a new ModuleLayer, and use ModuleFinder with this path. > But still, I see no way to change the classloader for this jar. > 1. The simplelogger.properties is the default configuration, and I want to make > it possible to override these values. > The simplelogger offers the option to use SystemProperties to override values > from simplelogger.properties. > I want to avoid that the System.properties will be polluted with entities just > for logging. > As far as I know it is not possible to scope System.properties: they are global, > available for every part of the application. > Another idea I had was to extend the simplelogging.properties resources at > runtime using the SequenceInputStream, and for this I also need to have control > over the classloader, as it requires to override the > ClassLoader.getResourceAsStream() > It seems both challenges could be solved if could control the classloader (while > still being protected by jdk.internal.loader.Loader). > Am I missing some feature or it there another way to solve these issues? yes, as you said yourself, you box yourself in a corner by wanting to use Simplelogger which does not provide a great deal of ways to configure it. I'm sure they are other loggers that, while heavier, are more capable in term of configuration. You are cursed by your own knowledge in this case, as an application developer, you should pretend that ClassLoaders do not exist, like the rest of the application developers do. Let me try a trick, those are not the classloaders you are looking for :) > Best, > Robert regards, R?mi -------------- next part -------------- An HTML attachment was scrubbed... URL: From rfscholte at apache.org Sat Jan 27 11:13:58 2024 From: rfscholte at apache.org (Robert Scholte) Date: Sat, 27 Jan 2024 12:13:58 +0100 Subject: Custom Classloaders In-Reply-To: <39023581.113292571.1706268449187.JavaMail.zimbra@univ-eiffel.fr> References: <034201da5044$f893b980$e9bb2c80$@apache.org> <39023581.113292571.1706268449187.JavaMail.zimbra@univ-eiffel.fr> Message-ID: <004c01da5111$ec9e1d50$c5da57f0$@apache.org> Hi, With both the answers from Alan and R?mi it is clear to me that the implementation of loading configuration in the simplelogger doesn?t fit my requirements. It is called ?simple? for a reason. I prefer to depend on existing SLF4JProviders, but based on possible extra requirements it might be worth writing my own provider. Thanks, Robert _____ From: "Robert Scholte" > To: "jigsaw-dev" > Sent: Friday, January 26, 2024 11:46:51 AM Subject: Custom Classloaders Hi, Hello Robert, you were miss at last Devoxx.be, I?m working on a new application, using jlink to build it. One challenge is related to the simplelogger of SLF4J (as SLF4J-api is probably the most used logging api, the simplelogger implementation matches my requirements) There are 2 things that I would like to adjust: 1. Simplelogger uses the classloader to find the simplelogger.properties ( https://github.com/qos-ch/slf4j/blob/master/slf4j-simple/src/main/java/org/slf4j/simple/SimpleLoggerConfiguration.java#L107-L130 ). However, I want to externalize this file to ${java.home}/conf/simplelogger.properties, so users can adjust it to their preferences. If I could change the classloader I could control where to find this file. When bundling simplelogger with jlink the jar will end up in the AppClassloader due to the SimpleServiceProvider. The only option to avoid this, is to move this jar to a separate directory and when making a new ModuleLayer, and use ModuleFinder with this path. But still, I see no way to change the classloader for this jar. 2. The simplelogger.properties is the default configuration, and I want to make it possible to override these values. The simplelogger offers the option to use SystemProperties to override values from simplelogger.properties. I want to avoid that the System.properties will be polluted with entities just for logging. As far as I know it is not possible to scope System.properties: they are global, available for every part of the application. Another idea I had was to extend the simplelogging.properties resources at runtime using the SequenceInputStream, and for this I also need to have control over the classloader, as it requires to override the ClassLoader.getResourceAsStream() It seems both challenges could be solved if could control the classloader (while still being protected by jdk.internal.loader.Loader). Am I missing some feature or it there another way to solve these issues? yes, as you said yourself, you box yourself in a corner by wanting to use Simplelogger which does not provide a great deal of ways to configure it. I'm sure they are other loggers that, while heavier, are more capable in term of configuration. You are cursed by your own knowledge in this case, as an application developer, you should pretend that ClassLoaders do not exist, like the rest of the application developers do. Let me try a trick, those are not the classloaders you are looking for :) Best, Robert regards, R?mi -------------- next part -------------- An HTML attachment was scrubbed... URL: From simone.bordet at gmail.com Sat Jan 27 13:46:16 2024 From: simone.bordet at gmail.com (Simone Bordet) Date: Sat, 27 Jan 2024 14:46:16 +0100 Subject: Custom Classloaders In-Reply-To: <004c01da5111$ec9e1d50$c5da57f0$@apache.org> References: <034201da5044$f893b980$e9bb2c80$@apache.org> <39023581.113292571.1706268449187.JavaMail.zimbra@univ-eiffel.fr> <004c01da5111$ec9e1d50$c5da57f0$@apache.org> Message-ID: Robert, On Sat, Jan 27, 2024 at 12:14?PM Robert Scholte wrote: > > Hi, > > > > With both the answers from Alan and R?mi it is clear to me that the implementation of loading configuration in the simplelogger doesn?t fit my requirements. > > It is called ?simple? for a reason. > > I prefer to depend on existing SLF4JProviders, but based on possible extra requirements it might be worth writing my own provider. Just to say that Jetty has written one, and it's a proper JPMS module: https://central.sonatype.com/artifact/org.eclipse.jetty/jetty-slf4j-impl Not sure if it meets 100% your requirements, but we do support system property log category configuration, etc. Let us know if we can improve it :) -- Simone Bordet --- Finally, no matter how good the architecture and design are, to deliver bug-free software with optimal performance and reliability, the implementation technique must be flawless. Victoria Livschitz From alex.buckley at oracle.com Tue Jan 30 19:05:01 2024 From: alex.buckley at oracle.com (Alex Buckley) Date: Tue, 30 Jan 2024 11:05:01 -0800 Subject: Java extensibility and JPMS (ServiceLoader) In-Reply-To: References: Message-ID: <432557a1-299d-4c97-ad66-b3bfa4560cdd@oracle.com> On 1/19/2024 5:02 AM, Tomas Langer wrote: > Helidon currently has around 300 modules with module-info.java. In > general, this has improved our module structure and design. > Yet, we are now encountering some major issues related to extensibility. > ?I will put down a few points that are problematic, and explain each in > detail further in the e-mail (it is quite long, sorry about that). > > 1. provider implementations cannot be code generated without major problems Annotation processing is designed to avoid mutating the elements, so it would be a fundamental change to allow mutation of module elements in the annotation processing API. It's on the framework to post-process module-info.class so it has `provides` clauses. The framework can use the ClassFile API to do this. It's not "weird" for a framework to modify module-info.class to ensure that code in the module has the right execution environment. Another example would be a tool that injects calls to a logging API into user code, then has to post-process module-info.class to add `requires logging.lib;`. > 2. the provider interface module MUST be on module path, even if it > could have `requires static` Relaxing module resolution to allow a module to `provides` an interface that it can't access is do-able, but the implications are unknown. It would be unfortunate if resolution succeeded but then unforeseen exceptions occur when faraway code tries to access the missing interface. We'll leave JDK-8299504 open for now, but we would need evidence that a large number of users are finding the current rules unworkable before actively looking at `provides` again. > 3. the provider implementation must be public with public constructor > 4. duality of definition between module path and class path These requests for ServiceLoader to (i) support package-private providers and (ii) inspect module-info.class files for `provides` clauses in JARs on the classpath both stem from insisting that modular JARs can be deployed on the classpath without losing functionality. This is a net-new requirement on the module system. It's relatively low risk, but also low reward, so we aren't going to investigate it further. Alex From tomas.langer at oracle.com Tue Jan 30 19:39:23 2024 From: tomas.langer at oracle.com (Tomas Langer) Date: Tue, 30 Jan 2024 19:39:23 +0000 Subject: Java extensibility and JPMS (ServiceLoader) In-Reply-To: <432557a1-299d-4c97-ad66-b3bfa4560cdd@oracle.com> References: <432557a1-299d-4c97-ad66-b3bfa4560cdd@oracle.com> Message-ID: Ad 1 (provider implementations cannot be code generated without major problems) I understand that annotation processing should not mutate elements, and I never suggested it should be done. I am looking for a solution provided by the language (whatever that may be). I am not sure if you are aware of the complexity of the proposed "It's on the framework to post-process module-info.class so it has `provides` clauses" Considering that we currently have multiple build frameworks, and for each of those the postprocessing must be done differently, we are also opening the possibility of the user not configuring the postprocessing to happen at all. Just the first part (Maven plugins, Gradle plugins, possible other systems) is a major maintenance issue. In addition, the compiled code deployed with the application differs from its source code, making troubleshooting more complicated. I understand it is easy to dismiss these problems and let us (as framework owners) handle it. You are in the end forcing us to do it differently for each framework - which in the end is a problem for our users. The result that other frameworks reached (and I was trying not to do), and we will switch to as well, is to design our own extensibility approach that is based on resources rather than on module-info.java. This makes it once again harder for users, as they cannot use the tools the language provides. Pushing us to post-processing the bytecode: if there was an event in the compiler with extensibility similar to annotation processors (i.e. ClassPostProcessor) using the ClassFile API, I can imagine this being done, as it is still part of a single compilation run (but as far as I know, there is no such possibility). Ad 2 (requires static on provider module) As far as I have seen the analysis of current Maven repository, Helidon is the only bigger framework that has embraced modularization. So "a large number of users" will not find the current rules unworkable, as there is not "a large number of users" using JPMS. As one of the users of it on a larger scale, I can tell you this is a major restriction for using JPMS together with ServiceLoader (and as mentioned above, we will stop using ServiceLoader because of this). Ad 3 (public provider) I understand this is not a critical feature Ad 4 (duality between module path and classpath) You have mentioned "It's relatively low risk, but also low reward, so we aren't going to investigate it further." Once again I am not sure whose "low reward" you have in mind. I understand that in the language and for language developers this may not mean much. But for the users, frameworks etc. this is quite a big thing - we are now required to do another post-processing of our libraries to ensure we have both (module-info provides and META-INF/services). The reward is that new libraries can work both on classpath and on module path. And considering how many people use JPMS, this is quite a critical behavior for anything we create. Tomas ________________________________ Od: jigsaw-dev za u?ivatele Alex Buckley Odesl?no: ?ter? 30. ledna 2024 20:05 Komu: jigsaw-dev P?edm?t: Re: Java extensibility and JPMS (ServiceLoader) On 1/19/2024 5:02 AM, Tomas Langer wrote: > Helidon currently has around 300 modules with module-info.java. In > general, this has improved our module structure and design. > Yet, we are now encountering some major issues related to extensibility. > I will put down a few points that are problematic, and explain each in > detail further in the e-mail (it is quite long, sorry about that). > > 1. provider implementations cannot be code generated without major problems Annotation processing is designed to avoid mutating the elements, so it would be a fundamental change to allow mutation of module elements in the annotation processing API. It's on the framework to post-process module-info.class so it has `provides` clauses. The framework can use the ClassFile API to do this. It's not "weird" for a framework to modify module-info.class to ensure that code in the module has the right execution environment. Another example would be a tool that injects calls to a logging API into user code, then has to post-process module-info.class to add `requires logging.lib;`. > 2. the provider interface module MUST be on module path, even if it > could have `requires static` Relaxing module resolution to allow a module to `provides` an interface that it can't access is do-able, but the implications are unknown. It would be unfortunate if resolution succeeded but then unforeseen exceptions occur when faraway code tries to access the missing interface. We'll leave JDK-8299504 open for now, but we would need evidence that a large number of users are finding the current rules unworkable before actively looking at `provides` again. > 3. the provider implementation must be public with public constructor > 4. duality of definition between module path and class path These requests for ServiceLoader to (i) support package-private providers and (ii) inspect module-info.class files for `provides` clauses in JARs on the classpath both stem from insisting that modular JARs can be deployed on the classpath without losing functionality. This is a net-new requirement on the module system. It's relatively low risk, but also low reward, so we aren't going to investigate it further. Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From josiahnoel at gmail.com Wed Jan 31 01:25:48 2024 From: josiahnoel at gmail.com (Josiah Noel) Date: Tue, 30 Jan 2024 20:25:48 -0500 Subject: Java extensibility and JPMS (ServiceLoader) In-Reply-To: References: <432557a1-299d-4c97-ad66-b3bfa4560cdd@oracle.com> Message-ID: > > Annotation processing is designed to avoid mutating the elements, so it > would be a fundamental change to allow mutation of module elements We're well aware that processors shouldn't modify code, we're talking about adding some extension to the service loader that we *can *generate. We simply want to regain the ability to register services at compile-time. It's on the framework to post-process > module-info.class so it has `provides` clauses. The framework can use > the ClassFile API to do this. > As Tomas has stated, there are a myriad of build systems so postprocessors are quite a maintenance issue. The result that other frameworks reached (and I was trying not to do), is > to design our own extensibility approach that is based on resources I wonder what are these other frameworks that have gone this route? On Tue, Jan 30, 2024 at 2:39?PM Tomas Langer wrote: > Ad 1 (provider implementations cannot be code generated without major > problems) > I understand that annotation processing should not mutate elements, and I > never suggested it should be done. I am looking for a solution provided by > the language (whatever that may be). > > I am not sure if you are aware of the complexity of the proposed "It's on > the framework to post-process > module-info.class so it has `provides` clauses" > > Considering that we currently have multiple build frameworks, and for each > of those the postprocessing must be done differently, we are also opening > the possibility of the user not configuring the postprocessing to happen at > all. Just the first part (Maven plugins, Gradle plugins, possible other > systems) is a major maintenance issue. > In addition, the compiled code deployed with the application differs from > its source code, making troubleshooting more complicated. > > I understand it is easy to dismiss these problems and let us (as framework > owners) handle it. You are in the end forcing us to do it differently for > each framework - which in the end is a problem for our users. > > The result that other frameworks reached (and I was trying not to do), and > we will switch to as well, is to design our own extensibility approach that > is based on resources rather than on module-info.java. This makes it once > again harder for users, as they cannot use the tools the language provides. > > Pushing us to post-processing the bytecode: if there was an event in the > compiler with extensibility similar to annotation processors (i.e. > ClassPostProcessor) using the ClassFile API, I can imagine this being done, > as it is still part of a single compilation run (but as far as I know, > there is no such possibility). > > Ad 2 (requires static on provider module) > As far as I have seen the analysis of current Maven repository, Helidon is > the only bigger framework that has embraced modularization. > So "a large number of users" will not find the current rules unworkable, > as there is not "a large number of users" using JPMS. > As one of the users of it on a larger scale, I can tell you this is a > major restriction for using JPMS together with ServiceLoader (and as > mentioned above, we will stop using ServiceLoader because of this). > > Ad 3 (public provider) > I understand this is not a critical feature > > Ad 4 (duality between module path and classpath) > You have mentioned "It's relatively low risk, but also low reward, so we > aren't going to investigate it further." > Once again I am not sure whose "low reward" you have in mind. I understand > that in the language and for language developers this may not mean much. > But for the users, frameworks etc. this is quite a big thing - we are now > required to do another post-processing of our libraries to ensure we have > both (module-info provides and META-INF/services). > The reward is that new libraries can work both on classpath and on module > path. And considering how many people use JPMS, this is quite a critical > behavior for anything we create. > > Tomas > > ------------------------------ > *Od:* jigsaw-dev za u?ivatele Alex Buckley < > alex.buckley at oracle.com> > *Odesl?no:* ?ter? 30. ledna 2024 20:05 > *Komu:* jigsaw-dev > *P?edm?t:* Re: Java extensibility and JPMS (ServiceLoader) > > On 1/19/2024 5:02 AM, Tomas Langer wrote: > > Helidon currently has around 300 modules with module-info.java. In > > general, this has improved our module structure and design. > > Yet, we are now encountering some major issues related to extensibility. > > I will put down a few points that are problematic, and explain each in > > detail further in the e-mail (it is quite long, sorry about that). > > > > 1. provider implementations cannot be code generated without major > problems > > Annotation processing is designed to avoid mutating the elements, so it > would be a fundamental change to allow mutation of module elements in > the annotation processing API. It's on the framework to post-process > module-info.class so it has `provides` clauses. The framework can use > the ClassFile API to do this. > > It's not "weird" for a framework to modify module-info.class to ensure > that code in the module has the right execution environment. Another > example would be a tool that injects calls to a logging API into user > code, then has to post-process module-info.class to add `requires > logging.lib;`. > > > 2. the provider interface module MUST be on module path, even if it > > could have `requires static` > > Relaxing module resolution to allow a module to `provides` an interface > that it can't access is do-able, but the implications are unknown. It > would be unfortunate if resolution succeeded but then unforeseen > exceptions occur when faraway code tries to access the missing > interface. We'll leave JDK-8299504 open for now, but we would need > evidence that a large number of users are finding the current rules > unworkable before actively looking at `provides` again. > > > 3. the provider implementation must be public with public constructor > > 4. duality of definition between module path and class path > > These requests for ServiceLoader to (i) support package-private > providers and (ii) inspect module-info.class files for `provides` > clauses in JARs on the classpath both stem from insisting that modular > JARs can be deployed on the classpath without losing functionality. This > is a net-new requirement on the module system. It's relatively low risk, > but also low reward, so we aren't going to investigate it further. > > Alex > -------------- next part -------------- An HTML attachment was scrubbed... URL: From piotr.karwasz at gmail.com Wed Jan 31 07:47:30 2024 From: piotr.karwasz at gmail.com (Piotr P. Karwasz) Date: Wed, 31 Jan 2024 08:47:30 +0100 Subject: Are modifiers `static` and `transitive` incompatible? Message-ID: Hello, The `requires` instruction can have two qualifiers `static` and `transitive` that separately make perfect sense. For a module `foo`: * `requires static bar;`, makes resolution of the `bar` module optional at runtime, * `requires transitive bar;`, allows each module that reads `foo` to also read `bar`. As far as I understand this means that `bar` must be present at both compile time and runtime. There is however a third option allowed: ```java module foo { requires static transitive bar; } ``` What is this combination supposed to do? Why is it allowed? According to the Javadoc: > `requires` directives that have the `static` modifier express an optional dependence at run time. If a module declares that it `requires static M` then resolution does not search the observable modules for M to satisfy the dependency. However, if M is recursively enumerated at step 1 then all modules that are enumerated and `requires static M` will read M. where "step 1" describes the computation of the closure of the set of root modules by the `X requires transitive Y` relation. If I interpret this correctly, if a `requires` directive is both `static` and `transitive`, the `foo` module is **not optional**, since "step 1" requires its presence. We often end up with such directives, when using the `bnd-maven-plugin`, which maps the OSGi `resolution:=optional` directive to `static` and the `uses:=...` directive to `transitive`. If we mark a package as optional, but the package uses public types from `bar`, we end up with `static transitive`. Is this a bug of the plugin or does `static transitive` have a meaning? Piotr [1] https://github.com/bndtools/bnd/tree/master/maven-plugins/bnd-maven-plugin From tomas.langer at oracle.com Wed Jan 31 12:02:46 2024 From: tomas.langer at oracle.com (Tomas Langer) Date: Wed, 31 Jan 2024 12:02:46 +0000 Subject: [External] : Re: Java extensibility and JPMS (ServiceLoader) In-Reply-To: References: <432557a1-299d-4c97-ad66-b3bfa4560cdd@oracle.com> Message-ID: Other frameworks I am aware of that have custom service loading: * hk2 * WebLogic * Jersey * Micronaut Tomas ________________________________ Od: Josiah Noel Odesl?no: st?eda 31. ledna 2024 2:25 Komu: Tomas Langer Kopie: Alex Buckley ; jigsaw-dev P?edm?t: [External] : Re: Java extensibility and JPMS (ServiceLoader) Annotation processing is designed to avoid mutating the elements, so it would be a fundamental change to allow mutation of module elements We're well aware that processors shouldn't modify code, we're talking about adding some extension to the service loader that we can generate. We simply want to regain the ability to register services at compile-time. It's on the framework to post-process module-info.class so it has `provides` clauses. The framework can use the ClassFile API to do this. As Tomas has stated, there are a myriad of build systems so postprocessors are quite a maintenance issue. The result that other frameworks reached (and I was trying not to do), is to design our own extensibility approach that is based on resources I wonder what are these other frameworks that have gone this route? On Tue, Jan 30, 2024 at 2:39?PM Tomas Langer > wrote: Ad 1 (provider implementations cannot be code generated without major problems) I understand that annotation processing should not mutate elements, and I never suggested it should be done. I am looking for a solution provided by the language (whatever that may be). I am not sure if you are aware of the complexity of the proposed "It's on the framework to post-process module-info.class so it has `provides` clauses" Considering that we currently have multiple build frameworks, and for each of those the postprocessing must be done differently, we are also opening the possibility of the user not configuring the postprocessing to happen at all. Just the first part (Maven plugins, Gradle plugins, possible other systems) is a major maintenance issue. In addition, the compiled code deployed with the application differs from its source code, making troubleshooting more complicated. I understand it is easy to dismiss these problems and let us (as framework owners) handle it. You are in the end forcing us to do it differently for each framework - which in the end is a problem for our users. The result that other frameworks reached (and I was trying not to do), and we will switch to as well, is to design our own extensibility approach that is based on resources rather than on module-info.java. This makes it once again harder for users, as they cannot use the tools the language provides. Pushing us to post-processing the bytecode: if there was an event in the compiler with extensibility similar to annotation processors (i.e. ClassPostProcessor) using the ClassFile API, I can imagine this being done, as it is still part of a single compilation run (but as far as I know, there is no such possibility). Ad 2 (requires static on provider module) As far as I have seen the analysis of current Maven repository, Helidon is the only bigger framework that has embraced modularization. So "a large number of users" will not find the current rules unworkable, as there is not "a large number of users" using JPMS. As one of the users of it on a larger scale, I can tell you this is a major restriction for using JPMS together with ServiceLoader (and as mentioned above, we will stop using ServiceLoader because of this). Ad 3 (public provider) I understand this is not a critical feature Ad 4 (duality between module path and classpath) You have mentioned "It's relatively low risk, but also low reward, so we aren't going to investigate it further." Once again I am not sure whose "low reward" you have in mind. I understand that in the language and for language developers this may not mean much. But for the users, frameworks etc. this is quite a big thing - we are now required to do another post-processing of our libraries to ensure we have both (module-info provides and META-INF/services). The reward is that new libraries can work both on classpath and on module path. And considering how many people use JPMS, this is quite a critical behavior for anything we create. Tomas ________________________________ Od: jigsaw-dev > za u?ivatele Alex Buckley > Odesl?no: ?ter? 30. ledna 2024 20:05 Komu: jigsaw-dev > P?edm?t: Re: Java extensibility and JPMS (ServiceLoader) On 1/19/2024 5:02 AM, Tomas Langer wrote: > Helidon currently has around 300 modules with module-info.java. In > general, this has improved our module structure and design. > Yet, we are now encountering some major issues related to extensibility. > I will put down a few points that are problematic, and explain each in > detail further in the e-mail (it is quite long, sorry about that). > > 1. provider implementations cannot be code generated without major problems Annotation processing is designed to avoid mutating the elements, so it would be a fundamental change to allow mutation of module elements in the annotation processing API. It's on the framework to post-process module-info.class so it has `provides` clauses. The framework can use the ClassFile API to do this. It's not "weird" for a framework to modify module-info.class to ensure that code in the module has the right execution environment. Another example would be a tool that injects calls to a logging API into user code, then has to post-process module-info.class to add `requires logging.lib;`. > 2. the provider interface module MUST be on module path, even if it > could have `requires static` Relaxing module resolution to allow a module to `provides` an interface that it can't access is do-able, but the implications are unknown. It would be unfortunate if resolution succeeded but then unforeseen exceptions occur when faraway code tries to access the missing interface. We'll leave JDK-8299504 open for now, but we would need evidence that a large number of users are finding the current rules unworkable before actively looking at `provides` again. > 3. the provider implementation must be public with public constructor > 4. duality of definition between module path and class path These requests for ServiceLoader to (i) support package-private providers and (ii) inspect module-info.class files for `provides` clauses in JARs on the classpath both stem from insisting that modular JARs can be deployed on the classpath without losing functionality. This is a net-new requirement on the module system. It's relatively low risk, but also low reward, so we aren't going to investigate it further. Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Wed Jan 31 15:41:29 2024 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 31 Jan 2024 15:41:29 +0000 Subject: Are modifiers `static` and `transitive` incompatible? In-Reply-To: References: Message-ID: <19ce7383-1bcc-438e-b7f2-11c45f4882eb@oracle.com> On 31/01/2024 07:47, Piotr P. Karwasz wrote: > Hello, > > The `requires` instruction can have two qualifiers `static` and > `transitive` that separately make perfect sense. For a module `foo`: > > * `requires static bar;`, makes resolution of the `bar` module > optional at runtime, > * `requires transitive bar;`, allows each module that reads `foo` to > also read `bar`. As far as I understand this means that `bar` must be > present at both compile time and runtime. > > There is however a third option allowed: > > ```java > module foo { > requires static transitive bar; > } > ``` It may be surprising but they do work together. If bar is resolved then foo will read bar. If there is another module baz that requires foo (and doesn't explicitly require bar) then baz will read foo, and if bar is resolved, then baz will read bar. > : > We often end up with such directives, when using the > `bnd-maven-plugin`, which maps the OSGi `resolution:=optional` > directive to `static` and the `uses:=...` directive to `transitive`. > If we mark a package as optional, but the package uses public types > from `bar`, we end up with `static transitive`. > > Is this a bug of the plugin or does `static transitive` have a meaning? > I think it's more of a case that it is fragile for modules like foo to have bar types foo API signatures.? `requires static` is okay for things like annotations with a retention policy of "source" but going beyond that means giving up a bit on reliable configuration. In the example, foo probably has a reflection guard and is careful in their use of bar types. However, foo's user if baz and the maintainers of baz are unaware of these shenanigans. This creates a risk that baz code will get a CNFE at runtime. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: