From pavelturk2000 at gmail.com  Mon Jan  1 15:51:24 2024
From: pavelturk2000 at gmail.com (PavelTurk)
Date: Mon, 1 Jan 2024 17:51:24 +0200
Subject: How to get reads from module?
Message-ID: <54fb9136-c7f4-1d65-8ac7-3287c13633e9@gmail.com>

Hello all. I've asked this question, however I got no answer. So I decided to ask the question again
providing more details.

Consider the?following?layer?graph:

1)?boot
2)?layerA?(parent?-?boot)?has moduleX
3)?layerB?(parent?-?boot)?has?moduleX
4)?layerC?(parents?-?layerA,?layerB)?has?moduleY

As?I?understand?it?is?a?possible?situation.?ModuleY?requires?moduleX?that?is?present?in?two?parent?layers.
Now, I have a reference to moduleY: var moduleY = ....; And I need to open foo package of moduleX to moduleY
dynamically:

ModuleLayer.Controller.addOpens?(moduleX, foo, moduleY)//I have foo and moduleY

The?question?-?how?can?I?find?out?from?which?layer?moduleX is used for moduleY so I could use `addOpens(..)`?

Best regards, Pavel

From michal at kleczek.org  Tue Jan  2 07:56:34 2024
From: michal at kleczek.org (=?utf-8?Q?Micha=C5=82_K=C5=82eczek?=)
Date: Tue, 2 Jan 2024 08:56:34 +0100
Subject: How to get reads from module?
In-Reply-To: <54fb9136-c7f4-1d65-8ac7-3287c13633e9@gmail.com>
References: <54fb9136-c7f4-1d65-8ac7-3287c13633e9@gmail.com>
Message-ID: <F8E2A7F5-3E1D-4118-B0AD-03FA79786BFD@kleczek.org>

How about moduleY.layer().findModule(?moduleX?)?


Micha? 

> On 1 Jan 2024, at 16:52, PavelTurk <pavelturk2000 at gmail.com> wrote:
> 
> ?Hello all. I've asked this question, however I got no answer. So I decided to ask the question again
> providing more details.
> 
> Consider the following layer graph:
> 
> 1) boot
> 2) layerA (parent - boot) has moduleX
> 3) layerB (parent - boot) has moduleX
> 4) layerC (parents - layerA, layerB) has moduleY
> 
> As I understand it is a possible situation. ModuleY requires moduleX that is present in two parent layers.
> Now, I have a reference to moduleY: var moduleY = ....; And I need to open foo package of moduleX to moduleY
> dynamically:
> 
> ModuleLayer.Controller.addOpens?(moduleX, foo, moduleY)//I have foo and moduleY
> 
> The question - how can I find out from which layer moduleX is used for moduleY so I could use `addOpens(..)`?
> 
> Best regards, Pavel

From Alan.Bateman at oracle.com  Tue Jan  2 08:03:19 2024
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Tue, 2 Jan 2024 08:03:19 +0000
Subject: A couple of questions
In-Reply-To: <7214b7d4-4002-1e0d-d7ee-f00eed90382f@gmail.com>
References: <7214b7d4-4002-1e0d-d7ee-f00eed90382f@gmail.com>
Message-ID: <331fd5e4-660f-485e-bf4b-5eecf4fa3487@oracle.com>

On 29/12/2023 11:47, PavelTurk wrote:
> Hello all. Could anyone help me with the following questions:
>
> 1. Let's suppose I have a reference to an unnamed module: 
> module.getName() == null . But I need at least some information about it.
> How can I get file name of this module - jar/war etc?

If you have a reference to an unnamed module then you can get its class 
loader with Module::getClassLoader. There's unnamed module for each 
class loader, 1:1 relationship. You can get the set of packages for the 
classes defined in the unnamed module with Module::getPackages which may 
help get closer to what you are looking for. That said, there is no API 
to expose the class path or the locations where the class loader has 
loaded classes. The class bytes can come from anywhere.

What is the context for this question? Agents/tooling can use APIs to 
enumerate all classes loaded in the VM and most of these classes will 
have a URL code source. I think that is the closest to what you are 
looking for (if I understand your question correctly).


>
> 2. Let's suppose I have the following layer graph:
>
> 1) boot
> 2) layerA (parent - boot) has moduleX
> 3) layerB (parent - boot) has moduleX
> 4) layerC (parents - layerA, layerB) has moduleY
>
> As I understand it is a possible situation. ModuleY requires moduleX 
> that is present in two parent layers.
> The question - how can I find out from which layer moduleX will be 
> used for moduleY?

The Configuration objects for layerA,?layerB and?layerC encapsulate the 
readability graphs used to the these module layers. Use 
Configuration::modules to get the set the resolved modules in the 
configuration, and ResovledRead::read to get the set of modules that are 
read. This will allow you to see which modules in C read "moduleX" in A 
and which modules in C read "moduleX" in B.

-Alan.



From Alan.Bateman at oracle.com  Tue Jan  2 11:40:34 2024
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Tue, 2 Jan 2024 11:40:34 +0000
Subject: How to get reads from module?
In-Reply-To: <F8E2A7F5-3E1D-4118-B0AD-03FA79786BFD@kleczek.org>
References: <54fb9136-c7f4-1d65-8ac7-3287c13633e9@gmail.com>
 <F8E2A7F5-3E1D-4118-B0AD-03FA79786BFD@kleczek.org>
Message-ID: <6b06a360-0069-488d-8b95-920e42f0fc64@oracle.com>

On 02/01/2024 07:56, Micha? K?eczek wrote:
> How about moduleY.layer().findModule(?moduleX?)?
>
That does a depth first search to find the "moduleX". I think Pavel is 
looking for the moduleX that is read by moduleY. This can be done in 
model world with the Configuration objects as they have the readability 
graphs to answer the question.

-Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240102/2a6594a2/attachment.htm>

From michal at kleczek.org  Tue Jan  2 11:55:31 2024
From: michal at kleczek.org (=?utf-8?Q?Micha=C5=82_K=C5=82eczek?=)
Date: Tue, 2 Jan 2024 12:55:31 +0100
Subject: How to get reads from module?
In-Reply-To: <6b06a360-0069-488d-8b95-920e42f0fc64@oracle.com>
References: <6b06a360-0069-488d-8b95-920e42f0fc64@oracle.com>
Message-ID: <B211666D-DBB9-41A3-ABE7-797FBA0AC38D@kleczek.org>

Now I am lost. Wouldn?t it find the same module?

Micha? 

> On 2 Jan 2024, at 12:40, Alan Bateman <Alan.Bateman at oracle.com> wrote:
> 
> ? On 02/01/2024 07:56, Micha? K?eczek wrote:
>> How about moduleY.layer().findModule(?moduleX?)?
>> 
> That does a depth first search to find the "moduleX". I think Pavel is looking for the moduleX that is read by moduleY. This can be done in model world with the Configuration objects as they have the readability graphs to answer the question.
> 
> -Alan

From Alan.Bateman at oracle.com  Tue Jan  2 12:43:13 2024
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Tue, 2 Jan 2024 12:43:13 +0000
Subject: How to get reads from module?
In-Reply-To: <B211666D-DBB9-41A3-ABE7-797FBA0AC38D@kleczek.org>
References: <6b06a360-0069-488d-8b95-920e42f0fc64@oracle.com>
 <B211666D-DBB9-41A3-ABE7-797FBA0AC38D@kleczek.org>
Message-ID: <10ada413-a882-4883-b44f-a5514427f60f@oracle.com>

On 02/01/2024 11:55, Micha? K?eczek wrote:
> Now I am lost. Wouldn?t it find the same module?
>
This is a readability graph rather than a dependency graph. They are the 
same thing until you add implied readability ("requires transitive") to 
the picture. In that case, it's possible for a module in layerC to 
require (and thus read) a module in layer A. That module in layer A, in 
turn, requires-transitive another module in layer A and thus our module 
in layerC reads two modules in layerA. It doesn't matter if there is a 
module in layerB with the same name as the second module in layerA, it's 
okay as long as there isn't a scenario where a module reads two (or 
more) modules with the same name.

-Alan.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240102/0c764172/attachment.htm>

From michal at kleczek.org  Tue Jan  2 12:45:23 2024
From: michal at kleczek.org (=?utf-8?Q?Micha=C5=82_K=C5=82eczek?=)
Date: Tue, 2 Jan 2024 13:45:23 +0100
Subject: How to get reads from module?
In-Reply-To: <10ada413-a882-4883-b44f-a5514427f60f@oracle.com>
References: <6b06a360-0069-488d-8b95-920e42f0fc64@oracle.com>
 <B211666D-DBB9-41A3-ABE7-797FBA0AC38D@kleczek.org>
 <10ada413-a882-4883-b44f-a5514427f60f@oracle.com>
Message-ID: <F7FA0896-1A97-4A64-BF1A-900406E7AC98@kleczek.org>



> On 2 Jan 2024, at 13:43, Alan Bateman <Alan.Bateman at oracle.com> wrote:
> 
> On 02/01/2024 11:55, Micha? K?eczek wrote:
>> Now I am lost. Wouldn?t it find the same module?
>> 
> This is a readability graph rather than a dependency graph. They are the same thing until you add implied readability ("requires transitive") to the picture.

Ahh, right.

Thanks!

?
Michal



From pavelturk2000 at gmail.com  Wed Jan 10 10:29:33 2024
From: pavelturk2000 at gmail.com (PavelTurk)
Date: Wed, 10 Jan 2024 12:29:33 +0200
Subject: A couple of questions
In-Reply-To: <331fd5e4-660f-485e-bf4b-5eecf4fa3487@oracle.com>
References: <7214b7d4-4002-1e0d-d7ee-f00eed90382f@gmail.com>
 <331fd5e4-660f-485e-bf4b-5eecf4fa3487@oracle.com>
Message-ID: <c663d18b-e2a1-a231-c786-a96763f72c55@gmail.com>


>>
>> 2. Let's suppose I have the following layer graph:
>>
>> 1) boot
>> 2) layerA (parent - boot) has moduleX
>> 3) layerB (parent - boot) has moduleX
>> 4) layerC (parents - layerA, layerB) has moduleY
>>
>> As I understand it is a possible situation. ModuleY requires moduleX that is present in two parent layers.
>> The question - how can I find out from which layer moduleX will be used for moduleY?
>
> The Configuration objects for layerA,?layerB and?layerC encapsulate the readability graphs used to the these module layers. Use Configuration::modules to get the set the resolved modules in the configuration, and ResovledRead::read to get the set of modules that are read. This will allow you to see which modules in C read "moduleX" in A and which modules in C read "moduleX" in B.
>

Still can't get reference to read modules:

 ??????? layerC.configuration().modules().stream().forEach((m) -> {
 ??????????? for (ResolvedModule r : m.reads()) {
 ??????????????? //LINE X
 ??????????? }
 ??????? });

At LINE X I need to get `Module` from `ResolvedModule` to use Module.getLayer() method. How to do it?

Best regards, Pavel

From Alan.Bateman at oracle.com  Thu Jan 11 13:30:02 2024
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Thu, 11 Jan 2024 13:30:02 +0000
Subject: A couple of questions
In-Reply-To: <c663d18b-e2a1-a231-c786-a96763f72c55@gmail.com>
References: <7214b7d4-4002-1e0d-d7ee-f00eed90382f@gmail.com>
 <331fd5e4-660f-485e-bf4b-5eecf4fa3487@oracle.com>
 <c663d18b-e2a1-a231-c786-a96763f72c55@gmail.com>
Message-ID: <f046c26f-96e1-46f7-b8cc-70d7dc51cb7d@oracle.com>

On 10/01/2024 10:29, PavelTurk wrote:
> :
>
> Still can't get reference to read modules:
>
> ??????? layerC.configuration().modules().stream().forEach((m) -> {
> ??????????? for (ResolvedModule r : m.reads()) {
> ??????????????? //LINE X
> ??????????? }
> ??????? });
>
> At LINE X I need to get `Module` from `ResolvedModule` to use 
> Module.getLayer() method. How to do it?

Starting in model world (Configuration/ResolvedModule) is right and you 
first need to find the Configuration containing the moduleX that moduleY 
reads, this should do it:

 ??????? Configuration cf = layerC.configuration()
 ??????????????? .findModule("moduleY")
 ??????????????? .orElseThrow()
 ??????????????? .reads().stream()
 ??????????????? .filter(m -> m.name().equals("moduleX"))
 ??????????????? .map(ResolvedModule::configuration)
 ??????????????? .findAny()
 ??????????????? .orElseThrow();

Once you have the Configuration then you need to find the ModuleLayer 
that was reifed from that Configuration, this should do it:

 ??????? ModuleLayer layer = layerC.parents().stream()
 ??????????????? .filter(l -> l.configuration() == cf)
 ??????????????? .findAny()
 ??????????????? .orElseThrow();

Can you try it?

-Alan



From pavelturk2000 at gmail.com  Thu Jan 11 21:57:05 2024
From: pavelturk2000 at gmail.com (PavelTurk)
Date: Thu, 11 Jan 2024 23:57:05 +0200
Subject: A couple of questions
In-Reply-To: <f046c26f-96e1-46f7-b8cc-70d7dc51cb7d@oracle.com>
References: <7214b7d4-4002-1e0d-d7ee-f00eed90382f@gmail.com>
 <331fd5e4-660f-485e-bf4b-5eecf4fa3487@oracle.com>
 <c663d18b-e2a1-a231-c786-a96763f72c55@gmail.com>
 <f046c26f-96e1-46f7-b8cc-70d7dc51cb7d@oracle.com>
Message-ID: <cf4747f5-3788-777d-bb69-14c90c4a5e22@gmail.com>


> Starting in model world (Configuration/ResolvedModule) is right and you first need to find the Configuration containing the moduleX that moduleY reads, this should do it:
>
Yes, I understand it, I sent a piece of real application code, so it was a little different.
> Once you have the Configuration then you need to find the ModuleLayer that was reifed from that Configuration, this should do it:
>
> ??????? ModuleLayer layer = layerC.parents().stream()
> ??????????????? .filter(l -> l.configuration() == cf)
> ??????????????? .findAny()
> ??????????????? .orElseThrow();
>
> Can you try it?
>
Yes, it worked. Thank you very much for your help. I didn't understand that we could map layer and modules of this layer by configuration.
So I just created Map where as a key I used the configuration of all layers and after that
for (var readModule : m.reads()) {
 ??? var layer = moduleLayersByConfiguration.get(readModule.configuration());
}

Several notes:

1. Your code `ModuleLayer layer = layerC.parents().stream()...` didn't find layer if moduleX was in boot layer, but not in direct parents of layerC. In our example 4
layers:
1)?boot
2)?layerA?(parent?-?boot)?has?moduleX
3)?layerB?(parent?-?boot)?has?moduleX
4)?layerC?(parents?-?layerA,?layerB)?has?moduleY

2. Please see this diagram https://i.stack.imgur.com/ss2os.png that was generated for real application using reads and ResolvedModules (the topic of this thread).
Here we have two layers - the right is a boot layer and the left is a webserver layer. The program must show reads of com.google.gson. Reads are shown using dashed arrows.
a) we see that gson reads 5 modules. At the same time gson requires only 1 module:
module com.google.gson {
 ??? exports com.google.gson;
 ??? exports com.google.gson.annotations;
 ??? exports com.google.gson.reflect;
 ??? exports com.google.gson.stream;

 ??? requires transitive java.sql;
}

Do I understand it correctly that extra 4 modules are read via java.sql?

3. Do I understand it correctly if some explicit module (with module-info) from webserver layer will require any automatic module from webserver layer then this explicit
module will read ALL automatic modules from both layers?

Best regards, Pavel

From Alan.Bateman at oracle.com  Fri Jan 12 14:23:14 2024
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Fri, 12 Jan 2024 14:23:14 +0000
Subject: A couple of questions
In-Reply-To: <cf4747f5-3788-777d-bb69-14c90c4a5e22@gmail.com>
References: <7214b7d4-4002-1e0d-d7ee-f00eed90382f@gmail.com>
 <331fd5e4-660f-485e-bf4b-5eecf4fa3487@oracle.com>
 <c663d18b-e2a1-a231-c786-a96763f72c55@gmail.com>
 <f046c26f-96e1-46f7-b8cc-70d7dc51cb7d@oracle.com>
 <cf4747f5-3788-777d-bb69-14c90c4a5e22@gmail.com>
Message-ID: <9d0bdf6e-1532-4cf7-af07-56811cf4407d@oracle.com>

On 11/01/2024 21:57, PavelTurk wrote:
> :
>
> a) we see that gson reads 5 modules. At the same time gson requires 
> only 1 module:
> module com.google.gson {
> ??? exports com.google.gson;
> ??? exports com.google.gson.annotations;
> ??? exports com.google.gson.reflect;
> ??? exports com.google.gson.stream;
>
> ??? requires transitive java.sql;
> }
>
> Do I understand it correctly that extra 4 modules are read via java.sql?

In the API docs for java.sql you will see that it requires transitive 
java.logging, java.transaction.xa, and transitive java.xml. This means 
that com.google.gson will read java.sql, java.logging, 
java.transaction.xa, and transitive java.xml. In addition, every module 
reads java.base so that explains the edges you see in the readability graph.


>
> 3. Do I understand it correctly if some explicit module (with 
> module-info) from webserver layer will require any automatic module 
> from webserver layer then this explicit
> module will read ALL automatic modules from both layers?

If an automatic module is enumerated during resolution when all 
observable automatic modules are enumerated. If an explicit module 
requires an automatic module then the explicit module will read all 
enumerated automatic modules.

That said, there is an open issue around readability when there is a 
stack or tree of configurations when there are modules with the same 
name in different configurations. The science is right when they are all 
explicit modules but there is an issue when there are automatic modules 
in configurations other than the configuration for the boot layer and 
where the module names aren't globally unique.? This is the issue we 
were discussing in other thread where it will require both specification 
and implementation changes to address.

-Alan

From pavelturk2000 at gmail.com  Mon Jan 15 16:03:21 2024
From: pavelturk2000 at gmail.com (PavelTurk)
Date: Mon, 15 Jan 2024 18:03:21 +0200
Subject: What is the practical use of JPMS layers?
Message-ID: <68259665-8305-c594-3e15-b905c48dbaeb@gmail.com>

Hello all.

JPMS is not only about modules, but it also supports layers. What is the practical use of JPMS layers?

Best regards, Pavel

From sormuras at gmail.com  Tue Jan 16 07:10:58 2024
From: sormuras at gmail.com (Christian Stein)
Date: Tue, 16 Jan 2024 08:10:58 +0100
Subject: What is the practical use of JPMS layers?
In-Reply-To: <68259665-8305-c594-3e15-b905c48dbaeb@gmail.com>
References: <68259665-8305-c594-3e15-b905c48dbaeb@gmail.com>
Message-ID: <CAJi8+g7P_GicofHmQy4UeMn4_2LZNpwKXS-NEPPDCCjQGv0fYg@mail.gmail.com>

An exemplary use of module layers is documented and implemented here:
https://github.com/moditect/layrry

Copied from its "Why Layrry?" section:

This is where Layrry comes in: utilizing the notion of module layers, it
> provides a declarative approach as well as an API for assembling
> modularized applications, organized in module layers. The JARs to be
> included are described using Maven GAV (group id, artifact id, version)
> coordinates, solving the issue of retrieving all required JARs in the right
> version.



Module layers allow to use different versions of one and the same module in
> different layers of an application (as long as they are not exposed in a
> conflicting way on module API boundaries).


Cheers,
Christian

On Mon, Jan 15, 2024 at 6:58?PM PavelTurk <pavelturk2000 at gmail.com> wrote:

> Hello all.
>
> JPMS is not only about modules, but it also supports layers. What is the
> practical use of JPMS layers?
>
> Best regards, Pavel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240116/4e993948/attachment-0001.htm>

From pavelturk2000 at gmail.com  Wed Jan 17 17:22:08 2024
From: pavelturk2000 at gmail.com (PavelTurk)
Date: Wed, 17 Jan 2024 19:22:08 +0200
Subject: Is it possible to add JRE and JDK modules in child layers?
Message-ID: <cc74de05-9c08-f3be-4ec8-c1406c517dbc@gmail.com>

Hello all.

We use JPMS for our framework where we have the following layers:

1) boot layer: framework + JRE/JDK modules (to add them we use --add-modules ALL-DEFAULT)
2) child layers [with child layers..] where every child layer is dynamically added component.

As I understand Layrry framework does the same.? I asked several days ago about practical use
of JPMS layers and got only one response about Layrry. So, I can conclude that layers are used
primarily for creating dynamic substems, plugins, components, extensions etc.

However, we couldn't find a way to add JRE and JDK modules in child layers. Because of this we
have to add all default modules to boot layer what is a very very serious flaw.

Why do we need it? As components are created dynamically when/if they are required we don't
know what JRE/JDK modules will be required we create boot layer. That's why add all of them to
boot layer.

Could anyone say if it is possible to add JRE/JDK modules to child layers? Is there any way to do it?
Are there any plans to add such possibility?

Best regards, Pavel

From Alan.Bateman at oracle.com  Thu Jan 18 09:04:54 2024
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Thu, 18 Jan 2024 09:04:54 +0000
Subject: Is it possible to add JRE and JDK modules in child layers?
In-Reply-To: <cc74de05-9c08-f3be-4ec8-c1406c517dbc@gmail.com>
References: <cc74de05-9c08-f3be-4ec8-c1406c517dbc@gmail.com>
Message-ID: <e8afc4e3-c09b-436c-ad03-3acbfea9b1b2@oracle.com>

On 17/01/2024 17:22, PavelTurk wrote:
> Hello all.
>
> We use JPMS for our framework where we have the following layers:
>
> 1) boot layer: framework + JRE/JDK modules (to add them we use 
> --add-modules ALL-DEFAULT)
> 2) child layers [with child layers..] where every child layer is 
> dynamically added component.
>
> As I understand Layrry framework does the same.? I asked several days 
> ago about practical use
> of JPMS layers and got only one response about Layrry. So, I can 
> conclude that layers are used
> primarily for creating dynamic substems, plugins, components, 
> extensions etc.
>
> However, we couldn't find a way to add JRE and JDK modules in child 
> layers. Because of this we
> have to add all default modules to boot layer what is a very very 
> serious flaw.
>
> Why do we need it? As components are created dynamically when/if they 
> are required we don't
> know what JRE/JDK modules will be required we create boot layer. 
> That's why add all of them to
> boot layer.
>
> Could anyone say if it is possible to add JRE/JDK modules to child 
> layers? Is there any way to do it?
> Are there any plans to add such possibility?

This is the topic that was on the original requirements wish list as 
"Run-time augmentation of the platform modules". There isn't any general 
support for this in the JDK. One reason is that module layers, and in 
this case the boot layer, are immutable. Another reason is that 
integrity of the platform requires that classes in the java.* packages 
be defined to the boot or platform class loaders. In other words, you 
can't have child layers containing modules with java.* classes that are 
mapped to custom class loaders.

That said, there are two things to be aware of:

1. ALL-DEFAULT. You've already found that. As explained in JEP 261, this 
is there to support cases where a container module is in the boot layer 
but it does dynamic configuration with modules that require standard 
modules that might not be in the boot layer.

2. Java agents and the JMX/management agent. If an agent is started in a 
running VM, and the required java.instrument or java.management modules 
are not in the boot layer, then a child layer will magically comes into 
being with the required standard + JDK modules. This it possible because 
the starting of these agents is under the control of the VM with the 
original mapping of modules to class loaders.

To your question, you can't use the API to do dynamic configuration and 
create module layers that contain the standard modules. The solution for 
containers is ALL-DEFAULT to ensure that all modules in the system image 
that export an API are in the boot layer.

-Alan

From pavelturk2000 at gmail.com  Thu Jan 18 13:49:01 2024
From: pavelturk2000 at gmail.com (PavelTurk)
Date: Thu, 18 Jan 2024 15:49:01 +0200
Subject: Is it possible to add JRE and JDK modules in child layers?
In-Reply-To: <e8afc4e3-c09b-436c-ad03-3acbfea9b1b2@oracle.com>
References: <cc74de05-9c08-f3be-4ec8-c1406c517dbc@gmail.com>
 <e8afc4e3-c09b-436c-ad03-3acbfea9b1b2@oracle.com>
Message-ID: <18b34e7d-dad8-e57d-5328-083bf6cb734e@gmail.com>

Hi Alan,

Thank you very much for your answer.

You wrote:
> This is the topic that was on the original requirements wish list as "Run-time augmentation of the platform modules". There isn't any general support for this in the JDK. One reason is that module layers, and in this case the boot layer, are immutable. Another reason is that integrity of the platform requires that classes in the java.* packages be defined to the boot or platform class loaders. In other words, you can't have child layers containing modules with java.* classes that are mapped?to?custom?class?loaders. 

I think that that all layers (including boot layer) must be immutable. What about "Another reason is that integrity of the platform
requires that classes in the java.* packages be defined to the boot or platform class loaders".

Do I understand it correctly that it is ABSOLUTELY, ABSOLUTELY impossible to do find a solution for this problem?
I mean that all today's knowledge in software development are still not enough to find a way
a) to provide integrity of the platform and b) to load standard (JRE/JDK) modules in child layers?

Best regards, Pavel


From tomas.langer at oracle.com  Fri Jan 19 13:02:55 2024
From: tomas.langer at oracle.com (Tomas Langer)
Date: Fri, 19 Jan 2024 13:02:55 +0000
Subject: Java extensibility and JPMS (ServiceLoader)
Message-ID: <CH0PR10MB51006744C4F05C3C19FCBB06FA702@CH0PR10MB5100.namprd10.prod.outlook.com>

Helidon currently has around 300 modules with module-info.java. In general, this has improved our module structure and design.
Yet, we are now encountering some major issues related to extensibility.
 I will put down a few points that are problematic, and explain each in detail further in the e-mail (it is quite long, sorry about that).

1. provider implementations cannot be code generated without major problems
2. the provider interface module MUST be on module path, even if it could have `requires static`
3. the provider implementation must be public with public constructor
4. duality of definition between module path and class path

I am trying to propose a solution within the bounds of the current service loader design. Of course there may be other solutions (both with current design, or even creating a brand new extensibility solution in Java, this is just to illustrate it).

The first two issues are quite major, as they force us to recommend not to use JPMS to our users...

We could design our own extensibility approach, though that would require the use of reflection to instantiate the services, and we would have to live with the limits of the module system (where Java ServiceLoader works around a few of them).
I feel the right way is to use what Java provides, so I would welcome any help (and possibly changes in the language) to support our use cases.

Thanks,
Tomas Langer
Architect, project Helidon


Ad 1 - Code generation
-------------------------------------
Problem: We cannot code generate service implementations (well we can, but the user must handcraft them in module-info.java, so we end up running the APT, generating a service, failing the compiler to tell the user to add the service, compiling again every time a new service is added).
Possible solution: Provide extensibility to module-info.java that can be code generated
Without JPMS: it just works, as `META-INF/services` files can be code generated without issues

Details:
What I do not see is how we are supposed to do extensibility through annotation processing.
There are a lot of usecases for this, such as:
- generating code for serializers/deserializers for objects that persist to JSON, XML, YAML
- generating code for database entities
- generating descriptor for services in a service registry

We actually want to implement these three use cases, and it is a major pain for the user - I would not mind much if this hurt us, as framework developers, but we must force the user to take action by breaking the compilation, or come up with some really weird solution (such as source code modification using some preprocessor before compilation, or postprocessor running on bytecode to re-generate module-info.class)


Ad 2 - Provider module cannot be optional dependency
-------------------------------------
Problem: We cannot declare `requires static` on a module that has the ServiceLoader provider interface (or abstract class)
Possible solution: Change the rules for JPMS to allow this
Without JPMS: it just works, as `META-INF/services` to not impose any classpath structure

Details:
If a module (my.json) defines a provider interface (let's say `JsonSerializer`), and I create a module with `MyJsonSerializer` (provides JsonSerializer witih MyJsonSerializer), currently I MUST do the following "requires my.json". If I do a "requires static my.json" I fail to start the JVM if that module is not on module path.

The service CANNOT be used unless the module is on module path (as anybody attempting to load it must declare `uses` in their module info with a proper `requires` on the `my.json` module.

So what are my options right now?
- have a "requires my.json" and just dump the module on all my users (not so good - people may want to use my library without JSON altogether, I may also provide support for XML, YAML - all of these would need to be on module path)
- create a module for each (one for JSON, XML, YAML + my library) - resulting in 4 modules (and this may grow if I decide to support other format); this looks kind of OK on the first look, but with the number of modules we have, and the number of features we support, this gets out of hand really really quickly; this approach is also very user unfriendly, as now the user needs to understand 4 modules instead of just 1, and use the right ones at the right time).
- considering the number of modules we already have, this would make our project unmaintainable (and unusable for users)

As JPMS already allows "static" dependencies, there should be no reason not to allow it in this case as well. We can break the module system even now - just use a class from static dependency in a public class - this will fail at runtime only. The service loader is not different (this is a reaction to text in https://bugs.openjdk.org/browse/JDK-8299504).


Ad 3 - Provider implementation must be public with public constructor
-------------------------------------
Problem: This creates a new public API that we may not want to expose, or document
Possible solution: Change the rules to allow package local service provider implementations with package local constructors
Without JPMS: Same issue, even more problematic as there is no restrictions on package visibility

Details:
Provider implementations are not supposed to be visible to users - they are not public API of my module (the fact that I provide a service is part of my public API).
Right now there is only one option to work around this, and it only works in JPMS, and in my opinion it brings in even more problems - put the provider implementation in an un-exported package.
The problem with this approach is that now the provider implementation MUST use only public methods of my module, thus creating even more public APIs, where if I just put it in my exported package, I can use package private methods of my other classes to implement the service (so I pay the price of having one public class with one public constructor agains multiple public classes and public methods). Also the "hiding" in unexported package is lost when on classpath anyway...


Ad 4 - Duality of definition between classpath and module path
-------------------------------------
Problem: To support services, we MUST declare them twice - once in `provides` in module-info.java, once through META-INF/services
Possible solution: Java could read module descriptors even when running in classpath mode to add service implementations and merge it with META-INF/services information
Without JPMS: it just works, as `META-INF/services` is always honored on classpath and for non-JPMS modules on module path

Details:
This is again quite a pain for us as framework develoepers, and a pitfall for users. When we started with JPMS, we had both created manually, which obviously ended in a huge inconsistent mess.
So now we have a custom Maven plugin, that creates META-INF/services files based on the content in module-info.java and fails on inconsistencies.
I do not consider this a nice solution for us, and definitely not for end users. Also there is no way to find out that you forgot to add one (or the other), as JVM just does not care. So basically you end up with a runtime issue that is really hard to troubleshoot.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240119/0791b794/attachment-0001.htm>

From Alan.Bateman at oracle.com  Fri Jan 19 14:30:20 2024
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Fri, 19 Jan 2024 14:30:20 +0000
Subject: Is it possible to add JRE and JDK modules in child layers?
In-Reply-To: <18b34e7d-dad8-e57d-5328-083bf6cb734e@gmail.com>
References: <cc74de05-9c08-f3be-4ec8-c1406c517dbc@gmail.com>
 <e8afc4e3-c09b-436c-ad03-3acbfea9b1b2@oracle.com>
 <18b34e7d-dad8-e57d-5328-083bf6cb734e@gmail.com>
Message-ID: <29624b9e-910c-4193-8b01-5814d1cae6b4@oracle.com>

On 18/01/2024 13:49, PavelTurk wrote:
> :
>
> Do I understand it correctly that it is ABSOLUTELY, ABSOLUTELY 
> impossible to do find a solution for this problem?
> I mean that all today's knowledge in software development are still 
> not enough to find a way
> a) to provide integrity of the platform and b) to load standard 
> (JRE/JDK) modules in child layers?

I didn't say it was impossible, instead I listed down some of the 
challenges with finding a workable solution to this type of use case. 
The science isn't too difficult, it's mostly about slicing a module 
graph and reifying the slices as a stack of ModuleLayers so that the 
ModuleLayer.Controller handed out to the container is for a ModueLayer 
that doesn't include the modules mapped to the boot or platform class 
loaders. The harder part is coming up with an API that looks like it has 
always been there.

For now, deploying the container with --add-modules ALL-DEFAULT is not 
terrible. It ensures that all modules in the run-time image that export 
an API are in the boot layer so there shouldn't be any issues with 
modules in child layers requiring modules that the container doesn't 
transitively require.

-Alan

From josiahnoel at gmail.com  Fri Jan 19 15:17:30 2024
From: josiahnoel at gmail.com (Josiah Noel)
Date: Fri, 19 Jan 2024 10:17:30 -0500
Subject: Java extensibility and JPMS (ServiceLoader)
In-Reply-To: <CH0PR10MB51006744C4F05C3C19FCBB06FA702@CH0PR10MB5100.namprd10.prod.outlook.com>
References: <CH0PR10MB51006744C4F05C3C19FCBB06FA702@CH0PR10MB5100.namprd10.prod.outlook.com>
Message-ID: <CAJ_t5UZ88v-Z5WmR5rui4H8dTOnxkx3FF08V5YRmxKp9XXW_HA@mail.gmail.com>

I help maintain a few service-loader-based modular annotation processor
libraries with the avaje framework, so I have also run into this and had to
work around it.

*Ad 1:*
So this is indeed an inconvenience but in my experience, it's not a crazy
one. When processing is over we check the module-info's directives to throw
compilation warnings like:
*[ERROR]
/M:/Dev/avaje-helidon-nima-api-example/src/main/java/module-info.java:[1,1]
Missing `provides io.ava*
*je.http.client.HttpClient.GeneratedComponent with
com.jojo.helidon.api.client.httpclient.GeneratedHttpCompon*
*ent;`*
*[ERROR]
/M:/Dev/avaje-helidon-nima-api-example/src/main/java/module-info.java:[1,1]
Missing `provides io.ava*
*je.jsonb.Jsonb.GeneratedComponent with
com.jojo.helidon.api.jsonb.GeneratedJsonComponent;`*
*[ERROR]
/M:/Dev/avaje-helidon-nima-api-example/src/main/java/module-info.java:[1,1]
Missing `provides io.ava*
*je.validation.Validator.GeneratedComponent with
com.jojo.helidon.api.controller.valid.GeneratedValidatorComp*
*onent;`*
*[ERROR]
/M:/Dev/avaje-helidon-nima-api-example/src/main/java/module-info.java:[1,1]
Missing "provides io.ava*
*je.inject.spi.Module with com.jojo.helidon.api.ApiModule;"*

The *real problem *with checking the module info in a processor is that
ModuleElement is bugged such that if you call *ModuleElement.getDirectives*
on the project module It breaks compilation
<https://bugs.openjdk.org/browse/JDK-8315125>. (they called this one a
duplicate but I still have the problem even in JDK 22-ea and JDK 23-ea)

To get around this I have to use Filer to retrieve the module-info's
sources file and parse it as a string to avoid calling *getDirectives. * I've
built libraries to do this to make this easier, but the problem remains
that if any other processor uses *getDirectives*, compilation will still
break.

*Ad 2:*
I asked this question here a while back, and what I got out of it is that
we needed to do some form of circular dependency if we truly wanted
optional services, the tech for compiling multi-module jars isn't here yet.
It's a pain to deploy when we change the plugins, but we rarely do so this
has worked fine for us. Example: avaje-validator
<https://github.com/avaje/avaje-validator> provides a plugin for avaje
inject that itself depends on avaje-validator.

*Ad 3: *
Yeah I got nothing, we never had to do this.

*Ad 4: *
This one was the simplest, we can define an annotation to go on the user's
service class and process them to generate META-INF files and validate the
module-info. In this way, one cannot forget to add the proper module-info
information. In some of the libs we do this with their processors, but we
also have a dedicated library for handling this sort of thing.


On Fri, Jan 19, 2024 at 8:04?AM Tomas Langer <tomas.langer at oracle.com>
wrote:

> Helidon currently has around 300 modules with module-info.java. In
> general, this has improved our module structure and design.
> Yet, we are now encountering some major issues related to extensibility.
>  I will put down a few points that are problematic, and explain each in
> detail further in the e-mail (it is quite long, sorry about that).
>
> 1. provider implementations cannot be code generated without major problems
> 2. the provider interface module MUST be on module path, even if it could
> have `requires static`
> 3. the provider implementation must be public with public constructor
> 4. duality of definition between module path and class path
>
> I am trying to propose a solution within the bounds of the current service
> loader design. Of course there may be other solutions (both with current
> design, or even creating a brand new extensibility solution in Java, this
> is just to illustrate it).
>
> The first two issues are quite major, as they force us to recommend not to
> use JPMS to our users...
>
> We could design our own extensibility approach, though that would require
> the use of reflection to instantiate the services, and we would have to
> live with the limits of the module system (where Java ServiceLoader works
> around a few of them).
> I feel the right way is to use what Java provides, so I would welcome any
> help (and possibly changes in the language) to support our use cases.
>
> Thanks,
> Tomas Langer
> Architect, project Helidon
>
>
> *Ad 1 - Code generation*
> -------------------------------------
> Problem: We cannot code generate service implementations (well we can, but
> the user must handcraft them in module-info.java, so we end up running the
> APT, generating a service, failing the compiler to tell the user to add the
> service, compiling again every time a new service is added).
> Possible solution: Provide extensibility to module-info.java that can be
> code generated
> Without JPMS: it just works, as `META-INF/services` files can be code
> generated without issues
>
> Details:
> What I do not see is how we are supposed to do extensibility through
> annotation processing.
> There are a lot of usecases for this, such as:
> - generating code for serializers/deserializers for objects that persist
> to JSON, XML, YAML
> - generating code for database entities
> - generating descriptor for services in a service registry
>
> We actually want to implement these three use cases, and it is a major
> pain for the user - I would not mind much if this hurt us, as framework
> developers, but we must force the user to take action by breaking the
> compilation, or come up with some really weird solution (such as source
> code modification using some preprocessor before compilation, or
> postprocessor running on bytecode to re-generate module-info.class)
>
>
> *Ad 2 - Provider module cannot be optional dependency*
> -------------------------------------
> Problem: We cannot declare `requires static` on a module that has the
> ServiceLoader provider interface (or abstract class)
> Possible solution: Change the rules for JPMS to allow this
> Without JPMS: it just works, as `META-INF/services` to not impose any
> classpath structure
>
> Details:
> If a module (my.json) defines a provider interface (let's say
> `JsonSerializer`), and I create a module with `MyJsonSerializer` (provides
> JsonSerializer witih MyJsonSerializer), currently I MUST do the following
> "requires my.json". If I do a "requires static my.json" I fail to start the
> JVM if that module is not on module path.
>
> The service CANNOT be used unless the module is on module path (as anybody
> attempting to load it must declare `uses` in their module info with a
> proper `requires` on the `my.json` module.
>
> So what are my options right now?
> - have a "requires my.json" and just dump the module on all my users (not
> so good - people may want to use my library without JSON altogether, I may
> also provide support for XML, YAML - all of these would need to be on
> module path)
> - create a module for each (one for JSON, XML, YAML + my library) -
> resulting in 4 modules (and this may grow if I decide to support other
> format); this looks kind of OK on the first look, but with the number of
> modules we have, and the number of features we support, this gets out of
> hand really really quickly; this approach is also very user unfriendly, as
> now the user needs to understand 4 modules instead of just 1, and use the
> right ones at the right time).
> - considering the number of modules we already have, this would make our
> project unmaintainable (and unusable for users)
>
> As JPMS already allows "static" dependencies, there should be no reason
> not to allow it in this case as well. We can break the module system even
> now - just use a class from static dependency in a public class - this will
> fail at runtime only. The service loader is not different (this is a
> reaction to text in https://bugs.openjdk.org/browse/JDK-8299504).
>
>
> *Ad 3 - Provider implementation must be public with public constructor*
> -------------------------------------
> Problem: This creates a new public API that we may not want to expose, or
> document
> Possible solution: Change the rules to allow package local service
> provider implementations with package local constructors
> Without JPMS: Same issue, even more problematic as there is no
> restrictions on package visibility
>
> Details:
> Provider implementations are not supposed to be visible to users - they
> are not public API of my module (the fact that I provide a service is part
> of my public API).
> Right now there is only one option to work around this, and it only works
> in JPMS, and in my opinion it brings in even more problems - put the
> provider implementation in an un-exported package.
> The problem with this approach is that now the provider implementation
> MUST use only public methods of my module, thus creating even more public
> APIs, where if I just put it in my exported package, I can use package
> private methods of my other classes to implement the service (so I pay the
> price of having one public class with one public constructor agains
> multiple public classes and public methods). Also the "hiding" in
> unexported package is lost when on classpath anyway...
>
>
> *Ad 4 - Duality of definition between classpath and module path*
> -------------------------------------
> Problem: To support services, we MUST declare them twice - once in
> `provides` in module-info.java, once through META-INF/services
> Possible solution: Java could read module descriptors even when running in
> classpath mode to add service implementations and merge it with
> META-INF/services information
> Without JPMS: it just works, as `META-INF/services` is always honored on
> classpath and for non-JPMS modules on module path
>
> Details:
> This is again quite a pain for us as framework develoepers, and a pitfall
> for users. When we started with JPMS, we had both created manually, which
> obviously ended in a huge inconsistent mess.
> So now we have a custom Maven plugin, that creates META-INF/services files
> based on the content in module-info.java and fails on inconsistencies.
> I do not consider this a nice solution for us, and definitely not for end
> users. Also there is no way to find out that you forgot to add one (or the
> other), as JVM just does not care. So basically you end up with a runtime
> issue that is really hard to troubleshoot.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240119/3b667438/attachment-0001.htm>

From johannes.spangenberg at hotmail.de  Sun Jan 21 01:19:02 2024
From: johannes.spangenberg at hotmail.de (Johannes Spangenberg)
Date: Sun, 21 Jan 2024 02:19:02 +0100
Subject: Java extensibility and JPMS (ServiceLoader)
In-Reply-To: <CH0PR10MB51006744C4F05C3C19FCBB06FA702@CH0PR10MB5100.namprd10.prod.outlook.com>
References: <CH0PR10MB51006744C4F05C3C19FCBB06FA702@CH0PR10MB5100.namprd10.prod.outlook.com>
Message-ID: <AM0PR09MB23214AF0743DA37454DF2D50E9762@AM0PR09MB2321.eurprd09.prod.outlook.com>

> *Ad 2 - Provider module cannot be optional dependency*
> -------------------------------------
> Problem: We cannot declare `requires static` on a module that has the 
> ServiceLoader provider interface (or abstract class)

Note that there was a relative long discussion about this exact topic in 
April 2023:

https://mail.openjdk.org/pipermail/jigsaw-dev/2023-April/014846.html

I personally think that the current implementation is too restrictive as 
I see very little gain in this runtime error, but there was also a 
noticeable opposition.

> *Ad 4 - Duality of definition between classpath and module path*
> -------------------------------------
> Problem: To support services, we MUST declare them twice - once in 
> `provides` in module-info.java, once through META-INF/services
> Possible solution: Java could read module descriptors even when 
> running in classpath mode to add service implementations and merge it 
> with META-INF/services information

While I did also run into this issue, I would favor a solution during 
the build phase. Maybe some project still uses Java 8, where the JDK 
does not yet know about the module-info. The behavior should not change 
between different versions of the runtime.

PS: I am not affiliated with the project, besides being subscribed to 
this mailing list.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240121/7e8fd4bf/attachment.htm>

From josiahnoel at gmail.com  Sun Jan 21 04:16:22 2024
From: josiahnoel at gmail.com (Josiah Noel)
Date: Sat, 20 Jan 2024 23:16:22 -0500
Subject: Java extensibility and JPMS (ServiceLoader)
In-Reply-To: <AM0PR09MB23214AF0743DA37454DF2D50E9762@AM0PR09MB2321.eurprd09.prod.outlook.com>
References: <CH0PR10MB51006744C4F05C3C19FCBB06FA702@CH0PR10MB5100.namprd10.prod.outlook.com>
 <AM0PR09MB23214AF0743DA37454DF2D50E9762@AM0PR09MB2321.eurprd09.prod.outlook.com>
Message-ID: <CAJ_t5UYrAt4=8CkhT9ZDF3Fqn0+PWFEz6SKc4SC6gJk8TH_y4w@mail.gmail.com>

>
> Note that there was a relative long discussion about this exact topic in
> April 2023:
>

Should multi-module jars be implemented, most issues with problem 2 would
be resolved without needing to resort to circular dependencies or requiring
the user to remember a bunch of extra service modules. Most apps deploying
as a jar would have the optional services work essentially as if they were
on the classpath. The one exception to this case I've found is that
applications deployed as fully modular jlink runtime images require the
optional service modules to be explicitly defined in the module-info or
added to the jlink command. (which is a pain for me, as I like using
jlink to add my application modules to the runtime)

This too I've had to deal with by scanning the module info in the processor
and giving compiler warnings if a service is detected in the processing
environment and not specified in the module-info. It's unfortunate though
that this limitation will continue to exist.

While I did also run into this issue, I would favor a solution during the
> build phase. Maybe some project still uses Java 8, where the JDK does not
> yet know about the module-info. The behavior should not change between
> different versions of the runtime.


Well, Helidon's baseline is now JDK 21, so Java 8 compatibility shouldn't
be a concern. I still think that auto-generating service files with an
annotation processor is the way to go if an alternative easier solution
cannot be found in this discussion.

On Sat, Jan 20, 2024 at 8:19?PM Johannes Spangenberg <
johannes.spangenberg at hotmail.de> wrote:

> *Ad 2 - Provider module cannot be optional dependency*
> -------------------------------------
> Problem: We cannot declare `requires static` on a module that has the
> ServiceLoader provider interface (or abstract class)
>
> Note that there was a relative long discussion about this exact topic in
> April 2023:
>
> https://mail.openjdk.org/pipermail/jigsaw-dev/2023-April/014846.html
>
> I personally think that the current implementation is too restrictive as I
> see very little gain in this runtime error, but there was also a noticeable
> opposition.
>
> *Ad 4 - Duality of definition between classpath and module path*
> -------------------------------------
> Problem: To support services, we MUST declare them twice - once in
> `provides` in module-info.java, once through META-INF/services
> Possible solution: Java could read module descriptors even when running in
> classpath mode to add service implementations and merge it with
> META-INF/services information
>
> While I did also run into this issue, I would favor a solution during the
> build phase. Maybe some project still uses Java 8, where the JDK does not
> yet know about the module-info. The behavior should not change between
> different versions of the runtime.
>
> PS: I am not affiliated with the project, besides being subscribed to this
> mailing list.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240120/78aedf90/attachment-0001.htm>

From pavelturk2000 at gmail.com  Mon Jan 22 10:07:37 2024
From: pavelturk2000 at gmail.com (PavelTurk)
Date: Mon, 22 Jan 2024 12:07:37 +0200
Subject: Is it possible to add JRE and JDK modules in child layers?
In-Reply-To: <29624b9e-910c-4193-8b01-5814d1cae6b4@oracle.com>
References: <cc74de05-9c08-f3be-4ec8-c1406c517dbc@gmail.com>
 <e8afc4e3-c09b-436c-ad03-3acbfea9b1b2@oracle.com>
 <18b34e7d-dad8-e57d-5328-083bf6cb734e@gmail.com>
 <29624b9e-910c-4193-8b01-5814d1cae6b4@oracle.com>
Message-ID: <0c2552cc-895b-96e6-1524-5488ed420749@gmail.com>

Alan, thank you very much for your explanations.

I created an issue https://bugs.openjdk.org/browse/JDK-8324272
So, everyone who is interested in this topic can track it.

Best regards, Pavel

On 1/19/24 4:30 PM, Alan Bateman wrote:
> On 18/01/2024 13:49, PavelTurk wrote:
>> :
>>
>> Do I understand it correctly that it is ABSOLUTELY, ABSOLUTELY impossible to do find a solution for this problem?
>> I mean that all today's knowledge in software development are still not enough to find a way
>> a) to provide integrity of the platform and b) to load standard (JRE/JDK) modules in child layers?
>
> I didn't say it was impossible, instead I listed down some of the challenges with finding a workable solution to this type of use case. The science isn't too difficult, it's mostly about slicing a module graph and reifying the slices as a stack of ModuleLayers so that the ModuleLayer.Controller handed out to the container is for a ModueLayer that doesn't include the modules mapped to the boot or platform class loaders. The harder part is coming up with an API that looks like it has always been there.
>
> For now, deploying the container with --add-modules ALL-DEFAULT is not terrible. It ensures that all modules in the run-time image that export an API are in the boot layer so there shouldn't be any issues with modules in child layers requiring modules that the container doesn't transitively require.
>
> -Alan


From alex.buckley at oracle.com  Mon Jan 22 20:13:39 2024
From: alex.buckley at oracle.com (Alex Buckley)
Date: Mon, 22 Jan 2024 12:13:39 -0800
Subject: Java extensibility and JPMS (ServiceLoader)
In-Reply-To: <CH0PR10MB51006744C4F05C3C19FCBB06FA702@CH0PR10MB5100.namprd10.prod.outlook.com>
References: <CH0PR10MB51006744C4F05C3C19FCBB06FA702@CH0PR10MB5100.namprd10.prod.outlook.com>
Message-ID: <f03aab4b-1e88-46b7-bb69-363df62589b4@oracle.com>

Tomas,

A clarification for point 3:

On 1/19/2024 5:02 AM, Tomas Langer wrote:
> *Ad 3 - Provider implementation must be public with public constructor*
> -------------------------------------
> Details:
> Provider implementations are not supposed to be visible to users - they 
> are not public API of my module (the fact that I provide a service is 
> part of my public API).
> Right now there is only one option to work around this, and it only 
> works in JPMS, and in my opinion it brings in even more problems - put 
> the provider implementation in an un-exported package.
> The problem with this approach is that now the provider implementation 
> MUST use only public methods of my module, thus creating even more 
> public APIs, where if I just put it in my exported package, I can use 
> package private methods of my other classes to implement the service (so 
> I pay the price of having one public class with one public constructor 
> agains multiple public classes and public methods). Also the "hiding" in 
> unexported package is lost when on classpath anyway...

I don't understand what you mean by "the provider implementation MUST 
use only public methods of my module". You have an arrangement of 
modules and packages in mind that I simply cannot see -- please share it.

Alex

From alex.buckley at oracle.com  Tue Jan 23 01:35:40 2024
From: alex.buckley at oracle.com (Alex Buckley)
Date: Mon, 22 Jan 2024 17:35:40 -0800
Subject: Java extensibility and JPMS (ServiceLoader)
In-Reply-To: <CH0PR10MB51004A2F226B02B975163EE1FA752@CH0PR10MB5100.namprd10.prod.outlook.com>
References: <CH0PR10MB51006744C4F05C3C19FCBB06FA702@CH0PR10MB5100.namprd10.prod.outlook.com>
 <f03aab4b-1e88-46b7-bb69-363df62589b4@oracle.com>
 <CH0PR10MB51004A2F226B02B975163EE1FA752@CH0PR10MB5100.namprd10.prod.outlook.com>
Message-ID: <9d838c78-fe31-48ce-a9a7-30051c354439@oracle.com>

Tomas,

I realize that you want to allow your modular JARs to be deployed on the 
classpath and still enjoy some degree of encapsulation.

However, mapping packages 1:1 to modules isn't the right way to use the 
module system. You get to formalize the system structure, and will be 
forced to break any legacy cycles between packages, but every module 
will export 100% of its packages, so there's no real information hiding. 
The maintainer of a module in one subsystem can freely require a module 
in another subsystem and access the exported-but-not-really-supported 
package of that other subsystem, so over time, a ball of mud will re-emerge.

Having "internal" packages is a _feature_ of modules, not a problem. 
It's only a problem if those modules are deployed on the classpath, and 
we don't recommend doing that.

On point 3: The requirement that a provider class must be public with a 
public ctor comes from the original design of ServiceLoader. This 
resulted in providers that were "too public", so we used the module 
system to solve that problem: you can take existing providers, "hide" 
them in unexported/internal packages of modules, and service consumers 
are not impacted at all.

Except for the small addition of "provider methods" as an alternative to 
public ctors, we took ServiceLoader in Java 9 as we found it. Evolving 
ServiceLoader in Java 23+ to allow providers with very limited 
accessibility -- package-access classes / ctors / provider methods -- is 
not incompatible with anything in the module system. In fact, keeping 
the circle of code that can instantiate providers as tight as possible 
is a good thing. However, we haven't heard the request for 
package-access providers from many people, and working on that feature 
would mean not working on something else, so it's unlikely that the 
feature will appear in the foreseeable future.

Alex

On 1/22/2024 3:30 PM, Tomas Langer wrote:
> We have a single module/single package approach, so all internal classes 
> are hidden by visibility (package local) and not by package (as that 
> cannot be enforced on classpath.
> Using more than one package usually results in internal packages and in 
> requirement to have some methods public, so classes in other packages of 
> your module can access them (where we can live with package local only...)
> 
> Tomas
> 
> Zasl?no z Outlooku pro Android <https://aka.ms/AAb9ysg>
> ------------------------------------------------------------------------
> *From:* jigsaw-dev <jigsaw-dev-retn at openjdk.org> on behalf of Alex 
> Buckley <alex.buckley at oracle.com>
> *Sent:* Monday, January 22, 2024 9:13:39 PM
> *To:* jigsaw-dev at openjdk.org <jigsaw-dev at openjdk.org>
> *Subject:* Re: Java extensibility and JPMS (ServiceLoader)
> Tomas,
> 
> A clarification for point 3:
> 
> On 1/19/2024 5:02 AM, Tomas Langer wrote:
>> *Ad 3 - Provider implementation must be public with public constructor*
>> -------------------------------------
>> Details:
>> Provider implementations are not supposed to be visible to users - they 
>> are not public API of my module (the fact that I provide a service is 
>> part of my public API).
>> Right now there is only one option to work around this, and it only 
>> works in JPMS, and in my opinion it brings in even more problems - put 
>> the provider implementation in an un-exported package.
>> The problem with this approach is that now the provider implementation 
>> MUST use only public methods of my module, thus creating even more 
>> public APIs, where if I just put it in my exported package, I can use 
>> package private methods of my other classes to implement the service (so 
>> I pay the price of having one public class with one public constructor 
>> agains multiple public classes and public methods). Also the "hiding" in 
>> unexported package is lost when on classpath anyway...
> 
> I don't understand what you mean by "the provider implementation MUST
> use only public methods of my module". You have an arrangement of
> modules and packages in mind that I simply cannot see -- please share it.
> 
> Alex

From tomas.langer at oracle.com  Thu Jan 25 12:34:42 2024
From: tomas.langer at oracle.com (Tomas Langer)
Date: Thu, 25 Jan 2024 12:34:42 +0000
Subject: Starting a module with main class from a different module
Message-ID: <CH0PR10MB51001B61925FE6B6E1EC1EF8FA7A2@CH0PR10MB5100.namprd10.prod.outlook.com>

Is there a way to define a main class on a module if it comes from a different one?

Currently we need to start the JVM as follows:
java ?module-path ... --add-modules my.modular.application --module io.helidon.microprofile.cdi/io.helidon.microprofile.cdi.Main

What we do:

  *   our microprofile implementation bootstraps a framework, that discovers its component using service loader, it also contains the Main class needed to start
  *   the "my.modular.application" provides such components (such as an HTTP endpoint), but does not need a main class

Ideally, I would like to run "my.modular.application" to have a simpler command line.
I cannot do that, as the main class is from a different module (and I think that cannot be added to the module metadata).
So I have to run the main class from the CDI module, and use `--add-modules`, as otherwise my application module would not end up on the module path (as it is not the "main" module started).


Is there some way please to simplify this? Or do we need to create a Main class for each application, even if it would just delegate to the "cdi.Main"?

When running on classpath, there is no problem, as the `Main-Class` attribute in manifest can point to a class in another jar, and the `Class-Path` attribute in the manifest can contain the jar that has the main class, so on classpath we can do:
java -jar my.modular.application.jar

Thanks
Tomas Langer
project Helidon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240125/626f18aa/attachment.htm>

From Alan.Bateman at oracle.com  Thu Jan 25 15:11:51 2024
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Thu, 25 Jan 2024 15:11:51 +0000
Subject: Starting a module with main class from a different module
In-Reply-To: <CH0PR10MB51001B61925FE6B6E1EC1EF8FA7A2@CH0PR10MB5100.namprd10.prod.outlook.com>
References: <CH0PR10MB51001B61925FE6B6E1EC1EF8FA7A2@CH0PR10MB5100.namprd10.prod.outlook.com>
Message-ID: <ce17e2bd-68ca-4e0c-a8c9-3e9f9e4ffc0b@oracle.com>

On 25/01/2024 12:34, Tomas Langer wrote:
> Is there a way to define a main class on a module if it comes from a 
> different one?
>
> Currently we need to start the JVM as follows:
> *java ?module-path ... --add-modules my.modular.application --module 
> io.helidon.microprofile.cdi/io.helidon.microprofile.cdi.Main*
>
> What we do:
>
>   * our microprofile implementation bootstraps a framework, that
>     discovers its component using service loader, it also contains the
>     Main class needed to start
>   * the "my.modular.application" provides such components (such as an
>     HTTP endpoint), but does not need a main class
>
>
> Ideally, I would like to run "my.modular.application" to have a 
> simpler command line.
> I cannot do that, as the main class is from a different module (and I 
> think that cannot be added to the module metadata).
> So I have to run the main class from the CDI module, and use 
> `--add-modules`, as otherwise my application module would not end up 
> on the module path (as it is not the "main" module started).
>
>
> Is there some way please to simplify this? Or do we need to create a 
> Main class for each application, even if it would just delegate to the 
> "cdi.Main"?
>

Is the application module discovered with service loader too? In that 
case, the --add-modules shouldn't be needed, instead just make it 
observable on the module path and service binding will ensure that it 
resolved.

-Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240125/6ad8cfd7/attachment-0001.htm>

From rfscholte at apache.org  Fri Jan 26 10:46:51 2024
From: rfscholte at apache.org (Robert Scholte)
Date: Fri, 26 Jan 2024 11:46:51 +0100
Subject: Custom Classloaders
Message-ID: <034201da5044$f893b980$e9bb2c80$@apache.org>

Hi,

 

I'm working on a new application, using jlink to build it.

One challenge is related to the simplelogger of SLF4J (as SLF4J-api is
probably the most used logging api, the simplelogger implementation matches
my requirements)

There are 2 things that I would like to adjust:

1.	Simplelogger uses the classloader to find the
simplelogger.properties (
https://github.com/qos-ch/slf4j/blob/master/slf4j-simple/src/main/java/org/s
lf4j/simple/SimpleLoggerConfiguration.java#L107-L130 ). 

However, I want to externalize this file to
${java.home}/conf/simplelogger.properties, so users can adjust it to their
preferences. If I could change the classloader I could control where to find
this file.

When bundling simplelogger with jlink the jar will end up in the
AppClassloader due to the SimpleServiceProvider. 

The only option to avoid this, is to move this jar to a separate directory
and when making a new ModuleLayer, and use ModuleFinder with this path. 

But still, I see no way to change the classloader for this jar.

 

2.	The simplelogger.properties is the default configuration, and I want
to make it possible to override these values. 

The simplelogger offers the option to use SystemProperties to override
values from simplelogger.properties.

I want to avoid that the System.properties will be polluted with entities
just for logging.

As far as I know it is not possible to scope System.properties: they are
global, available for every part of the application.

Another idea I had was to extend the simplelogging.properties resources at
runtime using the SequenceInputStream, and for this I also need to have
control over the classloader, as it requires to override the
ClassLoader.getResourceAsStream()

 

It seems both challenges could be solved if could control the classloader
(while still being protected by jdk.internal.loader.Loader).

 

Am I missing some feature or it there another way to solve these issues?

 

Best,

Robert 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240126/3bc16193/attachment.htm>

From Alan.Bateman at oracle.com  Fri Jan 26 11:12:13 2024
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Fri, 26 Jan 2024 11:12:13 +0000
Subject: Custom Classloaders
In-Reply-To: <034201da5044$f893b980$e9bb2c80$@apache.org>
References: <034201da5044$f893b980$e9bb2c80$@apache.org>
Message-ID: <a2e5fc51-6e3a-42db-9de0-9ced0b638b13@oracle.com>



On 26/01/2024 10:46, Robert Scholte wrote:
>
> Hi,
>
> I?m working on a new application, using jlink to build it.
>
> One challenge is related to the simplelogger of SLF4J (as SLF4J-api is 
> probably the most used logging api, the simplelogger implementation 
> matches my requirements)
>
> There are 2 things that I would like to adjust:
>
>  1. Simplelogger uses the classloader to find the
>     simplelogger.properties (
>     https://github.com/qos-ch/slf4j/blob/master/slf4j-simple/src/main/java/org/slf4j/simple/SimpleLoggerConfiguration.java#L107-L130
>     ).
>
> However, I want to externalize this file to 
> ${java.home}/conf/simplelogger.properties, so users can adjust it to 
> their preferences. If I could change the classloader I could control 
> where to find this file.
>
> When bundling simplelogger with jlink the jar will end up in the 
> AppClassloader due to the SimpleServiceProvider.
>
> The only option to avoid this, is to move this jar to a separate 
> directory and when making a new ModuleLayer, and use ModuleFinder with 
> this path.
>
> But still, I see no way to change the classloader for this jar.
>
>  2. The simplelogger.properties is the default configuration, and I
>     want to make it possible to override these values.
>
> The simplelogger offers the option to use SystemProperties to override 
> values from simplelogger.properties.
>
> I want to avoid that the System.properties will be polluted with 
> entities just for logging.
>
> As far as I know it is not possible to scope System.properties: they 
> are global, available for every part of the application.
>
> Another idea I had was to extend the simplelogging.properties 
> resources at runtime using the SequenceInputStream, and for this I 
> also need to have control over the classloader, as it requires to 
> override the ClassLoader.getResourceAsStream()
>
> It seems both challenges could be solved if could control the 
> classloader (while still being protected by jdk.internal.loader.Loader).
>
> Am I missing some feature or it there another way to solve these issues?
>
>
Hi Robert,

The code you linked to using the current Thread's context ClassLoader 
(CCL) to find the resource. So not the same thing as the class loader 
that defined SimpleLoggerConfiguration. At a guess, that code look like 
it is intended to support bundling of logging configuration with an 
application. It means code can set the current Thread's CCL, initialize 
SL4J to so it find the resource (config file) with that class loader, 
and reset it after.

To be editable in ${java.home}/conf would require 
changing?SimpleLoggerConfiguration.loadProperties to look for the 
properties file there. The defining classer for 
SimpleLoggerConfiguration won't change that as Class.getResource or 
ClassLoader.getResource doesn't find resources there.

-Alan




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240126/24e4329b/attachment-0001.htm>

From forax at univ-mlv.fr  Fri Jan 26 11:27:29 2024
From: forax at univ-mlv.fr (Remi Forax)
Date: Fri, 26 Jan 2024 12:27:29 +0100 (CET)
Subject: Custom Classloaders
In-Reply-To: <034201da5044$f893b980$e9bb2c80$@apache.org>
References: <034201da5044$f893b980$e9bb2c80$@apache.org>
Message-ID: <39023581.113292571.1706268449187.JavaMail.zimbra@univ-eiffel.fr>

> From: "Robert Scholte" <rfscholte at apache.org>
> To: "jigsaw-dev" <jigsaw-dev at openjdk.org>
> Sent: Friday, January 26, 2024 11:46:51 AM
> Subject: Custom Classloaders

> Hi,

Hello Robert, 
you were miss at last Devoxx.be, 

> I?m working on a new application, using jlink to build it.

> One challenge is related to the simplelogger of SLF4J (as SLF4J-api is probably
> the most used logging api, the simplelogger implementation matches my
> requirements)

> There are 2 things that I would like to adjust:

>    1. Simplelogger uses the classloader to find the simplelogger.properties ( [
>    https://github.com/qos-ch/slf4j/blob/master/slf4j-simple/src/main/java/org/slf4j/simple/SimpleLoggerConfiguration.java#L107-L130
>    |
>    https://github.com/qos-ch/slf4j/blob/master/slf4j-simple/src/main/java/org/slf4j/simple/SimpleLoggerConfiguration.java#L107-L130
>     ] ).

> However, I want to externalize this file to
> ${java.home}/conf/simplelogger.properties, so users can adjust it to their
> preferences. If I could change the classloader I could control where to find
> this file.

> When bundling simplelogger with jlink the jar will end up in the AppClassloader
> due to the SimpleServiceProvider.

> The only option to avoid this, is to move this jar to a separate directory and
> when making a new ModuleLayer, and use ModuleFinder with this path.

> But still, I see no way to change the classloader for this jar.

>    1. The simplelogger.properties is the default configuration, and I want to make
>     it possible to override these values.

> The simplelogger offers the option to use SystemProperties to override values
> from simplelogger.properties.

> I want to avoid that the System.properties will be polluted with entities just
> for logging.

> As far as I know it is not possible to scope System.properties: they are global,
> available for every part of the application.

> Another idea I had was to extend the simplelogging.properties resources at
> runtime using the SequenceInputStream, and for this I also need to have control
> over the classloader, as it requires to override the
> ClassLoader.getResourceAsStream()

> It seems both challenges could be solved if could control the classloader (while
> still being protected by jdk.internal.loader.Loader).

> Am I missing some feature or it there another way to solve these issues?

yes, 
as you said yourself, you box yourself in a corner by wanting to use Simplelogger which does not provide a great deal of ways to configure it. 
I'm sure they are other loggers that, while heavier, are more capable in term of configuration. 

You are cursed by your own knowledge in this case, as an application developer, you should pretend that ClassLoaders do not exist, like the rest of the application developers do. 

Let me try a trick, those are not the classloaders you are looking for :) 

> Best,

> Robert

regards, 
R?mi 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240126/2d62afe9/attachment.htm>

From rfscholte at apache.org  Sat Jan 27 11:13:58 2024
From: rfscholte at apache.org (Robert Scholte)
Date: Sat, 27 Jan 2024 12:13:58 +0100
Subject: Custom Classloaders
In-Reply-To: <39023581.113292571.1706268449187.JavaMail.zimbra@univ-eiffel.fr>
References: <034201da5044$f893b980$e9bb2c80$@apache.org>
 <39023581.113292571.1706268449187.JavaMail.zimbra@univ-eiffel.fr>
Message-ID: <004c01da5111$ec9e1d50$c5da57f0$@apache.org>

Hi,

 

With both the answers from Alan and R?mi it is clear to me that the implementation of loading configuration in the simplelogger doesn?t fit my requirements.

It is called ?simple? for a reason.

I prefer to depend on existing SLF4JProviders, but based on possible extra requirements it might be worth writing my own provider.

 

Thanks,

Robert

 

  _____  

From: "Robert Scholte" <rfscholte at apache.org <mailto:rfscholte at apache.org> >
To: "jigsaw-dev" <jigsaw-dev at openjdk.org <mailto:jigsaw-dev at openjdk.org> >
Sent: Friday, January 26, 2024 11:46:51 AM
Subject: Custom Classloaders

Hi,

 

Hello Robert,

you were miss at last Devoxx.be,

 

 

I?m working on a new application, using jlink to build it.

One challenge is related to the simplelogger of SLF4J (as SLF4J-api is probably the most used logging api, the simplelogger implementation matches my requirements)

There are 2 things that I would like to adjust:

1.       Simplelogger uses the classloader to find the simplelogger.properties ( https://github.com/qos-ch/slf4j/blob/master/slf4j-simple/src/main/java/org/slf4j/simple/SimpleLoggerConfiguration.java#L107-L130 ). 

However, I want to externalize this file to ${java.home}/conf/simplelogger.properties, so users can adjust it to their preferences. If I could change the classloader I could control where to find this file.

When bundling simplelogger with jlink the jar will end up in the AppClassloader due to the SimpleServiceProvider. 

The only option to avoid this, is to move this jar to a separate directory and when making a new ModuleLayer, and use ModuleFinder with this path. 

But still, I see no way to change the classloader for this jar.

2.       The simplelogger.properties is the default configuration, and I want to make it possible to override these values. 

The simplelogger offers the option to use SystemProperties to override values from simplelogger.properties.

I want to avoid that the System.properties will be polluted with entities just for logging.

As far as I know it is not possible to scope System.properties: they are global, available for every part of the application.

Another idea I had was to extend the simplelogging.properties resources at runtime using the SequenceInputStream, and for this I also need to have control over the classloader, as it requires to override the ClassLoader.getResourceAsStream()

It seems both challenges could be solved if could control the classloader (while still being protected by jdk.internal.loader.Loader).

Am I missing some feature or it there another way to solve these issues?

 

yes, 

as you said yourself, you box yourself in a corner by wanting to use Simplelogger which does not provide a great deal of ways to configure it.

I'm sure they are other loggers that, while heavier, are more capable in term of configuration.

 

You are cursed by your own knowledge in this case, as an application developer, you should pretend that ClassLoaders do not exist, like the rest of the application developers do.

 

Let me try a trick, those are not the classloaders you are looking for :)

 

Best,

Robert 

 

regards,

R?mi

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240127/26caeb53/attachment-0001.htm>

From simone.bordet at gmail.com  Sat Jan 27 13:46:16 2024
From: simone.bordet at gmail.com (Simone Bordet)
Date: Sat, 27 Jan 2024 14:46:16 +0100
Subject: Custom Classloaders
In-Reply-To: <004c01da5111$ec9e1d50$c5da57f0$@apache.org>
References: <034201da5044$f893b980$e9bb2c80$@apache.org>
 <39023581.113292571.1706268449187.JavaMail.zimbra@univ-eiffel.fr>
 <004c01da5111$ec9e1d50$c5da57f0$@apache.org>
Message-ID: <CAFWmRJ0wQdZ=MkS3_k-GkMemy-fNfKx1-zvUws9c8=b2T9v1Pw@mail.gmail.com>

Robert,

On Sat, Jan 27, 2024 at 12:14?PM Robert Scholte <rfscholte at apache.org> wrote:
>
> Hi,
>
>
>
> With both the answers from Alan and R?mi it is clear to me that the implementation of loading configuration in the simplelogger doesn?t fit my requirements.
>
> It is called ?simple? for a reason.
>
> I prefer to depend on existing SLF4JProviders, but based on possible extra requirements it might be worth writing my own provider.

Just to say that Jetty has written one, and it's a proper JPMS module:
https://central.sonatype.com/artifact/org.eclipse.jetty/jetty-slf4j-impl

Not sure if it meets 100% your requirements, but we do support system
property log category configuration, etc.

Let us know if we can improve it :)
-- 
Simone Bordet
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz

From alex.buckley at oracle.com  Tue Jan 30 19:05:01 2024
From: alex.buckley at oracle.com (Alex Buckley)
Date: Tue, 30 Jan 2024 11:05:01 -0800
Subject: Java extensibility and JPMS (ServiceLoader)
In-Reply-To: <CH0PR10MB51006744C4F05C3C19FCBB06FA702@CH0PR10MB5100.namprd10.prod.outlook.com>
References: <CH0PR10MB51006744C4F05C3C19FCBB06FA702@CH0PR10MB5100.namprd10.prod.outlook.com>
Message-ID: <432557a1-299d-4c97-ad66-b3bfa4560cdd@oracle.com>

On 1/19/2024 5:02 AM, Tomas Langer wrote:
> Helidon currently has around 300 modules with module-info.java. In 
> general, this has improved our module structure and design.
> Yet, we are now encountering some major issues related to extensibility.
>  ?I will put down a few points that are problematic, and explain each in 
> detail further in the e-mail (it is quite long, sorry about that).
> 
> 1. provider implementations cannot be code generated without major problems

Annotation processing is designed to avoid mutating the elements, so it 
would be a fundamental change to allow mutation of module elements in 
the annotation processing API. It's on the framework to post-process 
module-info.class so it has `provides` clauses. The framework can use 
the ClassFile API to do this.

It's not "weird" for a framework to modify module-info.class to ensure 
that code in the module has the right execution environment. Another 
example would be a tool that injects calls to a logging API into user 
code, then has to post-process module-info.class to add `requires 
logging.lib;`.

> 2. the provider interface module MUST be on module path, even if it 
> could have `requires static`

Relaxing module resolution to allow a module to `provides` an interface 
that it can't access is do-able, but the implications are unknown. It 
would be unfortunate if resolution succeeded but then unforeseen 
exceptions occur when faraway code tries to access the missing 
interface. We'll leave JDK-8299504 open for now, but we would need 
evidence that a large number of users are finding the current rules 
unworkable before actively looking at `provides` again.

> 3. the provider implementation must be public with public constructor
> 4. duality of definition between module path and class path

These requests for ServiceLoader to (i) support package-private 
providers and (ii) inspect module-info.class files for `provides` 
clauses in JARs on the classpath both stem from insisting that modular 
JARs can be deployed on the classpath without losing functionality. This 
is a net-new requirement on the module system. It's relatively low risk, 
but also low reward, so we aren't going to investigate it further.

Alex

From tomas.langer at oracle.com  Tue Jan 30 19:39:23 2024
From: tomas.langer at oracle.com (Tomas Langer)
Date: Tue, 30 Jan 2024 19:39:23 +0000
Subject: Java extensibility and JPMS (ServiceLoader)
In-Reply-To: <432557a1-299d-4c97-ad66-b3bfa4560cdd@oracle.com>
References: <CH0PR10MB51006744C4F05C3C19FCBB06FA702@CH0PR10MB5100.namprd10.prod.outlook.com>
 <432557a1-299d-4c97-ad66-b3bfa4560cdd@oracle.com>
Message-ID: <BN0PR10MB5095CBCE766E5E3924BE621CFA7D2@BN0PR10MB5095.namprd10.prod.outlook.com>

Ad 1 (provider implementations cannot be code generated without major problems)
I understand that annotation processing should not mutate elements, and I never suggested it should be done. I am looking for a solution provided by the language (whatever that may be).

I am not sure if you are aware of the complexity of the proposed "It's on the framework to post-process
module-info.class so it has `provides` clauses"

Considering that we currently have multiple build frameworks, and for each of those the postprocessing must be done differently, we are also opening the possibility of the user not configuring the postprocessing to happen at all. Just the first part (Maven plugins, Gradle plugins, possible other systems) is a major maintenance issue.
In addition, the compiled code deployed with the application differs from its source code, making troubleshooting more complicated.

I understand it is easy to dismiss these problems and let us (as framework owners) handle it. You are in the end forcing us to do it differently for each framework - which in the end is a problem for our users.

The result that other frameworks reached (and I was trying not to do), and we will switch to as well, is to design our own extensibility approach that is based on resources rather than on module-info.java. This makes it once again harder for users, as they cannot use the tools the language provides.

Pushing us to post-processing the bytecode: if there was an event in the compiler with extensibility similar to annotation processors (i.e. ClassPostProcessor) using the ClassFile API, I can imagine this being done, as it is still part of a single compilation run (but as far as I know, there is no such possibility).

Ad 2 (requires static on provider module)
As far as I have seen the analysis of current Maven repository, Helidon is the only bigger framework that has embraced modularization.
So "a large number of users" will not find the current rules unworkable, as there is not "a large number of users" using JPMS.
As one of the users of it on a larger scale, I can tell you this is a major restriction for using JPMS together with ServiceLoader (and as mentioned above, we will stop using ServiceLoader because of this).

Ad 3 (public provider)
I understand this is not a critical feature

Ad 4 (duality between module path and classpath)
You have mentioned "It's relatively low risk, but also low reward, so we aren't going to investigate it further."
Once again I am not sure whose "low reward" you have in mind. I understand that in the language and for language developers this may not mean much. But for the users, frameworks etc. this is quite a big thing - we are now required to do another post-processing of our libraries to ensure we have both (module-info provides and META-INF/services).
The reward is that new libraries can work both on classpath and on module path. And considering how many people use JPMS, this is quite a critical behavior for anything we create.

Tomas

________________________________
Od: jigsaw-dev <jigsaw-dev-retn at openjdk.org> za u?ivatele Alex Buckley <alex.buckley at oracle.com>
Odesl?no: ?ter? 30. ledna 2024 20:05
Komu: jigsaw-dev <jigsaw-dev at openjdk.java.net>
P?edm?t: Re: Java extensibility and JPMS (ServiceLoader)

On 1/19/2024 5:02 AM, Tomas Langer wrote:
> Helidon currently has around 300 modules with module-info.java. In
> general, this has improved our module structure and design.
> Yet, we are now encountering some major issues related to extensibility.
>   I will put down a few points that are problematic, and explain each in
> detail further in the e-mail (it is quite long, sorry about that).
>
> 1. provider implementations cannot be code generated without major problems

Annotation processing is designed to avoid mutating the elements, so it
would be a fundamental change to allow mutation of module elements in
the annotation processing API. It's on the framework to post-process
module-info.class so it has `provides` clauses. The framework can use
the ClassFile API to do this.

It's not "weird" for a framework to modify module-info.class to ensure
that code in the module has the right execution environment. Another
example would be a tool that injects calls to a logging API into user
code, then has to post-process module-info.class to add `requires
logging.lib;`.

> 2. the provider interface module MUST be on module path, even if it
> could have `requires static`

Relaxing module resolution to allow a module to `provides` an interface
that it can't access is do-able, but the implications are unknown. It
would be unfortunate if resolution succeeded but then unforeseen
exceptions occur when faraway code tries to access the missing
interface. We'll leave JDK-8299504 open for now, but we would need
evidence that a large number of users are finding the current rules
unworkable before actively looking at `provides` again.

> 3. the provider implementation must be public with public constructor
> 4. duality of definition between module path and class path

These requests for ServiceLoader to (i) support package-private
providers and (ii) inspect module-info.class files for `provides`
clauses in JARs on the classpath both stem from insisting that modular
JARs can be deployed on the classpath without losing functionality. This
is a net-new requirement on the module system. It's relatively low risk,
but also low reward, so we aren't going to investigate it further.

Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240130/549a17b4/attachment-0001.htm>

From josiahnoel at gmail.com  Wed Jan 31 01:25:48 2024
From: josiahnoel at gmail.com (Josiah Noel)
Date: Tue, 30 Jan 2024 20:25:48 -0500
Subject: Java extensibility and JPMS (ServiceLoader)
In-Reply-To: <BN0PR10MB5095CBCE766E5E3924BE621CFA7D2@BN0PR10MB5095.namprd10.prod.outlook.com>
References: <CH0PR10MB51006744C4F05C3C19FCBB06FA702@CH0PR10MB5100.namprd10.prod.outlook.com>
 <432557a1-299d-4c97-ad66-b3bfa4560cdd@oracle.com>
 <BN0PR10MB5095CBCE766E5E3924BE621CFA7D2@BN0PR10MB5095.namprd10.prod.outlook.com>
Message-ID: <CAJ_t5UYbE6yScVv_WGWwcU+FqG13MfG-3hOawDnO=vn-FjceJw@mail.gmail.com>

>
> Annotation processing is designed to avoid mutating the elements, so it
> would be a fundamental change to allow mutation of module elements


We're well aware that processors shouldn't modify code, we're talking about
adding some extension to the service loader that we *can *generate. We
simply want to regain the ability to register services at compile-time.

It's on the framework to post-process
> module-info.class so it has `provides` clauses. The framework can use
> the ClassFile API to do this.
>

As Tomas has stated, there are a myriad of build systems so postprocessors
are quite a maintenance issue.


The result that other frameworks reached (and I was trying not to do),  is
> to design our own extensibility approach that is based on resources


I wonder what are these other frameworks that have gone this route?


On Tue, Jan 30, 2024 at 2:39?PM Tomas Langer <tomas.langer at oracle.com>
wrote:

> Ad 1 (provider implementations cannot be code generated without major
> problems)
> I understand that annotation processing should not mutate elements, and I
> never suggested it should be done. I am looking for a solution provided by
> the language (whatever that may be).
>
> I am not sure if you are aware of the complexity of the proposed "It's on
> the framework to post-process
> module-info.class so it has `provides` clauses"
>
> Considering that we currently have multiple build frameworks, and for each
> of those the postprocessing must be done differently, we are also opening
> the possibility of the user not configuring the postprocessing to happen at
> all. Just the first part (Maven plugins, Gradle plugins, possible other
> systems) is a major maintenance issue.
> In addition, the compiled code deployed with the application differs from
> its source code, making troubleshooting more complicated.
>
> I understand it is easy to dismiss these problems and let us (as framework
> owners) handle it. You are in the end forcing us to do it differently for
> each framework - which in the end is a problem for our users.
>
> The result that other frameworks reached (and I was trying not to do), and
> we will switch to as well, is to design our own extensibility approach that
> is based on resources rather than on module-info.java. This makes it once
> again harder for users, as they cannot use the tools the language provides.
>
> Pushing us to post-processing the bytecode: if there was an event in the
> compiler with extensibility similar to annotation processors (i.e.
> ClassPostProcessor) using the ClassFile API, I can imagine this being done,
> as it is still part of a single compilation run (but as far as I know,
> there is no such possibility).
>
> Ad 2 (requires static on provider module)
> As far as I have seen the analysis of current Maven repository, Helidon is
> the only bigger framework that has embraced modularization.
> So "a large number of users" will not find the current rules unworkable,
> as there is not "a large number of users" using JPMS.
> As one of the users of it on a larger scale, I can tell you this is a
> major restriction for using JPMS together with ServiceLoader (and as
> mentioned above, we will stop using ServiceLoader because of this).
>
> Ad 3 (public provider)
> I understand this is not a critical feature
>
> Ad 4 (duality between module path and classpath)
> You have mentioned "It's relatively low risk, but also low reward, so we
> aren't going to investigate it further."
> Once again I am not sure whose "low reward" you have in mind. I understand
> that in the language and for language developers this may not mean much.
> But for the users, frameworks etc. this is quite a big thing - we are now
> required to do another post-processing of our libraries to ensure we have
> both (module-info provides and META-INF/services).
> The reward is that new libraries can work both on classpath and on module
> path. And considering how many people use JPMS, this is quite a critical
> behavior for anything we create.
>
> Tomas
>
> ------------------------------
> *Od:* jigsaw-dev <jigsaw-dev-retn at openjdk.org> za u?ivatele Alex Buckley <
> alex.buckley at oracle.com>
> *Odesl?no:* ?ter? 30. ledna 2024 20:05
> *Komu:* jigsaw-dev <jigsaw-dev at openjdk.java.net>
> *P?edm?t:* Re: Java extensibility and JPMS (ServiceLoader)
>
> On 1/19/2024 5:02 AM, Tomas Langer wrote:
> > Helidon currently has around 300 modules with module-info.java. In
> > general, this has improved our module structure and design.
> > Yet, we are now encountering some major issues related to extensibility.
> >   I will put down a few points that are problematic, and explain each in
> > detail further in the e-mail (it is quite long, sorry about that).
> >
> > 1. provider implementations cannot be code generated without major
> problems
>
> Annotation processing is designed to avoid mutating the elements, so it
> would be a fundamental change to allow mutation of module elements in
> the annotation processing API. It's on the framework to post-process
> module-info.class so it has `provides` clauses. The framework can use
> the ClassFile API to do this.
>
> It's not "weird" for a framework to modify module-info.class to ensure
> that code in the module has the right execution environment. Another
> example would be a tool that injects calls to a logging API into user
> code, then has to post-process module-info.class to add `requires
> logging.lib;`.
>
> > 2. the provider interface module MUST be on module path, even if it
> > could have `requires static`
>
> Relaxing module resolution to allow a module to `provides` an interface
> that it can't access is do-able, but the implications are unknown. It
> would be unfortunate if resolution succeeded but then unforeseen
> exceptions occur when faraway code tries to access the missing
> interface. We'll leave JDK-8299504 open for now, but we would need
> evidence that a large number of users are finding the current rules
> unworkable before actively looking at `provides` again.
>
> > 3. the provider implementation must be public with public constructor
> > 4. duality of definition between module path and class path
>
> These requests for ServiceLoader to (i) support package-private
> providers and (ii) inspect module-info.class files for `provides`
> clauses in JARs on the classpath both stem from insisting that modular
> JARs can be deployed on the classpath without losing functionality. This
> is a net-new requirement on the module system. It's relatively low risk,
> but also low reward, so we aren't going to investigate it further.
>
> Alex
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240130/7e6f80b5/attachment.htm>

From piotr.karwasz at gmail.com  Wed Jan 31 07:47:30 2024
From: piotr.karwasz at gmail.com (Piotr P. Karwasz)
Date: Wed, 31 Jan 2024 08:47:30 +0100
Subject: Are modifiers `static` and `transitive` incompatible?
Message-ID: <CAFmygFYHgOFszk0nVZW1ctPJwf3XdgOAX=W6W+SchHKnq4-VGQ@mail.gmail.com>

Hello,

The `requires` instruction can have two qualifiers `static` and
`transitive` that separately make perfect sense. For a module `foo`:

 * `requires static bar;`, makes resolution of the `bar` module
optional at runtime,
 * `requires transitive bar;`, allows each module that reads `foo` to
also read `bar`. As far as I understand this means that `bar` must be
present at both compile time and runtime.

There is however a third option allowed:

```java
module foo {
    requires static transitive bar;
}
```

What is this combination supposed to do? Why is it allowed?

According to the Javadoc:

> `requires` directives that have the `static` modifier express an optional dependence at run time. If a module declares that it `requires static M` then resolution does not search the observable modules for M to satisfy the dependency. However, if M is recursively enumerated at step 1 then all modules that are enumerated and `requires static M` will read M.

where "step 1" describes the computation of the closure of the set of
root modules by the `X requires transitive Y` relation.

If I interpret this correctly, if a `requires` directive is both
`static` and `transitive`, the `foo` module is **not optional**, since
"step 1" requires its presence.

We often end up with such directives, when using the
`bnd-maven-plugin`, which maps the OSGi `resolution:=optional`
directive to `static` and the `uses:=...` directive to `transitive`.
If we mark a package as optional, but the package uses public types
from `bar`, we end up with `static transitive`.

Is this a bug of the plugin or does `static transitive` have a meaning?

Piotr

[1] https://github.com/bndtools/bnd/tree/master/maven-plugins/bnd-maven-plugin

From tomas.langer at oracle.com  Wed Jan 31 12:02:46 2024
From: tomas.langer at oracle.com (Tomas Langer)
Date: Wed, 31 Jan 2024 12:02:46 +0000
Subject: [External] : Re: Java extensibility and JPMS (ServiceLoader)
In-Reply-To: <CAJ_t5UYbE6yScVv_WGWwcU+FqG13MfG-3hOawDnO=vn-FjceJw@mail.gmail.com>
References: <CH0PR10MB51006744C4F05C3C19FCBB06FA702@CH0PR10MB5100.namprd10.prod.outlook.com>
 <432557a1-299d-4c97-ad66-b3bfa4560cdd@oracle.com>
 <BN0PR10MB5095CBCE766E5E3924BE621CFA7D2@BN0PR10MB5095.namprd10.prod.outlook.com>
 <CAJ_t5UYbE6yScVv_WGWwcU+FqG13MfG-3hOawDnO=vn-FjceJw@mail.gmail.com>
Message-ID: <CH0PR10MB5100FC90F38739C4D9009ECEFA7C2@CH0PR10MB5100.namprd10.prod.outlook.com>

Other frameworks I am aware of that have custom service loading:

  *   hk2
  *   WebLogic
  *   Jersey
  *   Micronaut

Tomas

________________________________
Od: Josiah Noel <josiahnoel at gmail.com>
Odesl?no: st?eda 31. ledna 2024 2:25
Komu: Tomas Langer <tomas.langer at oracle.com>
Kopie: Alex Buckley <alex.buckley at oracle.com>; jigsaw-dev <jigsaw-dev at openjdk.java.net>
P?edm?t: [External] : Re: Java extensibility and JPMS (ServiceLoader)

Annotation processing is designed to avoid mutating the elements, so it
would be a fundamental change to allow mutation of module elements

We're well aware that processors shouldn't modify code, we're talking about adding some extension to the service loader that we can generate. We simply want to regain the ability to register services at compile-time.

It's on the framework to post-process
module-info.class so it has `provides` clauses. The framework can use
the ClassFile API to do this.

As Tomas has stated, there are a myriad of build systems so postprocessors are quite a maintenance issue.


The result that other frameworks reached (and I was trying not to do),  is to design our own extensibility approach that is based on resources

I wonder what are these other frameworks that have gone this route?


On Tue, Jan 30, 2024 at 2:39?PM Tomas Langer <tomas.langer at oracle.com<mailto:tomas.langer at oracle.com>> wrote:
Ad 1 (provider implementations cannot be code generated without major problems)
I understand that annotation processing should not mutate elements, and I never suggested it should be done. I am looking for a solution provided by the language (whatever that may be).

I am not sure if you are aware of the complexity of the proposed "It's on the framework to post-process
module-info.class so it has `provides` clauses"

Considering that we currently have multiple build frameworks, and for each of those the postprocessing must be done differently, we are also opening the possibility of the user not configuring the postprocessing to happen at all. Just the first part (Maven plugins, Gradle plugins, possible other systems) is a major maintenance issue.
In addition, the compiled code deployed with the application differs from its source code, making troubleshooting more complicated.

I understand it is easy to dismiss these problems and let us (as framework owners) handle it. You are in the end forcing us to do it differently for each framework - which in the end is a problem for our users.

The result that other frameworks reached (and I was trying not to do), and we will switch to as well, is to design our own extensibility approach that is based on resources rather than on module-info.java. This makes it once again harder for users, as they cannot use the tools the language provides.

Pushing us to post-processing the bytecode: if there was an event in the compiler with extensibility similar to annotation processors (i.e. ClassPostProcessor) using the ClassFile API, I can imagine this being done, as it is still part of a single compilation run (but as far as I know, there is no such possibility).

Ad 2 (requires static on provider module)
As far as I have seen the analysis of current Maven repository, Helidon is the only bigger framework that has embraced modularization.
So "a large number of users" will not find the current rules unworkable, as there is not "a large number of users" using JPMS.
As one of the users of it on a larger scale, I can tell you this is a major restriction for using JPMS together with ServiceLoader (and as mentioned above, we will stop using ServiceLoader because of this).

Ad 3 (public provider)
I understand this is not a critical feature

Ad 4 (duality between module path and classpath)
You have mentioned "It's relatively low risk, but also low reward, so we aren't going to investigate it further."
Once again I am not sure whose "low reward" you have in mind. I understand that in the language and for language developers this may not mean much. But for the users, frameworks etc. this is quite a big thing - we are now required to do another post-processing of our libraries to ensure we have both (module-info provides and META-INF/services).
The reward is that new libraries can work both on classpath and on module path. And considering how many people use JPMS, this is quite a critical behavior for anything we create.

Tomas

________________________________
Od: jigsaw-dev <jigsaw-dev-retn at openjdk.org<mailto:jigsaw-dev-retn at openjdk.org>> za u?ivatele Alex Buckley <alex.buckley at oracle.com<mailto:alex.buckley at oracle.com>>
Odesl?no: ?ter? 30. ledna 2024 20:05
Komu: jigsaw-dev <jigsaw-dev at openjdk.java.net<mailto:jigsaw-dev at openjdk.java.net>>
P?edm?t: Re: Java extensibility and JPMS (ServiceLoader)

On 1/19/2024 5:02 AM, Tomas Langer wrote:
> Helidon currently has around 300 modules with module-info.java. In
> general, this has improved our module structure and design.
> Yet, we are now encountering some major issues related to extensibility.
>   I will put down a few points that are problematic, and explain each in
> detail further in the e-mail (it is quite long, sorry about that).
>
> 1. provider implementations cannot be code generated without major problems

Annotation processing is designed to avoid mutating the elements, so it
would be a fundamental change to allow mutation of module elements in
the annotation processing API. It's on the framework to post-process
module-info.class so it has `provides` clauses. The framework can use
the ClassFile API to do this.

It's not "weird" for a framework to modify module-info.class to ensure
that code in the module has the right execution environment. Another
example would be a tool that injects calls to a logging API into user
code, then has to post-process module-info.class to add `requires
logging.lib;`.

> 2. the provider interface module MUST be on module path, even if it
> could have `requires static`

Relaxing module resolution to allow a module to `provides` an interface
that it can't access is do-able, but the implications are unknown. It
would be unfortunate if resolution succeeded but then unforeseen
exceptions occur when faraway code tries to access the missing
interface. We'll leave JDK-8299504 open for now, but we would need
evidence that a large number of users are finding the current rules
unworkable before actively looking at `provides` again.

> 3. the provider implementation must be public with public constructor
> 4. duality of definition between module path and class path

These requests for ServiceLoader to (i) support package-private
providers and (ii) inspect module-info.class files for `provides`
clauses in JARs on the classpath both stem from insisting that modular
JARs can be deployed on the classpath without losing functionality. This
is a net-new requirement on the module system. It's relatively low risk,
but also low reward, so we aren't going to investigate it further.

Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240131/515176f0/attachment-0001.htm>

From Alan.Bateman at oracle.com  Wed Jan 31 15:41:29 2024
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Wed, 31 Jan 2024 15:41:29 +0000
Subject: Are modifiers `static` and `transitive` incompatible?
In-Reply-To: <CAFmygFYHgOFszk0nVZW1ctPJwf3XdgOAX=W6W+SchHKnq4-VGQ@mail.gmail.com>
References: <CAFmygFYHgOFszk0nVZW1ctPJwf3XdgOAX=W6W+SchHKnq4-VGQ@mail.gmail.com>
Message-ID: <19ce7383-1bcc-438e-b7f2-11c45f4882eb@oracle.com>



On 31/01/2024 07:47, Piotr P. Karwasz wrote:
> Hello,
>
> The `requires` instruction can have two qualifiers `static` and
> `transitive` that separately make perfect sense. For a module `foo`:
>
>   * `requires static bar;`, makes resolution of the `bar` module
> optional at runtime,
>   * `requires transitive bar;`, allows each module that reads `foo` to
> also read `bar`. As far as I understand this means that `bar` must be
> present at both compile time and runtime.
>
> There is however a third option allowed:
>
> ```java
> module foo {
>      requires static transitive bar;
> }
> ```
It may be surprising but they do work together. If bar is resolved then 
foo will read bar. If there is another module baz that requires foo (and 
doesn't explicitly require bar) then baz will read foo, and if bar is 
resolved, then baz will read bar.


> :
> We often end up with such directives, when using the
> `bnd-maven-plugin`, which maps the OSGi `resolution:=optional`
> directive to `static` and the `uses:=...` directive to `transitive`.
> If we mark a package as optional, but the package uses public types
> from `bar`, we end up with `static transitive`.
>
> Is this a bug of the plugin or does `static transitive` have a meaning?
>
I think it's more of a case that it is fragile for modules like foo to 
have bar types foo API signatures.? `requires static` is okay for things 
like annotations with a retention policy of "source" but going beyond 
that means giving up a bit on reliable configuration. In the example, 
foo probably has a reflection guard and is careful in their use of bar 
types. However, foo's user if baz and the maintainers of baz are unaware 
of these shenanigans. This creates a risk that baz code will get a CNFE 
at runtime.

-Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240131/16653554/attachment.htm>