From thihup at gmail.com Sat Jan 2 03:21:55 2021 From: thihup at gmail.com (Thiago Henrique Hupner) Date: Sat, 2 Jan 2021 00:21:55 -0300 Subject: Should ClassLoader::getResouces return the same resource twice? Message-ID: Hi! I'm working to integrate the JPMS into our application server in a way that the applications can run in a layer each, but I faced something strange: our current behavior without layers, calling ClassLoader::getResources in the classloader provided by us, it will return a return "foo". In the conversion to layer, it was clear to me that I would need to use the ModuleLayer::defineModulesWithOneLoader, so the application classloading behavior wouldn't change and the ModuleReader would use the same source of resources that the classloader is using. However, calling the ClassLoader::getResources in the classloader provided by the Module system, it is finding the resource in itself and on the parent, so it finds the same resource twice. As our applications aren't expected to work with the same resource twice, it broke in some places where we cannot change. I've created a simple example of what is occurring [1]. I know there are behavior specific for getting a class if it is in a module, but I don't know if this may be a bug in the resource loading mechanism. In the example, the returned values are different to illustrate, but in my case, it returns two exact URLs for the same resource as the source for the module reader and the classloader is the same. [1] https://gist.github.com/Thihup/8e53bc991272452e63a6055f05690ba5 From Alan.Bateman at oracle.com Sat Jan 2 07:07:15 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 2 Jan 2021 07:07:15 +0000 Subject: Should ClassLoader::getResouces return the same resource twice? In-Reply-To: References: Message-ID: On 02/01/2021 03:21, Thiago Henrique Hupner wrote: > : > > I've created a simple example of what is occurring [1]. > I know there are behavior specific for getting a class if it is in a module, > but I don't know if this may be a bug in the resource loading mechanism. > In the example, the returned values are different to illustrate, but in my > case, it > returns two exact URLs for the same resource as the source for the module > reader > and the classloader is the same. The behavior you observe with the example is correct. ClassLoader.getResources locates the resource by searching the lass loader delegation chain and in the example there are two "foo" resources. The resource in module fake.module is located because it is an automatic module that opens all its packages unconditionally. The second resource is located by searching the parent class loader, a URLClassLoader in the example that also locates "foo".? Which "foo" did you expect to locate? If code in fake.module just wants to locate the resource in its own module then it should use getResource rather than getResources. -Alan. From thihup at gmail.com Sat Jan 2 12:59:19 2021 From: thihup at gmail.com (Thiago Henrique Hupner) Date: Sat, 2 Jan 2021 09:59:19 -0300 Subject: Should ClassLoader::getResouces return the same resource twice? In-Reply-To: References: Message-ID: I guess a little context can make more things clear: The servlet spec requires that all jars from WEB-INF/lib be available to the same classloader. The resource, in particular, is "META-INF/web-fragment.xml" Each jar can contain its own. So, using getResources make sense in order of parsing each. However, what is happening is if I have two JARs each with its own META-INF/web-fragment.xml, using the ModuleReader it is returning four resources, so it parses more than it should and it fails to parse the same resource twice. I'll have a try only exposing the ".class" files in the ModuleReader, so the Loader will be able to create the classes and it will read the resources from the classloader. I'm using my own implementations of the ModuleReader, ModuleFinder and ModuleDescriptor because all of our resources are in memory I wasn't able to use the ModuleFinder.of() because it requires a filesystem. Em s?b., 2 de jan. de 2021 ?s 04:07, Alan Bateman escreveu: > On 02/01/2021 03:21, Thiago Henrique Hupner wrote: > > : > > > > I've created a simple example of what is occurring [1]. > > I know there are behavior specific for getting a class if it is in a > module, > > but I don't know if this may be a bug in the resource loading mechanism. > > In the example, the returned values are different to illustrate, but in > my > > case, it > > returns two exact URLs for the same resource as the source for the module > > reader > > and the classloader is the same. > The behavior you observe with the example is correct. > ClassLoader.getResources locates the resource by searching the lass > loader delegation chain and in the example there are two "foo" > resources. The resource in module fake.module is located because it is > an automatic module that opens all its packages unconditionally. The > second resource is located by searching the parent class loader, a > URLClassLoader in the example that also locates "foo". Which "foo" did > you expect to locate? If code in fake.module just wants to locate the > resource in its own module then it should use getResource rather than > getResources. > > -Alan. > From james.laskey at oracle.com Sat Jan 2 13:09:01 2021 From: james.laskey at oracle.com (James Laskey) Date: Sat, 2 Jan 2021 09:09:01 -0400 Subject: Should ClassLoader::getResouces return the same resource twice? In-Reply-To: References: Message-ID: As an aside, creating a FileSystem is not an onerous task. https://stackoverflow.com/questions/22966176/creating-a-custom-filesystem-implementation-in-java Cheers, ? Jim ?? > On Jan 2, 2021, at 8:59 AM, Thiago Henrique Hupner wrote: > > ?I guess a little context can make more things clear: > The servlet spec requires that all jars from WEB-INF/lib > be available to the same classloader. > The resource, in particular, is "META-INF/web-fragment.xml" > Each jar can contain its own. So, using getResources make sense > in order of parsing each. However, what is happening is if I have two JARs > each with its own META-INF/web-fragment.xml, using the ModuleReader > it is returning four resources, so it parses more than it should and it > fails > to parse the same resource twice. > > I'll have a try only exposing the ".class" files in the ModuleReader, > so the Loader will be able to create the classes and it will read the > resources > from the classloader. > > I'm using my own implementations of the ModuleReader, ModuleFinder > and ModuleDescriptor because all of our resources are in memory I wasn't > able to > use the ModuleFinder.of() because it requires a filesystem. > >> Em s?b., 2 de jan. de 2021 ?s 04:07, Alan Bateman >> escreveu: >> >>> On 02/01/2021 03:21, Thiago Henrique Hupner wrote: >>> : >>> >>> I've created a simple example of what is occurring [1]. >>> I know there are behavior specific for getting a class if it is in a >> module, >>> but I don't know if this may be a bug in the resource loading mechanism. >>> In the example, the returned values are different to illustrate, but in >> my >>> case, it >>> returns two exact URLs for the same resource as the source for the module >>> reader >>> and the classloader is the same. >> The behavior you observe with the example is correct. >> ClassLoader.getResources locates the resource by searching the lass >> loader delegation chain and in the example there are two "foo" >> resources. The resource in module fake.module is located because it is >> an automatic module that opens all its packages unconditionally. The >> second resource is located by searching the parent class loader, a >> URLClassLoader in the example that also locates "foo". Which "foo" did >> you expect to locate? If code in fake.module just wants to locate the >> resource in its own module then it should use getResource rather than >> getResources. >> >> -Alan. >> From Alan.Bateman at oracle.com Sat Jan 2 17:49:21 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 2 Jan 2021 17:49:21 +0000 Subject: Should ClassLoader::getResouces return the same resource twice? In-Reply-To: References: Message-ID: On 02/01/2021 12:59, Thiago Henrique Hupner wrote: > I guess a little context can make more things clear: > The servlet spec requires that all jars from WEB-INF/lib > be available to the same classloader. > The resource, in particular, is "META-INF/web-fragment.xml" > Each jar can contain its own. So, using getResources make sense > in order of parsing each. However, what is happening is if I have two JARs > each with its own META-INF/web-fragment.xml, using the ModuleReader > it is returning four resources, so it parses more than it should and > it fails > to parse the same resource twice. > Which parent class loader are you specifying to defineModulesWithOneLoader? It it possible that it locates the resource in the two JAR files? -Alan From thihup at gmail.com Sat Jan 2 20:06:56 2021 From: thihup at gmail.com (Thiago Henrique Hupner) Date: Sat, 2 Jan 2021 17:06:56 -0300 Subject: Should ClassLoader::getResouces return the same resource twice? In-Reply-To: References: Message-ID: Yes, the parent loader locates two resources and it is used in the defineModulesWithOneLoader. So the behavior is the following: module-a.jar -> META-INF/web-fragment.xml module-b.jar -> META-INF/web-fragment.xml Creating a URLClassloader with these two jars and calling the getResources, it returns two URLs. If then I create a module for each jar, the contents of each module will be the same as the content found by the URLClassloader. Using that classloader as the parent in the defineModulesWithOneLoader gives the following behavior: Calling getResources with the loader created by the ModuleLayer, it will search the resources in all the modules of that layer (module.a and module.b), and then delegate to the parent (URLClassloader), which will find the same resources, thus, doubling the results. Now describing it make clear to me that is something not easy to fix, so by now, I worked around it by creating a ModuleReader that only find the ".class" files in the module and the other resources will be available by the parent, however, I don't know if this solution is the best. Em s?b., 2 de jan. de 2021 ?s 14:51, Alan Bateman escreveu: > On 02/01/2021 12:59, Thiago Henrique Hupner wrote: > > I guess a little context can make more things clear: > > The servlet spec requires that all jars from WEB-INF/lib > > be available to the same classloader. > > The resource, in particular, is "META-INF/web-fragment.xml" > > Each jar can contain its own. So, using getResources make sense > > in order of parsing each. However, what is happening is if I have two > JARs > > each with its own META-INF/web-fragment.xml, using the ModuleReader > > it is returning four resources, so it parses more than it should and > > it fails > > to parse the same resource twice. > > > Which parent class loader are you specifying to > defineModulesWithOneLoader? It it possible that it locates the resource > in the two JAR files? > > -Alan > From johannes.spangenberg at hotmail.de Sun Jan 3 04:21:50 2021 From: johannes.spangenberg at hotmail.de (Johannes Spangenberg) Date: Sun, 3 Jan 2021 05:21:50 +0100 Subject: Should ClassLoader::getResouces return the same resource twice? In-Reply-To: References: Message-ID: Note that you are actually loading the JARs twice which means that you might have two versions of the same classses loaded, not only of the resources. Is this intentional? I still don't know what you want to achieve with your class loader and module layer. Am 02.01.2021 um 21:06 schrieb Thiago Henrique Hupner: > Yes, the parent loader locates two resources and it is used in the > defineModulesWithOneLoader. > > So the behavior is the following: > module-a.jar -> META-INF/web-fragment.xml > module-b.jar -> META-INF/web-fragment.xml > > Creating a URLClassloader with these two jars and calling the getResources, > it returns two URLs. > > If then I create a module for each jar, the contents of each module will be > the same as the content > found by the URLClassloader. > Using that classloader as the parent in the defineModulesWithOneLoader > gives the following behavior: > > Calling getResources with the loader created by the ModuleLayer, > it will search the resources in all the modules of that layer (module.a and > module.b), > and then delegate to the parent (URLClassloader), which will find the same > resources, > thus, doubling the results. > > Now describing it make clear to me that is something not easy to fix, > so by now, I worked around it by creating a ModuleReader that only > find the ".class" files in the module and the other resources will be > available by the parent, > however, I don't know if this solution is the best. > > > Em s?b., 2 de jan. de 2021 ?s 14:51, Alan Bateman > escreveu: > >> On 02/01/2021 12:59, Thiago Henrique Hupner wrote: >>> I guess a little context can make more things clear: >>> The servlet spec requires that all jars from WEB-INF/lib >>> be available to the same classloader. >>> The resource, in particular, is "META-INF/web-fragment.xml" >>> Each jar can contain its own. So, using getResources make sense >>> in order of parsing each. However, what is happening is if I have two >> JARs >>> each with its own META-INF/web-fragment.xml, using the ModuleReader >>> it is returning four resources, so it parses more than it should and >>> it fails >>> to parse the same resource twice. >>> >> Which parent class loader are you specifying to >> defineModulesWithOneLoader? It it possible that it locates the resource >> in the two JAR files? >> >> -Alan >> From thihup at gmail.com Sun Jan 3 13:10:20 2021 From: thihup at gmail.com (Thiago Henrique Hupner) Date: Sun, 3 Jan 2021 10:10:20 -0300 Subject: Should ClassLoader::getResouces return the same resource twice? In-Reply-To: References: Message-ID: Yes, that is intentional. So I guess the correct behavior is to find the jars only via the ModuleFinder, and use the classloader only for nonmodular jars, is this right? In the Servlet spec, it is allowed to have split packages, from the WEB-INF/classes and WEB-INF/lib/ jars, for instance, so it is not as trivial as running an application in the command line. Besides being allowed split packages, the classes in the WEB-INF/classes must override the same classes in the WEB-INF/lib (something like the --patch-module) My current behavior is trying to find the modules using my ModuleFinder (most of the modules will be an automatic module) and if it is not allowed for some reason (invalid package name, invalid services, so on) it fallbacks to the classloader to still find all the things correctly. Unfortunately, I cannot only remove the jars from my classloader and delegate to the classloader provided by the ModuleLayer because we use a custom classloader, and have some important information stored in it, so by removing the jars from the classloader all applications would stop working. I hope made it a little more clear. By now it is clear that it is loading the same resource twice because the same jar is in the classloader and in the module layer classloader. Em dom., 3 de jan. de 2021 ?s 01:24, Johannes Spangenberg < johannes.spangenberg at hotmail.de> escreveu: > Note that you are actually loading the JARs twice which means that you > might have two versions of the same classses loaded, not only of the > resources. Is this intentional? I still don't know what you want to > achieve with your class loader and module layer. > > Am 02.01.2021 um 21:06 schrieb Thiago Henrique Hupner: > > Yes, the parent loader locates two resources and it is used in the > > defineModulesWithOneLoader. > > > > So the behavior is the following: > > module-a.jar -> META-INF/web-fragment.xml > > module-b.jar -> META-INF/web-fragment.xml > > > > Creating a URLClassloader with these two jars and calling the > getResources, > > it returns two URLs. > > > > If then I create a module for each jar, the contents of each module will > be > > the same as the content > > found by the URLClassloader. > > Using that classloader as the parent in the defineModulesWithOneLoader > > gives the following behavior: > > > > Calling getResources with the loader created by the ModuleLayer, > > it will search the resources in all the modules of that layer (module.a > and > > module.b), > > and then delegate to the parent (URLClassloader), which will find the > same > > resources, > > thus, doubling the results. > > > > Now describing it make clear to me that is something not easy to fix, > > so by now, I worked around it by creating a ModuleReader that only > > find the ".class" files in the module and the other resources will be > > available by the parent, > > however, I don't know if this solution is the best. > > > > > > Em s?b., 2 de jan. de 2021 ?s 14:51, Alan Bateman < > Alan.Bateman at oracle.com> > > escreveu: > > > >> On 02/01/2021 12:59, Thiago Henrique Hupner wrote: > >>> I guess a little context can make more things clear: > >>> The servlet spec requires that all jars from WEB-INF/lib > >>> be available to the same classloader. > >>> The resource, in particular, is "META-INF/web-fragment.xml" > >>> Each jar can contain its own. So, using getResources make sense > >>> in order of parsing each. However, what is happening is if I have two > >> JARs > >>> each with its own META-INF/web-fragment.xml, using the ModuleReader > >>> it is returning four resources, so it parses more than it should and > >>> it fails > >>> to parse the same resource twice. > >>> > >> Which parent class loader are you specifying to > >> defineModulesWithOneLoader? It it possible that it locates the resource > >> in the two JAR files? > >> > >> -Alan > >> > From Alan.Bateman at oracle.com Mon Jan 4 08:27:04 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 4 Jan 2021 08:27:04 +0000 Subject: Making jlink Plug-in API public In-Reply-To: References: <63363488-5389-6829-d83d-2abc9c37eb68@oracle.com> Message-ID: <64b699ed-3dff-0643-d884-0b2dd2a5a135@oracle.com> On 29/12/2020 19:29, Gunnar Morling wrote: > : > > That's interesting; what is missing from your PoV to make the API an > incubating one? Getting consensus that it's the right thing to do, and then the commitment to doing it. The latter requires re-examining the internal plugin API, looking at the issues that have come up since JDK 9, going over each of the API elements, iteration, and then setting it on a course to incubate for a release or two, gathering feedback, then eventually make it permanent and maintaining it. On one hard, it could be interesting to have an eco system of useful link-time plugins.. On the other hand, I think many of the plugins will be tied to core modules so cannot be maintained outside of the jdk repo. Also many of the plugins that we have experience with to date involve generating classes or modifying existing class files at link time so there may be an argument to wait until the JDK has a bytecode API. > : > > I could see myself contributing to a plug-in around signature > validation, depending on how involved that task would be. I did a > quick PoC and discuss it here: > > https://www.morling.dev/blog/jlinks-missing-link-api-signature-validation/ > > > This is using the existing Animal Sniffer tool for a basic signature > check. I'm not sure how much effort it'd be to fully implement this, > and also catch things like added non-virtual methods in a base > class/interface, changes to type parameters, and more. Could existing > code from the JDK be re-used for that task? Nice blog and PoC! Yes, I think this is worth exploring more as static analysis to detect mismatch with compile-time could be useful. A possible starting point is something that scans a module path, it doesn't have to start out as a jlink plugin. I could imagine `java --validate-modules` making use of this to do deep analysis for example. -Alan From Alan.Bateman at oracle.com Mon Jan 4 08:41:21 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 4 Jan 2021 08:41:21 +0000 Subject: Should ClassLoader::getResouces return the same resource twice? In-Reply-To: References: Message-ID: On 03/01/2021 13:10, Thiago Henrique Hupner wrote: > : > > I hope made it a little more clear. By now it is clear that it is loading > the same > resource twice because the same jar is in the classloader and in the module > layer > classloader. If I read your mail correctly you've got a class path today and you want to "move" the well behaved components to a module path, leaving the remaining the components that can't work as automatic modules on the class path. It's equivalent to specifying both a module path and class path to the java command line launcher. You also mention that you have to use a custom class loader. If that class loader were updated to support modules and load classes into its unnamed module then you should be able to get this to work. This is very advanced territory and would be approximately equivalent to the application class loader in the JDK where it supports both modules and the class path. I don't think your current approach of using a URLClassLoader as a parent will work as it doesn't support the bidirectional delegation that would be need to get this to work. -Alan. From thihup at gmail.com Mon Jan 4 11:38:45 2021 From: thihup at gmail.com (Thiago Henrique Hupner) Date: Mon, 4 Jan 2021 08:38:45 -0300 Subject: Should ClassLoader::getResouces return the same resource twice? In-Reply-To: References: Message-ID: Yes, that's exactly it. Sorry for all the noise in the previous emails. I'm not using a URLClassLoader in my application, I've only used it as an example because it is very easy to override to return a specific value (and not have to implement all the other methods). So, I guess I will need to have a look at the application classloader and update my current classloader to be able to do the bidirectional delegation. Do you have any recommendations about where to look? Thanks! Em seg., 4 de jan. de 2021 ?s 05:48, Alan Bateman escreveu: > On 03/01/2021 13:10, Thiago Henrique Hupner wrote: > > : > > > > I hope made it a little more clear. By now it is clear that it is loading > > the same > > resource twice because the same jar is in the classloader and in the > module > > layer > > classloader. > If I read your mail correctly you've got a class path today and you want > to "move" the well behaved components to a module path, leaving the > remaining the components that can't work as automatic modules on the > class path. It's equivalent to specifying both a module path and class > path to the java command line launcher. You also mention that you have > to use a custom class loader. If that class loader were updated to > support modules and load classes into its unnamed module then you should > be able to get this to work. This is very advanced territory and would > be approximately equivalent to the application class loader in the JDK > where it supports both modules and the class path. I don't think your > current approach of using a URLClassLoader as a parent will work as it > doesn't support the bidirectional delegation that would be need to get > this to work. > > -Alan. > From johannes.spangenberg at hotmail.de Mon Jan 4 12:41:18 2021 From: johannes.spangenberg at hotmail.de (Johannes Spangenberg) Date: Mon, 4 Jan 2021 13:41:18 +0100 Subject: Should ClassLoader::getResouces return the same resource twice? In-Reply-To: References: Message-ID: The bidirectional delegation is an important point. Beside that, you also mentioned that your custom class loader stores additional information. > Unfortunately, I cannot only remove the jars from my classloader and > delegate > to the classloader provided by the ModuleLayer because we use a custom > classloader, > and have some important information stored in it, so by removing the > jars from the classloader > all applications would stop working. If you really need this information as part of the class loader, I fear you need to create custom class loaders for your modules together with ModuleLayer.defineModules [1] instead of ModuleLayer.defineModulesWithOneLoader. The class loader for ModuleLayer.defineModulesWithOneLoader is not public. I guess you would need to reimplement it. Maybe you can find other locations to store the information. [1] https://docs.oracle.com/en/java/javase/15/docs/api/java.base/java/lang/ModuleLayer.html#defineModules(java.lang.module.Configuration,java.util.List,java.util.function.Function) Am 04.01.2021 um 12:38 schrieb Thiago Henrique Hupner: > Yes, that's exactly it. Sorry for all the noise in the previous emails. > > I'm not using a URLClassLoader in my application, I've only used it > as an example because it is very easy to override to return a specific > value (and not have to implement all the other methods). > > So, I guess I will need to have a look at the application classloader > and update my current classloader to be able to do the bidirectional > delegation. Do you have any recommendations about where to look? > > Thanks! > > Em seg., 4 de jan. de 2021 ?s 05:48, Alan Bateman > escreveu: > >> On 03/01/2021 13:10, Thiago Henrique Hupner wrote: >>> : >>> >>> I hope made it a little more clear. By now it is clear that it is loading >>> the same >>> resource twice because the same jar is in the classloader and in the >> module >>> layer >>> classloader. >> If I read your mail correctly you've got a class path today and you want >> to "move" the well behaved components to a module path, leaving the >> remaining the components that can't work as automatic modules on the >> class path. It's equivalent to specifying both a module path and class >> path to the java command line launcher. You also mention that you have >> to use a custom class loader. If that class loader were updated to >> support modules and load classes into its unnamed module then you should >> be able to get this to work. This is very advanced territory and would >> be approximately equivalent to the application class loader in the JDK >> where it supports both modules and the class path. I don't think your >> current approach of using a URLClassLoader as a parent will work as it >> doesn't support the bidirectional delegation that would be need to get >> this to work. >> >> -Alan. >> From thihup at gmail.com Mon Jan 4 13:51:18 2021 From: thihup at gmail.com (Thiago Henrique Hupner) Date: Mon, 4 Jan 2021 10:51:18 -0300 Subject: Should ClassLoader::getResouces return the same resource twice? In-Reply-To: References: Message-ID: Looks like the ModuleLayer.defineModules is the thing I needed. With it, I can use my current classloader to load all the modules and it can find the classes from both module and unnamed module. However, I'm having some trouble with the ServiceLoader now, In the configuration, I've called resourceAndBind, but I'm getting "Provider foo.bar not found" There is some trick to get a class from an upper layer to bind to the a class in the boot layer? Thanks! Em seg., 4 de jan. de 2021 ?s 09:41, Johannes Spangenberg < johannes.spangenberg at hotmail.de> escreveu: > The bidirectional delegation is an important point. Beside that, you > also mentioned that your custom class loader stores additional information. > > > Unfortunately, I cannot only remove the jars from my classloader and > > delegate > > to the classloader provided by the ModuleLayer because we use a custom > > classloader, > > and have some important information stored in it, so by removing the > > jars from the classloader > > all applications would stop working. > > If you really need this information as part of the class loader, I fear > you need to create custom class loaders for your modules together with > ModuleLayer.defineModules [1] instead of > ModuleLayer.defineModulesWithOneLoader. The class loader for > ModuleLayer.defineModulesWithOneLoader is not public. I guess you would > need to reimplement it. Maybe you can find other locations to store the > information. > > [1] > > https://docs.oracle.com/en/java/javase/15/docs/api/java.base/java/lang/ModuleLayer.html#defineModules(java.lang.module.Configuration,java.util.List,java.util.function.Function) > > Am 04.01.2021 um 12:38 schrieb Thiago Henrique Hupner: > > Yes, that's exactly it. Sorry for all the noise in the previous emails. > > > > I'm not using a URLClassLoader in my application, I've only used it > > as an example because it is very easy to override to return a specific > > value (and not have to implement all the other methods). > > > > So, I guess I will need to have a look at the application classloader > > and update my current classloader to be able to do the bidirectional > > delegation. Do you have any recommendations about where to look? > > > > Thanks! > > > > Em seg., 4 de jan. de 2021 ?s 05:48, Alan Bateman < > Alan.Bateman at oracle.com> > > escreveu: > > > >> On 03/01/2021 13:10, Thiago Henrique Hupner wrote: > >>> : > >>> > >>> I hope made it a little more clear. By now it is clear that it is > loading > >>> the same > >>> resource twice because the same jar is in the classloader and in the > >> module > >>> layer > >>> classloader. > >> If I read your mail correctly you've got a class path today and you want > >> to "move" the well behaved components to a module path, leaving the > >> remaining the components that can't work as automatic modules on the > >> class path. It's equivalent to specifying both a module path and class > >> path to the java command line launcher. You also mention that you have > >> to use a custom class loader. If that class loader were updated to > >> support modules and load classes into its unnamed module then you should > >> be able to get this to work. This is very advanced territory and would > >> be approximately equivalent to the application class loader in the JDK > >> where it supports both modules and the class path. I don't think your > >> current approach of using a URLClassLoader as a parent will work as it > >> doesn't support the bidirectional delegation that would be need to get > >> this to work. > >> > >> -Alan. > >> > From thihup at gmail.com Mon Jan 4 14:13:58 2021 From: thihup at gmail.com (Thiago Henrique Hupner) Date: Mon, 4 Jan 2021 11:13:58 -0300 Subject: Should ClassLoader::getResouces return the same resource twice? In-Reply-To: References: Message-ID: The trick is that now I'm using my classloader to load the modules, I need to override the ClassLoader#findClass(String module, String className) So the service loader finds the classes successfully. Em seg., 4 de jan. de 2021 ?s 10:51, Thiago Henrique Hupner < thihup at gmail.com> escreveu: > Looks like the ModuleLayer.defineModules is the thing I needed. > With it, I can use my current classloader to load all the modules > and it can find the classes from both module and unnamed module. > > However, I'm having some trouble with the ServiceLoader now, > In the configuration, I've called resourceAndBind, but I'm getting > "Provider foo.bar not found" > > There is some trick to get a class from an upper layer to bind to the > a class in the boot layer? > > Thanks! > > Em seg., 4 de jan. de 2021 ?s 09:41, Johannes Spangenberg < > johannes.spangenberg at hotmail.de> escreveu: > >> The bidirectional delegation is an important point. Beside that, you >> also mentioned that your custom class loader stores additional >> information. >> >> > Unfortunately, I cannot only remove the jars from my classloader and >> > delegate >> > to the classloader provided by the ModuleLayer because we use a custom >> > classloader, >> > and have some important information stored in it, so by removing the >> > jars from the classloader >> > all applications would stop working. >> >> If you really need this information as part of the class loader, I fear >> you need to create custom class loaders for your modules together with >> ModuleLayer.defineModules [1] instead of >> ModuleLayer.defineModulesWithOneLoader. The class loader for >> ModuleLayer.defineModulesWithOneLoader is not public. I guess you would >> need to reimplement it. Maybe you can find other locations to store the >> information. >> >> [1] >> >> https://docs.oracle.com/en/java/javase/15/docs/api/java.base/java/lang/ModuleLayer.html#defineModules(java.lang.module.Configuration,java.util.List,java.util.function.Function) >> >> Am 04.01.2021 um 12:38 schrieb Thiago Henrique Hupner: >> > Yes, that's exactly it. Sorry for all the noise in the previous emails. >> > >> > I'm not using a URLClassLoader in my application, I've only used it >> > as an example because it is very easy to override to return a specific >> > value (and not have to implement all the other methods). >> > >> > So, I guess I will need to have a look at the application classloader >> > and update my current classloader to be able to do the bidirectional >> > delegation. Do you have any recommendations about where to look? >> > >> > Thanks! >> > >> > Em seg., 4 de jan. de 2021 ?s 05:48, Alan Bateman < >> Alan.Bateman at oracle.com> >> > escreveu: >> > >> >> On 03/01/2021 13:10, Thiago Henrique Hupner wrote: >> >>> : >> >>> >> >>> I hope made it a little more clear. By now it is clear that it is >> loading >> >>> the same >> >>> resource twice because the same jar is in the classloader and in the >> >> module >> >>> layer >> >>> classloader. >> >> If I read your mail correctly you've got a class path today and you >> want >> >> to "move" the well behaved components to a module path, leaving the >> >> remaining the components that can't work as automatic modules on the >> >> class path. It's equivalent to specifying both a module path and class >> >> path to the java command line launcher. You also mention that you have >> >> to use a custom class loader. If that class loader were updated to >> >> support modules and load classes into its unnamed module then you >> should >> >> be able to get this to work. This is very advanced territory and would >> >> be approximately equivalent to the application class loader in the JDK >> >> where it supports both modules and the class path. I don't think your >> >> current approach of using a URLClassLoader as a parent will work as it >> >> doesn't support the bidirectional delegation that would be need to get >> >> this to work. >> >> >> >> -Alan. >> >> >> > From johannes.spangenberg at hotmail.de Mon Jan 4 15:24:44 2021 From: johannes.spangenberg at hotmail.de (Johannes Spangenberg) Date: Mon, 4 Jan 2021 16:24:44 +0100 Subject: Should ClassLoader::getResouces return the same resource twice? In-Reply-To: References: Message-ID: > Looks like the ModuleLayer.defineModules is the thing I needed. > With it, I can use my current classloader to load all the modules > and it can find the classes from both module and unnamed module. Note that class loaders for modules behave differently then original class loaders. For example they will favor local class definitions over class definitions of the parent loader. Not sure if there are other important differences. The loader is implemented in jdk.internal.loader.Loader [1]. > There is some trick to get a class from an upper layer to bind to the > a class in the boot layer? You can pass the created module layer as first argument to ServiceLoader.load?(ModuleLayer, Class) [2]. [1] https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/jdk/internal/loader/Loader.java [2] https://docs.oracle.com/en/java/javase/15/docs/api/java.base/java/util/ServiceLoader.html#load(java.lang.ModuleLayer,java.lang.Class) From thihup at gmail.com Wed Jan 6 19:15:03 2021 From: thihup at gmail.com (Thiago Henrique Hupner) Date: Wed, 6 Jan 2021 16:15:03 -0300 Subject: IllegalAccessError with --patch-module Message-ID: Hi! I've noticed something strange, but I don't know if it is a bug: If we have two jars: module.a with the class com.foo.Bar module.b with the classes com.foo.Bar and com.foo.Main. Running: java -p module-a.jar --patch-module=module.a=module-b.jar -m module.a/com.foo.Main it throws: Error: Unable to load main class com.foo.Main in module module.a java.lang.IllegalAccessError: superclass access check failed: class com.foo.Main (in module module.a) cannot access class java.lang.Object (in module java.base) because module java.base does not export java.lang to module module.a Even with --add-exports=java.base/java.lang=module.a it doesn't work. Is this a bug? I have a real usage where the main class is the "patched" module to override the com.foo.Bar. Thanks! From Alan.Bateman at oracle.com Wed Jan 6 19:56:02 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 6 Jan 2021 19:56:02 +0000 Subject: IllegalAccessError with --patch-module In-Reply-To: References: Message-ID: <480cb5be-a186-b137-06b1-16372e4b10fe@oracle.com> On 06/01/2021 19:15, Thiago Henrique Hupner wrote: > Hi! > > I've noticed something strange, but I don't know if it is a bug: > If we have two jars: > module.a with the class com.foo.Bar > module.b with the classes com.foo.Bar and com.foo.Main. > > Running: > java -p module-a.jar --patch-module=module.a=module-b.jar -m > module.a/com.foo.Main > > it throws: > Error: Unable to load main class com.foo.Main in module module.a > java.lang.IllegalAccessError: superclass access check failed: class > com.foo.Main (in module module.a) cannot access class java.lang.Object (in > module java.base) because module java.base does not export java.lang to > module module.a > > Even with --add-exports=java.base/java.lang=module.a it doesn't work. > > Is this a bug? I have a real usage where the main class is the "patched" > module to override the com.foo.Bar. > I'm not aware of any issues in this area. Can you create a test case as that might be easier to discuss? (I'm guessing the module names aren't "module.a" and "module.b" as that won't compile. Also I would expect this type of patching to emit a warning that the module-info.class in module-b.jar is ignored). -Alan From thihup at gmail.com Wed Jan 6 19:57:56 2021 From: thihup at gmail.com (Thiago Henrique Hupner) Date: Wed, 6 Jan 2021 16:57:56 -0300 Subject: IllegalAccessError with --patch-module In-Reply-To: <480cb5be-a186-b137-06b1-16372e4b10fe@oracle.com> References: <480cb5be-a186-b137-06b1-16372e4b10fe@oracle.com> Message-ID: Sorry, they are automatic modules. I'll create a test case. Em qua., 6 de jan. de 2021 ?s 16:56, Alan Bateman escreveu: > On 06/01/2021 19:15, Thiago Henrique Hupner wrote: > > Hi! > > > > I've noticed something strange, but I don't know if it is a bug: > > If we have two jars: > > module.a with the class com.foo.Bar > > module.b with the classes com.foo.Bar and com.foo.Main. > > > > Running: > > java -p module-a.jar --patch-module=module.a=module-b.jar -m > > module.a/com.foo.Main > > > > it throws: > > Error: Unable to load main class com.foo.Main in module module.a > > java.lang.IllegalAccessError: superclass access check failed: > class > > com.foo.Main (in module module.a) cannot access class java.lang.Object > (in > > module java.base) because module java.base does not export java.lang to > > module module.a > > > > Even with --add-exports=java.base/java.lang=module.a it doesn't work. > > > > Is this a bug? I have a real usage where the main class is the "patched" > > module to override the com.foo.Bar. > > > I'm not aware of any issues in this area. Can you create a test case as > that might be easier to discuss? (I'm guessing the module names aren't > "module.a" and "module.b" as that won't compile. Also I would expect > this type of patching to emit a warning that the module-info.class in > module-b.jar is ignored). > > -Alan > From Alan.Bateman at oracle.com Wed Jan 6 20:25:53 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 6 Jan 2021 20:25:53 +0000 Subject: IllegalAccessError with --patch-module In-Reply-To: References: <480cb5be-a186-b137-06b1-16372e4b10fe@oracle.com> Message-ID: <76ccf2e8-4ed7-30bf-6134-8270da193a9b@oracle.com> On 06/01/2021 19:57, Thiago Henrique Hupner wrote: > Sorry, they are automatic modules. > > I'll create a test case. > Also if you can include the output with --show-module-resolution then it might help track this down quickly (running without any explicit modules on the module path is, on the surface, a bit unusual but I wouldn't expect any issues). -Alan From thihup at gmail.com Wed Jan 6 21:47:05 2021 From: thihup at gmail.com (Thiago Henrique Hupner) Date: Wed, 6 Jan 2021 18:47:05 -0300 Subject: IllegalAccessError with --patch-module In-Reply-To: <76ccf2e8-4ed7-30bf-6134-8270da193a9b@oracle.com> References: <480cb5be-a186-b137-06b1-16372e4b10fe@oracle.com> <76ccf2e8-4ed7-30bf-6134-8270da193a9b@oracle.com> Message-ID: This is getting very confusing. If you use any jar as an automatic module in the "--module-path" (-p) and add the main class in another jar (in this example jar-b.jar), patch both and the error happens. The main class is literally just an empty main method [1]. The error happens even if we use an empty jar. Running: /jvms/jdk-15/bin/java -p /tmp/activation.jar --show-module-resolution --patch-module=activation=target/jar-b.jar -m activation/com.foo.Main root activation file:///tmp/activation.jar automatic Error: Unable to load main class com.foo.Main in module activation java.lang.IllegalAccessError: superclass access check failed: class com.foo.Main (in module activation) cannot access class java.lang.Object (in module java.base) because module java.base does not export java.lang to module activation However, it works if we --add-modules java.base java -p /tmp/activation.jar --add-modules=java.base --show-module-resolution --patch-module=activation=target/jar-b.jar -m activation/com.foo.Main root activation file:///tmp/activation.jar automatic root java.base jrt:/java.base java.base binds java.logging jrt:/java.logging java.base binds java.management jrt:/java.management java.base binds jdk.security.auth jrt:/jdk.security.auth java.base binds jdk.charsets jrt:/jdk.charsets java.base binds jdk.zipfs jrt:/jdk.zipfs java.base binds jdk.localedata jrt:/jdk.localedata [...] Looks like if the automatic module is patched it loses the requires java.base java -p /tmp/activation.jar --add-modules=java.base --describe-module=activation --patch-module=activation=target/jar-b.jar WARNING: Unknown module: activation specified to --patch-module activation file:///tmp/activation.jar automatic contains com.foo contains com.sun.activation.registries contains com.sun.activation.viewers contains javax.activation java -p /tmp/activation.jar --add-modules=java.base --describe-module=activation activation file:///tmp/activation.jar automatic requires java.base mandated contains com.sun.activation.registries contains com.sun.activation.viewers contains javax.activation [1] package com.foo; public class Main { public static void main(String[] args) throws Exception{ } } Em qua., 6 de jan. de 2021 ?s 17:26, Alan Bateman escreveu: > On 06/01/2021 19:57, Thiago Henrique Hupner wrote: > > Sorry, they are automatic modules. > > > > I'll create a test case. > > > Also if you can include the output with --show-module-resolution then it > might help track this down quickly (running without any explicit modules > on the module path is, on the surface, a bit unusual but I wouldn't > expect any issues). > > -Alan > From info at j-kuhn.de Wed Jan 6 23:49:45 2021 From: info at j-kuhn.de (Johannes Kuhn) Date: Thu, 7 Jan 2021 00:49:45 +0100 Subject: IllegalAccessError with --patch-module In-Reply-To: References: <480cb5be-a186-b137-06b1-16372e4b10fe@oracle.com> <76ccf2e8-4ed7-30bf-6134-8270da193a9b@oracle.com> Message-ID: <65f95a91-a0eb-d475-b051-9d39bc2e6c2d@j-kuhn.de> Indeed a bug. Happens under the following conditions: * Patched module is an automatic module * Patch adds one or more additional packages to the module. Cause are the following lines: https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/jdk/internal/module/ModulePatcher.java#L154 This creates a module builder without strict mode. The following lines won't copy the original dependencies over if it is an automatic module. https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/lang/module/ModuleDescriptor.java#L2130 The dependency to java.base is not added - because the module builder is not strict. The simple fix of adding a requires(... "java.base" ...) doesn't work - because automatic modules can't require anything: https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/lang/module/ModuleDescriptor.java#L1554 Proposed fix: * Change requires to not check if it is an automatic module if the builder is not strict. * Move the copying of the requires out of the if. - Johannes On 06-Jan-21 22:47, Thiago Henrique Hupner wrote: > This is getting very confusing. > > If you use any jar as an automatic module in the "--module-path" (-p) and > add the main class in another jar (in this example jar-b.jar), patch both > and the error happens. > The main class is literally just an empty main method [1]. > The error happens even if we use an empty jar. > > Running: > /jvms/jdk-15/bin/java -p /tmp/activation.jar --show-module-resolution > --patch-module=activation=target/jar-b.jar -m activation/com.foo.Main > root activation file:///tmp/activation.jar automatic > Error: Unable to load main class com.foo.Main in module activation > java.lang.IllegalAccessError: superclass access check failed: class > com.foo.Main (in module activation) cannot access class java.lang.Object > (in module java.base) because module java.base does not export java.lang to > module activation > > However, it works if we --add-modules java.base > > java -p /tmp/activation.jar --add-modules=java.base > --show-module-resolution --patch-module=activation=target/jar-b.jar -m > activation/com.foo.Main > root activation file:///tmp/activation.jar automatic > root java.base jrt:/java.base > java.base binds java.logging jrt:/java.logging > java.base binds java.management jrt:/java.management > java.base binds jdk.security.auth jrt:/jdk.security.auth > java.base binds jdk.charsets jrt:/jdk.charsets > java.base binds jdk.zipfs jrt:/jdk.zipfs > java.base binds jdk.localedata jrt:/jdk.localedata > [...] > > Looks like if the automatic module is patched it loses the requires > java.base > > java -p /tmp/activation.jar --add-modules=java.base > --describe-module=activation --patch-module=activation=target/jar-b.jar > > WARNING: Unknown module: activation specified to --patch-module > activation file:///tmp/activation.jar automatic > contains com.foo > contains com.sun.activation.registries > contains com.sun.activation.viewers > contains javax.activation > > java -p /tmp/activation.jar --add-modules=java.base > --describe-module=activation > activation file:///tmp/activation.jar automatic > requires java.base mandated > contains com.sun.activation.registries > contains com.sun.activation.viewers > contains javax.activation > > > [1] > package com.foo; > > public class Main { > public static void main(String[] args) throws Exception{ > } > } > > Em qua., 6 de jan. de 2021 ?s 17:26, Alan Bateman > escreveu: > >> On 06/01/2021 19:57, Thiago Henrique Hupner wrote: >>> Sorry, they are automatic modules. >>> >>> I'll create a test case. >>> >> Also if you can include the output with --show-module-resolution then it >> might help track this down quickly (running without any explicit modules >> on the module path is, on the surface, a bit unusual but I wouldn't >> expect any issues). >> >> -Alan >> From Alan.Bateman at oracle.com Thu Jan 7 10:01:48 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 7 Jan 2021 10:01:48 +0000 Subject: IllegalAccessError with --patch-module In-Reply-To: <65f95a91-a0eb-d475-b051-9d39bc2e6c2d@j-kuhn.de> References: <480cb5be-a186-b137-06b1-16372e4b10fe@oracle.com> <76ccf2e8-4ed7-30bf-6134-8270da193a9b@oracle.com> <65f95a91-a0eb-d475-b051-9d39bc2e6c2d@j-kuhn.de> Message-ID: <33698ffb-99b9-7b82-a1f6-7cac5350d4c5@oracle.com> On 06/01/2021 23:49, Johannes Kuhn wrote: > Indeed a bug. > > Happens under the following conditions: > * Patched module is an automatic module > * Patch adds one or more additional packages to the module. > > : > > Proposed fix: > * Change requires to not check if it is an automatic module if the > builder is not strict. > * Move the copying of the requires out of the if. Thanks for the mails, it is bug. The additional information that the patch adds packages to the automatic module was important to understand the mails. One other thing is that the bug does not surface when the set of root modules includes an explicit module or an unpatched automatic module. So very obscure scenario that probably explains why there hasn't been reported before now. The patcher should be using strict mode when re-building the module descriptor for a patched automatic modules. The non-strict mode is for explicit modules as the identifiers (that may not be valid in the Java Language) are read from the module-info.class. I'll create a bug for this. -Alan. From info at j-kuhn.de Thu Jan 7 13:06:47 2021 From: info at j-kuhn.de (Johannes Kuhn) Date: Thu, 7 Jan 2021 14:06:47 +0100 Subject: IllegalAccessError with --patch-module In-Reply-To: <33698ffb-99b9-7b82-a1f6-7cac5350d4c5@oracle.com> References: <480cb5be-a186-b137-06b1-16372e4b10fe@oracle.com> <76ccf2e8-4ed7-30bf-6134-8270da193a9b@oracle.com> <65f95a91-a0eb-d475-b051-9d39bc2e6c2d@j-kuhn.de> <33698ffb-99b9-7b82-a1f6-7cac5350d4c5@oracle.com> Message-ID: On 07-Jan-21 11:01, Alan Bateman wrote: > On 06/01/2021 23:49, Johannes Kuhn wrote: >> Indeed a bug. >> >> Happens under the following conditions: >> * Patched module is an automatic module >> * Patch adds one or more additional packages to the module. >> >> : >> >> Proposed fix: >> * Change requires to not check if it is an automatic module if the >> builder is not strict. >> * Move the copying of the requires out of the if. > > Thanks for the mails, it is bug. The additional information that the > patch adds packages to the automatic module was important to understand > the mails. One other thing is that the bug does not surface when the set > of root modules includes an explicit module or an unpatched automatic > module. So very obscure scenario that probably explains why there hasn't > been reported before now. > > The patcher should be using strict mode when re-building the module > descriptor for a patched automatic modules. The non-strict mode is for > explicit modules as the identifiers (that may not be valid in the Java > Language) are read from the module-info.class. I'll create a bug for this. Even better fix. To be honest - I don't quite understand the bug - but I was able to reproduce it. So I did manually retrace the code to see where the difference was happening, and found the lines above - which was "good enough" for an explanation. Is there a way to debug the early java initialization? I didn't got that working, so had to do it by hand. > > -Alan. - Johannes From Alan.Bateman at oracle.com Thu Jan 7 19:04:10 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 7 Jan 2021 19:04:10 +0000 Subject: IllegalAccessError with --patch-module In-Reply-To: References: <480cb5be-a186-b137-06b1-16372e4b10fe@oracle.com> <76ccf2e8-4ed7-30bf-6134-8270da193a9b@oracle.com> <65f95a91-a0eb-d475-b051-9d39bc2e6c2d@j-kuhn.de> <33698ffb-99b9-7b82-a1f6-7cac5350d4c5@oracle.com> Message-ID: <5130e26e-fc7d-e21e-c951-21add9b044b2@oracle.com> On 07/01/2021 13:06, Johannes Kuhn wrote: > : > > To be honest - I don't quite understand the bug - but I was able to > reproduce it. > So I did manually retrace the code to see where the difference was > happening, and found the lines above - which was "good enough" for an > explanation. Resolution involves recursively enumerating the "requires" directives of a set of root modules (think transitive closure). The root modules are the module you specify to -m or the modules specified to --add-modules. Every module requires java.base so it will always be resolved and will also be in the the boot layer. The bug in the patcher code results in a root module that does not require java.base and since they are no other root modules, it means the boot layer doesn't have java.base, oops! But, the java.base has been partly defined to the VM in very early startup and normally the creation of the boot layer will complete the initialization, including setting up exports. The IllegalAccessError trying to link the super class arises because the exports aren't setup, meaning java.lang is not exported so no classes outside of java.base can be linked. > > Is there a way to debug the early java initialization? I didn't got > that working, so had to do it by hand. The debugger needs the VM to be fully initialized before it can debug Java code. In JVM TI speak this is the "live phase". There are early phases where tooling and agents can get events and do some operations during VM startup but it's not enough for a Java debugger. -Alan. From info at j-kuhn.de Thu Jan 7 20:29:41 2021 From: info at j-kuhn.de (Johannes Kuhn) Date: Thu, 7 Jan 2021 21:29:41 +0100 Subject: IllegalAccessError with --patch-module In-Reply-To: <5130e26e-fc7d-e21e-c951-21add9b044b2@oracle.com> References: <480cb5be-a186-b137-06b1-16372e4b10fe@oracle.com> <76ccf2e8-4ed7-30bf-6134-8270da193a9b@oracle.com> <65f95a91-a0eb-d475-b051-9d39bc2e6c2d@j-kuhn.de> <33698ffb-99b9-7b82-a1f6-7cac5350d4c5@oracle.com> <5130e26e-fc7d-e21e-c951-21add9b044b2@oracle.com> Message-ID: On 07-Jan-21 20:04, Alan Bateman wrote> [...] The bug in the patcher code results in a > root module that does not require java.base and since they are no other > root modules, it means the boot layer doesn't have java.base, oops! But, > the java.base has been partly defined to the VM in very early startup > and normally the creation of the boot layer will complete the > initialization, including setting up exports. [...] Gotcha. That was the missing piece. Thank you. >> Is there a way to debug the early java initialization? I didn't got >> that working, so had to do it by hand. > The debugger needs the VM to be fully initialized before it can debug > Java code. In JVM TI speak this is the "live phase". There are early > phases where tooling and agents can get events and do some operations > during VM startup but it's not enough for a Java debugger. Was worth a shot to ask. Thanks. > -Alan. - Johannes