From johan.vos at gluonhq.com Wed Sep 15 08:45:26 2021 From: johan.vos at gluonhq.com (Johan Vos) Date: Wed, 15 Sep 2021 10:45:26 +0200 Subject: Modules with platform specific parts Message-ID: Hi, There have been discussions in the past about how to deal with platform-specific parts (java code, native code, resources) in modules. There is no standard for this, and afaik no recommendation. In the OpenJFX project, we upload jars with module info to maven central, and we have plugins for maven and gradle to deal with them at compiletime and at runtime. However, the lack of a standard recommendation forces us to change the internal behavior every now and then. The latest change we made (removing automatic module names from empty modules [1]) caused issues in the 17 release, when developers compile JavaFX modular apps [2] Before we make many other changes, I would like to have a better view on what would be the recommended approach, so here is my summary. Suggestions are highly appreciated. Let's assume we have a component that contains some platform-independent Java code, some platform-dependent Java code, and some platform-dependent native code. To make the example concrete, let's use the javafx.graphics "module" from the OpenJFX project, which contains exactly that. There are a number of options, and before thinking about the best way for tools do deal with this situation, it would be good to have a recommended approach for those "hybrid" modules. 1. There is a single module (i.e. only one Module). All code, no matter on what platform or in which layer, will report the same value for class.getModule().getName(). This would lead to a single "javafx.graphics" conceptual module, but there will be a number of physical module files that are different (with different class files and native code). Those different modules should obvisouly be mutually exclusive in a runtime. 2. There are 2 modules: the platform-independent Java code goes into one module (let's call that javafx.graphics.api) and a second module is named javafx.graphics.platform and contains the platform-dependent Java code and the native code. In this approach, developers use the javafx.graphics.api module to compile against, and at runtime the javafx.graphics.platform module is required. Again, that second module will have a number of different physical implementations. As an extension to this, we might add the Service Provider Interface approach for loading platform-specific modules/bits at runtime. 3. We create one module for each platform. There is no "javafx.graphics" module in this case, but there is a "javafx.graphics.linux.aarch64" module for example. Doing so, there is a tight coupling between one conceptual module and one physical module. A clear drawback of this is that this is a real challenge at compiletime. Developers (who are only using generic API's) need to compile against a platform-specific module. 4. We use 2 artifacts: an "empty" one and then a number of implementation specific ones. The difference with option 2 is that the empty "module" exists solely for the purpose of tools, which can detect what implementation specific module(s) need to be loaded at compile/runtime. We currently use option 4, but in my opinion, option 2 would be the better approach. Thanks, - Johan [1] https://bugs.openjdk.java.net/browse/JDK-8264998 [2] https://mail.openjdk.java.net/pipermail/openjfx-dev/2021-September/031934.html From michal at kleczek.org Wed Sep 15 09:15:33 2021 From: michal at kleczek.org (=?utf-8?Q?Micha=C5=82_K=C5=82eczek?=) Date: Wed, 15 Sep 2021 11:15:33 +0200 Subject: Modules with platform specific parts In-Reply-To: References: Message-ID: <9B6B055F-8CD1-45B6-A6CA-EE980DB74994@kleczek.org> Hi, Hi, There is another option: API module and multiple platform specific modules (different names like javafx.graphics.linux.aarch64) that require API module and provide services. API module does not ?require? any platform specific module but uses ServiceLoader to find a proper implementation at runtime. Thanks, Michal > On 15 Sep 2021, at 10:45, Johan Vos wrote: > > Hi, > > There have been discussions in the past about how to deal with > platform-specific parts (java code, native code, resources) in modules. > There is no standard for this, and afaik no recommendation. In the OpenJFX > project, we upload jars with module info to maven central, and we have > plugins for maven and gradle to deal with them at compiletime and at > runtime. > > However, the lack of a standard recommendation forces us to change the > internal behavior every now and then. The latest change we made (removing > automatic module names from empty modules [1]) caused issues in the 17 > release, when developers compile JavaFX modular apps [2] > > Before we make many other changes, I would like to have a better view on > what would be the recommended approach, so here is my summary. Suggestions > are highly appreciated. > > Let's assume we have a component that contains some > platform-independent Java code, some platform-dependent Java code, and > some platform-dependent native code. To make the example concrete, > let's use the javafx.graphics "module" from the OpenJFX project, which > contains exactly that. > > There are a number of options, and before thinking about the best way > for tools do deal with this situation, it would be good to have a > recommended approach for those "hybrid" modules. > > 1. There is a single module (i.e. only one Module). All code, no > matter on what platform or in which layer, will report the same value > for class.getModule().getName(). This would lead to a single > "javafx.graphics" conceptual module, but there will be a number of > physical module files that are different (with different class files > and native code). Those different modules should obvisouly be mutually > exclusive in a runtime. > > 2. There are 2 modules: the platform-independent Java code goes into > one module (let's call that javafx.graphics.api) and a second module > is named javafx.graphics.platform and contains the platform-dependent > Java code and the native code. > In this approach, developers use the javafx.graphics.api module to > compile against, and at runtime the javafx.graphics.platform module is > required. Again, that second module will have a number of different > physical implementations. As an extension to this, we might add the > Service Provider Interface approach for loading platform-specific > modules/bits at runtime. > > 3. We create one module for each platform. There is no > "javafx.graphics" module in this case, but there is a > "javafx.graphics.linux.aarch64" module for example. > Doing so, there is a tight coupling between one conceptual module and > one physical module. A clear drawback of this is that this is a real > challenge at compiletime. Developers (who are only using generic > API's) need to compile against a platform-specific module. > > 4. We use 2 artifacts: an "empty" one and then a number of > implementation specific ones. The difference with option 2 is that the > empty "module" exists solely for the purpose of tools, which can > detect what implementation specific module(s) need to be loaded at > compile/runtime. > > We currently use option 4, but in my opinion, option 2 would be the > better approach. > > Thanks, > > - Johan > > [1] https://bugs.openjdk.java.net/browse/JDK-8264998 > [2] > https://mail.openjdk.java.net/pipermail/openjfx-dev/2021-September/031934.html From Alan.Bateman at oracle.com Wed Sep 15 13:55:23 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 15 Sep 2021 14:55:23 +0100 Subject: Modules with platform specific parts In-Reply-To: References: Message-ID: <86876ebf-64b4-f886-e10e-2bcf972ee12c@oracle.com> On 15/09/2021 09:45, Johan Vos wrote: > Hi, > > There have been discussions in the past about how to deal with > platform-specific parts (java code, native code, resources) in modules. > There is no standard for this, and afaik no recommendation. In the OpenJFX > project, we upload jars with module info to maven central, and we have > plugins for maven and gradle to deal with them at compiletime and at > runtime. > Project Panama and the foreign linker API may be another motivation to re-visit this topic. One general concern is adding more complexity to the JAR format. I could imagine platform specific sections adding another dimension of complexity to MR JARs, modular JARs, modular MR JARs, ...? Also every bag we nail on could have implications for compile time, packaging time, jlink, and run-time. The position we took on this in JDK 9 is that modules can be platform specific. There is a JDK-specific class file attribute named "ModuleTarget" for the OS and/or architecture.? The jmod tool has an option to specify the OS/arch when creating a JMOD. Post resolution checks will catch issues where someone accidentally deploys a module for the different OS/architecture. I have not looked at the JavaFX modules recently but at some point I think they were packaged as JMOD files and it might be that the platform was specified at that time. If you've since moved to modular JARs then I could imagine this getting lost. -Alan From hg at apteryx.fr Wed Sep 15 16:09:42 2021 From: hg at apteryx.fr (=?UTF-8?Q?Herv=c3=a9_Guillemet?=) Date: Wed, 15 Sep 2021 18:09:42 +0200 Subject: Modules with platform specific parts In-Reply-To: <86876ebf-64b4-f886-e10e-2bcf972ee12c@oracle.com> References: <86876ebf-64b4-f886-e10e-2bcf972ee12c@oracle.com> Message-ID: Hi, Thank you Johan for starting this discussion. The current situation is indeed problematic and a (de facto or not) standard should emerge. IMO, beyond being easy to use, the selected mechanism should allow to build multi-platform images (for instance for MacOSX on both aarch64 and x86_64). > 1. There is a single module (i.e. only one Module). All code, no > matter on what platform or in which layer, will report the same value > for class.getModule().getName(). This would lead to a single > "javafx.graphics" conceptual module, but there will be a number of > physical module files that are different (with different class files > and native code). Those different modules should obvisouly be mutually > exclusive in a runtime. This option doesn?t allow multi-platform images. Also the duplication of all common Java classes in each platform jars is not ideal. > 2. There are 2 modules: the platform-independent Java code goes into > one module (let's call that javafx.graphics.api) and a second module > is named javafx.graphics.platform and contains the platform-dependent > Java code and the native code. > In this approach, developers use the javafx.graphics.api module to > compile against, and at runtime the javafx.graphics.platform module is > required. Again, that second module will have a number of different > physical implementations. As an extension to this, we might add the > Service Provider Interface approach for loading platform-specific > modules/bits at runtime. This option doesn?t allow multi-platform images. > 3. We create one module for each platform. There is no > "javafx.graphics" module in this case, but there is a > "javafx.graphics.linux.aarch64" module for example. > Doing so, there is a tight coupling between one conceptual module and > one physical module. A clear drawback of this is that this is a real > challenge at compiletime. Developers (who are only using generic > API's) need to compile against a platform-specific module. Indeed. The building of the module path can be automatized with Maven on another build system, but the module descriptor should also be changed for each platform you are building for. Multi-platform images won?t work either since packages would be split, and common classes are also duplicated. I don?t see any advantage compared to option 1. > 4. We use 2 artifacts: an "empty" one and then a number of > implementation specific ones. The difference with option 2 is that the > empty "module" exists solely for the purpose of tools, which can > detect what implementation specific module(s) need to be loaded at > compile/runtime. Like mentioned in your referenced links, this causes problems. I haven?t worked with OpenJFX 17 but with OpenJFX 11-16, and to be able to build JLink images I had either to use the JavaFX maven plugin (which takes care of empty modules but one may wish to use alternate plugins because of some limitations [1-4]) or explicitly depend directly on the implementation-specific artifact, and remove indirect dependency towards the empty modules with maven configurations like : org.openjfx javafx-controls ${javafx.platform} 16 org.openjfx * I?m adding the option currently used by JavaCPP: 5. There is one main/API module (javafx.graphics or javafx.graphics.api), any number of platform-specific modules (javafx.graphics.linux.aarch64, etc?), and an empty module (javafx.graphics.platform) similar to the current one in OpenJFX. There is a 1-1 coupling between modules and artifacts. Users declares maven dependencies towards the empty module but the module descriptor "requires" the API module. The build system takes care or populating the module path according to the value of a property (javafx.platform) that could be multi-valued. Compared to option 2, it allows for multi-platform builds, but users must take care of the javafx.platform property and the platform modules must be brought into the module graph by an --add-modules command line argument. The special value ALL-MODULE-PATH can be used to benefit from the path calculation of the build system. Slighlty OOT, I believe that the selected mechanisms should provide a way to ship the native libraries pre-extracted in a JLink image. There is no technical reason from the user point of view to slow down the first run with library extraction and to silently pollute the home directory with a never cleaned .javacpp or .openjfx directory, when the library could simply come pre-extracted in the image. That means that the artifact with the native libraries have Maven scope ?provided?, or that exists some stripped version of the artifact without the libraries if the artifact must also contain Java classes or other resources. I finally link another OOT but close issue that was faced by JavaCPP users trying to run their App within an OSGi framework and that shows how tricky it can be to enumerate the bundled native libraries and extract them to the file system in some not-mainstream cases, and that we do need some common standard solution : [5]. -- Herv? Guillemet [1] https://github.com/javafxports/openjdk-jfx/issues/387 [2] https://github.com/openjfx/javafx-maven-plugin/issues/91 [3] https://github.com/openjfx/javafx-maven-plugin/pull/92 [4] https://github.com/openjfx/javafx-maven-plugin/pull/109 [5] https://github.com/bytedeco/javacpp/pull/511 From kevin.rushforth at oracle.com Wed Sep 15 23:59:47 2021 From: kevin.rushforth at oracle.com (Kevin Rushforth) Date: Wed, 15 Sep 2021 16:59:47 -0700 Subject: Modules with platform specific parts In-Reply-To: References: Message-ID: <4312ea13-2faa-8e61-f673-234d73c2036a@oracle.com> Thanks for starting this discussion. One thing to point out is that for JavaFX we actually use two different approaches depending on whether you are talking about the maven artifacts or the downloadable artifacts. The maven artifacts, which is what Johan was primarily referring to, use option 4 as he mentioned. The modules are delivered as a set of modular jar files. Two things are worth noting. First, from an application POV there is only the javafx.graphics module; the tooling uses the dummy jar file, but ultimately chooses which of the platform-specific modular jars to load, each of which has the module name of "javafx.graphics". Second, there are no jmod files on maven central (although since we pack the native code in the jar files for the modular jars that go on maven central, it "sort of" works to point jlink at the platform-specific modular jars). The downloadable artifacts consist of an "sdk" bundle and a "jmods" bundle for each platform. Once downloaded and unzipped, the SDK contains a set of modular jars (along with the native libraries for that platform), while the jmods bundle contains a set of jmods. The modular jars in the SDK and the jmod files in the jmods bundle are all named simply "javafx.graphics", etc. So this is really option 1. If we had a blank sheet of paper (i.e., if we were defining the tools to work with modules as well as the modules themselves), option 1 with some standardized way of selecting the specific physical implementation for your platform would seem ideal to me. A library like JavaFX -- or any other library with platform-specific bits -- wouldn't have to split its modules into "api modules" and "implementation modules". This more closely matches what happens in the JDK itself for modules that are delivered with the platform. There is only a single "java.desktop" module regardless of platform, with different physical instances of that module for each platform (OS x arch). However, it seems that this option is running into an impedance mismatch with existing tools like maven, gradle, the various IDEs, etc. It seems like there ought to be a solution to this, but maybe there are just too many compatibility constraints with the existing tools. -- Kevin On 9/15/2021 1:45 AM, Johan Vos wrote: > Hi, > > There have been discussions in the past about how to deal with > platform-specific parts (java code, native code, resources) in modules. > There is no standard for this, and afaik no recommendation. In the OpenJFX > project, we upload jars with module info to maven central, and we have > plugins for maven and gradle to deal with them at compiletime and at > runtime. > > However, the lack of a standard recommendation forces us to change the > internal behavior every now and then. The latest change we made (removing > automatic module names from empty modules [1]) caused issues in the 17 > release, when developers compile JavaFX modular apps [2] > > Before we make many other changes, I would like to have a better view on > what would be the recommended approach, so here is my summary. Suggestions > are highly appreciated. > > Let's assume we have a component that contains some > platform-independent Java code, some platform-dependent Java code, and > some platform-dependent native code. To make the example concrete, > let's use the javafx.graphics "module" from the OpenJFX project, which > contains exactly that. > > There are a number of options, and before thinking about the best way > for tools do deal with this situation, it would be good to have a > recommended approach for those "hybrid" modules. > > 1. There is a single module (i.e. only one Module). All code, no > matter on what platform or in which layer, will report the same value > for class.getModule().getName(). This would lead to a single > "javafx.graphics" conceptual module, but there will be a number of > physical module files that are different (with different class files > and native code). Those different modules should obvisouly be mutually > exclusive in a runtime. > > 2. There are 2 modules: the platform-independent Java code goes into > one module (let's call that javafx.graphics.api) and a second module > is named javafx.graphics.platform and contains the platform-dependent > Java code and the native code. > In this approach, developers use the javafx.graphics.api module to > compile against, and at runtime the javafx.graphics.platform module is > required. Again, that second module will have a number of different > physical implementations. As an extension to this, we might add the > Service Provider Interface approach for loading platform-specific > modules/bits at runtime. > > 3. We create one module for each platform. There is no > "javafx.graphics" module in this case, but there is a > "javafx.graphics.linux.aarch64" module for example. > Doing so, there is a tight coupling between one conceptual module and > one physical module. A clear drawback of this is that this is a real > challenge at compiletime. Developers (who are only using generic > API's) need to compile against a platform-specific module. > > 4. We use 2 artifacts: an "empty" one and then a number of > implementation specific ones. The difference with option 2 is that the > empty "module" exists solely for the purpose of tools, which can > detect what implementation specific module(s) need to be loaded at > compile/runtime. > > We currently use option 4, but in my opinion, option 2 would be the > better approach. > > Thanks, > > - Johan > > [1] https://bugs.openjdk.java.net/browse/JDK-8264998 > [2] > https://mail.openjdk.java.net/pipermail/openjfx-dev/2021-September/031934.html From kevin.rushforth at oracle.com Thu Sep 16 00:10:03 2021 From: kevin.rushforth at oracle.com (Kevin Rushforth) Date: Wed, 15 Sep 2021 17:10:03 -0700 Subject: Modules with platform specific parts In-Reply-To: <9B6B055F-8CD1-45B6-A6CA-EE980DB74994@kleczek.org> References: <9B6B055F-8CD1-45B6-A6CA-EE980DB74994@kleczek.org> Message-ID: If we do eventually have to go with multiple, differently-named modules, this would seem a sensible approach. I think it's roughly what Johan was alluding to when he spoke of extending option 2 to use an SPI. In this case, we would refactor the platform-specific code into different named modules (that would be a fair bit of work, but let's ignore that for now), and the existing javafx.graphics module would become the API module. Somewhat related to this: Of the four options, I think option 3 is a non-starter. The application developer should never have to "requires", or even know about the existence of, platform-specific modules. This means that we either need to stick with different physical modules all of which are named the same, or else platform-specific modules that are loaded using some sort of SPI, so that they don't have to be known at compile / link time. -- Kevin On 9/15/2021 2:15 AM, Micha? K?eczek wrote: > Hi, > > Hi, > > There is another option: > > API module and multiple platform specific modules (different names like javafx.graphics.linux.aarch64) that require API module and provide services. > > API module does not ?require? any platform specific module but uses ServiceLoader to find a proper implementation at runtime. > > Thanks, > Michal > > >> On 15 Sep 2021, at 10:45, Johan Vos wrote: >> >> Hi, >> >> There have been discussions in the past about how to deal with >> platform-specific parts (java code, native code, resources) in modules. >> There is no standard for this, and afaik no recommendation. In the OpenJFX >> project, we upload jars with module info to maven central, and we have >> plugins for maven and gradle to deal with them at compiletime and at >> runtime. >> >> However, the lack of a standard recommendation forces us to change the >> internal behavior every now and then. The latest change we made (removing >> automatic module names from empty modules [1]) caused issues in the 17 >> release, when developers compile JavaFX modular apps [2] >> >> Before we make many other changes, I would like to have a better view on >> what would be the recommended approach, so here is my summary. Suggestions >> are highly appreciated. >> >> Let's assume we have a component that contains some >> platform-independent Java code, some platform-dependent Java code, and >> some platform-dependent native code. To make the example concrete, >> let's use the javafx.graphics "module" from the OpenJFX project, which >> contains exactly that. >> >> There are a number of options, and before thinking about the best way >> for tools do deal with this situation, it would be good to have a >> recommended approach for those "hybrid" modules. >> >> 1. There is a single module (i.e. only one Module). All code, no >> matter on what platform or in which layer, will report the same value >> for class.getModule().getName(). This would lead to a single >> "javafx.graphics" conceptual module, but there will be a number of >> physical module files that are different (with different class files >> and native code). Those different modules should obvisouly be mutually >> exclusive in a runtime. >> >> 2. There are 2 modules: the platform-independent Java code goes into >> one module (let's call that javafx.graphics.api) and a second module >> is named javafx.graphics.platform and contains the platform-dependent >> Java code and the native code. >> In this approach, developers use the javafx.graphics.api module to >> compile against, and at runtime the javafx.graphics.platform module is >> required. Again, that second module will have a number of different >> physical implementations. As an extension to this, we might add the >> Service Provider Interface approach for loading platform-specific >> modules/bits at runtime. >> >> 3. We create one module for each platform. There is no >> "javafx.graphics" module in this case, but there is a >> "javafx.graphics.linux.aarch64" module for example. >> Doing so, there is a tight coupling between one conceptual module and >> one physical module. A clear drawback of this is that this is a real >> challenge at compiletime. Developers (who are only using generic >> API's) need to compile against a platform-specific module. >> >> 4. We use 2 artifacts: an "empty" one and then a number of >> implementation specific ones. The difference with option 2 is that the >> empty "module" exists solely for the purpose of tools, which can >> detect what implementation specific module(s) need to be loaded at >> compile/runtime. >> >> We currently use option 4, but in my opinion, option 2 would be the >> better approach. >> >> Thanks, >> >> - Johan >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8264998 >> [2] >> https://mail.openjdk.java.net/pipermail/openjfx-dev/2021-September/031934.html From kevin.rushforth at oracle.com Thu Sep 16 00:13:19 2021 From: kevin.rushforth at oracle.com (Kevin Rushforth) Date: Wed, 15 Sep 2021 17:13:19 -0700 Subject: Modules with platform specific parts In-Reply-To: <86876ebf-64b4-f886-e10e-2bcf972ee12c@oracle.com> References: <86876ebf-64b4-f886-e10e-2bcf972ee12c@oracle.com> Message-ID: <8f837410-b515-b027-bab3-31a499605fe4@oracle.com> We do have jmods available for download, but not in maven central. I'm not well-versed enough with maven / gradle support for jmods to know whether it would even be an option, but even we do have them, we still need the modular jars for the cases where the developer doesn't end up running jlink. -- Kevin On 9/15/2021 6:55 AM, Alan Bateman wrote: > On 15/09/2021 09:45, Johan Vos wrote: >> Hi, >> >> There have been discussions in the past about how to deal with >> platform-specific parts (java code, native code, resources) in modules. >> There is no standard for this, and afaik no recommendation. In the >> OpenJFX >> project, we upload jars with module info to maven central, and we have >> plugins for maven and gradle to deal with them at compiletime and at >> runtime. >> > Project Panama and the foreign linker API may be another motivation to > re-visit this topic. One general concern is adding more complexity to > the JAR format. I could imagine platform specific sections adding > another dimension of complexity to MR JARs, modular JARs, modular MR > JARs, ...? Also every bag we nail on could have implications for > compile time, packaging time, jlink, and run-time. > > The position we took on this in JDK 9 is that modules can be platform > specific. There is a JDK-specific class file attribute named > "ModuleTarget" for the OS and/or architecture.? The jmod tool has an > option to specify the OS/arch when creating a JMOD. Post resolution > checks will catch issues where someone accidentally deploys a module > for the different OS/architecture. I have not looked at the JavaFX > modules recently but at some point I think they were packaged as JMOD > files and it might be that the platform was specified at that time. If > you've since moved to modular JARs then I could imagine this getting > lost. > > -Alan > From samuel.audet at gmail.com Thu Sep 16 00:54:57 2021 From: samuel.audet at gmail.com (Samuel Audet) Date: Thu, 16 Sep 2021 09:54:57 +0900 Subject: Modules with platform specific parts In-Reply-To: <8f837410-b515-b027-bab3-31a499605fe4@oracle.com> References: <86876ebf-64b4-f886-e10e-2bcf972ee12c@oracle.com> <8f837410-b515-b027-bab3-31a499605fe4@oracle.com> Message-ID: <356c9cd7-268f-7f0c-47bd-5ea434a8c637@gmail.com> We can publish JMODs, or any other binaries for that matter, on Maven repositories, for example, here's a random one I've just pulled out: https://search.maven.org/artifact/com.github.nullterminated/trylambda/1.4/jmod But why do we need to maintain 2 different sets of archives containing essentially the same thing? Android has been doing something similar: When native libraries are involved, we need to feed it AAR files instead of normal JAR files for end users to get all the benefits of the platform. However, we don't need to do that in the case of, for example, OSGi. With some additional metadata and a lot of hacking, we can get it to manage JAR files containing native libraries just fine, so it *is* possible. JAR is the common denominator, JMOD is not. If it wants to remain relevant, OpenJDK should really consider having a broader discussion about this. That is, rather than simply saying that everything is hopelessly broken, and that we have to throw everything out the window, let's maybe try to fix the "old and broken", because redoing everything from scratch just isn't going to happen. If the Java platform cannot maintain some semblance of backward compatibility, developers are going to be tempted to just abandon it and start new projects with something else. Please, please, do consider fixing the JDK instead of talking about coming up with incompatible "solutions"! Samuel On 9/16/21 9:13 AM, Kevin Rushforth wrote: > We do have jmods available for download, but not in maven central. I'm > not well-versed enough with maven / gradle support for jmods to know > whether it would even be an option, but even we do have them, we still > need the modular jars for the cases where the developer doesn't end up > running jlink. > > -- Kevin > > > On 9/15/2021 6:55 AM, Alan Bateman wrote: >> On 15/09/2021 09:45, Johan Vos wrote: >>> Hi, >>> >>> There have been discussions in the past about how to deal with >>> platform-specific parts (java code, native code, resources) in modules. >>> There is no standard for this, and afaik no recommendation. In the OpenJFX >>> project, we upload jars with module info to maven central, and we have >>> plugins for maven and gradle to deal with them at compiletime and at >>> runtime. >>> >> Project Panama and the foreign linker API may be another motivation to >> re-visit this topic. One general concern is adding more complexity to >> the JAR format. I could imagine platform specific sections adding >> another dimension of complexity to MR JARs, modular JARs, modular MR >> JARs, ...? Also every bag we nail on could have implications for >> compile time, packaging time, jlink, and run-time. >> >> The position we took on this in JDK 9 is that modules can be platform >> specific. There is a JDK-specific class file attribute named >> "ModuleTarget" for the OS and/or architecture.? The jmod tool has an >> option to specify the OS/arch when creating a JMOD. Post resolution >> checks will catch issues where someone accidentally deploys a module >> for the different OS/architecture. I have not looked at the JavaFX >> modules recently but at some point I think they were packaged as JMOD >> files and it might be that the platform was specified at that time. If >> you've since moved to modular JARs then I could imagine this getting >> lost. >> >> -Alan >> > From johan.vos at gluonhq.com Thu Sep 16 07:24:17 2021 From: johan.vos at gluonhq.com (Johan Vos) Date: Thu, 16 Sep 2021 09:24:17 +0200 Subject: Modules with platform specific parts In-Reply-To: <4312ea13-2faa-8e61-f673-234d73c2036a@oracle.com> References: <4312ea13-2faa-8e61-f673-234d73c2036a@oracle.com> Message-ID: Indeed, I didn't make it clear that we distribute JavaFX in 2 ways: the "standalone SDK" and the "maven artifacts". There are no issues with the standalone SDK approach, and I should have mentioned that. But apart for the core JDK modules, the SDK approach makes it much harder for developers to maintain dependencies. This is why we have tools like maven, to deal with changing dependencies instead of requiring developers to download, install and maintain SDK's for every dependency they have. I believe we want to make sure the current SDK approach is not compromised, while at the same time the "maven" approach is improved. Again, I'm not sure this requires changes in the OpenJDK, but at least recommendations would be useful. - Johan On Thu, Sep 16, 2021 at 2:00 AM Kevin Rushforth wrote: > Thanks for starting this discussion. > > One thing to point out is that for JavaFX we actually use two different > approaches depending on whether you are talking about the maven > artifacts or the downloadable artifacts. > > The maven artifacts, which is what Johan was primarily referring to, use > option 4 as he mentioned. The modules are delivered as a set of modular > jar files. Two things are worth noting. First, from an application POV > there is only the javafx.graphics module; the tooling uses the dummy jar > file, but ultimately chooses which of the platform-specific modular jars > to load, each of which has the module name of "javafx.graphics". Second, > there are no jmod files on maven central (although since we pack the > native code in the jar files for the modular jars that go on maven > central, it "sort of" works to point jlink at the platform-specific > modular jars). > > The downloadable artifacts consist of an "sdk" bundle and a "jmods" > bundle for each platform. Once downloaded and unzipped, the SDK contains > a set of modular jars (along with the native libraries for that > platform), while the jmods bundle contains a set of jmods. The modular > jars in the SDK and the jmod files in the jmods bundle are all named > simply "javafx.graphics", etc. So this is really option 1. > > If we had a blank sheet of paper (i.e., if we were defining the tools to > work with modules as well as the modules themselves), option 1 with some > standardized way of selecting the specific physical implementation for > your platform would seem ideal to me. A library like JavaFX -- or any > other library with platform-specific bits -- wouldn't have to split its > modules into "api modules" and "implementation modules". This more > closely matches what happens in the JDK itself for modules that are > delivered with the platform. There is only a single "java.desktop" > module regardless of platform, with different physical instances of that > module for each platform (OS x arch). > > However, it seems that this option is running into an impedance mismatch > with existing tools like maven, gradle, the various IDEs, etc. It seems > like there ought to be a solution to this, but maybe there are just too > many compatibility constraints with the existing tools. > > -- Kevin > > > On 9/15/2021 1:45 AM, Johan Vos wrote: > > Hi, > > > > There have been discussions in the past about how to deal with > > platform-specific parts (java code, native code, resources) in modules. > > There is no standard for this, and afaik no recommendation. In the > OpenJFX > > project, we upload jars with module info to maven central, and we have > > plugins for maven and gradle to deal with them at compiletime and at > > runtime. > > > > However, the lack of a standard recommendation forces us to change the > > internal behavior every now and then. The latest change we made (removing > > automatic module names from empty modules [1]) caused issues in the 17 > > release, when developers compile JavaFX modular apps [2] > > > > Before we make many other changes, I would like to have a better view on > > what would be the recommended approach, so here is my summary. > Suggestions > > are highly appreciated. > > > > Let's assume we have a component that contains some > > platform-independent Java code, some platform-dependent Java code, and > > some platform-dependent native code. To make the example concrete, > > let's use the javafx.graphics "module" from the OpenJFX project, which > > contains exactly that. > > > > There are a number of options, and before thinking about the best way > > for tools do deal with this situation, it would be good to have a > > recommended approach for those "hybrid" modules. > > > > 1. There is a single module (i.e. only one Module). All code, no > > matter on what platform or in which layer, will report the same value > > for class.getModule().getName(). This would lead to a single > > "javafx.graphics" conceptual module, but there will be a number of > > physical module files that are different (with different class files > > and native code). Those different modules should obvisouly be mutually > > exclusive in a runtime. > > > > 2. There are 2 modules: the platform-independent Java code goes into > > one module (let's call that javafx.graphics.api) and a second module > > is named javafx.graphics.platform and contains the platform-dependent > > Java code and the native code. > > In this approach, developers use the javafx.graphics.api module to > > compile against, and at runtime the javafx.graphics.platform module is > > required. Again, that second module will have a number of different > > physical implementations. As an extension to this, we might add the > > Service Provider Interface approach for loading platform-specific > > modules/bits at runtime. > > > > 3. We create one module for each platform. There is no > > "javafx.graphics" module in this case, but there is a > > "javafx.graphics.linux.aarch64" module for example. > > Doing so, there is a tight coupling between one conceptual module and > > one physical module. A clear drawback of this is that this is a real > > challenge at compiletime. Developers (who are only using generic > > API's) need to compile against a platform-specific module. > > > > 4. We use 2 artifacts: an "empty" one and then a number of > > implementation specific ones. The difference with option 2 is that the > > empty "module" exists solely for the purpose of tools, which can > > detect what implementation specific module(s) need to be loaded at > > compile/runtime. > > > > We currently use option 4, but in my opinion, option 2 would be the > > better approach. > > > > Thanks, > > > > - Johan > > > > [1] https://bugs.openjdk.java.net/browse/JDK-8264998 > > [2] > > > https://mail.openjdk.java.net/pipermail/openjfx-dev/2021-September/031934.html > > From johan.vos at gluonhq.com Thu Sep 16 07:33:11 2021 From: johan.vos at gluonhq.com (Johan Vos) Date: Thu, 16 Sep 2021 09:33:11 +0200 Subject: Modules with platform specific parts In-Reply-To: References: <86876ebf-64b4-f886-e10e-2bcf972ee12c@oracle.com> Message-ID: Right, I didn't even mention multi-platform builds. That is a very valid concern, and it makes the situation a bit more complex. We currently have different jar files for every different platform/architecture combination. There is an exception though: we have 1 build for all iOS combinations: {32/64 bit, intel/arm}. One of the reasons for this is that market stores often require a multi-architecture build to be uploaded, so that the same applications work on all their supported archs. We currently don't do this yet for MacOS, where we have different jars for AMD64 and AARCH64. Maven uses the concept "classifier" for this. But that is just an unstructured string. We might need is a more structured approach in which a jar (or a jmod?) can declare one or more classifiers that it provides. It's then up to the tooling to wire things together, but at the very least, tools should be able to understand what the modules are providing (and not only which packages). - Johan On Wed, Sep 15, 2021 at 6:10 PM Herv? Guillemet wrote: > Hi, > > Thank you Johan for starting this discussion. The current situation > is indeed problematic and a (de facto or not) standard should emerge. > > IMO, beyond being easy to use, the selected mechanism should allow to > build multi-platform images (for instance for MacOSX on both aarch64 > and x86_64). > > > 1. There is a single module (i.e. only one Module). All code, no > > matter on what platform or in which layer, will report the same value > > for class.getModule().getName(). This would lead to a single > > "javafx.graphics" conceptual module, but there will be a number of > > physical module files that are different (with different class files > > and native code). Those different modules should obvisouly be mutually > > exclusive in a runtime. > > This option doesn?t allow multi-platform images. Also the duplication of > all common Java classes in each platform jars is not ideal. > > > 2. There are 2 modules: the platform-independent Java code goes into > > one module (let's call that javafx.graphics.api) and a second module > > is named javafx.graphics.platform and contains the platform-dependent > > Java code and the native code. > > In this approach, developers use the javafx.graphics.api module to > > compile against, and at runtime the javafx.graphics.platform module is > > required. Again, that second module will have a number of different > > physical implementations. As an extension to this, we might add the > > Service Provider Interface approach for loading platform-specific > > modules/bits at runtime. > > This option doesn?t allow multi-platform images. > > > 3. We create one module for each platform. There is no > > "javafx.graphics" module in this case, but there is a > > "javafx.graphics.linux.aarch64" module for example. > > Doing so, there is a tight coupling between one conceptual module and > > one physical module. A clear drawback of this is that this is a real > > challenge at compiletime. Developers (who are only using generic > > API's) need to compile against a platform-specific module. > > Indeed. The building of the module path can be automatized with > Maven on another build system, but the module descriptor should also > be changed for each platform you are building for. > Multi-platform images won?t work either since packages would be split, > and common classes are also duplicated. I don?t see any advantage compared > to option 1. > > > 4. We use 2 artifacts: an "empty" one and then a number of > > implementation specific ones. The difference with option 2 is that the > > empty "module" exists solely for the purpose of tools, which can > > detect what implementation specific module(s) need to be loaded at > > compile/runtime. > > Like mentioned in your referenced links, this causes problems. > I haven?t worked with OpenJFX 17 but with OpenJFX 11-16, and to be > able to build JLink images I had either to use the JavaFX maven > plugin (which takes care of empty modules but one may wish to use > alternate plugins because of some limitations [1-4]) or explicitly > depend directly on the implementation-specific artifact, and remove > indirect dependency towards the empty modules with maven configurations > like : > > > org.openjfx > javafx-controls > ${javafx.platform} > 16 > > > org.openjfx > * > > > > > > I?m adding the option currently used by JavaCPP: > > 5. There is one main/API module (javafx.graphics or > javafx.graphics.api), any number of platform-specific modules > (javafx.graphics.linux.aarch64, etc?), and an empty module > (javafx.graphics.platform) similar to the current one in OpenJFX. > There is a 1-1 coupling between modules and artifacts. > Users declares maven dependencies towards the empty module but the > module descriptor "requires" the API module. The build system takes > care or populating the module path according to the value of a > property (javafx.platform) that could be multi-valued. > > Compared to option 2, it allows for multi-platform builds, but > users must take care of the javafx.platform property and the platform > modules must be brought into the module graph by an --add-modules > command line argument. > The special value ALL-MODULE-PATH can be used to benefit from the path > calculation of the build system. > > Slighlty OOT, I believe that the selected mechanisms should provide a > way to ship the native libraries pre-extracted in a JLink image. There > is no technical reason from the user point of view to slow down the > first run with library extraction and to silently pollute the home > directory with a never cleaned .javacpp or .openjfx directory, when the > library could simply come pre-extracted in the image. That means that > the artifact with the native libraries have Maven scope ?provided?, or > that exists some stripped version of the artifact without the libraries > if the artifact must also contain Java classes or other resources. > > I finally link another OOT but close issue that was faced by JavaCPP > users trying to run their App within an OSGi framework and that shows > how tricky it can be to enumerate the bundled native libraries and > extract them to the file system in some not-mainstream cases, and that > we do need some common standard solution : [5]. > > -- > Herv? Guillemet > > > [1] https://github.com/javafxports/openjdk-jfx/issues/387 > [2] https://github.com/openjfx/javafx-maven-plugin/issues/91 > [3] https://github.com/openjfx/javafx-maven-plugin/pull/92 > [4] https://github.com/openjfx/javafx-maven-plugin/pull/109 > [5] https://github.com/bytedeco/javacpp/pull/511 > > From johan.vos at gluonhq.com Thu Sep 16 07:35:27 2021 From: johan.vos at gluonhq.com (Johan Vos) Date: Thu, 16 Sep 2021 09:35:27 +0200 Subject: Modules with platform specific parts In-Reply-To: <356c9cd7-268f-7f0c-47bd-5ea434a8c637@gmail.com> References: <86876ebf-64b4-f886-e10e-2bcf972ee12c@oracle.com> <8f837410-b515-b027-bab3-31a499605fe4@oracle.com> <356c9cd7-268f-7f0c-47bd-5ea434a8c637@gmail.com> Message-ID: On Thu, Sep 16, 2021 at 2:55 AM Samuel Audet wrote: > > If it wants to remain relevant, OpenJDK should really consider having a > broader discussion about this. > ... > Please, please, do consider fixing the JDK instead of talking about > coming up with incompatible "solutions"! > I totally agree, but I believe this is exactly what we are doing now? - Johan From mike at plan99.net Thu Sep 16 08:58:37 2021 From: mike at plan99.net (Mike Hearn) Date: Thu, 16 Sep 2021 10:58:37 +0200 Subject: Modules with platform specific parts In-Reply-To: References: <86876ebf-64b4-f886-e10e-2bcf972ee12c@oracle.com> <8f837410-b515-b027-bab3-31a499605fe4@oracle.com> <356c9cd7-268f-7f0c-47bd-5ea434a8c637@gmail.com> Message-ID: A simple approach would be to teach the app classloader to read JMODs and extract native libraries on the fly, and to preferentially check a certain sub-directory for classes before others (for OS-specific class files). The original Jigsaw JEPs put that out of scope but it's what people make their JARs do anyway. Revisiting and formalizing that would allow existing tools to simply treat jmods as modular jars, which is a very minimal change to their code, and then jlink can continue pre-extracting libraries for startup time improvements, signing etc. From samuel.audet at gmail.com Fri Sep 17 10:51:52 2021 From: samuel.audet at gmail.com (Samuel Audet) Date: Fri, 17 Sep 2021 19:51:52 +0900 Subject: Modules with platform specific parts In-Reply-To: References: <86876ebf-64b4-f886-e10e-2bcf972ee12c@oracle.com> <8f837410-b515-b027-bab3-31a499605fe4@oracle.com> <356c9cd7-268f-7f0c-47bd-5ea434a8c637@gmail.com> Message-ID: I certainly hope so! But I don't see anyone, for example, from Google representing Android, so what do we hope to accomplish, exactly? Let's start by making the goals clear. Samuel On Thu, Sep 16, 2021, 16:35 Johan Vos wrote: > > > On Thu, Sep 16, 2021 at 2:55 AM Samuel Audet > wrote: > >> >> If it wants to remain relevant, OpenJDK should really consider having a >> broader discussion about this. >> > ... > >> Please, please, do consider fixing the JDK instead of talking about >> coming up with incompatible "solutions"! >> > > I totally agree, but I believe this is exactly what we are doing now? > > - Johan > From craigraw at gmail.com Mon Sep 20 06:52:13 2021 From: craigraw at gmail.com (Craig Raw) Date: Mon, 20 Sep 2021 08:52:13 +0200 Subject: jlink SystemModulesPlugin use of hashCode breaks reproducibility Message-ID: Hi, While I've noticed significant improvements in making reproducible builds with JDK 16, there is a remaining issue I'm stumped by. The issue is this: Using jlink invokes the SystemModulesPlugin, which creates .class files containing module descriptors for all the modules of the application. Specifically, these .class files are SystemModules$all.class and SystemModules$default.class. The class files create instances of jdk.internal.module.Builder to create the ModuleDescriptor objects. The ModuleDescriptors are created by use of the Builder.build(int hashCode) method. The problem is this: the value of the hashCode parameter can change when building across different systems, changing the bytecode of the SystemModules$*.class files, and therefore the modules file created by jlink. It is unclear to me why the hashCode should differ. The SystemModulesPlugin invokes ModuleDescriptor.hashCode() on line 1116, and that method should be deterministic. However, I do note that the value is cached inside ModuleDescriptor, and in fact a ModuleDescriptor can be constructed with a provided hashCode value. Perhaps an outdated value is cached? My other consideration is that integer overflow is behaving differently on some systems, but I am targeting 64 bit systems only, and this issue is occurring across similar Debian-based platforms running identical JDK versions (adoptopenjdk-16-hotspot=16.0.1+9). The issue is illustrated with an example here: https://github.com/sparrowwallet/sparrow/issues/197 Note that this issue is intermittent - many users are able to create reproducible builds of this project, but some are not. For reference, the steps to creating a reproducible build are documented here (note only the 1.5.0-beta1 release is reproducible): https://github.com/sparrowwallet/sparrow/blob/master/docs/reproducible.md Any assistance in resolving this is appreciated. Regards, Craig From robin.bygrave at gmail.com Mon Sep 20 08:55:04 2021 From: robin.bygrave at gmail.com (Rob Bygrave) Date: Mon, 20 Sep 2021 20:55:04 +1200 Subject: module resource loading - opens {package} when {package} only holds resources Message-ID: Hi, I have a case where I believe I have a module where I would like to use: opens {package}; ... *where {package} only contains resources*, there are no .class files in {package}. Currently when {package} only contains resources, we get an InvalidModuleDescriptorException error: Error occurred during initialization of boot layer java.lang.module.FindException: Error reading module: /home/rob/github/avaje/test-modules/target/classes Caused by: java.lang.module.InvalidModuleDescriptorException: Package dbmigration not found in module module test.modules { ... opens dbmigration; } *Background:* I have a library that runs database sql scripts ("database migrations"). A module will have a resources directory (typically called dbmigration) that contains resources (sql scripts). The library loads the sql script resources and executes them when required. *Options that do work:* A) Open the entire module B) Add a dummy java class into the dbmigration package which then allows us to specify *opens dbmigration; *in the module-info C) Change the resource path from dbmigration to for example db-migration - change the resource path to any non-valid package name, as Alexey Gavrilov pointed out this is equivalent to opening it to all other modules. D) Put extra code (that is otherwise unnecessary) into the same module as the resources to load them and make them available to the library. *Question* Is there any chance that we can use opens for a resource path, when there is no package per say? Can we change to allows opens for resource only paths? If we could, then this to me would seem like the nicest option/approach for this case. Thanks, Rob. From Alan.Bateman at oracle.com Mon Sep 20 09:21:53 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 20 Sep 2021 10:21:53 +0100 Subject: module resource loading - opens {package} when {package} only holds resources In-Reply-To: References: Message-ID: <7b655246-31f2-cc83-41b9-8cecfa9d8c83@oracle.com> On 20/09/2021 09:55, Rob Bygrave wrote: > Hi, > I have a case where I believe I have a module where I would like to use: > > opens {package}; > > ... *where {package} only contains resources*, there are no .class files in > {package}. Currently when {package} only contains resources, we get an > InvalidModuleDescriptorException error: > > Error occurred during initialization of boot layer > java.lang.module.FindException: Error reading module: > /home/rob/github/avaje/test-modules/target/classes > Caused by: java.lang.module.InvalidModuleDescriptorException: Package > dbmigration not found in module > > module test.modules { > ... > opens dbmigration; > } You can open a package that only contains resources. The directory structure above hints that this is a Maven project, it is possible that the "dbmigration" resources have not been copied into target/classes? -Alan From Alan.Bateman at oracle.com Mon Sep 20 09:54:37 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 20 Sep 2021 10:54:37 +0100 Subject: jlink SystemModulesPlugin use of hashCode breaks reproducibility In-Reply-To: References: Message-ID: <7fd1ac15-360d-c4b1-29f2-61daf407d49c@oracle.com> On 20/09/2021 07:52, Craig Raw wrote: > Hi, > > While I've noticed significant improvements in making reproducible builds > with JDK 16, there is a remaining issue I'm stumped by. The issue is this: > > Using jlink invokes the SystemModulesPlugin, which creates .class files > containing module descriptors for all the modules of the application. > Specifically, these .class files are SystemModules$all.class and > SystemModules$default.class. The class files create instances of > jdk.internal.module.Builder to create the ModuleDescriptor objects. The > ModuleDescriptors are created by use of the Builder.build(int hashCode) > method. The problem is this: the value of the hashCode parameter can change > when building across different systems, changing the bytecode of the > SystemModules$*.class files, and therefore the modules file created by > jlink. > > It is unclear to me why the hashCode should differ. The SystemModulesPlugin > invokes ModuleDescriptor.hashCode() on line 1116, and that method should be > deterministic. However, I do note that the value is cached inside > ModuleDescriptor, and in fact a ModuleDescriptor can be constructed with a > provided hashCode value. Perhaps an outdated value is cached? My other > consideration is that integer overflow is behaving differently on some > systems, but I am targeting 64 bit systems only, and this issue is > occurring across similar Debian-based platforms running identical JDK > versions (adoptopenjdk-16-hotspot=16.0.1+9). > > The issue is illustrated with an example here: > https://github.com/sparrowwallet/sparrow/issues/197 > > Note that this issue is intermittent - many users are able to create > reproducible builds of this project, but some are not. For reference, the > steps to creating a reproducible build are documented here (note only the > 1.5.0-beta1 release is reproducible): > https://github.com/sparrowwallet/sparrow/blob/master/docs/reproducible.md > > Any assistance in resolving this is appreciated. We fixed a lots of issues with reproducible builds in recent releases, including several issues related to iteration order that were problematic for the jlink plugins. Off hand I can't think of any issues with the ModuleDescriptor hashCode. It is computed at link time and should be deterministic. If I were to guess then then this may be something to do with the module version recorded at compile-time at that is one of the components that the hash is based on. That's just a guess, I think I would need to see the module path specified to jlink to have a better idea. BTW: Have you had any reports with JDK 17 yet? -Alan From robin.bygrave at gmail.com Mon Sep 20 09:58:47 2021 From: robin.bygrave at gmail.com (Rob Bygrave) Date: Mon, 20 Sep 2021 21:58:47 +1200 Subject: module resource loading - opens {package} when {package} only holds resources In-Reply-To: <7b655246-31f2-cc83-41b9-8cecfa9d8c83@oracle.com> References: <7b655246-31f2-cc83-41b9-8cecfa9d8c83@oracle.com> Message-ID: *> "dbmigration" resources have not been copied into target/classes?* target/classes/dbmigration exists *BUT* ... now I noticed that dbmigration only contained other sub-directories (which then contain the resources). So putting an empty junk.txt file into dbmigration fixes the problem. So if the resource path only contains subdirectories then we get the InvalidModuleDescriptorException, to open resource paths they need to contain at least one file "at the top level". Hmmm, I'll have to ponder that - it seems pretty subtle. target/classes/dbmigration ??? postgres *(directory)* ? ??? 1.0_initial.sql ? ??? idx_postgres.migrations ??? mysql *(directory)* ? ??? 1.0_initial.sql ? ??? idx_mysql.migrations ??? junk.txt *